2024-09-17T18:08:53.081 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-17T18:08:53.098 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-17T18:08:53.183 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909241 branch: wip-bharath10-testing-2024-09-17-1348 description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{centos_latest} thrashers/force-sync-many workloads/rados_api_tests} email: skanta@redhat.com first_in_suite: false job_id: '7909241' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-bharath10-testing-2024-09-17-1348 ceph: conf: client: debug monc: 20 debug ms: 1 global: debug ms: 1 debug objecter: 20 debug rados: 20 mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr1: false ms bind msgr2: true ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: mon ms inject internal delays: 0.002 ms inject socket failures: 2500 ms type: async mgr: debug mgr: 20 debug monc: 10 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 5 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore write v2: false bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(TOO_FEW_PGS\) - \(POOL_APP_NOT_ENABLED\) - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_FULL\) - \(SLOW_OPS\) - \(MON_DOWN\) - \(PG_ - \(SMALLER_PGP_NUM\) - slow request sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-bharath10-testing-2024-09-17-1348 sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 3421 sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath10-testing-2024-09-17-1348 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 targets: smithi121.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKY1CQNsro7aKTRL/8UMG2MgcryOjAsnegIPQyDkS9cHfqpi5LCbliE0naFgvoSCiZKh01CDd8RnPgMaDKnsLx4= smithi173.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOlQCiEh3GppOaCSXMZmC2+fxY2g/LhDplZ5JvlQ+2GJ0nn86Xwb0ff5DeaUoMeEw+s+IC9zfAHIo3cgwan8J+g= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 90 store_thrash: true thrash_delay: 1 thrash_many: true - workunit: clients: client.0: - rados/test.sh unit_test_scan teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-17_15:01:33 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-17T18:08:53.183 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa; will attempt to use it 2024-09-17T18:08:53.184 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa/tasks 2024-09-17T18:08:53.184 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-17T18:08:53.186 INFO:teuthology.task.internal:Checking packages... 2024-09-17T18:08:53.209 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '1889a165b5128787ec693450a3d1c5df5d6dc203' 2024-09-17T18:08:53.209 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-17T18:08:53.209 INFO:teuthology.packaging:ref: None 2024-09-17T18:08:53.209 INFO:teuthology.packaging:tag: None 2024-09-17T18:08:53.209 INFO:teuthology.packaging:branch: wip-bharath10-testing-2024-09-17-1348 2024-09-17T18:08:53.209 INFO:teuthology.packaging:sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T18:08:53.209 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-bharath10-testing-2024-09-17-1348 2024-09-17T18:08:53.352 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4975.g1889a165 2024-09-17T18:08:53.353 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-17T18:08:53.360 INFO:teuthology.task.internal:no buildpackages task found 2024-09-17T18:08:53.360 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-17T18:08:53.380 INFO:teuthology.task.internal:Saving configuration 2024-09-17T18:08:53.395 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-17T18:08:53.402 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-17T18:08:53.427 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909241', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-17 18:03:54.949041', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKY1CQNsro7aKTRL/8UMG2MgcryOjAsnegIPQyDkS9cHfqpi5LCbliE0naFgvoSCiZKh01CDd8RnPgMaDKnsLx4='} 2024-09-17T18:08:53.450 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909241', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-17 18:03:54.950918', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOlQCiEh3GppOaCSXMZmC2+fxY2g/LhDplZ5JvlQ+2GJ0nn86Xwb0ff5DeaUoMeEw+s+IC9zfAHIo3cgwan8J+g='} 2024-09-17T18:08:53.450 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-17T18:08:53.456 INFO:teuthology.task.internal:roles: ubuntu@smithi121.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2024-09-17T18:08:53.456 INFO:teuthology.task.internal:roles: ubuntu@smithi173.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2024-09-17T18:08:53.456 INFO:teuthology.run_tasks:Running task console_log... 2024-09-17T18:08:53.529 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3ad41dfbe0>, signals=[15]) 2024-09-17T18:08:53.529 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-17T18:08:53.536 INFO:teuthology.task.internal:Opening connections... 2024-09-17T18:08:53.536 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi121.front.sepia.ceph.com 2024-09-17T18:08:53.537 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T18:08:53.615 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi173.front.sepia.ceph.com 2024-09-17T18:08:53.616 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T18:08:53.692 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-17T18:08:53.698 DEBUG:teuthology.orchestra.run.smithi121:> uname -m 2024-09-17T18:08:53.716 INFO:teuthology.orchestra.run.smithi121.stdout:x86_64 2024-09-17T18:08:53.717 DEBUG:teuthology.orchestra.run.smithi121:> cat /etc/os-release 2024-09-17T18:08:53.774 INFO:teuthology.orchestra.run.smithi121.stdout:NAME="CentOS Stream" 2024-09-17T18:08:53.774 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION="9" 2024-09-17T18:08:53.775 INFO:teuthology.orchestra.run.smithi121.stdout:ID="centos" 2024-09-17T18:08:53.775 INFO:teuthology.orchestra.run.smithi121.stdout:ID_LIKE="rhel fedora" 2024-09-17T18:08:53.775 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_ID="9" 2024-09-17T18:08:53.775 INFO:teuthology.orchestra.run.smithi121.stdout:PLATFORM_ID="platform:el9" 2024-09-17T18:08:53.775 INFO:teuthology.orchestra.run.smithi121.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-17T18:08:53.775 INFO:teuthology.orchestra.run.smithi121.stdout:ANSI_COLOR="0;31" 2024-09-17T18:08:53.775 INFO:teuthology.orchestra.run.smithi121.stdout:LOGO="fedora-logo-icon" 2024-09-17T18:08:53.775 INFO:teuthology.orchestra.run.smithi121.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-17T18:08:53.775 INFO:teuthology.orchestra.run.smithi121.stdout:HOME_URL="https://centos.org/" 2024-09-17T18:08:53.775 INFO:teuthology.orchestra.run.smithi121.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-17T18:08:53.775 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-17T18:08:53.775 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-17T18:08:53.776 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server 2024-09-17T18:08:53.797 DEBUG:teuthology.orchestra.run.smithi173:> uname -m 2024-09-17T18:08:53.816 INFO:teuthology.orchestra.run.smithi173.stdout:x86_64 2024-09-17T18:08:53.816 DEBUG:teuthology.orchestra.run.smithi173:> cat /etc/os-release 2024-09-17T18:08:53.875 INFO:teuthology.orchestra.run.smithi173.stdout:NAME="CentOS Stream" 2024-09-17T18:08:53.875 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION="9" 2024-09-17T18:08:53.875 INFO:teuthology.orchestra.run.smithi173.stdout:ID="centos" 2024-09-17T18:08:53.875 INFO:teuthology.orchestra.run.smithi173.stdout:ID_LIKE="rhel fedora" 2024-09-17T18:08:53.875 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION_ID="9" 2024-09-17T18:08:53.875 INFO:teuthology.orchestra.run.smithi173.stdout:PLATFORM_ID="platform:el9" 2024-09-17T18:08:53.875 INFO:teuthology.orchestra.run.smithi173.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-17T18:08:53.876 INFO:teuthology.orchestra.run.smithi173.stdout:ANSI_COLOR="0;31" 2024-09-17T18:08:53.876 INFO:teuthology.orchestra.run.smithi173.stdout:LOGO="fedora-logo-icon" 2024-09-17T18:08:53.876 INFO:teuthology.orchestra.run.smithi173.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-17T18:08:53.876 INFO:teuthology.orchestra.run.smithi173.stdout:HOME_URL="https://centos.org/" 2024-09-17T18:08:53.876 INFO:teuthology.orchestra.run.smithi173.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-17T18:08:53.876 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-17T18:08:53.876 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-17T18:08:53.876 INFO:teuthology.lock.ops:Updating smithi173.front.sepia.ceph.com on lock server 2024-09-17T18:08:53.897 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-17T18:08:53.910 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-17T18:08:53.916 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-17T18:08:53.916 DEBUG:teuthology.orchestra.run.smithi121:> test '!' -e /home/ubuntu/cephtest 2024-09-17T18:08:53.918 DEBUG:teuthology.orchestra.run.smithi173:> test '!' -e /home/ubuntu/cephtest 2024-09-17T18:08:53.935 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-17T18:08:53.941 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-17T18:08:53.941 DEBUG:teuthology.orchestra.run.smithi121:> test -z $(ls -A /var/lib/ceph) 2024-09-17T18:08:53.976 DEBUG:teuthology.orchestra.run.smithi173:> test -z $(ls -A /var/lib/ceph) 2024-09-17T18:08:54.084 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-17T18:08:54.139 INFO:teuthology.run_tasks:Running task kernel... 2024-09-17T18:08:54.153 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-17T18:08:54.153 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-17T18:08:54.153 DEBUG:teuthology.orchestra.run.smithi121:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-17T18:08:54.154 DEBUG:teuthology.orchestra.run.smithi173:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-17T18:08:54.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T18:08:54.171 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2024-09-17T18:08:54.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T18:08:54.172 DEBUG:teuthology.orchestra.run.smithi173:> uname -r 2024-09-17T18:08:54.229 INFO:teuthology.orchestra.run.smithi121.stdout:5.14.0-503.el9.x86_64 2024-09-17T18:08:54.230 INFO:teuthology.task.kernel:Running kernel on smithi121: 5.14.0-503.el9.x86_64 2024-09-17T18:08:54.230 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum install -y kernel 2024-09-17T18:08:54.231 INFO:teuthology.orchestra.run.smithi173.stdout:5.14.0-503.el9.x86_64 2024-09-17T18:08:54.232 INFO:teuthology.task.kernel:Running kernel on smithi173: 5.14.0-503.el9.x86_64 2024-09-17T18:08:54.232 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum install -y kernel 2024-09-17T18:08:57.727 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - BaseOS 4.9 MB/s | 8.3 MB 00:01 2024-09-17T18:08:57.830 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 8.3 MB 00:01 2024-09-17T18:08:59.403 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - AppStream 33 MB/s | 20 MB 00:00 2024-09-17T18:09:00.148 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - AppStream 16 MB/s | 20 MB 00:01 2024-09-17T18:09:03.729 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.5 MB 00:00 2024-09-17T18:09:05.155 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - Extras packages 94 kB/s | 19 kB 00:00 2024-09-17T18:09:06.225 INFO:teuthology.orchestra.run.smithi173.stdout:Extra Packages for Enterprise Linux 24 MB/s | 23 MB 00:00 2024-09-17T18:09:06.431 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - CRB 2.6 MB/s | 6.5 MB 00:02 2024-09-17T18:09:08.181 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 19 kB 00:00 2024-09-17T18:09:09.078 INFO:teuthology.orchestra.run.smithi121.stdout:Extra Packages for Enterprise Linux 29 MB/s | 23 MB 00:00 2024-09-17T18:09:11.606 INFO:teuthology.orchestra.run.smithi173.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-17T18:09:13.521 INFO:teuthology.orchestra.run.smithi173.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-17T18:09:13.521 INFO:teuthology.orchestra.run.smithi173.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-17T18:09:13.606 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T18:09:13.616 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:09:13.616 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-09-17T18:09:13.616 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:09:13.616 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T18:09:13.617 INFO:teuthology.orchestra.run.smithi173.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-17T18:09:13.617 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-09-17T18:09:13.617 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-17T18:09:13.617 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-17T18:09:13.617 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-17T18:09:13.617 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:09:13.617 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T18:09:13.617 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:09:13.617 INFO:teuthology.orchestra.run.smithi173.stdout:Install 4 Packages 2024-09-17T18:09:13.617 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:09:13.618 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 79 M 2024-09-17T18:09:13.618 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 126 M 2024-09-17T18:09:13.618 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T18:09:14.008 INFO:teuthology.orchestra.run.smithi173.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 286 kB/s | 36 kB 00:00 2024-09-17T18:09:14.533 INFO:teuthology.orchestra.run.smithi121.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-17T18:09:16.431 INFO:teuthology.orchestra.run.smithi121.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-17T18:09:16.431 INFO:teuthology.orchestra.run.smithi121.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-17T18:09:16.515 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-17T18:09:16.525 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:09:16.525 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-09-17T18:09:16.525 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:09:16.526 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-17T18:09:16.526 INFO:teuthology.orchestra.run.smithi121.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-17T18:09:16.526 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-09-17T18:09:16.526 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-17T18:09:16.526 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-17T18:09:16.526 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-17T18:09:16.526 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:09:16.526 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-17T18:09:16.526 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:09:16.526 INFO:teuthology.orchestra.run.smithi121.stdout:Install 4 Packages 2024-09-17T18:09:16.526 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:09:16.527 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 79 M 2024-09-17T18:09:16.527 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 126 M 2024-09-17T18:09:16.527 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-17T18:09:16.643 INFO:teuthology.orchestra.run.smithi121.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 388 kB/s | 36 kB 00:00 2024-09-17T18:09:17.118 INFO:teuthology.orchestra.run.smithi121.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 28 MB/s | 16 MB 00:00 2024-09-17T18:09:17.299 INFO:teuthology.orchestra.run.smithi173.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 4.6 MB/s | 16 MB 00:03 2024-09-17T18:09:17.501 INFO:teuthology.orchestra.run.smithi121.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 33 MB/s | 29 MB 00:00 2024-09-17T18:09:17.827 INFO:teuthology.orchestra.run.smithi121.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 27 MB/s | 35 MB 00:01 2024-09-17T18:09:17.827 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-17T18:09:17.827 INFO:teuthology.orchestra.run.smithi121.stdout:Total 61 MB/s | 79 MB 00:01 2024-09-17T18:09:18.518 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-17T18:09:18.864 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-17T18:09:18.864 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-17T18:09:19.822 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-17T18:09:19.822 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-17T18:09:20.007 INFO:teuthology.orchestra.run.smithi173.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 4.8 MB/s | 29 MB 00:05 2024-09-17T18:09:20.324 INFO:teuthology.orchestra.run.smithi173.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 5.4 MB/s | 35 MB 00:06 2024-09-17T18:09:20.324 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T18:09:20.324 INFO:teuthology.orchestra.run.smithi173.stdout:Total 12 MB/s | 79 MB 00:06 2024-09-17T18:09:21.010 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T18:09:21.372 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T18:09:21.373 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T18:09:22.275 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-17T18:09:22.361 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T18:09:22.361 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T18:09:22.900 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-17T18:09:22.985 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T18:09:24.783 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T18:09:25.459 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-17T18:09:25.541 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T18:09:25.743 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T18:09:25.873 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T18:09:28.304 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T18:09:28.438 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T18:09:30.683 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T18:09:30.714 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T18:09:33.251 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T18:09:33.289 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T18:09:35.544 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T18:09:38.332 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T18:10:13.780 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T18:10:13.820 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-17T18:10:14.121 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T18:10:14.121 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-17T18:10:14.121 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T18:10:14.121 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T18:10:14.960 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T18:10:14.961 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:10:14.961 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-17T18:10:14.961 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-17T18:10:14.961 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-17T18:10:14.961 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-17T18:10:14.961 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-17T18:10:14.961 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:10:14.961 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-17T18:10:15.335 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T18:10:15.420 INFO:teuthology.orchestra.run.smithi121.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T18:10:15.420 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T18:10:15.420 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-17T18:10:15.420 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-17T18:10:15.421 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-17T18:10:15.421 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-17T18:10:15.421 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi121.front.sepia.ceph.com, path=None, version=distro) 2024-09-17T18:10:15.421 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T18:10:15.483 INFO:teuthology.orchestra.run.smithi121.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T18:10:15.483 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T18:10:15.483 DEBUG:teuthology.orchestra.run.smithi121:> sudo rpm -qi grub2-tools 2024-09-17T18:10:15.525 INFO:teuthology.orchestra.run.smithi121.stdout:Name : grub2-tools 2024-09-17T18:10:15.525 INFO:teuthology.orchestra.run.smithi121.stdout:Epoch : 1 2024-09-17T18:10:15.525 INFO:teuthology.orchestra.run.smithi121.stdout:Version : 2.06 2024-09-17T18:10:15.525 INFO:teuthology.orchestra.run.smithi121.stdout:Release : 82.el9 2024-09-17T18:10:15.525 INFO:teuthology.orchestra.run.smithi121.stdout:Architecture: x86_64 2024-09-17T18:10:15.525 INFO:teuthology.orchestra.run.smithi121.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-17T18:10:15.525 INFO:teuthology.orchestra.run.smithi121.stdout:Group : Unspecified 2024-09-17T18:10:15.525 INFO:teuthology.orchestra.run.smithi121.stdout:Size : 8274670 2024-09-17T18:10:15.525 INFO:teuthology.orchestra.run.smithi121.stdout:License : GPLv3+ 2024-09-17T18:10:15.526 INFO:teuthology.orchestra.run.smithi121.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-17T18:10:15.526 INFO:teuthology.orchestra.run.smithi121.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-17T18:10:15.526 INFO:teuthology.orchestra.run.smithi121.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-17T18:10:15.526 INFO:teuthology.orchestra.run.smithi121.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-17T18:10:15.526 INFO:teuthology.orchestra.run.smithi121.stdout:Packager : builder@centos.org 2024-09-17T18:10:15.526 INFO:teuthology.orchestra.run.smithi121.stdout:Vendor : CentOS 2024-09-17T18:10:15.526 INFO:teuthology.orchestra.run.smithi121.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-17T18:10:15.526 INFO:teuthology.orchestra.run.smithi121.stdout:Summary : Support tools for GRUB. 2024-09-17T18:10:15.526 INFO:teuthology.orchestra.run.smithi121.stdout:Description : 2024-09-17T18:10:15.526 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:10:15.526 INFO:teuthology.orchestra.run.smithi121.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-17T18:10:15.526 INFO:teuthology.orchestra.run.smithi121.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-17T18:10:15.526 INFO:teuthology.orchestra.run.smithi121.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-17T18:10:15.526 INFO:teuthology.orchestra.run.smithi121.stdout:hardware devices. 2024-09-17T18:10:15.526 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:10:15.527 INFO:teuthology.orchestra.run.smithi121.stdout:This subpackage provides tools for support of all platforms. 2024-09-17T18:10:15.527 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-17T18:10:15.527 INFO:teuthology.task.kernel:Updating grub on smithi121 to boot 5.14.0-508.el9.x86_64 2024-09-17T18:10:15.528 DEBUG:teuthology.orchestra.run.smithi121:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-17T18:10:16.203 INFO:teuthology.orchestra.run.smithi121.stderr:Generating grub configuration file ... 2024-09-17T18:10:16.689 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T18:10:16.737 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-17T18:10:17.078 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T18:10:17.078 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-17T18:10:17.078 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T18:10:17.078 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T18:10:17.521 INFO:teuthology.orchestra.run.smithi121.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-17T18:10:17.553 INFO:teuthology.orchestra.run.smithi121.stderr:done 2024-09-17T18:10:17.556 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2024-09-17T18:10:17.572 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.hqEPZx4jN0 2024-09-17T18:10:17.572 DEBUG:teuthology.orchestra.run.smithi121:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.hqEPZx4jN0 2024-09-17T18:10:17.639 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.hqEPZx4jN0 2024-09-17T18:10:17.774 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.hqEPZx4jN0 is 6KB 2024-09-17T18:10:17.822 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.hqEPZx4jN0 2024-09-17T18:10:17.837 DEBUG:teuthology.orchestra.run.smithi121:> sudo /bin/ls /boot/loader/entries || true 2024-09-17T18:10:17.914 INFO:teuthology.orchestra.run.smithi121.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-17T18:10:17.914 INFO:teuthology.orchestra.run.smithi121.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-17T18:10:17.914 INFO:teuthology.orchestra.run.smithi121.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-17T18:10:17.914 INFO:teuthology.orchestra.run.smithi121.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-17T18:10:17.915 DEBUG:teuthology.orchestra.run.smithi121:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-17T18:10:17.928 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T18:10:17.928 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:10:17.928 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T18:10:17.928 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-17T18:10:17.928 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-17T18:10:17.928 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-17T18:10:17.929 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-17T18:10:17.929 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:10:17.929 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T18:10:18.070 DEBUG:teuthology.orchestra.run.smithi121:> sudo shutdown -r now 2024-09-17T18:10:18.261 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T18:10:18.345 INFO:teuthology.orchestra.run.smithi173.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T18:10:18.346 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T18:10:18.346 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-17T18:10:18.346 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-17T18:10:18.346 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-17T18:10:18.346 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-17T18:10:18.346 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi173.front.sepia.ceph.com, path=None, version=distro) 2024-09-17T18:10:18.346 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T18:10:18.404 INFO:teuthology.orchestra.run.smithi173.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T18:10:18.404 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T18:10:18.404 DEBUG:teuthology.orchestra.run.smithi173:> sudo rpm -qi grub2-tools 2024-09-17T18:10:18.436 INFO:teuthology.orchestra.run.smithi173.stdout:Name : grub2-tools 2024-09-17T18:10:18.437 INFO:teuthology.orchestra.run.smithi173.stdout:Epoch : 1 2024-09-17T18:10:18.437 INFO:teuthology.orchestra.run.smithi173.stdout:Version : 2.06 2024-09-17T18:10:18.437 INFO:teuthology.orchestra.run.smithi173.stdout:Release : 82.el9 2024-09-17T18:10:18.437 INFO:teuthology.orchestra.run.smithi173.stdout:Architecture: x86_64 2024-09-17T18:10:18.437 INFO:teuthology.orchestra.run.smithi173.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-17T18:10:18.437 INFO:teuthology.orchestra.run.smithi173.stdout:Group : Unspecified 2024-09-17T18:10:18.437 INFO:teuthology.orchestra.run.smithi173.stdout:Size : 8274670 2024-09-17T18:10:18.437 INFO:teuthology.orchestra.run.smithi173.stdout:License : GPLv3+ 2024-09-17T18:10:18.437 INFO:teuthology.orchestra.run.smithi173.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-17T18:10:18.437 INFO:teuthology.orchestra.run.smithi173.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-17T18:10:18.437 INFO:teuthology.orchestra.run.smithi173.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-17T18:10:18.438 INFO:teuthology.orchestra.run.smithi173.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-17T18:10:18.438 INFO:teuthology.orchestra.run.smithi173.stdout:Packager : builder@centos.org 2024-09-17T18:10:18.438 INFO:teuthology.orchestra.run.smithi173.stdout:Vendor : CentOS 2024-09-17T18:10:18.438 INFO:teuthology.orchestra.run.smithi173.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-17T18:10:18.438 INFO:teuthology.orchestra.run.smithi173.stdout:Summary : Support tools for GRUB. 2024-09-17T18:10:18.438 INFO:teuthology.orchestra.run.smithi173.stdout:Description : 2024-09-17T18:10:18.438 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:10:18.438 INFO:teuthology.orchestra.run.smithi173.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-17T18:10:18.438 INFO:teuthology.orchestra.run.smithi173.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-17T18:10:18.438 INFO:teuthology.orchestra.run.smithi173.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-17T18:10:18.438 INFO:teuthology.orchestra.run.smithi173.stdout:hardware devices. 2024-09-17T18:10:18.438 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:10:18.438 INFO:teuthology.orchestra.run.smithi173.stdout:This subpackage provides tools for support of all platforms. 2024-09-17T18:10:18.439 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-17T18:10:18.439 INFO:teuthology.task.kernel:Updating grub on smithi173 to boot 5.14.0-508.el9.x86_64 2024-09-17T18:10:18.439 DEBUG:teuthology.orchestra.run.smithi173:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-17T18:10:19.032 INFO:teuthology.orchestra.run.smithi173.stderr:Generating grub configuration file ... 2024-09-17T18:10:20.312 INFO:teuthology.orchestra.run.smithi173.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-17T18:10:20.344 INFO:teuthology.orchestra.run.smithi173.stderr:done 2024-09-17T18:10:20.345 DEBUG:teuthology.orchestra.run.smithi173:> mktemp 2024-09-17T18:10:20.361 INFO:teuthology.orchestra.run.smithi173.stdout:/tmp/tmp.pxslU5vOZw 2024-09-17T18:10:20.361 DEBUG:teuthology.orchestra.run.smithi173:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.pxslU5vOZw 2024-09-17T18:10:20.428 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0666 /tmp/tmp.pxslU5vOZw 2024-09-17T18:10:20.573 DEBUG:teuthology.orchestra.remote:smithi173:/tmp/tmp.pxslU5vOZw is 6KB 2024-09-17T18:10:20.622 DEBUG:teuthology.orchestra.run.smithi173:> rm -fr /tmp/tmp.pxslU5vOZw 2024-09-17T18:10:20.637 DEBUG:teuthology.orchestra.run.smithi173:> sudo /bin/ls /boot/loader/entries || true 2024-09-17T18:10:20.699 INFO:teuthology.orchestra.run.smithi173.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-17T18:10:20.699 INFO:teuthology.orchestra.run.smithi173.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-17T18:10:20.700 INFO:teuthology.orchestra.run.smithi173.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-17T18:10:20.700 INFO:teuthology.orchestra.run.smithi173.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-17T18:10:20.701 DEBUG:teuthology.orchestra.run.smithi173:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-17T18:10:20.851 DEBUG:teuthology.orchestra.run.smithi173:> sudo shutdown -r now 2024-09-17T18:10:48.097 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-17T18:10:48.097 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-09-17T18:10:48.098 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T18:10:50.857 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-17T18:10:50.857 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-09-17T18:10:50.857 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T18:11:06.429 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.121 2024-09-17T18:11:09.245 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-09-17T18:11:15.431 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-09-17T18:11:15.432 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T18:11:18.249 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-09-17T18:11:18.249 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T18:11:18.493 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.121 2024-09-17T18:11:21.309 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-09-17T18:11:30.498 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-09-17T18:11:30.499 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T18:11:33.313 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-09-17T18:11:33.313 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T18:11:36.381 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-09-17T18:11:51.389 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-09-17T18:11:51.390 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T18:11:54.465 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-09-17T18:12:02.336 DEBUG:teuthology.orchestra.run.smithi121:> true 2024-09-17T18:12:02.798 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-09-17T18:12:02.798 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-17T18:12:02.799 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2024-09-17T18:12:02.837 INFO:teuthology.orchestra.run.smithi121.stdout:5.14.0-508.el9.x86_64 2024-09-17T18:12:02.837 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-17T18:12:02.837 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-17T18:12:02.838 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-17T18:12:03.838 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-17T18:12:03.838 DEBUG:teuthology.orchestra.run.smithi121:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-17T18:12:03.901 INFO:teuthology.orchestra.run.smithi121.stdout:ttyS1 2024-09-17T18:12:03.925 DEBUG:teuthology.parallel:result is None 2024-09-17T18:12:12.466 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-09-17T18:12:12.467 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T18:12:12.997 DEBUG:teuthology.orchestra.run.smithi173:> true 2024-09-17T18:12:13.329 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-09-17T18:12:13.329 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-508.el9.x86_64"... 2024-09-17T18:12:13.329 DEBUG:teuthology.orchestra.run.smithi173:> uname -r 2024-09-17T18:12:13.345 INFO:teuthology.orchestra.run.smithi173.stdout:5.14.0-508.el9.x86_64 2024-09-17T18:12:13.346 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-17T18:12:13.346 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-17T18:12:13.346 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-17T18:12:14.347 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-17T18:12:14.347 DEBUG:teuthology.orchestra.run.smithi173:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-17T18:12:14.405 INFO:teuthology.orchestra.run.smithi173.stdout:ttyS1 2024-09-17T18:12:14.437 DEBUG:teuthology.parallel:result is None 2024-09-17T18:12:14.438 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-17T18:12:14.446 INFO:teuthology.task.internal:Creating test directory... 2024-09-17T18:12:14.446 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-17T18:12:14.449 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-17T18:12:14.466 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-17T18:12:14.492 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-17T18:12:14.540 INFO:teuthology.task.internal:Creating archive directory... 2024-09-17T18:12:14.541 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-17T18:12:14.543 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-17T18:12:14.581 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-17T18:12:14.590 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-17T18:12:14.590 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-17T18:12:14.611 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-17T18:12:14.643 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:12:14.657 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:12:14.658 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:12:14.666 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:12:14.669 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-17T18:12:14.677 INFO:teuthology.task.internal:Configuring sudo... 2024-09-17T18:12:14.677 DEBUG:teuthology.orchestra.run.smithi121:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-17T18:12:14.702 DEBUG:teuthology.orchestra.run.smithi173:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-17T18:12:14.737 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-17T18:12:14.746 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-17T18:12:14.746 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-17T18:12:14.770 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-17T18:12:14.794 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T18:12:14.867 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T18:12:14.927 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T18:12:15.010 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T18:12:15.053 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-17T18:12:15.054 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-17T18:12:15.121 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T18:12:15.166 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T18:12:15.228 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T18:12:15.311 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T18:12:15.356 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T18:12:15.356 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-17T18:12:15.424 DEBUG:teuthology.orchestra.run.smithi121:> sudo service rsyslog restart 2024-09-17T18:12:15.426 DEBUG:teuthology.orchestra.run.smithi173:> sudo service rsyslog restart 2024-09-17T18:12:15.476 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T18:12:15.510 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T18:12:15.992 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-17T18:12:16.002 INFO:teuthology.task.internal:Starting timer... 2024-09-17T18:12:16.002 INFO:teuthology.run_tasks:Running task pcp... 2024-09-17T18:12:16.019 INFO:teuthology.run_tasks:Running task selinux... 2024-09-17T18:12:16.030 DEBUG:teuthology.orchestra.run.smithi121:> sudo service auditd rotate 2024-09-17T18:12:16.081 INFO:teuthology.orchestra.run.smithi121.stdout:Rotating logs: 2024-09-17T18:12:16.164 DEBUG:teuthology.orchestra.run.smithi173:> sudo service auditd rotate 2024-09-17T18:12:16.210 INFO:teuthology.orchestra.run.smithi173.stdout:Rotating logs: 2024-09-17T18:12:16.212 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-17T18:12:16.212 DEBUG:teuthology.orchestra.run.smithi121:> /usr/sbin/getenforce 2024-09-17T18:12:16.237 INFO:teuthology.orchestra.run.smithi121.stdout:Permissive 2024-09-17T18:12:16.238 DEBUG:teuthology.orchestra.run.smithi173:> /usr/sbin/getenforce 2024-09-17T18:12:16.264 INFO:teuthology.orchestra.run.smithi173.stdout:Permissive 2024-09-17T18:12:16.265 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi121.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi173.front.sepia.ceph.com': 'permissive'} 2024-09-17T18:12:16.265 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T18:12:16.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T18:12:16.305 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T18:12:16.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T18:12:16.334 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-17T18:12:16.334 DEBUG:teuthology.orchestra.run.smithi121:> sudo /usr/sbin/setenforce permissive 2024-09-17T18:12:16.381 DEBUG:teuthology.orchestra.run.smithi173:> sudo /usr/sbin/setenforce permissive 2024-09-17T18:12:16.411 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-17T18:12:16.423 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-17T18:12:16.435 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-17T18:12:16.437 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi121.front.sepia.ceph.com,smithi173.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-17T18:16:02.747 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi121.front.sepia.ceph.com'), Remote(name='ubuntu@smithi173.front.sepia.ceph.com')] 2024-09-17T18:16:02.748 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-09-17T18:16:02.749 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T18:16:02.828 DEBUG:teuthology.orchestra.run.smithi121:> true 2024-09-17T18:16:02.908 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-09-17T18:16:02.908 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-09-17T18:16:02.909 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T18:16:02.987 DEBUG:teuthology.orchestra.run.smithi173:> true 2024-09-17T18:16:03.028 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-09-17T18:16:03.028 INFO:teuthology.run_tasks:Running task clock... 2024-09-17T18:16:03.039 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-17T18:16:03.039 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-17T18:16:03.039 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T18:16:03.042 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-17T18:16:03.042 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T18:16:03.068 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-17T18:16:03.082 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-17T18:16:03.098 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-17T18:16:03.107 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: ntpd: command not found 2024-09-17T18:16:03.112 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-17T18:16:03.117 INFO:teuthology.orchestra.run.smithi121.stdout:506 Cannot talk to daemon 2024-09-17T18:16:03.131 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-17T18:16:03.136 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: ntpd: command not found 2024-09-17T18:16:03.145 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-17T18:16:03.147 INFO:teuthology.orchestra.run.smithi173.stdout:506 Cannot talk to daemon 2024-09-17T18:16:03.161 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-17T18:16:03.175 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-17T18:16:03.203 INFO:teuthology.orchestra.run.smithi121.stderr:bash: line 1: ntpq: command not found 2024-09-17T18:16:03.205 INFO:teuthology.orchestra.run.smithi121.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T18:16:03.206 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================== 2024-09-17T18:16:03.206 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T18:16:03.206 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T18:16:03.206 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T18:16:03.206 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T18:16:03.232 INFO:teuthology.orchestra.run.smithi173.stderr:bash: line 1: ntpq: command not found 2024-09-17T18:16:03.235 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T18:16:03.235 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-09-17T18:16:03.235 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T18:16:03.235 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T18:16:03.235 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T18:16:03.235 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T18:16:03.236 INFO:teuthology.run_tasks:Running task install... 2024-09-17T18:16:03.248 DEBUG:teuthology.task.install:project ceph 2024-09-17T18:16:03.249 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'}} 2024-09-17T18:16:03.249 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'} 2024-09-17T18:16:03.249 INFO:teuthology.task.install:Using flavor: default 2024-09-17T18:16:03.260 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-17T18:16:03.261 INFO:teuthology.task.install:extra packages: [] 2024-09-17T18:16:03.261 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203', 'tag': None, 'wait_for_package': False} 2024-09-17T18:16:03.262 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T18:16:03.264 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203', 'tag': None, 'wait_for_package': False} 2024-09-17T18:16:03.265 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T18:16:03.419 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/ 2024-09-17T18:16:03.419 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4975.g1889a165 2024-09-17T18:16:03.423 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/ 2024-09-17T18:16:03.424 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4975.g1889a165 2024-09-17T18:16:03.532 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-17T18:16:03.533 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T18:16:03.533 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-17T18:16:03.545 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-17T18:16:03.546 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-17T18:16:03.546 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-17T18:16:03.559 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-17T18:16:03.560 DEBUG:teuthology.orchestra.run.smithi173:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1889a165b5128787ec693450a3d1c5df5d6dc203/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-17T18:16:03.573 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-17T18:16:03.574 DEBUG:teuthology.orchestra.run.smithi121:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1889a165b5128787ec693450a3d1c5df5d6dc203/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-17T18:16:03.628 DEBUG:teuthology.orchestra.run.smithi173:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-17T18:16:03.639 DEBUG:teuthology.orchestra.run.smithi121:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-17T18:16:03.703 DEBUG:teuthology.orchestra.run.smithi173:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-17T18:16:03.714 DEBUG:teuthology.orchestra.run.smithi121:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-17T18:16:03.767 INFO:teuthology.orchestra.run.smithi173.stdout:check_obsoletes = 1 2024-09-17T18:16:03.769 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum clean all 2024-09-17T18:16:03.778 INFO:teuthology.orchestra.run.smithi121.stdout:check_obsoletes = 1 2024-09-17T18:16:03.780 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum clean all 2024-09-17T18:16:03.989 INFO:teuthology.orchestra.run.smithi173.stdout:45 files removed 2024-09-17T18:16:04.007 INFO:teuthology.orchestra.run.smithi121.stdout:45 files removed 2024-09-17T18:16:04.016 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-radosgw 2024-09-17T18:16:04.036 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-radosgw 2024-09-17T18:16:04.568 INFO:teuthology.orchestra.run.smithi173.stdout:ceph packages for x86_64 315 kB/s | 84 kB 00:00 2024-09-17T18:16:04.588 INFO:teuthology.orchestra.run.smithi121.stdout:ceph packages for x86_64 313 kB/s | 84 kB 00:00 2024-09-17T18:16:04.861 INFO:teuthology.orchestra.run.smithi173.stdout:ceph noarch packages 98 kB/s | 24 kB 00:00 2024-09-17T18:16:04.872 INFO:teuthology.orchestra.run.smithi121.stdout:ceph noarch packages 101 kB/s | 24 kB 00:00 2024-09-17T18:16:05.136 INFO:teuthology.orchestra.run.smithi173.stdout:ceph source packages 8.0 kB/s | 1.9 kB 00:00 2024-09-17T18:16:05.150 INFO:teuthology.orchestra.run.smithi121.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-17T18:16:06.017 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - BaseOS 9.8 MB/s | 8.3 MB 00:00 2024-09-17T18:16:06.025 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - BaseOS 9.8 MB/s | 8.3 MB 00:00 2024-09-17T18:16:09.201 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - AppStream 9.6 MB/s | 20 MB 00:02 2024-09-17T18:16:10.645 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - AppStream 5.7 MB/s | 20 MB 00:03 2024-09-17T18:16:13.993 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - CRB 6.5 MB/s | 6.5 MB 00:01 2024-09-17T18:16:15.443 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - Extras packages 86 kB/s | 19 kB 00:00 2024-09-17T18:16:16.356 INFO:teuthology.orchestra.run.smithi121.stdout:Extra Packages for Enterprise Linux 29 MB/s | 23 MB 00:00 2024-09-17T18:16:16.604 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - CRB 3.0 MB/s | 6.5 MB 00:02 2024-09-17T18:16:18.196 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - Extras packages 77 kB/s | 19 kB 00:00 2024-09-17T18:16:19.351 INFO:teuthology.orchestra.run.smithi173.stdout:Extra Packages for Enterprise Linux 22 MB/s | 23 MB 00:01 2024-09-17T18:16:21.745 INFO:teuthology.orchestra.run.smithi121.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-17T18:16:23.978 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-17T18:16:23.982 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:16:23.982 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-09-17T18:16:23.982 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:16:23.982 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-17T18:16:23.982 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw x86_64 2:19.3.0-4975.g1889a165.el9 ceph 11 M 2024-09-17T18:16:23.982 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrading: 2024-09-17T18:16:23.983 INFO:teuthology.orchestra.run.smithi121.stdout: librados2 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 3.5 M 2024-09-17T18:16:23.983 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 3.3 M 2024-09-17T18:16:23.983 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-09-17T18:16:23.983 INFO:teuthology.orchestra.run.smithi121.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-17T18:16:23.983 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base x86_64 2:19.3.0-4975.g1889a165.el9 ceph 5.5 M 2024-09-17T18:16:23.983 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common x86_64 2:19.3.0-4975.g1889a165.el9 ceph 22 M 2024-09-17T18:16:23.983 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux x86_64 2:19.3.0-4975.g1889a165.el9 ceph 25 k 2024-09-17T18:16:23.983 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-17T18:16:23.983 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-17T18:16:23.983 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-17T18:16:23.984 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 756 k 2024-09-17T18:16:23.984 INFO:teuthology.orchestra.run.smithi121.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-17T18:16:23.984 INFO:teuthology.orchestra.run.smithi121.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-17T18:16:23.984 INFO:teuthology.orchestra.run.smithi121.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-17T18:16:23.984 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-17T18:16:23.984 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 508 k 2024-09-17T18:16:23.984 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-17T18:16:23.984 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 5.5 M 2024-09-17T18:16:23.984 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-17T18:16:23.984 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-17T18:16:23.984 INFO:teuthology.orchestra.run.smithi121.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-17T18:16:23.984 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse x86_64 2:19.3.0-4975.g1889a165.el9 ceph 45 k 2024-09-17T18:16:23.985 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common x86_64 2:19.3.0-4975.g1889a165.el9 ceph 143 k 2024-09-17T18:16:23.985 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs x86_64 2:19.3.0-4975.g1889a165.el9 ceph 161 k 2024-09-17T18:16:23.985 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-17T18:16:23.985 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados x86_64 2:19.3.0-4975.g1889a165.el9 ceph 321 k 2024-09-17T18:16:23.985 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd x86_64 2:19.3.0-4975.g1889a165.el9 ceph 305 k 2024-09-17T18:16:23.985 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw x86_64 2:19.3.0-4975.g1889a165.el9 ceph 99 k 2024-09-17T18:16:23.985 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-17T18:16:23.985 INFO:teuthology.orchestra.run.smithi121.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-17T18:16:23.985 INFO:teuthology.orchestra.run.smithi121.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-17T18:16:23.985 INFO:teuthology.orchestra.run.smithi121.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-17T18:16:23.986 INFO:teuthology.orchestra.run.smithi121.stdout:Installing weak dependencies: 2024-09-17T18:16:23.986 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-17T18:16:23.986 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:16:23.986 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-17T18:16:23.986 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:16:23.986 INFO:teuthology.orchestra.run.smithi121.stdout:Install 31 Packages 2024-09-17T18:16:23.986 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrade 2 Packages 2024-09-17T18:16:23.986 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:16:23.987 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 62 M 2024-09-17T18:16:23.987 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-17T18:16:24.765 INFO:teuthology.orchestra.run.smithi173.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-09-17T18:16:25.171 INFO:teuthology.orchestra.run.smithi121.stdout:(1/33): ceph-base-19.3.0-4975.g1889a165.el9.x86 9.8 MB/s | 5.5 MB 00:00 2024-09-17T18:16:25.213 INFO:teuthology.orchestra.run.smithi121.stdout:(2/33): ceph-selinux-19.3.0-4975.g1889a165.el9. 605 kB/s | 25 kB 00:00 2024-09-17T18:16:25.296 INFO:teuthology.orchestra.run.smithi121.stdout:(3/33): libcephfs2-19.3.0-4975.g1889a165.el9.x8 8.9 MB/s | 756 kB 00:00 2024-09-17T18:16:25.372 INFO:teuthology.orchestra.run.smithi121.stdout:(4/33): libradosstriper1-19.3.0-4975.g1889a165. 6.6 MB/s | 508 kB 00:00 2024-09-17T18:16:25.539 INFO:teuthology.orchestra.run.smithi121.stdout:(5/33): ceph-radosgw-19.3.0-4975.g1889a165.el9. 12 MB/s | 11 MB 00:00 2024-09-17T18:16:25.764 INFO:teuthology.orchestra.run.smithi121.stdout:(6/33): ceph-common-19.3.0-4975.g1889a165.el9.x 19 MB/s | 22 MB 00:01 2024-09-17T18:16:25.789 INFO:teuthology.orchestra.run.smithi121.stdout:(7/33): python3-ceph-argparse-19.3.0-4975.g1889 180 kB/s | 45 kB 00:00 2024-09-17T18:16:25.815 INFO:teuthology.orchestra.run.smithi121.stdout:(8/33): python3-ceph-common-19.3.0-4975.g1889a1 2.8 MB/s | 143 kB 00:00 2024-09-17T18:16:25.840 INFO:teuthology.orchestra.run.smithi121.stdout:(9/33): python3-cephfs-19.3.0-4975.g1889a165.el 3.1 MB/s | 161 kB 00:00 2024-09-17T18:16:25.873 INFO:teuthology.orchestra.run.smithi121.stdout:(10/33): python3-rados-19.3.0-4975.g1889a165.el 5.4 MB/s | 321 kB 00:00 2024-09-17T18:16:25.899 INFO:teuthology.orchestra.run.smithi121.stdout:(11/33): python3-rbd-19.3.0-4975.g1889a165.el9. 5.1 MB/s | 305 kB 00:00 2024-09-17T18:16:25.924 INFO:teuthology.orchestra.run.smithi121.stdout:(12/33): python3-rgw-19.3.0-4975.g1889a165.el9. 1.9 MB/s | 99 kB 00:00 2024-09-17T18:16:25.999 INFO:teuthology.orchestra.run.smithi121.stdout:(13/33): librgw2-19.3.0-4975.g1889a165.el9.x86_ 8.8 MB/s | 5.5 MB 00:00 2024-09-17T18:16:26.391 INFO:teuthology.orchestra.run.smithi121.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 217 kB/s | 107 kB 00:00 2024-09-17T18:16:26.491 INFO:teuthology.orchestra.run.smithi121.stdout:(15/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 326 kB/s | 160 kB 00:00 2024-09-17T18:16:26.516 INFO:teuthology.orchestra.run.smithi121.stdout:(16/33): libnbd-1.20.2-2.el9.x86_64.rpm 288 kB/s | 171 kB 00:00 2024-09-17T18:16:26.542 INFO:teuthology.orchestra.run.smithi121.stdout:(17/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 301 kB/s | 45 kB 00:00 2024-09-17T18:16:26.650 INFO:teuthology.orchestra.run.smithi121.stdout:(18/33): python3-prettytable-0.7.2-27.el9.noarc 390 kB/s | 42 kB 00:00 2024-09-17T18:16:26.717 INFO:teuthology.orchestra.run.smithi121.stdout:(19/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.4 MB/s | 292 kB 00:00 2024-09-17T18:16:26.834 INFO:teuthology.orchestra.run.smithi121.stdout:(20/33): qatlib-service-24.02.0-1.el9.x86_64.rp 308 kB/s | 36 kB 00:00 2024-09-17T18:16:26.860 INFO:teuthology.orchestra.run.smithi121.stdout:(21/33): qatlib-24.02.0-1.el9.x86_64.rpm 1.0 MB/s | 221 kB 00:00 2024-09-17T18:16:26.885 INFO:teuthology.orchestra.run.smithi121.stdout:(22/33): librdkafka-1.6.1-102.el9.x86_64.rpm 1.6 MB/s | 662 kB 00:00 2024-09-17T18:16:26.952 INFO:teuthology.orchestra.run.smithi121.stdout:(23/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 3.3 MB/s | 308 kB 00:00 2024-09-17T18:16:26.977 INFO:teuthology.orchestra.run.smithi121.stdout:(24/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 333 kB/s | 47 kB 00:00 2024-09-17T18:16:26.999 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T18:16:27.002 INFO:teuthology.orchestra.run.smithi121.stdout:(25/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 515 kB/s | 26 kB 00:00 2024-09-17T18:16:27.016 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:16:27.016 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-09-17T18:16:27.016 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:16:27.017 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T18:16:27.017 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw x86_64 2:19.3.0-4975.g1889a165.el9 ceph 11 M 2024-09-17T18:16:27.017 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrading: 2024-09-17T18:16:27.017 INFO:teuthology.orchestra.run.smithi173.stdout: librados2 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 3.5 M 2024-09-17T18:16:27.017 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 3.3 M 2024-09-17T18:16:27.017 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-09-17T18:16:27.017 INFO:teuthology.orchestra.run.smithi173.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-17T18:16:27.017 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base x86_64 2:19.3.0-4975.g1889a165.el9 ceph 5.5 M 2024-09-17T18:16:27.017 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common x86_64 2:19.3.0-4975.g1889a165.el9 ceph 22 M 2024-09-17T18:16:27.017 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux x86_64 2:19.3.0-4975.g1889a165.el9 ceph 25 k 2024-09-17T18:16:27.017 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-17T18:16:27.017 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-17T18:16:27.017 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-17T18:16:27.018 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 756 k 2024-09-17T18:16:27.018 INFO:teuthology.orchestra.run.smithi173.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-17T18:16:27.018 INFO:teuthology.orchestra.run.smithi173.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-17T18:16:27.018 INFO:teuthology.orchestra.run.smithi173.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-17T18:16:27.018 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-17T18:16:27.018 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 508 k 2024-09-17T18:16:27.018 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-17T18:16:27.018 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 5.5 M 2024-09-17T18:16:27.018 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-17T18:16:27.018 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-17T18:16:27.018 INFO:teuthology.orchestra.run.smithi173.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-17T18:16:27.018 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse x86_64 2:19.3.0-4975.g1889a165.el9 ceph 45 k 2024-09-17T18:16:27.018 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common x86_64 2:19.3.0-4975.g1889a165.el9 ceph 143 k 2024-09-17T18:16:27.019 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs x86_64 2:19.3.0-4975.g1889a165.el9 ceph 161 k 2024-09-17T18:16:27.019 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-17T18:16:27.019 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados x86_64 2:19.3.0-4975.g1889a165.el9 ceph 321 k 2024-09-17T18:16:27.019 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd x86_64 2:19.3.0-4975.g1889a165.el9 ceph 305 k 2024-09-17T18:16:27.019 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw x86_64 2:19.3.0-4975.g1889a165.el9 ceph 99 k 2024-09-17T18:16:27.019 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-17T18:16:27.019 INFO:teuthology.orchestra.run.smithi173.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-17T18:16:27.019 INFO:teuthology.orchestra.run.smithi173.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-17T18:16:27.019 INFO:teuthology.orchestra.run.smithi173.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-17T18:16:27.019 INFO:teuthology.orchestra.run.smithi173.stdout:Installing weak dependencies: 2024-09-17T18:16:27.019 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-17T18:16:27.019 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:16:27.019 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T18:16:27.019 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:16:27.020 INFO:teuthology.orchestra.run.smithi173.stdout:Install 31 Packages 2024-09-17T18:16:27.020 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrade 2 Packages 2024-09-17T18:16:27.020 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:16:27.020 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 62 M 2024-09-17T18:16:27.020 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T18:16:27.036 INFO:teuthology.orchestra.run.smithi121.stdout:(26/33): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-09-17T18:16:27.070 INFO:teuthology.orchestra.run.smithi121.stdout:(27/33): liboath-2.6.7-2.el9.x86_64.rpm 525 kB/s | 49 kB 00:00 2024-09-17T18:16:27.112 INFO:teuthology.orchestra.run.smithi121.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 4.5 MB/s | 191 kB 00:00 2024-09-17T18:16:27.154 INFO:teuthology.orchestra.run.smithi121.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.1 MB/s | 839 kB 00:00 2024-09-17T18:16:27.237 INFO:teuthology.orchestra.run.smithi121.stdout:(30/33): libarrow-9.0.0-11.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-09-17T18:16:27.279 INFO:teuthology.orchestra.run.smithi121.stdout:(31/33): thrift-0.15.0-2.el9.x86_64.rpm 9.5 MB/s | 1.6 MB 00:00 2024-09-17T18:16:27.596 INFO:teuthology.orchestra.run.smithi121.stdout:(32/33): librados2-19.3.0-4975.g1889a165.el9.x8 7.8 MB/s | 3.5 MB 00:00 2024-09-17T18:16:27.796 INFO:teuthology.orchestra.run.smithi121.stdout:(33/33): librbd1-19.3.0-4975.g1889a165.el9.x86_ 5.8 MB/s | 3.3 MB 00:00 2024-09-17T18:16:27.801 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-17T18:16:27.801 INFO:teuthology.orchestra.run.smithi121.stdout:Total 16 MB/s | 62 MB 00:03 2024-09-17T18:16:27.913 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-17T18:16:27.950 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-17T18:16:27.950 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-17T18:16:28.623 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-17T18:16:28.624 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-17T18:16:28.866 INFO:teuthology.orchestra.run.smithi173.stdout:(1/33): ceph-radosgw-19.3.0-4975.g1889a165.el9. 7.1 MB/s | 11 MB 00:01 2024-09-17T18:16:28.908 INFO:teuthology.orchestra.run.smithi173.stdout:(2/33): ceph-selinux-19.3.0-4975.g1889a165.el9. 609 kB/s | 25 kB 00:00 2024-09-17T18:16:29.033 INFO:teuthology.orchestra.run.smithi173.stdout:(3/33): libcephfs2-19.3.0-4975.g1889a165.el9.x8 5.9 MB/s | 756 kB 00:00 2024-09-17T18:16:29.100 INFO:teuthology.orchestra.run.smithi173.stdout:(4/33): libradosstriper1-19.3.0-4975.g1889a165. 7.5 MB/s | 508 kB 00:00 2024-09-17T18:16:29.617 INFO:teuthology.orchestra.run.smithi173.stdout:(5/33): ceph-common-19.3.0-4975.g1889a165.el9.x 9.5 MB/s | 22 MB 00:02 2024-09-17T18:16:29.700 INFO:teuthology.orchestra.run.smithi173.stdout:(6/33): ceph-base-19.3.0-4975.g1889a165.el9.x86 2.3 MB/s | 5.5 MB 00:02 2024-09-17T18:16:29.717 INFO:teuthology.orchestra.run.smithi173.stdout:(7/33): python3-ceph-argparse-19.3.0-4975.g1889 450 kB/s | 45 kB 00:00 2024-09-17T18:16:29.793 INFO:teuthology.orchestra.run.smithi173.stdout:(8/33): librgw2-19.3.0-4975.g1889a165.el9.x86_6 7.9 MB/s | 5.5 MB 00:00 2024-09-17T18:16:29.818 INFO:teuthology.orchestra.run.smithi173.stdout:(9/33): python3-ceph-common-19.3.0-4975.g1889a1 1.2 MB/s | 143 kB 00:00 2024-09-17T18:16:29.843 INFO:teuthology.orchestra.run.smithi173.stdout:(10/33): python3-cephfs-19.3.0-4975.g1889a165.e 1.3 MB/s | 161 kB 00:00 2024-09-17T18:16:29.868 INFO:teuthology.orchestra.run.smithi173.stdout:(11/33): python3-rados-19.3.0-4975.g1889a165.el 4.2 MB/s | 321 kB 00:00 2024-09-17T18:16:29.894 INFO:teuthology.orchestra.run.smithi173.stdout:(12/33): python3-rgw-19.3.0-4975.g1889a165.el9. 1.9 MB/s | 99 kB 00:00 2024-09-17T18:16:29.919 INFO:teuthology.orchestra.run.smithi173.stdout:(13/33): python3-rbd-19.3.0-4975.g1889a165.el9. 3.0 MB/s | 305 kB 00:00 2024-09-17T18:16:30.086 INFO:teuthology.orchestra.run.smithi173.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 492 kB/s | 107 kB 00:00 2024-09-17T18:16:30.111 INFO:teuthology.orchestra.run.smithi173.stdout:(15/33): libnbd-1.20.2-2.el9.x86_64.rpm 785 kB/s | 171 kB 00:00 2024-09-17T18:16:30.136 INFO:teuthology.orchestra.run.smithi173.stdout:(16/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 737 kB/s | 160 kB 00:00 2024-09-17T18:16:30.154 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-17T18:16:30.161 INFO:teuthology.orchestra.run.smithi173.stdout:(17/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 600 kB/s | 45 kB 00:00 2024-09-17T18:16:30.212 INFO:teuthology.orchestra.run.smithi173.stdout:(18/33): python3-prettytable-0.7.2-27.el9.noarc 843 kB/s | 42 kB 00:00 2024-09-17T18:16:30.237 INFO:teuthology.orchestra.run.smithi173.stdout:(19/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-09-17T18:16:30.238 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-17T18:16:30.268 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-17T18:16:30.291 INFO:teuthology.orchestra.run.smithi173.stdout:(20/33): librdkafka-1.6.1-102.el9.x86_64.rpm 3.7 MB/s | 662 kB 00:00 2024-09-17T18:16:30.301 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-17T18:16:30.313 INFO:teuthology.orchestra.run.smithi173.stdout:(21/33): qatlib-24.02.0-1.el9.x86_64.rpm 2.1 MB/s | 221 kB 00:00 2024-09-17T18:16:30.324 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-17T18:16:30.338 INFO:teuthology.orchestra.run.smithi173.stdout:(22/33): qatlib-service-24.02.0-1.el9.x86_64.rp 357 kB/s | 36 kB 00:00 2024-09-17T18:16:30.344 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-17T18:16:30.363 INFO:teuthology.orchestra.run.smithi173.stdout:(23/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 630 kB/s | 47 kB 00:00 2024-09-17T18:16:30.422 INFO:teuthology.orchestra.run.smithi173.stdout:(24/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 3.6 MB/s | 308 kB 00:00 2024-09-17T18:16:30.447 INFO:teuthology.orchestra.run.smithi173.stdout:(25/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 311 kB/s | 26 kB 00:00 2024-09-17T18:16:30.472 INFO:teuthology.orchestra.run.smithi173.stdout:(26/33): liboath-2.6.7-2.el9.x86_64.rpm 971 kB/s | 49 kB 00:00 2024-09-17T18:16:30.489 INFO:teuthology.orchestra.run.smithi173.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-09-17T18:16:30.517 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-17T18:16:30.523 INFO:teuthology.orchestra.run.smithi173.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-17T18:16:30.548 INFO:teuthology.orchestra.run.smithi173.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 11 MB/s | 839 kB 00:00 2024-09-17T18:16:30.597 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-17T18:16:30.606 INFO:teuthology.orchestra.run.smithi173.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-09-17T18:16:30.690 INFO:teuthology.orchestra.run.smithi173.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-09-17T18:16:30.866 INFO:teuthology.orchestra.run.smithi173.stdout:(32/33): librbd1-19.3.0-4975.g1889a165.el9.x86_ 13 MB/s | 3.3 MB 00:00 2024-09-17T18:16:30.963 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-17T18:16:31.053 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librados2-2:19.3.0-4975.g1889a165.el9.x86_64 7/35 2024-09-17T18:16:31.129 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-2:19.3.0-4975.g1889a165.el9.x86_64 7/35 2024-09-17T18:16:31.220 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 8/35 2024-09-17T18:16:31.246 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 9/35 2024-09-17T18:16:31.291 INFO:teuthology.orchestra.run.smithi173.stdout:(33/33): librados2-19.3.0-4975.g1889a165.el9.x8 4.7 MB/s | 3.5 MB 00:00 2024-09-17T18:16:31.295 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T18:16:31.295 INFO:teuthology.orchestra.run.smithi173.stdout:Total 14 MB/s | 62 MB 00:04 2024-09-17T18:16:31.301 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 9/35 2024-09-17T18:16:31.338 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-17T18:16:31.382 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-argparse-2:19.3.0-4975.g1889a165.el9. 11/35 2024-09-17T18:16:31.410 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T18:16:31.446 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T18:16:31.446 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T18:16:31.451 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 12/35 2024-09-17T18:16:31.474 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_6 13/35 2024-09-17T18:16:31.537 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_6 13/35 2024-09-17T18:16:31.572 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-17T18:16:31.683 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-17T18:16:31.724 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-17T18:16:31.764 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-17T18:16:31.797 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-17T18:16:31.826 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-17T18:16:31.858 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-17T18:16:31.888 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-17T18:16:32.129 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T18:16:32.130 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T18:16:32.212 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-17T18:16:32.247 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 23/35 2024-09-17T18:16:32.322 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 23/35 2024-09-17T18:16:32.361 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 24/35 2024-09-17T18:16:32.397 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-17T18:16:32.909 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-17T18:16:32.933 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 27/35 2024-09-17T18:16:33.005 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 27/35 2024-09-17T18:16:33.082 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 28/35 2024-09-17T18:16:33.110 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-common-2:19.3.0-4975.g1889a165.el9.x8 29/35 2024-09-17T18:16:33.653 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T18:16:33.728 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-17T18:16:33.762 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-17T18:16:33.787 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-17T18:16:33.810 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-17T18:16:33.832 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-17T18:16:34.029 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-17T18:16:34.082 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-17T18:16:34.448 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-17T18:16:34.522 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librados2-2:19.3.0-4975.g1889a165.el9.x86_64 7/35 2024-09-17T18:16:34.606 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-2:19.3.0-4975.g1889a165.el9.x86_64 7/35 2024-09-17T18:16:34.711 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 8/35 2024-09-17T18:16:34.732 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 9/35 2024-09-17T18:16:34.797 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 9/35 2024-09-17T18:16:34.837 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-17T18:16:34.886 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-argparse-2:19.3.0-4975.g1889a165.el9. 11/35 2024-09-17T18:16:34.972 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 12/35 2024-09-17T18:16:35.001 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_6 13/35 2024-09-17T18:16:35.065 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_6 13/35 2024-09-17T18:16:35.099 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-17T18:16:35.217 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-17T18:16:35.257 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-17T18:16:35.305 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-17T18:16:35.332 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-17T18:16:35.361 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-17T18:16:35.394 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-17T18:16:35.400 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 30/35 2024-09-17T18:16:35.431 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-17T18:16:35.432 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 30/35 2024-09-17T18:16:35.760 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-17T18:16:35.808 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 23/35 2024-09-17T18:16:35.898 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 23/35 2024-09-17T18:16:35.929 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 24/35 2024-09-17T18:16:35.966 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-17T18:16:36.007 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 30/35 2024-09-17T18:16:36.044 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 31/35 2024-09-17T18:16:36.142 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 31/35 2024-09-17T18:16:36.142 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-17T18:16:36.142 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-17T18:16:36.142 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:16:36.170 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 32/35 2024-09-17T18:16:36.453 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-17T18:16:36.485 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 27/35 2024-09-17T18:16:36.556 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 27/35 2024-09-17T18:16:36.625 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 28/35 2024-09-17T18:16:36.654 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-common-2:19.3.0-4975.g1889a165.el9.x8 29/35 2024-09-17T18:16:38.954 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 30/35 2024-09-17T18:16:39.000 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 30/35 2024-09-17T18:16:39.568 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 30/35 2024-09-17T18:16:39.603 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 31/35 2024-09-17T18:16:39.695 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 31/35 2024-09-17T18:16:39.695 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-17T18:16:39.695 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-17T18:16:39.695 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:16:39.788 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 32/35 2024-09-17T18:16:46.081 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 32/35 2024-09-17T18:16:46.081 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /sys 2024-09-17T18:16:46.081 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /proc 2024-09-17T18:16:46.081 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /mnt 2024-09-17T18:16:46.081 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /var/tmp 2024-09-17T18:16:46.081 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /home 2024-09-17T18:16:46.082 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /root 2024-09-17T18:16:46.082 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /tmp 2024-09-17T18:16:46.082 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:16:47.115 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 33/35 2024-09-17T18:16:47.141 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 33/35 2024-09-17T18:16:47.141 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T18:16:47.141 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-17T18:16:47.141 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T18:16:47.141 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T18:16:47.141 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:16:47.143 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-17T18:16:47.226 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-17T18:16:47.226 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-17T18:16:48.436 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-17T18:16:48.436 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 1/35 2024-09-17T18:16:48.436 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 2/35 2024-09-17T18:16:48.436 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 3/35 2024-09-17T18:16:48.437 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 4/35 2024-09-17T18:16:48.437 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 5/35 2024-09-17T18:16:48.437 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_6 6/35 2024-09-17T18:16:48.437 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 7/35 2024-09-17T18:16:48.437 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4975.g1889a165.el9. 8/35 2024-09-17T18:16:48.437 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-common-2:19.3.0-4975.g1889a165.el9.x8 9/35 2024-09-17T18:16:48.437 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 10/35 2024-09-17T18:16:48.437 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 11/35 2024-09-17T18:16:48.437 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 12/35 2024-09-17T18:16:48.437 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 13/35 2024-09-17T18:16:48.437 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-17T18:16:48.437 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-17T18:16:48.437 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-17T18:16:48.438 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-17T18:16:48.438 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-17T18:16:48.438 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-17T18:16:48.438 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-17T18:16:48.438 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-17T18:16:48.438 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-17T18:16:48.438 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-17T18:16:48.438 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-17T18:16:48.438 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-17T18:16:48.438 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-17T18:16:48.438 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-17T18:16:48.438 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-17T18:16:48.438 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-17T18:16:48.438 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-17T18:16:48.439 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-17T18:16:48.439 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-2:19.3.0-4975.g1889a165.el9.x86_64 32/35 2024-09-17T18:16:48.439 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-17T18:16:48.439 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 34/35 2024-09-17T18:16:49.381 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-17T18:16:49.382 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:16:49.382 INFO:teuthology.orchestra.run.smithi121.stdout:Upgraded: 2024-09-17T18:16:49.382 INFO:teuthology.orchestra.run.smithi121.stdout: librados2-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:16:49.382 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:16:49.382 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-17T18:16:49.382 INFO:teuthology.orchestra.run.smithi121.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-17T18:16:49.382 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:16:49.383 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:16:49.383 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:16:49.383 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:16:49.383 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-17T18:16:49.383 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-17T18:16:49.383 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-17T18:16:49.384 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:16:49.384 INFO:teuthology.orchestra.run.smithi121.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-17T18:16:49.384 INFO:teuthology.orchestra.run.smithi121.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-17T18:16:49.384 INFO:teuthology.orchestra.run.smithi121.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-17T18:16:49.384 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-17T18:16:49.384 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:16:49.384 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-17T18:16:49.385 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:16:49.385 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-17T18:16:49.385 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-17T18:16:49.385 INFO:teuthology.orchestra.run.smithi121.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-17T18:16:49.385 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:16:49.385 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:16:49.385 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:16:49.386 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-17T18:16:49.386 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:16:49.386 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:16:49.386 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:16:49.386 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-17T18:16:49.386 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-17T18:16:49.386 INFO:teuthology.orchestra.run.smithi121.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-17T18:16:49.386 INFO:teuthology.orchestra.run.smithi121.stdout: re2-1:20211101-20.el9.x86_64 2024-09-17T18:16:49.386 INFO:teuthology.orchestra.run.smithi121.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-17T18:16:49.386 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:16:49.386 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-17T18:16:49.718 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-test 2024-09-17T18:16:49.877 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 32/35 2024-09-17T18:16:49.877 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /sys 2024-09-17T18:16:49.877 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /proc 2024-09-17T18:16:49.878 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /mnt 2024-09-17T18:16:49.878 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /var/tmp 2024-09-17T18:16:49.878 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /home 2024-09-17T18:16:49.878 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /root 2024-09-17T18:16:49.878 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /tmp 2024-09-17T18:16:49.878 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:16:50.300 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:29 ago on Tue 17 Sep 2024 06:16:21 PM UTC. 2024-09-17T18:16:50.432 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-17T18:16:50.433 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:16:50.433 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-09-17T18:16:50.433 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:16:50.433 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-17T18:16:50.433 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test x86_64 2:19.3.0-4975.g1889a165.el9 ceph 46 M 2024-09-17T18:16:50.433 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-09-17T18:16:50.434 INFO:teuthology.orchestra.run.smithi121.stdout: libcephsqlite x86_64 2:19.3.0-4975.g1889a165.el9 ceph 162 k 2024-09-17T18:16:50.434 INFO:teuthology.orchestra.run.smithi121.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-17T18:16:50.434 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-17T18:16:50.434 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:16:50.434 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-17T18:16:50.434 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:16:50.434 INFO:teuthology.orchestra.run.smithi121.stdout:Install 4 Packages 2024-09-17T18:16:50.434 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:16:50.435 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 46 M 2024-09-17T18:16:50.435 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 201 M 2024-09-17T18:16:50.435 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-17T18:16:50.747 INFO:teuthology.orchestra.run.smithi121.stdout:(1/4): libcephsqlite-19.3.0-4975.g1889a165.el9. 770 kB/s | 162 kB 00:00 2024-09-17T18:16:50.773 INFO:teuthology.orchestra.run.smithi121.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.3 MB/s | 304 kB 00:00 2024-09-17T18:16:50.853 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 33/35 2024-09-17T18:16:50.881 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 33/35 2024-09-17T18:16:50.881 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T18:16:50.881 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-17T18:16:50.881 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T18:16:50.881 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T18:16:50.882 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:16:50.883 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-17T18:16:50.898 INFO:teuthology.orchestra.run.smithi121.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 424 kB/s | 64 kB 00:00 2024-09-17T18:16:50.988 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-17T18:16:50.989 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-17T18:16:52.262 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-17T18:16:52.263 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 1/35 2024-09-17T18:16:52.263 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 2/35 2024-09-17T18:16:52.263 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 3/35 2024-09-17T18:16:52.263 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 4/35 2024-09-17T18:16:52.264 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 5/35 2024-09-17T18:16:52.264 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_6 6/35 2024-09-17T18:16:52.265 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 7/35 2024-09-17T18:16:52.265 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4975.g1889a165.el9. 8/35 2024-09-17T18:16:52.265 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-common-2:19.3.0-4975.g1889a165.el9.x8 9/35 2024-09-17T18:16:52.265 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 10/35 2024-09-17T18:16:52.265 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 11/35 2024-09-17T18:16:52.266 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 12/35 2024-09-17T18:16:52.266 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 13/35 2024-09-17T18:16:52.266 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-17T18:16:52.266 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-17T18:16:52.266 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-17T18:16:52.267 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-17T18:16:52.267 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-17T18:16:52.267 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-17T18:16:52.267 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-17T18:16:52.267 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-17T18:16:52.267 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-17T18:16:52.268 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-17T18:16:52.268 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-17T18:16:52.268 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-17T18:16:52.268 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-17T18:16:52.268 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-17T18:16:52.268 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-17T18:16:52.269 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-17T18:16:52.269 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-17T18:16:52.269 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-17T18:16:52.269 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-2:19.3.0-4975.g1889a165.el9.x86_64 32/35 2024-09-17T18:16:52.269 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-17T18:16:52.269 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 34/35 2024-09-17T18:16:52.573 INFO:teuthology.orchestra.run.smithi121.stdout:(4/4): ceph-test-19.3.0-4975.g1889a165.el9.x86_ 22 MB/s | 46 MB 00:02 2024-09-17T18:16:52.576 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-17T18:16:52.576 INFO:teuthology.orchestra.run.smithi121.stdout:Total 21 MB/s | 46 MB 00:02 2024-09-17T18:16:52.603 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-17T18:16:52.622 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-17T18:16:52.623 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-17T18:16:53.003 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-17T18:16:53.003 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-17T18:16:53.130 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-17T18:16:53.130 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:16:53.130 INFO:teuthology.orchestra.run.smithi173.stdout:Upgraded: 2024-09-17T18:16:53.130 INFO:teuthology.orchestra.run.smithi173.stdout: librados2-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:16:53.130 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:16:53.131 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T18:16:53.131 INFO:teuthology.orchestra.run.smithi173.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-17T18:16:53.131 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:16:53.131 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:16:53.131 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:16:53.131 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:16:53.131 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-17T18:16:53.131 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-17T18:16:53.131 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-17T18:16:53.131 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:16:53.131 INFO:teuthology.orchestra.run.smithi173.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-17T18:16:53.131 INFO:teuthology.orchestra.run.smithi173.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-17T18:16:53.131 INFO:teuthology.orchestra.run.smithi173.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-17T18:16:53.131 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-17T18:16:53.132 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:16:53.132 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-17T18:16:53.132 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:16:53.132 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-17T18:16:53.132 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-17T18:16:53.132 INFO:teuthology.orchestra.run.smithi173.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-17T18:16:53.132 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:16:53.132 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:16:53.132 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:16:53.132 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-17T18:16:53.132 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:16:53.132 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:16:53.132 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:16:53.132 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-17T18:16:53.133 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-17T18:16:53.133 INFO:teuthology.orchestra.run.smithi173.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-17T18:16:53.133 INFO:teuthology.orchestra.run.smithi173.stdout: re2-1:20211101-20.el9.x86_64 2024-09-17T18:16:53.133 INFO:teuthology.orchestra.run.smithi173.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-17T18:16:53.133 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:16:53.133 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T18:16:53.452 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-test 2024-09-17T18:16:53.507 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-17T18:16:53.582 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-17T18:16:53.631 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-17T18:16:53.658 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 3/4 2024-09-17T18:16:54.035 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:30 ago on Tue 17 Sep 2024 06:16:24 PM UTC. 2024-09-17T18:16:54.168 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T18:16:54.168 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:16:54.168 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-09-17T18:16:54.168 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:16:54.168 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T18:16:54.169 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test x86_64 2:19.3.0-4975.g1889a165.el9 ceph 46 M 2024-09-17T18:16:54.169 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-09-17T18:16:54.169 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite x86_64 2:19.3.0-4975.g1889a165.el9 ceph 162 k 2024-09-17T18:16:54.169 INFO:teuthology.orchestra.run.smithi173.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-17T18:16:54.169 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-17T18:16:54.169 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:16:54.169 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T18:16:54.169 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:16:54.169 INFO:teuthology.orchestra.run.smithi173.stdout:Install 4 Packages 2024-09-17T18:16:54.169 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:16:54.170 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 46 M 2024-09-17T18:16:54.170 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 201 M 2024-09-17T18:16:54.170 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T18:16:54.646 INFO:teuthology.orchestra.run.smithi173.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.4 MB/s | 304 kB 00:00 2024-09-17T18:16:54.671 INFO:teuthology.orchestra.run.smithi173.stdout:(2/4): libcephsqlite-19.3.0-4975.g1889a165.el9. 663 kB/s | 162 kB 00:00 2024-09-17T18:16:54.696 INFO:teuthology.orchestra.run.smithi173.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-17T18:16:55.855 INFO:teuthology.orchestra.run.smithi173.stdout:(4/4): ceph-test-19.3.0-4975.g1889a165.el9.x86_ 32 MB/s | 46 MB 00:01 2024-09-17T18:16:55.857 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T18:16:55.857 INFO:teuthology.orchestra.run.smithi173.stdout:Total 27 MB/s | 46 MB 00:01 2024-09-17T18:16:55.885 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T18:16:55.906 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T18:16:55.906 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T18:16:56.273 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T18:16:56.273 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T18:16:56.786 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T18:16:56.855 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-17T18:16:56.899 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-17T18:16:56.932 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 3/4 2024-09-17T18:16:58.033 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 3/4 2024-09-17T18:16:58.123 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 4/4 2024-09-17T18:17:00.418 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 4/4 2024-09-17T18:17:00.418 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 1/4 2024-09-17T18:17:00.418 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 2/4 2024-09-17T18:17:00.418 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-17T18:17:00.722 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-17T18:17:00.722 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:00.722 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-17T18:17:00.722 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:17:00.722 INFO:teuthology.orchestra.run.smithi121.stdout: libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:17:00.722 INFO:teuthology.orchestra.run.smithi121.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-17T18:17:00.722 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-17T18:17:00.722 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:00.722 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-17T18:17:01.039 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph 2024-09-17T18:17:01.298 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 3/4 2024-09-17T18:17:01.381 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 4/4 2024-09-17T18:17:01.641 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:40 ago on Tue 17 Sep 2024 06:16:21 PM UTC. 2024-09-17T18:17:01.784 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-17T18:17:01.786 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:17:01.787 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-09-17T18:17:01.787 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:17:01.787 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-17T18:17:01.787 INFO:teuthology.orchestra.run.smithi121.stdout: ceph x86_64 2:19.3.0-4975.g1889a165.el9 ceph 6.6 k 2024-09-17T18:17:01.787 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-09-17T18:17:01.787 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds x86_64 2:19.3.0-4975.g1889a165.el9 ceph 2.4 M 2024-09-17T18:17:01.787 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr x86_64 2:19.3.0-4975.g1889a165.el9 ceph 1.5 M 2024-09-17T18:17:01.787 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 291 k 2024-09-17T18:17:01.787 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon x86_64 2:19.3.0-4975.g1889a165.el9 ceph 4.8 M 2024-09-17T18:17:01.787 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd x86_64 2:19.3.0-4975.g1889a165.el9 ceph 17 M 2024-09-17T18:17:01.787 INFO:teuthology.orchestra.run.smithi121.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-17T18:17:01.788 INFO:teuthology.orchestra.run.smithi121.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-17T18:17:01.788 INFO:teuthology.orchestra.run.smithi121.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-17T18:17:01.788 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-17T18:17:01.788 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-17T18:17:01.788 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-17T18:17:01.788 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-17T18:17:01.788 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-17T18:17:01.788 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-17T18:17:01.788 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-17T18:17:01.788 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-collections 2024-09-17T18:17:01.788 INFO:teuthology.orchestra.run.smithi121.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-17T18:17:01.789 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-17T18:17:01.789 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-17T18:17:01.789 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-17T18:17:01.789 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-17T18:17:01.789 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-17T18:17:01.789 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-17T18:17:01.789 INFO:teuthology.orchestra.run.smithi121.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-17T18:17:01.789 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-17T18:17:01.789 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-17T18:17:01.789 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-17T18:17:01.789 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-17T18:17:01.790 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-17T18:17:01.790 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-17T18:17:01.790 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-17T18:17:01.790 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-17T18:17:01.790 INFO:teuthology.orchestra.run.smithi121.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-17T18:17:01.790 INFO:teuthology.orchestra.run.smithi121.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-17T18:17:01.790 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-17T18:17:01.790 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-17T18:17:01.790 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-17T18:17:01.790 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:01.790 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-17T18:17:01.791 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:17:01.791 INFO:teuthology.orchestra.run.smithi121.stdout:Install 37 Packages 2024-09-17T18:17:01.791 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:01.791 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 31 M 2024-09-17T18:17:01.791 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 109 M 2024-09-17T18:17:01.792 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-17T18:17:02.649 INFO:teuthology.orchestra.run.smithi121.stdout:(1/37): ceph-19.3.0-4975.g1889a165.el9.x86_64.r 55 kB/s | 6.6 kB 00:00 2024-09-17T18:17:02.833 INFO:teuthology.orchestra.run.smithi121.stdout:(2/37): ceph-mgr-19.3.0-4975.g1889a165.el9.x86_ 5.0 MB/s | 1.5 MB 00:00 2024-09-17T18:17:02.875 INFO:teuthology.orchestra.run.smithi121.stdout:(3/37): ceph-mds-19.3.0-4975.g1889a165.el9.x86_ 6.9 MB/s | 2.4 MB 00:00 2024-09-17T18:17:02.942 INFO:teuthology.orchestra.run.smithi121.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4975.g1889 4.3 MB/s | 291 kB 00:00 2024-09-17T18:17:03.017 INFO:teuthology.orchestra.run.smithi121.stdout:(5/37): ceph-mon-19.3.0-4975.g1889a165.el9.x86_ 13 MB/s | 4.8 MB 00:00 2024-09-17T18:17:03.134 INFO:teuthology.orchestra.run.smithi121.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.3 MB/s | 253 kB 00:00 2024-09-17T18:17:03.176 INFO:teuthology.orchestra.run.smithi121.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-17T18:17:03.218 INFO:teuthology.orchestra.run.smithi121.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 3.2 MB/s | 135 kB 00:00 2024-09-17T18:17:03.260 INFO:teuthology.orchestra.run.smithi121.stdout:(9/37): python3-cryptography-36.0.1-4.el9.x86_6 5.1 MB/s | 1.2 MB 00:00 2024-09-17T18:17:03.285 INFO:teuthology.orchestra.run.smithi121.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 1.8 MB/s | 125 kB 00:00 2024-09-17T18:17:03.477 INFO:teuthology.orchestra.run.smithi121.stdout:(11/37): ceph-osd-19.3.0-4975.g1889a165.el9.x86 27 MB/s | 17 MB 00:00 2024-09-17T18:17:03.503 INFO:teuthology.orchestra.run.smithi121.stdout:(12/37): python3-urllib3-1.26.5-6.el9.noarch.rp 886 kB/s | 215 kB 00:00 2024-09-17T18:17:03.603 INFO:teuthology.orchestra.run.smithi121.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 348 kB/s | 35 kB 00:00 2024-09-17T18:17:03.636 INFO:teuthology.orchestra.run.smithi121.stdout:(14/37): lua-5.4.4-4.el9.x86_64.rpm 537 kB/s | 188 kB 00:00 2024-09-17T18:17:03.658 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 4/4 2024-09-17T18:17:03.659 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 1/4 2024-09-17T18:17:03.659 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 2/4 2024-09-17T18:17:03.659 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-17T18:17:03.662 INFO:teuthology.orchestra.run.smithi121.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 936 kB/s | 172 kB 00:00 2024-09-17T18:17:03.687 INFO:teuthology.orchestra.run.smithi121.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-17T18:17:03.712 INFO:teuthology.orchestra.run.smithi121.stdout:(17/37): python3-packaging-20.9-5.el9.noarch.rp 706 kB/s | 77 kB 00:00 2024-09-17T18:17:03.737 INFO:teuthology.orchestra.run.smithi121.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 552 kB/s | 42 kB 00:00 2024-09-17T18:17:03.779 INFO:teuthology.orchestra.run.smithi121.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-09-17T18:17:03.805 INFO:teuthology.orchestra.run.smithi121.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 650 kB/s | 43 kB 00:00 2024-09-17T18:17:03.830 INFO:teuthology.orchestra.run.smithi121.stdout:(21/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 3.5 MB/s | 172 kB 00:00 2024-09-17T18:17:03.857 INFO:teuthology.orchestra.run.smithi121.stdout:(22/37): lua-devel-5.4.4-4.el9.x86_64.rpm 133 kB/s | 22 kB 00:00 2024-09-17T18:17:03.881 INFO:teuthology.orchestra.run.smithi121.stdout:(23/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 212 kB/s | 11 kB 00:00 2024-09-17T18:17:03.914 INFO:teuthology.orchestra.run.smithi121.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 875 kB/s | 23 kB 00:00 2024-09-17T18:17:03.939 INFO:teuthology.orchestra.run.smithi121.stdout:(25/37): python3-jaraco-classes-3.2.1-5.el9.noa 211 kB/s | 18 kB 00:00 2024-09-17T18:17:03.964 INFO:teuthology.orchestra.run.smithi121.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 402 kB/s | 19 kB 00:00 2024-09-17T18:17:03.990 INFO:teuthology.orchestra.run.smithi121.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 789 kB/s | 20 kB 00:00 2024-09-17T18:17:04.015 INFO:teuthology.orchestra.run.smithi121.stdout:(28/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.7 MB/s | 358 kB 00:00 2024-09-17T18:17:04.027 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-17T18:17:04.027 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:04.028 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T18:17:04.028 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:17:04.028 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:17:04.028 INFO:teuthology.orchestra.run.smithi173.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-17T18:17:04.028 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-17T18:17:04.028 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:04.028 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T18:17:04.041 INFO:teuthology.orchestra.run.smithi121.stdout:(29/37): python3-logutils-0.3.5-21.el9.noarch.r 610 kB/s | 46 kB 00:00 2024-09-17T18:17:04.066 INFO:teuthology.orchestra.run.smithi121.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-17T18:17:04.091 INFO:teuthology.orchestra.run.smithi121.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 328 kB/s | 16 kB 00:00 2024-09-17T18:17:04.117 INFO:teuthology.orchestra.run.smithi121.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-17T18:17:04.142 INFO:teuthology.orchestra.run.smithi121.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 710 kB/s | 36 kB 00:00 2024-09-17T18:17:04.175 INFO:teuthology.orchestra.run.smithi121.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-09-17T18:17:04.201 INFO:teuthology.orchestra.run.smithi121.stdout:(35/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-09-17T18:17:04.217 INFO:teuthology.orchestra.run.smithi121.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 480 kB/s | 20 kB 00:00 2024-09-17T18:17:04.260 INFO:teuthology.orchestra.run.smithi121.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2024-09-17T18:17:04.265 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-17T18:17:04.265 INFO:teuthology.orchestra.run.smithi121.stdout:Total 12 MB/s | 31 MB 00:02 2024-09-17T18:17:04.276 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph 2024-09-17T18:17:04.410 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-17T18:17:04.450 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-17T18:17:04.450 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-17T18:17:04.877 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:40 ago on Tue 17 Sep 2024 06:16:24 PM UTC. 2024-09-17T18:17:04.950 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-17T18:17:04.951 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-17T18:17:05.025 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T18:17:05.028 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:17:05.028 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-09-17T18:17:05.028 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:17:05.028 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T18:17:05.028 INFO:teuthology.orchestra.run.smithi173.stdout: ceph x86_64 2:19.3.0-4975.g1889a165.el9 ceph 6.6 k 2024-09-17T18:17:05.028 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-09-17T18:17:05.028 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds x86_64 2:19.3.0-4975.g1889a165.el9 ceph 2.4 M 2024-09-17T18:17:05.028 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr x86_64 2:19.3.0-4975.g1889a165.el9 ceph 1.5 M 2024-09-17T18:17:05.028 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 291 k 2024-09-17T18:17:05.029 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon x86_64 2:19.3.0-4975.g1889a165.el9 ceph 4.8 M 2024-09-17T18:17:05.029 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd x86_64 2:19.3.0-4975.g1889a165.el9 ceph 17 M 2024-09-17T18:17:05.029 INFO:teuthology.orchestra.run.smithi173.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-17T18:17:05.029 INFO:teuthology.orchestra.run.smithi173.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-17T18:17:05.029 INFO:teuthology.orchestra.run.smithi173.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-17T18:17:05.029 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-17T18:17:05.029 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-17T18:17:05.029 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-17T18:17:05.029 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-17T18:17:05.029 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-17T18:17:05.029 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-17T18:17:05.029 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-17T18:17:05.029 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-collections 2024-09-17T18:17:05.030 INFO:teuthology.orchestra.run.smithi173.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-17T18:17:05.030 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-17T18:17:05.030 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-17T18:17:05.030 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-17T18:17:05.030 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-17T18:17:05.030 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-17T18:17:05.030 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-17T18:17:05.030 INFO:teuthology.orchestra.run.smithi173.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-17T18:17:05.030 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-17T18:17:05.030 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-17T18:17:05.030 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-17T18:17:05.030 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-17T18:17:05.030 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-17T18:17:05.031 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-17T18:17:05.031 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-17T18:17:05.031 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-17T18:17:05.031 INFO:teuthology.orchestra.run.smithi173.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-17T18:17:05.031 INFO:teuthology.orchestra.run.smithi173.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-17T18:17:05.031 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-17T18:17:05.031 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-17T18:17:05.031 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-17T18:17:05.031 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:05.031 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T18:17:05.031 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:17:05.031 INFO:teuthology.orchestra.run.smithi173.stdout:Install 37 Packages 2024-09-17T18:17:05.031 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:05.032 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 31 M 2024-09-17T18:17:05.032 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 109 M 2024-09-17T18:17:05.032 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T18:17:05.931 INFO:teuthology.orchestra.run.smithi173.stdout:(1/37): ceph-19.3.0-4975.g1889a165.el9.x86_64.r 52 kB/s | 6.6 kB 00:00 2024-09-17T18:17:06.131 INFO:teuthology.orchestra.run.smithi173.stdout:(2/37): ceph-mgr-19.3.0-4975.g1889a165.el9.x86_ 4.7 MB/s | 1.5 MB 00:00 2024-09-17T18:17:06.190 INFO:teuthology.orchestra.run.smithi173.stdout:(3/37): ceph-mds-19.3.0-4975.g1889a165.el9.x86_ 6.2 MB/s | 2.4 MB 00:00 2024-09-17T18:17:06.290 INFO:teuthology.orchestra.run.smithi173.stdout:(4/37): ceph-mon-19.3.0-4975.g1889a165.el9.x86_ 13 MB/s | 4.8 MB 00:00 2024-09-17T18:17:06.302 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-17T18:17:06.315 INFO:teuthology.orchestra.run.smithi173.stdout:(5/37): ceph-mgr-modules-core-19.3.0-4975.g1889 2.3 MB/s | 291 kB 00:00 2024-09-17T18:17:06.371 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-17T18:17:06.412 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-17T18:17:06.443 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-17T18:17:06.482 INFO:teuthology.orchestra.run.smithi173.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.3 MB/s | 253 kB 00:00 2024-09-17T18:17:06.515 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-17T18:17:06.533 INFO:teuthology.orchestra.run.smithi173.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-09-17T18:17:06.595 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-17T18:17:06.625 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-17T18:17:06.700 INFO:teuthology.orchestra.run.smithi173.stdout:(8/37): ceph-osd-19.3.0-4975.g1889a165.el9.x86_ 30 MB/s | 17 MB 00:00 2024-09-17T18:17:06.725 INFO:teuthology.orchestra.run.smithi173.stdout:(9/37): python3-pycparser-2.20-6.el9.noarch.rpm 705 kB/s | 135 kB 00:00 2024-09-17T18:17:06.726 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-17T18:17:06.767 INFO:teuthology.orchestra.run.smithi173.stdout:(10/37): python3-cryptography-36.0.1-4.el9.x86_ 2.7 MB/s | 1.2 MB 00:00 2024-09-17T18:17:06.803 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-17T18:17:06.809 INFO:teuthology.orchestra.run.smithi173.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 2.5 MB/s | 215 kB 00:00 2024-09-17T18:17:06.853 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-17T18:17:06.888 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-17T18:17:06.909 INFO:teuthology.orchestra.run.smithi173.stdout:(12/37): python3-requests-2.25.1-8.el9.noarch.r 598 kB/s | 125 kB 00:00 2024-09-17T18:17:06.918 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-17T18:17:06.934 INFO:teuthology.orchestra.run.smithi173.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 1.1 MB/s | 188 kB 00:00 2024-09-17T18:17:06.952 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-17T18:17:06.959 INFO:teuthology.orchestra.run.smithi173.stdout:(14/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-09-17T18:17:06.986 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-17T18:17:06.993 INFO:teuthology.orchestra.run.smithi173.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 1.3 MB/s | 77 kB 00:00 2024-09-17T18:17:07.018 INFO:teuthology.orchestra.run.smithi173.stdout:(16/37): python3-markupsafe-1.1.1-12.el9.x86_64 319 kB/s | 35 kB 00:00 2024-09-17T18:17:07.026 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-17T18:17:07.044 INFO:teuthology.orchestra.run.smithi173.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 607 kB/s | 51 kB 00:00 2024-09-17T18:17:07.061 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-17T18:17:07.069 INFO:teuthology.orchestra.run.smithi173.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 551 kB/s | 42 kB 00:00 2024-09-17T18:17:07.096 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-17T18:17:07.111 INFO:teuthology.orchestra.run.smithi173.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-09-17T18:17:07.136 INFO:teuthology.orchestra.run.smithi173.stdout:(20/37): lua-devel-5.4.4-4.el9.x86_64.rpm 190 kB/s | 22 kB 00:00 2024-09-17T18:17:07.144 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-17T18:17:07.170 INFO:teuthology.orchestra.run.smithi173.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 433 kB/s | 43 kB 00:00 2024-09-17T18:17:07.177 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-17T18:17:07.195 INFO:teuthology.orchestra.run.smithi173.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 461 kB/s | 11 kB 00:00 2024-09-17T18:17:07.223 INFO:teuthology.orchestra.run.smithi173.stdout:(23/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2024-09-17T18:17:07.249 INFO:teuthology.orchestra.run.smithi173.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 330 kB/s | 18 kB 00:00 2024-09-17T18:17:07.258 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-17T18:17:07.274 INFO:teuthology.orchestra.run.smithi173.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 458 kB/s | 23 kB 00:00 2024-09-17T18:17:07.299 INFO:teuthology.orchestra.run.smithi173.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 385 kB/s | 19 kB 00:00 2024-09-17T18:17:07.325 INFO:teuthology.orchestra.run.smithi173.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-09-17T18:17:07.350 INFO:teuthology.orchestra.run.smithi173.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 923 kB/s | 46 kB 00:00 2024-09-17T18:17:07.375 INFO:teuthology.orchestra.run.smithi173.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-17T18:17:07.401 INFO:teuthology.orchestra.run.smithi173.stdout:(30/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-09-17T18:17:07.405 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-17T18:17:07.426 INFO:teuthology.orchestra.run.smithi173.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-09-17T18:17:07.451 INFO:teuthology.orchestra.run.smithi173.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-17T18:17:07.476 INFO:teuthology.orchestra.run.smithi173.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-09-17T18:17:07.484 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-17T18:17:07.502 INFO:teuthology.orchestra.run.smithi173.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-17T18:17:07.527 INFO:teuthology.orchestra.run.smithi173.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 805 kB/s | 20 kB 00:00 2024-09-17T18:17:07.537 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-17T18:17:07.552 INFO:teuthology.orchestra.run.smithi173.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-17T18:17:07.576 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-17T18:17:07.577 INFO:teuthology.orchestra.run.smithi173.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-17T18:17:07.582 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T18:17:07.582 INFO:teuthology.orchestra.run.smithi173.stdout:Total 12 MB/s | 31 MB 00:02 2024-09-17T18:17:07.633 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-17T18:17:07.698 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-17T18:17:07.737 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T18:17:07.776 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T18:17:07.776 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T18:17:07.893 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-17T18:17:07.941 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-17T18:17:08.020 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-17T18:17:08.109 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-17T18:17:08.141 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-17T18:17:08.259 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T18:17:08.260 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T18:17:08.276 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-17T18:17:08.438 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4975.g1889a165.el9. 32/37 2024-09-17T18:17:08.468 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 33/37 2024-09-17T18:17:08.494 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 33/37 2024-09-17T18:17:08.495 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T18:17:08.495 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-17T18:17:08.495 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T18:17:08.495 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T18:17:08.495 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:09.642 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T18:17:09.714 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-17T18:17:09.764 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-17T18:17:09.799 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-17T18:17:09.871 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-17T18:17:09.923 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-17T18:17:09.956 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-17T18:17:09.960 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 34/37 2024-09-17T18:17:09.987 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 34/37 2024-09-17T18:17:09.987 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T18:17:09.987 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-17T18:17:09.987 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T18:17:09.987 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T18:17:09.988 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:10.049 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-17T18:17:10.127 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-17T18:17:10.166 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-17T18:17:10.202 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-17T18:17:10.232 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-17T18:17:10.266 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-17T18:17:10.300 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-17T18:17:10.341 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-17T18:17:10.384 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-17T18:17:10.418 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-17T18:17:10.452 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 35/37 2024-09-17T18:17:10.469 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-17T18:17:10.476 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 35/37 2024-09-17T18:17:10.476 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T18:17:10.476 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-17T18:17:10.476 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T18:17:10.476 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T18:17:10.476 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:10.507 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-17T18:17:10.580 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-17T18:17:10.703 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 36/37 2024-09-17T18:17:10.726 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-17T18:17:10.726 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 36/37 2024-09-17T18:17:10.727 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T18:17:10.727 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-17T18:17:10.727 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T18:17:10.727 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T18:17:10.727 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:10.791 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-2:19.3.0-4975.g1889a165.el9.x86_64 37/37 2024-09-17T18:17:10.816 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-17T18:17:10.869 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-17T18:17:10.907 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-17T18:17:10.956 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-17T18:17:11.011 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-17T18:17:11.196 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-17T18:17:11.244 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-17T18:17:11.322 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-17T18:17:11.410 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-17T18:17:11.445 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-17T18:17:11.573 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-17T18:17:11.741 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4975.g1889a165.el9. 32/37 2024-09-17T18:17:11.771 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 33/37 2024-09-17T18:17:11.797 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 33/37 2024-09-17T18:17:11.797 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T18:17:11.797 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-17T18:17:11.797 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T18:17:11.797 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T18:17:11.797 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:12.602 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-2:19.3.0-4975.g1889a165.el9.x86_64 37/37 2024-09-17T18:17:12.602 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-2:19.3.0-4975.g1889a165.el9.x86_64 1/37 2024-09-17T18:17:12.602 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 2/37 2024-09-17T18:17:12.602 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 3/37 2024-09-17T18:17:12.602 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 4/37 2024-09-17T18:17:12.603 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 5/37 2024-09-17T18:17:12.603 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4975.g1889a165.el9. 6/37 2024-09-17T18:17:12.603 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-17T18:17:12.603 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-17T18:17:12.603 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-17T18:17:12.603 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-17T18:17:12.603 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-17T18:17:12.603 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-17T18:17:12.603 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-17T18:17:12.603 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-17T18:17:12.603 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-17T18:17:12.603 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-17T18:17:12.604 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-17T18:17:12.604 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-17T18:17:12.604 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-17T18:17:12.604 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-17T18:17:12.604 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-17T18:17:12.604 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-17T18:17:12.604 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-17T18:17:12.604 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-17T18:17:12.604 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-17T18:17:12.604 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-17T18:17:12.604 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-17T18:17:12.604 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-17T18:17:12.606 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-17T18:17:12.606 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-17T18:17:12.606 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-17T18:17:12.606 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-17T18:17:12.606 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-17T18:17:12.606 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-17T18:17:12.606 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-17T18:17:12.606 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-17T18:17:13.263 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 34/37 2024-09-17T18:17:13.291 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 34/37 2024-09-17T18:17:13.291 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T18:17:13.291 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-17T18:17:13.291 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T18:17:13.291 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T18:17:13.291 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:13.624 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-17T18:17:13.625 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:13.625 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-17T18:17:13.625 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:17:13.625 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:17:13.625 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:17:13.625 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T18:17:13.625 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:17:13.625 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:17:13.625 INFO:teuthology.orchestra.run.smithi121.stdout: lua-5.4.4-4.el9.x86_64 2024-09-17T18:17:13.625 INFO:teuthology.orchestra.run.smithi121.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-17T18:17:13.626 INFO:teuthology.orchestra.run.smithi121.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-17T18:17:13.626 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-17T18:17:13.626 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-17T18:17:13.626 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-17T18:17:13.626 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-17T18:17:13.626 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-17T18:17:13.626 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-17T18:17:13.626 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-17T18:17:13.626 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-17T18:17:13.626 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-17T18:17:13.626 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-17T18:17:13.626 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-17T18:17:13.627 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-17T18:17:13.627 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-17T18:17:13.627 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-17T18:17:13.627 INFO:teuthology.orchestra.run.smithi121.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-17T18:17:13.627 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-17T18:17:13.627 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ply-3.11-14.el9.noarch 2024-09-17T18:17:13.627 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-17T18:17:13.627 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-17T18:17:13.627 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-17T18:17:13.627 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-17T18:17:13.627 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-17T18:17:13.627 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-17T18:17:13.628 INFO:teuthology.orchestra.run.smithi121.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-17T18:17:13.628 INFO:teuthology.orchestra.run.smithi121.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-17T18:17:13.628 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-17T18:17:13.628 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-17T18:17:13.628 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-17T18:17:13.628 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:13.628 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-17T18:17:13.755 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 35/37 2024-09-17T18:17:13.779 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 35/37 2024-09-17T18:17:13.779 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T18:17:13.779 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-17T18:17:13.780 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T18:17:13.780 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T18:17:13.780 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:13.991 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-base 2024-09-17T18:17:14.006 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 36/37 2024-09-17T18:17:14.030 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 36/37 2024-09-17T18:17:14.030 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T18:17:14.030 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-17T18:17:14.030 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T18:17:14.030 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T18:17:14.031 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:14.095 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-2:19.3.0-4975.g1889a165.el9.x86_64 37/37 2024-09-17T18:17:14.586 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:53 ago on Tue 17 Sep 2024 06:16:21 PM UTC. 2024-09-17T18:17:14.675 INFO:teuthology.orchestra.run.smithi121.stdout:Package ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T18:17:14.717 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-17T18:17:14.718 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-09-17T18:17:14.718 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-17T18:17:14.779 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install cephadm 2024-09-17T18:17:15.350 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:54 ago on Tue 17 Sep 2024 06:16:21 PM UTC. 2024-09-17T18:17:15.480 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-17T18:17:15.480 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:17:15.481 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-09-17T18:17:15.481 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:17:15.481 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-17T18:17:15.481 INFO:teuthology.orchestra.run.smithi121.stdout: cephadm noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 784 k 2024-09-17T18:17:15.481 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:15.481 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-17T18:17:15.481 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:17:15.481 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-09-17T18:17:15.481 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:15.481 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 784 k 2024-09-17T18:17:15.481 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 791 k 2024-09-17T18:17:15.481 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-17T18:17:15.743 INFO:teuthology.orchestra.run.smithi121.stdout:cephadm-19.3.0-4975.g1889a165.el9.noarch.rpm 2.9 MB/s | 784 kB 00:00 2024-09-17T18:17:15.744 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-17T18:17:15.744 INFO:teuthology.orchestra.run.smithi121.stdout:Total 2.9 MB/s | 784 kB 00:00 2024-09-17T18:17:15.744 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-17T18:17:15.749 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-17T18:17:15.749 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-17T18:17:15.755 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-17T18:17:15.756 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-17T18:17:15.827 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-17T18:17:15.896 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-2:19.3.0-4975.g1889a165.el9.x86_64 37/37 2024-09-17T18:17:15.896 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-2:19.3.0-4975.g1889a165.el9.x86_64 1/37 2024-09-17T18:17:15.896 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 2/37 2024-09-17T18:17:15.896 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 3/37 2024-09-17T18:17:15.897 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 4/37 2024-09-17T18:17:15.897 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 5/37 2024-09-17T18:17:15.897 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4975.g1889a165.el9. 6/37 2024-09-17T18:17:15.897 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-17T18:17:15.897 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-17T18:17:15.897 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-17T18:17:15.897 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-17T18:17:15.897 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-17T18:17:15.897 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-17T18:17:15.897 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-17T18:17:15.897 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-17T18:17:15.897 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-17T18:17:15.897 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-17T18:17:15.897 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-17T18:17:15.898 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-17T18:17:15.898 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-17T18:17:15.898 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-17T18:17:15.898 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-17T18:17:15.898 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-17T18:17:15.898 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-17T18:17:15.898 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-17T18:17:15.898 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-17T18:17:15.898 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-17T18:17:15.898 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-17T18:17:15.898 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-17T18:17:15.898 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-17T18:17:15.898 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-17T18:17:15.899 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-17T18:17:15.899 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-17T18:17:15.899 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-17T18:17:15.899 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-17T18:17:15.899 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-17T18:17:15.899 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-17T18:17:16.263 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T18:17:16.355 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T18:17:16.679 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T18:17:16.927 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-17T18:17:16.928 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:16.928 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T18:17:16.928 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:17:16.928 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:17:16.928 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:17:16.928 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T18:17:16.928 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:17:16.928 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:17:16.928 INFO:teuthology.orchestra.run.smithi173.stdout: lua-5.4.4-4.el9.x86_64 2024-09-17T18:17:16.928 INFO:teuthology.orchestra.run.smithi173.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-17T18:17:16.929 INFO:teuthology.orchestra.run.smithi173.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-17T18:17:16.929 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-17T18:17:16.929 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-17T18:17:16.929 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-17T18:17:16.929 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-17T18:17:16.929 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-17T18:17:16.929 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-17T18:17:16.929 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-17T18:17:16.930 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-17T18:17:16.930 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-17T18:17:16.930 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-17T18:17:16.930 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-17T18:17:16.930 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-17T18:17:16.930 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-17T18:17:16.930 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-17T18:17:16.930 INFO:teuthology.orchestra.run.smithi173.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-17T18:17:16.930 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-17T18:17:16.930 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ply-3.11-14.el9.noarch 2024-09-17T18:17:16.931 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-17T18:17:16.931 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-17T18:17:16.931 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-17T18:17:16.931 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-17T18:17:16.931 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-17T18:17:16.931 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-17T18:17:16.931 INFO:teuthology.orchestra.run.smithi173.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-17T18:17:16.931 INFO:teuthology.orchestra.run.smithi173.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-17T18:17:16.931 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-17T18:17:16.931 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-17T18:17:16.931 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-17T18:17:16.932 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:16.932 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T18:17:16.932 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T18:17:16.933 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:16.933 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-17T18:17:16.933 INFO:teuthology.orchestra.run.smithi121.stdout: cephadm-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T18:17:16.933 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:16.933 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-17T18:17:17.131 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-immutable-object-cache 2024-09-17T18:17:17.329 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-base 2024-09-17T18:17:17.724 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:56 ago on Tue 17 Sep 2024 06:16:21 PM UTC. 2024-09-17T18:17:17.856 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-17T18:17:17.857 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:17:17.857 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-09-17T18:17:17.857 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:17:17.857 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-17T18:17:17.857 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4975.g1889a165.el9 ceph 145 k 2024-09-17T18:17:17.857 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:17.857 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-17T18:17:17.857 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:17:17.858 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-09-17T18:17:17.858 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:17.858 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 145 k 2024-09-17T18:17:17.858 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 432 k 2024-09-17T18:17:17.858 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-17T18:17:17.912 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:53 ago on Tue 17 Sep 2024 06:16:24 PM UTC. 2024-09-17T18:17:18.001 INFO:teuthology.orchestra.run.smithi173.stdout:Package ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T18:17:18.043 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T18:17:18.044 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-09-17T18:17:18.045 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T18:17:18.068 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-immutable-object-cache-19.3.0-4975.g1889a1 692 kB/s | 145 kB 00:00 2024-09-17T18:17:18.068 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-17T18:17:18.069 INFO:teuthology.orchestra.run.smithi121.stdout:Total 686 kB/s | 145 kB 00:00 2024-09-17T18:17:18.069 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-17T18:17:18.076 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-17T18:17:18.077 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-17T18:17:18.111 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install cephadm 2024-09-17T18:17:18.134 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-17T18:17:18.134 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-17T18:17:18.288 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-17T18:17:18.386 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4975.g1889a165. 1/1 2024-09-17T18:17:18.411 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4975.g1889a165. 1/1 2024-09-17T18:17:18.411 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T18:17:18.411 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-17T18:17:18.411 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:18.689 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:54 ago on Tue 17 Sep 2024 06:16:24 PM UTC. 2024-09-17T18:17:18.820 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T18:17:18.821 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:17:18.821 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-09-17T18:17:18.821 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:17:18.821 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T18:17:18.822 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 784 k 2024-09-17T18:17:18.822 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:18.822 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T18:17:18.822 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:17:18.822 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-09-17T18:17:18.822 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:18.822 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 784 k 2024-09-17T18:17:18.822 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 791 k 2024-09-17T18:17:18.822 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T18:17:19.073 INFO:teuthology.orchestra.run.smithi173.stdout:cephadm-19.3.0-4975.g1889a165.el9.noarch.rpm 3.1 MB/s | 784 kB 00:00 2024-09-17T18:17:19.074 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T18:17:19.074 INFO:teuthology.orchestra.run.smithi173.stdout:Total 3.0 MB/s | 784 kB 00:00 2024-09-17T18:17:19.074 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T18:17:19.079 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T18:17:19.079 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T18:17:19.085 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T18:17:19.086 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T18:17:19.155 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T18:17:19.313 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4975.g1889a165. 1/1 2024-09-17T18:17:19.313 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:19.313 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-17T18:17:19.313 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:17:19.313 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:19.313 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-17T18:17:19.534 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr 2024-09-17T18:17:19.644 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T18:17:19.735 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T18:17:20.071 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T18:17:20.120 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:59 ago on Tue 17 Sep 2024 06:16:21 PM UTC. 2024-09-17T18:17:20.213 INFO:teuthology.orchestra.run.smithi121.stdout:Package ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T18:17:20.257 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-17T18:17:20.258 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-09-17T18:17:20.258 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-17T18:17:20.316 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-dashboard 2024-09-17T18:17:20.346 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T18:17:20.346 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:20.346 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T18:17:20.346 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T18:17:20.346 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:20.346 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T18:17:20.560 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-immutable-object-cache 2024-09-17T18:17:20.888 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:59 ago on Tue 17 Sep 2024 06:16:21 PM UTC. 2024-09-17T18:17:21.020 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-17T18:17:21.021 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:17:21.021 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-09-17T18:17:21.021 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:17:21.021 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-17T18:17:21.021 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 14 M 2024-09-17T18:17:21.021 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-09-17T18:17:21.021 INFO:teuthology.orchestra.run.smithi121.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-17T18:17:21.022 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 34 k 2024-09-17T18:17:21.022 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 16 k 2024-09-17T18:17:21.022 INFO:teuthology.orchestra.run.smithi121.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-17T18:17:21.022 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-17T18:17:21.022 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-17T18:17:21.022 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-17T18:17:21.022 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-17T18:17:21.022 INFO:teuthology.orchestra.run.smithi121.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-17T18:17:21.022 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-17T18:17:21.022 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-17T18:17:21.022 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:21.022 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-17T18:17:21.022 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:17:21.023 INFO:teuthology.orchestra.run.smithi121.stdout:Install 12 Packages 2024-09-17T18:17:21.023 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:21.023 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 19 M 2024-09-17T18:17:21.023 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 159 M 2024-09-17T18:17:21.024 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-17T18:17:21.145 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:57 ago on Tue 17 Sep 2024 06:16:24 PM UTC. 2024-09-17T18:17:21.279 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T18:17:21.279 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:17:21.280 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2024-09-17T18:17:21.280 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:17:21.280 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T18:17:21.280 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4975.g1889a165.el9 ceph 145 k 2024-09-17T18:17:21.280 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:21.280 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T18:17:21.280 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:17:21.280 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-09-17T18:17:21.280 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:21.280 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 145 k 2024-09-17T18:17:21.280 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 432 k 2024-09-17T18:17:21.280 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T18:17:21.396 INFO:teuthology.orchestra.run.smithi121.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4975.g18 242 kB/s | 34 kB 00:00 2024-09-17T18:17:21.421 INFO:teuthology.orchestra.run.smithi121.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4975.g188 99 kB/s | 16 kB 00:00 2024-09-17T18:17:21.487 INFO:teuthology.orchestra.run.smithi173.stdout:ceph-immutable-object-cache-19.3.0-4975.g1889a1 701 kB/s | 145 kB 00:00 2024-09-17T18:17:21.488 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T18:17:21.488 INFO:teuthology.orchestra.run.smithi173.stdout:Total 695 kB/s | 145 kB 00:00 2024-09-17T18:17:21.488 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T18:17:21.496 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T18:17:21.496 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T18:17:21.553 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T18:17:21.553 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T18:17:21.622 INFO:teuthology.orchestra.run.smithi121.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.3 MB/s | 269 kB 00:00 2024-09-17T18:17:21.706 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T18:17:21.788 INFO:teuthology.orchestra.run.smithi121.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.6 MB/s | 1.0 MB 00:00 2024-09-17T18:17:21.813 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4975.g1889a165. 1/1 2024-09-17T18:17:21.822 INFO:teuthology.orchestra.run.smithi121.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.2 MB/s | 863 kB 00:00 2024-09-17T18:17:21.839 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4975.g1889a165. 1/1 2024-09-17T18:17:21.839 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T18:17:21.840 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-17T18:17:21.840 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:21.989 INFO:teuthology.orchestra.run.smithi121.stdout:(6/12): ceph-mgr-dashboard-19.3.0-4975.g1889a16 19 MB/s | 14 MB 00:00 2024-09-17T18:17:22.031 INFO:teuthology.orchestra.run.smithi121.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 96 kB/s | 19 kB 00:00 2024-09-17T18:17:22.056 INFO:teuthology.orchestra.run.smithi121.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.0 MB/s | 548 kB 00:00 2024-09-17T18:17:22.082 INFO:teuthology.orchestra.run.smithi121.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-17T18:17:22.107 INFO:teuthology.orchestra.run.smithi121.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 1.9 MB/s | 144 kB 00:00 2024-09-17T18:17:22.132 INFO:teuthology.orchestra.run.smithi121.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-17T18:17:22.190 INFO:teuthology.orchestra.run.smithi121.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 10 MB/s | 2.0 MB 00:00 2024-09-17T18:17:22.194 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-17T18:17:22.194 INFO:teuthology.orchestra.run.smithi121.stdout:Total 16 MB/s | 19 MB 00:01 2024-09-17T18:17:22.241 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-17T18:17:22.260 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-17T18:17:22.260 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-17T18:17:22.554 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-17T18:17:22.554 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-17T18:17:22.778 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4975.g1889a165. 1/1 2024-09-17T18:17:22.778 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:22.778 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T18:17:22.778 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:17:22.779 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:22.779 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T18:17:22.991 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr 2024-09-17T18:17:23.165 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-17T18:17:23.241 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-17T18:17:23.285 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-17T18:17:23.332 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-17T18:17:23.362 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-17T18:17:23.472 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-17T18:17:23.584 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:59 ago on Tue 17 Sep 2024 06:16:24 PM UTC. 2024-09-17T18:17:23.604 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-17T18:17:23.675 INFO:teuthology.orchestra.run.smithi173.stdout:Package ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T18:17:23.706 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-17T18:17:23.717 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T18:17:23.718 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-09-17T18:17:23.718 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T18:17:23.759 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-17T18:17:23.785 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-dashboard 2024-09-17T18:17:23.787 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-17T18:17:23.820 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4975.g1889a165.el9 10/12 2024-09-17T18:17:24.358 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:00 ago on Tue 17 Sep 2024 06:16:24 PM UTC. 2024-09-17T18:17:24.493 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T18:17:24.494 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:17:24.494 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-09-17T18:17:24.494 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:17:24.494 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T18:17:24.494 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 14 M 2024-09-17T18:17:24.494 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-09-17T18:17:24.494 INFO:teuthology.orchestra.run.smithi173.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-17T18:17:24.494 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 34 k 2024-09-17T18:17:24.495 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 16 k 2024-09-17T18:17:24.495 INFO:teuthology.orchestra.run.smithi173.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-17T18:17:24.495 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-17T18:17:24.495 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-17T18:17:24.495 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-17T18:17:24.495 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-17T18:17:24.495 INFO:teuthology.orchestra.run.smithi173.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-17T18:17:24.495 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-17T18:17:24.495 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-17T18:17:24.495 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:24.495 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T18:17:24.495 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:17:24.495 INFO:teuthology.orchestra.run.smithi173.stdout:Install 12 Packages 2024-09-17T18:17:24.495 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:24.496 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 19 M 2024-09-17T18:17:24.496 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 159 M 2024-09-17T18:17:24.496 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T18:17:24.791 INFO:teuthology.orchestra.run.smithi173.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4975.g18 238 kB/s | 34 kB 00:00 2024-09-17T18:17:24.816 INFO:teuthology.orchestra.run.smithi173.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4975.g188 97 kB/s | 16 kB 00:00 2024-09-17T18:17:24.983 INFO:teuthology.orchestra.run.smithi173.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.6 MB/s | 269 kB 00:00 2024-09-17T18:17:25.033 INFO:teuthology.orchestra.run.smithi173.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.2 MB/s | 1.0 MB 00:00 2024-09-17T18:17:25.084 INFO:teuthology.orchestra.run.smithi173.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 11 MB/s | 548 kB 00:00 2024-09-17T18:17:25.101 INFO:teuthology.orchestra.run.smithi173.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 1.1 MB/s | 19 kB 00:00 2024-09-17T18:17:25.168 INFO:teuthology.orchestra.run.smithi173.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 31 MB/s | 2.0 MB 00:00 2024-09-17T18:17:25.193 INFO:teuthology.orchestra.run.smithi173.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.7 MB/s | 144 kB 00:00 2024-09-17T18:17:25.210 INFO:teuthology.orchestra.run.smithi173.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.8 MB/s | 31 kB 00:00 2024-09-17T18:17:25.244 INFO:teuthology.orchestra.run.smithi173.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 5.5 MB/s | 188 kB 00:00 2024-09-17T18:17:25.277 INFO:teuthology.orchestra.run.smithi173.stdout:(11/12): protobuf-compiler-3.14.0-13.el9.x86_64 2.9 MB/s | 863 kB 00:00 2024-09-17T18:17:25.427 INFO:teuthology.orchestra.run.smithi173.stdout:(12/12): ceph-mgr-dashboard-19.3.0-4975.g1889a1 18 MB/s | 14 MB 00:00 2024-09-17T18:17:25.431 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T18:17:25.431 INFO:teuthology.orchestra.run.smithi173.stdout:Total 21 MB/s | 19 MB 00:00 2024-09-17T18:17:25.480 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T18:17:25.499 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T18:17:25.499 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T18:17:25.763 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T18:17:25.764 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T18:17:26.360 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T18:17:26.445 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-17T18:17:26.488 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-17T18:17:26.535 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-17T18:17:26.565 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-17T18:17:26.674 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-17T18:17:26.807 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-17T18:17:26.894 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-17T18:17:26.945 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-17T18:17:26.974 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-17T18:17:26.998 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4975.g1889a165.el 11/12 2024-09-17T18:17:27.005 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4975.g1889a165.el9 10/12 2024-09-17T18:17:27.057 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noa 12/12 2024-09-17T18:17:28.884 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noa 12/12 2024-09-17T18:17:28.884 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4975.g1889a165.el 1/12 2024-09-17T18:17:28.884 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noa 2/12 2024-09-17T18:17:28.884 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4975.g1889a165.el9 3/12 2024-09-17T18:17:28.884 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-17T18:17:28.884 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-17T18:17:28.884 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-17T18:17:28.884 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-17T18:17:28.884 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-17T18:17:28.884 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-17T18:17:28.885 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-17T18:17:28.885 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-17T18:17:29.203 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-17T18:17:29.203 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:29.204 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-17T18:17:29.204 INFO:teuthology.orchestra.run.smithi121.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-17T18:17:29.204 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-grafana-dashboards-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T18:17:29.204 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T18:17:29.204 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-prometheus-alerts-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T18:17:29.204 INFO:teuthology.orchestra.run.smithi121.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-17T18:17:29.204 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-17T18:17:29.204 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-17T18:17:29.204 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-17T18:17:29.204 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-17T18:17:29.204 INFO:teuthology.orchestra.run.smithi121.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-17T18:17:29.204 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-17T18:17:29.205 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-17T18:17:29.205 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:29.205 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-17T18:17:29.457 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-17T18:17:30.024 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:09 ago on Tue 17 Sep 2024 06:16:21 PM UTC. 2024-09-17T18:17:30.155 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-17T18:17:30.156 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================= 2024-09-17T18:17:30.156 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-09-17T18:17:30.156 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================= 2024-09-17T18:17:30.157 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-17T18:17:30.157 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 7.4 M 2024-09-17T18:17:30.157 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-09-17T18:17:30.157 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-17T18:17:30.157 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-17T18:17:30.157 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-17T18:17:30.157 INFO:teuthology.orchestra.run.smithi121.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-17T18:17:30.157 INFO:teuthology.orchestra.run.smithi121.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-17T18:17:30.157 INFO:teuthology.orchestra.run.smithi121.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-17T18:17:30.157 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-17T18:17:30.157 INFO:teuthology.orchestra.run.smithi121.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-17T18:17:30.157 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-17T18:17:30.157 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-17T18:17:30.158 INFO:teuthology.orchestra.run.smithi121.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-17T18:17:30.158 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:30.158 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-17T18:17:30.158 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================= 2024-09-17T18:17:30.158 INFO:teuthology.orchestra.run.smithi121.stdout:Install 12 Packages 2024-09-17T18:17:30.158 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:30.158 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 38 M 2024-09-17T18:17:30.159 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 212 M 2024-09-17T18:17:30.159 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-17T18:17:30.197 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4975.g1889a165.el 11/12 2024-09-17T18:17:30.251 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noa 12/12 2024-09-17T18:17:30.347 INFO:teuthology.orchestra.run.smithi121.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.3 MB/s | 191 kB 00:00 2024-09-17T18:17:30.389 INFO:teuthology.orchestra.run.smithi121.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.1 MB/s | 801 kB 00:00 2024-09-17T18:17:30.673 INFO:teuthology.orchestra.run.smithi121.stdout:(3/12): ceph-mgr-diskprediction-local-19.3.0-49 16 MB/s | 7.4 MB 00:00 2024-09-17T18:17:30.931 INFO:teuthology.orchestra.run.smithi121.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 56 kB/s | 33 kB 00:00 2024-09-17T18:17:31.165 INFO:teuthology.orchestra.run.smithi121.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 38 kB/s | 18 kB 00:00 2024-09-17T18:17:31.515 INFO:teuthology.orchestra.run.smithi121.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 66 kB/s | 38 kB 00:00 2024-09-17T18:17:32.212 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noa 12/12 2024-09-17T18:17:32.212 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4975.g1889a165.el 1/12 2024-09-17T18:17:32.212 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noa 2/12 2024-09-17T18:17:32.212 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4975.g1889a165.el9 3/12 2024-09-17T18:17:32.212 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-17T18:17:32.212 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-17T18:17:32.212 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-17T18:17:32.213 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-17T18:17:32.213 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-17T18:17:32.213 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-17T18:17:32.213 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-17T18:17:32.213 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-17T18:17:32.341 INFO:teuthology.orchestra.run.smithi121.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 297 kB/s | 245 kB 00:00 2024-09-17T18:17:32.561 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-17T18:17:32.561 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:32.561 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T18:17:32.561 INFO:teuthology.orchestra.run.smithi173.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-17T18:17:32.561 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-grafana-dashboards-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T18:17:32.561 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T18:17:32.561 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-prometheus-alerts-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T18:17:32.561 INFO:teuthology.orchestra.run.smithi173.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-17T18:17:32.561 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-17T18:17:32.562 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-17T18:17:32.562 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-17T18:17:32.562 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-17T18:17:32.562 INFO:teuthology.orchestra.run.smithi173.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-17T18:17:32.562 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-17T18:17:32.562 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-17T18:17:32.562 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:32.562 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T18:17:32.791 INFO:teuthology.orchestra.run.smithi121.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 3.1 MB/s | 4.9 MB 00:01 2024-09-17T18:17:32.799 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-17T18:17:32.925 INFO:teuthology.orchestra.run.smithi121.stdout:(9/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 1.2 MB/s | 3.0 MB 00:02 2024-09-17T18:17:33.372 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:09 ago on Tue 17 Sep 2024 06:16:24 PM UTC. 2024-09-17T18:17:33.500 INFO:teuthology.orchestra.run.smithi121.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.4 MB/s | 5.1 MB 00:01 2024-09-17T18:17:33.503 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T18:17:33.504 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================= 2024-09-17T18:17:33.505 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-09-17T18:17:33.505 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================= 2024-09-17T18:17:33.505 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T18:17:33.505 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 7.4 M 2024-09-17T18:17:33.505 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-09-17T18:17:33.505 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-17T18:17:33.505 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-17T18:17:33.505 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-17T18:17:33.505 INFO:teuthology.orchestra.run.smithi173.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-17T18:17:33.505 INFO:teuthology.orchestra.run.smithi173.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-17T18:17:33.506 INFO:teuthology.orchestra.run.smithi173.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-17T18:17:33.506 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-17T18:17:33.506 INFO:teuthology.orchestra.run.smithi173.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-17T18:17:33.506 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-17T18:17:33.506 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-17T18:17:33.506 INFO:teuthology.orchestra.run.smithi173.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-17T18:17:33.506 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:33.506 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T18:17:33.506 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================= 2024-09-17T18:17:33.506 INFO:teuthology.orchestra.run.smithi173.stdout:Install 12 Packages 2024-09-17T18:17:33.507 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:33.507 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 38 M 2024-09-17T18:17:33.507 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 212 M 2024-09-17T18:17:33.507 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T18:17:33.633 INFO:teuthology.orchestra.run.smithi121.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 298 kB/s | 250 kB 00:00 2024-09-17T18:17:33.858 INFO:teuthology.orchestra.run.smithi173.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.0 MB/s | 191 kB 00:00 2024-09-17T18:17:33.942 INFO:teuthology.orchestra.run.smithi173.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 392 kB/s | 33 kB 00:00 2024-09-17T18:17:33.967 INFO:teuthology.orchestra.run.smithi173.stdout:(3/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.6 MB/s | 801 kB 00:00 2024-09-17T18:17:34.043 INFO:teuthology.orchestra.run.smithi173.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 243 kB/s | 18 kB 00:00 2024-09-17T18:17:34.127 INFO:teuthology.orchestra.run.smithi173.stdout:(5/12): ceph-mgr-diskprediction-local-19.3.0-49 16 MB/s | 7.4 MB 00:00 2024-09-17T18:17:34.160 INFO:teuthology.orchestra.run.smithi173.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 329 kB/s | 38 kB 00:00 2024-09-17T18:17:34.219 INFO:teuthology.orchestra.run.smithi173.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-09-17T18:17:34.286 INFO:teuthology.orchestra.run.smithi173.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.9 MB/s | 245 kB 00:00 2024-09-17T18:17:34.361 INFO:teuthology.orchestra.run.smithi173.stdout:(9/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 5.1 MB 00:00 2024-09-17T18:17:34.386 INFO:teuthology.orchestra.run.smithi173.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.4 MB/s | 250 kB 00:00 2024-09-17T18:17:34.417 INFO:teuthology.orchestra.run.smithi121.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-09-17T18:17:34.418 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-17T18:17:34.418 INFO:teuthology.orchestra.run.smithi121.stdout:Total 9.0 MB/s | 38 MB 00:04 2024-09-17T18:17:34.478 INFO:teuthology.orchestra.run.smithi173.stdout:(11/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 14 MB/s | 4.9 MB 00:00 2024-09-17T18:17:34.703 INFO:teuthology.orchestra.run.smithi173.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 48 MB/s | 16 MB 00:00 2024-09-17T18:17:34.704 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T18:17:34.704 INFO:teuthology.orchestra.run.smithi173.stdout:Total 32 MB/s | 38 MB 00:01 2024-09-17T18:17:34.773 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-17T18:17:34.799 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-17T18:17:34.799 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-17T18:17:35.061 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T18:17:35.087 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T18:17:35.088 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T18:17:35.225 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-17T18:17:35.226 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-17T18:17:35.508 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T18:17:35.509 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T18:17:35.995 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-17T18:17:36.060 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-17T18:17:36.098 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-17T18:17:36.199 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-17T18:17:36.241 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-17T18:17:36.267 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T18:17:36.346 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-17T18:17:36.384 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-17T18:17:36.425 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-17T18:17:36.448 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-17T18:17:36.483 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-17T18:17:36.519 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-17T18:17:36.531 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-17T18:17:36.704 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-17T18:17:36.727 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-17T18:17:36.828 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-17T18:17:37.181 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-17T18:17:37.305 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-17T18:17:37.473 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-17T18:17:37.589 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-17T18:17:38.567 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-17T18:17:38.814 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-17T18:17:39.609 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-17T18:17:39.632 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a 12/12 2024-09-17T18:17:39.836 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-17T18:17:39.858 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a 12/12 2024-09-17T18:17:42.214 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a 12/12 2024-09-17T18:17:42.214 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a 1/12 2024-09-17T18:17:42.214 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-17T18:17:42.214 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-17T18:17:42.214 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-17T18:17:42.215 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-17T18:17:42.215 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-17T18:17:42.215 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-17T18:17:42.215 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-17T18:17:42.215 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-17T18:17:42.215 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-17T18:17:42.215 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-17T18:17:42.460 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a 12/12 2024-09-17T18:17:42.460 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a 1/12 2024-09-17T18:17:42.460 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-17T18:17:42.460 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-17T18:17:42.460 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-17T18:17:42.460 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-17T18:17:42.460 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-17T18:17:42.460 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-17T18:17:42.461 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-17T18:17:42.461 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-17T18:17:42.461 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-17T18:17:42.461 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-17T18:17:42.599 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-17T18:17:42.600 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:42.600 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-17T18:17:42.600 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T18:17:42.600 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-17T18:17:42.600 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-17T18:17:42.600 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-17T18:17:42.600 INFO:teuthology.orchestra.run.smithi121.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-17T18:17:42.600 INFO:teuthology.orchestra.run.smithi121.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-17T18:17:42.600 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-17T18:17:42.600 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-17T18:17:42.600 INFO:teuthology.orchestra.run.smithi121.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-17T18:17:42.600 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-17T18:17:42.600 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-17T18:17:42.601 INFO:teuthology.orchestra.run.smithi121.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-17T18:17:42.601 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:42.601 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-17T18:17:42.896 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-17T18:17:42.896 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:42.896 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T18:17:42.896 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T18:17:42.896 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-17T18:17:42.896 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-17T18:17:42.896 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-17T18:17:42.896 INFO:teuthology.orchestra.run.smithi173.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-17T18:17:42.896 INFO:teuthology.orchestra.run.smithi173.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-17T18:17:42.897 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-17T18:17:42.897 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-17T18:17:42.897 INFO:teuthology.orchestra.run.smithi173.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-17T18:17:42.897 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-17T18:17:42.897 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-17T18:17:42.897 INFO:teuthology.orchestra.run.smithi173.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-17T18:17:42.897 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:42.897 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T18:17:42.990 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-rook 2024-09-17T18:17:43.280 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-rook 2024-09-17T18:17:43.587 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:22 ago on Tue 17 Sep 2024 06:16:21 PM UTC. 2024-09-17T18:17:43.719 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-17T18:17:43.720 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:17:43.720 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-09-17T18:17:43.721 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:17:43.721 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-17T18:17:43.721 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-rook noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 49 k 2024-09-17T18:17:43.721 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-09-17T18:17:43.721 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-17T18:17:43.721 INFO:teuthology.orchestra.run.smithi121.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-17T18:17:43.721 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-17T18:17:43.721 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-17T18:17:43.721 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-17T18:17:43.721 INFO:teuthology.orchestra.run.smithi121.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-17T18:17:43.721 INFO:teuthology.orchestra.run.smithi121.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-17T18:17:43.722 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-17T18:17:43.722 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-17T18:17:43.722 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-17T18:17:43.722 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-17T18:17:43.722 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-17T18:17:43.722 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:43.722 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-17T18:17:43.722 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:17:43.722 INFO:teuthology.orchestra.run.smithi121.stdout:Install 13 Packages 2024-09-17T18:17:43.722 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:43.723 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 2.2 M 2024-09-17T18:17:43.723 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 26 M 2024-09-17T18:17:43.723 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-17T18:17:43.872 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:19 ago on Tue 17 Sep 2024 06:16:24 PM UTC. 2024-09-17T18:17:44.006 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T18:17:44.007 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:17:44.007 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-09-17T18:17:44.007 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:17:44.007 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T18:17:44.007 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-rook noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 49 k 2024-09-17T18:17:44.007 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-09-17T18:17:44.007 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-17T18:17:44.007 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-17T18:17:44.007 INFO:teuthology.orchestra.run.smithi173.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-17T18:17:44.007 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-17T18:17:44.007 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-17T18:17:44.008 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-17T18:17:44.008 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-17T18:17:44.008 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-17T18:17:44.008 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-17T18:17:44.008 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-17T18:17:44.008 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-17T18:17:44.008 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-17T18:17:44.008 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:44.008 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T18:17:44.008 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:17:44.008 INFO:teuthology.orchestra.run.smithi173.stdout:Install 13 Packages 2024-09-17T18:17:44.008 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:44.009 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 2.2 M 2024-09-17T18:17:44.009 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 26 M 2024-09-17T18:17:44.009 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T18:17:44.224 INFO:teuthology.orchestra.run.smithi121.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 236 kB/s | 19 kB 00:00 2024-09-17T18:17:44.249 INFO:teuthology.orchestra.run.smithi121.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 241 kB/s | 26 kB 00:00 2024-09-17T18:17:44.308 INFO:teuthology.orchestra.run.smithi121.stdout:(3/13): ceph-mgr-rook-19.3.0-4975.g1889a165.el9 298 kB/s | 49 kB 00:00 2024-09-17T18:17:44.333 INFO:teuthology.orchestra.run.smithi121.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-09-17T18:17:44.359 INFO:teuthology.orchestra.run.smithi121.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-09-17T18:17:44.413 INFO:teuthology.orchestra.run.smithi121.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 680 kB/s | 54 kB 00:00 2024-09-17T18:17:44.459 INFO:teuthology.orchestra.run.smithi121.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 321 kB/s | 32 kB 00:00 2024-09-17T18:17:44.485 INFO:teuthology.orchestra.run.smithi121.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 196 kB/s | 14 kB 00:00 2024-09-17T18:17:44.510 INFO:teuthology.orchestra.run.smithi121.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.4 MB/s | 279 kB 00:00 2024-09-17T18:17:44.543 INFO:teuthology.orchestra.run.smithi121.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.6 MB/s | 220 kB 00:00 2024-09-17T18:17:44.569 INFO:teuthology.orchestra.run.smithi121.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.8 MB/s | 90 kB 00:00 2024-09-17T18:17:44.594 INFO:teuthology.orchestra.run.smithi121.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 705 kB/s | 59 kB 00:00 2024-09-17T18:17:44.628 INFO:teuthology.orchestra.run.smithi121.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.2 MB/s | 1.0 MB 00:00 2024-09-17T18:17:44.631 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-17T18:17:44.631 INFO:teuthology.orchestra.run.smithi121.stdout:Total 2.4 MB/s | 2.2 MB 00:00 2024-09-17T18:17:44.696 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-17T18:17:44.712 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-17T18:17:44.712 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-17T18:17:44.743 INFO:teuthology.orchestra.run.smithi173.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 290 kB/s | 26 kB 00:00 2024-09-17T18:17:44.768 INFO:teuthology.orchestra.run.smithi173.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 169 kB/s | 19 kB 00:00 2024-09-17T18:17:44.800 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-17T18:17:44.802 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-17T18:17:44.852 INFO:teuthology.orchestra.run.smithi173.stdout:(3/13): ceph-mgr-rook-19.3.0-4975.g1889a165.el9 250 kB/s | 49 kB 00:00 2024-09-17T18:17:44.885 INFO:teuthology.orchestra.run.smithi173.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-09-17T18:17:44.911 INFO:teuthology.orchestra.run.smithi173.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.5 MB/s | 222 kB 00:00 2024-09-17T18:17:44.944 INFO:teuthology.orchestra.run.smithi173.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 914 kB/s | 54 kB 00:00 2024-09-17T18:17:44.978 INFO:teuthology.orchestra.run.smithi173.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 482 kB/s | 32 kB 00:00 2024-09-17T18:17:45.003 INFO:teuthology.orchestra.run.smithi173.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 249 kB/s | 14 kB 00:00 2024-09-17T18:17:45.029 INFO:teuthology.orchestra.run.smithi173.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 4.4 MB/s | 220 kB 00:00 2024-09-17T18:17:45.054 INFO:teuthology.orchestra.run.smithi173.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-17T18:17:45.087 INFO:teuthology.orchestra.run.smithi173.stdout:(11/13): python3-pyasn1-modules-0.4.8-6.el9.noa 1.2 MB/s | 279 kB 00:00 2024-09-17T18:17:45.113 INFO:teuthology.orchestra.run.smithi173.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-09-17T18:17:45.146 INFO:teuthology.orchestra.run.smithi173.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.2 MB/s | 1.0 MB 00:00 2024-09-17T18:17:45.148 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T18:17:45.148 INFO:teuthology.orchestra.run.smithi173.stdout:Total 1.9 MB/s | 2.2 MB 00:01 2024-09-17T18:17:45.214 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T18:17:45.230 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T18:17:45.230 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T18:17:45.310 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-17T18:17:45.317 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T18:17:45.318 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T18:17:45.482 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-17T18:17:45.537 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-17T18:17:45.592 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-17T18:17:45.638 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-17T18:17:45.679 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-17T18:17:45.797 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-17T18:17:45.817 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T18:17:45.907 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-17T18:17:45.956 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-17T18:17:45.997 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-17T18:17:46.048 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-17T18:17:46.103 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-17T18:17:46.148 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-17T18:17:46.190 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-17T18:17:46.292 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-17T18:17:46.403 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-17T18:17:46.459 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-17T18:17:46.666 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-17T18:17:46.720 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-17T18:17:46.754 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-17T18:17:46.802 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-17T18:17:46.824 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 13/13 2024-09-17T18:17:47.192 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-17T18:17:47.241 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-17T18:17:47.272 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-17T18:17:47.320 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-17T18:17:47.342 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 13/13 2024-09-17T18:17:47.615 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 13/13 2024-09-17T18:17:47.615 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 1/13 2024-09-17T18:17:47.615 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-17T18:17:47.615 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-17T18:17:47.615 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-17T18:17:47.615 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-17T18:17:47.615 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-17T18:17:47.615 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-17T18:17:47.615 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-17T18:17:47.616 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-17T18:17:47.616 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-17T18:17:47.616 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-17T18:17:47.616 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-17T18:17:47.938 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-17T18:17:47.938 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:47.938 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-17T18:17:47.938 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T18:17:47.938 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-17T18:17:47.938 INFO:teuthology.orchestra.run.smithi121.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-17T18:17:47.938 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-17T18:17:47.938 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-17T18:17:47.938 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-17T18:17:47.938 INFO:teuthology.orchestra.run.smithi121.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-17T18:17:47.938 INFO:teuthology.orchestra.run.smithi121.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-17T18:17:47.939 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-17T18:17:47.939 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-17T18:17:47.939 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-17T18:17:47.939 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-17T18:17:47.939 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-17T18:17:47.939 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:47.939 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-17T18:17:48.052 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 13/13 2024-09-17T18:17:48.052 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 1/13 2024-09-17T18:17:48.052 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-17T18:17:48.052 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-17T18:17:48.052 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-17T18:17:48.053 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-17T18:17:48.053 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-17T18:17:48.053 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-17T18:17:48.053 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-17T18:17:48.053 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-17T18:17:48.053 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-17T18:17:48.053 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-17T18:17:48.053 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-17T18:17:48.202 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-cephadm 2024-09-17T18:17:48.392 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-17T18:17:48.392 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:48.392 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T18:17:48.392 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T18:17:48.393 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-17T18:17:48.393 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-17T18:17:48.393 INFO:teuthology.orchestra.run.smithi173.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-17T18:17:48.393 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-17T18:17:48.393 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-17T18:17:48.393 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-17T18:17:48.393 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-17T18:17:48.393 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-17T18:17:48.393 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-17T18:17:48.393 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-17T18:17:48.393 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-17T18:17:48.393 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-17T18:17:48.393 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:48.393 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T18:17:48.636 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-cephadm 2024-09-17T18:17:48.767 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:27 ago on Tue 17 Sep 2024 06:16:21 PM UTC. 2024-09-17T18:17:48.898 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-17T18:17:48.898 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:17:48.898 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-09-17T18:17:48.899 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:17:48.899 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-17T18:17:48.899 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 154 k 2024-09-17T18:17:48.899 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-09-17T18:17:48.899 INFO:teuthology.orchestra.run.smithi121.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-17T18:17:48.899 INFO:teuthology.orchestra.run.smithi121.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-17T18:17:48.899 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-17T18:17:48.899 INFO:teuthology.orchestra.run.smithi121.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-17T18:17:48.899 INFO:teuthology.orchestra.run.smithi121.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-17T18:17:48.899 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:48.899 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-17T18:17:48.899 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:17:48.899 INFO:teuthology.orchestra.run.smithi121.stdout:Install 6 Packages 2024-09-17T18:17:48.900 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:48.900 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 7.0 M 2024-09-17T18:17:48.900 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 33 M 2024-09-17T18:17:48.900 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-17T18:17:49.229 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:25 ago on Tue 17 Sep 2024 06:16:24 PM UTC. 2024-09-17T18:17:49.280 INFO:teuthology.orchestra.run.smithi121.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-09-17T18:17:49.305 INFO:teuthology.orchestra.run.smithi121.stdout:(2/6): ceph-mgr-cephadm-19.3.0-4975.g1889a165.e 726 kB/s | 154 kB 00:00 2024-09-17T18:17:49.355 INFO:teuthology.orchestra.run.smithi121.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 7.1 MB/s | 548 kB 00:00 2024-09-17T18:17:49.361 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T18:17:49.362 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:17:49.362 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-09-17T18:17:49.362 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:17:49.363 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T18:17:49.363 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 154 k 2024-09-17T18:17:49.363 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-09-17T18:17:49.363 INFO:teuthology.orchestra.run.smithi173.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-17T18:17:49.363 INFO:teuthology.orchestra.run.smithi173.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-17T18:17:49.363 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-17T18:17:49.364 INFO:teuthology.orchestra.run.smithi173.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-17T18:17:49.364 INFO:teuthology.orchestra.run.smithi173.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-17T18:17:49.364 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:49.364 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T18:17:49.364 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:17:49.365 INFO:teuthology.orchestra.run.smithi173.stdout:Install 6 Packages 2024-09-17T18:17:49.365 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:49.366 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 7.0 M 2024-09-17T18:17:49.366 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 33 M 2024-09-17T18:17:49.366 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T18:17:49.381 INFO:teuthology.orchestra.run.smithi121.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 766 kB/s | 58 kB 00:00 2024-09-17T18:17:49.456 INFO:teuthology.orchestra.run.smithi121.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 741 kB/s | 74 kB 00:00 2024-09-17T18:17:49.556 INFO:teuthology.orchestra.run.smithi121.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 13 MB/s | 6.0 MB 00:00 2024-09-17T18:17:49.559 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-17T18:17:49.559 INFO:teuthology.orchestra.run.smithi121.stdout:Total 11 MB/s | 7.0 MB 00:00 2024-09-17T18:17:49.628 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-17T18:17:49.640 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-17T18:17:49.640 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-17T18:17:49.738 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-17T18:17:49.738 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-17T18:17:49.931 INFO:teuthology.orchestra.run.smithi173.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-09-17T18:17:49.956 INFO:teuthology.orchestra.run.smithi173.stdout:(2/6): ceph-mgr-cephadm-19.3.0-4975.g1889a165.e 661 kB/s | 154 kB 00:00 2024-09-17T18:17:50.006 INFO:teuthology.orchestra.run.smithi173.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 7.1 MB/s | 548 kB 00:00 2024-09-17T18:17:50.031 INFO:teuthology.orchestra.run.smithi173.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 767 kB/s | 58 kB 00:00 2024-09-17T18:17:50.039 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-17T18:17:50.057 INFO:teuthology.orchestra.run.smithi173.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.5 MB/s | 74 kB 00:00 2024-09-17T18:17:50.202 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-17T18:17:50.249 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-17T18:17:50.780 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-17T18:17:50.860 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-17T18:17:50.928 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-17T18:17:50.970 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 6/6 2024-09-17T18:17:51.714 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 6/6 2024-09-17T18:17:51.714 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/6 2024-09-17T18:17:51.714 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-17T18:17:51.714 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-17T18:17:51.714 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-17T18:17:51.714 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-17T18:17:51.882 INFO:teuthology.orchestra.run.smithi173.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 2.8 MB/s | 6.0 MB 00:02 2024-09-17T18:17:51.885 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T18:17:51.885 INFO:teuthology.orchestra.run.smithi173.stdout:Total 2.8 MB/s | 7.0 MB 00:02 2024-09-17T18:17:51.960 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T18:17:51.972 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T18:17:51.973 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T18:17:51.982 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-17T18:17:51.982 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:51.982 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-17T18:17:51.982 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T18:17:51.982 INFO:teuthology.orchestra.run.smithi121.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-17T18:17:51.982 INFO:teuthology.orchestra.run.smithi121.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-17T18:17:51.983 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-17T18:17:51.983 INFO:teuthology.orchestra.run.smithi121.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-17T18:17:51.983 INFO:teuthology.orchestra.run.smithi121.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-17T18:17:51.983 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:51.983 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-17T18:17:52.075 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T18:17:52.076 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T18:17:52.243 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-fuse 2024-09-17T18:17:52.386 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T18:17:52.559 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-17T18:17:52.608 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-17T18:17:52.819 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:31 ago on Tue 17 Sep 2024 06:16:21 PM UTC. 2024-09-17T18:17:52.953 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-17T18:17:52.953 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:17:52.953 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-09-17T18:17:52.953 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:17:52.954 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-17T18:17:52.954 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse x86_64 2:19.3.0-4975.g1889a165.el9 ceph 899 k 2024-09-17T18:17:52.954 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-09-17T18:17:52.954 INFO:teuthology.orchestra.run.smithi121.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-17T18:17:52.954 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:52.954 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-17T18:17:52.954 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:17:52.954 INFO:teuthology.orchestra.run.smithi121.stdout:Install 2 Packages 2024-09-17T18:17:52.954 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:52.955 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 979 k 2024-09-17T18:17:52.955 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 2.9 M 2024-09-17T18:17:52.955 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-17T18:17:53.132 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-17T18:17:53.220 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-17T18:17:53.296 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-17T18:17:53.321 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 6/6 2024-09-17T18:17:53.425 INFO:teuthology.orchestra.run.smithi121.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 414 kB/s | 80 kB 00:00 2024-09-17T18:17:53.483 INFO:teuthology.orchestra.run.smithi121.stdout:(2/2): ceph-fuse-19.3.0-4975.g1889a165.el9.x86_ 3.5 MB/s | 899 kB 00:00 2024-09-17T18:17:53.484 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-17T18:17:53.484 INFO:teuthology.orchestra.run.smithi121.stdout:Total 1.8 MB/s | 979 kB 00:00 2024-09-17T18:17:53.496 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-17T18:17:53.518 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-17T18:17:53.518 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-17T18:17:53.605 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-17T18:17:53.605 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-17T18:17:53.809 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-17T18:17:53.953 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-17T18:17:54.024 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 2/2 2024-09-17T18:17:54.106 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 6/6 2024-09-17T18:17:54.106 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/6 2024-09-17T18:17:54.106 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-17T18:17:54.106 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-17T18:17:54.106 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-17T18:17:54.106 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-17T18:17:54.365 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-17T18:17:54.365 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:54.366 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T18:17:54.366 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T18:17:54.366 INFO:teuthology.orchestra.run.smithi173.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-17T18:17:54.366 INFO:teuthology.orchestra.run.smithi173.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-17T18:17:54.366 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-17T18:17:54.366 INFO:teuthology.orchestra.run.smithi173.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-17T18:17:54.366 INFO:teuthology.orchestra.run.smithi173.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-17T18:17:54.366 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:54.366 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T18:17:54.616 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-fuse 2024-09-17T18:17:54.763 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 2/2 2024-09-17T18:17:54.763 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/2 2024-09-17T18:17:55.026 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-17T18:17:55.026 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:55.027 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-17T18:17:55.027 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-17T18:17:55.027 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:55.027 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-17T18:17:55.193 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:31 ago on Tue 17 Sep 2024 06:16:24 PM UTC. 2024-09-17T18:17:55.313 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-volume 2024-09-17T18:17:55.324 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T18:17:55.325 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:17:55.325 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-09-17T18:17:55.325 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:17:55.325 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T18:17:55.325 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse x86_64 2:19.3.0-4975.g1889a165.el9 ceph 899 k 2024-09-17T18:17:55.325 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-09-17T18:17:55.325 INFO:teuthology.orchestra.run.smithi173.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-17T18:17:55.326 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:55.326 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T18:17:55.326 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:17:55.326 INFO:teuthology.orchestra.run.smithi173.stdout:Install 2 Packages 2024-09-17T18:17:55.326 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:55.326 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 979 k 2024-09-17T18:17:55.326 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 2.9 M 2024-09-17T18:17:55.326 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T18:17:55.576 INFO:teuthology.orchestra.run.smithi173.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 525 kB/s | 80 kB 00:00 2024-09-17T18:17:55.709 INFO:teuthology.orchestra.run.smithi173.stdout:(2/2): ceph-fuse-19.3.0-4975.g1889a165.el9.x86_ 3.1 MB/s | 899 kB 00:00 2024-09-17T18:17:55.709 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T18:17:55.710 INFO:teuthology.orchestra.run.smithi173.stdout:Total 2.5 MB/s | 979 kB 00:00 2024-09-17T18:17:55.722 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T18:17:55.743 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T18:17:55.743 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T18:17:55.826 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T18:17:55.827 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T18:17:55.907 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:34 ago on Tue 17 Sep 2024 06:16:21 PM UTC. 2024-09-17T18:17:56.015 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T18:17:56.042 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-17T18:17:56.043 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:17:56.043 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-09-17T18:17:56.043 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:17:56.043 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-17T18:17:56.043 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-volume noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 285 k 2024-09-17T18:17:56.043 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:56.043 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-17T18:17:56.043 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:17:56.043 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-09-17T18:17:56.043 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:56.044 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 285 k 2024-09-17T18:17:56.044 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 1.3 M 2024-09-17T18:17:56.044 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-17T18:17:56.161 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-17T18:17:56.232 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 2/2 2024-09-17T18:17:56.247 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-volume-19.3.0-4975.g1889a165.el9.noarch.rp 1.4 MB/s | 285 kB 00:00 2024-09-17T18:17:56.247 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-17T18:17:56.248 INFO:teuthology.orchestra.run.smithi121.stdout:Total 1.4 MB/s | 285 kB 00:00 2024-09-17T18:17:56.248 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-17T18:17:56.256 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-17T18:17:56.257 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-17T18:17:56.285 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-17T18:17:56.285 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-17T18:17:56.519 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-17T18:17:56.599 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T18:17:56.615 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T18:17:56.615 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T18:17:56.615 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-17T18:17:56.615 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:56.897 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 2/2 2024-09-17T18:17:56.897 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/2 2024-09-17T18:17:57.176 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-17T18:17:57.176 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:57.176 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T18:17:57.176 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-17T18:17:57.176 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:57.176 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T18:17:57.475 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-volume 2024-09-17T18:17:57.632 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T18:17:57.633 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:57.633 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-17T18:17:57.633 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T18:17:57.633 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:57.633 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-17T18:17:57.888 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librados-devel 2024-09-17T18:17:58.071 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:34 ago on Tue 17 Sep 2024 06:16:24 PM UTC. 2024-09-17T18:17:58.204 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T18:17:58.205 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:17:58.205 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-09-17T18:17:58.205 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:17:58.205 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T18:17:58.206 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-volume noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 285 k 2024-09-17T18:17:58.206 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:58.206 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T18:17:58.206 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:17:58.206 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-09-17T18:17:58.206 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:58.206 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 285 k 2024-09-17T18:17:58.206 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 1.3 M 2024-09-17T18:17:58.206 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T18:17:58.405 INFO:teuthology.orchestra.run.smithi173.stdout:ceph-volume-19.3.0-4975.g1889a165.el9.noarch.rp 1.4 MB/s | 285 kB 00:00 2024-09-17T18:17:58.406 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T18:17:58.406 INFO:teuthology.orchestra.run.smithi173.stdout:Total 1.4 MB/s | 285 kB 00:00 2024-09-17T18:17:58.406 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T18:17:58.414 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T18:17:58.414 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T18:17:58.445 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T18:17:58.446 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T18:17:58.476 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:37 ago on Tue 17 Sep 2024 06:16:21 PM UTC. 2024-09-17T18:17:58.610 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-17T18:17:58.610 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:17:58.610 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-09-17T18:17:58.610 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:17:58.610 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-17T18:17:58.610 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel x86_64 2:19.3.0-4975.g1889a165.el9 ceph 126 k 2024-09-17T18:17:58.611 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:58.611 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-17T18:17:58.611 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:17:58.611 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-09-17T18:17:58.611 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:58.611 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 126 k 2024-09-17T18:17:58.612 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 456 k 2024-09-17T18:17:58.612 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-17T18:17:58.661 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T18:17:58.732 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T18:17:58.748 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T18:17:58.748 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T18:17:58.748 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-17T18:17:58.748 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:58.802 INFO:teuthology.orchestra.run.smithi121.stdout:librados-devel-19.3.0-4975.g1889a165.el9.x86_64 664 kB/s | 126 kB 00:00 2024-09-17T18:17:58.803 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-17T18:17:58.803 INFO:teuthology.orchestra.run.smithi121.stdout:Total 659 kB/s | 126 kB 00:00 2024-09-17T18:17:58.803 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-17T18:17:58.808 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-17T18:17:58.809 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-17T18:17:58.874 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-17T18:17:58.874 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-17T18:17:59.034 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-17T18:17:59.125 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T18:17:59.599 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T18:17:59.741 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T18:17:59.741 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:59.741 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T18:17:59.741 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T18:17:59.741 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:17:59.741 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T18:17:59.897 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T18:17:59.897 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:59.897 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-17T18:17:59.897 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:17:59.897 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:17:59.897 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-17T18:18:00.031 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install librados-devel 2024-09-17T18:18:00.134 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install libcephfs2 2024-09-17T18:18:00.634 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:36 ago on Tue 17 Sep 2024 06:16:24 PM UTC. 2024-09-17T18:18:00.723 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:39 ago on Tue 17 Sep 2024 06:16:21 PM UTC. 2024-09-17T18:18:00.768 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T18:18:00.768 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:18:00.768 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2024-09-17T18:18:00.768 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:18:00.768 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T18:18:00.768 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel x86_64 2:19.3.0-4975.g1889a165.el9 ceph 126 k 2024-09-17T18:18:00.768 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:00.768 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T18:18:00.768 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:18:00.769 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-09-17T18:18:00.769 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:00.769 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 126 k 2024-09-17T18:18:00.769 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 456 k 2024-09-17T18:18:00.769 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T18:18:00.814 INFO:teuthology.orchestra.run.smithi121.stdout:Package libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T18:18:00.858 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-17T18:18:00.859 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-09-17T18:18:00.859 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-17T18:18:00.934 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install libcephfs-devel 2024-09-17T18:18:00.968 INFO:teuthology.orchestra.run.smithi173.stdout:librados-devel-19.3.0-4975.g1889a165.el9.x86_64 635 kB/s | 126 kB 00:00 2024-09-17T18:18:00.968 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T18:18:00.969 INFO:teuthology.orchestra.run.smithi173.stdout:Total 630 kB/s | 126 kB 00:00 2024-09-17T18:18:00.969 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T18:18:00.974 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T18:18:00.974 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T18:18:01.038 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T18:18:01.038 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T18:18:01.198 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T18:18:01.313 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T18:18:01.516 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:40 ago on Tue 17 Sep 2024 06:16:21 PM UTC. 2024-09-17T18:18:01.649 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-17T18:18:01.649 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:18:01.649 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-09-17T18:18:01.650 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:18:01.650 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-17T18:18:01.650 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel x86_64 2:19.3.0-4975.g1889a165.el9 ceph 32 k 2024-09-17T18:18:01.650 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:01.650 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-17T18:18:01.650 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:18:01.650 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-09-17T18:18:01.650 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:01.650 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 32 k 2024-09-17T18:18:01.650 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 142 k 2024-09-17T18:18:01.650 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-17T18:18:01.706 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T18:18:01.807 INFO:teuthology.orchestra.run.smithi121.stdout:libcephfs-devel-19.3.0-4975.g1889a165.el9.x86_6 203 kB/s | 32 kB 00:00 2024-09-17T18:18:01.807 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-17T18:18:01.807 INFO:teuthology.orchestra.run.smithi121.stdout:Total 201 kB/s | 32 kB 00:00 2024-09-17T18:18:01.807 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-17T18:18:01.811 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-17T18:18:01.811 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-17T18:18:01.840 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-17T18:18:01.840 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-17T18:18:01.929 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-17T18:18:01.959 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T18:18:01.959 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:01.959 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T18:18:01.959 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:18:01.959 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:01.959 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T18:18:02.039 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T18:18:02.233 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install libcephfs2 2024-09-17T18:18:02.438 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T18:18:02.664 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T18:18:02.664 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:02.664 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-17T18:18:02.664 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:18:02.664 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:02.664 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-17T18:18:02.821 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:38 ago on Tue 17 Sep 2024 06:16:24 PM UTC. 2024-09-17T18:18:02.847 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librados2 2024-09-17T18:18:02.913 INFO:teuthology.orchestra.run.smithi173.stdout:Package libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T18:18:02.958 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T18:18:02.959 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-09-17T18:18:02.959 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T18:18:03.017 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install libcephfs-devel 2024-09-17T18:18:03.425 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:42 ago on Tue 17 Sep 2024 06:16:21 PM UTC. 2024-09-17T18:18:03.515 INFO:teuthology.orchestra.run.smithi121.stdout:Package librados2-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T18:18:03.558 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-17T18:18:03.559 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-09-17T18:18:03.559 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-17T18:18:03.597 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:39 ago on Tue 17 Sep 2024 06:16:24 PM UTC. 2024-09-17T18:18:03.623 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librbd1 2024-09-17T18:18:03.730 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T18:18:03.730 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:18:03.731 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2024-09-17T18:18:03.731 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:18:03.731 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T18:18:03.731 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel x86_64 2:19.3.0-4975.g1889a165.el9 ceph 32 k 2024-09-17T18:18:03.731 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:03.731 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T18:18:03.731 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:18:03.731 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-09-17T18:18:03.731 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:03.731 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 32 k 2024-09-17T18:18:03.731 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 142 k 2024-09-17T18:18:03.731 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T18:18:03.880 INFO:teuthology.orchestra.run.smithi173.stdout:libcephfs-devel-19.3.0-4975.g1889a165.el9.x86_6 214 kB/s | 32 kB 00:00 2024-09-17T18:18:03.881 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T18:18:03.881 INFO:teuthology.orchestra.run.smithi173.stdout:Total 212 kB/s | 32 kB 00:00 2024-09-17T18:18:03.881 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T18:18:03.884 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T18:18:03.884 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T18:18:03.896 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T18:18:03.896 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T18:18:04.001 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T18:18:04.119 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T18:18:04.203 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:43 ago on Tue 17 Sep 2024 06:16:21 PM UTC. 2024-09-17T18:18:04.293 INFO:teuthology.orchestra.run.smithi121.stdout:Package librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T18:18:04.336 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-17T18:18:04.337 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-09-17T18:18:04.337 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-17T18:18:04.401 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-rados 2024-09-17T18:18:04.467 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T18:18:04.689 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T18:18:04.689 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:04.689 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T18:18:04.689 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:18:04.690 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:04.690 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T18:18:04.869 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install librados2 2024-09-17T18:18:04.973 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:43 ago on Tue 17 Sep 2024 06:16:21 PM UTC. 2024-09-17T18:18:05.063 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T18:18:05.106 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-17T18:18:05.107 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-09-17T18:18:05.107 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-17T18:18:05.164 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-rgw 2024-09-17T18:18:05.449 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:41 ago on Tue 17 Sep 2024 06:16:24 PM UTC. 2024-09-17T18:18:05.539 INFO:teuthology.orchestra.run.smithi173.stdout:Package librados2-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T18:18:05.581 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T18:18:05.582 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-09-17T18:18:05.582 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T18:18:05.645 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install librbd1 2024-09-17T18:18:05.742 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:44 ago on Tue 17 Sep 2024 06:16:21 PM UTC. 2024-09-17T18:18:05.832 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T18:18:05.874 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-17T18:18:05.875 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-09-17T18:18:05.875 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-17T18:18:05.942 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-cephfs 2024-09-17T18:18:06.220 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:42 ago on Tue 17 Sep 2024 06:16:24 PM UTC. 2024-09-17T18:18:06.310 INFO:teuthology.orchestra.run.smithi173.stdout:Package librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T18:18:06.353 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T18:18:06.354 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-09-17T18:18:06.354 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T18:18:06.411 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-rados 2024-09-17T18:18:06.519 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:45 ago on Tue 17 Sep 2024 06:16:21 PM UTC. 2024-09-17T18:18:06.608 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T18:18:06.651 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-17T18:18:06.652 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-09-17T18:18:06.652 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-17T18:18:06.719 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-rbd 2024-09-17T18:18:07.003 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:42 ago on Tue 17 Sep 2024 06:16:24 PM UTC. 2024-09-17T18:18:07.092 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T18:18:07.135 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T18:18:07.136 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-09-17T18:18:07.137 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T18:18:07.190 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-rgw 2024-09-17T18:18:07.297 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:46 ago on Tue 17 Sep 2024 06:16:21 PM UTC. 2024-09-17T18:18:07.387 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T18:18:07.430 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-17T18:18:07.431 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-09-17T18:18:07.431 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-17T18:18:07.498 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-fuse 2024-09-17T18:18:07.769 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:43 ago on Tue 17 Sep 2024 06:16:24 PM UTC. 2024-09-17T18:18:07.858 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T18:18:07.900 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T18:18:07.901 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-09-17T18:18:07.901 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T18:18:07.959 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-cephfs 2024-09-17T18:18:08.068 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:47 ago on Tue 17 Sep 2024 06:16:21 PM UTC. 2024-09-17T18:18:08.200 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-17T18:18:08.200 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:18:08.201 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-09-17T18:18:08.201 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:18:08.201 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-17T18:18:08.201 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse x86_64 2:19.3.0-4975.g1889a165.el9 ceph 85 k 2024-09-17T18:18:08.201 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:08.201 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-17T18:18:08.201 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:18:08.201 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-09-17T18:18:08.201 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:08.201 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 85 k 2024-09-17T18:18:08.201 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 231 k 2024-09-17T18:18:08.201 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-17T18:18:08.401 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-fuse-19.3.0-4975.g1889a165.el9.x86_64.rpm 428 kB/s | 85 kB 00:00 2024-09-17T18:18:08.401 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-17T18:18:08.401 INFO:teuthology.orchestra.run.smithi121.stdout:Total 425 kB/s | 85 kB 00:00 2024-09-17T18:18:08.401 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-17T18:18:08.408 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-17T18:18:08.408 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-17T18:18:08.471 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-17T18:18:08.471 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-17T18:18:08.543 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:44 ago on Tue 17 Sep 2024 06:16:24 PM UTC. 2024-09-17T18:18:08.613 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-17T18:18:08.631 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T18:18:08.674 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T18:18:08.676 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-09-17T18:18:08.676 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T18:18:08.714 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T18:18:08.741 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-rbd 2024-09-17T18:18:09.101 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T18:18:09.321 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:45 ago on Tue 17 Sep 2024 06:16:24 PM UTC. 2024-09-17T18:18:09.365 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T18:18:09.365 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:09.365 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-17T18:18:09.366 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:18:09.366 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:09.366 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-17T18:18:09.409 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T18:18:09.453 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T18:18:09.455 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-09-17T18:18:09.455 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T18:18:09.519 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install rbd-fuse 2024-09-17T18:18:09.639 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-mirror 2024-09-17T18:18:10.095 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:46 ago on Tue 17 Sep 2024 06:16:24 PM UTC. 2024-09-17T18:18:10.226 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T18:18:10.227 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:18:10.227 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-09-17T18:18:10.227 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:18:10.227 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T18:18:10.227 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse x86_64 2:19.3.0-4975.g1889a165.el9 ceph 85 k 2024-09-17T18:18:10.227 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:10.227 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T18:18:10.227 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:18:10.227 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-09-17T18:18:10.227 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:10.228 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 85 k 2024-09-17T18:18:10.228 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 231 k 2024-09-17T18:18:10.228 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T18:18:10.253 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:49 ago on Tue 17 Sep 2024 06:16:21 PM UTC. 2024-09-17T18:18:10.394 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-17T18:18:10.394 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:18:10.395 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-09-17T18:18:10.395 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:18:10.395 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-17T18:18:10.395 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror x86_64 2:19.3.0-4975.g1889a165.el9 ceph 3.2 M 2024-09-17T18:18:10.395 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:10.395 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-17T18:18:10.395 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:18:10.395 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-09-17T18:18:10.395 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:10.395 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 3.2 M 2024-09-17T18:18:10.395 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 14 M 2024-09-17T18:18:10.395 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-17T18:18:10.416 INFO:teuthology.orchestra.run.smithi173.stdout:rbd-fuse-19.3.0-4975.g1889a165.el9.x86_64.rpm 453 kB/s | 85 kB 00:00 2024-09-17T18:18:10.417 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T18:18:10.417 INFO:teuthology.orchestra.run.smithi173.stdout:Total 450 kB/s | 85 kB 00:00 2024-09-17T18:18:10.417 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T18:18:10.424 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T18:18:10.424 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T18:18:10.487 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T18:18:10.487 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T18:18:10.636 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T18:18:10.731 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-mirror-19.3.0-4975.g1889a165.el9.x86_64.rpm 9.5 MB/s | 3.2 MB 00:00 2024-09-17T18:18:10.731 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-17T18:18:10.732 INFO:teuthology.orchestra.run.smithi121.stdout:Total 9.5 MB/s | 3.2 MB 00:00 2024-09-17T18:18:10.732 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-17T18:18:10.738 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T18:18:10.741 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-17T18:18:10.741 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-17T18:18:10.822 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-17T18:18:10.822 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-17T18:18:11.082 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T18:18:11.258 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-17T18:18:11.315 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T18:18:11.325 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T18:18:11.326 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:11.326 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T18:18:11.326 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:18:11.326 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:11.326 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T18:18:11.341 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T18:18:11.341 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T18:18:11.341 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-17T18:18:11.341 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T18:18:11.341 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T18:18:11.341 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:11.621 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install rbd-mirror 2024-09-17T18:18:12.241 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:48 ago on Tue 17 Sep 2024 06:16:24 PM UTC. 2024-09-17T18:18:12.380 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T18:18:12.381 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:18:12.381 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2024-09-17T18:18:12.381 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:18:12.381 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T18:18:12.381 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror x86_64 2:19.3.0-4975.g1889a165.el9 ceph 3.2 M 2024-09-17T18:18:12.381 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:12.381 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T18:18:12.381 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:18:12.381 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-09-17T18:18:12.382 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:12.382 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 3.2 M 2024-09-17T18:18:12.382 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 14 M 2024-09-17T18:18:12.382 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T18:18:12.440 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T18:18:12.440 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:12.440 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-17T18:18:12.440 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:18:12.440 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:12.440 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-17T18:18:12.676 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-nbd 2024-09-17T18:18:12.695 INFO:teuthology.orchestra.run.smithi173.stdout:rbd-mirror-19.3.0-4975.g1889a165.el9.x86_64.rpm 10 MB/s | 3.2 MB 00:00 2024-09-17T18:18:12.696 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T18:18:12.696 INFO:teuthology.orchestra.run.smithi173.stdout:Total 10 MB/s | 3.2 MB 00:00 2024-09-17T18:18:12.696 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T18:18:12.705 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T18:18:12.706 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T18:18:12.787 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T18:18:12.787 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T18:18:13.226 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T18:18:13.275 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:52 ago on Tue 17 Sep 2024 06:16:21 PM UTC. 2024-09-17T18:18:13.287 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T18:18:13.313 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T18:18:13.313 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T18:18:13.313 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-17T18:18:13.313 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T18:18:13.313 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T18:18:13.314 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:13.408 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-17T18:18:13.409 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:18:13.409 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-09-17T18:18:13.409 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:18:13.409 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-17T18:18:13.409 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd x86_64 2:19.3.0-4975.g1889a165.el9 ceph 170 k 2024-09-17T18:18:13.409 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:13.409 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-17T18:18:13.409 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-17T18:18:13.409 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-09-17T18:18:13.409 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:13.409 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 170 k 2024-09-17T18:18:13.410 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 490 k 2024-09-17T18:18:13.410 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-17T18:18:13.626 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-nbd-19.3.0-4975.g1889a165.el9.x86_64.rpm 788 kB/s | 170 kB 00:00 2024-09-17T18:18:13.626 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-17T18:18:13.626 INFO:teuthology.orchestra.run.smithi121.stdout:Total 783 kB/s | 170 kB 00:00 2024-09-17T18:18:13.627 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-17T18:18:13.634 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-17T18:18:13.634 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-17T18:18:13.698 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-17T18:18:13.698 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-17T18:18:13.847 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-17T18:18:13.940 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T18:18:14.388 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T18:18:14.401 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T18:18:14.401 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:14.401 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T18:18:14.401 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:18:14.402 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:14.402 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T18:18:14.640 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install rbd-nbd 2024-09-17T18:18:14.658 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T18:18:14.658 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:14.658 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-17T18:18:14.658 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:18:14.658 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:14.658 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-17T18:18:14.883 DEBUG:teuthology.parallel:result is None 2024-09-17T18:18:15.240 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:51 ago on Tue 17 Sep 2024 06:16:24 PM UTC. 2024-09-17T18:18:15.372 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T18:18:15.373 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:18:15.373 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-09-17T18:18:15.373 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:18:15.373 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T18:18:15.373 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd x86_64 2:19.3.0-4975.g1889a165.el9 ceph 170 k 2024-09-17T18:18:15.374 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:15.374 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T18:18:15.374 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T18:18:15.374 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-09-17T18:18:15.374 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:15.374 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 170 k 2024-09-17T18:18:15.374 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 490 k 2024-09-17T18:18:15.374 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T18:18:15.590 INFO:teuthology.orchestra.run.smithi173.stdout:rbd-nbd-19.3.0-4975.g1889a165.el9.x86_64.rpm 787 kB/s | 170 kB 00:00 2024-09-17T18:18:15.591 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T18:18:15.591 INFO:teuthology.orchestra.run.smithi173.stdout:Total 782 kB/s | 170 kB 00:00 2024-09-17T18:18:15.591 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T18:18:15.599 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T18:18:15.599 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T18:18:15.662 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T18:18:15.662 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T18:18:15.810 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T18:18:15.921 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T18:18:16.342 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T18:18:16.593 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T18:18:16.593 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:16.593 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T18:18:16.593 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T18:18:16.594 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:16.594 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T18:18:16.844 DEBUG:teuthology.parallel:result is None 2024-09-17T18:18:16.845 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T18:18:16.990 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-17T18:18:17.016 INFO:teuthology.orchestra.run.smithi121.stdout:19.3.0-4975.g1889a165.el9 2024-09-17T18:18:17.017 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4975.g1889a165.el9 2024-09-17T18:18:17.017 INFO:teuthology.task.install:The correct ceph version 19.3.0-4975.g1889a165 is installed. 2024-09-17T18:18:17.019 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T18:18:17.159 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-17T18:18:17.181 INFO:teuthology.orchestra.run.smithi173.stdout:19.3.0-4975.g1889a165.el9 2024-09-17T18:18:17.181 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4975.g1889a165.el9 2024-09-17T18:18:17.181 INFO:teuthology.task.install:The correct ceph version 19.3.0-4975.g1889a165 is installed. 2024-09-17T18:18:17.183 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-17T18:18:17.183 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-17T18:18:17.183 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-17T18:18:17.214 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T18:18:17.214 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-17T18:18:17.249 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-17T18:18:17.250 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-17T18:18:17.250 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/daemon-helper 2024-09-17T18:18:17.281 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-17T18:18:17.349 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T18:18:17.349 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/daemon-helper 2024-09-17T18:18:17.374 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-17T18:18:17.440 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-17T18:18:17.440 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-17T18:18:17.440 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-17T18:18:17.467 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-17T18:18:17.534 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T18:18:17.534 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-17T18:18:17.581 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-17T18:18:17.646 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-17T18:18:17.646 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-17T18:18:17.646 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/stdin-killer 2024-09-17T18:18:17.672 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-17T18:18:17.738 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T18:18:17.738 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/stdin-killer 2024-09-17T18:18:17.763 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-17T18:18:17.827 INFO:teuthology.run_tasks:Running task ceph... 2024-09-17T18:18:17.913 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-17T18:18:17.913 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 777 /var/log/ceph 2024-09-17T18:18:17.916 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 777 /var/log/ceph 2024-09-17T18:18:17.940 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-17T18:18:17.940 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-17T18:18:17.982 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-17T18:18:18.006 INFO:tasks.ceph:Creating extra log directories... 2024-09-17T18:18:18.006 DEBUG:teuthology.orchestra.run.smithi121:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-17T18:18:18.048 DEBUG:teuthology.orchestra.run.smithi173:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-17T18:18:18.080 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-17T18:18:18.080 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 5}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2': False, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(TOO_FEW_PGS\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_FULL\\)', '\\(SLOW_OPS\\)', '\\(MON_DOWN\\)', '\\(PG_', '\\(SMALLER_PGP_NUM\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-17T18:18:18.081 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909241', 'branch': 'wip-bharath10-testing-2024-09-17-1348', 'description': 'rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{centos_latest} thrashers/force-sync-many workloads/rados_api_tests}', 'email': 'skanta@redhat.com', 'first_in_suite': False, 'job_id': '7909241', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-bharath10-testing-2024-09-17-1348'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 5}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2': False, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(TOO_FEW_PGS\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_FULL\\)', '\\(SLOW_OPS\\)', '\\(MON_DOWN\\)', '\\(PG_', '\\(SMALLER_PGP_NUM\\)', 'slow request'], 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath10-testing-2024-09-17-1348', 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'}}, 'owner': 'scheduled_skanta@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 3421, 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-bharath10-testing-2024-09-17-1348', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203', 'targets': {'smithi121.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKY1CQNsro7aKTRL/8UMG2MgcryOjAsnegIPQyDkS9cHfqpi5LCbliE0naFgvoSCiZKh01CDd8RnPgMaDKnsLx4=', 'smithi173.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOlQCiEh3GppOaCSXMZmC2+fxY2g/LhDplZ5JvlQ+2GJ0nn86Xwb0ff5DeaUoMeEw+s+IC9zfAHIo3cgwan8J+g='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 90, 'store_thrash': True, 'thrash_delay': 1, 'thrash_many': True}}, {'workunit': {'clients': {'client.0': ['rados/test.sh unit_test_scan']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-17_15:01:33', 'tube': 'smithi', 'user': 'skanta', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-17T18:18:18.081 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-17T18:18:18.114 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-17T18:18:18.129 DEBUG:teuthology.orchestra.run.smithi121:> sudo install -d -m0777 -- /var/run/ceph 2024-09-17T18:18:18.170 DEBUG:teuthology.orchestra.run.smithi173:> sudo install -d -m0777 -- /var/run/ceph 2024-09-17T18:18:18.194 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-17T18:18:18.194 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/scratch_devs of=/dev/stdout 2024-09-17T18:18:18.248 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-17T18:18:18.248 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_1 2024-09-17T18:18:18.303 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-17T18:18:18.304 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T18:18:18.304 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-09-17T18:18:18.304 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T18:18:18.304 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T18:18:18.304 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-09-17 18:18:11.600967387 +0000 2024-09-17T18:18:18.304 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-09-17 18:15:41.028631249 +0000 2024-09-17T18:18:18.304 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-09-17 18:15:41.028631249 +0000 2024-09-17T18:18:18.304 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-09-17 18:15:41.028631249 +0000 2024-09-17T18:18:18.304 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-17T18:18:18.366 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-09-17T18:18:18.366 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-09-17T18:18:18.366 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000183599 s, 2.8 MB/s 2024-09-17T18:18:18.367 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-17T18:18:18.423 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_2 2024-09-17T18:18:18.479 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-17T18:18:18.479 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T18:18:18.480 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-17T18:18:18.480 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T18:18:18.480 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T18:18:18.480 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-09-17 18:18:11.601967375 +0000 2024-09-17T18:18:18.480 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-09-17 18:15:41.348627413 +0000 2024-09-17T18:18:18.480 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-09-17 18:15:41.348627413 +0000 2024-09-17T18:18:18.480 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-09-17 18:15:41.348627413 +0000 2024-09-17T18:18:18.480 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-17T18:18:18.542 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-09-17T18:18:18.542 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-09-17T18:18:18.542 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.00021589 s, 2.4 MB/s 2024-09-17T18:18:18.544 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-17T18:18:18.599 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_3 2024-09-17T18:18:18.654 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-17T18:18:18.654 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T18:18:18.654 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-17T18:18:18.654 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T18:18:18.654 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T18:18:18.654 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-09-17 18:18:11.601967375 +0000 2024-09-17T18:18:18.655 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-09-17 18:15:41.697623229 +0000 2024-09-17T18:18:18.655 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-09-17 18:15:41.697623229 +0000 2024-09-17T18:18:18.655 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-09-17 18:15:41.697623229 +0000 2024-09-17T18:18:18.655 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-17T18:18:18.717 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-09-17T18:18:18.717 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-09-17T18:18:18.717 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000197778 s, 2.6 MB/s 2024-09-17T18:18:18.718 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-17T18:18:18.774 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_4 2024-09-17T18:18:18.829 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-17T18:18:18.829 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T18:18:18.829 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-17T18:18:18.829 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T18:18:18.830 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T18:18:18.830 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-09-17 18:18:11.601967375 +0000 2024-09-17T18:18:18.830 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-09-17 18:15:42.045619056 +0000 2024-09-17T18:18:18.830 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-09-17 18:15:42.045619056 +0000 2024-09-17T18:18:18.830 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-09-17 18:15:42.045619056 +0000 2024-09-17T18:18:18.830 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-17T18:18:18.892 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-09-17T18:18:18.892 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-09-17T18:18:18.892 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000209101 s, 2.4 MB/s 2024-09-17T18:18:18.894 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-17T18:18:18.950 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-17T18:18:18.950 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T18:18:18.950 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/scratch_devs of=/dev/stdout 2024-09-17T18:18:18.968 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-17T18:18:18.969 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_1 2024-09-17T18:18:19.027 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-17T18:18:19.027 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T18:18:19.027 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-09-17T18:18:19.027 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T18:18:19.027 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T18:18:19.027 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-09-17 18:18:13.561817263 +0000 2024-09-17T18:18:19.027 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-09-17 18:15:41.972909358 +0000 2024-09-17T18:18:19.027 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-09-17 18:15:41.972909358 +0000 2024-09-17T18:18:19.027 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-09-17 18:15:41.972909358 +0000 2024-09-17T18:18:19.027 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-17T18:18:19.093 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-09-17T18:18:19.093 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-09-17T18:18:19.094 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000212083 s, 2.4 MB/s 2024-09-17T18:18:19.095 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-17T18:18:19.155 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_2 2024-09-17T18:18:19.213 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-17T18:18:19.213 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T18:18:19.214 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-17T18:18:19.214 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T18:18:19.214 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T18:18:19.214 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-09-17 18:18:13.561817263 +0000 2024-09-17T18:18:19.214 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-09-17 18:15:42.307901878 +0000 2024-09-17T18:18:19.214 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-09-17 18:15:42.307901878 +0000 2024-09-17T18:18:19.214 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-09-17 18:15:42.307901878 +0000 2024-09-17T18:18:19.214 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-17T18:18:19.280 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-09-17T18:18:19.280 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-09-17T18:18:19.280 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000245914 s, 2.1 MB/s 2024-09-17T18:18:19.281 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-17T18:18:19.340 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_3 2024-09-17T18:18:19.396 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-17T18:18:19.397 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T18:18:19.397 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-17T18:18:19.397 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T18:18:19.397 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T18:18:19.397 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-09-17 18:18:13.561817263 +0000 2024-09-17T18:18:19.397 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-09-17 18:15:42.629894688 +0000 2024-09-17T18:18:19.397 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-09-17 18:15:42.629894688 +0000 2024-09-17T18:18:19.397 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-09-17 18:15:42.629894688 +0000 2024-09-17T18:18:19.398 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-17T18:18:19.462 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-09-17T18:18:19.462 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-09-17T18:18:19.462 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000172968 s, 3.0 MB/s 2024-09-17T18:18:19.463 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-17T18:18:19.521 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_4 2024-09-17T18:18:19.576 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-17T18:18:19.577 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T18:18:19.577 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-17T18:18:19.577 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T18:18:19.577 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T18:18:19.577 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-09-17 18:18:13.562817241 +0000 2024-09-17T18:18:19.577 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-09-17 18:15:42.947887587 +0000 2024-09-17T18:18:19.577 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-09-17 18:15:42.947887587 +0000 2024-09-17T18:18:19.577 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-09-17 18:15:42.947887587 +0000 2024-09-17T18:18:19.577 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-17T18:18:19.641 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-09-17T18:18:19.641 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-09-17T18:18:19.641 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000201523 s, 2.5 MB/s 2024-09-17T18:18:19.642 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-17T18:18:19.698 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-17T18:18:19.698 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi121.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi173.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-17T18:18:19.698 INFO:tasks.ceph:Generating config... 2024-09-17T18:18:19.699 INFO:tasks.ceph:[client] debug monc = 20 2024-09-17T18:18:19.700 INFO:tasks.ceph:[client] debug ms = 1 2024-09-17T18:18:19.700 INFO:tasks.ceph:[global] debug ms = 1 2024-09-17T18:18:19.700 INFO:tasks.ceph:[global] debug objecter = 20 2024-09-17T18:18:19.700 INFO:tasks.ceph:[global] debug rados = 20 2024-09-17T18:18:19.700 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-17T18:18:19.700 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-17T18:18:19.700 INFO:tasks.ceph:[global] ms bind msgr1 = False 2024-09-17T18:18:19.700 INFO:tasks.ceph:[global] ms bind msgr2 = True 2024-09-17T18:18:19.700 INFO:tasks.ceph:[global] ms inject delay max = 1 2024-09-17T18:18:19.700 INFO:tasks.ceph:[global] ms inject delay probability = 0.005 2024-09-17T18:18:19.700 INFO:tasks.ceph:[global] ms inject delay type = mon 2024-09-17T18:18:19.700 INFO:tasks.ceph:[global] ms inject internal delays = 0.002 2024-09-17T18:18:19.700 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2024-09-17T18:18:19.701 INFO:tasks.ceph:[global] ms type = async 2024-09-17T18:18:19.701 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-17T18:18:19.701 INFO:tasks.ceph:[mgr] debug monc = 10 2024-09-17T18:18:19.701 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-17T18:18:19.701 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-17T18:18:19.701 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-17T18:18:19.701 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-17T18:18:19.701 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2024-09-17T18:18:19.701 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-17T18:18:19.701 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-17T18:18:19.701 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-17T18:18:19.701 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2024-09-17T18:18:19.701 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2024-09-17T18:18:19.702 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-17T18:18:19.702 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-09-17T18:18:19.702 INFO:tasks.ceph:[mon] paxos service trim min = 5 2024-09-17T18:18:19.702 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-17T18:18:19.702 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-09-17T18:18:19.702 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-17T18:18:19.702 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-17T18:18:19.702 INFO:tasks.ceph:[osd] bluestore write v2 = False 2024-09-17T18:18:19.702 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-17T18:18:19.702 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-17T18:18:19.702 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-17T18:18:19.702 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-17T18:18:19.702 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-17T18:18:19.702 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-17T18:18:19.703 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-17T18:18:19.703 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-17T18:18:19.703 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-17T18:18:19.703 INFO:tasks.ceph:[osd] osd class default list = * 2024-09-17T18:18:19.703 INFO:tasks.ceph:[osd] osd class load list = * 2024-09-17T18:18:19.703 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-17T18:18:19.703 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-17T18:18:19.703 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-17T18:18:19.703 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-17T18:18:19.703 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-17T18:18:19.703 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-17T18:18:19.703 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-17T18:18:19.703 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-17T18:18:19.704 INFO:tasks.ceph:Setting up mon.a... 2024-09-17T18:18:19.704 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-17T18:18:19.750 INFO:teuthology.orchestra.run.smithi121.stdout:creating /etc/ceph/ceph.keyring 2024-09-17T18:18:19.753 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-17T18:18:19.796 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-17T18:18:19.862 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.121'), ('mon.c', '[v2:172.21.15.121:3301,v1:172.21.15.121:6790]'), ('mon.b', '172.21.15.173')] 2024-09-17T18:18:19.862 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.121,[v2:172.21.15.121:3301,v1:172.21.15.121:6790],172.21.15.173', 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2': False, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 10}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 5}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-09-17T18:18:19.863 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-17T18:18:19.863 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-17T18:18:19.916 DEBUG:teuthology.orchestra.run.smithi121:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.121 --addv c '[v2:172.21.15.121:3301,v1:172.21.15.121:6790]' --add b 172.21.15.173 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-17T18:18:20.002 INFO:teuthology.orchestra.run.smithi121.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-17T18:18:20.003 INFO:teuthology.orchestra.run.smithi121.stdout:monmaptool: generated fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:20.003 INFO:teuthology.orchestra.run.smithi121.stdout:setting min_mon_release = quincy 2024-09-17T18:18:20.003 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 0 2024-09-17T18:18:20.003 INFO:teuthology.orchestra.run.smithi121.stdout:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:20.003 INFO:teuthology.orchestra.run.smithi121.stdout:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:20.003 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:20.003 INFO:teuthology.orchestra.run.smithi121.stdout:min_mon_release 17 (quincy) 2024-09-17T18:18:20.003 INFO:teuthology.orchestra.run.smithi121.stdout:election_strategy: 1 2024-09-17T18:18:20.003 INFO:teuthology.orchestra.run.smithi121.stdout:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:20.003 INFO:teuthology.orchestra.run.smithi121.stdout:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:20.003 INFO:teuthology.orchestra.run.smithi121.stdout:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:20.004 INFO:teuthology.orchestra.run.smithi121.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-17T18:18:20.005 DEBUG:teuthology.orchestra.run.smithi121:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-17T18:18:20.024 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b... 2024-09-17T18:18:20.025 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T18:18:20.066 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T18:18:20.106 INFO:teuthology.orchestra.run.smithi121.stdout:[global] 2024-09-17T18:18:20.106 INFO:teuthology.orchestra.run.smithi121.stdout: chdir = "" 2024-09-17T18:18:20.106 INFO:teuthology.orchestra.run.smithi121.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T18:18:20.107 INFO:teuthology.orchestra.run.smithi121.stdout: auth supported = cephx 2024-09-17T18:18:20.107 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:20.107 INFO:teuthology.orchestra.run.smithi121.stdout: filestore xattr use omap = true 2024-09-17T18:18:20.107 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:20.107 INFO:teuthology.orchestra.run.smithi121.stdout: mon clock drift allowed = 1.000 2024-09-17T18:18:20.107 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:20.107 INFO:teuthology.orchestra.run.smithi121.stdout: osd crush chooseleaf type = 0 2024-09-17T18:18:20.107 INFO:teuthology.orchestra.run.smithi121.stdout: auth debug = true 2024-09-17T18:18:20.107 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:20.107 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on old message = true 2024-09-17T18:18:20.107 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on bug = true 2024-09-17T18:18:20.107 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:20.107 INFO:teuthology.orchestra.run.smithi121.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T18:18:20.107 INFO:teuthology.orchestra.run.smithi121.stdout: mon pg warn max object skew = 0 2024-09-17T18:18:20.107 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:20.108 INFO:teuthology.orchestra.run.smithi121.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T18:18:20.108 INFO:teuthology.orchestra.run.smithi121.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T18:18:20.108 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:20.108 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default size = 2 2024-09-17T18:18:20.108 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:20.108 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow primary affinity = true 2024-09-17T18:18:20.108 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow pg remap = true 2024-09-17T18:18:20.108 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on legacy crush tunables = false 2024-09-17T18:18:20.108 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on crush straw calc version zero = false 2024-09-17T18:18:20.108 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on no sortbitwise = false 2024-09-17T18:18:20.108 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on osd down out interval zero = false 2024-09-17T18:18:20.108 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on too few osds = false 2024-09-17T18:18:20.108 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T18:18:20.109 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T18:18:20.109 INFO:teuthology.orchestra.run.smithi121.stdout: mon_allow_pool_size_one = true 2024-09-17T18:18:20.109 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:20.109 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T18:18:20.109 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:20.109 INFO:teuthology.orchestra.run.smithi121.stdout: osd default data pool replay window = 5 2024-09-17T18:18:20.109 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:20.109 INFO:teuthology.orchestra.run.smithi121.stdout: mon allow pool delete = true 2024-09-17T18:18:20.109 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:20.109 INFO:teuthology.orchestra.run.smithi121.stdout: mon cluster log file level = debug 2024-09-17T18:18:20.109 INFO:teuthology.orchestra.run.smithi121.stdout: debug asserts on shutdown = true 2024-09-17T18:18:20.109 INFO:teuthology.orchestra.run.smithi121.stdout: mon health detail to clog = false 2024-09-17T18:18:20.109 INFO:teuthology.orchestra.run.smithi121.stdout: mon host = "172.21.15.121,[v2:172.21.15.121:3301,v1:172.21.15.121:6790],172.21.15.173" 2024-09-17T18:18:20.109 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-09-17T18:18:20.110 INFO:teuthology.orchestra.run.smithi121.stdout: debug objecter = 20 2024-09-17T18:18:20.110 INFO:teuthology.orchestra.run.smithi121.stdout: debug rados = 20 2024-09-17T18:18:20.110 INFO:teuthology.orchestra.run.smithi121.stdout: mon client directed command retry = 5 2024-09-17T18:18:20.110 INFO:teuthology.orchestra.run.smithi121.stdout: mon election default strategy = 1 2024-09-17T18:18:20.110 INFO:teuthology.orchestra.run.smithi121.stdout: ms bind msgr1 = False 2024-09-17T18:18:20.110 INFO:teuthology.orchestra.run.smithi121.stdout: ms bind msgr2 = True 2024-09-17T18:18:20.110 INFO:teuthology.orchestra.run.smithi121.stdout: ms inject delay max = 1 2024-09-17T18:18:20.110 INFO:teuthology.orchestra.run.smithi121.stdout: ms inject delay probability = 0.005 2024-09-17T18:18:20.110 INFO:teuthology.orchestra.run.smithi121.stdout: ms inject delay type = mon 2024-09-17T18:18:20.110 INFO:teuthology.orchestra.run.smithi121.stdout: ms inject internal delays = 0.002 2024-09-17T18:18:20.110 INFO:teuthology.orchestra.run.smithi121.stdout: ms inject socket failures = 2500 2024-09-17T18:18:20.110 INFO:teuthology.orchestra.run.smithi121.stdout: ms type = async 2024-09-17T18:18:20.110 INFO:teuthology.orchestra.run.smithi121.stdout: fsid = a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:20.110 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:20.111 INFO:teuthology.orchestra.run.smithi121.stdout:[osd] 2024-09-17T18:18:20.111 INFO:teuthology.orchestra.run.smithi121.stdout: osd journal size = 100 2024-09-17T18:18:20.111 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:20.111 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub load threshold = 5.0 2024-09-17T18:18:20.111 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub max interval = 600 2024-09-17T18:18:20.111 INFO:teuthology.orchestra.run.smithi121.stdout: osd mclock profile = high_recovery_ops 2024-09-17T18:18:20.111 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:20.111 INFO:teuthology.orchestra.run.smithi121.stdout: osd recover clone overlap = true 2024-09-17T18:18:20.111 INFO:teuthology.orchestra.run.smithi121.stdout: osd recovery max chunk = 1048576 2024-09-17T18:18:20.111 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:20.111 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug shutdown = true 2024-09-17T18:18:20.111 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug op order = true 2024-09-17T18:18:20.111 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug verify stray on activate = true 2024-09-17T18:18:20.111 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug trim objects = true 2024-09-17T18:18:20.112 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:20.112 INFO:teuthology.orchestra.run.smithi121.stdout: osd open classes on start = true 2024-09-17T18:18:20.112 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug pg log writeout = true 2024-09-17T18:18:20.112 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:20.112 INFO:teuthology.orchestra.run.smithi121.stdout: osd deep scrub update digest min age = 30 2024-09-17T18:18:20.112 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:20.112 INFO:teuthology.orchestra.run.smithi121.stdout: osd map max advance = 10 2024-09-17T18:18:20.112 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:20.112 INFO:teuthology.orchestra.run.smithi121.stdout: journal zero on create = true 2024-09-17T18:18:20.112 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:20.112 INFO:teuthology.orchestra.run.smithi121.stdout: filestore ondisk finisher threads = 3 2024-09-17T18:18:20.112 INFO:teuthology.orchestra.run.smithi121.stdout: filestore apply finisher threads = 3 2024-09-17T18:18:20.112 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:20.112 INFO:teuthology.orchestra.run.smithi121.stdout: bdev debug aio = true 2024-09-17T18:18:20.112 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug misdirected ops = true 2024-09-17T18:18:20.113 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore block size = 96636764160 2024-09-17T18:18:20.113 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore compression algorithm = snappy 2024-09-17T18:18:20.113 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore compression mode = aggressive 2024-09-17T18:18:20.113 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore fsck on mount = True 2024-09-17T18:18:20.113 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore write v2 = False 2024-09-17T18:18:20.113 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore zero block detection = True 2024-09-17T18:18:20.113 INFO:teuthology.orchestra.run.smithi121.stdout: debug bluefs = 20 2024-09-17T18:18:20.113 INFO:teuthology.orchestra.run.smithi121.stdout: debug bluestore = 20 2024-09-17T18:18:20.113 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-09-17T18:18:20.113 INFO:teuthology.orchestra.run.smithi121.stdout: debug osd = 20 2024-09-17T18:18:20.113 INFO:teuthology.orchestra.run.smithi121.stdout: debug rocksdb = 10 2024-09-17T18:18:20.113 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T18:18:20.113 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd full ratio = 0.9 2024-09-17T18:18:20.113 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd nearfull ratio = 0.8 2024-09-17T18:18:20.113 INFO:teuthology.orchestra.run.smithi121.stdout: osd class default list = * 2024-09-17T18:18:20.114 INFO:teuthology.orchestra.run.smithi121.stdout: osd class load list = * 2024-09-17T18:18:20.114 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug verify cached snaps = True 2024-09-17T18:18:20.114 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug verify missing on start = True 2024-09-17T18:18:20.114 INFO:teuthology.orchestra.run.smithi121.stdout: osd failsafe full ratio = 0.95 2024-09-17T18:18:20.114 INFO:teuthology.orchestra.run.smithi121.stdout: osd mclock override recovery settings = True 2024-09-17T18:18:20.114 INFO:teuthology.orchestra.run.smithi121.stdout: osd objectstore = bluestore 2024-09-17T18:18:20.114 INFO:teuthology.orchestra.run.smithi121.stdout: osd op queue = debug_random 2024-09-17T18:18:20.114 INFO:teuthology.orchestra.run.smithi121.stdout: osd op queue cut off = debug_random 2024-09-17T18:18:20.114 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:20.114 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr] 2024-09-17T18:18:20.114 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-09-17T18:18:20.114 INFO:teuthology.orchestra.run.smithi121.stdout: debug mgr = 20 2024-09-17T18:18:20.114 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2024-09-17T18:18:20.114 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2024-09-17T18:18:20.115 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2024-09-17T18:18:20.115 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2024-09-17T18:18:20.115 INFO:teuthology.orchestra.run.smithi121.stdout: mgr/telemetry/nag = false 2024-09-17T18:18:20.115 INFO:teuthology.orchestra.run.smithi121.stdout: debug monc = 10 2024-09-17T18:18:20.115 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:20.115 INFO:teuthology.orchestra.run.smithi121.stdout:[mon] 2024-09-17T18:18:20.115 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-09-17T18:18:20.115 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2024-09-17T18:18:20.115 INFO:teuthology.orchestra.run.smithi121.stdout: debug paxos = 20 2024-09-17T18:18:20.115 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2024-09-17T18:18:20.115 INFO:teuthology.orchestra.run.smithi121.stdout: mon data avail warn = 5 2024-09-17T18:18:20.115 INFO:teuthology.orchestra.run.smithi121.stdout: mon mgr mkfs grace = 240 2024-09-17T18:18:20.115 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2024-09-17T18:18:20.115 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd reporter subtree level = osd 2024-09-17T18:18:20.115 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd prime pg temp = true 2024-09-17T18:18:20.116 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2024-09-17T18:18:20.116 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:20.116 INFO:teuthology.orchestra.run.smithi121.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T18:18:20.116 INFO:teuthology.orchestra.run.smithi121.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T18:18:20.116 INFO:teuthology.orchestra.run.smithi121.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T18:18:20.116 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:20.116 INFO:teuthology.orchestra.run.smithi121.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T18:18:20.116 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T18:18:20.116 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T18:18:20.116 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:20.116 INFO:teuthology.orchestra.run.smithi121.stdout: # 1m isn't quite enough 2024-09-17T18:18:20.116 INFO:teuthology.orchestra.run.smithi121.stdout: mon_down_mkfs_grace = 2m 2024-09-17T18:18:20.117 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:20.117 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_filestore_osds = false 2024-09-17T18:18:20.117 INFO:teuthology.orchestra.run.smithi121.stdout: mon min osdmap epochs = 25 2024-09-17T18:18:20.117 INFO:teuthology.orchestra.run.smithi121.stdout: mon osdmap full prune interval = 2 2024-09-17T18:18:20.117 INFO:teuthology.orchestra.run.smithi121.stdout: mon osdmap full prune min = 15 2024-09-17T18:18:20.117 INFO:teuthology.orchestra.run.smithi121.stdout: mon osdmap full prune txsize = 2 2024-09-17T18:18:20.117 INFO:teuthology.orchestra.run.smithi121.stdout: mon scrub inject crc mismatch = 0.01 2024-09-17T18:18:20.117 INFO:teuthology.orchestra.run.smithi121.stdout: mon scrub inject missing keys = 0.05 2024-09-17T18:18:20.117 INFO:teuthology.orchestra.run.smithi121.stdout: mon scrub interval = 300 2024-09-17T18:18:20.117 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on pool no app = False 2024-09-17T18:18:20.117 INFO:teuthology.orchestra.run.smithi121.stdout: paxos service trim min = 5 2024-09-17T18:18:20.117 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:20.117 INFO:teuthology.orchestra.run.smithi121.stdout:[client] 2024-09-17T18:18:20.117 INFO:teuthology.orchestra.run.smithi121.stdout: rgw cache enabled = true 2024-09-17T18:18:20.117 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable ops log = true 2024-09-17T18:18:20.118 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable usage log = true 2024-09-17T18:18:20.118 INFO:teuthology.orchestra.run.smithi121.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T18:18:20.118 INFO:teuthology.orchestra.run.smithi121.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T18:18:20.118 INFO:teuthology.orchestra.run.smithi121.stdout: debug monc = 20 2024-09-17T18:18:20.118 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-09-17T18:18:20.118 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.a] 2024-09-17T18:18:20.118 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.c] 2024-09-17T18:18:20.118 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.b] 2024-09-17T18:18:20.119 INFO:teuthology.orchestra.run.smithi173.stdout:[global] 2024-09-17T18:18:20.119 INFO:teuthology.orchestra.run.smithi173.stdout: chdir = "" 2024-09-17T18:18:20.119 INFO:teuthology.orchestra.run.smithi173.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T18:18:20.119 INFO:teuthology.orchestra.run.smithi173.stdout: auth supported = cephx 2024-09-17T18:18:20.119 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:20.119 INFO:teuthology.orchestra.run.smithi173.stdout: filestore xattr use omap = true 2024-09-17T18:18:20.119 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:20.119 INFO:teuthology.orchestra.run.smithi173.stdout: mon clock drift allowed = 1.000 2024-09-17T18:18:20.119 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:20.119 INFO:teuthology.orchestra.run.smithi173.stdout: osd crush chooseleaf type = 0 2024-09-17T18:18:20.119 INFO:teuthology.orchestra.run.smithi173.stdout: auth debug = true 2024-09-17T18:18:20.119 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:20.120 INFO:teuthology.orchestra.run.smithi173.stdout: ms die on old message = true 2024-09-17T18:18:20.120 INFO:teuthology.orchestra.run.smithi173.stdout: ms die on bug = true 2024-09-17T18:18:20.120 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:20.120 INFO:teuthology.orchestra.run.smithi173.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T18:18:20.120 INFO:teuthology.orchestra.run.smithi173.stdout: mon pg warn max object skew = 0 2024-09-17T18:18:20.120 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:20.120 INFO:teuthology.orchestra.run.smithi173.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T18:18:20.120 INFO:teuthology.orchestra.run.smithi173.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T18:18:20.120 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:20.120 INFO:teuthology.orchestra.run.smithi173.stdout: osd pool default size = 2 2024-09-17T18:18:20.120 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:20.120 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd allow primary affinity = true 2024-09-17T18:18:20.120 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd allow pg remap = true 2024-09-17T18:18:20.120 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on legacy crush tunables = false 2024-09-17T18:18:20.121 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on crush straw calc version zero = false 2024-09-17T18:18:20.121 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on no sortbitwise = false 2024-09-17T18:18:20.121 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on osd down out interval zero = false 2024-09-17T18:18:20.121 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on too few osds = false 2024-09-17T18:18:20.121 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T18:18:20.121 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T18:18:20.121 INFO:teuthology.orchestra.run.smithi173.stdout: mon_allow_pool_size_one = true 2024-09-17T18:18:20.121 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:20.121 INFO:teuthology.orchestra.run.smithi173.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T18:18:20.121 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:20.121 INFO:teuthology.orchestra.run.smithi173.stdout: osd default data pool replay window = 5 2024-09-17T18:18:20.121 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:20.121 INFO:teuthology.orchestra.run.smithi173.stdout: mon allow pool delete = true 2024-09-17T18:18:20.121 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:20.122 INFO:teuthology.orchestra.run.smithi173.stdout: mon cluster log file level = debug 2024-09-17T18:18:20.122 INFO:teuthology.orchestra.run.smithi173.stdout: debug asserts on shutdown = true 2024-09-17T18:18:20.122 INFO:teuthology.orchestra.run.smithi173.stdout: mon health detail to clog = false 2024-09-17T18:18:20.122 INFO:teuthology.orchestra.run.smithi173.stdout: mon host = "172.21.15.121,[v2:172.21.15.121:3301,v1:172.21.15.121:6790],172.21.15.173" 2024-09-17T18:18:20.122 INFO:teuthology.orchestra.run.smithi173.stdout: debug ms = 1 2024-09-17T18:18:20.122 INFO:teuthology.orchestra.run.smithi173.stdout: debug objecter = 20 2024-09-17T18:18:20.122 INFO:teuthology.orchestra.run.smithi173.stdout: debug rados = 20 2024-09-17T18:18:20.122 INFO:teuthology.orchestra.run.smithi173.stdout: mon client directed command retry = 5 2024-09-17T18:18:20.122 INFO:teuthology.orchestra.run.smithi173.stdout: mon election default strategy = 1 2024-09-17T18:18:20.122 INFO:teuthology.orchestra.run.smithi173.stdout: ms bind msgr1 = False 2024-09-17T18:18:20.122 INFO:teuthology.orchestra.run.smithi173.stdout: ms bind msgr2 = True 2024-09-17T18:18:20.122 INFO:teuthology.orchestra.run.smithi173.stdout: ms inject delay max = 1 2024-09-17T18:18:20.122 INFO:teuthology.orchestra.run.smithi173.stdout: ms inject delay probability = 0.005 2024-09-17T18:18:20.122 INFO:teuthology.orchestra.run.smithi173.stdout: ms inject delay type = mon 2024-09-17T18:18:20.123 INFO:teuthology.orchestra.run.smithi173.stdout: ms inject internal delays = 0.002 2024-09-17T18:18:20.123 INFO:teuthology.orchestra.run.smithi173.stdout: ms inject socket failures = 2500 2024-09-17T18:18:20.123 INFO:teuthology.orchestra.run.smithi173.stdout: ms type = async 2024-09-17T18:18:20.123 INFO:teuthology.orchestra.run.smithi173.stdout: fsid = a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:20.123 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:20.123 INFO:teuthology.orchestra.run.smithi173.stdout:[osd] 2024-09-17T18:18:20.123 INFO:teuthology.orchestra.run.smithi173.stdout: osd journal size = 100 2024-09-17T18:18:20.123 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:20.123 INFO:teuthology.orchestra.run.smithi173.stdout: osd scrub load threshold = 5.0 2024-09-17T18:18:20.123 INFO:teuthology.orchestra.run.smithi173.stdout: osd scrub max interval = 600 2024-09-17T18:18:20.123 INFO:teuthology.orchestra.run.smithi173.stdout: osd mclock profile = high_recovery_ops 2024-09-17T18:18:20.123 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:20.123 INFO:teuthology.orchestra.run.smithi173.stdout: osd recover clone overlap = true 2024-09-17T18:18:20.123 INFO:teuthology.orchestra.run.smithi173.stdout: osd recovery max chunk = 1048576 2024-09-17T18:18:20.123 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:20.124 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug shutdown = true 2024-09-17T18:18:20.124 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug op order = true 2024-09-17T18:18:20.124 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug verify stray on activate = true 2024-09-17T18:18:20.124 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug trim objects = true 2024-09-17T18:18:20.124 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:20.124 INFO:teuthology.orchestra.run.smithi173.stdout: osd open classes on start = true 2024-09-17T18:18:20.124 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug pg log writeout = true 2024-09-17T18:18:20.124 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:20.124 INFO:teuthology.orchestra.run.smithi173.stdout: osd deep scrub update digest min age = 30 2024-09-17T18:18:20.124 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:20.124 INFO:teuthology.orchestra.run.smithi173.stdout: osd map max advance = 10 2024-09-17T18:18:20.124 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:20.124 INFO:teuthology.orchestra.run.smithi173.stdout: journal zero on create = true 2024-09-17T18:18:20.124 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:20.125 INFO:teuthology.orchestra.run.smithi173.stdout: filestore ondisk finisher threads = 3 2024-09-17T18:18:20.125 INFO:teuthology.orchestra.run.smithi173.stdout: filestore apply finisher threads = 3 2024-09-17T18:18:20.125 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:20.125 INFO:teuthology.orchestra.run.smithi173.stdout: bdev debug aio = true 2024-09-17T18:18:20.125 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug misdirected ops = true 2024-09-17T18:18:20.125 INFO:teuthology.orchestra.run.smithi173.stdout: bluestore block size = 96636764160 2024-09-17T18:18:20.125 INFO:teuthology.orchestra.run.smithi173.stdout: bluestore compression algorithm = snappy 2024-09-17T18:18:20.125 INFO:teuthology.orchestra.run.smithi173.stdout: bluestore compression mode = aggressive 2024-09-17T18:18:20.125 INFO:teuthology.orchestra.run.smithi173.stdout: bluestore fsck on mount = True 2024-09-17T18:18:20.125 INFO:teuthology.orchestra.run.smithi173.stdout: bluestore write v2 = False 2024-09-17T18:18:20.125 INFO:teuthology.orchestra.run.smithi173.stdout: bluestore zero block detection = True 2024-09-17T18:18:20.125 INFO:teuthology.orchestra.run.smithi173.stdout: debug bluefs = 20 2024-09-17T18:18:20.125 INFO:teuthology.orchestra.run.smithi173.stdout: debug bluestore = 20 2024-09-17T18:18:20.125 INFO:teuthology.orchestra.run.smithi173.stdout: debug ms = 1 2024-09-17T18:18:20.125 INFO:teuthology.orchestra.run.smithi173.stdout: debug osd = 20 2024-09-17T18:18:20.126 INFO:teuthology.orchestra.run.smithi173.stdout: debug rocksdb = 10 2024-09-17T18:18:20.126 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T18:18:20.126 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd full ratio = 0.9 2024-09-17T18:18:20.126 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd nearfull ratio = 0.8 2024-09-17T18:18:20.126 INFO:teuthology.orchestra.run.smithi173.stdout: osd class default list = * 2024-09-17T18:18:20.126 INFO:teuthology.orchestra.run.smithi173.stdout: osd class load list = * 2024-09-17T18:18:20.126 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug verify cached snaps = True 2024-09-17T18:18:20.126 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug verify missing on start = True 2024-09-17T18:18:20.126 INFO:teuthology.orchestra.run.smithi173.stdout: osd failsafe full ratio = 0.95 2024-09-17T18:18:20.126 INFO:teuthology.orchestra.run.smithi173.stdout: osd mclock override recovery settings = True 2024-09-17T18:18:20.126 INFO:teuthology.orchestra.run.smithi173.stdout: osd objectstore = bluestore 2024-09-17T18:18:20.126 INFO:teuthology.orchestra.run.smithi173.stdout: osd op queue = debug_random 2024-09-17T18:18:20.126 INFO:teuthology.orchestra.run.smithi173.stdout: osd op queue cut off = debug_random 2024-09-17T18:18:20.126 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:20.127 INFO:teuthology.orchestra.run.smithi173.stdout:[mgr] 2024-09-17T18:18:20.127 INFO:teuthology.orchestra.run.smithi173.stdout: debug ms = 1 2024-09-17T18:18:20.127 INFO:teuthology.orchestra.run.smithi173.stdout: debug mgr = 20 2024-09-17T18:18:20.127 INFO:teuthology.orchestra.run.smithi173.stdout: debug mon = 20 2024-09-17T18:18:20.127 INFO:teuthology.orchestra.run.smithi173.stdout: debug auth = 20 2024-09-17T18:18:20.127 INFO:teuthology.orchestra.run.smithi173.stdout: mon reweight min pgs per osd = 4 2024-09-17T18:18:20.127 INFO:teuthology.orchestra.run.smithi173.stdout: mon reweight min bytes per osd = 10 2024-09-17T18:18:20.127 INFO:teuthology.orchestra.run.smithi173.stdout: mgr/telemetry/nag = false 2024-09-17T18:18:20.127 INFO:teuthology.orchestra.run.smithi173.stdout: debug monc = 10 2024-09-17T18:18:20.127 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:20.127 INFO:teuthology.orchestra.run.smithi173.stdout:[mon] 2024-09-17T18:18:20.127 INFO:teuthology.orchestra.run.smithi173.stdout: debug ms = 1 2024-09-17T18:18:20.127 INFO:teuthology.orchestra.run.smithi173.stdout: debug mon = 20 2024-09-17T18:18:20.127 INFO:teuthology.orchestra.run.smithi173.stdout: debug paxos = 20 2024-09-17T18:18:20.127 INFO:teuthology.orchestra.run.smithi173.stdout: debug auth = 20 2024-09-17T18:18:20.128 INFO:teuthology.orchestra.run.smithi173.stdout: mon data avail warn = 5 2024-09-17T18:18:20.128 INFO:teuthology.orchestra.run.smithi173.stdout: mon mgr mkfs grace = 240 2024-09-17T18:18:20.128 INFO:teuthology.orchestra.run.smithi173.stdout: mon reweight min pgs per osd = 4 2024-09-17T18:18:20.128 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd reporter subtree level = osd 2024-09-17T18:18:20.128 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd prime pg temp = true 2024-09-17T18:18:20.128 INFO:teuthology.orchestra.run.smithi173.stdout: mon reweight min bytes per osd = 10 2024-09-17T18:18:20.128 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:20.128 INFO:teuthology.orchestra.run.smithi173.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T18:18:20.128 INFO:teuthology.orchestra.run.smithi173.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T18:18:20.128 INFO:teuthology.orchestra.run.smithi173.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T18:18:20.128 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:20.128 INFO:teuthology.orchestra.run.smithi173.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T18:18:20.128 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T18:18:20.128 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T18:18:20.128 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:20.129 INFO:teuthology.orchestra.run.smithi173.stdout: # 1m isn't quite enough 2024-09-17T18:18:20.129 INFO:teuthology.orchestra.run.smithi173.stdout: mon_down_mkfs_grace = 2m 2024-09-17T18:18:20.129 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:20.129 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_filestore_osds = false 2024-09-17T18:18:20.129 INFO:teuthology.orchestra.run.smithi173.stdout: mon min osdmap epochs = 25 2024-09-17T18:18:20.129 INFO:teuthology.orchestra.run.smithi173.stdout: mon osdmap full prune interval = 2 2024-09-17T18:18:20.129 INFO:teuthology.orchestra.run.smithi173.stdout: mon osdmap full prune min = 15 2024-09-17T18:18:20.129 INFO:teuthology.orchestra.run.smithi173.stdout: mon osdmap full prune txsize = 2 2024-09-17T18:18:20.129 INFO:teuthology.orchestra.run.smithi173.stdout: mon scrub inject crc mismatch = 0.01 2024-09-17T18:18:20.129 INFO:teuthology.orchestra.run.smithi173.stdout: mon scrub inject missing keys = 0.05 2024-09-17T18:18:20.129 INFO:teuthology.orchestra.run.smithi173.stdout: mon scrub interval = 300 2024-09-17T18:18:20.129 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on pool no app = False 2024-09-17T18:18:20.129 INFO:teuthology.orchestra.run.smithi173.stdout: paxos service trim min = 5 2024-09-17T18:18:20.130 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T18:18:20.130 INFO:teuthology.orchestra.run.smithi173.stdout:[client] 2024-09-17T18:18:20.130 INFO:teuthology.orchestra.run.smithi173.stdout: rgw cache enabled = true 2024-09-17T18:18:20.130 INFO:teuthology.orchestra.run.smithi173.stdout: rgw enable ops log = true 2024-09-17T18:18:20.130 INFO:teuthology.orchestra.run.smithi173.stdout: rgw enable usage log = true 2024-09-17T18:18:20.130 INFO:teuthology.orchestra.run.smithi173.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T18:18:20.130 INFO:teuthology.orchestra.run.smithi173.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T18:18:20.130 INFO:teuthology.orchestra.run.smithi173.stdout: debug monc = 20 2024-09-17T18:18:20.130 INFO:teuthology.orchestra.run.smithi173.stdout: debug ms = 1 2024-09-17T18:18:20.130 INFO:teuthology.orchestra.run.smithi173.stdout:[mon.a] 2024-09-17T18:18:20.130 INFO:teuthology.orchestra.run.smithi173.stdout:[mon.c] 2024-09-17T18:18:20.130 INFO:teuthology.orchestra.run.smithi173.stdout:[mon.b] 2024-09-17T18:18:20.132 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-17T18:18:20.132 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-17T18:18:20.226 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-17T18:18:20.226 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-17T18:18:20.226 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-17T18:18:20.241 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-17T18:18:20.241 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-17T18:18:20.297 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi121.front.sepia.ceph.com 2024-09-17T18:18:20.297 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-17T18:18:20.297 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-17T18:18:20.297 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-17T18:18:20.371 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-17T18:18:20.372 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-17T18:18:20.426 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi173.front.sepia.ceph.com 2024-09-17T18:18:20.427 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T18:18:20.427 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-17T18:18:20.427 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-17T18:18:20.464 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T18:18:20.464 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-17T18:18:20.521 INFO:tasks.ceph:Setting up mon nodes... 2024-09-17T18:18:20.522 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-17T18:18:20.522 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-17T18:18:20.647 INFO:teuthology.orchestra.run.smithi173.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-17T18:18:20.650 INFO:tasks.ceph:Setting up mds nodes... 2024-09-17T18:18:20.650 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-17T18:18:20.651 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-17T18:18:20.693 INFO:teuthology.orchestra.run.smithi173.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-17T18:18:20.706 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-17T18:18:20.706 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi121.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi173.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-17T18:18:20.707 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-17T18:18:20.732 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-17T18:18:20.732 INFO:tasks.ceph:role: osd.0 2024-09-17T18:18:20.732 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi121.front.sepia.ceph.com 2024-09-17T18:18:20.732 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-17T18:18:20.804 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T18:18:20.804 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T18:18:20.804 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T18:18:20.804 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T18:18:20.804 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T18:18:20.804 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2024-09-17T18:18:20.804 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T18:18:20.804 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T18:18:20.804 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T18:18:20.805 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T18:18:21.284 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2024-09-17T18:18:21.285 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2024-09-17T18:18:21.285 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-17T18:18:21.402 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-17T18:18:21.431 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-17T18:18:21.496 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-17T18:18:21.496 INFO:tasks.ceph:role: osd.1 2024-09-17T18:18:21.496 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi121.front.sepia.ceph.com 2024-09-17T18:18:21.496 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-17T18:18:21.568 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T18:18:21.568 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T18:18:21.568 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T18:18:21.568 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T18:18:21.568 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T18:18:21.568 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2024-09-17T18:18:21.569 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T18:18:21.569 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T18:18:21.569 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T18:18:21.569 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T18:18:22.030 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2024-09-17T18:18:22.032 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2024-09-17T18:18:22.033 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-17T18:18:22.107 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-17T18:18:22.176 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-17T18:18:22.242 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-17T18:18:22.242 INFO:tasks.ceph:role: osd.2 2024-09-17T18:18:22.242 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi121.front.sepia.ceph.com 2024-09-17T18:18:22.243 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-17T18:18:22.314 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T18:18:22.315 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T18:18:22.315 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T18:18:22.315 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T18:18:22.315 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T18:18:22.315 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2024-09-17T18:18:22.315 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T18:18:22.315 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T18:18:22.315 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T18:18:22.315 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T18:18:22.796 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2024-09-17T18:18:22.798 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2024-09-17T18:18:22.798 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-17T18:18:22.903 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-17T18:18:22.933 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T18:18:23.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:23.026+0000 7f400d00d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-17T18:18:23.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:23.031+0000 7f400d00d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-17T18:18:23.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:23.031+0000 7f400d00d640 -1 bdev(0x556c6d3bd800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-17T18:18:23.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:23.031+0000 7f400d00d640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-17T18:18:26.681 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-17T18:18:26.710 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T18:18:26.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:26.798+0000 7fc7a3a15640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-17T18:18:26.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:26.798+0000 7fc7a3a15640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-17T18:18:26.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:26.798+0000 7fc7a3a15640 -1 bdev(0x560e7ed1d800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-17T18:18:26.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:26.798+0000 7fc7a3a15640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-17T18:18:29.962 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-17T18:18:29.992 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T18:18:30.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:30.082+0000 7f1217e0c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-17T18:18:30.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:30.082+0000 7f1217e0c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-17T18:18:30.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:30.082+0000 7f1217e0c640 -1 bdev(0x55979477f800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-17T18:18:30.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:30.082+0000 7f1217e0c640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-17T18:18:33.726 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-17T18:18:33.753 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-17T18:18:33.784 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-17T18:18:33.784 INFO:tasks.ceph:role: osd.3 2024-09-17T18:18:33.784 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi173.front.sepia.ceph.com 2024-09-17T18:18:33.784 DEBUG:teuthology.orchestra.run.smithi173:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-17T18:18:33.860 INFO:teuthology.orchestra.run.smithi173.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T18:18:33.860 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T18:18:33.860 INFO:teuthology.orchestra.run.smithi173.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T18:18:33.860 INFO:teuthology.orchestra.run.smithi173.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T18:18:33.860 INFO:teuthology.orchestra.run.smithi173.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T18:18:33.860 INFO:teuthology.orchestra.run.smithi173.stdout: = sunit=0 swidth=0 blks 2024-09-17T18:18:33.861 INFO:teuthology.orchestra.run.smithi173.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T18:18:33.861 INFO:teuthology.orchestra.run.smithi173.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T18:18:33.861 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T18:18:33.861 INFO:teuthology.orchestra.run.smithi173.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T18:18:34.374 INFO:teuthology.orchestra.run.smithi173.stdout:Discarding blocks...Done. 2024-09-17T18:18:34.376 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi173.front.sepia.ceph.com -o noatime 2024-09-17T18:18:34.377 DEBUG:teuthology.orchestra.run.smithi173:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-17T18:18:34.521 DEBUG:teuthology.orchestra.run.smithi173:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-17T18:18:34.555 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-17T18:18:34.622 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-17T18:18:34.623 INFO:tasks.ceph:role: osd.4 2024-09-17T18:18:34.623 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi173.front.sepia.ceph.com 2024-09-17T18:18:34.623 DEBUG:teuthology.orchestra.run.smithi173:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-17T18:18:34.696 INFO:teuthology.orchestra.run.smithi173.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T18:18:34.697 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T18:18:34.697 INFO:teuthology.orchestra.run.smithi173.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T18:18:34.697 INFO:teuthology.orchestra.run.smithi173.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T18:18:34.697 INFO:teuthology.orchestra.run.smithi173.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T18:18:34.697 INFO:teuthology.orchestra.run.smithi173.stdout: = sunit=0 swidth=0 blks 2024-09-17T18:18:34.697 INFO:teuthology.orchestra.run.smithi173.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T18:18:34.697 INFO:teuthology.orchestra.run.smithi173.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T18:18:34.697 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T18:18:34.697 INFO:teuthology.orchestra.run.smithi173.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T18:18:35.141 INFO:teuthology.orchestra.run.smithi173.stdout:Discarding blocks...Done. 2024-09-17T18:18:35.144 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi173.front.sepia.ceph.com -o noatime 2024-09-17T18:18:35.145 DEBUG:teuthology.orchestra.run.smithi173:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-17T18:18:35.242 DEBUG:teuthology.orchestra.run.smithi173:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-17T18:18:35.274 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-17T18:18:35.340 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-17T18:18:35.340 INFO:tasks.ceph:role: osd.5 2024-09-17T18:18:35.340 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi173.front.sepia.ceph.com 2024-09-17T18:18:35.340 DEBUG:teuthology.orchestra.run.smithi173:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-17T18:18:35.413 INFO:teuthology.orchestra.run.smithi173.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T18:18:35.413 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T18:18:35.413 INFO:teuthology.orchestra.run.smithi173.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T18:18:35.413 INFO:teuthology.orchestra.run.smithi173.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T18:18:35.414 INFO:teuthology.orchestra.run.smithi173.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T18:18:35.414 INFO:teuthology.orchestra.run.smithi173.stdout: = sunit=0 swidth=0 blks 2024-09-17T18:18:35.414 INFO:teuthology.orchestra.run.smithi173.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T18:18:35.414 INFO:teuthology.orchestra.run.smithi173.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T18:18:35.414 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T18:18:35.414 INFO:teuthology.orchestra.run.smithi173.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T18:18:35.898 INFO:teuthology.orchestra.run.smithi173.stdout:Discarding blocks...Done. 2024-09-17T18:18:35.901 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi173.front.sepia.ceph.com -o noatime 2024-09-17T18:18:35.901 DEBUG:teuthology.orchestra.run.smithi173:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-09-17T18:18:36.027 DEBUG:teuthology.orchestra.run.smithi173:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-17T18:18:36.060 DEBUG:teuthology.orchestra.run.smithi173:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T18:18:36.157 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:36.155+0000 7fa990d4d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-17T18:18:36.158 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:36.155+0000 7fa990d4d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-17T18:18:36.158 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:36.155+0000 7fa990d4d640 -1 bdev(0x55c3f5329800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-17T18:18:36.158 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:36.155+0000 7fa990d4d640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-17T18:18:39.316 DEBUG:teuthology.orchestra.run.smithi173:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-17T18:18:39.345 DEBUG:teuthology.orchestra.run.smithi173:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T18:18:39.437 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:39.435+0000 7f069340d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-17T18:18:39.437 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:39.435+0000 7f069340d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-17T18:18:39.437 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:39.435+0000 7f069340d640 -1 bdev(0x5560e40c3800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-17T18:18:39.437 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:39.435+0000 7f069340d640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-17T18:18:42.353 DEBUG:teuthology.orchestra.run.smithi173:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-17T18:18:42.383 DEBUG:teuthology.orchestra.run.smithi173:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T18:18:42.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:42.471+0000 7f000541c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-17T18:18:42.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:42.471+0000 7f000541c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-17T18:18:42.474 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:42.471+0000 7f000541c640 -1 bdev(0x564b41841800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-17T18:18:42.474 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:42.471+0000 7f000541c640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-17T18:18:45.380 DEBUG:teuthology.orchestra.run.smithi173:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-17T18:18:45.409 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-17T18:18:45.409 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-17T18:18:45.409 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-17T18:18:45.439 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-17T18:18:45.439 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-17T18:18:45.509 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-17T18:18:45.509 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-17T18:18:45.577 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T18:18:45.577 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-17T18:18:45.606 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T18:18:45.606 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-17T18:18:45.675 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T18:18:45.675 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-17T18:18:45.743 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T18:18:45.743 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-17T18:18:45.811 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T18:18:45.811 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-17T18:18:45.867 INFO:tasks.ceph:Adding keys to all mons... 2024-09-17T18:18:45.867 DEBUG:teuthology.orchestra.run.smithi121:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-17T18:18:45.870 DEBUG:teuthology.orchestra.run.smithi173:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-17T18:18:45.911 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.0] 2024-09-17T18:18:45.911 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDvx+lmXf+wARAA13nP4Y0ToV6VkIagvwD+NA== 2024-09-17T18:18:45.911 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.1] 2024-09-17T18:18:45.911 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDyx+lm0xiqLxAAhRbunzM5AvQOtUHqNC/H/g== 2024-09-17T18:18:45.912 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.2] 2024-09-17T18:18:45.912 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQD2x+lm5XsCBRAALy2RhaMe4uNF/LqyWR9esQ== 2024-09-17T18:18:45.912 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr.x] 2024-09-17T18:18:45.912 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDsx+lmbYKPJhAABH7fxyTI2ojlZKkzPqigEA== 2024-09-17T18:18:45.912 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.3] 2024-09-17T18:18:45.912 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQD8x+lmuuBOCRAA0MezgNvqYOYZ9xPLuMvBcw== 2024-09-17T18:18:45.912 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.4] 2024-09-17T18:18:45.912 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQD/x+lm7DP8GRAAtBHYDNqY+tx05V60uJsvWA== 2024-09-17T18:18:45.912 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.5] 2024-09-17T18:18:45.912 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQACyOlmw0ImHBAAuqfUD1dvBS6yNBf4QtbLnw== 2024-09-17T18:18:45.912 INFO:teuthology.orchestra.run.smithi121.stdout:[client.0] 2024-09-17T18:18:45.912 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDsx+lmpQdOKRAAfmSXyTsSq9auSsPtx5yNbw== 2024-09-17T18:18:45.932 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.0] 2024-09-17T18:18:45.932 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQDvx+lmXf+wARAA13nP4Y0ToV6VkIagvwD+NA== 2024-09-17T18:18:45.933 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.1] 2024-09-17T18:18:45.933 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQDyx+lm0xiqLxAAhRbunzM5AvQOtUHqNC/H/g== 2024-09-17T18:18:45.933 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.2] 2024-09-17T18:18:45.933 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQD2x+lm5XsCBRAALy2RhaMe4uNF/LqyWR9esQ== 2024-09-17T18:18:45.933 INFO:teuthology.orchestra.run.smithi173.stdout:[mgr.x] 2024-09-17T18:18:45.933 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQDsx+lmbYKPJhAABH7fxyTI2ojlZKkzPqigEA== 2024-09-17T18:18:45.933 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.3] 2024-09-17T18:18:45.933 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQD8x+lmuuBOCRAA0MezgNvqYOYZ9xPLuMvBcw== 2024-09-17T18:18:45.933 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.4] 2024-09-17T18:18:45.933 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQD/x+lm7DP8GRAAtBHYDNqY+tx05V60uJsvWA== 2024-09-17T18:18:45.933 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.5] 2024-09-17T18:18:45.933 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQACyOlmw0ImHBAAuqfUD1dvBS6yNBf4QtbLnw== 2024-09-17T18:18:45.933 INFO:teuthology.orchestra.run.smithi173.stdout:[client.0] 2024-09-17T18:18:45.934 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQDsx+lmpQdOKRAAfmSXyTsSq9auSsPtx5yNbw== 2024-09-17T18:18:45.934 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T18:18:45.956 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T18:18:46.029 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T18:18:46.031 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T18:18:46.083 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T18:18:46.086 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T18:18:46.136 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-17T18:18:46.139 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-17T18:18:46.190 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T18:18:46.192 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T18:18:46.243 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T18:18:46.245 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T18:18:46.295 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T18:18:46.298 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T18:18:46.349 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T18:18:46.351 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T18:18:46.405 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-17T18:18:46.405 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-17T18:18:46.429 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-17T18:18:46.524 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-17T18:18:46.549 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-17T18:18:46.614 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-17T18:18:46.710 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-17T18:18:46.734 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-17T18:18:46.759 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-17T18:18:46.857 DEBUG:teuthology.orchestra.run.smithi173:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-17T18:18:46.881 DEBUG:teuthology.orchestra.run.smithi121:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-17T18:18:46.884 DEBUG:teuthology.orchestra.run.smithi173:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-17T18:18:46.937 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-17T18:18:46.937 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-17T18:18:46.938 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-17T18:18:46.940 INFO:tasks.ceph.mon.a:Started 2024-09-17T18:18:46.940 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-17T18:18:46.940 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-17T18:18:46.942 INFO:tasks.ceph.mon.c:Started 2024-09-17T18:18:46.942 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-17T18:18:46.942 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-17T18:18:46.980 INFO:tasks.ceph.mon.b:Started 2024-09-17T18:18:46.980 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-17T18:18:46.981 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-17T18:18:46.981 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-17T18:18:46.983 INFO:tasks.ceph.mgr.x:Started 2024-09-17T18:18:46.983 DEBUG:tasks.ceph:set 0 configs 2024-09-17T18:18:46.983 INFO:tasks.ceph:Setting crush tunables to default 2024-09-17T18:18:46.983 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd crush tunables default 2024-09-17T18:18:47.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.080+0000 7f917b631640 10 monclient: get_monmap_and_config 2024-09-17T18:18:47.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.080+0000 7f917b631640 10 monclient: build_initial_monmap 2024-09-17T18:18:47.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.080+0000 7f917b631640 10 monclient: monmap: 2024-09-17T18:18:47.082 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:18:47.082 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:47.082 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:47.081194+0000 2024-09-17T18:18:47.082 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:47.081194+0000 2024-09-17T18:18:47.083 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:18:47.083 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:18:47.083 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:18:47.083 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:18:47.083 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:18:47.083 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:18:47.083 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:18:47.083 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:18:47.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.081+0000 7f917b631640 1 Processor -- start 2024-09-17T18:18:47.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.081+0000 7f917b631640 1 -- start start 2024-09-17T18:18:47.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.081+0000 7f917b631640 10 monclient: init 2024-09-17T18:18:47.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.081+0000 7f917b631640 10 monclient: _reopen_session rank -1 2024-09-17T18:18:47.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.081+0000 7f917b631640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-17T18:18:47.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.081+0000 7f917b631640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f91740a5b90 0x7f91740a5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:47.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.081+0000 7f917b631640 10 monclient(hunting): picked mon.noname-c con 0x7f91740a5b90 addr v2:172.21.15.173:3300/0 2024-09-17T18:18:47.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.081+0000 7f917b631640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f91740a71c0 0x7f91740a64b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:47.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.081+0000 7f917b631640 10 monclient(hunting): picked mon.noname-b con 0x7f91740a71c0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:18:47.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.081+0000 7f917b631640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f91741389c0 addr v1:172.21.15.121:6789/0 2024-09-17T18:18:47.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.081+0000 7f917b631640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f917409cb90 con 0x7f91741389c0 2024-09-17T18:18:47.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.081+0000 7f917b631640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:47.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.081+0000 7f917b631640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f917409d4c0 con 0x7f91740a5b90 2024-09-17T18:18:47.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.081+0000 7f917b631640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:47.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.081+0000 7f917b631640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f91740a6c50 con 0x7f91740a71c0 2024-09-17T18:18:47.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.081+0000 7f917b631640 10 monclient(hunting): _renew_subs 2024-09-17T18:18:47.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.081+0000 7f917b631640 10 monclient(hunting): authenticate will time out at 730.797850s 2024-09-17T18:18:47.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.081+0000 7f9178bcc640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f91740a71c0 0x7f91740a64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:47.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.081+0000 7f9179bce640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f91741389c0 0x7f9174138dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:40202/0 (socket says 172.21.15.121:40202) 2024-09-17T18:18:47.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.081+0000 7f9179bce640 1 -- 172.21.15.121:0/1882254377 learned_addr learned my addr 172.21.15.121:0/1882254377 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:18:47.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.082+0000 7f9178bcc640 10 monclient(hunting): get_auth_request con 0x7f91740a71c0 auth_method 0 2024-09-17T18:18:47.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.082+0000 7f9178bcc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:47.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.082+0000 7f9178bcc640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:47.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.082+0000 7f9178bcc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:47.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.082+0000 7f9178bcc640 1 -- 172.21.15.121:0/1882254377 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f91740a71c0 msgr2=0x7f91740a64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T18:18:47.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.082+0000 7f9178bcc640 1 -- 172.21.15.121:0/1882254377 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f91740a71c0 msgr2=0x7f91740a64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:18:47.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.082+0000 7f9178bcc640 1 --2- 172.21.15.121:0/1882254377 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f91740a71c0 0x7f91740a64b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:18:47.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.082+0000 7f9178bcc640 1 --2- 172.21.15.121:0/1882254377 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f91740a71c0 0x7f91740a64b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:18:47.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.082+0000 7f91793cd640 1 --2- 172.21.15.121:0/1882254377 >> v2:172.21.15.173:3300/0 conn(0x7f91740a5b90 0x7f91740a5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:47.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.082+0000 7f91793cd640 10 monclient(hunting): get_auth_request con 0x7f91740a5b90 auth_method 0 2024-09-17T18:18:47.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.082+0000 7f91793cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:47.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.082+0000 7f91793cd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:47.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.082+0000 7f91793cd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:47.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.082+0000 7f91793cd640 1 -- 172.21.15.121:0/1882254377 >> v2:172.21.15.173:3300/0 conn(0x7f91740a5b90 msgr2=0x7f91740a5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:18:47.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.082+0000 7f91793cd640 1 -- 172.21.15.121:0/1882254377 >> v2:172.21.15.173:3300/0 conn(0x7f91740a5b90 msgr2=0x7f91740a5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:18:47.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.082+0000 7f91793cd640 1 --2- 172.21.15.121:0/1882254377 >> v2:172.21.15.173:3300/0 conn(0x7f91740a5b90 0x7f91740a5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:18:47.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.082+0000 7f91793cd640 1 --2- 172.21.15.121:0/1882254377 >> v2:172.21.15.173:3300/0 conn(0x7f91740a5b90 0x7f91740a5f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:18:47.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.084+0000 7f9179bce640 1 -- 172.21.15.121:0/1882254377 >> v1:172.21.15.121:6789/0 conn(0x7f91741389c0 legacy=0x7f9174138dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:18:47.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.256+0000 7f916bfff640 1 -- 172.21.15.121:0/1882254377 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1495022942 0 0) 0x7f917409cb90 con 0x7f91741389c0 2024-09-17T18:18:47.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.256+0000 7f916bfff640 10 monclient(hunting): _negotiate 2024-09-17T18:18:47.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.256+0000 7f916bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:47.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.256+0000 7f916bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:47.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.256+0000 7f916bfff640 10 monclient(hunting): my global_id is 4098 2024-09-17T18:18:47.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.256+0000 7f916bfff640 1 -- 172.21.15.121:0/1882254377 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9160003520 con 0x7f91741389c0 2024-09-17T18:18:47.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f916bfff640 1 -- 172.21.15.121:0/1882254377 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 448469020 0 0) 0x7f9160003520 con 0x7f91741389c0 2024-09-17T18:18:47.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f916bfff640 1 -- 172.21.15.121:0/1882254377 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f91740a71c0 msgr2=0x7f91740a64b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:18:47.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f916bfff640 1 --2- 172.21.15.121:0/1882254377 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f91740a71c0 0x7f91740a64b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:47.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f916bfff640 1 -- 172.21.15.121:0/1882254377 >> v2:172.21.15.173:3300/0 conn(0x7f91740a5b90 msgr2=0x7f91740a5f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:18:47.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f916bfff640 1 --2- 172.21.15.121:0/1882254377 >> v2:172.21.15.173:3300/0 conn(0x7f91740a5b90 0x7f91740a5f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:47.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f916bfff640 10 monclient: _finish_hunting 0 2024-09-17T18:18:47.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f916bfff640 1 monclient: found mon.noname-a-legacy 2024-09-17T18:18:47.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f916bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:18:47.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f916bfff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.121:6789/0 2024-09-17T18:18:47.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f916bfff640 1 -- 172.21.15.121:0/1882254377 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9174070550 con 0x7f91741389c0 2024-09-17T18:18:47.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f916bfff640 10 monclient: _finish_auth 0 2024-09-17T18:18:47.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f916bfff640 10 monclient: _check_auth_tickets 2024-09-17T18:18:47.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f916bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:18:47.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f916bfff640 1 -- 172.21.15.121:0/1882254377 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4002019445 0 0) 0x7f916c001d20 con 0x7f91741389c0 2024-09-17T18:18:47.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f917b631640 5 monclient: authenticate success, global_id 4098 2024-09-17T18:18:47.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f917b631640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:18:47.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f916bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:47.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f916bfff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T18:18:47.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f916bfff640 10 monclient: dump: 2024-09-17T18:18:47.264 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:18:47.264 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:47.264 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:47.264 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:47.264 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:47.264 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:18:47.264 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:47.264 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:47.264 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:47.264 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:18:47.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f916bfff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] but i'm connected to v1:172.21.15.121:6789/0, reconnecting 2024-09-17T18:18:47.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f916bfff640 10 monclient: _reopen_session rank -1 2024-09-17T18:18:47.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f916bfff640 1 -- 172.21.15.121:0/1882254377 >> v1:172.21.15.121:6789/0 conn(0x7f91741389c0 legacy=0x7f9174138dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:47.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f916bfff640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:18:47.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f916bfff640 1 --2- 172.21.15.121:0/1882254377 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9160003ad0 0x7f9160003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:47.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f916bfff640 10 monclient(hunting): picked mon.a con 0x7f9160003ad0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:18:47.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f916bfff640 1 --2- 172.21.15.121:0/1882254377 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9160004800 0x7f9160004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:47.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f916bfff640 10 monclient(hunting): picked mon.b con 0x7f9160004800 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:18:47.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f916bfff640 1 --2- 172.21.15.121:0/1882254377 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f91740a71c0 0x7f916000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:47.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f916bfff640 10 monclient(hunting): picked mon.c con 0x7f91740a71c0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:18:47.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f916bfff640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:47.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f916bfff640 1 -- 172.21.15.121:0/1882254377 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f917409d4c0 con 0x7f9160003ad0 2024-09-17T18:18:47.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f916bfff640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:47.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f916bfff640 1 -- 172.21.15.121:0/1882254377 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f91600055d0 con 0x7f9160004800 2024-09-17T18:18:47.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f916bfff640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:47.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f916bfff640 1 -- 172.21.15.121:0/1882254377 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f916000bf00 con 0x7f91740a71c0 2024-09-17T18:18:47.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f916bfff640 10 monclient(hunting): _renew_subs 2024-09-17T18:18:47.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f916bfff640 10 monclient(hunting): _finish_auth 0 2024-09-17T18:18:47.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f917b631640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-17T18:18:47.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f91793cd640 1 --2- 172.21.15.121:0/1882254377 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9160003ad0 0x7f9160003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:47.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f91793cd640 10 monclient(hunting): get_auth_request con 0x7f9160003ad0 auth_method 0 2024-09-17T18:18:47.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f91793cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:47.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f91793cd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:47.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f91793cd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:18:47.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f9179bce640 1 --2- 172.21.15.121:0/1882254377 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f91740a71c0 0x7f916000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:47.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f9178bcc640 1 --2- 172.21.15.121:0/1882254377 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9160004800 0x7f9160004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:47.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f9178bcc640 10 monclient(hunting): get_auth_request con 0x7f9160004800 auth_method 0 2024-09-17T18:18:47.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f9178bcc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:47.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f9178bcc640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:47.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f9178bcc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:18:47.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f9179bce640 10 monclient(hunting): get_auth_request con 0x7f91740a71c0 auth_method 0 2024-09-17T18:18:47.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f9179bce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:47.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f9179bce640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:47.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f9179bce640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:18:47.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f91793cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:47.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f91793cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:47.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f91793cd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:18:47.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f9178bcc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:47.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f9178bcc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:47.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f9178bcc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:18:47.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f9179bce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:47.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f9179bce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:47.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f9179bce640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:18:47.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f91793cd640 10 monclient(hunting): handle_auth_done global_id 4098 payload 293 2024-09-17T18:18:47.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f91793cd640 1 -- 172.21.15.121:0/1882254377 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f91740a71c0 msgr2=0x7f916000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:18:47.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f91793cd640 1 --2- 172.21.15.121:0/1882254377 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f91740a71c0 0x7f916000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:47.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f91793cd640 1 -- 172.21.15.121:0/1882254377 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9160004800 msgr2=0x7f9160004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:18:47.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f91793cd640 1 --2- 172.21.15.121:0/1882254377 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9160004800 0x7f9160004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:47.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f91793cd640 10 monclient: _finish_hunting 0 2024-09-17T18:18:47.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f91793cd640 1 monclient: found mon.a 2024-09-17T18:18:47.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f91793cd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:18:47.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f91793cd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:18:47.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f91793cd640 1 -- 172.21.15.121:0/1882254377 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9160003520 con 0x7f9160003ad0 2024-09-17T18:18:47.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f91793cd640 10 monclient: _finish_auth 0 2024-09-17T18:18:47.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f91793cd640 10 monclient: _check_auth_tickets 2024-09-17T18:18:47.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.257+0000 7f91793cd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:18:47.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.260+0000 7f91793cd640 1 -- 172.21.15.121:0/1882254377 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9160003ad0 msgr2=0x7f9160003ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:18:47.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.260+0000 7f91793cd640 1 --2- 172.21.15.121:0/1882254377 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9160003ad0 0x7f9160003ef0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f917000ba10 tx=0x7f9160005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8656749612a8cab3 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:18:47.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.260+0000 7f916bfff640 1 -- 172.21.15.121:0/1882254377 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9170030070 con 0x7f9160003ad0 2024-09-17T18:18:47.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.260+0000 7f916bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:47.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.260+0000 7f916bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:18:47.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.260+0000 7f916bfff640 10 monclient: dump: 2024-09-17T18:18:47.274 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:18:47.274 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:47.275 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:47.275 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:47.275 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:47.275 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:18:47.275 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:47.275 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:47.275 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:47.275 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:18:47.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.260+0000 7f916bfff640 1 -- 172.21.15.121:0/1882254377 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9170034600 con 0x7f9160003ad0 2024-09-17T18:18:47.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.260+0000 7f917b631640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:18:47.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.260+0000 7f916bfff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:18:47.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.260+0000 7f917b631640 10 monclient: get_monmap_and_config success 2024-09-17T18:18:47.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.260+0000 7f916bfff640 1 -- 172.21.15.121:0/1882254377 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f917000b5e0 con 0x7f9160003ad0 2024-09-17T18:18:47.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.260+0000 7f917b631640 4 set_mon_vals no callback set 2024-09-17T18:18:47.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.260+0000 7f916bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:47.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.260+0000 7f916bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:18:47.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.260+0000 7f916bfff640 10 monclient: dump: 2024-09-17T18:18:47.276 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:18:47.276 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:47.276 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:47.276 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:47.276 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:47.276 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:18:47.276 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:47.276 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:47.276 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:47.276 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:18:47.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.261+0000 7f917b631640 10 monclient: shutdown 2024-09-17T18:18:47.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.261+0000 7f917b631640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:18:47.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.261+0000 7f917b631640 1 -- 172.21.15.121:0/1882254377 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9160003ad0 msgr2=0x7f9160003ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:47.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.261+0000 7f917b631640 1 --2- 172.21.15.121:0/1882254377 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9160003ad0 0x7f9160003ef0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f917000ba10 tx=0x7f9160005830 comp rx=0 tx=0).stop 2024-09-17T18:18:47.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.261+0000 7f9179bce640 1 -- 172.21.15.121:0/1882254377 reap_dead start 2024-09-17T18:18:47.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.261+0000 7f917b631640 1 -- 172.21.15.121:0/1882254377 shutdown_connections 2024-09-17T18:18:47.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.261+0000 7f917b631640 1 -- 172.21.15.121:0/1882254377 >> 172.21.15.121:0/1882254377 conn(0x7f91740a7c90 msgr2=0x7f91740a80a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:18:47.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.261+0000 7f917b631640 1 -- 172.21.15.121:0/1882254377 shutdown_connections 2024-09-17T18:18:47.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.261+0000 7f917b631640 1 -- 172.21.15.121:0/1882254377 wait complete. 2024-09-17T18:18:47.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.262+0000 7f917b631640 10 monclient: build_initial_monmap 2024-09-17T18:18:47.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.262+0000 7f917b631640 10 monclient: monmap: 2024-09-17T18:18:47.277 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:18:47.277 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:18:47.277 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:18:47.278 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:18:47.278 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:18:47.278 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:18:47.278 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:18:47.278 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:18:47.278 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:18:47.278 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:18:47.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.262+0000 7f917b631640 1 librados: starting msgr at 2024-09-17T18:18:47.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.262+0000 7f917b631640 1 librados: starting objecter 2024-09-17T18:18:47.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.262+0000 7f917b631640 1 Processor -- start 2024-09-17T18:18:47.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.262+0000 7f917b631640 1 -- start start 2024-09-17T18:18:47.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.262+0000 7f917b631640 1 librados: setting wanted keys 2024-09-17T18:18:47.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.262+0000 7f917b631640 1 librados: calling monclient init 2024-09-17T18:18:47.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.262+0000 7f917b631640 10 monclient: init 2024-09-17T18:18:47.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.262+0000 7f917b631640 10 monclient: _reopen_session rank -1 2024-09-17T18:18:47.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.262+0000 7f917b631640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T18:18:47.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.262+0000 7f917b631640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9160003ad0 0x7f9174126430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:47.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.262+0000 7f917b631640 10 monclient(hunting): picked mon.noname-b con 0x7f9160003ad0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:18:47.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.262+0000 7f917b631640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9174123f00 0x7f9174124360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:47.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.262+0000 7f917b631640 10 monclient(hunting): picked mon.noname-a con 0x7f9174123f00 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:18:47.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.262+0000 7f917b631640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f91741248a0 0x7f9174124d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:47.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.262+0000 7f917b631640 10 monclient(hunting): picked mon.noname-c con 0x7f91741248a0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:18:47.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.262+0000 7f917b631640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:47.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.262+0000 7f917b631640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f91740a8ba0 con 0x7f9174123f00 2024-09-17T18:18:47.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.262+0000 7f917b631640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:47.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.262+0000 7f917b631640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f91740a8a20 con 0x7f9160003ad0 2024-09-17T18:18:47.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.262+0000 7f917b631640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:47.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.262+0000 7f917b631640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f91741394d0 con 0x7f91741248a0 2024-09-17T18:18:47.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.262+0000 7f917b631640 10 monclient(hunting): _renew_subs 2024-09-17T18:18:47.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.262+0000 7f917b631640 10 monclient(hunting): authenticate will time out at 730.978916s 2024-09-17T18:18:47.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.262+0000 7f9178bcc640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9174123f00 0x7f9174124360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:47.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.263+0000 7f9179bce640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f91741248a0 0x7f9174124d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:47.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.263+0000 7f9178bcc640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9174123f00 0x7f9174124360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39346/0 (socket says 172.21.15.121:39346) 2024-09-17T18:18:47.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.263+0000 7f91793cd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9160003ad0 0x7f9174126430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:47.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.263+0000 7f9178bcc640 1 -- 172.21.15.121:0/3547678878 learned_addr learned my addr 172.21.15.121:0/3547678878 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:18:47.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.263+0000 7f9178bcc640 10 monclient(hunting): get_auth_request con 0x7f9174123f00 auth_method 0 2024-09-17T18:18:47.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.263+0000 7f9178bcc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:47.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.263+0000 7f9178bcc640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:47.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.263+0000 7f9178bcc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:47.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.263+0000 7f9179bce640 10 monclient(hunting): get_auth_request con 0x7f91741248a0 auth_method 0 2024-09-17T18:18:47.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.263+0000 7f9179bce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:47.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.263+0000 7f9179bce640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:47.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.263+0000 7f9179bce640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:47.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.263+0000 7f91793cd640 10 monclient(hunting): get_auth_request con 0x7f9160003ad0 auth_method 0 2024-09-17T18:18:47.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.263+0000 7f91793cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:47.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.263+0000 7f91793cd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:47.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.263+0000 7f91793cd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:47.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.263+0000 7f9178bcc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:47.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.263+0000 7f9178bcc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:47.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.263+0000 7f9178bcc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:18:47.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.263+0000 7f9179bce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:47.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.263+0000 7f9179bce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:47.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.263+0000 7f9179bce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:18:47.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.263+0000 7f91793cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:47.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.263+0000 7f91793cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:47.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.263+0000 7f91793cd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:18:47.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.263+0000 7f9178bcc640 10 monclient(hunting): handle_auth_done global_id 4104 payload 824 2024-09-17T18:18:47.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.263+0000 7f9178bcc640 1 -- 172.21.15.121:0/3547678878 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f91741248a0 msgr2=0x7f9174124d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:47.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.263+0000 7f9178bcc640 1 --2- 172.21.15.121:0/3547678878 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f91741248a0 0x7f9174124d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:47.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.263+0000 7f9178bcc640 1 -- 172.21.15.121:0/3547678878 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9160003ad0 msgr2=0x7f9174126430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:47.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.263+0000 7f9178bcc640 1 --2- 172.21.15.121:0/3547678878 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9160003ad0 0x7f9174126430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:47.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.263+0000 7f9178bcc640 10 monclient: _finish_hunting 0 2024-09-17T18:18:47.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.263+0000 7f9178bcc640 1 monclient: found mon.noname-a 2024-09-17T18:18:47.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.263+0000 7f9178bcc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:18:47.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.263+0000 7f9178bcc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:18:47.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.263+0000 7f9178bcc640 1 -- 172.21.15.121:0/3547678878 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9174126dc0 con 0x7f9174123f00 2024-09-17T18:18:47.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.265+0000 7f9178bcc640 1 -- 172.21.15.121:0/3547678878 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9174123f00 msgr2=0x7f9174124360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:18:47.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.265+0000 7f9178bcc640 1 --2- 172.21.15.121:0/3547678878 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9174123f00 0x7f9174124360 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f91640108c0 tx=0x7f9164010d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:18:47.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.265+0000 7f9169ffb640 10 client.?.objecter ms_handle_connect 0x7f9174123f00 2024-09-17T18:18:47.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.265+0000 7f9169ffb640 10 client.?.objecter resend_mon_ops 2024-09-17T18:18:47.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.265+0000 7f9169ffb640 1 -- 172.21.15.121:0/3547678878 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f916401a020 con 0x7f9174123f00 2024-09-17T18:18:47.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.265+0000 7f9169ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:47.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.265+0000 7f9169ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:18:47.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.265+0000 7f9169ffb640 10 monclient: dump: 2024-09-17T18:18:47.283 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:18:47.283 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:47.283 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:47.283 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:47.283 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:47.283 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:18:47.283 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:47.283 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:47.283 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:47.283 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:18:47.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.265+0000 7f9169ffb640 10 monclient: _finish_auth 0 2024-09-17T18:18:47.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.265+0000 7f9169ffb640 10 monclient: _check_auth_tickets 2024-09-17T18:18:47.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.265+0000 7f9169ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:18:47.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.265+0000 7f9169ffb640 1 -- 172.21.15.121:0/3547678878 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f916400ebd0 con 0x7f9174123f00 2024-09-17T18:18:47.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.265+0000 7f917b631640 5 monclient: authenticate success, global_id 4104 2024-09-17T18:18:47.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.265+0000 7f917b631640 10 monclient: _renew_subs 2024-09-17T18:18:47.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.265+0000 7f917b631640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:18:47.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.265+0000 7f917b631640 1 -- 172.21.15.121:0/3547678878 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91740a2380 con 0x7f9174123f00 2024-09-17T18:18:47.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.265+0000 7f9169ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:18:47.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.265+0000 7f9169ffb640 1 -- 172.21.15.121:0/3547678878 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9164016480 con 0x7f9174123f00 2024-09-17T18:18:47.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.265+0000 7f9169ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:47.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.265+0000 7f9169ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:18:47.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.265+0000 7f9169ffb640 10 monclient: dump: 2024-09-17T18:18:47.285 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:18:47.285 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:47.285 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:47.285 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:47.285 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:47.285 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:18:47.285 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:47.285 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:47.285 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:47.285 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:18:47.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.265+0000 7f916b7fe640 4 set_mon_vals no callback set 2024-09-17T18:18:47.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.265+0000 7f917b631640 10 client.4104.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:18:47.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.265+0000 7f917b631640 10 monclient: _renew_subs 2024-09-17T18:18:47.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.265+0000 7f917b631640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:18:47.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.265+0000 7f917b631640 1 -- 172.21.15.121:0/3547678878 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9174127c30 con 0x7f9174123f00 2024-09-17T18:18:47.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.265+0000 7f917b631640 1 librados: init done 2024-09-17T18:18:47.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.266+0000 7f9169ffb640 1 -- 172.21.15.121:0/3547678878 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f916401d020 con 0x7f9174123f00 2024-09-17T18:18:47.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.266+0000 7f9169ffb640 1 -- 172.21.15.121:0/3547678878 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9164014a10 con 0x7f9174123f00 2024-09-17T18:18:47.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.266+0000 7f9169ffb640 10 client.4104.objecter ms_dispatch 0x7f9174000b90 osd_map(1..1 src has 1..1) 2024-09-17T18:18:47.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.266+0000 7f9169ffb640 3 client.4104.objecter handle_osd_map got epochs [1,1] > 0 2024-09-17T18:18:47.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.266+0000 7f9169ffb640 3 client.4104.objecter handle_osd_map decoding full epoch 1 2024-09-17T18:18:47.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.266+0000 7f9169ffb640 20 client.4104.objecter dump_active .. 0 homeless 2024-09-17T18:18:47.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.266+0000 7f917b631640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:18:47.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.266+0000 7f917b631640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:18:47.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.266+0000 7f917b631640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:18:47.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.266+0000 7f917b631640 1 -- 172.21.15.121:0/3547678878 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91741252a0 con 0x7f9174123f00 2024-09-17T18:18:47.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.268+0000 7f9169ffb640 1 -- 172.21.15.121:0/3547678878 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f9164024020 con 0x7f9174123f00 2024-09-17T18:18:47.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.268+0000 7f9169ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:18:47.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.268+0000 7f9169ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:18:47.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.331+0000 7f917b631640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-17T18:18:47.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.331+0000 7f917b631640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-17T18:18:47.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.331+0000 7f917b631640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:18:47.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.331+0000 7f917b631640 1 -- 172.21.15.121:0/3547678878 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f9174138dc0 con 0x7f9174123f00 2024-09-17T18:18:47.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.332+0000 7f9169ffb640 1 -- 172.21.15.121:0/3547678878 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f9164014d30 con 0x7f9174123f00 2024-09-17T18:18:47.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.332+0000 7f9169ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-17T18:18:47.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.332+0000 7f9169ffb640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-09-17T18:18:47.334 INFO:teuthology.orchestra.run.smithi121.stderr:adjusted tunables profile to default 2024-09-17T18:18:47.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.333+0000 7f917b631640 10 librados: watch_flush enter 2024-09-17T18:18:47.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.333+0000 7f917b631640 10 librados: watch_flush exit 2024-09-17T18:18:47.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.333+0000 7f917b631640 20 client.4104.objecter shutdown clearing up homeless session... 2024-09-17T18:18:47.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.333+0000 7f917b631640 10 client.4104.objecter successfully canceled tick 2024-09-17T18:18:47.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.333+0000 7f917b631640 10 monclient: shutdown 2024-09-17T18:18:47.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.333+0000 7f917b631640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:18:47.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.333+0000 7f917b631640 1 -- 172.21.15.121:0/3547678878 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9174123f00 msgr2=0x7f9174124360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:47.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.333+0000 7f917b631640 1 --2- 172.21.15.121:0/3547678878 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9174123f00 0x7f9174124360 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f91640108c0 tx=0x7f9164010d80 comp rx=0 tx=0).stop 2024-09-17T18:18:47.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.333+0000 7f917b631640 1 -- 172.21.15.121:0/3547678878 shutdown_connections 2024-09-17T18:18:47.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.333+0000 7f917b631640 1 --2- 172.21.15.121:0/3547678878 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f91741248a0 0x7f9174124d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:47.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.333+0000 7f917b631640 1 --2- 172.21.15.121:0/3547678878 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9174123f00 0x7f9174124360 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:47.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.333+0000 7f917b631640 1 --2- 172.21.15.121:0/3547678878 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9160003ad0 0x7f9174126430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:47.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.333+0000 7f917b631640 1 -- 172.21.15.121:0/3547678878 >> 172.21.15.121:0/3547678878 conn(0x7f91740a7c90 msgr2=0x7f917409e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:18:47.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.333+0000 7f917b631640 1 -- 172.21.15.121:0/3547678878 shutdown_connections 2024-09-17T18:18:47.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.333+0000 7f917b631640 1 -- 172.21.15.121:0/3547678878 wait complete. 2024-09-17T18:18:47.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:47.333+0000 7f917b631640 1 librados: shutdown 2024-09-17T18:18:47.345 INFO:tasks.ceph:check_enable_crimson: False 2024-09-17T18:18:47.345 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-17T18:18:47.346 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-17T18:18:47.346 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-17T18:18:47.370 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-17T18:18:47.370 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-17T18:18:47.435 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-17T18:18:47.435 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-17T18:18:47.500 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T18:18:47.500 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-17T18:18:47.525 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T18:18:47.525 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-17T18:18:47.592 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T18:18:47.592 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-17T18:18:47.657 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster ceph osd new 7bf586d1-05ac-4c7e-8b88-313f07a76801 0 2024-09-17T18:18:47.795 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.793+0000 7ff327144640 10 monclient: get_monmap_and_config 2024-09-17T18:18:47.796 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.793+0000 7ff327144640 10 monclient: build_initial_monmap 2024-09-17T18:18:47.796 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.793+0000 7ff327144640 10 monclient: monmap: 2024-09-17T18:18:47.796 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-09-17T18:18:47.796 INFO:teuthology.orchestra.run.smithi173.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:47.796 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-09-17T18:18:47.794585+0000 2024-09-17T18:18:47.796 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-09-17T18:18:47.794585+0000 2024-09-17T18:18:47.796 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-09-17T18:18:47.796 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-09-17T18:18:47.796 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:18:47.796 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:18:47.796 INFO:teuthology.orchestra.run.smithi173.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:18:47.796 INFO:teuthology.orchestra.run.smithi173.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:18:47.797 INFO:teuthology.orchestra.run.smithi173.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:18:47.797 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T18:18:47.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.794+0000 7ff327144640 1 Processor -- start 2024-09-17T18:18:47.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.794+0000 7ff327144640 1 -- start start 2024-09-17T18:18:47.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.794+0000 7ff327144640 10 monclient: init 2024-09-17T18:18:47.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.794+0000 7ff327144640 10 monclient: _reopen_session rank -1 2024-09-17T18:18:47.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.794+0000 7ff327144640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-17T18:18:47.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.794+0000 7ff327144640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7ff3200a5b90 0x7ff3200a5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:47.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.794+0000 7ff327144640 10 monclient(hunting): picked mon.noname-c con 0x7ff3200a5b90 addr v2:172.21.15.173:3300/0 2024-09-17T18:18:47.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.794+0000 7ff327144640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff3200a71c0 addr v1:172.21.15.173:6789/0 2024-09-17T18:18:47.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.794+0000 7ff327144640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff3200a64b0 addr v1:172.21.15.121:6789/0 2024-09-17T18:18:47.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.794+0000 7ff327144640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3201397e0 con 0x7ff3200a64b0 2024-09-17T18:18:47.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.794+0000 7ff327144640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff32009cb90 con 0x7ff3200a71c0 2024-09-17T18:18:47.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.794+0000 7ff327144640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:47.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.794+0000 7ff327144640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7ff32009d4c0 con 0x7ff3200a5b90 2024-09-17T18:18:47.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.794+0000 7ff327144640 10 monclient(hunting): _renew_subs 2024-09-17T18:18:47.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.794+0000 7ff327144640 10 monclient(hunting): authenticate will time out at 725.586987s 2024-09-17T18:18:47.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.795+0000 7ff324ee0640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7ff3200a5b90 0x7ff3200a5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:47.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.795+0000 7ff324ee0640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7ff3200a5b90 0x7ff3200a5f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:58080/0 (socket says 172.21.15.173:58080) 2024-09-17T18:18:47.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.795+0000 7ff31ffff640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7ff3200a71c0 0x7ff3201389c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:33796/0 (socket says 172.21.15.173:33796) 2024-09-17T18:18:47.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.795+0000 7ff324ee0640 1 -- 172.21.15.173:0/2343319134 learned_addr learned my addr 172.21.15.173:0/2343319134 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-09-17T18:18:47.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.795+0000 7ff324ee0640 10 monclient(hunting): get_auth_request con 0x7ff3200a5b90 auth_method 0 2024-09-17T18:18:47.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.795+0000 7ff324ee0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:47.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.795+0000 7ff324ee0640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:47.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.795+0000 7ff324ee0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:47.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.795+0000 7ff324ee0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:47.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.795+0000 7ff324ee0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:47.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.795+0000 7ff324ee0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:18:47.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.795+0000 7ff324ee0640 10 monclient(hunting): handle_auth_done global_id 4108 payload 274 2024-09-17T18:18:47.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.795+0000 7ff324ee0640 1 -- 172.21.15.173:0/2343319134 >> v1:172.21.15.173:6789/0 conn(0x7ff3200a71c0 legacy=0x7ff3201389c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:18:47.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.795+0000 7ff324ee0640 1 -- 172.21.15.173:0/2343319134 >> v1:172.21.15.121:6789/0 conn(0x7ff3200a64b0 legacy=0x7ff3201390d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:18:47.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.795+0000 7ff324ee0640 10 monclient: _finish_hunting 0 2024-09-17T18:18:47.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.795+0000 7ff324ee0640 1 monclient: found mon.noname-c 2024-09-17T18:18:47.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.795+0000 7ff324ee0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:18:47.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.795+0000 7ff324ee0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.173:3300/0 2024-09-17T18:18:47.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.795+0000 7ff324ee0640 1 -- 172.21.15.173:0/2343319134 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff320070550 con 0x7ff3200a5b90 2024-09-17T18:18:47.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.798+0000 7ff324ee0640 1 -- 172.21.15.173:0/2343319134 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff3200a5b90 msgr2=0x7ff3200a5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:18:47.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.798+0000 7ff324ee0640 1 --2- 172.21.15.173:0/2343319134 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff3200a5b90 0x7ff3200a5f70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff314009870 tx=0x7ff31402a0a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7dd1aa08289f4943 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:18:47.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.798+0000 7ff31f7fe640 1 -- 172.21.15.173:0/2343319134 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff314037070 con 0x7ff3200a5b90 2024-09-17T18:18:47.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.798+0000 7ff31f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:47.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.798+0000 7ff31f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:18:47.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.798+0000 7ff31f7fe640 10 monclient: dump: 2024-09-17T18:18:47.808 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-09-17T18:18:47.808 INFO:teuthology.orchestra.run.smithi173.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:47.808 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:47.808 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:47.808 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:47.808 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-09-17T18:18:47.808 INFO:teuthology.orchestra.run.smithi173.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:47.808 INFO:teuthology.orchestra.run.smithi173.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:47.808 INFO:teuthology.orchestra.run.smithi173.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:47.808 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T18:18:47.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.798+0000 7ff31f7fe640 10 monclient: _finish_auth 0 2024-09-17T18:18:47.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.798+0000 7ff31f7fe640 10 monclient: _check_auth_tickets 2024-09-17T18:18:47.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.798+0000 7ff31f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:18:47.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.798+0000 7ff31f7fe640 1 -- 172.21.15.173:0/2343319134 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff314032640 con 0x7ff3200a5b90 2024-09-17T18:18:47.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.798+0000 7ff327144640 5 monclient: authenticate success, global_id 4108 2024-09-17T18:18:47.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.798+0000 7ff327144640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:18:47.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.798+0000 7ff31f7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:18:47.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.798+0000 7ff327144640 10 monclient: get_monmap_and_config success 2024-09-17T18:18:47.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.798+0000 7ff31f7fe640 1 -- 172.21.15.173:0/2343319134 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff314032a10 con 0x7ff3200a5b90 2024-09-17T18:18:47.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.798+0000 7ff31f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:47.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.798+0000 7ff327144640 4 set_mon_vals no callback set 2024-09-17T18:18:47.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.798+0000 7ff31f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:18:47.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.798+0000 7ff31f7fe640 10 monclient: dump: 2024-09-17T18:18:47.809 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-09-17T18:18:47.809 INFO:teuthology.orchestra.run.smithi173.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:47.810 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:47.810 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:47.810 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:47.810 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-09-17T18:18:47.810 INFO:teuthology.orchestra.run.smithi173.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:47.810 INFO:teuthology.orchestra.run.smithi173.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:47.810 INFO:teuthology.orchestra.run.smithi173.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:47.810 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T18:18:47.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.798+0000 7ff327144640 10 monclient: shutdown 2024-09-17T18:18:47.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.798+0000 7ff327144640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:18:47.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.798+0000 7ff327144640 1 -- 172.21.15.173:0/2343319134 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff3200a5b90 msgr2=0x7ff3200a5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:47.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.798+0000 7ff327144640 1 --2- 172.21.15.173:0/2343319134 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff3200a5b90 0x7ff3200a5f70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff314009870 tx=0x7ff31402a0a0 comp rx=0 tx=0).stop 2024-09-17T18:18:47.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.798+0000 7ff327144640 1 -- 172.21.15.173:0/2343319134 shutdown_connections 2024-09-17T18:18:47.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.798+0000 7ff327144640 1 --2- 172.21.15.173:0/2343319134 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff3200a5b90 0x7ff3200a5f70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:47.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.798+0000 7ff327144640 1 -- 172.21.15.173:0/2343319134 >> 172.21.15.173:0/2343319134 conn(0x7ff3200a7c90 msgr2=0x7ff3200a80a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:18:47.811 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.798+0000 7ff327144640 1 -- 172.21.15.173:0/2343319134 shutdown_connections 2024-09-17T18:18:47.811 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.798+0000 7ff327144640 1 -- 172.21.15.173:0/2343319134 wait complete. 2024-09-17T18:18:47.811 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.799+0000 7ff327144640 10 monclient: build_initial_monmap 2024-09-17T18:18:47.811 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.799+0000 7ff327144640 10 monclient: monmap: 2024-09-17T18:18:47.811 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-09-17T18:18:47.811 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:18:47.811 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 0.000000 2024-09-17T18:18:47.811 INFO:teuthology.orchestra.run.smithi173.stderr:created 0.000000 2024-09-17T18:18:47.811 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-09-17T18:18:47.811 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-09-17T18:18:47.811 INFO:teuthology.orchestra.run.smithi173.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:18:47.811 INFO:teuthology.orchestra.run.smithi173.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:18:47.811 INFO:teuthology.orchestra.run.smithi173.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:18:47.811 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T18:18:47.811 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.799+0000 7ff327144640 1 librados: starting msgr at 2024-09-17T18:18:47.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.799+0000 7ff327144640 1 librados: starting objecter 2024-09-17T18:18:47.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.799+0000 7ff327144640 1 Processor -- start 2024-09-17T18:18:47.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.799+0000 7ff327144640 1 -- start start 2024-09-17T18:18:47.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.799+0000 7ff327144640 1 librados: setting wanted keys 2024-09-17T18:18:47.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.799+0000 7ff327144640 1 librados: calling monclient init 2024-09-17T18:18:47.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.799+0000 7ff327144640 10 monclient: init 2024-09-17T18:18:47.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff327144640 10 monclient: _reopen_session rank -1 2024-09-17T18:18:47.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff327144640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T18:18:47.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff327144640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff3200a5b90 0x7ff3201387c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:47.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff327144640 10 monclient(hunting): picked mon.noname-c con 0x7ff3200a5b90 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:18:47.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff327144640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff3200a64b0 0x7ff320138d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:47.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff327144640 10 monclient(hunting): picked mon.noname-b con 0x7ff3200a64b0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:18:47.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff327144640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3200a71c0 0x7ff3201263a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:47.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff327144640 10 monclient(hunting): picked mon.noname-a con 0x7ff3200a71c0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:18:47.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff327144640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:47.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff327144640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff3200a6e90 con 0x7ff3200a71c0 2024-09-17T18:18:47.813 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff327144640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:47.813 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff327144640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff3200a8a20 con 0x7ff3200a64b0 2024-09-17T18:18:47.813 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff327144640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:47.813 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff327144640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff32009f970 con 0x7ff3200a5b90 2024-09-17T18:18:47.813 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff327144640 10 monclient(hunting): _renew_subs 2024-09-17T18:18:47.813 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff327144640 10 monclient(hunting): authenticate will time out at 725.592460s 2024-09-17T18:18:47.813 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff31ffff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff3200a64b0 0x7ff320138d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:47.813 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff31ffff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff3200a64b0 0x7ff320138d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:58088/0 (socket says 172.21.15.173:58088) 2024-09-17T18:18:47.813 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff31ffff640 1 -- 172.21.15.173:0/1970909842 learned_addr learned my addr 172.21.15.173:0/1970909842 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-09-17T18:18:47.813 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff31ffff640 10 monclient(hunting): get_auth_request con 0x7ff3200a64b0 auth_method 0 2024-09-17T18:18:47.813 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff31ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:47.813 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff31ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:47.813 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff31ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:47.813 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff31ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:47.814 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff31ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:47.814 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff31ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:18:47.814 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff324ee0640 1 --2- 172.21.15.173:0/1970909842 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff3200a5b90 0x7ff3201387c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:47.814 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff3256e1640 1 --2- 172.21.15.173:0/1970909842 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3200a71c0 0x7ff3201263a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:47.814 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff324ee0640 10 monclient(hunting): get_auth_request con 0x7ff3200a5b90 auth_method 0 2024-09-17T18:18:47.814 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff324ee0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:47.814 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff324ee0640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:47.814 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff324ee0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:47.814 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff3256e1640 10 monclient(hunting): get_auth_request con 0x7ff3200a71c0 auth_method 0 2024-09-17T18:18:47.814 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff3256e1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:47.814 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff3256e1640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:47.814 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff3256e1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:47.814 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff3256e1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:47.814 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff3256e1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:47.814 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff3256e1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:18:47.815 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff324ee0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:47.815 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff324ee0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:47.815 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff324ee0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:18:47.815 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff31ffff640 10 monclient(hunting): handle_auth_done global_id 4111 payload 824 2024-09-17T18:18:47.815 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff31ffff640 1 -- 172.21.15.173:0/1970909842 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff3200a5b90 msgr2=0x7ff3201387c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:47.815 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff31ffff640 1 --2- 172.21.15.173:0/1970909842 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff3200a5b90 0x7ff3201387c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:47.815 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff31ffff640 1 -- 172.21.15.173:0/1970909842 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3200a71c0 msgr2=0x7ff3201263a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:47.815 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff31ffff640 1 --2- 172.21.15.173:0/1970909842 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3200a71c0 0x7ff3201263a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:47.815 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff31ffff640 10 monclient: _finish_hunting 0 2024-09-17T18:18:47.815 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff31ffff640 1 monclient: found mon.noname-b 2024-09-17T18:18:47.815 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff31ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:18:47.815 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff31ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.173:3300/0 2024-09-17T18:18:47.815 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.800+0000 7ff31ffff640 1 -- 172.21.15.173:0/1970909842 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3200a2380 con 0x7ff3200a64b0 2024-09-17T18:18:47.815 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.803+0000 7ff31ffff640 1 -- 172.21.15.173:0/1970909842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff3200a64b0 msgr2=0x7ff320138d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:18:47.815 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.803+0000 7ff31ffff640 1 --2- 172.21.15.173:0/1970909842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff3200a64b0 0x7ff320138d00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff30c00c450 tx=0x7ff30c00c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:18:47.815 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.803+0000 7ff31d7fa640 10 client.?.objecter ms_handle_connect 0x7ff3200a64b0 2024-09-17T18:18:47.816 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.803+0000 7ff31d7fa640 10 client.?.objecter resend_mon_ops 2024-09-17T18:18:47.816 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.803+0000 7ff31d7fa640 1 -- 172.21.15.173:0/1970909842 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff30c017070 con 0x7ff3200a64b0 2024-09-17T18:18:47.816 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.803+0000 7ff31d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:47.816 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.803+0000 7ff31d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:18:47.816 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.803+0000 7ff31d7fa640 10 monclient: dump: 2024-09-17T18:18:47.816 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-09-17T18:18:47.816 INFO:teuthology.orchestra.run.smithi173.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:47.816 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:47.816 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:47.816 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:47.816 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-09-17T18:18:47.816 INFO:teuthology.orchestra.run.smithi173.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:47.816 INFO:teuthology.orchestra.run.smithi173.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:47.816 INFO:teuthology.orchestra.run.smithi173.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:47.816 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T18:18:47.817 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.803+0000 7ff31d7fa640 10 monclient: _finish_auth 0 2024-09-17T18:18:47.817 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.803+0000 7ff31d7fa640 10 monclient: _check_auth_tickets 2024-09-17T18:18:47.817 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.803+0000 7ff31d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:18:47.817 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.803+0000 7ff327144640 5 monclient: authenticate success, global_id 4111 2024-09-17T18:18:47.817 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.803+0000 7ff31d7fa640 1 -- 172.21.15.173:0/1970909842 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff30c005810 con 0x7ff3200a64b0 2024-09-17T18:18:47.817 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.803+0000 7ff31d7fa640 10 monclient: handle_config config(0 keys) 2024-09-17T18:18:47.817 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.803+0000 7ff31d7fa640 1 -- 172.21.15.173:0/1970909842 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff30c007690 con 0x7ff3200a64b0 2024-09-17T18:18:47.817 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.803+0000 7ff327144640 10 monclient: _renew_subs 2024-09-17T18:18:47.817 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.803+0000 7ff327144640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:18:47.817 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.803+0000 7ff327144640 1 -- 172.21.15.173:0/1970909842 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff320139240 con 0x7ff3200a64b0 2024-09-17T18:18:47.817 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.803+0000 7ff31effd640 4 set_mon_vals no callback set 2024-09-17T18:18:47.817 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.803+0000 7ff31d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:47.817 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.803+0000 7ff31d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:18:47.817 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.803+0000 7ff31d7fa640 10 monclient: dump: 2024-09-17T18:18:47.818 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-09-17T18:18:47.818 INFO:teuthology.orchestra.run.smithi173.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:47.818 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:47.818 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:47.818 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:47.818 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-09-17T18:18:47.818 INFO:teuthology.orchestra.run.smithi173.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:47.818 INFO:teuthology.orchestra.run.smithi173.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:47.818 INFO:teuthology.orchestra.run.smithi173.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:47.818 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T18:18:47.818 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.803+0000 7ff327144640 10 client.4111.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:18:47.818 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.803+0000 7ff327144640 10 monclient: _renew_subs 2024-09-17T18:18:47.818 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.803+0000 7ff327144640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:18:47.818 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.803+0000 7ff327144640 1 -- 172.21.15.173:0/1970909842 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff32012a1c0 con 0x7ff3200a64b0 2024-09-17T18:18:47.819 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.803+0000 7ff327144640 1 librados: init done 2024-09-17T18:18:47.819 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.803+0000 7ff31d7fa640 1 -- 172.21.15.173:0/1970909842 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff30c01a070 con 0x7ff3200a64b0 2024-09-17T18:18:47.819 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.803+0000 7ff31d7fa640 1 -- 172.21.15.173:0/1970909842 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7ff30c012740 con 0x7ff3200a64b0 2024-09-17T18:18:47.819 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.803+0000 7ff31d7fa640 10 client.4111.objecter ms_dispatch 0x7ff320000b90 osd_map(2..2 src has 1..2) 2024-09-17T18:18:47.819 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.803+0000 7ff31d7fa640 3 client.4111.objecter handle_osd_map got epochs [2,2] > 0 2024-09-17T18:18:47.819 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.803+0000 7ff31d7fa640 3 client.4111.objecter handle_osd_map decoding full epoch 2 2024-09-17T18:18:47.819 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.803+0000 7ff31d7fa640 20 client.4111.objecter dump_active .. 0 homeless 2024-09-17T18:18:47.819 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.804+0000 7ff327144640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:18:47.819 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.804+0000 7ff327144640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:18:47.819 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.804+0000 7ff327144640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:18:47.819 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.804+0000 7ff327144640 1 -- 172.21.15.173:0/1970909842 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3200a5f70 con 0x7ff3200a64b0 2024-09-17T18:18:47.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.806+0000 7ff31d7fa640 1 -- 172.21.15.173:0/1970909842 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7ff30c021020 con 0x7ff3200a64b0 2024-09-17T18:18:47.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.806+0000 7ff31d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:18:47.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.806+0000 7ff31d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:18:47.929 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.927+0000 7ff327144640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "7bf586d1-05ac-4c7e-8b88-313f07a76801", "id": 0}] 2024-09-17T18:18:47.929 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.927+0000 7ff327144640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "7bf586d1-05ac-4c7e-8b88-313f07a76801", "id": 0}] 2024-09-17T18:18:47.929 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.927+0000 7ff327144640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:18:47.929 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.927+0000 7ff327144640 1 -- 172.21.15.173:0/1970909842 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd new", "uuid": "7bf586d1-05ac-4c7e-8b88-313f07a76801", "id": 0} v 0) -- 0x7ff3200a0040 con 0x7ff3200a64b0 2024-09-17T18:18:47.935 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.933+0000 7ff31d7fa640 1 -- 172.21.15.173:0/1970909842 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7bf586d1-05ac-4c7e-8b88-313f07a76801", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7ff30c012b90 con 0x7ff3200a64b0 2024-09-17T18:18:47.935 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.933+0000 7ff31d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "7bf586d1-05ac-4c7e-8b88-313f07a76801", "id": 0}] 2024-09-17T18:18:47.935 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.933+0000 7ff31d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:18:47.935 INFO:teuthology.orchestra.run.smithi173.stdout:0 2024-09-17T18:18:47.937 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.934+0000 7ff327144640 10 librados: watch_flush enter 2024-09-17T18:18:47.937 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.934+0000 7ff327144640 10 librados: watch_flush exit 2024-09-17T18:18:47.937 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.934+0000 7ff327144640 20 client.4111.objecter shutdown clearing up homeless session... 2024-09-17T18:18:47.937 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.934+0000 7ff327144640 10 client.4111.objecter successfully canceled tick 2024-09-17T18:18:47.937 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.934+0000 7ff327144640 10 monclient: shutdown 2024-09-17T18:18:47.937 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.934+0000 7ff327144640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:18:47.937 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.934+0000 7ff327144640 1 -- 172.21.15.173:0/1970909842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff3200a64b0 msgr2=0x7ff320138d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:47.937 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.934+0000 7ff327144640 1 --2- 172.21.15.173:0/1970909842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff3200a64b0 0x7ff320138d00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff30c00c450 tx=0x7ff30c00c910 comp rx=0 tx=0).stop 2024-09-17T18:18:47.937 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.934+0000 7ff327144640 1 -- 172.21.15.173:0/1970909842 shutdown_connections 2024-09-17T18:18:47.938 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.934+0000 7ff327144640 1 --2- 172.21.15.173:0/1970909842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff3200a64b0 0x7ff320138d00 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:47.938 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.934+0000 7ff327144640 1 --2- 172.21.15.173:0/1970909842 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3200a71c0 0x7ff3201263a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:47.938 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.934+0000 7ff327144640 1 --2- 172.21.15.173:0/1970909842 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff3200a5b90 0x7ff3201387c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:47.938 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.934+0000 7ff327144640 1 -- 172.21.15.173:0/1970909842 >> 172.21.15.173:0/1970909842 conn(0x7ff3200a7c90 msgr2=0x7ff3200a9f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:18:47.938 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.934+0000 7ff327144640 1 -- 172.21.15.173:0/1970909842 shutdown_connections 2024-09-17T18:18:47.938 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.934+0000 7ff327144640 1 -- 172.21.15.173:0/1970909842 wait complete. 2024-09-17T18:18:47.938 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:47.934+0000 7ff327144640 1 librados: shutdown 2024-09-17T18:18:47.947 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster ceph osd new e26c6163-1eda-4200-82e1-57b752fd8a46 1 2024-09-17T18:18:48.045 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.042+0000 7f678badd640 10 monclient: get_monmap_and_config 2024-09-17T18:18:48.045 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.042+0000 7f678badd640 10 monclient: build_initial_monmap 2024-09-17T18:18:48.045 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.042+0000 7f678badd640 10 monclient: monmap: 2024-09-17T18:18:48.045 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-09-17T18:18:48.045 INFO:teuthology.orchestra.run.smithi173.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:48.045 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-09-17T18:18:48.043694+0000 2024-09-17T18:18:48.045 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-09-17T18:18:48.043694+0000 2024-09-17T18:18:48.045 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-09-17T18:18:48.045 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-09-17T18:18:48.045 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:18:48.046 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:18:48.046 INFO:teuthology.orchestra.run.smithi173.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:18:48.046 INFO:teuthology.orchestra.run.smithi173.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:18:48.046 INFO:teuthology.orchestra.run.smithi173.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:18:48.046 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T18:18:48.046 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.043+0000 7f678badd640 1 Processor -- start 2024-09-17T18:18:48.048 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.043+0000 7f678badd640 1 -- start start 2024-09-17T18:18:48.048 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.043+0000 7f678badd640 10 monclient: init 2024-09-17T18:18:48.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.043+0000 7f678badd640 10 monclient: _reopen_session rank -1 2024-09-17T18:18:48.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.044+0000 7f678badd640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-17T18:18:48.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.044+0000 7f678badd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f67840a5b90 addr v1:172.21.15.173:6789/0 2024-09-17T18:18:48.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.044+0000 7f678badd640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f67840a71c0 0x7f678413dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:48.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.044+0000 7f678badd640 10 monclient(hunting): picked mon.noname-a con 0x7f67840a71c0 addr v2:172.21.15.121:3300/0 2024-09-17T18:18:48.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.044+0000 7f678badd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f67840a6680 0x7f67841401c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:48.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.044+0000 7f678badd640 10 monclient(hunting): picked mon.noname-b con 0x7f67840a6680 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:18:48.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.044+0000 7f678badd640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f678409cb90 con 0x7f67840a5b90 2024-09-17T18:18:48.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.044+0000 7f678badd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:48.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.044+0000 7f678badd640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f678409d4c0 con 0x7f67840a71c0 2024-09-17T18:18:48.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.044+0000 7f678badd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:48.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.044+0000 7f678badd640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f67840a6c50 con 0x7f67840a6680 2024-09-17T18:18:48.050 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.044+0000 7f678badd640 10 monclient(hunting): _renew_subs 2024-09-17T18:18:48.050 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.044+0000 7f678badd640 10 monclient(hunting): authenticate will time out at 725.836111s 2024-09-17T18:18:48.050 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.044+0000 7f6789879640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f67840a5b90 0x7f67840a5f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:33804/0 (socket says 172.21.15.173:33804) 2024-09-17T18:18:48.050 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.044+0000 7f6789879640 1 -- 172.21.15.173:0/3218179172 learned_addr learned my addr 172.21.15.173:0/3218179172 (peer_addr_for_me v1:172.21.15.173:0/0) 2024-09-17T18:18:48.050 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.044+0000 7f678a07a640 1 --2- 172.21.15.173:0/3218179172 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f67840a6680 0x7f67841401c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:48.050 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.044+0000 7f6789078640 1 --2- 172.21.15.173:0/3218179172 >> v2:172.21.15.121:3300/0 conn(0x7f67840a71c0 0x7f678413dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:48.050 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.044+0000 7f678a07a640 10 monclient(hunting): get_auth_request con 0x7f67840a6680 auth_method 0 2024-09-17T18:18:48.050 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.044+0000 7f678a07a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:48.050 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.044+0000 7f678a07a640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:48.050 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.044+0000 7f678a07a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:48.050 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.044+0000 7f6789078640 10 monclient(hunting): get_auth_request con 0x7f67840a71c0 auth_method 0 2024-09-17T18:18:48.050 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.044+0000 7f6789078640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:48.050 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.044+0000 7f6789078640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:48.051 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.044+0000 7f6789078640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:48.051 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.044+0000 7f678a07a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:48.051 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.044+0000 7f678a07a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:48.051 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.044+0000 7f678a07a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:18:48.051 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.044+0000 7f6789078640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:48.051 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.044+0000 7f6789078640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:48.051 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.044+0000 7f6789078640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:18:48.051 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.044+0000 7f678a07a640 10 monclient(hunting): handle_auth_done global_id 4109 payload 274 2024-09-17T18:18:48.051 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.044+0000 7f678a07a640 1 -- 172.21.15.173:0/3218179172 >> v2:172.21.15.121:3300/0 conn(0x7f67840a71c0 msgr2=0x7f678413dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:18:48.051 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.044+0000 7f678a07a640 1 --2- 172.21.15.173:0/3218179172 >> v2:172.21.15.121:3300/0 conn(0x7f67840a71c0 0x7f678413dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:48.051 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.044+0000 7f678a07a640 1 -- 172.21.15.173:0/3218179172 >> v1:172.21.15.173:6789/0 conn(0x7f67840a5b90 legacy=0x7f67840a5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:18:48.051 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.044+0000 7f678a07a640 10 monclient: _finish_hunting 0 2024-09-17T18:18:48.052 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.044+0000 7f678a07a640 1 monclient: found mon.noname-b 2024-09-17T18:18:48.052 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.044+0000 7f678a07a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:18:48.052 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.044+0000 7f678a07a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:18:48.052 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.044+0000 7f678a07a640 1 -- 172.21.15.173:0/3218179172 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6784070550 con 0x7f67840a6680 2024-09-17T18:18:48.058 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.047+0000 7f678a07a640 1 -- 172.21.15.173:0/3218179172 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f67840a6680 msgr2=0x7f67841401c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:18:48.058 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.047+0000 7f678a07a640 1 --2- 172.21.15.173:0/3218179172 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f67840a6680 0x7f67841401c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f677c009d90 tx=0x7f677c02a490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5255e9b1c6b42 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:18:48.058 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.047+0000 7f6788877640 1 -- 172.21.15.173:0/3218179172 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f677c007950 con 0x7f67840a6680 2024-09-17T18:18:48.058 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.047+0000 7f6788877640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:48.058 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.047+0000 7f6788877640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:18:48.058 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.047+0000 7f6788877640 10 monclient: dump: 2024-09-17T18:18:48.058 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-09-17T18:18:48.058 INFO:teuthology.orchestra.run.smithi173.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:48.058 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:48.058 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:48.058 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:48.058 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-09-17T18:18:48.059 INFO:teuthology.orchestra.run.smithi173.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:48.059 INFO:teuthology.orchestra.run.smithi173.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:48.059 INFO:teuthology.orchestra.run.smithi173.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:48.059 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T18:18:48.059 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.047+0000 7f6788877640 10 monclient: _finish_auth 0 2024-09-17T18:18:48.059 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.047+0000 7f6788877640 10 monclient: _check_auth_tickets 2024-09-17T18:18:48.059 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.047+0000 7f6788877640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:18:48.059 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.047+0000 7f6788877640 1 -- 172.21.15.173:0/3218179172 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f677c007af0 con 0x7f67840a6680 2024-09-17T18:18:48.059 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.047+0000 7f678badd640 5 monclient: authenticate success, global_id 4109 2024-09-17T18:18:48.059 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.047+0000 7f678badd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:18:48.059 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.047+0000 7f6788877640 10 monclient: handle_config config(0 keys) 2024-09-17T18:18:48.059 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.047+0000 7f678badd640 10 monclient: get_monmap_and_config success 2024-09-17T18:18:48.059 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.047+0000 7f678badd640 4 set_mon_vals no callback set 2024-09-17T18:18:48.060 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.047+0000 7f6788877640 1 -- 172.21.15.173:0/3218179172 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f677c031400 con 0x7f67840a6680 2024-09-17T18:18:48.060 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.047+0000 7f6788877640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:48.060 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.047+0000 7f6788877640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:18:48.060 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.047+0000 7f6788877640 10 monclient: dump: 2024-09-17T18:18:48.060 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-09-17T18:18:48.060 INFO:teuthology.orchestra.run.smithi173.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:48.060 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:48.060 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:48.060 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:48.060 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-09-17T18:18:48.060 INFO:teuthology.orchestra.run.smithi173.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:48.060 INFO:teuthology.orchestra.run.smithi173.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:48.060 INFO:teuthology.orchestra.run.smithi173.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:48.060 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T18:18:48.061 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.048+0000 7f678badd640 10 monclient: shutdown 2024-09-17T18:18:48.061 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.048+0000 7f678badd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:18:48.061 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.048+0000 7f678badd640 1 -- 172.21.15.173:0/3218179172 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f67840a6680 msgr2=0x7f67841401c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:48.061 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.048+0000 7f678badd640 1 --2- 172.21.15.173:0/3218179172 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f67840a6680 0x7f67841401c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f677c009d90 tx=0x7f677c02a490 comp rx=0 tx=0).stop 2024-09-17T18:18:48.061 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.048+0000 7f678badd640 1 -- 172.21.15.173:0/3218179172 shutdown_connections 2024-09-17T18:18:48.061 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.048+0000 7f678badd640 1 --2- 172.21.15.173:0/3218179172 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f67840a6680 0x7f67841401c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:48.061 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.048+0000 7f678badd640 1 --2- 172.21.15.173:0/3218179172 >> v2:172.21.15.121:3300/0 conn(0x7f67840a71c0 0x7f678413dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:48.061 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.048+0000 7f678badd640 1 -- 172.21.15.173:0/3218179172 >> 172.21.15.173:0/3218179172 conn(0x7f67840a7c90 msgr2=0x7f67840a80a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:18:48.061 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.048+0000 7f678badd640 1 -- 172.21.15.173:0/3218179172 shutdown_connections 2024-09-17T18:18:48.061 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.048+0000 7f678badd640 1 -- 172.21.15.173:0/3218179172 wait complete. 2024-09-17T18:18:48.061 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.049+0000 7f678badd640 10 monclient: build_initial_monmap 2024-09-17T18:18:48.061 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.049+0000 7f678badd640 10 monclient: monmap: 2024-09-17T18:18:48.061 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-09-17T18:18:48.062 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:18:48.062 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 0.000000 2024-09-17T18:18:48.062 INFO:teuthology.orchestra.run.smithi173.stderr:created 0.000000 2024-09-17T18:18:48.062 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-09-17T18:18:48.062 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-09-17T18:18:48.062 INFO:teuthology.orchestra.run.smithi173.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:18:48.062 INFO:teuthology.orchestra.run.smithi173.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:18:48.062 INFO:teuthology.orchestra.run.smithi173.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:18:48.062 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T18:18:48.062 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.049+0000 7f678badd640 1 librados: starting msgr at 2024-09-17T18:18:48.062 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.049+0000 7f678badd640 1 librados: starting objecter 2024-09-17T18:18:48.062 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.049+0000 7f678badd640 1 Processor -- start 2024-09-17T18:18:48.062 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.049+0000 7f678badd640 1 -- start start 2024-09-17T18:18:48.062 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.049+0000 7f678badd640 1 librados: setting wanted keys 2024-09-17T18:18:48.063 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.049+0000 7f678badd640 1 librados: calling monclient init 2024-09-17T18:18:48.063 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.049+0000 7f678badd640 10 monclient: init 2024-09-17T18:18:48.063 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.049+0000 7f678badd640 10 monclient: _reopen_session rank -1 2024-09-17T18:18:48.063 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.049+0000 7f678badd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T18:18:48.063 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.049+0000 7f678badd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f67840a5b90 0x7f6784071b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:48.063 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.049+0000 7f678badd640 10 monclient(hunting): picked mon.noname-c con 0x7f67840a5b90 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:18:48.063 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.049+0000 7f678badd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67840a6680 0x7f6784074610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:48.063 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.049+0000 7f678badd640 10 monclient(hunting): picked mon.noname-b con 0x7f67840a6680 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:18:48.063 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.049+0000 7f678badd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67840a71c0 0x7f678413dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:48.063 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.049+0000 7f678badd640 10 monclient(hunting): picked mon.noname-a con 0x7f67840a71c0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:18:48.063 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.049+0000 7f678badd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:48.063 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.049+0000 7f678badd640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f67840a8ba0 con 0x7f67840a71c0 2024-09-17T18:18:48.063 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.049+0000 7f678badd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:48.063 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.049+0000 7f678badd640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f67840a8a20 con 0x7f67840a6680 2024-09-17T18:18:48.064 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.049+0000 7f678badd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:48.064 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.049+0000 7f678badd640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6784140700 con 0x7f67840a5b90 2024-09-17T18:18:48.064 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.049+0000 7f678badd640 10 monclient(hunting): _renew_subs 2024-09-17T18:18:48.064 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.049+0000 7f678badd640 10 monclient(hunting): authenticate will time out at 725.841956s 2024-09-17T18:18:48.064 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.050+0000 7f6789078640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67840a6680 0x7f6784074610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:48.064 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.050+0000 7f6789078640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67840a6680 0x7f6784074610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:58104/0 (socket says 172.21.15.173:58104) 2024-09-17T18:18:48.064 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.050+0000 7f6789078640 1 -- 172.21.15.173:0/1351185137 learned_addr learned my addr 172.21.15.173:0/1351185137 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-09-17T18:18:48.064 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.050+0000 7f6789078640 10 monclient(hunting): get_auth_request con 0x7f67840a6680 auth_method 0 2024-09-17T18:18:48.064 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.050+0000 7f6789078640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:48.064 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.050+0000 7f6789078640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:48.064 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.050+0000 7f6789078640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:48.064 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.050+0000 7f6789879640 1 --2- 172.21.15.173:0/1351185137 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f67840a5b90 0x7f6784071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:48.064 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.050+0000 7f678a07a640 1 --2- 172.21.15.173:0/1351185137 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67840a71c0 0x7f678413dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:48.065 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.050+0000 7f6789879640 10 monclient(hunting): get_auth_request con 0x7f67840a5b90 auth_method 0 2024-09-17T18:18:48.065 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.050+0000 7f6789879640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:48.065 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.050+0000 7f6789879640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:48.065 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.050+0000 7f6789879640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:48.065 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.050+0000 7f678a07a640 10 monclient(hunting): get_auth_request con 0x7f67840a71c0 auth_method 0 2024-09-17T18:18:48.065 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.050+0000 7f678a07a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:48.065 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.050+0000 7f678a07a640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:48.065 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.050+0000 7f678a07a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:48.065 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.050+0000 7f6789078640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:48.065 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.050+0000 7f6789078640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:48.065 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.050+0000 7f6789078640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:18:48.065 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.050+0000 7f6789879640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:48.065 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.050+0000 7f6789879640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:48.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.050+0000 7f6789879640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:18:48.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.050+0000 7f678a07a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:48.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.050+0000 7f678a07a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:48.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.050+0000 7f678a07a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:18:48.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.050+0000 7f6789078640 10 monclient(hunting): handle_auth_done global_id 4114 payload 824 2024-09-17T18:18:48.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.050+0000 7f6789078640 1 -- 172.21.15.173:0/1351185137 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f67840a5b90 msgr2=0x7f6784071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:48.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.050+0000 7f6789078640 1 --2- 172.21.15.173:0/1351185137 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f67840a5b90 0x7f6784071b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:48.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.050+0000 7f6789078640 1 -- 172.21.15.173:0/1351185137 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67840a71c0 msgr2=0x7f678413dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:48.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.050+0000 7f6789078640 1 --2- 172.21.15.173:0/1351185137 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67840a71c0 0x7f678413dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:48.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.050+0000 7f6789078640 10 monclient: _finish_hunting 0 2024-09-17T18:18:48.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.050+0000 7f6789078640 1 monclient: found mon.noname-b 2024-09-17T18:18:48.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.050+0000 7f6789078640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:18:48.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.050+0000 7f6789078640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.173:3300/0 2024-09-17T18:18:48.067 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.050+0000 7f6789078640 1 -- 172.21.15.173:0/1351185137 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67840a2380 con 0x7f67840a6680 2024-09-17T18:18:48.067 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.050+0000 7f6789879640 1 --2- 172.21.15.173:0/1351185137 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f67840a5b90 0x7f6784071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:18:48.067 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.050+0000 7f678a07a640 1 --2- 172.21.15.173:0/1351185137 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67840a71c0 0x7f678413dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:18:48.067 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.052+0000 7f6789078640 1 -- 172.21.15.173:0/1351185137 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67840a6680 msgr2=0x7f6784074610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:18:48.067 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.052+0000 7f6789078640 1 --2- 172.21.15.173:0/1351185137 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67840a6680 0x7f6784074610 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f677400e9b0 tx=0x7f677400ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:18:48.067 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.052+0000 7f677a7fc640 10 client.?.objecter ms_handle_connect 0x7f67840a6680 2024-09-17T18:18:48.067 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.052+0000 7f677a7fc640 10 client.?.objecter resend_mon_ops 2024-09-17T18:18:48.067 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.052+0000 7f677a7fc640 1 -- 172.21.15.173:0/1351185137 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6774017070 con 0x7f67840a6680 2024-09-17T18:18:48.067 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.052+0000 7f677a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:48.067 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.052+0000 7f677a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:18:48.067 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.052+0000 7f677a7fc640 10 monclient: dump: 2024-09-17T18:18:48.067 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-09-17T18:18:48.067 INFO:teuthology.orchestra.run.smithi173.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:48.067 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:48.068 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:48.068 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:48.068 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-09-17T18:18:48.068 INFO:teuthology.orchestra.run.smithi173.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:48.068 INFO:teuthology.orchestra.run.smithi173.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:48.068 INFO:teuthology.orchestra.run.smithi173.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:48.068 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T18:18:48.068 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.052+0000 7f677a7fc640 10 monclient: _finish_auth 0 2024-09-17T18:18:48.068 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.052+0000 7f677a7fc640 10 monclient: _check_auth_tickets 2024-09-17T18:18:48.068 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.052+0000 7f677a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:18:48.068 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.052+0000 7f678badd640 5 monclient: authenticate success, global_id 4114 2024-09-17T18:18:48.068 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.052+0000 7f678badd640 10 monclient: _renew_subs 2024-09-17T18:18:48.068 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.052+0000 7f678badd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:18:48.069 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.052+0000 7f678badd640 1 -- 172.21.15.173:0/1351185137 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f678413f810 con 0x7f67840a6680 2024-09-17T18:18:48.069 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.052+0000 7f677a7fc640 1 -- 172.21.15.173:0/1351185137 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f677400cbb0 con 0x7f67840a6680 2024-09-17T18:18:48.069 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.052+0000 7f677a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:18:48.069 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.052+0000 7f677a7fc640 1 -- 172.21.15.173:0/1351185137 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6774013470 con 0x7f67840a6680 2024-09-17T18:18:48.069 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.052+0000 7f677a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:48.069 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.052+0000 7f677a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:18:48.069 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.052+0000 7f677a7fc640 10 monclient: dump: 2024-09-17T18:18:48.069 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-09-17T18:18:48.069 INFO:teuthology.orchestra.run.smithi173.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:48.069 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:48.069 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:48.069 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:48.069 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-09-17T18:18:48.069 INFO:teuthology.orchestra.run.smithi173.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:48.070 INFO:teuthology.orchestra.run.smithi173.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:48.070 INFO:teuthology.orchestra.run.smithi173.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:48.070 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T18:18:48.070 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.052+0000 7f677bfff640 4 set_mon_vals no callback set 2024-09-17T18:18:48.070 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.052+0000 7f678badd640 10 client.4114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:18:48.070 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.052+0000 7f678badd640 10 monclient: _renew_subs 2024-09-17T18:18:48.070 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.052+0000 7f678badd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:18:48.070 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.052+0000 7f678badd640 1 -- 172.21.15.173:0/1351185137 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6784072510 con 0x7f67840a6680 2024-09-17T18:18:48.070 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.052+0000 7f678badd640 1 librados: init done 2024-09-17T18:18:48.070 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.053+0000 7f677a7fc640 1 -- 172.21.15.173:0/1351185137 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f677401a070 con 0x7f67840a6680 2024-09-17T18:18:48.070 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.053+0000 7f677a7fc640 1 -- 172.21.15.173:0/1351185137 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f67740063d0 con 0x7f67840a6680 2024-09-17T18:18:48.070 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.053+0000 7f677a7fc640 10 client.4114.objecter ms_dispatch 0x7f6784000b90 osd_map(3..3 src has 1..3) 2024-09-17T18:18:48.070 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.053+0000 7f677a7fc640 3 client.4114.objecter handle_osd_map got epochs [3,3] > 0 2024-09-17T18:18:48.070 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.053+0000 7f677a7fc640 3 client.4114.objecter handle_osd_map decoding full epoch 3 2024-09-17T18:18:48.071 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.053+0000 7f677a7fc640 20 client.4114.objecter dump_active .. 0 homeless 2024-09-17T18:18:48.071 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.053+0000 7f678badd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:18:48.071 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.053+0000 7f678badd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:18:48.071 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.053+0000 7f678badd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:18:48.071 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.053+0000 7f678badd640 1 -- 172.21.15.173:0/1351185137 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67840a63b0 con 0x7f67840a6680 2024-09-17T18:18:48.071 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.055+0000 7f677a7fc640 1 -- 172.21.15.173:0/1351185137 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f67840a63b0 con 0x7f67840a6680 2024-09-17T18:18:48.071 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.055+0000 7f677a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:18:48.071 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.055+0000 7f677a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:18:48.127 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.124+0000 7f678badd640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e26c6163-1eda-4200-82e1-57b752fd8a46", "id": 1}] 2024-09-17T18:18:48.127 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.124+0000 7f678badd640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e26c6163-1eda-4200-82e1-57b752fd8a46", "id": 1}] 2024-09-17T18:18:48.127 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.124+0000 7f678badd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:18:48.127 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.124+0000 7f678badd640 1 -- 172.21.15.173:0/1351185137 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e26c6163-1eda-4200-82e1-57b752fd8a46", "id": 1} v 0) -- 0x7f67840a5f70 con 0x7f67840a6680 2024-09-17T18:18:48.143 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.141+0000 7f677a7fc640 1 -- 172.21.15.173:0/1351185137 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e26c6163-1eda-4200-82e1-57b752fd8a46", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f6774013e20 con 0x7f67840a6680 2024-09-17T18:18:48.143 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.141+0000 7f677a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e26c6163-1eda-4200-82e1-57b752fd8a46", "id": 1}] 2024-09-17T18:18:48.143 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.141+0000 7f677a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:18:48.143 INFO:teuthology.orchestra.run.smithi173.stdout:1 2024-09-17T18:18:48.145 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.142+0000 7f678badd640 10 librados: watch_flush enter 2024-09-17T18:18:48.145 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.142+0000 7f678badd640 10 librados: watch_flush exit 2024-09-17T18:18:48.145 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.142+0000 7f678badd640 20 client.4114.objecter shutdown clearing up homeless session... 2024-09-17T18:18:48.145 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.142+0000 7f678badd640 10 client.4114.objecter successfully canceled tick 2024-09-17T18:18:48.145 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.142+0000 7f678badd640 10 monclient: shutdown 2024-09-17T18:18:48.145 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.142+0000 7f678badd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:18:48.145 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.142+0000 7f678badd640 1 -- 172.21.15.173:0/1351185137 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67840a6680 msgr2=0x7f6784074610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:48.146 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.142+0000 7f678badd640 1 --2- 172.21.15.173:0/1351185137 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67840a6680 0x7f6784074610 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f677400e9b0 tx=0x7f677400ee70 comp rx=0 tx=0).stop 2024-09-17T18:18:48.146 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.142+0000 7f678badd640 1 -- 172.21.15.173:0/1351185137 shutdown_connections 2024-09-17T18:18:48.146 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.142+0000 7f678badd640 1 --2- 172.21.15.173:0/1351185137 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67840a6680 0x7f6784074610 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:48.146 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.142+0000 7f678badd640 1 --2- 172.21.15.173:0/1351185137 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67840a71c0 0x7f678413dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:48.146 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.142+0000 7f678badd640 1 --2- 172.21.15.173:0/1351185137 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f67840a5b90 0x7f6784071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:48.146 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.142+0000 7f678badd640 1 -- 172.21.15.173:0/1351185137 >> 172.21.15.173:0/1351185137 conn(0x7f67840a7c90 msgr2=0x7f67840a1390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:18:48.146 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.142+0000 7f678badd640 1 -- 172.21.15.173:0/1351185137 shutdown_connections 2024-09-17T18:18:48.146 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.142+0000 7f678badd640 1 -- 172.21.15.173:0/1351185137 wait complete. 2024-09-17T18:18:48.146 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.142+0000 7f678badd640 1 librados: shutdown 2024-09-17T18:18:48.154 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster ceph osd new 6fb45798-21d4-449e-9f68-7210d984e036 2 2024-09-17T18:18:48.251 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.249+0000 7f1c119e2640 10 monclient: get_monmap_and_config 2024-09-17T18:18:48.251 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.249+0000 7f1c119e2640 10 monclient: build_initial_monmap 2024-09-17T18:18:48.251 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.249+0000 7f1c119e2640 10 monclient: monmap: 2024-09-17T18:18:48.251 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-09-17T18:18:48.251 INFO:teuthology.orchestra.run.smithi173.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:48.251 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-09-17T18:18:48.250138+0000 2024-09-17T18:18:48.251 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-09-17T18:18:48.250138+0000 2024-09-17T18:18:48.251 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-09-17T18:18:48.252 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-09-17T18:18:48.252 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:18:48.252 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:18:48.252 INFO:teuthology.orchestra.run.smithi173.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:18:48.252 INFO:teuthology.orchestra.run.smithi173.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:18:48.252 INFO:teuthology.orchestra.run.smithi173.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:18:48.252 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T18:18:48.257 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.249+0000 7f1c119e2640 1 Processor -- start 2024-09-17T18:18:48.257 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.249+0000 7f1c119e2640 1 -- start start 2024-09-17T18:18:48.257 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.249+0000 7f1c119e2640 10 monclient: init 2024-09-17T18:18:48.257 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.250+0000 7f1c119e2640 10 monclient: _reopen_session rank -1 2024-09-17T18:18:48.257 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.250+0000 7f1c119e2640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-17T18:18:48.257 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.250+0000 7f1c119e2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1c0c0a5b90 addr v1:172.21.15.121:6789/0 2024-09-17T18:18:48.257 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.250+0000 7f1c119e2640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1c0c0a71c0 0x7f1c0c13dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:48.257 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.250+0000 7f1c119e2640 10 monclient(hunting): picked mon.noname-a con 0x7f1c0c0a71c0 addr v2:172.21.15.121:3300/0 2024-09-17T18:18:48.258 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.250+0000 7f1c119e2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1c0c0a6680 addr v1:172.21.15.173:6789/0 2024-09-17T18:18:48.258 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.250+0000 7f1c119e2640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c0c09cb90 con 0x7f1c0c0a5b90 2024-09-17T18:18:48.258 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.250+0000 7f1c119e2640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c0c070550 con 0x7f1c0c0a6680 2024-09-17T18:18:48.258 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.250+0000 7f1c119e2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:48.258 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.250+0000 7f1c119e2640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f1c0c09d4c0 con 0x7f1c0c0a71c0 2024-09-17T18:18:48.258 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.250+0000 7f1c119e2640 10 monclient(hunting): _renew_subs 2024-09-17T18:18:48.258 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.250+0000 7f1c119e2640 10 monclient(hunting): authenticate will time out at 726.042398s 2024-09-17T18:18:48.258 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.250+0000 7f1c0b7fe640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f1c0c0a6680 0x7f1c0c1401c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:33816/0 (socket says 172.21.15.173:33816) 2024-09-17T18:18:48.258 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.250+0000 7f1c0b7fe640 1 -- 172.21.15.173:0/2876097369 learned_addr learned my addr 172.21.15.173:0/2876097369 (peer_addr_for_me v1:172.21.15.173:0/0) 2024-09-17T18:18:48.258 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.250+0000 7f1c0a7fc640 1 --2- 172.21.15.173:0/2876097369 >> v2:172.21.15.121:3300/0 conn(0x7f1c0c0a71c0 0x7f1c0c13dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:48.259 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.250+0000 7f1c0a7fc640 10 monclient(hunting): get_auth_request con 0x7f1c0c0a71c0 auth_method 0 2024-09-17T18:18:48.259 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.250+0000 7f1c0a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:48.259 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.250+0000 7f1c0a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:48.259 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.250+0000 7f1c0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:48.259 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.252+0000 7f1c0b7fe640 1 -- 172.21.15.173:0/2876097369 >> v1:172.21.15.173:6789/0 conn(0x7f1c0c0a6680 legacy=0x7f1c0c1401c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:18:48.259 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.252+0000 7f1c0affd640 1 -- 172.21.15.173:0/2876097369 >> v1:172.21.15.121:6789/0 conn(0x7f1c0c0a5b90 legacy=0x7f1c0c0a5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:18:48.259 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.252+0000 7f1c09ffb640 1 -- 172.21.15.173:0/2876097369 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2170860497 0 0) 0x7f1c0c070550 con 0x7f1c0c0a6680 2024-09-17T18:18:48.259 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.252+0000 7f1c09ffb640 10 monclient(hunting): _negotiate 2024-09-17T18:18:48.259 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.252+0000 7f1c09ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:48.259 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.252+0000 7f1c09ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:48.259 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.252+0000 7f1c09ffb640 10 monclient(hunting): my global_id is 4117 2024-09-17T18:18:48.259 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.252+0000 7f1c09ffb640 1 -- 172.21.15.173:0/2876097369 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1bf8003520 con 0x7f1c0c0a6680 2024-09-17T18:18:48.260 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.252+0000 7f1c0a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:48.260 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.252+0000 7f1c0a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:48.260 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.252+0000 7f1c0a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:18:48.260 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c09ffb640 1 -- 172.21.15.173:0/2876097369 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1195290306 0 0) 0x7f1bf8003520 con 0x7f1c0c0a6680 2024-09-17T18:18:48.260 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c09ffb640 1 -- 172.21.15.173:0/2876097369 >> v2:172.21.15.121:3300/0 conn(0x7f1c0c0a71c0 msgr2=0x7f1c0c13dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:18:48.260 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c09ffb640 1 --2- 172.21.15.173:0/2876097369 >> v2:172.21.15.121:3300/0 conn(0x7f1c0c0a71c0 0x7f1c0c13dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:48.260 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c09ffb640 1 -- 172.21.15.173:0/2876097369 >> v1:172.21.15.121:6789/0 conn(0x7f1c0c0a5b90 legacy=0x7f1c0c0a5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:48.260 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c09ffb640 10 monclient: _finish_hunting 0 2024-09-17T18:18:48.260 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c09ffb640 1 monclient: found mon.noname-c-legacy 2024-09-17T18:18:48.260 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c09ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:18:48.260 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c09ffb640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.173:6789/0 2024-09-17T18:18:48.260 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c09ffb640 1 -- 172.21.15.173:0/2876097369 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c0c0a6e90 con 0x7f1c0c0a6680 2024-09-17T18:18:48.260 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c09ffb640 10 monclient: _finish_auth 0 2024-09-17T18:18:48.260 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c09ffb640 10 monclient: _check_auth_tickets 2024-09-17T18:18:48.261 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:18:48.261 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c09ffb640 1 -- 172.21.15.173:0/2876097369 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4002019445 0 0) 0x7f1c04001d20 con 0x7f1c0c0a6680 2024-09-17T18:18:48.261 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c119e2640 5 monclient: authenticate success, global_id 4117 2024-09-17T18:18:48.261 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c119e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:18:48.261 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c09ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:48.261 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c09ffb640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-17T18:18:48.261 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c09ffb640 10 monclient: dump: 2024-09-17T18:18:48.261 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-09-17T18:18:48.261 INFO:teuthology.orchestra.run.smithi173.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:48.261 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:48.261 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:48.261 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:48.261 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-09-17T18:18:48.261 INFO:teuthology.orchestra.run.smithi173.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:48.262 INFO:teuthology.orchestra.run.smithi173.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:48.262 INFO:teuthology.orchestra.run.smithi173.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:48.262 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T18:18:48.262 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c09ffb640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] but i'm connected to v1:172.21.15.173:6789/0, reconnecting 2024-09-17T18:18:48.262 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c09ffb640 10 monclient: _reopen_session rank -1 2024-09-17T18:18:48.262 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c09ffb640 1 -- 172.21.15.173:0/2876097369 >> v1:172.21.15.173:6789/0 conn(0x7f1c0c0a6680 legacy=0x7f1c0c1401c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:48.262 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c09ffb640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T18:18:48.262 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c09ffb640 1 --2- 172.21.15.173:0/2876097369 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1bf8003b20 0x7f1bf8003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:48.262 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c09ffb640 10 monclient(hunting): picked mon.a con 0x7f1bf8003b20 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:18:48.262 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c09ffb640 1 --2- 172.21.15.173:0/2876097369 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1bf8004800 0x7f1bf8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:48.262 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c09ffb640 10 monclient(hunting): picked mon.c con 0x7f1bf8004800 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:18:48.262 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c09ffb640 1 --2- 172.21.15.173:0/2876097369 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1bf80053c0 0x7f1bf800ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:48.262 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c09ffb640 10 monclient(hunting): picked mon.b con 0x7f1bf80053c0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:18:48.262 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c09ffb640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:48.262 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c09ffb640 1 -- 172.21.15.173:0/2876097369 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1c0c09d4c0 con 0x7f1bf8003b20 2024-09-17T18:18:48.263 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c09ffb640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:48.263 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c09ffb640 1 -- 172.21.15.173:0/2876097369 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf800c300 con 0x7f1bf80053c0 2024-09-17T18:18:48.263 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c09ffb640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:48.263 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c09ffb640 1 -- 172.21.15.173:0/2876097369 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1bf800c500 con 0x7f1bf8004800 2024-09-17T18:18:48.263 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c09ffb640 10 monclient(hunting): _renew_subs 2024-09-17T18:18:48.263 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c09ffb640 10 monclient(hunting): _finish_auth 0 2024-09-17T18:18:48.263 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c119e2640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-17T18:18:48.263 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c0b7fe640 1 --2- 172.21.15.173:0/2876097369 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1bf80053c0 0x7f1bf800ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:48.264 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c0b7fe640 10 monclient(hunting): get_auth_request con 0x7f1bf80053c0 auth_method 0 2024-09-17T18:18:48.264 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c0b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:48.264 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c0b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:48.264 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c0b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:18:48.264 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c0affd640 1 --2- 172.21.15.173:0/2876097369 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1bf8003b20 0x7f1bf8003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:48.264 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c0a7fc640 1 --2- 172.21.15.173:0/2876097369 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1bf8004800 0x7f1bf8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:48.264 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c0affd640 10 monclient(hunting): get_auth_request con 0x7f1bf8003b20 auth_method 0 2024-09-17T18:18:48.264 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c0affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:48.264 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c0affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:48.264 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c0affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:18:48.264 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c0b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:48.264 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c0b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:48.264 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c0b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:18:48.265 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c0a7fc640 10 monclient(hunting): get_auth_request con 0x7f1bf8004800 auth_method 0 2024-09-17T18:18:48.265 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c0a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:48.265 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c0a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:48.265 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c0a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:18:48.265 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c0affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:48.265 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c0affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:48.265 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c0affd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:18:48.265 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c0a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:48.265 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c0a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:48.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c0a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:18:48.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c0b7fe640 10 monclient(hunting): handle_auth_done global_id 4117 payload 293 2024-09-17T18:18:48.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c0b7fe640 1 -- 172.21.15.173:0/2876097369 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1bf8004800 msgr2=0x7f1bf8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:18:48.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c0b7fe640 1 --2- 172.21.15.173:0/2876097369 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1bf8004800 0x7f1bf8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:48.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c0b7fe640 1 -- 172.21.15.173:0/2876097369 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1bf8003b20 msgr2=0x7f1bf8003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:18:48.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c0b7fe640 1 --2- 172.21.15.173:0/2876097369 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1bf8003b20 0x7f1bf8003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:48.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c0b7fe640 10 monclient: _finish_hunting 0 2024-09-17T18:18:48.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c0b7fe640 1 monclient: found mon.b 2024-09-17T18:18:48.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:18:48.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c0b7fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:18:48.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c0b7fe640 1 -- 172.21.15.173:0/2876097369 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bf8003520 con 0x7f1bf80053c0 2024-09-17T18:18:48.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c0b7fe640 10 monclient: _finish_auth 0 2024-09-17T18:18:48.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c0b7fe640 10 monclient: _check_auth_tickets 2024-09-17T18:18:48.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.253+0000 7f1c0b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:18:48.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.254+0000 7f1c0b7fe640 1 -- 172.21.15.173:0/2876097369 reap_dead start 2024-09-17T18:18:48.270 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.256+0000 7f1c0b7fe640 1 -- 172.21.15.173:0/2876097369 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1bf80053c0 msgr2=0x7f1bf800ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:18:48.270 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.256+0000 7f1c0b7fe640 1 --2- 172.21.15.173:0/2876097369 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1bf80053c0 0x7f1bf800ba20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1c04001710 tx=0x7f1bf8005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30f1b99557d7db38 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:18:48.270 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.256+0000 7f1c09ffb640 1 -- 172.21.15.173:0/2876097369 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c0400a7e0 con 0x7f1bf80053c0 2024-09-17T18:18:48.270 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.256+0000 7f1c09ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:48.270 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.256+0000 7f1c09ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:18:48.270 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.256+0000 7f1c09ffb640 10 monclient: dump: 2024-09-17T18:18:48.271 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-09-17T18:18:48.271 INFO:teuthology.orchestra.run.smithi173.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:48.271 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:48.271 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:48.271 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:48.271 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-09-17T18:18:48.271 INFO:teuthology.orchestra.run.smithi173.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:48.271 INFO:teuthology.orchestra.run.smithi173.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:48.271 INFO:teuthology.orchestra.run.smithi173.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:48.271 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T18:18:48.271 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.256+0000 7f1c09ffb640 1 -- 172.21.15.173:0/2876097369 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c0402f070 con 0x7f1bf80053c0 2024-09-17T18:18:48.271 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.256+0000 7f1c119e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:18:48.271 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.256+0000 7f1c09ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:18:48.272 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.256+0000 7f1c119e2640 10 monclient: get_monmap_and_config success 2024-09-17T18:18:48.272 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.256+0000 7f1c09ffb640 1 -- 172.21.15.173:0/2876097369 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c04034470 con 0x7f1bf80053c0 2024-09-17T18:18:48.272 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.256+0000 7f1c09ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:48.272 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.256+0000 7f1c119e2640 4 set_mon_vals no callback set 2024-09-17T18:18:48.272 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.256+0000 7f1c09ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:18:48.272 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.256+0000 7f1c09ffb640 10 monclient: dump: 2024-09-17T18:18:48.272 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-09-17T18:18:48.272 INFO:teuthology.orchestra.run.smithi173.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:48.272 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:48.272 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:48.272 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:48.272 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-09-17T18:18:48.272 INFO:teuthology.orchestra.run.smithi173.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:48.273 INFO:teuthology.orchestra.run.smithi173.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:48.273 INFO:teuthology.orchestra.run.smithi173.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:48.273 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T18:18:48.273 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.257+0000 7f1c119e2640 10 monclient: shutdown 2024-09-17T18:18:48.273 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.257+0000 7f1c119e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:18:48.273 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.257+0000 7f1c119e2640 1 -- 172.21.15.173:0/2876097369 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1bf80053c0 msgr2=0x7f1bf800ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:48.273 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.257+0000 7f1c119e2640 1 --2- 172.21.15.173:0/2876097369 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1bf80053c0 0x7f1bf800ba20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1c04001710 tx=0x7f1bf8005830 comp rx=0 tx=0).stop 2024-09-17T18:18:48.273 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.257+0000 7f1c119e2640 1 -- 172.21.15.173:0/2876097369 shutdown_connections 2024-09-17T18:18:48.274 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.257+0000 7f1c119e2640 1 --2- 172.21.15.173:0/2876097369 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1bf80053c0 0x7f1bf800ba20 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:48.274 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.257+0000 7f1c119e2640 1 -- 172.21.15.173:0/2876097369 >> 172.21.15.173:0/2876097369 conn(0x7f1c0c0a7c90 msgr2=0x7f1c0c0a80a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:18:48.274 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.257+0000 7f1c119e2640 1 -- 172.21.15.173:0/2876097369 shutdown_connections 2024-09-17T18:18:48.274 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.257+0000 7f1c119e2640 1 -- 172.21.15.173:0/2876097369 wait complete. 2024-09-17T18:18:48.274 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.257+0000 7f1c119e2640 10 monclient: build_initial_monmap 2024-09-17T18:18:48.274 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.257+0000 7f1c119e2640 10 monclient: monmap: 2024-09-17T18:18:48.274 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-09-17T18:18:48.274 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:18:48.274 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 0.000000 2024-09-17T18:18:48.274 INFO:teuthology.orchestra.run.smithi173.stderr:created 0.000000 2024-09-17T18:18:48.274 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-09-17T18:18:48.274 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-09-17T18:18:48.274 INFO:teuthology.orchestra.run.smithi173.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:18:48.274 INFO:teuthology.orchestra.run.smithi173.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:18:48.275 INFO:teuthology.orchestra.run.smithi173.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:18:48.275 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T18:18:48.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.257+0000 7f1c119e2640 1 librados: starting msgr at 2024-09-17T18:18:48.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.257+0000 7f1c119e2640 1 librados: starting objecter 2024-09-17T18:18:48.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.257+0000 7f1c119e2640 1 Processor -- start 2024-09-17T18:18:48.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.258+0000 7f1c119e2640 1 -- start start 2024-09-17T18:18:48.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.258+0000 7f1c119e2640 1 librados: setting wanted keys 2024-09-17T18:18:48.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.258+0000 7f1c119e2640 1 librados: calling monclient init 2024-09-17T18:18:48.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.258+0000 7f1c119e2640 10 monclient: init 2024-09-17T18:18:48.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.258+0000 7f1c119e2640 10 monclient: _reopen_session rank -1 2024-09-17T18:18:48.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.258+0000 7f1c119e2640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T18:18:48.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.258+0000 7f1c119e2640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1bf80053c0 0x7f1c0c125f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:48.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.258+0000 7f1c119e2640 10 monclient(hunting): picked mon.noname-c con 0x7f1bf80053c0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:18:48.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.258+0000 7f1c119e2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c0c126450 0x7f1c0c1268d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:48.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.258+0000 7f1c119e2640 10 monclient(hunting): picked mon.noname-b con 0x7f1c0c126450 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:18:48.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.258+0000 7f1c119e2640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1c0c1d1a70 0x7f1c0c1d3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:48.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.258+0000 7f1c119e2640 10 monclient(hunting): picked mon.noname-a con 0x7f1c0c1d1a70 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:18:48.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.258+0000 7f1c119e2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:48.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.258+0000 7f1c119e2640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1c0c0a5f70 con 0x7f1c0c1d1a70 2024-09-17T18:18:48.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.258+0000 7f1c119e2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:48.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.258+0000 7f1c119e2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1c0c0a60f0 con 0x7f1c0c126450 2024-09-17T18:18:48.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.258+0000 7f1c119e2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:48.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.258+0000 7f1c119e2640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1c0c0a6270 con 0x7f1bf80053c0 2024-09-17T18:18:48.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.258+0000 7f1c119e2640 10 monclient(hunting): _renew_subs 2024-09-17T18:18:48.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.258+0000 7f1c119e2640 10 monclient(hunting): authenticate will time out at 726.050663s 2024-09-17T18:18:48.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.258+0000 7f1c0a7fc640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c0c126450 0x7f1c0c1268d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:48.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.258+0000 7f1c0a7fc640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c0c126450 0x7f1c0c1268d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:58128/0 (socket says 172.21.15.173:58128) 2024-09-17T18:18:48.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.258+0000 7f1c0a7fc640 1 -- 172.21.15.173:0/4081757408 learned_addr learned my addr 172.21.15.173:0/4081757408 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-09-17T18:18:48.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.258+0000 7f1c0a7fc640 10 monclient(hunting): get_auth_request con 0x7f1c0c126450 auth_method 0 2024-09-17T18:18:48.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.258+0000 7f1c0a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:48.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.258+0000 7f1c0a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:48.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.258+0000 7f1c0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:48.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.258+0000 7f1c0a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:48.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.258+0000 7f1c0a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:48.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.258+0000 7f1c0a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:18:48.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.258+0000 7f1c0b7fe640 1 --2- 172.21.15.173:0/4081757408 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1c0c1d1a70 0x7f1c0c1d3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:48.278 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.258+0000 7f1c0affd640 1 --2- 172.21.15.173:0/4081757408 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1bf80053c0 0x7f1c0c125f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:48.278 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.258+0000 7f1c0b7fe640 10 monclient(hunting): get_auth_request con 0x7f1c0c1d1a70 auth_method 0 2024-09-17T18:18:48.278 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.258+0000 7f1c0b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:48.278 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.258+0000 7f1c0b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:48.278 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.258+0000 7f1c0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:48.278 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.258+0000 7f1c0affd640 10 monclient(hunting): get_auth_request con 0x7f1bf80053c0 auth_method 0 2024-09-17T18:18:48.278 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.258+0000 7f1c0affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:48.278 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.258+0000 7f1c0affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:48.278 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.258+0000 7f1c0affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:48.278 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.259+0000 7f1c0a7fc640 10 monclient(hunting): handle_auth_done global_id 4120 payload 824 2024-09-17T18:18:48.278 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.259+0000 7f1c0a7fc640 1 -- 172.21.15.173:0/4081757408 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1bf80053c0 msgr2=0x7f1c0c125f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:48.278 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.259+0000 7f1c0a7fc640 1 --2- 172.21.15.173:0/4081757408 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1bf80053c0 0x7f1c0c125f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:48.278 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.259+0000 7f1c0a7fc640 1 -- 172.21.15.173:0/4081757408 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1c0c1d1a70 msgr2=0x7f1c0c1d3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:48.278 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.259+0000 7f1c0a7fc640 1 --2- 172.21.15.173:0/4081757408 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1c0c1d1a70 0x7f1c0c1d3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:48.279 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.259+0000 7f1c0a7fc640 10 monclient: _finish_hunting 0 2024-09-17T18:18:48.279 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.259+0000 7f1c0a7fc640 1 monclient: found mon.noname-b 2024-09-17T18:18:48.279 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.259+0000 7f1c0a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:18:48.279 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.259+0000 7f1c0a7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.173:3300/0 2024-09-17T18:18:48.279 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.259+0000 7f1c0a7fc640 1 -- 172.21.15.173:0/4081757408 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c0c129ce0 con 0x7f1c0c126450 2024-09-17T18:18:48.279 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.259+0000 7f1c0b7fe640 1 --2- 172.21.15.173:0/4081757408 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1c0c1d1a70 0x7f1c0c1d3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T18:18:48.279 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.259+0000 7f1c0a7fc640 1 -- 172.21.15.173:0/4081757408 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c0c126450 msgr2=0x7f1c0c1268d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T18:18:48.279 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.259+0000 7f1c0a7fc640 1 -- 172.21.15.173:0/4081757408 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c0c126450 msgr2=0x7f1c0c1268d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:18:48.279 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.259+0000 7f1c0a7fc640 1 --2- 172.21.15.173:0/4081757408 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c0c126450 0x7f1c0c1268d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f1bf40118e0 tx=0x7f1bf4011da0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:18:48.279 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.259+0000 7f1c0a7fc640 1 --2- 172.21.15.173:0/4081757408 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c0c126450 0x7f1c0c1268d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:18:48.464 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.460+0000 7f1c0a7fc640 1 --2- 172.21.15.173:0/4081757408 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c0c126450 0x7f1c0c1268d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:48.464 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.460+0000 7f1c0a7fc640 10 monclient: get_auth_request con 0x7f1c0c126450 auth_method 0 2024-09-17T18:18:48.464 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.460+0000 7f1c0a7fc640 0 --2- 172.21.15.173:0/4081757408 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c0c126450 0x7f1c0c1268d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-17T18:18:48.465 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.460+0000 7f1c0a7fc640 1 --2- 172.21.15.173:0/4081757408 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c0c126450 0x7f1c0c1268d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:48.465 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.460+0000 7f1c03fff640 10 monclient: ms_handle_reset current mon [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:18:48.465 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.460+0000 7f1c03fff640 10 monclient: _reopen_session rank -1 2024-09-17T18:18:48.465 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.460+0000 7f1c03fff640 1 -- 172.21.15.173:0/4081757408 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c0c126450 msgr2=0x7f1c0c1268d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:18:48.465 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.460+0000 7f1c03fff640 1 --2- 172.21.15.173:0/4081757408 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c0c126450 0x7f1c0c1268d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:48.465 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.460+0000 7f1c03fff640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T18:18:48.465 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.460+0000 7f1c03fff640 1 --2- 172.21.15.173:0/4081757408 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1bf80053c0 0x7f1bec0054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:48.465 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.460+0000 7f1c03fff640 10 monclient(hunting): picked mon.noname-c con 0x7f1bf80053c0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:18:48.465 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.460+0000 7f1c03fff640 1 --2- 172.21.15.173:0/4081757408 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1c0c1d1a70 0x7f1bec007f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:48.465 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.460+0000 7f1c03fff640 10 monclient(hunting): picked mon.noname-a con 0x7f1c0c1d1a70 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:18:48.465 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.460+0000 7f1c03fff640 1 --2- 172.21.15.173:0/4081757408 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1bec0089c0 0x7f1bec00ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:48.465 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.460+0000 7f1c03fff640 10 monclient(hunting): picked mon.noname-b con 0x7f1bec0089c0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:18:48.466 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.460+0000 7f1c03fff640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:48.466 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.460+0000 7f1c03fff640 1 -- 172.21.15.173:0/4081757408 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1bec00b980 con 0x7f1c0c1d1a70 2024-09-17T18:18:48.466 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.460+0000 7f1c03fff640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:48.466 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.460+0000 7f1c03fff640 1 -- 172.21.15.173:0/4081757408 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1bec00bb30 con 0x7f1bec0089c0 2024-09-17T18:18:48.466 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.460+0000 7f1c03fff640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:48.466 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.460+0000 7f1c03fff640 1 -- 172.21.15.173:0/4081757408 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1bec00bd30 con 0x7f1bf80053c0 2024-09-17T18:18:48.466 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.460+0000 7f1c03fff640 10 monclient(hunting): _renew_subs 2024-09-17T18:18:48.466 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.460+0000 7f1c0b7fe640 1 --2- 172.21.15.173:0/4081757408 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1bec0089c0 0x7f1bec00ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:48.466 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.461+0000 7f1c0b7fe640 10 monclient(hunting): get_auth_request con 0x7f1bec0089c0 auth_method 0 2024-09-17T18:18:48.466 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.461+0000 7f1c0b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:48.466 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.461+0000 7f1c0b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:48.466 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.461+0000 7f1c0b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:18:48.466 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.461+0000 7f1c0b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:48.467 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.461+0000 7f1c0b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:48.467 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.461+0000 7f1c0b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:18:48.467 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.461+0000 7f1c0affd640 1 --2- 172.21.15.173:0/4081757408 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1bf80053c0 0x7f1bec0054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:48.467 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.461+0000 7f1c0a7fc640 1 --2- 172.21.15.173:0/4081757408 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1c0c1d1a70 0x7f1bec007f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:48.467 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.461+0000 7f1c0affd640 10 monclient(hunting): get_auth_request con 0x7f1bf80053c0 auth_method 0 2024-09-17T18:18:48.467 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.461+0000 7f1c0affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:48.467 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.461+0000 7f1c0affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:48.467 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.461+0000 7f1c0affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:18:48.467 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.461+0000 7f1c0b7fe640 10 monclient(hunting): handle_auth_done global_id 4120 payload 293 2024-09-17T18:18:48.467 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.461+0000 7f1c0b7fe640 1 -- 172.21.15.173:0/4081757408 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1bf80053c0 msgr2=0x7f1bec0054d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:48.467 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.461+0000 7f1c0b7fe640 1 --2- 172.21.15.173:0/4081757408 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1bf80053c0 0x7f1bec0054d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:48.467 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.461+0000 7f1c0b7fe640 1 -- 172.21.15.173:0/4081757408 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1c0c1d1a70 msgr2=0x7f1bec007f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:48.467 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.461+0000 7f1c0b7fe640 1 --2- 172.21.15.173:0/4081757408 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1c0c1d1a70 0x7f1bec007f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:48.467 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.461+0000 7f1c0b7fe640 10 monclient: _finish_hunting 0 2024-09-17T18:18:48.468 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.461+0000 7f1c0b7fe640 1 monclient: found mon.noname-b 2024-09-17T18:18:48.468 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.461+0000 7f1c0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:18:48.468 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.461+0000 7f1c0b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.173:3300/0 2024-09-17T18:18:48.468 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.461+0000 7f1c0b7fe640 1 -- 172.21.15.173:0/4081757408 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bec00bff0 con 0x7f1bec0089c0 2024-09-17T18:18:48.468 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.461+0000 7f1c0a7fc640 10 monclient: get_auth_request con 0x7f1c0c1d1a70 auth_method 0 2024-09-17T18:18:48.468 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.461+0000 7f1c0a7fc640 1 --2- 172.21.15.173:0/4081757408 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1c0c1d1a70 0x7f1bec007f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T18:18:48.469 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.463+0000 7f1c0b7fe640 1 -- 172.21.15.173:0/4081757408 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1bec0089c0 msgr2=0x7f1bec00ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:18:48.469 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.463+0000 7f1c0b7fe640 1 --2- 172.21.15.173:0/4081757408 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1bec0089c0 0x7f1bec00ae50 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c06ef30 tx=0x7f1bf4009190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:18:48.470 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.463+0000 7f1c03fff640 10 client.?.objecter ms_handle_connect 0x7f1bec0089c0 2024-09-17T18:18:48.470 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.463+0000 7f1c03fff640 10 client.?.objecter resend_mon_ops 2024-09-17T18:18:48.470 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.465+0000 7f1c03fff640 1 -- 172.21.15.173:0/4081757408 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c0400a7e0 con 0x7f1bec0089c0 2024-09-17T18:18:48.470 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.465+0000 7f1c03fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:48.470 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.465+0000 7f1c03fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:18:48.470 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.465+0000 7f1c03fff640 10 monclient: dump: 2024-09-17T18:18:48.470 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-09-17T18:18:48.470 INFO:teuthology.orchestra.run.smithi173.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:48.470 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:48.470 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:48.470 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:48.470 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-09-17T18:18:48.470 INFO:teuthology.orchestra.run.smithi173.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:48.470 INFO:teuthology.orchestra.run.smithi173.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:48.471 INFO:teuthology.orchestra.run.smithi173.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:48.471 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T18:18:48.471 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.465+0000 7f1c03fff640 10 monclient: _finish_auth 0 2024-09-17T18:18:48.471 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.465+0000 7f1c03fff640 10 monclient: _check_auth_tickets 2024-09-17T18:18:48.471 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.465+0000 7f1c03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:18:48.471 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.465+0000 7f1c119e2640 5 monclient: authenticate success, global_id 4120 2024-09-17T18:18:48.471 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.465+0000 7f1c03fff640 1 -- 172.21.15.173:0/4081757408 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c0400d040 con 0x7f1bec0089c0 2024-09-17T18:18:48.471 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.465+0000 7f1c03fff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:18:48.471 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.465+0000 7f1c03fff640 1 -- 172.21.15.173:0/4081757408 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c04001ca0 con 0x7f1bec0089c0 2024-09-17T18:18:48.471 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.465+0000 7f1c119e2640 10 monclient: _renew_subs 2024-09-17T18:18:48.471 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.465+0000 7f1c119e2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:18:48.471 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.465+0000 7f1c119e2640 1 -- 172.21.15.173:0/4081757408 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c0c09f840 con 0x7f1bec0089c0 2024-09-17T18:18:48.471 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.465+0000 7f1c097fa640 4 set_mon_vals no callback set 2024-09-17T18:18:48.471 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.465+0000 7f1c03fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:48.472 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.465+0000 7f1c03fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:18:48.472 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.465+0000 7f1c03fff640 10 monclient: dump: 2024-09-17T18:18:48.472 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-09-17T18:18:48.472 INFO:teuthology.orchestra.run.smithi173.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:48.472 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:48.472 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:48.472 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:48.472 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-09-17T18:18:48.472 INFO:teuthology.orchestra.run.smithi173.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:48.472 INFO:teuthology.orchestra.run.smithi173.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:48.472 INFO:teuthology.orchestra.run.smithi173.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:48.472 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T18:18:48.472 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.465+0000 7f1c119e2640 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:18:48.472 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.465+0000 7f1c119e2640 10 monclient: _renew_subs 2024-09-17T18:18:48.472 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.465+0000 7f1c119e2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:18:48.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.465+0000 7f1c119e2640 1 -- 172.21.15.173:0/4081757408 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c0c127980 con 0x7f1bec0089c0 2024-09-17T18:18:48.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.465+0000 7f1c119e2640 1 librados: init done 2024-09-17T18:18:48.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.466+0000 7f1c03fff640 1 -- 172.21.15.173:0/4081757408 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1c04004b60 con 0x7f1bec0089c0 2024-09-17T18:18:48.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.466+0000 7f1c03fff640 1 -- 172.21.15.173:0/4081757408 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f1c04034c40 con 0x7f1bec0089c0 2024-09-17T18:18:48.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.466+0000 7f1c03fff640 10 client.4120.objecter ms_dispatch 0x7f1c0c000b90 osd_map(4..4 src has 1..4) 2024-09-17T18:18:48.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.466+0000 7f1c03fff640 3 client.4120.objecter handle_osd_map got epochs [4,4] > 0 2024-09-17T18:18:48.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.466+0000 7f1c03fff640 3 client.4120.objecter handle_osd_map decoding full epoch 4 2024-09-17T18:18:48.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.466+0000 7f1c03fff640 20 client.4120.objecter dump_active .. 0 homeless 2024-09-17T18:18:48.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.467+0000 7f1c119e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:18:48.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.467+0000 7f1c119e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:18:48.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.467+0000 7f1c119e2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:18:48.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.467+0000 7f1c119e2640 1 -- 172.21.15.173:0/4081757408 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c0c070550 con 0x7f1bec0089c0 2024-09-17T18:18:48.474 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.469+0000 7f1c03fff640 1 -- 172.21.15.173:0/4081757408 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f1c04060860 con 0x7f1bec0089c0 2024-09-17T18:18:48.474 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.469+0000 7f1c03fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:18:48.474 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.469+0000 7f1c03fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:18:48.561 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.559+0000 7f1c119e2640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6fb45798-21d4-449e-9f68-7210d984e036", "id": 2}] 2024-09-17T18:18:48.562 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.559+0000 7f1c119e2640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6fb45798-21d4-449e-9f68-7210d984e036", "id": 2}] 2024-09-17T18:18:48.562 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.559+0000 7f1c119e2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:18:48.562 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.559+0000 7f1c119e2640 1 -- 172.21.15.173:0/4081757408 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd new", "uuid": "6fb45798-21d4-449e-9f68-7210d984e036", "id": 2} v 0) -- 0x7f1c0c06cd40 con 0x7f1bec0089c0 2024-09-17T18:18:48.567 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.565+0000 7f1c03fff640 1 -- 172.21.15.173:0/4081757408 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6fb45798-21d4-449e-9f68-7210d984e036", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f1c0404e030 con 0x7f1bec0089c0 2024-09-17T18:18:48.567 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.565+0000 7f1c03fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6fb45798-21d4-449e-9f68-7210d984e036", "id": 2}] 2024-09-17T18:18:48.567 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.565+0000 7f1c03fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:18:48.568 INFO:teuthology.orchestra.run.smithi173.stdout:2 2024-09-17T18:18:48.569 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.566+0000 7f1c119e2640 10 librados: watch_flush enter 2024-09-17T18:18:48.569 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.566+0000 7f1c119e2640 10 librados: watch_flush exit 2024-09-17T18:18:48.569 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.566+0000 7f1c119e2640 20 client.4120.objecter shutdown clearing up homeless session... 2024-09-17T18:18:48.569 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.566+0000 7f1c119e2640 10 client.4120.objecter successfully canceled tick 2024-09-17T18:18:48.569 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.566+0000 7f1c119e2640 10 monclient: shutdown 2024-09-17T18:18:48.569 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.566+0000 7f1c119e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:18:48.569 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.566+0000 7f1c119e2640 1 -- 172.21.15.173:0/4081757408 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1bec0089c0 msgr2=0x7f1bec00ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:48.569 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.566+0000 7f1c119e2640 1 --2- 172.21.15.173:0/4081757408 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1bec0089c0 0x7f1bec00ae50 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c06ef30 tx=0x7f1bf4009190 comp rx=0 tx=0).stop 2024-09-17T18:18:48.569 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.566+0000 7f1c119e2640 1 -- 172.21.15.173:0/4081757408 shutdown_connections 2024-09-17T18:18:48.570 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.566+0000 7f1c119e2640 1 --2- 172.21.15.173:0/4081757408 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1bec0089c0 0x7f1bec00ae50 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:48.570 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.566+0000 7f1c119e2640 1 --2- 172.21.15.173:0/4081757408 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1c0c1d1a70 0x7f1bec007f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:48.570 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.566+0000 7f1c119e2640 1 --2- 172.21.15.173:0/4081757408 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1bf80053c0 0x7f1bec0054d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:48.570 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.566+0000 7f1c119e2640 1 -- 172.21.15.173:0/4081757408 >> 172.21.15.173:0/4081757408 conn(0x7f1c0c0a7c90 msgr2=0x7f1c0c0a0f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:18:48.570 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.566+0000 7f1c119e2640 1 -- 172.21.15.173:0/4081757408 shutdown_connections 2024-09-17T18:18:48.570 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.566+0000 7f1c119e2640 1 -- 172.21.15.173:0/4081757408 wait complete. 2024-09-17T18:18:48.570 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.566+0000 7f1c119e2640 1 librados: shutdown 2024-09-17T18:18:48.579 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster ceph osd new 3de6630d-4fb7-4f64-b4f9-e5c304b1be63 3 2024-09-17T18:18:48.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.676+0000 7fe06c450640 10 monclient: get_monmap_and_config 2024-09-17T18:18:48.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.676+0000 7fe06c450640 10 monclient: build_initial_monmap 2024-09-17T18:18:48.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.676+0000 7fe06c450640 10 monclient: monmap: 2024-09-17T18:18:48.679 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-09-17T18:18:48.679 INFO:teuthology.orchestra.run.smithi173.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:48.679 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-09-17T18:18:48.677454+0000 2024-09-17T18:18:48.679 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-09-17T18:18:48.677454+0000 2024-09-17T18:18:48.679 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-09-17T18:18:48.679 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-09-17T18:18:48.679 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:18:48.679 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:18:48.679 INFO:teuthology.orchestra.run.smithi173.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:18:48.679 INFO:teuthology.orchestra.run.smithi173.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:18:48.679 INFO:teuthology.orchestra.run.smithi173.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:18:48.679 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T18:18:48.681 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.677+0000 7fe06c450640 1 Processor -- start 2024-09-17T18:18:48.681 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.677+0000 7fe06c450640 1 -- start start 2024-09-17T18:18:48.681 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.677+0000 7fe06c450640 10 monclient: init 2024-09-17T18:18:48.681 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.677+0000 7fe06c450640 10 monclient: _reopen_session rank -1 2024-09-17T18:18:48.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.677+0000 7fe06c450640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-17T18:18:48.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.677+0000 7fe06c450640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe0640a5b90 0x7fe0640a5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:48.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.677+0000 7fe06c450640 10 monclient(hunting): picked mon.noname-a con 0x7fe0640a5b90 addr v2:172.21.15.121:3300/0 2024-09-17T18:18:48.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.677+0000 7fe06c450640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe0640a71c0 addr v1:172.21.15.121:6789/0 2024-09-17T18:18:48.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.677+0000 7fe06c450640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe0640a64b0 addr v1:172.21.15.173:6789/0 2024-09-17T18:18:48.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.677+0000 7fe06c450640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0641397e0 con 0x7fe0640a71c0 2024-09-17T18:18:48.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.677+0000 7fe06c450640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe06409cb90 con 0x7fe0640a64b0 2024-09-17T18:18:48.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.677+0000 7fe06c450640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:48.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.677+0000 7fe06c450640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fe06409d4c0 con 0x7fe0640a5b90 2024-09-17T18:18:48.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.677+0000 7fe06c450640 10 monclient(hunting): _renew_subs 2024-09-17T18:18:48.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.677+0000 7fe06c450640 10 monclient(hunting): authenticate will time out at 726.469954s 2024-09-17T18:18:48.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.678+0000 7fe06a9ed640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fe0640a64b0 0x7fe0641390d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:33832/0 (socket says 172.21.15.173:33832) 2024-09-17T18:18:48.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.678+0000 7fe06a9ed640 1 -- 172.21.15.173:0/3203073967 learned_addr learned my addr 172.21.15.173:0/3203073967 (peer_addr_for_me v1:172.21.15.173:0/0) 2024-09-17T18:18:48.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.678+0000 7fe06a1ec640 1 --2- 172.21.15.173:0/3203073967 >> v2:172.21.15.121:3300/0 conn(0x7fe0640a5b90 0x7fe0640a5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:48.683 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.678+0000 7fe06a1ec640 10 monclient(hunting): get_auth_request con 0x7fe0640a5b90 auth_method 0 2024-09-17T18:18:48.683 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.678+0000 7fe06a1ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:48.683 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.678+0000 7fe06a1ec640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:48.683 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.678+0000 7fe06a1ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:48.683 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.678+0000 7fe06a1ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:48.683 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.678+0000 7fe06a1ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:48.683 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.678+0000 7fe06a1ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:18:48.691 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe06a9ed640 1 -- 172.21.15.173:0/3203073967 >> v1:172.21.15.173:6789/0 conn(0x7fe0640a64b0 legacy=0x7fe0641390d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:18:48.692 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0691ea640 1 -- 172.21.15.173:0/3203073967 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2442996023 0 0) 0x7fe06409cb90 con 0x7fe0640a64b0 2024-09-17T18:18:48.692 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0691ea640 10 monclient(hunting): _negotiate 2024-09-17T18:18:48.692 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0691ea640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:48.692 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0691ea640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:48.692 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0691ea640 10 monclient(hunting): my global_id is 4123 2024-09-17T18:18:48.692 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0691ea640 1 -- 172.21.15.173:0/3203073967 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe050003520 con 0x7fe0640a64b0 2024-09-17T18:18:48.692 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0691ea640 1 -- 172.21.15.173:0/3203073967 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2010719824 0 0) 0x7fe050003520 con 0x7fe0640a64b0 2024-09-17T18:18:48.692 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0691ea640 1 -- 172.21.15.173:0/3203073967 >> v2:172.21.15.121:3300/0 conn(0x7fe0640a5b90 msgr2=0x7fe0640a5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:18:48.692 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0691ea640 1 --2- 172.21.15.173:0/3203073967 >> v2:172.21.15.121:3300/0 conn(0x7fe0640a5b90 0x7fe0640a5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:48.692 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0691ea640 1 -- 172.21.15.173:0/3203073967 >> v1:172.21.15.121:6789/0 conn(0x7fe0640a71c0 legacy=0x7fe0641389c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:18:48.692 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0691ea640 10 monclient: _finish_hunting 0 2024-09-17T18:18:48.692 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0691ea640 1 monclient: found mon.noname-c-legacy 2024-09-17T18:18:48.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0691ea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:18:48.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0691ea640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.173:6789/0 2024-09-17T18:18:48.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0691ea640 1 -- 172.21.15.173:0/3203073967 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe064070550 con 0x7fe0640a64b0 2024-09-17T18:18:48.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0691ea640 10 monclient: _finish_auth 0 2024-09-17T18:18:48.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0691ea640 10 monclient: _check_auth_tickets 2024-09-17T18:18:48.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0691ea640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:18:48.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0691ea640 1 -- 172.21.15.173:0/3203073967 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4002019445 0 0) 0x7fe05c001de0 con 0x7fe0640a64b0 2024-09-17T18:18:48.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe06c450640 5 monclient: authenticate success, global_id 4123 2024-09-17T18:18:48.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe06c450640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:18:48.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0691ea640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:48.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0691ea640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-17T18:18:48.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0691ea640 10 monclient: dump: 2024-09-17T18:18:48.693 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-09-17T18:18:48.693 INFO:teuthology.orchestra.run.smithi173.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:48.694 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:48.694 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:48.694 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:48.694 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-09-17T18:18:48.694 INFO:teuthology.orchestra.run.smithi173.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:48.694 INFO:teuthology.orchestra.run.smithi173.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:48.694 INFO:teuthology.orchestra.run.smithi173.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:48.694 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T18:18:48.694 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0691ea640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] but i'm connected to v1:172.21.15.173:6789/0, reconnecting 2024-09-17T18:18:48.694 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0691ea640 10 monclient: _reopen_session rank -1 2024-09-17T18:18:48.694 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0691ea640 1 -- 172.21.15.173:0/3203073967 >> v1:172.21.15.173:6789/0 conn(0x7fe0640a64b0 legacy=0x7fe0641390d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:48.694 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0691ea640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T18:18:48.694 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0691ea640 1 --2- 172.21.15.173:0/3203073967 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe050003a00 0x7fe050003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:48.695 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0691ea640 10 monclient(hunting): picked mon.b con 0x7fe050003a00 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:18:48.695 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0691ea640 1 --2- 172.21.15.173:0/3203073967 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe050004800 0x7fe050004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:48.695 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0691ea640 10 monclient(hunting): picked mon.a con 0x7fe050004800 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:18:48.695 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0691ea640 1 --2- 172.21.15.173:0/3203073967 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0500052e0 0x7fe05000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:48.695 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0691ea640 10 monclient(hunting): picked mon.c con 0x7fe0500052e0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:18:48.695 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0691ea640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:48.695 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0691ea640 1 -- 172.21.15.173:0/3203073967 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe06409d4c0 con 0x7fe050004800 2024-09-17T18:18:48.695 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0691ea640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:48.695 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0691ea640 1 -- 172.21.15.173:0/3203073967 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe05000c1e0 con 0x7fe050003a00 2024-09-17T18:18:48.695 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0691ea640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:48.695 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0691ea640 1 -- 172.21.15.173:0/3203073967 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe05000c3e0 con 0x7fe0500052e0 2024-09-17T18:18:48.695 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0691ea640 10 monclient(hunting): _renew_subs 2024-09-17T18:18:48.695 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0691ea640 10 monclient(hunting): _finish_auth 0 2024-09-17T18:18:48.696 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe06c450640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-17T18:18:48.696 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe06a1ec640 1 --2- 172.21.15.173:0/3203073967 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe050003a00 0x7fe050003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:48.696 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe06a1ec640 10 monclient(hunting): get_auth_request con 0x7fe050003a00 auth_method 0 2024-09-17T18:18:48.696 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe06a1ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:48.696 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe06a1ec640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:48.696 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe06a1ec640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:18:48.696 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe06a1ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:48.696 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe06a1ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:48.696 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe06a1ec640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:18:48.696 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe06a9ed640 1 --2- 172.21.15.173:0/3203073967 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0500052e0 0x7fe05000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:48.696 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0699eb640 1 --2- 172.21.15.173:0/3203073967 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe050004800 0x7fe050004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:48.696 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe06a9ed640 10 monclient(hunting): get_auth_request con 0x7fe0500052e0 auth_method 0 2024-09-17T18:18:48.696 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe06a9ed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:48.697 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe06a9ed640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:48.697 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe06a9ed640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:18:48.697 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0699eb640 10 monclient(hunting): get_auth_request con 0x7fe050004800 auth_method 0 2024-09-17T18:18:48.697 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0699eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:48.697 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0699eb640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:48.697 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.680+0000 7fe0699eb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:18:48.697 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.681+0000 7fe06a9ed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:48.697 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.681+0000 7fe06a9ed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:48.697 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.681+0000 7fe06a9ed640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:18:48.697 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.681+0000 7fe06a1ec640 10 monclient(hunting): handle_auth_done global_id 4123 payload 293 2024-09-17T18:18:48.697 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.681+0000 7fe06a1ec640 1 -- 172.21.15.173:0/3203073967 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0500052e0 msgr2=0x7fe05000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:18:48.697 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.681+0000 7fe06a1ec640 1 --2- 172.21.15.173:0/3203073967 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0500052e0 0x7fe05000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:48.697 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.681+0000 7fe06a1ec640 1 -- 172.21.15.173:0/3203073967 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe050004800 msgr2=0x7fe050004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:18:48.698 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.681+0000 7fe06a1ec640 1 --2- 172.21.15.173:0/3203073967 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe050004800 0x7fe050004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:48.698 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.681+0000 7fe06a1ec640 10 monclient: _finish_hunting 0 2024-09-17T18:18:48.698 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.681+0000 7fe06a1ec640 1 monclient: found mon.b 2024-09-17T18:18:48.698 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.681+0000 7fe06a1ec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:18:48.698 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.681+0000 7fe06a1ec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:18:48.698 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.681+0000 7fe06a1ec640 1 -- 172.21.15.173:0/3203073967 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe050003520 con 0x7fe050003a00 2024-09-17T18:18:48.698 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.681+0000 7fe06a1ec640 10 monclient: _finish_auth 0 2024-09-17T18:18:48.698 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.681+0000 7fe06a1ec640 10 monclient: _check_auth_tickets 2024-09-17T18:18:48.698 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.681+0000 7fe06a1ec640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:18:48.698 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.681+0000 7fe06a9ed640 1 -- 172.21.15.173:0/3203073967 reap_dead start 2024-09-17T18:18:48.698 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.681+0000 7fe0699eb640 1 --2- 172.21.15.173:0/3203073967 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe050004800 0x7fe050004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T18:18:48.698 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.683+0000 7fe06a1ec640 1 -- 172.21.15.173:0/3203073967 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe050003a00 msgr2=0x7fe050003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:18:48.698 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.683+0000 7fe06a1ec640 1 --2- 172.21.15.173:0/3203073967 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe050003a00 0x7fe050003e00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe060009a90 tx=0x7fe050005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94283c6eea90a89a server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:18:48.698 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.683+0000 7fe0691ea640 1 -- 172.21.15.173:0/3203073967 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe060036070 con 0x7fe050003a00 2024-09-17T18:18:48.699 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.683+0000 7fe0691ea640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:48.699 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.684+0000 7fe0691ea640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:18:48.699 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.684+0000 7fe0691ea640 10 monclient: dump: 2024-09-17T18:18:48.699 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-09-17T18:18:48.699 INFO:teuthology.orchestra.run.smithi173.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:48.699 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:48.699 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:48.699 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:48.699 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-09-17T18:18:48.699 INFO:teuthology.orchestra.run.smithi173.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:48.699 INFO:teuthology.orchestra.run.smithi173.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:48.699 INFO:teuthology.orchestra.run.smithi173.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:48.699 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T18:18:48.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.684+0000 7fe06c450640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:18:48.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.684+0000 7fe0691ea640 1 -- 172.21.15.173:0/3203073967 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe06002f510 con 0x7fe050003a00 2024-09-17T18:18:48.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.684+0000 7fe0691ea640 10 monclient: handle_config config(0 keys) 2024-09-17T18:18:48.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.684+0000 7fe06c450640 10 monclient: get_monmap_and_config success 2024-09-17T18:18:48.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.684+0000 7fe0691ea640 1 -- 172.21.15.173:0/3203073967 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe060009690 con 0x7fe050003a00 2024-09-17T18:18:48.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.684+0000 7fe06c450640 4 set_mon_vals no callback set 2024-09-17T18:18:48.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.684+0000 7fe0691ea640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:48.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.684+0000 7fe0691ea640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:18:48.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.684+0000 7fe0691ea640 10 monclient: dump: 2024-09-17T18:18:48.700 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-09-17T18:18:48.700 INFO:teuthology.orchestra.run.smithi173.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:48.700 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:48.700 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:48.701 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:48.701 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-09-17T18:18:48.701 INFO:teuthology.orchestra.run.smithi173.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:48.701 INFO:teuthology.orchestra.run.smithi173.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:48.701 INFO:teuthology.orchestra.run.smithi173.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:48.701 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T18:18:48.701 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.684+0000 7fe06c450640 10 monclient: shutdown 2024-09-17T18:18:48.701 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.684+0000 7fe06c450640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:18:48.701 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.684+0000 7fe06c450640 1 -- 172.21.15.173:0/3203073967 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe050003a00 msgr2=0x7fe050003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:48.701 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.684+0000 7fe06c450640 1 --2- 172.21.15.173:0/3203073967 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe050003a00 0x7fe050003e00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe060009a90 tx=0x7fe050005830 comp rx=0 tx=0).stop 2024-09-17T18:18:48.701 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.684+0000 7fe06c450640 1 -- 172.21.15.173:0/3203073967 shutdown_connections 2024-09-17T18:18:48.701 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.684+0000 7fe06c450640 1 --2- 172.21.15.173:0/3203073967 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe050003a00 0x7fe050003e00 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:48.701 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.684+0000 7fe06c450640 1 -- 172.21.15.173:0/3203073967 >> 172.21.15.173:0/3203073967 conn(0x7fe0640a7c90 msgr2=0x7fe0640a80a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:18:48.701 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.684+0000 7fe06c450640 1 -- 172.21.15.173:0/3203073967 shutdown_connections 2024-09-17T18:18:48.702 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.684+0000 7fe06c450640 1 -- 172.21.15.173:0/3203073967 wait complete. 2024-09-17T18:18:48.702 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.685+0000 7fe06c450640 10 monclient: build_initial_monmap 2024-09-17T18:18:48.702 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.685+0000 7fe06c450640 10 monclient: monmap: 2024-09-17T18:18:48.702 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-09-17T18:18:48.702 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:18:48.702 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 0.000000 2024-09-17T18:18:48.702 INFO:teuthology.orchestra.run.smithi173.stderr:created 0.000000 2024-09-17T18:18:48.702 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-09-17T18:18:48.702 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-09-17T18:18:48.702 INFO:teuthology.orchestra.run.smithi173.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:18:48.702 INFO:teuthology.orchestra.run.smithi173.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:18:48.702 INFO:teuthology.orchestra.run.smithi173.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:18:48.702 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T18:18:48.702 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.685+0000 7fe06c450640 1 librados: starting msgr at 2024-09-17T18:18:48.702 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.685+0000 7fe06c450640 1 librados: starting objecter 2024-09-17T18:18:48.703 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.685+0000 7fe06c450640 1 Processor -- start 2024-09-17T18:18:48.703 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.685+0000 7fe06c450640 1 -- start start 2024-09-17T18:18:48.703 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.685+0000 7fe06c450640 1 librados: setting wanted keys 2024-09-17T18:18:48.703 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.685+0000 7fe06c450640 1 librados: calling monclient init 2024-09-17T18:18:48.703 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.685+0000 7fe06c450640 10 monclient: init 2024-09-17T18:18:48.703 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.686+0000 7fe06c450640 10 monclient: _reopen_session rank -1 2024-09-17T18:18:48.703 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.686+0000 7fe06c450640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T18:18:48.703 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.686+0000 7fe06c450640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe050003a00 0x7fe0641cd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:48.703 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.686+0000 7fe06c450640 10 monclient(hunting): picked mon.noname-b con 0x7fe050003a00 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:18:48.703 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.686+0000 7fe06c450640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe064138de0 0x7fe064122750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:48.703 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.686+0000 7fe06c450640 10 monclient(hunting): picked mon.noname-c con 0x7fe064138de0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:18:48.703 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.686+0000 7fe06c450640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe064139260 0x7fe064122c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:48.703 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.686+0000 7fe06c450640 10 monclient(hunting): picked mon.noname-a con 0x7fe064139260 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:18:48.704 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.686+0000 7fe06c450640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:48.704 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.686+0000 7fe06c450640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe0640a6e90 con 0x7fe064139260 2024-09-17T18:18:48.704 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.686+0000 7fe06c450640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:48.704 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.686+0000 7fe06c450640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe0640a8a20 con 0x7fe050003a00 2024-09-17T18:18:48.704 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.686+0000 7fe06c450640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:48.704 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.686+0000 7fe06c450640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe0640a2380 con 0x7fe064138de0 2024-09-17T18:18:48.704 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.686+0000 7fe06c450640 10 monclient(hunting): _renew_subs 2024-09-17T18:18:48.704 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.686+0000 7fe06c450640 10 monclient(hunting): authenticate will time out at 726.478273s 2024-09-17T18:18:48.704 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.686+0000 7fe06a1ec640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe050003a00 0x7fe0641cd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:48.704 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.686+0000 7fe06a1ec640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe050003a00 0x7fe0641cd2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:58182/0 (socket says 172.21.15.173:58182) 2024-09-17T18:18:48.704 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.686+0000 7fe06a1ec640 1 -- 172.21.15.173:0/1776263058 learned_addr learned my addr 172.21.15.173:0/1776263058 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-09-17T18:18:48.704 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.686+0000 7fe06a1ec640 10 monclient(hunting): get_auth_request con 0x7fe050003a00 auth_method 0 2024-09-17T18:18:48.704 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.686+0000 7fe06a1ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:48.704 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.686+0000 7fe06a1ec640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:48.705 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.686+0000 7fe06a1ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:48.705 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.686+0000 7fe06a1ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:48.705 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.686+0000 7fe06a1ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:48.705 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.686+0000 7fe06a1ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:18:48.705 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.686+0000 7fe06a1ec640 10 monclient(hunting): handle_auth_done global_id 4126 payload 824 2024-09-17T18:18:48.705 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.686+0000 7fe06a1ec640 1 -- 172.21.15.173:0/1776263058 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe064138de0 msgr2=0x7fe064122750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:48.705 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.686+0000 7fe0699eb640 1 --2- 172.21.15.173:0/1776263058 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe064138de0 0x7fe064122750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:48.705 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.686+0000 7fe06a1ec640 1 --2- 172.21.15.173:0/1776263058 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe064138de0 0x7fe064122750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:48.705 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.686+0000 7fe06a1ec640 1 -- 172.21.15.173:0/1776263058 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe064139260 msgr2=0x7fe064122c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:48.705 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.686+0000 7fe06a1ec640 1 --2- 172.21.15.173:0/1776263058 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe064139260 0x7fe064122c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:48.705 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.686+0000 7fe06a1ec640 10 monclient: _finish_hunting 0 2024-09-17T18:18:48.705 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.686+0000 7fe06a1ec640 1 monclient: found mon.noname-b 2024-09-17T18:18:48.705 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.686+0000 7fe06a1ec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:18:48.706 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.686+0000 7fe06a1ec640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.173:3300/0 2024-09-17T18:18:48.706 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.686+0000 7fe06a1ec640 1 -- 172.21.15.173:0/1776263058 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe064124210 con 0x7fe050003a00 2024-09-17T18:18:48.706 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.688+0000 7fe06a1ec640 1 -- 172.21.15.173:0/1776263058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe050003a00 msgr2=0x7fe0641cd2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:18:48.706 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.688+0000 7fe06a1ec640 1 --2- 172.21.15.173:0/1776263058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe050003a00 0x7fe0641cd2a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe060034040 tx=0x7fe060004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:18:48.706 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.688+0000 7fe05affd640 10 client.?.objecter ms_handle_connect 0x7fe050003a00 2024-09-17T18:18:48.706 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.688+0000 7fe05affd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:18:48.706 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.688+0000 7fe05affd640 1 -- 172.21.15.173:0/1776263058 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe06003f030 con 0x7fe050003a00 2024-09-17T18:18:48.706 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.688+0000 7fe05affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:48.706 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.688+0000 7fe05affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:18:48.706 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.688+0000 7fe05affd640 10 monclient: dump: 2024-09-17T18:18:48.706 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-09-17T18:18:48.706 INFO:teuthology.orchestra.run.smithi173.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:48.706 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:48.706 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:48.706 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:48.707 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-09-17T18:18:48.707 INFO:teuthology.orchestra.run.smithi173.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:48.707 INFO:teuthology.orchestra.run.smithi173.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:48.707 INFO:teuthology.orchestra.run.smithi173.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:48.707 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T18:18:48.707 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.689+0000 7fe05affd640 10 monclient: _finish_auth 0 2024-09-17T18:18:48.707 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.689+0000 7fe05affd640 10 monclient: _check_auth_tickets 2024-09-17T18:18:48.707 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.689+0000 7fe05affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:18:48.707 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.689+0000 7fe06c450640 5 monclient: authenticate success, global_id 4126 2024-09-17T18:18:48.707 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.689+0000 7fe05affd640 1 -- 172.21.15.173:0/1776263058 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe060037040 con 0x7fe050003a00 2024-09-17T18:18:48.707 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.689+0000 7fe06c450640 10 monclient: _renew_subs 2024-09-17T18:18:48.707 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.689+0000 7fe06c450640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:18:48.707 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.689+0000 7fe06c450640 1 -- 172.21.15.173:0/1776263058 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe064126570 con 0x7fe050003a00 2024-09-17T18:18:48.707 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.689+0000 7fe05affd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:18:48.707 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.689+0000 7fe05affd640 1 -- 172.21.15.173:0/1776263058 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe060036070 con 0x7fe050003a00 2024-09-17T18:18:48.708 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.689+0000 7fe05affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:48.708 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.689+0000 7fe05affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:18:48.708 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.689+0000 7fe05affd640 10 monclient: dump: 2024-09-17T18:18:48.708 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-09-17T18:18:48.708 INFO:teuthology.orchestra.run.smithi173.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:48.708 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:48.708 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:48.708 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:48.708 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-09-17T18:18:48.708 INFO:teuthology.orchestra.run.smithi173.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:48.708 INFO:teuthology.orchestra.run.smithi173.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:48.708 INFO:teuthology.orchestra.run.smithi173.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:48.708 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T18:18:48.709 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.689+0000 7fe0689e9640 4 set_mon_vals no callback set 2024-09-17T18:18:48.709 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.689+0000 7fe06c450640 10 client.4126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:18:48.709 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.689+0000 7fe06c450640 10 monclient: _renew_subs 2024-09-17T18:18:48.709 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.689+0000 7fe06c450640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:18:48.709 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.689+0000 7fe06c450640 1 -- 172.21.15.173:0/1776263058 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0641cd7e0 con 0x7fe050003a00 2024-09-17T18:18:48.709 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.689+0000 7fe06c450640 1 librados: init done 2024-09-17T18:18:48.709 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.689+0000 7fe05affd640 1 -- 172.21.15.173:0/1776263058 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe060042070 con 0x7fe050003a00 2024-09-17T18:18:48.709 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.689+0000 7fe05affd640 1 -- 172.21.15.173:0/1776263058 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7fe060003ca0 con 0x7fe050003a00 2024-09-17T18:18:48.709 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.689+0000 7fe05affd640 10 client.4126.objecter ms_dispatch 0x7fe064000b90 osd_map(5..5 src has 1..5) 2024-09-17T18:18:48.709 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.689+0000 7fe05affd640 3 client.4126.objecter handle_osd_map got epochs [5,5] > 0 2024-09-17T18:18:48.709 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.689+0000 7fe05affd640 3 client.4126.objecter handle_osd_map decoding full epoch 5 2024-09-17T18:18:48.709 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.689+0000 7fe05affd640 20 client.4126.objecter dump_active .. 0 homeless 2024-09-17T18:18:48.709 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.690+0000 7fe06c450640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:18:48.709 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.690+0000 7fe06c450640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:18:48.709 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.690+0000 7fe06c450640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:18:48.710 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.690+0000 7fe06c450640 1 -- 172.21.15.173:0/1776263058 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0640a5f70 con 0x7fe050003a00 2024-09-17T18:18:48.710 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.692+0000 7fe05affd640 1 -- 172.21.15.173:0/1776263058 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fe06002fd20 con 0x7fe050003a00 2024-09-17T18:18:48.710 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.692+0000 7fe05affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:18:48.710 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.692+0000 7fe05affd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:18:48.765 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.762+0000 7fe06c450640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "3de6630d-4fb7-4f64-b4f9-e5c304b1be63", "id": 3}] 2024-09-17T18:18:48.765 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.763+0000 7fe06c450640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "3de6630d-4fb7-4f64-b4f9-e5c304b1be63", "id": 3}] 2024-09-17T18:18:48.765 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.763+0000 7fe06c450640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:18:48.765 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.763+0000 7fe06c450640 1 -- 172.21.15.173:0/1776263058 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd new", "uuid": "3de6630d-4fb7-4f64-b4f9-e5c304b1be63", "id": 3} v 0) -- 0x7fe064125f10 con 0x7fe050003a00 2024-09-17T18:18:48.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.770+0000 7fe05affd640 1 -- 172.21.15.173:0/1776263058 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3de6630d-4fb7-4f64-b4f9-e5c304b1be63", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7fe060003610 con 0x7fe050003a00 2024-09-17T18:18:48.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.770+0000 7fe05affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "3de6630d-4fb7-4f64-b4f9-e5c304b1be63", "id": 3}] 2024-09-17T18:18:48.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.770+0000 7fe05affd640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:18:48.773 INFO:teuthology.orchestra.run.smithi173.stdout:3 2024-09-17T18:18:48.774 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.771+0000 7fe06c450640 10 librados: watch_flush enter 2024-09-17T18:18:48.774 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.771+0000 7fe06c450640 10 librados: watch_flush exit 2024-09-17T18:18:48.774 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.771+0000 7fe06c450640 20 client.4126.objecter shutdown clearing up homeless session... 2024-09-17T18:18:48.774 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.771+0000 7fe06c450640 10 client.4126.objecter successfully canceled tick 2024-09-17T18:18:48.774 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.771+0000 7fe06c450640 10 monclient: shutdown 2024-09-17T18:18:48.775 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.771+0000 7fe06c450640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:18:48.775 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.771+0000 7fe06c450640 1 -- 172.21.15.173:0/1776263058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe050003a00 msgr2=0x7fe0641cd2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:48.775 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.771+0000 7fe06c450640 1 --2- 172.21.15.173:0/1776263058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe050003a00 0x7fe0641cd2a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe060034040 tx=0x7fe060004730 comp rx=0 tx=0).stop 2024-09-17T18:18:48.775 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.771+0000 7fe06c450640 1 -- 172.21.15.173:0/1776263058 shutdown_connections 2024-09-17T18:18:48.775 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.771+0000 7fe06c450640 1 --2- 172.21.15.173:0/1776263058 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe064139260 0x7fe064122c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:48.775 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.771+0000 7fe06c450640 1 --2- 172.21.15.173:0/1776263058 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe064138de0 0x7fe064122750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:48.775 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.771+0000 7fe06c450640 1 --2- 172.21.15.173:0/1776263058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe050003a00 0x7fe0641cd2a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:48.775 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.771+0000 7fe06c450640 1 -- 172.21.15.173:0/1776263058 >> 172.21.15.173:0/1776263058 conn(0x7fe0640a7c90 msgr2=0x7fe0640a1790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:18:48.775 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.771+0000 7fe06c450640 1 -- 172.21.15.173:0/1776263058 shutdown_connections 2024-09-17T18:18:48.775 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.771+0000 7fe06c450640 1 -- 172.21.15.173:0/1776263058 wait complete. 2024-09-17T18:18:48.775 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.771+0000 7fe06c450640 1 librados: shutdown 2024-09-17T18:18:48.783 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster ceph osd new cfa74383-b7c9-4ecd-8258-3792fbccc237 4 2024-09-17T18:18:48.881 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.879+0000 7fadb3c0f640 10 monclient: get_monmap_and_config 2024-09-17T18:18:48.882 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.879+0000 7fadb3c0f640 10 monclient: build_initial_monmap 2024-09-17T18:18:48.882 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.879+0000 7fadb3c0f640 10 monclient: monmap: 2024-09-17T18:18:48.882 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-09-17T18:18:48.882 INFO:teuthology.orchestra.run.smithi173.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:48.882 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-09-17T18:18:48.880600+0000 2024-09-17T18:18:48.882 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-09-17T18:18:48.880600+0000 2024-09-17T18:18:48.882 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-09-17T18:18:48.882 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-09-17T18:18:48.882 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:18:48.882 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:18:48.882 INFO:teuthology.orchestra.run.smithi173.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:18:48.882 INFO:teuthology.orchestra.run.smithi173.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:18:48.882 INFO:teuthology.orchestra.run.smithi173.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:18:48.882 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T18:18:48.884 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.880+0000 7fadb3c0f640 1 Processor -- start 2024-09-17T18:18:48.884 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.880+0000 7fadb3c0f640 1 -- start start 2024-09-17T18:18:48.885 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.880+0000 7fadb3c0f640 10 monclient: init 2024-09-17T18:18:48.885 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.880+0000 7fadb3c0f640 10 monclient: _reopen_session rank -1 2024-09-17T18:18:48.885 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.880+0000 7fadb3c0f640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-17T18:18:48.885 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.880+0000 7fadb3c0f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fadac0a5b90 addr v1:172.21.15.121:6789/0 2024-09-17T18:18:48.885 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.880+0000 7fadb3c0f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fadac0a71c0 0x7fadac13dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:48.885 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.880+0000 7fadb3c0f640 10 monclient(hunting): picked mon.noname-b con 0x7fadac0a71c0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:18:48.885 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.880+0000 7fadb3c0f640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fadac0a6680 0x7fadac1401c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:48.885 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.880+0000 7fadb3c0f640 10 monclient(hunting): picked mon.noname-c con 0x7fadac0a6680 addr v2:172.21.15.173:3300/0 2024-09-17T18:18:48.885 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.880+0000 7fadb3c0f640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fadac09cb90 con 0x7fadac0a5b90 2024-09-17T18:18:48.885 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.880+0000 7fadb3c0f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:48.885 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.880+0000 7fadb3c0f640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fadac09d4c0 con 0x7fadac0a6680 2024-09-17T18:18:48.885 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.880+0000 7fadb3c0f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:48.885 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.880+0000 7fadb3c0f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fadac0a6c50 con 0x7fadac0a71c0 2024-09-17T18:18:48.885 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.880+0000 7fadb3c0f640 10 monclient(hunting): _renew_subs 2024-09-17T18:18:48.886 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.880+0000 7fadb3c0f640 10 monclient(hunting): authenticate will time out at 726.672977s 2024-09-17T18:18:48.886 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.881+0000 7fadb21ac640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fadac0a6680 0x7fadac1401c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:48.886 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.881+0000 7fadb21ac640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fadac0a6680 0x7fadac1401c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:58188/0 (socket says 172.21.15.173:58188) 2024-09-17T18:18:48.886 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.881+0000 7fadb21ac640 1 -- 172.21.15.173:0/1996993455 learned_addr learned my addr 172.21.15.173:0/1996993455 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-09-17T18:18:48.886 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.881+0000 7fadb21ac640 10 monclient(hunting): get_auth_request con 0x7fadac0a6680 auth_method 0 2024-09-17T18:18:48.886 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.881+0000 7fadb21ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:48.886 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.881+0000 7fadb21ac640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:48.886 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.881+0000 7fadb21ac640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:48.886 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.881+0000 7fadb21ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:48.886 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.881+0000 7fadb21ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:48.886 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.881+0000 7fadb21ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:18:48.886 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.881+0000 7fadb11aa640 1 --2- 172.21.15.173:0/1996993455 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fadac0a71c0 0x7fadac13dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:48.886 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.881+0000 7fadb11aa640 10 monclient(hunting): get_auth_request con 0x7fadac0a71c0 auth_method 0 2024-09-17T18:18:48.886 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.881+0000 7fadb11aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:48.887 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.881+0000 7fadb11aa640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:48.887 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.881+0000 7fadb11aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:48.887 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.881+0000 7fadb21ac640 10 monclient(hunting): handle_auth_done global_id 4129 payload 274 2024-09-17T18:18:48.887 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.881+0000 7fadb21ac640 1 -- 172.21.15.173:0/1996993455 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fadac0a71c0 msgr2=0x7fadac13dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:18:48.887 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.881+0000 7fadb21ac640 1 --2- 172.21.15.173:0/1996993455 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fadac0a71c0 0x7fadac13dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:48.887 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.881+0000 7fadb21ac640 1 -- 172.21.15.173:0/1996993455 >> v1:172.21.15.121:6789/0 conn(0x7fadac0a5b90 legacy=0x7fadac0a5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:18:48.887 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.881+0000 7fadb21ac640 10 monclient: _finish_hunting 0 2024-09-17T18:18:48.887 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.881+0000 7fadb21ac640 1 monclient: found mon.noname-c 2024-09-17T18:18:48.887 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.881+0000 7fadb21ac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:18:48.887 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.881+0000 7fadb21ac640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.173:3300/0 2024-09-17T18:18:48.887 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.881+0000 7fadb21ac640 1 -- 172.21.15.173:0/1996993455 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadac070550 con 0x7fadac0a6680 2024-09-17T18:18:48.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.884+0000 7fadb21ac640 1 -- 172.21.15.173:0/1996993455 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fadac0a6680 msgr2=0x7fadac1401c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:18:48.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.884+0000 7fadb21ac640 1 --2- 172.21.15.173:0/1996993455 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fadac0a6680 0x7fadac1401c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fada4009e50 tx=0x7fada402a560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e96a2364c8e1e77 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:18:48.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.884+0000 7fadb09a9640 1 -- 172.21.15.173:0/1996993455 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada4036070 con 0x7fadac0a6680 2024-09-17T18:18:48.893 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.884+0000 7fadb09a9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:48.893 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.884+0000 7fadb09a9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:18:48.893 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.884+0000 7fadb09a9640 10 monclient: dump: 2024-09-17T18:18:48.893 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-09-17T18:18:48.893 INFO:teuthology.orchestra.run.smithi173.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:48.893 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:48.893 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:48.893 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:48.893 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-09-17T18:18:48.893 INFO:teuthology.orchestra.run.smithi173.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:48.893 INFO:teuthology.orchestra.run.smithi173.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:48.893 INFO:teuthology.orchestra.run.smithi173.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:48.893 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T18:18:48.894 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.884+0000 7fadb09a9640 10 monclient: _finish_auth 0 2024-09-17T18:18:48.894 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.884+0000 7fadb09a9640 10 monclient: _check_auth_tickets 2024-09-17T18:18:48.894 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.884+0000 7fadb09a9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:18:48.894 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.884+0000 7fadb09a9640 1 -- 172.21.15.173:0/1996993455 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fada4007800 con 0x7fadac0a6680 2024-09-17T18:18:48.894 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.884+0000 7fadb3c0f640 5 monclient: authenticate success, global_id 4129 2024-09-17T18:18:48.894 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.884+0000 7fadb3c0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:18:48.894 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.884+0000 7fadb09a9640 10 monclient: handle_config config(0 keys) 2024-09-17T18:18:48.894 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.884+0000 7fadb3c0f640 10 monclient: get_monmap_and_config success 2024-09-17T18:18:48.894 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.884+0000 7fadb09a9640 1 -- 172.21.15.173:0/1996993455 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada4007bd0 con 0x7fadac0a6680 2024-09-17T18:18:48.894 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.884+0000 7fadb09a9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:48.894 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.884+0000 7fadb3c0f640 4 set_mon_vals no callback set 2024-09-17T18:18:48.894 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.884+0000 7fadb09a9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:18:48.894 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.884+0000 7fadb09a9640 10 monclient: dump: 2024-09-17T18:18:48.894 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-09-17T18:18:48.895 INFO:teuthology.orchestra.run.smithi173.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:48.895 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:48.895 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:48.895 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:48.895 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-09-17T18:18:48.895 INFO:teuthology.orchestra.run.smithi173.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:48.895 INFO:teuthology.orchestra.run.smithi173.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:48.895 INFO:teuthology.orchestra.run.smithi173.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:48.895 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T18:18:48.895 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.884+0000 7fadb3c0f640 10 monclient: shutdown 2024-09-17T18:18:48.895 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.884+0000 7fadb3c0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:18:48.895 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.884+0000 7fadb3c0f640 1 -- 172.21.15.173:0/1996993455 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fadac0a6680 msgr2=0x7fadac1401c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:48.895 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.884+0000 7fadb3c0f640 1 --2- 172.21.15.173:0/1996993455 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fadac0a6680 0x7fadac1401c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fada4009e50 tx=0x7fada402a560 comp rx=0 tx=0).stop 2024-09-17T18:18:48.895 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.884+0000 7fadb3c0f640 1 -- 172.21.15.173:0/1996993455 shutdown_connections 2024-09-17T18:18:48.896 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.884+0000 7fadb3c0f640 1 --2- 172.21.15.173:0/1996993455 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fadac0a6680 0x7fadac1401c0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:48.896 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.884+0000 7fadb3c0f640 1 --2- 172.21.15.173:0/1996993455 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fadac0a71c0 0x7fadac13dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:48.896 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.884+0000 7fadb3c0f640 1 -- 172.21.15.173:0/1996993455 >> 172.21.15.173:0/1996993455 conn(0x7fadac0a7c90 msgr2=0x7fadac0a80a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:18:48.896 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.885+0000 7fadb3c0f640 1 -- 172.21.15.173:0/1996993455 shutdown_connections 2024-09-17T18:18:48.896 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.885+0000 7fadb3c0f640 1 -- 172.21.15.173:0/1996993455 wait complete. 2024-09-17T18:18:48.896 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.885+0000 7fadb3c0f640 10 monclient: build_initial_monmap 2024-09-17T18:18:48.896 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.885+0000 7fadb3c0f640 10 monclient: monmap: 2024-09-17T18:18:48.896 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-09-17T18:18:48.896 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:18:48.897 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 0.000000 2024-09-17T18:18:48.897 INFO:teuthology.orchestra.run.smithi173.stderr:created 0.000000 2024-09-17T18:18:48.897 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-09-17T18:18:48.897 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-09-17T18:18:48.897 INFO:teuthology.orchestra.run.smithi173.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:18:48.897 INFO:teuthology.orchestra.run.smithi173.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:18:48.897 INFO:teuthology.orchestra.run.smithi173.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:18:48.897 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T18:18:48.897 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.885+0000 7fadb3c0f640 1 librados: starting msgr at 2024-09-17T18:18:48.897 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.885+0000 7fadb3c0f640 1 librados: starting objecter 2024-09-17T18:18:48.897 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.885+0000 7fadb3c0f640 1 Processor -- start 2024-09-17T18:18:48.897 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.885+0000 7fadb3c0f640 1 -- start start 2024-09-17T18:18:48.897 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.885+0000 7fadb3c0f640 1 librados: setting wanted keys 2024-09-17T18:18:48.897 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.885+0000 7fadb3c0f640 1 librados: calling monclient init 2024-09-17T18:18:48.898 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.885+0000 7fadb3c0f640 10 monclient: init 2024-09-17T18:18:48.898 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb3c0f640 10 monclient: _reopen_session rank -1 2024-09-17T18:18:48.898 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb3c0f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T18:18:48.898 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb3c0f640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fadac0a5b90 0x7fadac13dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:48.898 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb3c0f640 10 monclient(hunting): picked mon.noname-a con 0x7fadac0a5b90 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:18:48.898 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb3c0f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fadac0a6680 0x7fadac121bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:48.898 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb3c0f640 10 monclient(hunting): picked mon.noname-c con 0x7fadac0a6680 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:18:48.898 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb3c0f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fadac0a71c0 0x7fadac138a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:48.898 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb3c0f640 10 monclient(hunting): picked mon.noname-b con 0x7fadac0a71c0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:18:48.898 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb3c0f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:48.898 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb3c0f640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fadac0a8ba0 con 0x7fadac0a5b90 2024-09-17T18:18:48.898 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb3c0f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:48.898 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb3c0f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fadac0a8a20 con 0x7fadac0a71c0 2024-09-17T18:18:48.898 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb3c0f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:48.899 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb3c0f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fadac140700 con 0x7fadac0a6680 2024-09-17T18:18:48.899 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb3c0f640 10 monclient(hunting): _renew_subs 2024-09-17T18:18:48.899 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb3c0f640 10 monclient(hunting): authenticate will time out at 726.678395s 2024-09-17T18:18:48.899 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb21ac640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fadac0a71c0 0x7fadac138a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:48.899 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb21ac640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fadac0a71c0 0x7fadac138a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:58204/0 (socket says 172.21.15.173:58204) 2024-09-17T18:18:48.899 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb21ac640 1 -- 172.21.15.173:0/3024028659 learned_addr learned my addr 172.21.15.173:0/3024028659 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-09-17T18:18:48.899 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb21ac640 10 monclient(hunting): get_auth_request con 0x7fadac0a71c0 auth_method 0 2024-09-17T18:18:48.899 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb21ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:48.899 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb21ac640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:48.899 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb21ac640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:48.899 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb21ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:48.899 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb21ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:48.899 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb21ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:18:48.899 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb11aa640 1 --2- 172.21.15.173:0/3024028659 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fadac0a6680 0x7fadac121bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:48.900 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb11aa640 1 -- 172.21.15.173:0/3024028659 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fadac0a6680 msgr2=0x7fadac121bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T18:18:48.900 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb11aa640 1 -- 172.21.15.173:0/3024028659 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fadac0a6680 msgr2=0x7fadac121bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:18:48.900 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb11aa640 1 --2- 172.21.15.173:0/3024028659 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fadac0a6680 0x7fadac121bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:18:48.900 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb19ab640 1 --2- 172.21.15.173:0/3024028659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fadac0a5b90 0x7fadac13dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:48.900 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb11aa640 1 --2- 172.21.15.173:0/3024028659 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fadac0a6680 0x7fadac121bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:18:48.900 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb19ab640 10 monclient(hunting): get_auth_request con 0x7fadac0a5b90 auth_method 0 2024-09-17T18:18:48.900 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb19ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:48.900 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb19ab640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:48.900 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb19ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:48.900 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb21ac640 10 monclient(hunting): handle_auth_done global_id 4132 payload 824 2024-09-17T18:18:48.900 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb21ac640 1 -- 172.21.15.173:0/3024028659 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fadac0a6680 msgr2=0x7fadac121bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:18:48.900 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb21ac640 1 --2- 172.21.15.173:0/3024028659 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fadac0a6680 0x7fadac121bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:48.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb21ac640 1 -- 172.21.15.173:0/3024028659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fadac0a5b90 msgr2=0x7fadac13dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:48.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb21ac640 1 --2- 172.21.15.173:0/3024028659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fadac0a5b90 0x7fadac13dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:48.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb21ac640 10 monclient: _finish_hunting 0 2024-09-17T18:18:48.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb21ac640 1 monclient: found mon.noname-b 2024-09-17T18:18:48.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb21ac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:18:48.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb21ac640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.173:3300/0 2024-09-17T18:18:48.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.886+0000 7fadb21ac640 1 -- 172.21.15.173:0/3024028659 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadac13f550 con 0x7fadac0a71c0 2024-09-17T18:18:48.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.888+0000 7fadb21ac640 1 -- 172.21.15.173:0/3024028659 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fadac0a71c0 msgr2=0x7fadac138a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:18:48.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.888+0000 7fadb21ac640 1 --2- 172.21.15.173:0/3024028659 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fadac0a71c0 0x7fadac138a20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fada402ab50 tx=0x7fada4004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:18:48.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.888+0000 7fada27fc640 10 client.?.objecter ms_handle_connect 0x7fadac0a71c0 2024-09-17T18:18:48.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.888+0000 7fada27fc640 10 client.?.objecter resend_mon_ops 2024-09-17T18:18:48.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.889+0000 7fada27fc640 1 -- 172.21.15.173:0/3024028659 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada4041020 con 0x7fadac0a71c0 2024-09-17T18:18:48.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.889+0000 7fada27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:48.902 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.889+0000 7fada27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:18:48.902 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.889+0000 7fada27fc640 10 monclient: dump: 2024-09-17T18:18:48.902 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-09-17T18:18:48.902 INFO:teuthology.orchestra.run.smithi173.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:48.902 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:48.902 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:48.902 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:48.902 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-09-17T18:18:48.902 INFO:teuthology.orchestra.run.smithi173.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:48.902 INFO:teuthology.orchestra.run.smithi173.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:48.902 INFO:teuthology.orchestra.run.smithi173.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:48.902 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T18:18:48.902 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.889+0000 7fada27fc640 10 monclient: _finish_auth 0 2024-09-17T18:18:48.902 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.889+0000 7fada27fc640 10 monclient: _check_auth_tickets 2024-09-17T18:18:48.902 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.889+0000 7fada27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:18:48.903 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.889+0000 7fadb3c0f640 5 monclient: authenticate success, global_id 4132 2024-09-17T18:18:48.903 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.889+0000 7fada27fc640 1 -- 172.21.15.173:0/3024028659 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fada4032950 con 0x7fadac0a71c0 2024-09-17T18:18:48.903 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.889+0000 7fada27fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:18:48.903 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.889+0000 7fadb3c0f640 10 monclient: _renew_subs 2024-09-17T18:18:48.903 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.889+0000 7fada27fc640 1 -- 172.21.15.173:0/3024028659 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada4036040 con 0x7fadac0a71c0 2024-09-17T18:18:48.903 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.889+0000 7fadb3c0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:18:48.903 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.889+0000 7fadb3c0f640 1 -- 172.21.15.173:0/3024028659 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadac140190 con 0x7fadac0a71c0 2024-09-17T18:18:48.903 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.889+0000 7fada27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:48.903 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.889+0000 7fada27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:18:48.903 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.889+0000 7fada27fc640 10 monclient: dump: 2024-09-17T18:18:48.903 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-09-17T18:18:48.903 INFO:teuthology.orchestra.run.smithi173.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:48.903 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:48.903 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:48.904 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:48.904 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-09-17T18:18:48.904 INFO:teuthology.orchestra.run.smithi173.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:48.904 INFO:teuthology.orchestra.run.smithi173.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:48.904 INFO:teuthology.orchestra.run.smithi173.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:48.904 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T18:18:48.904 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.889+0000 7fada3fff640 4 set_mon_vals no callback set 2024-09-17T18:18:48.904 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.889+0000 7fadb3c0f640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:18:48.904 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.889+0000 7fadb3c0f640 10 monclient: _renew_subs 2024-09-17T18:18:48.904 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.889+0000 7fadb3c0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:18:48.904 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.889+0000 7fadb3c0f640 1 -- 172.21.15.173:0/3024028659 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadac1233b0 con 0x7fadac0a71c0 2024-09-17T18:18:48.904 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.889+0000 7fadb3c0f640 1 librados: init done 2024-09-17T18:18:48.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.889+0000 7fada27fc640 1 -- 172.21.15.173:0/3024028659 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fada4044020 con 0x7fadac0a71c0 2024-09-17T18:18:48.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.889+0000 7fada27fc640 1 -- 172.21.15.173:0/3024028659 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7fada403d420 con 0x7fadac0a71c0 2024-09-17T18:18:48.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.889+0000 7fada27fc640 10 client.4132.objecter ms_dispatch 0x7fadac000b90 osd_map(6..6 src has 1..6) 2024-09-17T18:18:48.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.889+0000 7fada27fc640 3 client.4132.objecter handle_osd_map got epochs [6,6] > 0 2024-09-17T18:18:48.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.889+0000 7fada27fc640 3 client.4132.objecter handle_osd_map decoding full epoch 6 2024-09-17T18:18:48.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.889+0000 7fada27fc640 20 client.4132.objecter dump_active .. 0 homeless 2024-09-17T18:18:48.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.890+0000 7fadb3c0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:18:48.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.890+0000 7fadb3c0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:18:48.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.890+0000 7fadb3c0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:18:48.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.890+0000 7fadb3c0f640 1 -- 172.21.15.173:0/3024028659 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadac1231d0 con 0x7fadac0a71c0 2024-09-17T18:18:48.906 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.892+0000 7fada27fc640 1 -- 172.21.15.173:0/3024028659 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fadac1231d0 con 0x7fadac0a71c0 2024-09-17T18:18:48.906 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.892+0000 7fada27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:18:48.906 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.892+0000 7fada27fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:18:48.964 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.962+0000 7fadb3c0f640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "cfa74383-b7c9-4ecd-8258-3792fbccc237", "id": 4}] 2024-09-17T18:18:48.965 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.962+0000 7fadb3c0f640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "cfa74383-b7c9-4ecd-8258-3792fbccc237", "id": 4}] 2024-09-17T18:18:48.965 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.962+0000 7fadb3c0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:18:48.965 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.962+0000 7fadb3c0f640 1 -- 172.21.15.173:0/3024028659 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd new", "uuid": "cfa74383-b7c9-4ecd-8258-3792fbccc237", "id": 4} v 0) -- 0x7fadac0a5f70 con 0x7fadac0a71c0 2024-09-17T18:18:48.972 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.970+0000 7fada27fc640 1 -- 172.21.15.173:0/3024028659 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cfa74383-b7c9-4ecd-8258-3792fbccc237", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7fada404b020 con 0x7fadac0a71c0 2024-09-17T18:18:48.972 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.970+0000 7fada27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "cfa74383-b7c9-4ecd-8258-3792fbccc237", "id": 4}] 2024-09-17T18:18:48.972 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.970+0000 7fada27fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:18:48.973 INFO:teuthology.orchestra.run.smithi173.stdout:4 2024-09-17T18:18:48.974 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.971+0000 7fadb3c0f640 10 librados: watch_flush enter 2024-09-17T18:18:48.974 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.971+0000 7fadb3c0f640 10 librados: watch_flush exit 2024-09-17T18:18:48.974 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.971+0000 7fadb3c0f640 20 client.4132.objecter shutdown clearing up homeless session... 2024-09-17T18:18:48.974 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.971+0000 7fadb3c0f640 10 client.4132.objecter successfully canceled tick 2024-09-17T18:18:48.974 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.971+0000 7fadb3c0f640 10 monclient: shutdown 2024-09-17T18:18:48.974 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.971+0000 7fadb3c0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:18:48.974 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.971+0000 7fadb3c0f640 1 -- 172.21.15.173:0/3024028659 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fadac0a71c0 msgr2=0x7fadac138a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:48.974 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.971+0000 7fadb3c0f640 1 --2- 172.21.15.173:0/3024028659 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fadac0a71c0 0x7fadac138a20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fada402ab50 tx=0x7fada4004a40 comp rx=0 tx=0).stop 2024-09-17T18:18:48.975 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.971+0000 7fadb3c0f640 1 -- 172.21.15.173:0/3024028659 shutdown_connections 2024-09-17T18:18:48.975 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.971+0000 7fadb3c0f640 1 --2- 172.21.15.173:0/3024028659 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fadac0a71c0 0x7fadac138a20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:48.975 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.971+0000 7fadb3c0f640 1 --2- 172.21.15.173:0/3024028659 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fadac0a6680 0x7fadac121bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:48.975 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.971+0000 7fadb3c0f640 1 --2- 172.21.15.173:0/3024028659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fadac0a5b90 0x7fadac13dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:48.975 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.971+0000 7fadb3c0f640 1 -- 172.21.15.173:0/3024028659 >> 172.21.15.173:0/3024028659 conn(0x7fadac0a7c90 msgr2=0x7fadac0a0c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:18:48.975 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.971+0000 7fadb3c0f640 1 -- 172.21.15.173:0/3024028659 shutdown_connections 2024-09-17T18:18:48.975 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.971+0000 7fadb3c0f640 1 -- 172.21.15.173:0/3024028659 wait complete. 2024-09-17T18:18:48.975 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:48.971+0000 7fadb3c0f640 1 librados: shutdown 2024-09-17T18:18:48.984 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster ceph osd new 8bdb7768-a603-4f14-9650-47b503621215 5 2024-09-17T18:18:49.082 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.079+0000 7fe17b59e640 10 monclient: get_monmap_and_config 2024-09-17T18:18:49.082 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.080+0000 7fe17b59e640 10 monclient: build_initial_monmap 2024-09-17T18:18:49.082 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.080+0000 7fe17b59e640 10 monclient: monmap: 2024-09-17T18:18:49.082 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-09-17T18:18:49.082 INFO:teuthology.orchestra.run.smithi173.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:49.082 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-09-17T18:18:49.080911+0000 2024-09-17T18:18:49.083 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-09-17T18:18:49.080911+0000 2024-09-17T18:18:49.083 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-09-17T18:18:49.083 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-09-17T18:18:49.083 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:18:49.083 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:18:49.083 INFO:teuthology.orchestra.run.smithi173.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:18:49.083 INFO:teuthology.orchestra.run.smithi173.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:18:49.083 INFO:teuthology.orchestra.run.smithi173.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:18:49.083 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T18:18:49.085 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.080+0000 7fe17b59e640 1 Processor -- start 2024-09-17T18:18:49.085 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.080+0000 7fe17b59e640 1 -- start start 2024-09-17T18:18:49.085 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.080+0000 7fe17b59e640 10 monclient: init 2024-09-17T18:18:49.085 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.081+0000 7fe17b59e640 10 monclient: _reopen_session rank -1 2024-09-17T18:18:49.085 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.081+0000 7fe17b59e640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-17T18:18:49.086 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.081+0000 7fe17b59e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe17c06ff80 addr v1:172.21.15.173:6789/0 2024-09-17T18:18:49.086 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.081+0000 7fe17b59e640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fe17c09e680 0x7fe17c09ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:49.086 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.081+0000 7fe17b59e640 10 monclient(hunting): picked mon.noname-c con 0x7fe17c09e680 addr v2:172.21.15.173:3300/0 2024-09-17T18:18:49.086 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.081+0000 7fe17b59e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe17c13d5a0 addr v1:172.21.15.121:6789/0 2024-09-17T18:18:49.086 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.081+0000 7fe17b59e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe17c139010 con 0x7fe17c13d5a0 2024-09-17T18:18:49.086 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.081+0000 7fe17b59e640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe17c09ce70 con 0x7fe17c06ff80 2024-09-17T18:18:49.086 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.081+0000 7fe17b59e640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:49.086 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.081+0000 7fe17b59e640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fe17c0a7a40 con 0x7fe17c09e680 2024-09-17T18:18:49.086 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.081+0000 7fe17b59e640 10 monclient(hunting): _renew_subs 2024-09-17T18:18:49.086 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.081+0000 7fe17b59e640 10 monclient(hunting): authenticate will time out at 726.873332s 2024-09-17T18:18:49.086 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.081+0000 7fe173fff640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fe17c09e680 0x7fe17c09ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:49.086 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.081+0000 7fe173fff640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fe17c09e680 0x7fe17c09ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:58206/0 (socket says 172.21.15.173:58206) 2024-09-17T18:18:49.086 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.081+0000 7fe173fff640 1 -- 172.21.15.173:0/653927643 learned_addr learned my addr 172.21.15.173:0/653927643 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-09-17T18:18:49.087 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.081+0000 7fe173fff640 10 monclient(hunting): get_auth_request con 0x7fe17c09e680 auth_method 0 2024-09-17T18:18:49.087 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.081+0000 7fe173fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:49.087 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.081+0000 7fe173fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:49.087 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.081+0000 7fe173fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:49.087 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.081+0000 7fe17a59c640 1 --1- 172.21.15.173:0/653927643 >> v1:172.21.15.173:6789/0 conn(0x7fe17c06ff80 0x7fe17c138900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:33840/0 (socket says 172.21.15.173:33840) 2024-09-17T18:18:49.087 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.081+0000 7fe173fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:49.087 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.081+0000 7fe173fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:49.087 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.081+0000 7fe173fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:18:49.087 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.081+0000 7fe173fff640 10 monclient(hunting): handle_auth_done global_id 4135 payload 274 2024-09-17T18:18:49.087 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.081+0000 7fe173fff640 1 -- 172.21.15.173:0/653927643 >> v1:172.21.15.173:6789/0 conn(0x7fe17c06ff80 legacy=0x7fe17c138900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:18:49.087 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.081+0000 7fe173fff640 1 -- 172.21.15.173:0/653927643 >> v1:172.21.15.121:6789/0 conn(0x7fe17c13d5a0 legacy=0x7fe17c13f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:18:49.087 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.081+0000 7fe173fff640 10 monclient: _finish_hunting 0 2024-09-17T18:18:49.087 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.081+0000 7fe173fff640 1 monclient: found mon.noname-c 2024-09-17T18:18:49.087 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.081+0000 7fe173fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:18:49.088 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.081+0000 7fe173fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.173:3300/0 2024-09-17T18:18:49.088 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.081+0000 7fe173fff640 1 -- 172.21.15.173:0/653927643 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe17c0a8700 con 0x7fe17c09e680 2024-09-17T18:18:49.094 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.084+0000 7fe173fff640 1 -- 172.21.15.173:0/653927643 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe17c09e680 msgr2=0x7fe17c09ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:18:49.094 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.084+0000 7fe173fff640 1 --2- 172.21.15.173:0/653927643 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe17c09e680 0x7fe17c09ea80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe164009870 tx=0x7fe16402a0a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9df2b691db97e9c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:18:49.094 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.084+0000 7fe179d9b640 1 -- 172.21.15.173:0/653927643 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe164037070 con 0x7fe17c09e680 2024-09-17T18:18:49.095 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.084+0000 7fe179d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:49.095 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.084+0000 7fe179d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:18:49.095 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.084+0000 7fe179d9b640 10 monclient: dump: 2024-09-17T18:18:49.095 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-09-17T18:18:49.095 INFO:teuthology.orchestra.run.smithi173.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:49.095 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:49.095 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:49.095 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:49.095 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-09-17T18:18:49.095 INFO:teuthology.orchestra.run.smithi173.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:49.095 INFO:teuthology.orchestra.run.smithi173.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:49.095 INFO:teuthology.orchestra.run.smithi173.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:49.095 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T18:18:49.095 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.084+0000 7fe179d9b640 10 monclient: _finish_auth 0 2024-09-17T18:18:49.096 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.084+0000 7fe179d9b640 10 monclient: _check_auth_tickets 2024-09-17T18:18:49.096 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.084+0000 7fe179d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:18:49.096 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.084+0000 7fe179d9b640 1 -- 172.21.15.173:0/653927643 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe164032640 con 0x7fe17c09e680 2024-09-17T18:18:49.096 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.084+0000 7fe17b59e640 5 monclient: authenticate success, global_id 4135 2024-09-17T18:18:49.096 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.084+0000 7fe17b59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:18:49.096 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.084+0000 7fe179d9b640 10 monclient: handle_config config(0 keys) 2024-09-17T18:18:49.096 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.084+0000 7fe17b59e640 10 monclient: get_monmap_and_config success 2024-09-17T18:18:49.096 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.084+0000 7fe179d9b640 1 -- 172.21.15.173:0/653927643 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe164032a10 con 0x7fe17c09e680 2024-09-17T18:18:49.096 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.084+0000 7fe179d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:49.096 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.084+0000 7fe17b59e640 4 set_mon_vals no callback set 2024-09-17T18:18:49.096 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.084+0000 7fe179d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:18:49.096 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.084+0000 7fe179d9b640 10 monclient: dump: 2024-09-17T18:18:49.096 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-09-17T18:18:49.097 INFO:teuthology.orchestra.run.smithi173.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:49.097 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:49.097 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:49.097 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:49.097 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-09-17T18:18:49.097 INFO:teuthology.orchestra.run.smithi173.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:49.097 INFO:teuthology.orchestra.run.smithi173.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:49.097 INFO:teuthology.orchestra.run.smithi173.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:49.097 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T18:18:49.097 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.085+0000 7fe17b59e640 10 monclient: shutdown 2024-09-17T18:18:49.097 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.085+0000 7fe17b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:18:49.097 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.085+0000 7fe17b59e640 1 -- 172.21.15.173:0/653927643 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe17c09e680 msgr2=0x7fe17c09ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:49.097 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.085+0000 7fe17b59e640 1 --2- 172.21.15.173:0/653927643 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe17c09e680 0x7fe17c09ea80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe164009870 tx=0x7fe16402a0a0 comp rx=0 tx=0).stop 2024-09-17T18:18:49.098 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.085+0000 7fe17b59e640 1 -- 172.21.15.173:0/653927643 shutdown_connections 2024-09-17T18:18:49.098 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.085+0000 7fe17b59e640 1 --2- 172.21.15.173:0/653927643 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe17c09e680 0x7fe17c09ea80 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:49.098 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.085+0000 7fe17b59e640 1 -- 172.21.15.173:0/653927643 >> 172.21.15.173:0/653927643 conn(0x7fe17c0ab7c0 msgr2=0x7fe17c0a7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:18:49.098 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.085+0000 7fe17b59e640 1 -- 172.21.15.173:0/653927643 shutdown_connections 2024-09-17T18:18:49.098 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.085+0000 7fe17b59e640 1 -- 172.21.15.173:0/653927643 wait complete. 2024-09-17T18:18:49.098 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.086+0000 7fe17b59e640 10 monclient: build_initial_monmap 2024-09-17T18:18:49.098 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.086+0000 7fe17b59e640 10 monclient: monmap: 2024-09-17T18:18:49.098 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-09-17T18:18:49.098 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:18:49.098 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 0.000000 2024-09-17T18:18:49.098 INFO:teuthology.orchestra.run.smithi173.stderr:created 0.000000 2024-09-17T18:18:49.098 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-09-17T18:18:49.098 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-09-17T18:18:49.098 INFO:teuthology.orchestra.run.smithi173.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:18:49.099 INFO:teuthology.orchestra.run.smithi173.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:18:49.099 INFO:teuthology.orchestra.run.smithi173.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:18:49.099 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T18:18:49.099 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.086+0000 7fe17b59e640 1 librados: starting msgr at 2024-09-17T18:18:49.099 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.086+0000 7fe17b59e640 1 librados: starting objecter 2024-09-17T18:18:49.099 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.086+0000 7fe17b59e640 1 Processor -- start 2024-09-17T18:18:49.099 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.086+0000 7fe17b59e640 1 -- start start 2024-09-17T18:18:49.099 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.086+0000 7fe17b59e640 1 librados: setting wanted keys 2024-09-17T18:18:49.099 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.086+0000 7fe17b59e640 1 librados: calling monclient init 2024-09-17T18:18:49.099 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.086+0000 7fe17b59e640 10 monclient: init 2024-09-17T18:18:49.099 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.086+0000 7fe17b59e640 10 monclient: _reopen_session rank -1 2024-09-17T18:18:49.099 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.086+0000 7fe17b59e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T18:18:49.099 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.086+0000 7fe17b59e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe17c06ff80 0x7fe17c1cd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:49.099 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.086+0000 7fe17b59e640 10 monclient(hunting): picked mon.noname-a con 0x7fe17c06ff80 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:18:49.100 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.086+0000 7fe17b59e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe17c09e680 0x7fe17c0a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:49.100 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.086+0000 7fe17b59e640 10 monclient(hunting): picked mon.noname-c con 0x7fe17c09e680 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:18:49.100 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.086+0000 7fe17b59e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe17c13d5a0 0x7fe17c0a1c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:49.100 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.086+0000 7fe17b59e640 10 monclient(hunting): picked mon.noname-b con 0x7fe17c13d5a0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:18:49.100 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.086+0000 7fe17b59e640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:49.100 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.086+0000 7fe17b59e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe17c141230 con 0x7fe17c06ff80 2024-09-17T18:18:49.100 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.086+0000 7fe17b59e640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:49.100 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.086+0000 7fe17b59e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe17c1410b0 con 0x7fe17c13d5a0 2024-09-17T18:18:49.100 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.086+0000 7fe17b59e640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:49.100 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.086+0000 7fe17b59e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe17c1413b0 con 0x7fe17c09e680 2024-09-17T18:18:49.100 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.086+0000 7fe17b59e640 10 monclient(hunting): _renew_subs 2024-09-17T18:18:49.100 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.086+0000 7fe17b59e640 10 monclient(hunting): authenticate will time out at 726.878834s 2024-09-17T18:18:49.100 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.086+0000 7fe17ad9d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe17c13d5a0 0x7fe17c0a1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:49.101 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.086+0000 7fe17ad9d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe17c13d5a0 0x7fe17c0a1c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:58220/0 (socket says 172.21.15.173:58220) 2024-09-17T18:18:49.101 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.086+0000 7fe17ad9d640 1 -- 172.21.15.173:0/4015886809 learned_addr learned my addr 172.21.15.173:0/4015886809 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-09-17T18:18:49.101 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.086+0000 7fe17ad9d640 10 monclient(hunting): get_auth_request con 0x7fe17c13d5a0 auth_method 0 2024-09-17T18:18:49.101 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.086+0000 7fe17ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:49.101 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.086+0000 7fe17ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:49.101 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.086+0000 7fe17ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:49.101 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.087+0000 7fe17ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:49.101 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.087+0000 7fe17ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:49.101 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.087+0000 7fe17ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:18:49.101 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.087+0000 7fe173fff640 1 --2- 172.21.15.173:0/4015886809 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe17c09e680 0x7fe17c0a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:49.101 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.087+0000 7fe173fff640 10 monclient(hunting): get_auth_request con 0x7fe17c09e680 auth_method 0 2024-09-17T18:18:49.101 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.087+0000 7fe173fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:49.101 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.087+0000 7fe173fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:49.101 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.087+0000 7fe173fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:49.102 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.087+0000 7fe17a59c640 1 --2- 172.21.15.173:0/4015886809 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe17c06ff80 0x7fe17c1cd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:49.102 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.087+0000 7fe17a59c640 10 monclient(hunting): get_auth_request con 0x7fe17c06ff80 auth_method 0 2024-09-17T18:18:49.102 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.087+0000 7fe17a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:49.102 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.087+0000 7fe17a59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:49.102 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.087+0000 7fe17a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:49.102 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.087+0000 7fe17ad9d640 10 monclient(hunting): handle_auth_done global_id 4138 payload 824 2024-09-17T18:18:49.102 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.087+0000 7fe17ad9d640 1 -- 172.21.15.173:0/4015886809 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe17c09e680 msgr2=0x7fe17c0a16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:49.102 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.087+0000 7fe17ad9d640 1 --2- 172.21.15.173:0/4015886809 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe17c09e680 0x7fe17c0a16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:49.102 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.087+0000 7fe17ad9d640 1 -- 172.21.15.173:0/4015886809 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe17c06ff80 msgr2=0x7fe17c1cd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:49.102 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.087+0000 7fe17ad9d640 1 --2- 172.21.15.173:0/4015886809 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe17c06ff80 0x7fe17c1cd6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:49.102 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.087+0000 7fe17ad9d640 10 monclient: _finish_hunting 0 2024-09-17T18:18:49.102 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.087+0000 7fe17ad9d640 1 monclient: found mon.noname-b 2024-09-17T18:18:49.102 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.087+0000 7fe17ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:18:49.102 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.087+0000 7fe17ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.173:3300/0 2024-09-17T18:18:49.103 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.087+0000 7fe17ad9d640 1 -- 172.21.15.173:0/4015886809 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe17c0a01b0 con 0x7fe17c13d5a0 2024-09-17T18:18:49.103 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.087+0000 7fe173fff640 1 --2- 172.21.15.173:0/4015886809 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe17c09e680 0x7fe17c0a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T18:18:49.103 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.089+0000 7fe17ad9d640 1 -- 172.21.15.173:0/4015886809 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe17c13d5a0 msgr2=0x7fe17c0a1c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:18:49.103 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.089+0000 7fe17ad9d640 1 --2- 172.21.15.173:0/4015886809 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe17c13d5a0 0x7fe17c0a1c00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe17400c780 tx=0x7fe17400cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:18:49.103 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.089+0000 7fe172ffd640 10 client.?.objecter ms_handle_connect 0x7fe17c13d5a0 2024-09-17T18:18:49.103 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.089+0000 7fe172ffd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:18:49.103 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.089+0000 7fe172ffd640 1 -- 172.21.15.173:0/4015886809 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe174017070 con 0x7fe17c13d5a0 2024-09-17T18:18:49.103 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.089+0000 7fe172ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:49.103 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.089+0000 7fe172ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:18:49.103 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.089+0000 7fe172ffd640 10 monclient: dump: 2024-09-17T18:18:49.103 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-09-17T18:18:49.103 INFO:teuthology.orchestra.run.smithi173.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:49.103 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:49.103 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:49.104 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:49.104 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-09-17T18:18:49.104 INFO:teuthology.orchestra.run.smithi173.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:49.104 INFO:teuthology.orchestra.run.smithi173.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:49.104 INFO:teuthology.orchestra.run.smithi173.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:49.104 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T18:18:49.104 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.089+0000 7fe172ffd640 10 monclient: _finish_auth 0 2024-09-17T18:18:49.104 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.089+0000 7fe172ffd640 10 monclient: _check_auth_tickets 2024-09-17T18:18:49.104 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.089+0000 7fe172ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:18:49.104 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.089+0000 7fe172ffd640 1 -- 172.21.15.173:0/4015886809 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe17400ce30 con 0x7fe17c13d5a0 2024-09-17T18:18:49.104 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.089+0000 7fe17b59e640 5 monclient: authenticate success, global_id 4138 2024-09-17T18:18:49.104 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.089+0000 7fe172ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:18:49.104 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.089+0000 7fe17b59e640 10 monclient: _renew_subs 2024-09-17T18:18:49.105 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.089+0000 7fe172ffd640 1 -- 172.21.15.173:0/4015886809 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe174005290 con 0x7fe17c13d5a0 2024-09-17T18:18:49.105 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.089+0000 7fe17b59e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:18:49.105 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.089+0000 7fe17b59e640 1 -- 172.21.15.173:0/4015886809 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe17c0a2380 con 0x7fe17c13d5a0 2024-09-17T18:18:49.105 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.089+0000 7fe172ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:49.105 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.089+0000 7fe172ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:18:49.105 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.089+0000 7fe172ffd640 10 monclient: dump: 2024-09-17T18:18:49.105 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-09-17T18:18:49.105 INFO:teuthology.orchestra.run.smithi173.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:49.105 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:49.105 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:49.105 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:49.105 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-09-17T18:18:49.105 INFO:teuthology.orchestra.run.smithi173.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:49.106 INFO:teuthology.orchestra.run.smithi173.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:49.106 INFO:teuthology.orchestra.run.smithi173.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:49.106 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T18:18:49.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.089+0000 7fe17959a640 4 set_mon_vals no callback set 2024-09-17T18:18:49.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.089+0000 7fe17b59e640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:18:49.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.089+0000 7fe17b59e640 10 monclient: _renew_subs 2024-09-17T18:18:49.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.089+0000 7fe17b59e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:18:49.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.089+0000 7fe17b59e640 1 -- 172.21.15.173:0/4015886809 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe17c09fed0 con 0x7fe17c13d5a0 2024-09-17T18:18:49.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.089+0000 7fe17b59e640 1 librados: init done 2024-09-17T18:18:49.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.089+0000 7fe172ffd640 1 -- 172.21.15.173:0/4015886809 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe17401a070 con 0x7fe17c13d5a0 2024-09-17T18:18:49.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.089+0000 7fe172ffd640 1 -- 172.21.15.173:0/4015886809 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7fe174012980 con 0x7fe17c13d5a0 2024-09-17T18:18:49.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.089+0000 7fe172ffd640 10 client.4138.objecter ms_dispatch 0x7fe17c000b90 osd_map(7..7 src has 1..7) 2024-09-17T18:18:49.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.089+0000 7fe172ffd640 3 client.4138.objecter handle_osd_map got epochs [7,7] > 0 2024-09-17T18:18:49.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.089+0000 7fe172ffd640 3 client.4138.objecter handle_osd_map decoding full epoch 7 2024-09-17T18:18:49.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.089+0000 7fe172ffd640 20 client.4138.objecter dump_active .. 0 homeless 2024-09-17T18:18:49.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.090+0000 7fe17b59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:18:49.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.090+0000 7fe17b59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:18:49.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.090+0000 7fe17b59e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:18:49.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.090+0000 7fe17b59e640 1 -- 172.21.15.173:0/4015886809 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe17c09ea80 con 0x7fe17c13d5a0 2024-09-17T18:18:49.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.092+0000 7fe172ffd640 1 -- 172.21.15.173:0/4015886809 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fe174021020 con 0x7fe17c13d5a0 2024-09-17T18:18:49.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.092+0000 7fe172ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:18:49.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.092+0000 7fe172ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:18:49.165 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.163+0000 7fe17b59e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "8bdb7768-a603-4f14-9650-47b503621215", "id": 5}] 2024-09-17T18:18:49.165 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.163+0000 7fe17b59e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "8bdb7768-a603-4f14-9650-47b503621215", "id": 5}] 2024-09-17T18:18:49.165 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.163+0000 7fe17b59e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:18:49.165 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.163+0000 7fe17b59e640 1 -- 172.21.15.173:0/4015886809 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd new", "uuid": "8bdb7768-a603-4f14-9650-47b503621215", "id": 5} v 0) -- 0x7fe17c070610 con 0x7fe17c13d5a0 2024-09-17T18:18:49.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.167+0000 7fe172ffd640 1 -- 172.21.15.173:0/4015886809 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8bdb7768-a603-4f14-9650-47b503621215", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7fe1740056b0 con 0x7fe17c13d5a0 2024-09-17T18:18:49.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.167+0000 7fe172ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "8bdb7768-a603-4f14-9650-47b503621215", "id": 5}] 2024-09-17T18:18:49.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.167+0000 7fe172ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:18:49.170 INFO:teuthology.orchestra.run.smithi173.stdout:5 2024-09-17T18:18:49.171 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.168+0000 7fe17b59e640 10 librados: watch_flush enter 2024-09-17T18:18:49.171 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.168+0000 7fe17b59e640 10 librados: watch_flush exit 2024-09-17T18:18:49.171 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.168+0000 7fe17b59e640 20 client.4138.objecter shutdown clearing up homeless session... 2024-09-17T18:18:49.171 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.168+0000 7fe17b59e640 10 client.4138.objecter successfully canceled tick 2024-09-17T18:18:49.171 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.168+0000 7fe17b59e640 10 monclient: shutdown 2024-09-17T18:18:49.171 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.168+0000 7fe17b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:18:49.171 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.168+0000 7fe17b59e640 1 -- 172.21.15.173:0/4015886809 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe17c13d5a0 msgr2=0x7fe17c0a1c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:49.171 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.168+0000 7fe17b59e640 1 --2- 172.21.15.173:0/4015886809 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe17c13d5a0 0x7fe17c0a1c00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe17400c780 tx=0x7fe17400cc40 comp rx=0 tx=0).stop 2024-09-17T18:18:49.171 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.168+0000 7fe17b59e640 1 -- 172.21.15.173:0/4015886809 shutdown_connections 2024-09-17T18:18:49.172 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.168+0000 7fe17b59e640 1 --2- 172.21.15.173:0/4015886809 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe17c13d5a0 0x7fe17c0a1c00 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:49.172 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.168+0000 7fe17b59e640 1 --2- 172.21.15.173:0/4015886809 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe17c09e680 0x7fe17c0a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:49.172 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.168+0000 7fe17b59e640 1 --2- 172.21.15.173:0/4015886809 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe17c06ff80 0x7fe17c1cd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:49.172 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.168+0000 7fe17b59e640 1 -- 172.21.15.173:0/4015886809 >> 172.21.15.173:0/4015886809 conn(0x7fe17c0ab7c0 msgr2=0x7fe17c0a9630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:18:49.172 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.168+0000 7fe17b59e640 1 -- 172.21.15.173:0/4015886809 shutdown_connections 2024-09-17T18:18:49.172 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.168+0000 7fe17b59e640 1 -- 172.21.15.173:0/4015886809 wait complete. 2024-09-17T18:18:49.172 INFO:teuthology.orchestra.run.smithi173.stderr:2024-09-17T18:18:49.168+0000 7fe17b59e640 1 librados: shutdown 2024-09-17T18:18:49.180 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-17T18:18:49.180 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-17T18:18:49.183 INFO:tasks.ceph.osd.0:Started 2024-09-17T18:18:49.183 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-17T18:18:49.183 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-17T18:18:49.186 INFO:tasks.ceph.osd.1:Started 2024-09-17T18:18:49.186 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-17T18:18:49.186 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-17T18:18:49.188 INFO:tasks.ceph.osd.2:Started 2024-09-17T18:18:49.188 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-17T18:18:49.188 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-17T18:18:49.190 INFO:tasks.ceph.osd.3:Started 2024-09-17T18:18:49.190 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-17T18:18:49.190 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-17T18:18:49.192 INFO:tasks.ceph.osd.4:Started 2024-09-17T18:18:49.192 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-17T18:18:49.193 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-17T18:18:49.194 INFO:tasks.ceph.osd.5:Started 2024-09-17T18:18:49.195 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T18:18:49.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.304+0000 7f1e7e189640 10 monclient: get_monmap_and_config 2024-09-17T18:18:49.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.304+0000 7f1e7e189640 10 monclient: build_initial_monmap 2024-09-17T18:18:49.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.304+0000 7f1e7e189640 10 monclient: monmap: 2024-09-17T18:18:49.306 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:18:49.306 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:49.306 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:49.304708+0000 2024-09-17T18:18:49.306 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:49.304708+0000 2024-09-17T18:18:49.306 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:18:49.306 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:18:49.306 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:18:49.306 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:18:49.306 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:18:49.307 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:18:49.307 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:18:49.307 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:18:49.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.304+0000 7f1e7e189640 1 Processor -- start 2024-09-17T18:18:49.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.305+0000 7f1e7e189640 1 -- start start 2024-09-17T18:18:49.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.305+0000 7f1e7e189640 10 monclient: init 2024-09-17T18:18:49.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.305+0000 7f1e7e189640 10 monclient: _reopen_session rank -1 2024-09-17T18:18:49.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.305+0000 7f1e7e189640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-17T18:18:49.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.305+0000 7f1e7e189640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f1e780a8620 0x7f1e780a5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:49.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.305+0000 7f1e7e189640 10 monclient(hunting): picked mon.noname-c con 0x7f1e780a8620 addr v2:172.21.15.173:3300/0 2024-09-17T18:18:49.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.305+0000 7f1e7e189640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1e780a6350 0x7f1e780a6730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:49.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.305+0000 7f1e7e189640 10 monclient(hunting): picked mon.noname-a con 0x7f1e780a6350 addr v2:172.21.15.121:3300/0 2024-09-17T18:18:49.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.305+0000 7f1e7e189640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1e7813cbf0 0x7f1e7813efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:49.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.305+0000 7f1e7e189640 10 monclient(hunting): picked mon.noname-b con 0x7f1e7813cbf0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:18:49.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.305+0000 7f1e7e189640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:49.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.305+0000 7f1e7e189640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f1e780a6ca0 con 0x7f1e780a6350 2024-09-17T18:18:49.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.305+0000 7f1e7e189640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:49.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.305+0000 7f1e7e189640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f1e780a8a00 con 0x7f1e780a8620 2024-09-17T18:18:49.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.305+0000 7f1e7e189640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:49.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.305+0000 7f1e7e189640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1e7809f0e0 con 0x7f1e7813cbf0 2024-09-17T18:18:49.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.305+0000 7f1e7e189640 10 monclient(hunting): _renew_subs 2024-09-17T18:18:49.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.305+0000 7f1e7e189640 10 monclient(hunting): authenticate will time out at 733.021569s 2024-09-17T18:18:49.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.305+0000 7f1e77fff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1e7813cbf0 0x7f1e7813efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:49.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.305+0000 7f1e6effd640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1e780a6350 0x7f1e780a6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:49.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.305+0000 7f1e77fff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1e7813cbf0 0x7f1e7813efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:46464/0 (socket says 172.21.15.121:46464) 2024-09-17T18:18:49.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.305+0000 7f1e77fff640 1 -- 172.21.15.121:0/2477110162 learned_addr learned my addr 172.21.15.121:0/2477110162 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:18:49.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.305+0000 7f1e77fff640 10 monclient(hunting): get_auth_request con 0x7f1e7813cbf0 auth_method 0 2024-09-17T18:18:49.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.305+0000 7f1e77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:49.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.305+0000 7f1e77fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:49.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.305+0000 7f1e77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:49.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.305+0000 7f1e77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:49.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.305+0000 7f1e77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:49.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.305+0000 7f1e777fe640 1 --2- 172.21.15.121:0/2477110162 >> v2:172.21.15.173:3300/0 conn(0x7f1e780a8620 0x7f1e780a5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:49.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.305+0000 7f1e77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:18:49.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.305+0000 7f1e777fe640 10 monclient(hunting): get_auth_request con 0x7f1e780a8620 auth_method 0 2024-09-17T18:18:49.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.305+0000 7f1e777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:49.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.305+0000 7f1e777fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:49.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.305+0000 7f1e777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:49.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.305+0000 7f1e6effd640 10 monclient(hunting): get_auth_request con 0x7f1e780a6350 auth_method 0 2024-09-17T18:18:49.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.305+0000 7f1e6effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:49.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.305+0000 7f1e6effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:49.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.305+0000 7f1e6effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:49.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.305+0000 7f1e777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:49.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.305+0000 7f1e777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:49.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.305+0000 7f1e777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:18:49.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.306+0000 7f1e77fff640 10 monclient(hunting): handle_auth_done global_id 4130 payload 274 2024-09-17T18:18:49.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.306+0000 7f1e77fff640 1 -- 172.21.15.121:0/2477110162 >> v2:172.21.15.173:3300/0 conn(0x7f1e780a8620 msgr2=0x7f1e780a5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:18:49.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.306+0000 7f1e77fff640 1 --2- 172.21.15.121:0/2477110162 >> v2:172.21.15.173:3300/0 conn(0x7f1e780a8620 0x7f1e780a5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:49.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.306+0000 7f1e77fff640 1 -- 172.21.15.121:0/2477110162 >> v2:172.21.15.121:3300/0 conn(0x7f1e780a6350 msgr2=0x7f1e780a6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:18:49.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.306+0000 7f1e77fff640 1 --2- 172.21.15.121:0/2477110162 >> v2:172.21.15.121:3300/0 conn(0x7f1e780a6350 0x7f1e780a6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:49.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.306+0000 7f1e77fff640 10 monclient: _finish_hunting 0 2024-09-17T18:18:49.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.306+0000 7f1e77fff640 1 monclient: found mon.noname-b 2024-09-17T18:18:49.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.306+0000 7f1e77fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:18:49.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.306+0000 7f1e77fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:18:49.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.306+0000 7f1e77fff640 1 -- 172.21.15.121:0/2477110162 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e78070900 con 0x7f1e7813cbf0 2024-09-17T18:18:49.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.308+0000 7f1e77fff640 1 -- 172.21.15.121:0/2477110162 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1e7813cbf0 msgr2=0x7f1e7813efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:18:49.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.308+0000 7f1e77fff640 1 --2- 172.21.15.121:0/2477110162 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1e7813cbf0 0x7f1e7813efe0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1e78077c30 tx=0x7f1e7002a610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e81f997544b0d3d server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:18:49.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.309+0000 7f1e76ffd640 1 -- 172.21.15.121:0/2477110162 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e70036070 con 0x7f1e7813cbf0 2024-09-17T18:18:49.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.309+0000 7f1e76ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:49.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.309+0000 7f1e76ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:18:49.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.309+0000 7f1e76ffd640 10 monclient: dump: 2024-09-17T18:18:49.320 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:18:49.320 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:49.320 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:49.320 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:49.320 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:49.320 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:18:49.320 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:49.320 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:49.320 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:49.320 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:18:49.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.309+0000 7f1e76ffd640 10 monclient: _finish_auth 0 2024-09-17T18:18:49.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.309+0000 7f1e76ffd640 10 monclient: _check_auth_tickets 2024-09-17T18:18:49.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.309+0000 7f1e76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:18:49.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.309+0000 7f1e76ffd640 1 -- 172.21.15.121:0/2477110162 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e70007800 con 0x7f1e7813cbf0 2024-09-17T18:18:49.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.309+0000 7f1e7e189640 5 monclient: authenticate success, global_id 4130 2024-09-17T18:18:49.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.309+0000 7f1e7e189640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:18:49.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.309+0000 7f1e76ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:18:49.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.309+0000 7f1e7e189640 10 monclient: get_monmap_and_config success 2024-09-17T18:18:49.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.309+0000 7f1e76ffd640 1 -- 172.21.15.121:0/2477110162 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e70007bd0 con 0x7f1e7813cbf0 2024-09-17T18:18:49.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.309+0000 7f1e76ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:49.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.309+0000 7f1e7e189640 4 set_mon_vals no callback set 2024-09-17T18:18:49.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.309+0000 7f1e76ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:18:49.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.309+0000 7f1e76ffd640 10 monclient: dump: 2024-09-17T18:18:49.321 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:18:49.321 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:49.321 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:49.321 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:49.321 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:49.322 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:18:49.322 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:49.322 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:49.322 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:49.322 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:18:49.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.309+0000 7f1e7e189640 10 monclient: shutdown 2024-09-17T18:18:49.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.309+0000 7f1e7e189640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:18:49.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.309+0000 7f1e7e189640 1 -- 172.21.15.121:0/2477110162 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1e7813cbf0 msgr2=0x7f1e7813efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:49.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.309+0000 7f1e7e189640 1 --2- 172.21.15.121:0/2477110162 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1e7813cbf0 0x7f1e7813efe0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1e78077c30 tx=0x7f1e7002a610 comp rx=0 tx=0).stop 2024-09-17T18:18:49.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.309+0000 7f1e7e189640 1 -- 172.21.15.121:0/2477110162 shutdown_connections 2024-09-17T18:18:49.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.309+0000 7f1e7e189640 1 --2- 172.21.15.121:0/2477110162 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1e7813cbf0 0x7f1e7813efe0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:49.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.309+0000 7f1e7e189640 1 --2- 172.21.15.121:0/2477110162 >> v2:172.21.15.121:3300/0 conn(0x7f1e780a6350 0x7f1e780a6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:49.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.309+0000 7f1e7e189640 1 --2- 172.21.15.121:0/2477110162 >> v2:172.21.15.173:3300/0 conn(0x7f1e780a8620 0x7f1e780a5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:49.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.309+0000 7f1e7e189640 1 -- 172.21.15.121:0/2477110162 >> 172.21.15.121:0/2477110162 conn(0x7f1e780abcb0 msgr2=0x7f1e780a7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:18:49.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.309+0000 7f1e7e189640 1 -- 172.21.15.121:0/2477110162 shutdown_connections 2024-09-17T18:18:49.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.309+0000 7f1e7e189640 1 -- 172.21.15.121:0/2477110162 wait complete. 2024-09-17T18:18:49.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.310+0000 7f1e7e189640 10 monclient: build_initial_monmap 2024-09-17T18:18:49.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.310+0000 7f1e7e189640 10 monclient: monmap: 2024-09-17T18:18:49.323 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:18:49.323 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:18:49.323 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:18:49.323 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:18:49.323 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:18:49.323 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:18:49.323 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:18:49.323 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:18:49.323 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:18:49.323 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:18:49.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.310+0000 7f1e7e189640 1 librados: starting msgr at 2024-09-17T18:18:49.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.310+0000 7f1e7e189640 1 librados: starting objecter 2024-09-17T18:18:49.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.310+0000 7f1e7e189640 1 Processor -- start 2024-09-17T18:18:49.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.310+0000 7f1e7e189640 1 -- start start 2024-09-17T18:18:49.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.310+0000 7f1e7e189640 1 librados: setting wanted keys 2024-09-17T18:18:49.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.310+0000 7f1e7e189640 1 librados: calling monclient init 2024-09-17T18:18:49.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.310+0000 7f1e7e189640 10 monclient: init 2024-09-17T18:18:49.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e7e189640 10 monclient: _reopen_session rank -1 2024-09-17T18:18:49.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e7e189640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T18:18:49.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e7e189640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1e780a6350 0x7f1e78122110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:49.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e7e189640 10 monclient(hunting): picked mon.noname-c con 0x7f1e780a6350 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:18:49.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e7e189640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e780a8620 0x7f1e78122650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:49.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e7e189640 10 monclient(hunting): picked mon.noname-b con 0x7f1e780a8620 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:18:49.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e7e189640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1e7813cbf0 0x7f1e78122b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:49.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e7e189640 10 monclient(hunting): picked mon.noname-a con 0x7f1e7813cbf0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:18:49.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e7e189640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:49.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e7e189640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1e780a7490 con 0x7f1e7813cbf0 2024-09-17T18:18:49.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e7e189640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:49.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e7e189640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1e7809d930 con 0x7f1e780a8620 2024-09-17T18:18:49.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e7e189640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:49.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e7e189640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1e780a7610 con 0x7f1e780a6350 2024-09-17T18:18:49.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e7e189640 10 monclient(hunting): _renew_subs 2024-09-17T18:18:49.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e7e189640 10 monclient(hunting): authenticate will time out at 733.027193s 2024-09-17T18:18:49.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e777fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1e780a6350 0x7f1e78122110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:49.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e777fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1e780a6350 0x7f1e78122110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:46470/0 (socket says 172.21.15.121:46470) 2024-09-17T18:18:49.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e77fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1e7813cbf0 0x7f1e78122b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:49.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e777fe640 1 -- 172.21.15.121:0/3729150715 learned_addr learned my addr 172.21.15.121:0/3729150715 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:18:49.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e777fe640 10 monclient(hunting): get_auth_request con 0x7f1e780a6350 auth_method 0 2024-09-17T18:18:49.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:49.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e777fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:49.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:49.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e77fff640 10 monclient(hunting): get_auth_request con 0x7f1e7813cbf0 auth_method 0 2024-09-17T18:18:49.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:49.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e77fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:49.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:49.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:49.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:49.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:18:49.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:49.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:49.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e6effd640 1 --2- 172.21.15.121:0/3729150715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e780a8620 0x7f1e78122650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:49.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:18:49.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e6effd640 10 monclient(hunting): get_auth_request con 0x7f1e780a8620 auth_method 0 2024-09-17T18:18:49.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e6effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:49.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e6effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:49.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e6effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:49.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e6effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:49.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e6effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:49.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e6effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:18:49.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e777fe640 10 monclient(hunting): handle_auth_done global_id 4133 payload 824 2024-09-17T18:18:49.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e777fe640 1 -- 172.21.15.121:0/3729150715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e780a8620 msgr2=0x7f1e78122650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:49.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e777fe640 1 --2- 172.21.15.121:0/3729150715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e780a8620 0x7f1e78122650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:49.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e777fe640 1 -- 172.21.15.121:0/3729150715 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1e7813cbf0 msgr2=0x7f1e78122b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:49.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e777fe640 1 --2- 172.21.15.121:0/3729150715 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1e7813cbf0 0x7f1e78122b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:49.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e777fe640 10 monclient: _finish_hunting 0 2024-09-17T18:18:49.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e777fe640 1 monclient: found mon.noname-c 2024-09-17T18:18:49.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:18:49.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e777fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:18:49.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e777fe640 1 -- 172.21.15.121:0/3729150715 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e78125f30 con 0x7f1e780a6350 2024-09-17T18:18:49.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.311+0000 7f1e77fff640 1 --2- 172.21.15.121:0/3729150715 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1e7813cbf0 0x7f1e78122b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:18:49.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.313+0000 7f1e777fe640 1 -- 172.21.15.121:0/3729150715 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1e780a6350 msgr2=0x7f1e78122110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:18:49.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.313+0000 7f1e777fe640 1 --2- 172.21.15.121:0/3729150715 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1e780a6350 0x7f1e78122110 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1e6800e9e0 tx=0x7f1e6800eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:18:49.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.313+0000 7f1e74ff9640 10 client.?.objecter ms_handle_connect 0x7f1e780a6350 2024-09-17T18:18:49.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.313+0000 7f1e74ff9640 10 client.?.objecter resend_mon_ops 2024-09-17T18:18:49.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.313+0000 7f1e74ff9640 1 -- 172.21.15.121:0/3729150715 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e68017070 con 0x7f1e780a6350 2024-09-17T18:18:49.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.313+0000 7f1e74ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:49.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.313+0000 7f1e74ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:18:49.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.313+0000 7f1e74ff9640 10 monclient: dump: 2024-09-17T18:18:49.329 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:18:49.329 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:49.329 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:49.329 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:49.329 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:49.329 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:18:49.329 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:49.329 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:49.329 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:49.329 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:18:49.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.313+0000 7f1e74ff9640 10 monclient: _finish_auth 0 2024-09-17T18:18:49.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.314+0000 7f1e74ff9640 10 monclient: _check_auth_tickets 2024-09-17T18:18:49.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.314+0000 7f1e74ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:18:49.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.314+0000 7f1e74ff9640 1 -- 172.21.15.121:0/3729150715 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e6800cbe0 con 0x7f1e780a6350 2024-09-17T18:18:49.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.314+0000 7f1e74ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T18:18:49.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.314+0000 7f1e74ff9640 1 -- 172.21.15.121:0/3729150715 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e68013470 con 0x7f1e780a6350 2024-09-17T18:18:49.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.314+0000 7f1e74ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:49.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.314+0000 7f1e74ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:18:49.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.314+0000 7f1e74ff9640 10 monclient: dump: 2024-09-17T18:18:49.330 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:18:49.330 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:49.330 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:49.330 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:49.330 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:49.330 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:18:49.330 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:49.331 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:49.331 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:49.331 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:18:49.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.314+0000 7f1e7e189640 5 monclient: authenticate success, global_id 4133 2024-09-17T18:18:49.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.314+0000 7f1e7e189640 10 monclient: _renew_subs 2024-09-17T18:18:49.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.314+0000 7f1e767fc640 4 set_mon_vals no callback set 2024-09-17T18:18:49.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.314+0000 7f1e7e189640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:18:49.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.314+0000 7f1e7e189640 1 -- 172.21.15.121:0/3729150715 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e780a2910 con 0x7f1e780a6350 2024-09-17T18:18:49.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.314+0000 7f1e7e189640 10 client.4133.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:18:49.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.314+0000 7f1e7e189640 10 monclient: _renew_subs 2024-09-17T18:18:49.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.314+0000 7f1e7e189640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:18:49.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.314+0000 7f1e7e189640 1 -- 172.21.15.121:0/3729150715 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e78123bd0 con 0x7f1e780a6350 2024-09-17T18:18:49.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.314+0000 7f1e7e189640 1 librados: init done 2024-09-17T18:18:49.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.314+0000 7f1e74ff9640 1 -- 172.21.15.121:0/3729150715 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1e6801a070 con 0x7f1e780a6350 2024-09-17T18:18:49.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.314+0000 7f1e74ff9640 1 -- 172.21.15.121:0/3729150715 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f1e68013ca0 con 0x7f1e780a6350 2024-09-17T18:18:49.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.314+0000 7f1e74ff9640 10 client.4133.objecter ms_dispatch 0x7f1e78000b90 osd_map(8..8 src has 1..8) 2024-09-17T18:18:49.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.314+0000 7f1e74ff9640 3 client.4133.objecter handle_osd_map got epochs [8,8] > 0 2024-09-17T18:18:49.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.314+0000 7f1e74ff9640 3 client.4133.objecter handle_osd_map decoding full epoch 8 2024-09-17T18:18:49.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.314+0000 7f1e74ff9640 20 client.4133.objecter dump_active .. 0 homeless 2024-09-17T18:18:49.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.314+0000 7f1e7e189640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:18:49.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.315+0000 7f1e7e189640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:18:49.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.315+0000 7f1e7e189640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:18:49.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.315+0000 7f1e7e189640 1 -- 172.21.15.121:0/3729150715 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e780a6730 con 0x7f1e780a6350 2024-09-17T18:18:49.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.317+0000 7f1e74ff9640 1 -- 172.21.15.121:0/3729150715 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f1e68013610 con 0x7f1e780a6350 2024-09-17T18:18:49.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.317+0000 7f1e74ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:18:49.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.317+0000 7f1e74ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:18:49.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.374+0000 7f1e7e189640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-17T18:18:49.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.374+0000 7f1e7e189640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T18:18:49.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.374+0000 7f1e7e189640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:18:49.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.374+0000 7f1e7e189640 1 -- 172.21.15.121:0/3729150715 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1e7806cd40 con 0x7f1e780a6350 2024-09-17T18:18:49.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.375+0000 7f1e74ff9640 1 -- 172.21.15.121:0/3729150715 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (secure 0 0 0) 0x7f1e680137f0 con 0x7f1e780a6350 2024-09-17T18:18:49.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.375+0000 7f1e74ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T18:18:49.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.375+0000 7f1e74ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:18:49.377 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:18:49.377 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":8,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","created":"2024-09-17T18:18:47.237753+0000","modified":"2024-09-17T18:18:49.165275+0000","last_up_change":"0.000000","last_in_change":"2024-09-17T18:18:49.165275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bf586d1-05ac-4c7e-8b88-313f07a76801","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e26c6163-1eda-4200-82e1-57b752fd8a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6fb45798-21d4-449e-9f68-7210d984e036","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3de6630d-4fb7-4f64-b4f9-e5c304b1be63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cfa74383-b7c9-4ecd-8258-3792fbccc237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8bdb7768-a603-4f14-9650-47b503621215","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T18:18:49.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.375+0000 7f1e7e189640 10 librados: watch_flush enter 2024-09-17T18:18:49.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.375+0000 7f1e7e189640 10 librados: watch_flush exit 2024-09-17T18:18:49.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.375+0000 7f1e7e189640 20 client.4133.objecter shutdown clearing up homeless session... 2024-09-17T18:18:49.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.375+0000 7f1e7e189640 10 client.4133.objecter successfully canceled tick 2024-09-17T18:18:49.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.375+0000 7f1e7e189640 10 monclient: shutdown 2024-09-17T18:18:49.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.375+0000 7f1e7e189640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:18:49.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.375+0000 7f1e7e189640 1 -- 172.21.15.121:0/3729150715 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1e780a6350 msgr2=0x7f1e78122110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:49.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.375+0000 7f1e7e189640 1 --2- 172.21.15.121:0/3729150715 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1e780a6350 0x7f1e78122110 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1e6800e9e0 tx=0x7f1e6800eea0 comp rx=0 tx=0).stop 2024-09-17T18:18:49.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.376+0000 7f1e7e189640 1 -- 172.21.15.121:0/3729150715 shutdown_connections 2024-09-17T18:18:49.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.376+0000 7f1e7e189640 1 --2- 172.21.15.121:0/3729150715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e780a8620 0x7f1e78122650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:49.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.376+0000 7f1e7e189640 1 --2- 172.21.15.121:0/3729150715 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1e7813cbf0 0x7f1e78122b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:49.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.376+0000 7f1e7e189640 1 --2- 172.21.15.121:0/3729150715 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1e780a6350 0x7f1e78122110 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:49.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.376+0000 7f1e7e189640 1 -- 172.21.15.121:0/3729150715 >> 172.21.15.121:0/3729150715 conn(0x7f1e780abcb0 msgr2=0x7f1e780a31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:18:49.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.376+0000 7f1e7e189640 1 -- 172.21.15.121:0/3729150715 shutdown_connections 2024-09-17T18:18:49.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.376+0000 7f1e7e189640 1 -- 172.21.15.121:0/3729150715 wait complete. 2024-09-17T18:18:49.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:49.376+0000 7f1e7e189640 1 librados: shutdown 2024-09-17T18:18:49.387 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-17T18:18:49.388 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-17T18:18:49.991 DEBUG:teuthology.orchestra.run.smithi121:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-17T18:18:50.080 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.078+0000 7f19de00f640 10 monclient: get_monmap_and_config 2024-09-17T18:18:50.080 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.078+0000 7f19de00f640 10 monclient: build_initial_monmap 2024-09-17T18:18:50.080 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.078+0000 7f19de00f640 10 monclient: monmap: 2024-09-17T18:18:50.080 INFO:teuthology.misc.health.smithi121.stderr:epoch 0 2024-09-17T18:18:50.081 INFO:teuthology.misc.health.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:50.081 INFO:teuthology.misc.health.smithi121.stderr:last_changed 2024-09-17T18:18:50.079290+0000 2024-09-17T18:18:50.081 INFO:teuthology.misc.health.smithi121.stderr:created 2024-09-17T18:18:50.079290+0000 2024-09-17T18:18:50.081 INFO:teuthology.misc.health.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:18:50.081 INFO:teuthology.misc.health.smithi121.stderr:election_strategy: 1 2024-09-17T18:18:50.081 INFO:teuthology.misc.health.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:18:50.081 INFO:teuthology.misc.health.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:18:50.081 INFO:teuthology.misc.health.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:18:50.081 INFO:teuthology.misc.health.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:18:50.081 INFO:teuthology.misc.health.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:18:50.081 INFO:teuthology.misc.health.smithi121.stderr: 2024-09-17T18:18:50.084 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.079+0000 7f19de00f640 1 Processor -- start 2024-09-17T18:18:50.084 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.079+0000 7f19de00f640 1 -- start start 2024-09-17T18:18:50.084 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.079+0000 7f19de00f640 10 monclient: init 2024-09-17T18:18:50.084 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.079+0000 7f19de00f640 10 monclient: _reopen_session rank -1 2024-09-17T18:18:50.084 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.079+0000 7f19de00f640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-17T18:18:50.084 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.079+0000 7f19de00f640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f19d80735e0 0x7f19d80739c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:50.084 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.079+0000 7f19de00f640 10 monclient(hunting): picked mon.noname-a con 0x7f19d80735e0 addr v2:172.21.15.121:3300/0 2024-09-17T18:18:50.084 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.079+0000 7f19de00f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f19d8074270 addr v1:172.21.15.121:6789/0 2024-09-17T18:18:50.084 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.079+0000 7f19de00f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f19d80a1570 0x7f19d8140770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:50.084 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.079+0000 7f19de00f640 10 monclient(hunting): picked mon.noname-b con 0x7f19d80a1570 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:18:50.084 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.079+0000 7f19de00f640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19d806fe60 con 0x7f19d8074270 2024-09-17T18:18:50.084 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.079+0000 7f19de00f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:50.084 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.079+0000 7f19de00f640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f19d80a6390 con 0x7f19d80735e0 2024-09-17T18:18:50.085 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.079+0000 7f19de00f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:50.085 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.079+0000 7f19de00f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f19d80a7330 con 0x7f19d80a1570 2024-09-17T18:18:50.085 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.079+0000 7f19de00f640 10 monclient(hunting): _renew_subs 2024-09-17T18:18:50.085 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.079+0000 7f19de00f640 10 monclient(hunting): authenticate will time out at 733.795983s 2024-09-17T18:18:50.085 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.080+0000 7f19d77fe640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f19d80735e0 0x7f19d80739c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:50.085 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.080+0000 7f19d7fff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f19d80a1570 0x7f19d8140770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:50.085 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.080+0000 7f19d77fe640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f19d80735e0 0x7f19d80739c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39776/0 (socket says 172.21.15.121:39776) 2024-09-17T18:18:50.085 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.080+0000 7f19d77fe640 1 -- 172.21.15.121:0/2508911700 learned_addr learned my addr 172.21.15.121:0/2508911700 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:18:50.085 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.080+0000 7f19d77fe640 10 monclient(hunting): get_auth_request con 0x7f19d80735e0 auth_method 0 2024-09-17T18:18:50.085 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.080+0000 7f19d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:50.085 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.080+0000 7f19d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:50.085 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.080+0000 7f19d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:50.086 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.080+0000 7f19d7fff640 10 monclient(hunting): get_auth_request con 0x7f19d80a1570 auth_method 0 2024-09-17T18:18:50.086 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.080+0000 7f19d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:50.086 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.080+0000 7f19d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:50.086 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.080+0000 7f19d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:50.086 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.080+0000 7f19d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:50.086 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.080+0000 7f19d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:50.086 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.080+0000 7f19d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:18:50.086 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.080+0000 7f19d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:50.086 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.080+0000 7f19d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:50.086 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.080+0000 7f19d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:18:50.086 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.080+0000 7f19d7fff640 10 monclient(hunting): handle_auth_done global_id 4136 payload 274 2024-09-17T18:18:50.086 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.080+0000 7f19d7fff640 1 -- 172.21.15.121:0/2508911700 >> v2:172.21.15.121:3300/0 conn(0x7f19d80735e0 msgr2=0x7f19d80739c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:18:50.086 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.080+0000 7f19d7fff640 1 --2- 172.21.15.121:0/2508911700 >> v2:172.21.15.121:3300/0 conn(0x7f19d80735e0 0x7f19d80739c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:50.087 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.080+0000 7f19d7fff640 1 -- 172.21.15.121:0/2508911700 >> v1:172.21.15.121:6789/0 conn(0x7f19d8074270 legacy=0x7f19d80a0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:18:50.087 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.080+0000 7f19d7fff640 10 monclient: _finish_hunting 0 2024-09-17T18:18:50.087 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.080+0000 7f19d7fff640 1 monclient: found mon.noname-b 2024-09-17T18:18:50.087 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.080+0000 7f19d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:18:50.087 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.080+0000 7f19d7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:18:50.087 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.080+0000 7f19d7fff640 1 -- 172.21.15.121:0/2508911700 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19d806fb80 con 0x7f19d80a1570 2024-09-17T18:18:50.093 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.083+0000 7f19d7fff640 1 -- 172.21.15.121:0/2508911700 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f19d80a1570 msgr2=0x7f19d8140770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:18:50.093 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.083+0000 7f19d7fff640 1 --2- 172.21.15.121:0/2508911700 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f19d80a1570 0x7f19d8140770 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f19d00097a0 tx=0x7f19d0029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38ecad8f913b57ab server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:18:50.093 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.083+0000 7f19d67fc640 1 -- 172.21.15.121:0/2508911700 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d0037070 con 0x7f19d80a1570 2024-09-17T18:18:50.093 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.083+0000 7f19d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:50.093 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.083+0000 7f19d67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:18:50.093 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.083+0000 7f19d67fc640 10 monclient: dump: 2024-09-17T18:18:50.093 INFO:teuthology.misc.health.smithi121.stderr:epoch 1 2024-09-17T18:18:50.094 INFO:teuthology.misc.health.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:50.094 INFO:teuthology.misc.health.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:50.094 INFO:teuthology.misc.health.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:50.094 INFO:teuthology.misc.health.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:50.094 INFO:teuthology.misc.health.smithi121.stderr:election_strategy: 1 2024-09-17T18:18:50.094 INFO:teuthology.misc.health.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:50.094 INFO:teuthology.misc.health.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:50.094 INFO:teuthology.misc.health.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:50.095 INFO:teuthology.misc.health.smithi121.stderr: 2024-09-17T18:18:50.095 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.083+0000 7f19d67fc640 10 monclient: _finish_auth 0 2024-09-17T18:18:50.095 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.083+0000 7f19d67fc640 10 monclient: _check_auth_tickets 2024-09-17T18:18:50.095 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.083+0000 7f19d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:18:50.095 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.083+0000 7f19d67fc640 1 -- 172.21.15.121:0/2508911700 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19d0032440 con 0x7f19d80a1570 2024-09-17T18:18:50.095 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.083+0000 7f19de00f640 5 monclient: authenticate success, global_id 4136 2024-09-17T18:18:50.095 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.083+0000 7f19de00f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:18:50.095 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.083+0000 7f19d67fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:18:50.095 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.083+0000 7f19de00f640 10 monclient: get_monmap_and_config success 2024-09-17T18:18:50.095 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.083+0000 7f19d67fc640 1 -- 172.21.15.121:0/2508911700 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d0032810 con 0x7f19d80a1570 2024-09-17T18:18:50.095 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.083+0000 7f19d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:50.095 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.083+0000 7f19de00f640 4 set_mon_vals no callback set 2024-09-17T18:18:50.095 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.083+0000 7f19d67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:18:50.095 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.083+0000 7f19d67fc640 10 monclient: dump: 2024-09-17T18:18:50.096 INFO:teuthology.misc.health.smithi121.stderr:epoch 1 2024-09-17T18:18:50.096 INFO:teuthology.misc.health.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:50.096 INFO:teuthology.misc.health.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:50.096 INFO:teuthology.misc.health.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:50.096 INFO:teuthology.misc.health.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:50.096 INFO:teuthology.misc.health.smithi121.stderr:election_strategy: 1 2024-09-17T18:18:50.096 INFO:teuthology.misc.health.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:50.096 INFO:teuthology.misc.health.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:50.096 INFO:teuthology.misc.health.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:50.096 INFO:teuthology.misc.health.smithi121.stderr: 2024-09-17T18:18:50.096 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.083+0000 7f19de00f640 10 monclient: shutdown 2024-09-17T18:18:50.096 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.083+0000 7f19de00f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:18:50.096 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.083+0000 7f19de00f640 1 -- 172.21.15.121:0/2508911700 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f19d80a1570 msgr2=0x7f19d8140770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:50.096 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.083+0000 7f19de00f640 1 --2- 172.21.15.121:0/2508911700 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f19d80a1570 0x7f19d8140770 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f19d00097a0 tx=0x7f19d0029fd0 comp rx=0 tx=0).stop 2024-09-17T18:18:50.097 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.083+0000 7f19de00f640 1 -- 172.21.15.121:0/2508911700 shutdown_connections 2024-09-17T18:18:50.097 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.083+0000 7f19de00f640 1 --2- 172.21.15.121:0/2508911700 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f19d80a1570 0x7f19d8140770 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:50.097 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.083+0000 7f19de00f640 1 --2- 172.21.15.121:0/2508911700 >> v2:172.21.15.121:3300/0 conn(0x7f19d80735e0 0x7f19d80739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:50.097 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.083+0000 7f19de00f640 1 -- 172.21.15.121:0/2508911700 >> 172.21.15.121:0/2508911700 conn(0x7f19d80a7ce0 msgr2=0x7f19d80a80f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:18:50.097 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.083+0000 7f19de00f640 1 -- 172.21.15.121:0/2508911700 shutdown_connections 2024-09-17T18:18:50.097 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.083+0000 7f19de00f640 1 -- 172.21.15.121:0/2508911700 wait complete. 2024-09-17T18:18:50.097 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.084+0000 7f19de00f640 10 monclient: build_initial_monmap 2024-09-17T18:18:50.097 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.084+0000 7f19de00f640 10 monclient: monmap: 2024-09-17T18:18:50.097 INFO:teuthology.misc.health.smithi121.stderr:epoch 0 2024-09-17T18:18:50.097 INFO:teuthology.misc.health.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:18:50.097 INFO:teuthology.misc.health.smithi121.stderr:last_changed 0.000000 2024-09-17T18:18:50.097 INFO:teuthology.misc.health.smithi121.stderr:created 0.000000 2024-09-17T18:18:50.097 INFO:teuthology.misc.health.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:18:50.098 INFO:teuthology.misc.health.smithi121.stderr:election_strategy: 1 2024-09-17T18:18:50.098 INFO:teuthology.misc.health.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:18:50.098 INFO:teuthology.misc.health.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:18:50.098 INFO:teuthology.misc.health.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:18:50.098 INFO:teuthology.misc.health.smithi121.stderr: 2024-09-17T18:18:50.098 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.084+0000 7f19de00f640 1 librados: starting msgr at 2024-09-17T18:18:50.098 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.084+0000 7f19de00f640 1 librados: starting objecter 2024-09-17T18:18:50.098 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.084+0000 7f19de00f640 1 Processor -- start 2024-09-17T18:18:50.098 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.084+0000 7f19de00f640 1 -- start start 2024-09-17T18:18:50.098 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.084+0000 7f19de00f640 1 librados: setting wanted keys 2024-09-17T18:18:50.098 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19de00f640 1 librados: calling monclient init 2024-09-17T18:18:50.098 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19de00f640 10 monclient: init 2024-09-17T18:18:50.098 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19de00f640 10 monclient: _reopen_session rank -1 2024-09-17T18:18:50.099 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19de00f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:18:50.099 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19de00f640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19d80735e0 0x7f19d81cb7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:50.099 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19de00f640 10 monclient(hunting): picked mon.noname-a con 0x7f19d80735e0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:18:50.099 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19de00f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19d8074270 0x7f19d8135090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:50.099 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19de00f640 10 monclient(hunting): picked mon.noname-b con 0x7f19d8074270 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:18:50.099 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19de00f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f19d80a1570 0x7f19d81355d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:50.099 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19de00f640 10 monclient(hunting): picked mon.noname-c con 0x7f19d80a1570 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:18:50.099 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19de00f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:50.099 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19de00f640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f19d8074080 con 0x7f19d80735e0 2024-09-17T18:18:50.099 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19de00f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:50.099 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19de00f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f19d8073f00 con 0x7f19d8074270 2024-09-17T18:18:50.099 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19de00f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:50.099 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19de00f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f19d81388f0 con 0x7f19d80a1570 2024-09-17T18:18:50.099 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19de00f640 10 monclient(hunting): _renew_subs 2024-09-17T18:18:50.100 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19de00f640 10 monclient(hunting): authenticate will time out at 733.801443s 2024-09-17T18:18:50.100 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19d77fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19d80735e0 0x7f19d81cb7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:50.100 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19d7fff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f19d80a1570 0x7f19d81355d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:50.100 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19d77fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19d80735e0 0x7f19d81cb7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39792/0 (socket says 172.21.15.121:39792) 2024-09-17T18:18:50.100 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19d77fe640 1 -- 172.21.15.121:0/3134530417 learned_addr learned my addr 172.21.15.121:0/3134530417 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:18:50.100 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19d77fe640 10 monclient(hunting): get_auth_request con 0x7f19d80735e0 auth_method 0 2024-09-17T18:18:50.100 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:50.100 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:50.100 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:50.100 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19d7fff640 10 monclient(hunting): get_auth_request con 0x7f19d80a1570 auth_method 0 2024-09-17T18:18:50.100 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:50.100 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:50.100 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:50.101 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:50.101 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:50.101 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:18:50.101 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:50.101 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:50.101 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:18:50.101 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19d6ffd640 1 --2- 172.21.15.121:0/3134530417 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19d8074270 0x7f19d8135090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:50.101 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19d6ffd640 10 monclient(hunting): get_auth_request con 0x7f19d8074270 auth_method 0 2024-09-17T18:18:50.101 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:50.101 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:50.101 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:50.101 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19d77fe640 10 monclient(hunting): handle_auth_done global_id 4152 payload 824 2024-09-17T18:18:50.101 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19d77fe640 1 -- 172.21.15.121:0/3134530417 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f19d80a1570 msgr2=0x7f19d81355d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:50.102 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19d77fe640 1 --2- 172.21.15.121:0/3134530417 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f19d80a1570 0x7f19d81355d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:50.102 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19d77fe640 1 -- 172.21.15.121:0/3134530417 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19d8074270 msgr2=0x7f19d8135090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:50.102 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19d77fe640 1 --2- 172.21.15.121:0/3134530417 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19d8074270 0x7f19d8135090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:50.102 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19d77fe640 10 monclient: _finish_hunting 0 2024-09-17T18:18:50.102 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19d77fe640 1 monclient: found mon.noname-a 2024-09-17T18:18:50.102 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:18:50.102 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19d77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:18:50.102 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.085+0000 7f19d77fe640 1 -- 172.21.15.121:0/3134530417 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19d8074c70 con 0x7f19d80735e0 2024-09-17T18:18:50.102 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.088+0000 7f19d77fe640 1 -- 172.21.15.121:0/3134530417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19d80735e0 msgr2=0x7f19d81cb7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:18:50.102 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.088+0000 7f19d77fe640 1 --2- 172.21.15.121:0/3134530417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19d80735e0 0x7f19d81cb7f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f19c800eed0 tx=0x7f19c800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:18:50.102 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.088+0000 7f19bffff640 10 client.?.objecter ms_handle_connect 0x7f19d80735e0 2024-09-17T18:18:50.102 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.088+0000 7f19bffff640 10 client.?.objecter resend_mon_ops 2024-09-17T18:18:50.102 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.088+0000 7f19bffff640 1 -- 172.21.15.121:0/3134530417 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19c8009e60 con 0x7f19d80735e0 2024-09-17T18:18:50.102 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.088+0000 7f19bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:50.103 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.088+0000 7f19bffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:18:50.103 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.088+0000 7f19bffff640 10 monclient: dump: 2024-09-17T18:18:50.103 INFO:teuthology.misc.health.smithi121.stderr:epoch 1 2024-09-17T18:18:50.103 INFO:teuthology.misc.health.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:50.103 INFO:teuthology.misc.health.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:50.103 INFO:teuthology.misc.health.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:50.103 INFO:teuthology.misc.health.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:50.103 INFO:teuthology.misc.health.smithi121.stderr:election_strategy: 1 2024-09-17T18:18:50.103 INFO:teuthology.misc.health.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:50.103 INFO:teuthology.misc.health.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:50.103 INFO:teuthology.misc.health.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:50.103 INFO:teuthology.misc.health.smithi121.stderr: 2024-09-17T18:18:50.103 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.088+0000 7f19bffff640 10 monclient: _finish_auth 0 2024-09-17T18:18:50.103 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.088+0000 7f19bffff640 10 monclient: _check_auth_tickets 2024-09-17T18:18:50.104 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.088+0000 7f19bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:18:50.104 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.088+0000 7f19bffff640 1 -- 172.21.15.121:0/3134530417 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19c8019070 con 0x7f19d80735e0 2024-09-17T18:18:50.104 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.088+0000 7f19de00f640 5 monclient: authenticate success, global_id 4152 2024-09-17T18:18:50.104 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.088+0000 7f19de00f640 10 monclient: _renew_subs 2024-09-17T18:18:50.104 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.088+0000 7f19de00f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:18:50.104 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.088+0000 7f19de00f640 1 -- 172.21.15.121:0/3134530417 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19d8135b10 con 0x7f19d80735e0 2024-09-17T18:18:50.104 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.088+0000 7f19bffff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:18:50.104 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.088+0000 7f19bffff640 1 -- 172.21.15.121:0/3134530417 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19c8014600 con 0x7f19d80735e0 2024-09-17T18:18:50.104 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.088+0000 7f19bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:50.104 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.088+0000 7f19bffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:18:50.104 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.088+0000 7f19bffff640 10 monclient: dump: 2024-09-17T18:18:50.104 INFO:teuthology.misc.health.smithi121.stderr:epoch 1 2024-09-17T18:18:50.104 INFO:teuthology.misc.health.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:50.104 INFO:teuthology.misc.health.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:50.105 INFO:teuthology.misc.health.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:50.105 INFO:teuthology.misc.health.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:50.105 INFO:teuthology.misc.health.smithi121.stderr:election_strategy: 1 2024-09-17T18:18:50.105 INFO:teuthology.misc.health.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:50.105 INFO:teuthology.misc.health.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:50.105 INFO:teuthology.misc.health.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:50.105 INFO:teuthology.misc.health.smithi121.stderr: 2024-09-17T18:18:50.105 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.088+0000 7f19d5ffb640 4 set_mon_vals no callback set 2024-09-17T18:18:50.105 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.088+0000 7f19de00f640 10 client.4152.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:18:50.105 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.088+0000 7f19de00f640 10 monclient: _renew_subs 2024-09-17T18:18:50.105 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.088+0000 7f19de00f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:18:50.105 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.088+0000 7f19de00f640 1 -- 172.21.15.121:0/3134530417 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19d8122b50 con 0x7f19d80735e0 2024-09-17T18:18:50.105 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.088+0000 7f19de00f640 1 librados: init done 2024-09-17T18:18:50.105 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.088+0000 7f19bffff640 1 -- 172.21.15.121:0/3134530417 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f19c8005d10 con 0x7f19d80735e0 2024-09-17T18:18:50.106 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.088+0000 7f19bffff640 1 -- 172.21.15.121:0/3134530417 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f19c80147a0 con 0x7f19d80735e0 2024-09-17T18:18:50.106 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.088+0000 7f19bffff640 10 client.4152.objecter ms_dispatch 0x7f19d8000b90 osd_map(8..8 src has 1..8) 2024-09-17T18:18:50.106 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.088+0000 7f19bffff640 3 client.4152.objecter handle_osd_map got epochs [8,8] > 0 2024-09-17T18:18:50.106 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.088+0000 7f19bffff640 3 client.4152.objecter handle_osd_map decoding full epoch 8 2024-09-17T18:18:50.106 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.088+0000 7f19bffff640 20 client.4152.objecter dump_active .. 0 homeless 2024-09-17T18:18:50.106 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.089+0000 7f19de00f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:18:50.106 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.089+0000 7f19de00f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:18:50.106 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.089+0000 7f19de00f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:18:50.106 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.089+0000 7f19de00f640 1 -- 172.21.15.121:0/3134530417 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19d80a6540 con 0x7f19d80735e0 2024-09-17T18:18:50.106 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.091+0000 7f19d77fe640 1 --2- 172.21.15.121:0/3134530417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19d80735e0 0x7f19d81cb7f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f19c800eed0 tx=0x7f19c800b480 comp rx=0 tx=0).queue_received will delay after 1.7266e+09 on 0x7f19d80a6540 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) 2024-09-17T18:18:50.387 INFO:tasks.ceph.osd.0.smithi121.stderr:2024-09-17T18:18:50.385+0000 7f32ac433640 -1 Falling back to public interface 2024-09-17T18:18:50.400 INFO:tasks.ceph.osd.3.smithi173.stderr:2024-09-17T18:18:50.398+0000 7f299740c640 -1 Falling back to public interface 2024-09-17T18:18:50.656 INFO:tasks.ceph.osd.4.smithi173.stderr:2024-09-17T18:18:50.654+0000 7fbfd2014640 -1 Falling back to public interface 2024-09-17T18:18:50.658 INFO:tasks.ceph.osd.2.smithi121.stderr:2024-09-17T18:18:50.656+0000 7f1e464b0640 -1 Falling back to public interface 2024-09-17T18:18:50.660 INFO:tasks.ceph.osd.5.smithi173.stderr:2024-09-17T18:18:50.658+0000 7ffbe9743640 -1 Falling back to public interface 2024-09-17T18:18:50.884 INFO:tasks.ceph.osd.1.smithi121.stderr:2024-09-17T18:18:50.882+0000 7f1f194ed640 -1 Falling back to public interface 2024-09-17T18:18:50.921 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.919+0000 7f19bffff640 1 -- 172.21.15.121:0/3134530417 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f19d80a6540 con 0x7f19d80735e0 2024-09-17T18:18:50.921 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.919+0000 7f19bffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:18:50.921 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.919+0000 7f19bffff640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:18:50.979 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.977+0000 7f19de00f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-17T18:18:50.979 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.977+0000 7f19de00f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T18:18:50.979 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.977+0000 7f19de00f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:18:50.979 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.977+0000 7f19de00f640 1 -- 172.21.15.121:0/3134530417 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f19d80a1a20 con 0x7f19d80735e0 2024-09-17T18:18:50.980 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.978+0000 7f19bffff640 1 -- 172.21.15.121:0/3134530417 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (secure 0 0 0) 0x7f19c80149f0 con 0x7f19d80735e0 2024-09-17T18:18:50.980 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.978+0000 7f19bffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T18:18:50.980 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.978+0000 7f19bffff640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:18:50.980 INFO:teuthology.misc.health.smithi121.stdout: 2024-09-17T18:18:50.980 INFO:teuthology.misc.health.smithi121.stdout:{"epoch":8,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","created":"2024-09-17T18:18:47.237753+0000","modified":"2024-09-17T18:18:49.165275+0000","last_up_change":"0.000000","last_in_change":"2024-09-17T18:18:49.165275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bf586d1-05ac-4c7e-8b88-313f07a76801","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e26c6163-1eda-4200-82e1-57b752fd8a46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6fb45798-21d4-449e-9f68-7210d984e036","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3de6630d-4fb7-4f64-b4f9-e5c304b1be63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cfa74383-b7c9-4ecd-8258-3792fbccc237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8bdb7768-a603-4f14-9650-47b503621215","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T18:18:50.981 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.978+0000 7f19de00f640 10 librados: watch_flush enter 2024-09-17T18:18:50.981 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.978+0000 7f19de00f640 10 librados: watch_flush exit 2024-09-17T18:18:50.981 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.978+0000 7f19de00f640 20 client.4152.objecter shutdown clearing up homeless session... 2024-09-17T18:18:50.982 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.978+0000 7f19de00f640 10 client.4152.objecter successfully canceled tick 2024-09-17T18:18:50.982 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.978+0000 7f19de00f640 10 monclient: shutdown 2024-09-17T18:18:50.982 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.978+0000 7f19de00f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:18:50.982 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.978+0000 7f19de00f640 1 -- 172.21.15.121:0/3134530417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19d80735e0 msgr2=0x7f19d81cb7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:50.982 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.978+0000 7f19de00f640 1 --2- 172.21.15.121:0/3134530417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19d80735e0 0x7f19d81cb7f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f19c800eed0 tx=0x7f19c800b480 comp rx=0 tx=0).stop 2024-09-17T18:18:50.982 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.978+0000 7f19de00f640 1 -- 172.21.15.121:0/3134530417 shutdown_connections 2024-09-17T18:18:50.982 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.978+0000 7f19de00f640 1 --2- 172.21.15.121:0/3134530417 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19d8074270 0x7f19d8135090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:50.982 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.978+0000 7f19de00f640 1 --2- 172.21.15.121:0/3134530417 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f19d80a1570 0x7f19d81355d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:50.982 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.978+0000 7f19de00f640 1 --2- 172.21.15.121:0/3134530417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19d80735e0 0x7f19d81cb7f0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:50.982 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.978+0000 7f19de00f640 1 -- 172.21.15.121:0/3134530417 >> 172.21.15.121:0/3134530417 conn(0x7f19d80a7ce0 msgr2=0x7f19d80a4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:18:50.982 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.979+0000 7f19de00f640 1 -- 172.21.15.121:0/3134530417 shutdown_connections 2024-09-17T18:18:50.982 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.979+0000 7f19de00f640 1 -- 172.21.15.121:0/3134530417 wait complete. 2024-09-17T18:18:50.982 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:50.979+0000 7f19de00f640 1 librados: shutdown 2024-09-17T18:18:50.989 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-17T18:18:52.507 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-09-17T18:18:52.504+0000 7f091308e640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-17T18:18:52.510 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-09-17T18:18:52.508+0000 7f091308e640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-17T18:18:54.190 INFO:tasks.ceph.osd.0.smithi121.stderr:2024-09-17T18:18:54.188+0000 7f32ac433640 -1 osd.0 0 log_to_monitors true 2024-09-17T18:18:54.431 INFO:tasks.ceph.osd.5.smithi173.stderr:2024-09-17T18:18:54.428+0000 7ffbe9743640 -1 osd.5 0 log_to_monitors true 2024-09-17T18:18:54.461 INFO:tasks.ceph.osd.2.smithi121.stderr:2024-09-17T18:18:54.460+0000 7f1e464b0640 -1 osd.2 0 log_to_monitors true 2024-09-17T18:18:54.488 INFO:tasks.ceph.osd.3.smithi173.stderr:2024-09-17T18:18:54.485+0000 7f299740c640 -1 osd.3 0 log_to_monitors true 2024-09-17T18:18:54.954 INFO:tasks.ceph.osd.4.smithi173.stderr:2024-09-17T18:18:54.951+0000 7fbfd2014640 -1 osd.4 0 log_to_monitors true 2024-09-17T18:18:55.118 INFO:tasks.ceph.osd.1.smithi121.stderr:2024-09-17T18:18:55.117+0000 7f1f194ed640 -1 osd.1 0 log_to_monitors true 2024-09-17T18:18:55.507 INFO:tasks.ceph.osd.5.smithi173.stderr:2024-09-17T18:18:55.504+0000 7ffbe56d2640 -1 osd.5 0 waiting for initial osdmap 2024-09-17T18:18:55.510 INFO:tasks.ceph.osd.5.smithi173.stderr:2024-09-17T18:18:55.508+0000 7ffbe14f9640 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T18:18:56.095 INFO:tasks.ceph.osd.3.smithi173.stderr:2024-09-17T18:18:56.092+0000 7f29934dd640 -1 osd.3 0 waiting for initial osdmap 2024-09-17T18:18:56.097 INFO:tasks.ceph.osd.3.smithi173.stderr:2024-09-17T18:18:56.094+0000 7f298aae8640 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T18:18:56.197 INFO:tasks.ceph.osd.0.smithi121.stderr:2024-09-17T18:18:56.195+0000 7f32a83c2640 -1 osd.0 0 waiting for initial osdmap 2024-09-17T18:18:56.201 INFO:tasks.ceph.osd.0.smithi121.stderr:2024-09-17T18:18:56.199+0000 7f32a01e1640 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T18:18:56.244 INFO:tasks.ceph.osd.2.smithi121.stderr:2024-09-17T18:18:56.242+0000 7f1e42c53640 -1 osd.2 0 waiting for initial osdmap 2024-09-17T18:18:56.251 INFO:tasks.ceph.osd.2.smithi121.stderr:2024-09-17T18:18:56.249+0000 7f1e3a25e640 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T18:18:56.252 INFO:tasks.ceph.osd.2.smithi121.stderr:2024-09-17T18:18:56.250+0000 7f1e35d40640 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T18:18:56.508 INFO:tasks.ceph.osd.4.smithi173.stderr:2024-09-17T18:18:56.505+0000 7fbfcd99a640 -1 osd.4 0 waiting for initial osdmap 2024-09-17T18:18:56.511 INFO:tasks.ceph.osd.4.smithi173.stderr:2024-09-17T18:18:56.508+0000 7fbfc97c1640 -1 osd.4 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T18:18:57.084 INFO:tasks.ceph.osd.1.smithi121.stderr:2024-09-17T18:18:57.082+0000 7f1f1547c640 -1 osd.1 0 waiting for initial osdmap 2024-09-17T18:18:57.087 INFO:tasks.ceph.osd.1.smithi121.stderr:2024-09-17T18:18:57.085+0000 7f1f0d29b640 -1 osd.1 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T18:18:57.594 DEBUG:teuthology.orchestra.run.smithi121:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-17T18:18:57.681 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.680+0000 7ff62cb10640 10 monclient: get_monmap_and_config 2024-09-17T18:18:57.682 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.680+0000 7ff62cb10640 10 monclient: build_initial_monmap 2024-09-17T18:18:57.682 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.680+0000 7ff62cb10640 10 monclient: monmap: 2024-09-17T18:18:57.682 INFO:teuthology.misc.health.smithi121.stderr:epoch 0 2024-09-17T18:18:57.682 INFO:teuthology.misc.health.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:57.682 INFO:teuthology.misc.health.smithi121.stderr:last_changed 2024-09-17T18:18:57.680733+0000 2024-09-17T18:18:57.682 INFO:teuthology.misc.health.smithi121.stderr:created 2024-09-17T18:18:57.680733+0000 2024-09-17T18:18:57.682 INFO:teuthology.misc.health.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:18:57.682 INFO:teuthology.misc.health.smithi121.stderr:election_strategy: 1 2024-09-17T18:18:57.682 INFO:teuthology.misc.health.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:18:57.682 INFO:teuthology.misc.health.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:18:57.682 INFO:teuthology.misc.health.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:18:57.682 INFO:teuthology.misc.health.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:18:57.682 INFO:teuthology.misc.health.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:18:57.683 INFO:teuthology.misc.health.smithi121.stderr: 2024-09-17T18:18:57.685 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.680+0000 7ff62cb10640 1 Processor -- start 2024-09-17T18:18:57.685 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.680+0000 7ff62cb10640 1 -- start start 2024-09-17T18:18:57.685 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.680+0000 7ff62cb10640 10 monclient: init 2024-09-17T18:18:57.685 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.681+0000 7ff62cb10640 10 monclient: _reopen_session rank -1 2024-09-17T18:18:57.685 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.681+0000 7ff62cb10640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-17T18:18:57.685 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.681+0000 7ff62cb10640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff628070600 addr v1:172.21.15.121:6789/0 2024-09-17T18:18:57.685 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.681+0000 7ff62cb10640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff6280a6200 0x7ff62813daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:57.685 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.681+0000 7ff62cb10640 10 monclient(hunting): picked mon.noname-b con 0x7ff6280a6200 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:18:57.685 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.681+0000 7ff62cb10640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff6280a65e0 0x7ff628140040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:57.685 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.681+0000 7ff62cb10640 10 monclient(hunting): picked mon.noname-a con 0x7ff6280a65e0 addr v2:172.21.15.121:3300/0 2024-09-17T18:18:57.686 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.681+0000 7ff62cb10640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6280a2070 con 0x7ff628070600 2024-09-17T18:18:57.686 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.681+0000 7ff62cb10640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:57.686 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.681+0000 7ff62cb10640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7ff6280709e0 con 0x7ff6280a65e0 2024-09-17T18:18:57.686 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.681+0000 7ff62cb10640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:57.686 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.681+0000 7ff62cb10640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff6280a2b10 con 0x7ff6280a6200 2024-09-17T18:18:57.686 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.681+0000 7ff62cb10640 10 monclient(hunting): _renew_subs 2024-09-17T18:18:57.686 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.681+0000 7ff62cb10640 10 monclient(hunting): authenticate will time out at 741.397433s 2024-09-17T18:18:57.686 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.681+0000 7ff625d9b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff6280a6200 0x7ff62813daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:57.686 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.681+0000 7ff626d9d640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff6280a65e0 0x7ff628140040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:57.686 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.681+0000 7ff625d9b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff6280a6200 0x7ff62813daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:46534/0 (socket says 172.21.15.121:46534) 2024-09-17T18:18:57.686 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.681+0000 7ff625d9b640 1 -- 172.21.15.121:0/3503684190 learned_addr learned my addr 172.21.15.121:0/3503684190 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:18:57.686 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.681+0000 7ff625d9b640 10 monclient(hunting): get_auth_request con 0x7ff6280a6200 auth_method 0 2024-09-17T18:18:57.686 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.681+0000 7ff625d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:57.687 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.681+0000 7ff625d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:57.687 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.681+0000 7ff625d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:57.687 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.681+0000 7ff626d9d640 10 monclient(hunting): get_auth_request con 0x7ff6280a65e0 auth_method 0 2024-09-17T18:18:57.687 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.681+0000 7ff626d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:57.687 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.681+0000 7ff626d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:57.687 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.681+0000 7ff626d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:57.687 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.681+0000 7ff625d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:57.687 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.681+0000 7ff625d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:57.687 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.681+0000 7ff625d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:18:57.687 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.681+0000 7ff626d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:57.687 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.681+0000 7ff626d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:57.687 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.681+0000 7ff626d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:18:57.687 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.681+0000 7ff625d9b640 10 monclient(hunting): handle_auth_done global_id 4172 payload 274 2024-09-17T18:18:57.687 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.681+0000 7ff625d9b640 1 -- 172.21.15.121:0/3503684190 >> v2:172.21.15.121:3300/0 conn(0x7ff6280a65e0 msgr2=0x7ff628140040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:18:57.688 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.681+0000 7ff625d9b640 1 --2- 172.21.15.121:0/3503684190 >> v2:172.21.15.121:3300/0 conn(0x7ff6280a65e0 0x7ff628140040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:57.688 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.681+0000 7ff625d9b640 1 -- 172.21.15.121:0/3503684190 >> v1:172.21.15.121:6789/0 conn(0x7ff628070600 legacy=0x7ff6280a5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:18:57.688 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.681+0000 7ff625d9b640 10 monclient: _finish_hunting 0 2024-09-17T18:18:57.688 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.681+0000 7ff625d9b640 1 monclient: found mon.noname-b 2024-09-17T18:18:57.688 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.681+0000 7ff625d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:18:57.688 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.681+0000 7ff625d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:18:57.688 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.681+0000 7ff625d9b640 1 -- 172.21.15.121:0/3503684190 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6280a3c10 con 0x7ff6280a6200 2024-09-17T18:18:57.688 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.681+0000 7ff626d9d640 1 --2- 172.21.15.121:0/3503684190 >> v2:172.21.15.121:3300/0 conn(0x7ff6280a65e0 0x7ff628140040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:18:57.694 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.684+0000 7ff625d9b640 1 -- 172.21.15.121:0/3503684190 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff6280a6200 msgr2=0x7ff62813daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:18:57.694 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.684+0000 7ff625d9b640 1 --2- 172.21.15.121:0/3503684190 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff6280a6200 0x7ff62813daf0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff610009e50 tx=0x7ff61002a560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef8a8f03a53fe95b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:18:57.694 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.684+0000 7ff62559a640 1 -- 172.21.15.121:0/3503684190 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff610036070 con 0x7ff6280a6200 2024-09-17T18:18:57.694 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.684+0000 7ff62559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:57.694 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.684+0000 7ff62559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:18:57.695 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.684+0000 7ff62559a640 10 monclient: dump: 2024-09-17T18:18:57.695 INFO:teuthology.misc.health.smithi121.stderr:epoch 1 2024-09-17T18:18:57.695 INFO:teuthology.misc.health.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:57.695 INFO:teuthology.misc.health.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:57.695 INFO:teuthology.misc.health.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:57.695 INFO:teuthology.misc.health.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:57.695 INFO:teuthology.misc.health.smithi121.stderr:election_strategy: 1 2024-09-17T18:18:57.695 INFO:teuthology.misc.health.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:57.695 INFO:teuthology.misc.health.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:57.695 INFO:teuthology.misc.health.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:57.695 INFO:teuthology.misc.health.smithi121.stderr: 2024-09-17T18:18:57.695 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.684+0000 7ff62559a640 10 monclient: _finish_auth 0 2024-09-17T18:18:57.695 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.684+0000 7ff62559a640 10 monclient: _check_auth_tickets 2024-09-17T18:18:57.696 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.684+0000 7ff62559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:18:57.696 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.684+0000 7ff62559a640 1 -- 172.21.15.121:0/3503684190 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff610007800 con 0x7ff6280a6200 2024-09-17T18:18:57.696 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.684+0000 7ff62cb10640 5 monclient: authenticate success, global_id 4172 2024-09-17T18:18:57.696 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.684+0000 7ff62cb10640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:18:57.696 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.684+0000 7ff62559a640 10 monclient: handle_config config(0 keys) 2024-09-17T18:18:57.696 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.684+0000 7ff62cb10640 10 monclient: get_monmap_and_config success 2024-09-17T18:18:57.696 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.684+0000 7ff62559a640 1 -- 172.21.15.121:0/3503684190 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff610007bd0 con 0x7ff6280a6200 2024-09-17T18:18:57.696 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.684+0000 7ff62cb10640 4 set_mon_vals no callback set 2024-09-17T18:18:57.696 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.684+0000 7ff62559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:57.696 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.684+0000 7ff62559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:18:57.696 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.684+0000 7ff62559a640 10 monclient: dump: 2024-09-17T18:18:57.696 INFO:teuthology.misc.health.smithi121.stderr:epoch 1 2024-09-17T18:18:57.696 INFO:teuthology.misc.health.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:57.696 INFO:teuthology.misc.health.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:57.697 INFO:teuthology.misc.health.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:57.697 INFO:teuthology.misc.health.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:57.697 INFO:teuthology.misc.health.smithi121.stderr:election_strategy: 1 2024-09-17T18:18:57.697 INFO:teuthology.misc.health.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:57.697 INFO:teuthology.misc.health.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:57.697 INFO:teuthology.misc.health.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:57.697 INFO:teuthology.misc.health.smithi121.stderr: 2024-09-17T18:18:57.697 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.685+0000 7ff62cb10640 10 monclient: shutdown 2024-09-17T18:18:57.697 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.685+0000 7ff62cb10640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:18:57.697 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.685+0000 7ff62cb10640 1 -- 172.21.15.121:0/3503684190 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff6280a6200 msgr2=0x7ff62813daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:57.697 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.685+0000 7ff62cb10640 1 --2- 172.21.15.121:0/3503684190 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff6280a6200 0x7ff62813daf0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff610009e50 tx=0x7ff61002a560 comp rx=0 tx=0).stop 2024-09-17T18:18:57.697 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.685+0000 7ff62cb10640 1 -- 172.21.15.121:0/3503684190 shutdown_connections 2024-09-17T18:18:57.697 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.685+0000 7ff62cb10640 1 --2- 172.21.15.121:0/3503684190 >> v2:172.21.15.121:3300/0 conn(0x7ff6280a65e0 0x7ff628140040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:57.697 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.685+0000 7ff62cb10640 1 --2- 172.21.15.121:0/3503684190 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff6280a6200 0x7ff62813daf0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:57.698 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.685+0000 7ff62cb10640 1 -- 172.21.15.121:0/3503684190 >> 172.21.15.121:0/3503684190 conn(0x7ff62806d930 msgr2=0x7ff62806eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:18:57.698 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.685+0000 7ff62cb10640 1 -- 172.21.15.121:0/3503684190 shutdown_connections 2024-09-17T18:18:57.698 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.685+0000 7ff62cb10640 1 -- 172.21.15.121:0/3503684190 wait complete. 2024-09-17T18:18:57.698 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.686+0000 7ff62cb10640 10 monclient: build_initial_monmap 2024-09-17T18:18:57.698 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.686+0000 7ff62cb10640 10 monclient: monmap: 2024-09-17T18:18:57.698 INFO:teuthology.misc.health.smithi121.stderr:epoch 0 2024-09-17T18:18:57.698 INFO:teuthology.misc.health.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:18:57.698 INFO:teuthology.misc.health.smithi121.stderr:last_changed 0.000000 2024-09-17T18:18:57.698 INFO:teuthology.misc.health.smithi121.stderr:created 0.000000 2024-09-17T18:18:57.698 INFO:teuthology.misc.health.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:18:57.698 INFO:teuthology.misc.health.smithi121.stderr:election_strategy: 1 2024-09-17T18:18:57.698 INFO:teuthology.misc.health.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:18:57.698 INFO:teuthology.misc.health.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:18:57.698 INFO:teuthology.misc.health.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:18:57.699 INFO:teuthology.misc.health.smithi121.stderr: 2024-09-17T18:18:57.699 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.686+0000 7ff62cb10640 1 librados: starting msgr at 2024-09-17T18:18:57.699 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.686+0000 7ff62cb10640 1 librados: starting objecter 2024-09-17T18:18:57.699 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.686+0000 7ff62cb10640 1 Processor -- start 2024-09-17T18:18:57.699 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.686+0000 7ff62cb10640 1 -- start start 2024-09-17T18:18:57.699 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.686+0000 7ff62cb10640 1 librados: setting wanted keys 2024-09-17T18:18:57.699 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.686+0000 7ff62cb10640 1 librados: calling monclient init 2024-09-17T18:18:57.699 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.686+0000 7ff62cb10640 10 monclient: init 2024-09-17T18:18:57.699 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.686+0000 7ff62cb10640 10 monclient: _reopen_session rank -1 2024-09-17T18:18:57.699 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.686+0000 7ff62cb10640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T18:18:57.699 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.686+0000 7ff62cb10640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff628070600 0x7ff6281cd800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:57.699 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.686+0000 7ff62cb10640 10 monclient(hunting): picked mon.noname-b con 0x7ff628070600 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:18:57.699 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.686+0000 7ff62cb10640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6280a6200 0x7ff628138ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:57.699 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.686+0000 7ff62cb10640 10 monclient(hunting): picked mon.noname-a con 0x7ff6280a6200 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:18:57.700 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.686+0000 7ff62cb10640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff6280a65e0 0x7ff62813dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:57.700 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff62cb10640 10 monclient(hunting): picked mon.noname-c con 0x7ff6280a65e0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:18:57.700 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff62cb10640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:57.700 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff62cb10640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff628068e30 con 0x7ff6280a6200 2024-09-17T18:18:57.700 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff62cb10640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:57.700 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff62cb10640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff6280a6cc0 con 0x7ff628070600 2024-09-17T18:18:57.700 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff62cb10640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:57.700 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff62cb10640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff6281400e0 con 0x7ff6280a65e0 2024-09-17T18:18:57.700 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff62cb10640 10 monclient(hunting): _renew_subs 2024-09-17T18:18:57.700 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff62cb10640 10 monclient(hunting): authenticate will time out at 741.403026s 2024-09-17T18:18:57.700 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff625d9b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6280a6200 0x7ff628138ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:57.700 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff626d9d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff6280a65e0 0x7ff62813dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:57.700 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff625d9b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6280a6200 0x7ff628138ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39844/0 (socket says 172.21.15.121:39844) 2024-09-17T18:18:57.700 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff625d9b640 1 -- 172.21.15.121:0/750853594 learned_addr learned my addr 172.21.15.121:0/750853594 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:18:57.701 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff625d9b640 10 monclient(hunting): get_auth_request con 0x7ff6280a6200 auth_method 0 2024-09-17T18:18:57.701 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff625d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:57.701 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff625d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:57.701 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff625d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:57.701 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff626d9d640 10 monclient(hunting): get_auth_request con 0x7ff6280a65e0 auth_method 0 2024-09-17T18:18:57.701 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff626d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:57.701 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff626d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:57.701 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff626d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:57.701 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff625d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:57.701 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff625d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:57.701 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff625d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:18:57.701 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff626d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:57.701 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff626d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:57.701 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff626d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:18:57.702 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff62659c640 1 --2- 172.21.15.121:0/750853594 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff628070600 0x7ff6281cd800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:57.702 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff62659c640 10 monclient(hunting): get_auth_request con 0x7ff628070600 auth_method 0 2024-09-17T18:18:57.702 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff62659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:57.702 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff62659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:57.702 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff62659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:57.702 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff625d9b640 10 monclient(hunting): handle_auth_done global_id 4188 payload 824 2024-09-17T18:18:57.702 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff625d9b640 1 -- 172.21.15.121:0/750853594 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff6280a65e0 msgr2=0x7ff62813dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:57.702 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff625d9b640 1 --2- 172.21.15.121:0/750853594 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff6280a65e0 0x7ff62813dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:57.702 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff625d9b640 1 -- 172.21.15.121:0/750853594 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff628070600 msgr2=0x7ff6281cd800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:57.702 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff625d9b640 1 --2- 172.21.15.121:0/750853594 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff628070600 0x7ff6281cd800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:57.702 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff625d9b640 10 monclient: _finish_hunting 0 2024-09-17T18:18:57.702 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff625d9b640 1 monclient: found mon.noname-a 2024-09-17T18:18:57.702 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff625d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:18:57.703 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff625d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:18:57.703 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff625d9b640 1 -- 172.21.15.121:0/750853594 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6281ce700 con 0x7ff6280a6200 2024-09-17T18:18:57.703 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.687+0000 7ff626d9d640 1 --2- 172.21.15.121:0/750853594 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff6280a65e0 0x7ff62813dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:18:57.703 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.689+0000 7ff625d9b640 1 -- 172.21.15.121:0/750853594 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6280a6200 msgr2=0x7ff628138ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:18:57.703 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.689+0000 7ff625d9b640 1 --2- 172.21.15.121:0/750853594 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6280a6200 0x7ff628138ac0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff610034040 tx=0x7ff610004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:18:57.703 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.689+0000 7ff61effd640 10 client.?.objecter ms_handle_connect 0x7ff6280a6200 2024-09-17T18:18:57.703 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.689+0000 7ff61effd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:18:57.703 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.690+0000 7ff61effd640 1 -- 172.21.15.121:0/750853594 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff610036050 con 0x7ff6280a6200 2024-09-17T18:18:57.703 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.690+0000 7ff61effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:57.703 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.690+0000 7ff61effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:18:57.703 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.690+0000 7ff61effd640 10 monclient: dump: 2024-09-17T18:18:57.703 INFO:teuthology.misc.health.smithi121.stderr:epoch 1 2024-09-17T18:18:57.703 INFO:teuthology.misc.health.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:57.704 INFO:teuthology.misc.health.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:57.704 INFO:teuthology.misc.health.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:57.704 INFO:teuthology.misc.health.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:57.704 INFO:teuthology.misc.health.smithi121.stderr:election_strategy: 1 2024-09-17T18:18:57.704 INFO:teuthology.misc.health.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:57.704 INFO:teuthology.misc.health.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:57.704 INFO:teuthology.misc.health.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:57.704 INFO:teuthology.misc.health.smithi121.stderr: 2024-09-17T18:18:57.704 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.690+0000 7ff61effd640 10 monclient: _finish_auth 0 2024-09-17T18:18:57.704 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.690+0000 7ff61effd640 10 monclient: _check_auth_tickets 2024-09-17T18:18:57.704 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.690+0000 7ff61effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:18:57.704 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.690+0000 7ff62cb10640 5 monclient: authenticate success, global_id 4188 2024-09-17T18:18:57.704 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.690+0000 7ff61effd640 1 -- 172.21.15.121:0/750853594 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff610042070 con 0x7ff6280a6200 2024-09-17T18:18:57.704 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.690+0000 7ff62cb10640 10 monclient: _renew_subs 2024-09-17T18:18:57.705 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.690+0000 7ff62cb10640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:18:57.705 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.690+0000 7ff62cb10640 1 -- 172.21.15.121:0/750853594 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff62813f660 con 0x7ff6280a6200 2024-09-17T18:18:57.705 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.690+0000 7ff61effd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:18:57.705 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.690+0000 7ff61effd640 1 -- 172.21.15.121:0/750853594 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff610031e10 con 0x7ff6280a6200 2024-09-17T18:18:57.705 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.690+0000 7ff61effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:57.705 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.690+0000 7ff61effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:18:57.705 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.690+0000 7ff61effd640 10 monclient: dump: 2024-09-17T18:18:57.705 INFO:teuthology.misc.health.smithi121.stderr:epoch 1 2024-09-17T18:18:57.705 INFO:teuthology.misc.health.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:57.705 INFO:teuthology.misc.health.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:57.705 INFO:teuthology.misc.health.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:57.705 INFO:teuthology.misc.health.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:57.705 INFO:teuthology.misc.health.smithi121.stderr:election_strategy: 1 2024-09-17T18:18:57.705 INFO:teuthology.misc.health.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:57.705 INFO:teuthology.misc.health.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:57.706 INFO:teuthology.misc.health.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:57.706 INFO:teuthology.misc.health.smithi121.stderr: 2024-09-17T18:18:57.706 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.690+0000 7ff62cb10640 10 client.4188.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:18:57.706 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.690+0000 7ff624d99640 4 set_mon_vals no callback set 2024-09-17T18:18:57.706 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.690+0000 7ff62cb10640 10 monclient: _renew_subs 2024-09-17T18:18:57.706 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.690+0000 7ff62cb10640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:18:57.706 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.690+0000 7ff62cb10640 1 -- 172.21.15.121:0/750853594 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff628122cb0 con 0x7ff6280a6200 2024-09-17T18:18:57.706 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.690+0000 7ff62cb10640 1 librados: init done 2024-09-17T18:18:57.706 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.690+0000 7ff61effd640 1 -- 172.21.15.121:0/750853594 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 4) ==== 50940+0+0 (secure 0 0 0) 0x7ff610047460 con 0x7ff6280a6200 2024-09-17T18:18:57.706 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.690+0000 7ff62cb10640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:18:57.706 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.690+0000 7ff62cb10640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:18:57.706 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.690+0000 7ff62cb10640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:18:57.706 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.690+0000 7ff62cb10640 1 -- 172.21.15.121:0/750853594 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6280a5af0 con 0x7ff6280a6200 2024-09-17T18:18:57.707 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.691+0000 7ff61effd640 1 --2- 172.21.15.121:0/750853594 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff6080400e0 0x7ff6080425a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:57.707 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.691+0000 7ff61effd640 1 -- 172.21.15.121:0/750853594 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 2767+0+0 (secure 0 0 0) 0x7ff610073bd0 con 0x7ff6280a6200 2024-09-17T18:18:57.707 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.691+0000 7ff61effd640 10 client.4188.objecter ms_dispatch 0x7ff628000b90 osd_map(12..12 src has 1..12) 2024-09-17T18:18:57.707 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.691+0000 7ff61effd640 3 client.4188.objecter handle_osd_map got epochs [12,12] > 0 2024-09-17T18:18:57.707 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.691+0000 7ff61effd640 3 client.4188.objecter handle_osd_map decoding full epoch 12 2024-09-17T18:18:57.707 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.691+0000 7ff61effd640 20 client.4188.objecter dump_active .. 0 homeless 2024-09-17T18:18:57.707 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.691+0000 7ff62659c640 1 --2- 172.21.15.121:0/750853594 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff6080400e0 0x7ff6080425a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:57.707 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.691+0000 7ff62659c640 10 monclient: get_auth_request con 0x7ff6080400e0 auth_method 0 2024-09-17T18:18:57.708 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.694+0000 7ff61effd640 10 client.4188.objecter ms_handle_connect 0x7ff6080400e0 2024-09-17T18:18:57.708 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.694+0000 7ff62659c640 1 --2- 172.21.15.121:0/750853594 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff6080400e0 0x7ff6080425a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff620006fd0 tx=0x7ff620008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:18:57.708 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.696+0000 7ff61effd640 1 -- 172.21.15.121:0/750853594 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff610072dc0 con 0x7ff6280a6200 2024-09-17T18:18:57.708 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.696+0000 7ff61effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:18:57.708 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.696+0000 7ff61effd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:18:57.872 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.871+0000 7ff62cb10640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-17T18:18:57.873 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.871+0000 7ff62cb10640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T18:18:57.873 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.871+0000 7ff62cb10640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:18:57.873 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.871+0000 7ff62cb10640 1 -- 172.21.15.121:0/750853594 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff62806a580 con 0x7ff6280a6200 2024-09-17T18:18:57.873 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.871+0000 7ff61effd640 1 -- 172.21.15.121:0/750853594 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) ==== 74+0+7165 (secure 0 0 0) 0x7ff61005ee20 con 0x7ff6280a6200 2024-09-17T18:18:57.873 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.871+0000 7ff61effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T18:18:57.873 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.871+0000 7ff61effd640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:18:57.874 INFO:teuthology.misc.health.smithi121.stdout: 2024-09-17T18:18:57.874 INFO:teuthology.misc.health.smithi121.stdout:{"epoch":12,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","created":"2024-09-17T18:18:47.237753+0000","modified":"2024-09-17T18:18:57.511756+0000","last_up_change":"2024-09-17T18:18:57.511756+0000","last_in_change":"2024-09-17T18:18:49.165275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bf586d1-05ac-4c7e-8b88-313f07a76801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":1526257129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6801","nonce":1526257129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":1526257129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":1526257129}]},"public_addr":"172.21.15.121:6800/1526257129","cluster_addr":"172.21.15.121:6801/1526257129","heartbeat_back_addr":"172.21.15.121:6803/1526257129","heartbeat_front_addr":"172.21.15.121:6802/1526257129","state":["exists","up"]},{"osd":1,"uuid":"e26c6163-1eda-4200-82e1-57b752fd8a46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":196419920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6809","nonce":196419920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":196419920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":196419920}]},"public_addr":"172.21.15.121:6808/196419920","cluster_addr":"172.21.15.121:6809/196419920","heartbeat_back_addr":"172.21.15.121:6811/196419920","heartbeat_front_addr":"172.21.15.121:6810/196419920","state":["exists","up"]},{"osd":2,"uuid":"6fb45798-21d4-449e-9f68-7210d984e036","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2545987420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6805","nonce":2545987420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":2545987420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2545987420}]},"public_addr":"172.21.15.121:6804/2545987420","cluster_addr":"172.21.15.121:6805/2545987420","heartbeat_back_addr":"172.21.15.121:6807/2545987420","heartbeat_front_addr":"172.21.15.121:6806/2545987420","state":["exists","up"]},{"osd":3,"uuid":"3de6630d-4fb7-4f64-b4f9-e5c304b1be63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":3493964597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6801","nonce":3493964597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6803","nonce":3493964597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":3493964597}]},"public_addr":"172.21.15.173:6800/3493964597","cluster_addr":"172.21.15.173:6801/3493964597","heartbeat_back_addr":"172.21.15.173:6803/3493964597","heartbeat_front_addr":"172.21.15.173:6802/3493964597","state":["exists","up"]},{"osd":4,"uuid":"cfa74383-b7c9-4ecd-8258-3792fbccc237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":1961801317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6805","nonce":1961801317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6807","nonce":1961801317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":1961801317}]},"public_addr":"172.21.15.173:6804/1961801317","cluster_addr":"172.21.15.173:6805/1961801317","heartbeat_back_addr":"172.21.15.173:6807/1961801317","heartbeat_front_addr":"172.21.15.173:6806/1961801317","state":["exists","up"]},{"osd":5,"uuid":"8bdb7768-a603-4f14-9650-47b503621215","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1801998802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6809","nonce":1801998802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6811","nonce":1801998802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1801998802}]},"public_addr":"172.21.15.173:6808/1801998802","cluster_addr":"172.21.15.173:6809/1801998802","heartbeat_back_addr":"172.21.15.173:6811/1801998802","heartbeat_front_addr":"172.21.15.173:6810/1801998802","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T18:18:55.231353+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T18:18:56.242909+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T18:18:55.454088+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T18:18:55.389354+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T18:18:57.876 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.873+0000 7ff62cb10640 10 librados: watch_flush enter 2024-09-17T18:18:57.876 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.873+0000 7ff62cb10640 10 librados: watch_flush exit 2024-09-17T18:18:57.876 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.873+0000 7ff62cb10640 20 client.4188.objecter shutdown clearing up homeless session... 2024-09-17T18:18:57.876 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.873+0000 7ff62cb10640 10 client.4188.objecter successfully canceled tick 2024-09-17T18:18:57.876 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.873+0000 7ff62cb10640 1 -- 172.21.15.121:0/750853594 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff6080400e0 msgr2=0x7ff6080425a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:57.876 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.873+0000 7ff62cb10640 1 --2- 172.21.15.121:0/750853594 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff6080400e0 0x7ff6080425a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff620006fd0 tx=0x7ff620008040 comp rx=0 tx=0).stop 2024-09-17T18:18:57.876 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.873+0000 7ff62cb10640 10 monclient: shutdown 2024-09-17T18:18:57.877 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.873+0000 7ff62cb10640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:18:57.877 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.873+0000 7ff62cb10640 1 -- 172.21.15.121:0/750853594 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6280a6200 msgr2=0x7ff628138ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:57.877 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.873+0000 7ff62cb10640 1 --2- 172.21.15.121:0/750853594 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6280a6200 0x7ff628138ac0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff610034040 tx=0x7ff610004a40 comp rx=0 tx=0).stop 2024-09-17T18:18:57.877 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.873+0000 7ff62cb10640 1 -- 172.21.15.121:0/750853594 shutdown_connections 2024-09-17T18:18:57.877 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.873+0000 7ff62cb10640 1 --2- 172.21.15.121:0/750853594 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff6080400e0 0x7ff6080425a0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:57.877 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.873+0000 7ff62cb10640 1 --2- 172.21.15.121:0/750853594 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff6280a65e0 0x7ff62813dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:57.877 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.873+0000 7ff62cb10640 1 --2- 172.21.15.121:0/750853594 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6280a6200 0x7ff628138ac0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:57.877 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.873+0000 7ff62cb10640 1 --2- 172.21.15.121:0/750853594 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff628070600 0x7ff6281cd800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:57.877 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.874+0000 7ff62cb10640 1 -- 172.21.15.121:0/750853594 >> 172.21.15.121:0/750853594 conn(0x7ff62806d930 msgr2=0x7ff6280a8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:18:57.877 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.874+0000 7ff62cb10640 1 -- 172.21.15.121:0/750853594 shutdown_connections 2024-09-17T18:18:57.877 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.874+0000 7ff62cb10640 1 -- 172.21.15.121:0/750853594 wait complete. 2024-09-17T18:18:57.877 INFO:teuthology.misc.health.smithi121.stderr:2024-09-17T18:18:57.874+0000 7ff62cb10640 1 librados: shutdown 2024-09-17T18:18:57.884 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-09-17T18:18:57.885 INFO:tasks.ceph:Creating RBD pool 2024-09-17T18:18:57.885 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-17T18:18:57.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.980+0000 7f7c53305640 10 monclient: get_monmap_and_config 2024-09-17T18:18:57.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.980+0000 7f7c53305640 10 monclient: build_initial_monmap 2024-09-17T18:18:57.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.980+0000 7f7c53305640 10 monclient: monmap: 2024-09-17T18:18:57.982 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:18:57.982 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:57.982 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:57.980678+0000 2024-09-17T18:18:57.982 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:57.980678+0000 2024-09-17T18:18:57.982 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:18:57.982 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:18:57.982 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:18:57.982 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:18:57.982 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:18:57.982 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:18:57.982 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:18:57.983 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:18:57.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.980+0000 7f7c53305640 1 Processor -- start 2024-09-17T18:18:57.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.980+0000 7f7c53305640 1 -- start start 2024-09-17T18:18:57.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.980+0000 7f7c53305640 10 monclient: init 2024-09-17T18:18:57.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c53305640 10 monclient: _reopen_session rank -1 2024-09-17T18:18:57.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c53305640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-17T18:18:57.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c53305640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c4c0a5b90 0x7f7c4c0a5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:57.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c53305640 10 monclient(hunting): picked mon.noname-b con 0x7f7c4c0a5b90 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:18:57.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c53305640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7c4c0a7090 0x7f7c4c0a64b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:57.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c53305640 10 monclient(hunting): picked mon.noname-a con 0x7f7c4c0a7090 addr v2:172.21.15.121:3300/0 2024-09-17T18:18:57.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c53305640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7c4c138c40 addr v1:172.21.15.121:6789/0 2024-09-17T18:18:57.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c53305640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c4c0a6e70 con 0x7f7c4c138c40 2024-09-17T18:18:57.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c53305640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:57.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c53305640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f7c4c0a69f0 con 0x7f7c4c0a7090 2024-09-17T18:18:57.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c53305640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:57.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c53305640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7c4c09cde0 con 0x7f7c4c0a5b90 2024-09-17T18:18:57.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c53305640 10 monclient(hunting): _renew_subs 2024-09-17T18:18:57.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c53305640 10 monclient(hunting): authenticate will time out at 741.697437s 2024-09-17T18:18:57.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c510a1640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c4c0a5b90 0x7f7c4c0a5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:57.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c508a0640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7c4c0a7090 0x7f7c4c0a64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:57.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c510a1640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c4c0a5b90 0x7f7c4c0a5f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:46564/0 (socket says 172.21.15.121:46564) 2024-09-17T18:18:57.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c508a0640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7c4c0a7090 0x7f7c4c0a64b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39856/0 (socket says 172.21.15.121:39856) 2024-09-17T18:18:57.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c510a1640 1 -- 172.21.15.121:0/3792601956 learned_addr learned my addr 172.21.15.121:0/3792601956 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:18:57.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c510a1640 10 monclient(hunting): get_auth_request con 0x7f7c4c0a5b90 auth_method 0 2024-09-17T18:18:57.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c510a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:57.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c510a1640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:57.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c510a1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:57.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c518a2640 1 --1- 172.21.15.121:0/3792601956 >> v1:172.21.15.121:6789/0 conn(0x7f7c4c138c40 0x7f7c4c1390b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:38998/0 (socket says 172.21.15.121:38998) 2024-09-17T18:18:57.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c508a0640 10 monclient(hunting): get_auth_request con 0x7f7c4c0a7090 auth_method 0 2024-09-17T18:18:57.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c508a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:57.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c508a0640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:57.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c508a0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:57.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c510a1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:57.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c510a1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:57.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c510a1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:18:57.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c508a0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:57.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c508a0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:57.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c508a0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:18:57.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c510a1640 10 monclient(hunting): handle_auth_done global_id 4178 payload 274 2024-09-17T18:18:57.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c510a1640 1 -- 172.21.15.121:0/3792601956 >> v2:172.21.15.121:3300/0 conn(0x7f7c4c0a7090 msgr2=0x7f7c4c0a64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:18:57.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c510a1640 1 --2- 172.21.15.121:0/3792601956 >> v2:172.21.15.121:3300/0 conn(0x7f7c4c0a7090 0x7f7c4c0a64b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:57.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c510a1640 1 -- 172.21.15.121:0/3792601956 >> v1:172.21.15.121:6789/0 conn(0x7f7c4c138c40 legacy=0x7f7c4c1390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:18:57.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c510a1640 10 monclient: _finish_hunting 0 2024-09-17T18:18:57.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c510a1640 1 monclient: found mon.noname-b 2024-09-17T18:18:57.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c510a1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:18:57.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c510a1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:18:57.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c510a1640 1 -- 172.21.15.121:0/3792601956 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c4c09cb00 con 0x7f7c4c0a5b90 2024-09-17T18:18:57.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.981+0000 7f7c508a0640 1 --2- 172.21.15.121:0/3792601956 >> v2:172.21.15.121:3300/0 conn(0x7f7c4c0a7090 0x7f7c4c0a64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:18:57.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.984+0000 7f7c510a1640 1 -- 172.21.15.121:0/3792601956 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c4c0a5b90 msgr2=0x7f7c4c0a5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:18:57.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.984+0000 7f7c510a1640 1 --2- 172.21.15.121:0/3792601956 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c4c0a5b90 0x7f7c4c0a5f70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c070cf0 tx=0x7f7c4802a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d73fe017e8267c38 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:18:57.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.984+0000 7f7c43fff640 1 -- 172.21.15.121:0/3792601956 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c48037070 con 0x7f7c4c0a5b90 2024-09-17T18:18:57.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.984+0000 7f7c43fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:57.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.984+0000 7f7c43fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:18:57.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.984+0000 7f7c43fff640 10 monclient: dump: 2024-09-17T18:18:57.997 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:18:57.997 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:57.997 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:57.997 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:57.997 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:57.997 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:18:57.997 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:57.997 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:57.997 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:57.997 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:18:57.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.984+0000 7f7c43fff640 10 monclient: _finish_auth 0 2024-09-17T18:18:57.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.984+0000 7f7c43fff640 10 monclient: _check_auth_tickets 2024-09-17T18:18:57.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.984+0000 7f7c43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:18:57.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.984+0000 7f7c53305640 5 monclient: authenticate success, global_id 4178 2024-09-17T18:18:57.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.984+0000 7f7c43fff640 1 -- 172.21.15.121:0/3792601956 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c48032440 con 0x7f7c4c0a5b90 2024-09-17T18:18:57.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.984+0000 7f7c43fff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:18:57.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.984+0000 7f7c53305640 10 monclient: get_monmap_and_config success 2024-09-17T18:18:57.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.984+0000 7f7c43fff640 1 -- 172.21.15.121:0/3792601956 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c48032810 con 0x7f7c4c0a5b90 2024-09-17T18:18:57.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.984+0000 7f7c53305640 4 set_mon_vals no callback set 2024-09-17T18:18:57.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.984+0000 7f7c43fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:57.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.984+0000 7f7c43fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:18:57.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.984+0000 7f7c43fff640 10 monclient: dump: 2024-09-17T18:18:57.998 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:18:57.998 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:57.998 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:57.998 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:57.998 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:57.998 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:18:57.999 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:57.999 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:57.999 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:57.999 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:18:57.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.985+0000 7f7c53305640 10 monclient: shutdown 2024-09-17T18:18:57.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.985+0000 7f7c53305640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:18:57.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.985+0000 7f7c53305640 1 -- 172.21.15.121:0/3792601956 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c4c0a5b90 msgr2=0x7f7c4c0a5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:57.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.985+0000 7f7c53305640 1 --2- 172.21.15.121:0/3792601956 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c4c0a5b90 0x7f7c4c0a5f70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c070cf0 tx=0x7f7c4802a060 comp rx=0 tx=0).stop 2024-09-17T18:18:57.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.985+0000 7f7c53305640 1 -- 172.21.15.121:0/3792601956 shutdown_connections 2024-09-17T18:18:57.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.985+0000 7f7c53305640 1 --2- 172.21.15.121:0/3792601956 >> v2:172.21.15.121:3300/0 conn(0x7f7c4c0a7090 0x7f7c4c0a64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:57.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.985+0000 7f7c53305640 1 --2- 172.21.15.121:0/3792601956 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c4c0a5b90 0x7f7c4c0a5f70 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:57.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.985+0000 7f7c53305640 1 -- 172.21.15.121:0/3792601956 >> 172.21.15.121:0/3792601956 conn(0x7f7c4c0a7c90 msgr2=0x7f7c4c0a80a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:18:57.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.985+0000 7f7c53305640 1 -- 172.21.15.121:0/3792601956 shutdown_connections 2024-09-17T18:18:57.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.985+0000 7f7c53305640 1 -- 172.21.15.121:0/3792601956 wait complete. 2024-09-17T18:18:58.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.986+0000 7f7c53305640 10 monclient: build_initial_monmap 2024-09-17T18:18:58.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.986+0000 7f7c53305640 10 monclient: monmap: 2024-09-17T18:18:58.000 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:18:58.000 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:18:58.000 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:18:58.000 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:18:58.000 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:18:58.000 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:18:58.000 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:18:58.000 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:18:58.000 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:18:58.000 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:18:58.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.986+0000 7f7c53305640 1 librados: starting msgr at 2024-09-17T18:18:58.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.986+0000 7f7c53305640 1 librados: starting objecter 2024-09-17T18:18:58.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.986+0000 7f7c53305640 1 Processor -- start 2024-09-17T18:18:58.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.986+0000 7f7c53305640 1 -- start start 2024-09-17T18:18:58.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.986+0000 7f7c53305640 1 librados: setting wanted keys 2024-09-17T18:18:58.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.986+0000 7f7c53305640 1 librados: calling monclient init 2024-09-17T18:18:58.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.986+0000 7f7c53305640 10 monclient: init 2024-09-17T18:18:58.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.986+0000 7f7c53305640 10 monclient: _reopen_session rank -1 2024-09-17T18:18:58.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.986+0000 7f7c53305640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T18:18:58.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.986+0000 7f7c53305640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c4c0a5b90 0x7f7c4c1cdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:58.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.986+0000 7f7c53305640 10 monclient(hunting): picked mon.noname-b con 0x7f7c4c0a5b90 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:18:58.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.986+0000 7f7c53305640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c4c0a7090 0x7f7c4c1ce480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:58.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.986+0000 7f7c53305640 10 monclient(hunting): picked mon.noname-a con 0x7f7c4c0a7090 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:18:58.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.986+0000 7f7c53305640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c4c138c40 0x7f7c4c123650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:58.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.986+0000 7f7c53305640 10 monclient(hunting): picked mon.noname-c con 0x7f7c4c138c40 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:18:58.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.986+0000 7f7c53305640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:58.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.986+0000 7f7c53305640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c0a8ba0 con 0x7f7c4c0a7090 2024-09-17T18:18:58.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.986+0000 7f7c53305640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:58.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.986+0000 7f7c53305640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c0a8a20 con 0x7f7c4c0a5b90 2024-09-17T18:18:58.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.986+0000 7f7c53305640 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:58.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.986+0000 7f7c53305640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7c4c1397c0 con 0x7f7c4c138c40 2024-09-17T18:18:58.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.986+0000 7f7c53305640 10 monclient(hunting): _renew_subs 2024-09-17T18:18:58.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.986+0000 7f7c53305640 10 monclient(hunting): authenticate will time out at 741.702928s 2024-09-17T18:18:58.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.987+0000 7f7c508a0640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c4c0a7090 0x7f7c4c1ce480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:58.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.987+0000 7f7c518a2640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c4c138c40 0x7f7c4c123650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:58.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.987+0000 7f7c518a2640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c4c138c40 0x7f7c4c123650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:46570/0 (socket says 172.21.15.121:46570) 2024-09-17T18:18:58.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.987+0000 7f7c508a0640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c4c0a7090 0x7f7c4c1ce480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39864/0 (socket says 172.21.15.121:39864) 2024-09-17T18:18:58.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.987+0000 7f7c518a2640 1 -- 172.21.15.121:0/4076937458 learned_addr learned my addr 172.21.15.121:0/4076937458 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:18:58.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.987+0000 7f7c518a2640 10 monclient(hunting): get_auth_request con 0x7f7c4c138c40 auth_method 0 2024-09-17T18:18:58.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.987+0000 7f7c518a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:58.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.987+0000 7f7c518a2640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:58.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.987+0000 7f7c518a2640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:58.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.987+0000 7f7c508a0640 10 monclient(hunting): get_auth_request con 0x7f7c4c0a7090 auth_method 0 2024-09-17T18:18:58.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.987+0000 7f7c508a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:58.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.987+0000 7f7c508a0640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:58.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.987+0000 7f7c508a0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:58.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.987+0000 7f7c518a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:58.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.987+0000 7f7c518a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:58.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.987+0000 7f7c518a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:18:58.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.987+0000 7f7c508a0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:58.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.987+0000 7f7c508a0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:58.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.987+0000 7f7c508a0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:18:58.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.987+0000 7f7c518a2640 10 monclient(hunting): handle_auth_done global_id 4181 payload 824 2024-09-17T18:18:58.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.987+0000 7f7c510a1640 1 --2- 172.21.15.121:0/4076937458 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c4c0a5b90 0x7f7c4c1cdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:58.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.987+0000 7f7c518a2640 1 -- 172.21.15.121:0/4076937458 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c4c0a5b90 msgr2=0x7f7c4c1cdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:58.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.987+0000 7f7c518a2640 1 --2- 172.21.15.121:0/4076937458 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c4c0a5b90 0x7f7c4c1cdf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:58.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.987+0000 7f7c518a2640 1 -- 172.21.15.121:0/4076937458 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c4c0a7090 msgr2=0x7f7c4c1ce480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:58.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.987+0000 7f7c518a2640 1 --2- 172.21.15.121:0/4076937458 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c4c0a7090 0x7f7c4c1ce480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:58.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.987+0000 7f7c518a2640 10 monclient: _finish_hunting 0 2024-09-17T18:18:58.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.987+0000 7f7c518a2640 1 monclient: found mon.noname-c 2024-09-17T18:18:58.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.987+0000 7f7c518a2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:18:58.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.987+0000 7f7c518a2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:18:58.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.987+0000 7f7c518a2640 1 -- 172.21.15.121:0/4076937458 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c4c138420 con 0x7f7c4c138c40 2024-09-17T18:18:58.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.987+0000 7f7c510a1640 10 monclient: get_auth_request con 0x7f7c4c0a5b90 auth_method 0 2024-09-17T18:18:58.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.987+0000 7f7c508a0640 1 --2- 172.21.15.121:0/4076937458 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c4c0a7090 0x7f7c4c1ce480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:18:58.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.987+0000 7f7c510a1640 1 --2- 172.21.15.121:0/4076937458 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c4c0a5b90 0x7f7c4c1cdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T18:18:58.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.989+0000 7f7c518a2640 1 -- 172.21.15.121:0/4076937458 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c4c138c40 msgr2=0x7f7c4c123650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:18:58.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.989+0000 7f7c518a2640 1 --2- 172.21.15.121:0/4076937458 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c4c138c40 0x7f7c4c123650 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7c4400c780 tx=0x7f7c4400cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:18:58.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.989+0000 7f7c41ffb640 10 client.?.objecter ms_handle_connect 0x7f7c4c138c40 2024-09-17T18:18:58.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.989+0000 7f7c41ffb640 10 client.?.objecter resend_mon_ops 2024-09-17T18:18:58.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.989+0000 7f7c41ffb640 1 -- 172.21.15.121:0/4076937458 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c44017070 con 0x7f7c4c138c40 2024-09-17T18:18:58.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.989+0000 7f7c41ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:58.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.989+0000 7f7c41ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:18:58.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.989+0000 7f7c41ffb640 10 monclient: dump: 2024-09-17T18:18:58.005 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:18:58.005 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:58.005 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:58.005 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:58.005 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:58.005 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:18:58.005 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:58.005 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:58.006 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:58.006 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:18:58.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.989+0000 7f7c41ffb640 10 monclient: _finish_auth 0 2024-09-17T18:18:58.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.989+0000 7f7c41ffb640 10 monclient: _check_auth_tickets 2024-09-17T18:18:58.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.989+0000 7f7c41ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:18:58.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.989+0000 7f7c53305640 5 monclient: authenticate success, global_id 4181 2024-09-17T18:18:58.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.989+0000 7f7c41ffb640 1 -- 172.21.15.121:0/4076937458 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c4400a800 con 0x7f7c4c138c40 2024-09-17T18:18:58.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.989+0000 7f7c41ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:18:58.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.989+0000 7f7c53305640 10 monclient: _renew_subs 2024-09-17T18:18:58.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.989+0000 7f7c41ffb640 1 -- 172.21.15.121:0/4076937458 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4400abf0 con 0x7f7c4c138c40 2024-09-17T18:18:58.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.989+0000 7f7c53305640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:18:58.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.989+0000 7f7c53305640 1 -- 172.21.15.121:0/4076937458 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c4c0a2380 con 0x7f7c4c138c40 2024-09-17T18:18:58.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.989+0000 7f7c41ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:58.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.989+0000 7f7c41ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:18:58.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.989+0000 7f7c437fe640 4 set_mon_vals no callback set 2024-09-17T18:18:58.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.989+0000 7f7c41ffb640 10 monclient: dump: 2024-09-17T18:18:58.007 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:18:58.007 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:58.007 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:58.007 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:58.007 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:58.007 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:18:58.007 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:58.007 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:58.007 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:58.007 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:18:58.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.989+0000 7f7c53305640 10 client.4181.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:18:58.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.989+0000 7f7c53305640 10 monclient: _renew_subs 2024-09-17T18:18:58.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.989+0000 7f7c53305640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:18:58.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.989+0000 7f7c53305640 1 -- 172.21.15.121:0/4076937458 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c4c127470 con 0x7f7c4c138c40 2024-09-17T18:18:58.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.989+0000 7f7c53305640 1 librados: init done 2024-09-17T18:18:58.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.990+0000 7f7c41ffb640 1 -- 172.21.15.121:0/4076937458 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 4) ==== 50940+0+0 (secure 0 0 0) 0x7f7c4401a070 con 0x7f7c4c138c40 2024-09-17T18:18:58.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.990+0000 7f7c53305640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:18:58.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.990+0000 7f7c53305640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:18:58.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.990+0000 7f7c53305640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:18:58.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.990+0000 7f7c53305640 1 -- 172.21.15.121:0/4076937458 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c4c1390b0 con 0x7f7c4c138c40 2024-09-17T18:18:58.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.990+0000 7f7c41ffb640 1 --2- 172.21.15.121:0/4076937458 >> v2:172.21.15.173:6812/2423786037 conn(0x7f7c300400e0 0x7f7c300425a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:58.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.990+0000 7f7c41ffb640 1 -- 172.21.15.121:0/4076937458 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(12..12 src has 1..12) ==== 2767+0+0 (secure 0 0 0) 0x7f7c440522a0 con 0x7f7c4c138c40 2024-09-17T18:18:58.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.990+0000 7f7c41ffb640 10 client.4181.objecter ms_dispatch 0x7f7c4c000b90 osd_map(12..12 src has 1..12) 2024-09-17T18:18:58.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.990+0000 7f7c41ffb640 3 client.4181.objecter handle_osd_map got epochs [12,12] > 0 2024-09-17T18:18:58.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.990+0000 7f7c41ffb640 3 client.4181.objecter handle_osd_map decoding full epoch 12 2024-09-17T18:18:58.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.990+0000 7f7c41ffb640 20 client.4181.objecter dump_active .. 0 homeless 2024-09-17T18:18:58.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.991+0000 7f7c510a1640 1 --2- 172.21.15.121:0/4076937458 >> v2:172.21.15.173:6812/2423786037 conn(0x7f7c300400e0 0x7f7c300425a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:58.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.991+0000 7f7c510a1640 10 monclient: get_auth_request con 0x7f7c300400e0 auth_method 0 2024-09-17T18:18:58.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.993+0000 7f7c510a1640 1 --2- 172.21.15.121:0/4076937458 >> v2:172.21.15.173:6812/2423786037 conn(0x7f7c300400e0 0x7f7c300425a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7c48034000 tx=0x7f7c4802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:18:58.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.993+0000 7f7c41ffb640 10 client.4181.objecter ms_handle_connect 0x7f7c300400e0 2024-09-17T18:18:58.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.995+0000 7f7c41ffb640 1 -- 172.21.15.121:0/4076937458 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7c44051490 con 0x7f7c4c138c40 2024-09-17T18:18:58.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.995+0000 7f7c41ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:18:58.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:57.995+0000 7f7c41ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:18:58.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.169+0000 7f7c53305640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-17T18:18:58.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.169+0000 7f7c53305640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-17T18:18:58.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.169+0000 7f7c53305640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:18:58.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.169+0000 7f7c53305640 1 -- 172.21.15.121:0/4076937458 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7f7c4c0707b0 con 0x7f7c4c138c40 2024-09-17T18:18:58.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.523+0000 7f7c41ffb640 1 -- 172.21.15.121:0/4076937458 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v13) ==== 109+0+0 (secure 0 0 0) 0x7f7c44007070 con 0x7f7c4c138c40 2024-09-17T18:18:58.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.523+0000 7f7c41ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-17T18:18:58.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.523+0000 7f7c41ffb640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-09-17T18:18:58.525 INFO:teuthology.orchestra.run.smithi121.stderr:pool 'rbd' created 2024-09-17T18:18:58.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.525+0000 7f7c53305640 10 librados: watch_flush enter 2024-09-17T18:18:58.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.526+0000 7f7c53305640 10 librados: watch_flush exit 2024-09-17T18:18:58.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.526+0000 7f7c53305640 20 client.4181.objecter shutdown clearing up homeless session... 2024-09-17T18:18:58.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.526+0000 7f7c53305640 10 client.4181.objecter successfully canceled tick 2024-09-17T18:18:58.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.526+0000 7f7c53305640 1 -- 172.21.15.121:0/4076937458 >> v2:172.21.15.173:6812/2423786037 conn(0x7f7c300400e0 msgr2=0x7f7c300425a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:58.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.526+0000 7f7c53305640 1 --2- 172.21.15.121:0/4076937458 >> v2:172.21.15.173:6812/2423786037 conn(0x7f7c300400e0 0x7f7c300425a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7c48034000 tx=0x7f7c4802e000 comp rx=0 tx=0).stop 2024-09-17T18:18:58.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.526+0000 7f7c53305640 10 monclient: shutdown 2024-09-17T18:18:58.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.526+0000 7f7c53305640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:18:58.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.526+0000 7f7c53305640 1 -- 172.21.15.121:0/4076937458 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c4c138c40 msgr2=0x7f7c4c123650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:58.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.526+0000 7f7c53305640 1 --2- 172.21.15.121:0/4076937458 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c4c138c40 0x7f7c4c123650 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7c4400c780 tx=0x7f7c4400cc40 comp rx=0 tx=0).stop 2024-09-17T18:18:58.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.526+0000 7f7c53305640 1 -- 172.21.15.121:0/4076937458 shutdown_connections 2024-09-17T18:18:58.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.526+0000 7f7c53305640 1 --2- 172.21.15.121:0/4076937458 >> v2:172.21.15.173:6812/2423786037 conn(0x7f7c300400e0 0x7f7c300425a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:58.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.526+0000 7f7c53305640 1 --2- 172.21.15.121:0/4076937458 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c4c138c40 0x7f7c4c123650 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:58.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.526+0000 7f7c53305640 1 --2- 172.21.15.121:0/4076937458 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c4c0a7090 0x7f7c4c1ce480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:58.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.526+0000 7f7c53305640 1 --2- 172.21.15.121:0/4076937458 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c4c0a5b90 0x7f7c4c1cdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:58.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.526+0000 7f7c53305640 1 -- 172.21.15.121:0/4076937458 >> 172.21.15.121:0/4076937458 conn(0x7f7c4c0a7c90 msgr2=0x7f7c4c0a45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:18:58.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.526+0000 7f7c53305640 1 -- 172.21.15.121:0/4076937458 shutdown_connections 2024-09-17T18:18:58.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.526+0000 7f7c53305640 1 -- 172.21.15.121:0/4076937458 wait complete. 2024-09-17T18:18:58.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.526+0000 7f7c53305640 1 librados: shutdown 2024-09-17T18:18:58.538 DEBUG:teuthology.orchestra.run.smithi121:> rbd --cluster ceph pool init rbd 2024-09-17T18:18:58.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.606+0000 7ff0a8c3c140 10 monclient: get_monmap_and_config 2024-09-17T18:18:58.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.606+0000 7ff0a8c3c140 10 monclient: build_initial_monmap 2024-09-17T18:18:58.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.606+0000 7ff0a8c3c140 10 monclient: monmap: 2024-09-17T18:18:58.608 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:18:58.609 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:58.609 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:58.607295+0000 2024-09-17T18:18:58.609 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:58.607295+0000 2024-09-17T18:18:58.609 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:18:58.609 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:18:58.609 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:18:58.609 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:18:58.609 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:18:58.609 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:18:58.609 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:18:58.609 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:18:58.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.607+0000 7ff0a8c3c140 1 Processor -- start 2024-09-17T18:18:58.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.607+0000 7ff0a8c3c140 1 -- start start 2024-09-17T18:18:58.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.607+0000 7ff0a8c3c140 10 monclient: init 2024-09-17T18:18:58.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.607+0000 7ff0a8c3c140 10 monclient: _reopen_session rank -1 2024-09-17T18:18:58.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.607+0000 7ff0a8c3c140 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-17T18:18:58.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.607+0000 7ff0a8c3c140 1 --2- >> v2:172.21.15.121:3300/0 conn(0x557f355317a0 0x557f3552c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:58.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.607+0000 7ff0a8c3c140 10 monclient(hunting): picked mon.noname-a con 0x557f355317a0 addr v2:172.21.15.121:3300/0 2024-09-17T18:18:58.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.607+0000 7ff0a8c3c140 10 monclient(hunting): picked mon.noname-a-legacy con 0x557f35531b80 addr v1:172.21.15.121:6789/0 2024-09-17T18:18:58.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.607+0000 7ff0a8c3c140 1 --2- >> v2:172.21.15.173:3300/0 conn(0x557f355cbee0 0x557f355ce2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:58.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.607+0000 7ff0a8c3c140 10 monclient(hunting): picked mon.noname-c con 0x557f355cbee0 addr v2:172.21.15.173:3300/0 2024-09-17T18:18:58.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.607+0000 7ff0a8c3c140 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x557f353add90 con 0x557f35531b80 2024-09-17T18:18:58.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.607+0000 7ff0a8c3c140 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:58.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.607+0000 7ff0a8c3c140 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x557f353fda80 con 0x557f355317a0 2024-09-17T18:18:58.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.607+0000 7ff0a8c3c140 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:58.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.607+0000 7ff0a8c3c140 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x557f3540e650 con 0x557f355cbee0 2024-09-17T18:18:58.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.607+0000 7ff0a8c3c140 10 monclient(hunting): _renew_subs 2024-09-17T18:18:58.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.607+0000 7ff0a8c3c140 10 monclient(hunting): authenticate will time out at 742.323912s 2024-09-17T18:18:58.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.608+0000 7ff0a6c8c640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x557f355317a0 0x557f3552c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:58.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.608+0000 7ff0a6c8c640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x557f355317a0 0x557f3552c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39878/0 (socket says 172.21.15.121:39878) 2024-09-17T18:18:58.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.608+0000 7ff0a6c8c640 1 -- 172.21.15.121:0/1499333043 learned_addr learned my addr 172.21.15.121:0/1499333043 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:18:58.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.608+0000 7ff0a6c8c640 10 monclient(hunting): get_auth_request con 0x557f355317a0 auth_method 0 2024-09-17T18:18:58.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.608+0000 7ff0a6c8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:58.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.608+0000 7ff0a6c8c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:58.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.608+0000 7ff0a6c8c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:58.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.608+0000 7ff0a6c8c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:58.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.608+0000 7ff0a6c8c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:58.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.608+0000 7ff0a6c8c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:18:58.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.608+0000 7ff0a6c8c640 10 monclient(hunting): handle_auth_done global_id 4197 payload 274 2024-09-17T18:18:58.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.608+0000 7ff0a6c8c640 1 -- 172.21.15.121:0/1499333043 >> v2:172.21.15.173:3300/0 conn(0x557f355cbee0 msgr2=0x557f355ce2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:18:58.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.608+0000 7ff0a6c8c640 1 --2- 172.21.15.121:0/1499333043 >> v2:172.21.15.173:3300/0 conn(0x557f355cbee0 0x557f355ce2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:58.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.608+0000 7ff0a6c8c640 1 -- 172.21.15.121:0/1499333043 >> v1:172.21.15.121:6789/0 conn(0x557f35531b80 legacy=0x557f35531fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:18:58.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.608+0000 7ff0a6c8c640 10 monclient: _finish_hunting 0 2024-09-17T18:18:58.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.608+0000 7ff0a6c8c640 1 monclient: found mon.noname-a 2024-09-17T18:18:58.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.608+0000 7ff0a6c8c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:18:58.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.608+0000 7ff0a6c8c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:18:58.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.608+0000 7ff0a6c8c640 1 -- 172.21.15.121:0/1499333043 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f353c5410 con 0x557f355317a0 2024-09-17T18:18:58.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.612+0000 7ff0a6c8c640 1 -- 172.21.15.121:0/1499333043 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557f355317a0 msgr2=0x557f3552c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:18:58.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.612+0000 7ff0a6c8c640 1 --2- 172.21.15.121:0/1499333043 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557f355317a0 0x557f3552c390 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x557f35535d90 tx=0x7ff090045bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7da9a51322847b48 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:18:58.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.612+0000 7ff0a5c8a640 1 -- 172.21.15.121:0/1499333043 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff090054070 con 0x557f355317a0 2024-09-17T18:18:58.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.612+0000 7ff0a5c8a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:58.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.612+0000 7ff0a5c8a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:18:58.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.612+0000 7ff0a5c8a640 10 monclient: dump: 2024-09-17T18:18:58.618 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:18:58.618 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:58.618 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:58.618 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:58.618 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:58.618 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:18:58.618 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:58.618 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:58.618 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:58.618 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:18:58.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.612+0000 7ff0a5c8a640 10 monclient: _finish_auth 0 2024-09-17T18:18:58.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.612+0000 7ff0a5c8a640 10 monclient: _check_auth_tickets 2024-09-17T18:18:58.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.612+0000 7ff0a5c8a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:18:58.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.612+0000 7ff0a5c8a640 1 -- 172.21.15.121:0/1499333043 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0900506d0 con 0x557f355317a0 2024-09-17T18:18:58.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.612+0000 7ff0a8c3c140 5 monclient: authenticate success, global_id 4197 2024-09-17T18:18:58.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.612+0000 7ff0a8c3c140 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:18:58.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.612+0000 7ff0a5c8a640 10 monclient: handle_config config(0 keys) 2024-09-17T18:18:58.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.612+0000 7ff0a5c8a640 1 -- 172.21.15.121:0/1499333043 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff090050aa0 con 0x557f355317a0 2024-09-17T18:18:58.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.612+0000 7ff0a8c3c140 10 monclient: get_monmap_and_config success 2024-09-17T18:18:58.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.612+0000 7ff0a8c3c140 4 set_mon_vals no callback set 2024-09-17T18:18:58.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.612+0000 7ff0a5c8a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:58.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.612+0000 7ff0a5c8a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:18:58.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.612+0000 7ff0a5c8a640 10 monclient: dump: 2024-09-17T18:18:58.619 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:18:58.619 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:58.619 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:58.619 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:58.620 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:58.620 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:18:58.620 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:58.620 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:58.620 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:58.620 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:18:58.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.612+0000 7ff0a8c3c140 10 monclient: shutdown 2024-09-17T18:18:58.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.612+0000 7ff0a8c3c140 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:18:58.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.612+0000 7ff0a8c3c140 1 -- 172.21.15.121:0/1499333043 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557f355317a0 msgr2=0x557f3552c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:58.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.612+0000 7ff0a8c3c140 1 --2- 172.21.15.121:0/1499333043 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557f355317a0 0x557f3552c390 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x557f35535d90 tx=0x7ff090045bb0 comp rx=0 tx=0).stop 2024-09-17T18:18:58.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.612+0000 7ff0a8c3c140 1 -- 172.21.15.121:0/1499333043 shutdown_connections 2024-09-17T18:18:58.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.612+0000 7ff0a8c3c140 1 --2- 172.21.15.121:0/1499333043 >> v2:172.21.15.173:3300/0 conn(0x557f355cbee0 0x557f355ce2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:58.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.612+0000 7ff0a8c3c140 1 --2- 172.21.15.121:0/1499333043 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557f355317a0 0x557f3552c390 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:58.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.612+0000 7ff0a8c3c140 1 -- 172.21.15.121:0/1499333043 >> 172.21.15.121:0/1499333043 conn(0x557f35533100 msgr2=0x557f355143a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:18:58.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.612+0000 7ff0a8c3c140 1 -- 172.21.15.121:0/1499333043 shutdown_connections 2024-09-17T18:18:58.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.612+0000 7ff0a8c3c140 1 -- 172.21.15.121:0/1499333043 wait complete. 2024-09-17T18:18:58.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.613+0000 7ff0a8c3c140 10 monclient: build_initial_monmap 2024-09-17T18:18:58.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.613+0000 7ff0a8c3c140 10 monclient: monmap: 2024-09-17T18:18:58.621 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:18:58.621 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:18:58.621 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:18:58.621 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:18:58.622 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:18:58.622 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:18:58.622 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:18:58.622 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:18:58.622 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:18:58.622 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:18:58.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.613+0000 7ff0a8c3c140 1 librados: starting msgr at 2024-09-17T18:18:58.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.613+0000 7ff0a8c3c140 1 librados: starting objecter 2024-09-17T18:18:58.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.613+0000 7ff0a8c3c140 1 Processor -- start 2024-09-17T18:18:58.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.613+0000 7ff0a8c3c140 1 -- start start 2024-09-17T18:18:58.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.613+0000 7ff0a8c3c140 1 librados: setting wanted keys 2024-09-17T18:18:58.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.613+0000 7ff0a8c3c140 1 librados: calling monclient init 2024-09-17T18:18:58.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.613+0000 7ff0a8c3c140 10 monclient: init 2024-09-17T18:18:58.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a8c3c140 10 monclient: _reopen_session rank -1 2024-09-17T18:18:58.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a8c3c140 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T18:18:58.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a8c3c140 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557f355317a0 0x557f355aeac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:58.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a8c3c140 10 monclient(hunting): picked mon.noname-c con 0x557f355317a0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:18:58.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a8c3c140 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557f35531b80 0x557f355af000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:58.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a8c3c140 10 monclient(hunting): picked mon.noname-a con 0x557f35531b80 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:18:58.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a8c3c140 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557f355cbee0 0x557f355af540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:58.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a8c3c140 10 monclient(hunting): picked mon.noname-b con 0x557f355cbee0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:18:58.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a8c3c140 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:58.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a8c3c140 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x557f353a3390 con 0x557f35531b80 2024-09-17T18:18:58.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a8c3c140 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:58.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a8c3c140 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x557f35401890 con 0x557f355cbee0 2024-09-17T18:18:58.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a8c3c140 10 monclient(hunting): start opening mon connection 2024-09-17T18:18:58.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a8c3c140 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x557f35514120 con 0x557f355317a0 2024-09-17T18:18:58.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a8c3c140 10 monclient(hunting): _renew_subs 2024-09-17T18:18:58.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a8c3c140 10 monclient(hunting): authenticate will time out at 742.330163s 2024-09-17T18:18:58.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a6c8c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557f355317a0 0x557f355aeac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:58.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a648b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557f35531b80 0x557f355af000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:58.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a6c8c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557f355317a0 0x557f355aeac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:46576/0 (socket says 172.21.15.121:46576) 2024-09-17T18:18:58.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a6c8c640 1 -- 172.21.15.121:0/1646816398 learned_addr learned my addr 172.21.15.121:0/1646816398 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:18:58.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a6c8c640 10 monclient(hunting): get_auth_request con 0x557f355317a0 auth_method 0 2024-09-17T18:18:58.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a6c8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:58.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a6c8c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:58.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a6c8c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:58.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a648b640 10 monclient(hunting): get_auth_request con 0x557f35531b80 auth_method 0 2024-09-17T18:18:58.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a648b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:18:58.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a648b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:18:58.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a648b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:18:58.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a648b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:58.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a648b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:58.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a648b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:18:58.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a6c8c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:18:58.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a6c8c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:18:58.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a6c8c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:18:58.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a7eee640 1 --2- 172.21.15.121:0/1646816398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557f355cbee0 0x557f355af540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:58.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a648b640 10 monclient(hunting): handle_auth_done global_id 4200 payload 824 2024-09-17T18:18:58.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a648b640 1 -- 172.21.15.121:0/1646816398 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557f355317a0 msgr2=0x557f355aeac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:58.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a648b640 1 --2- 172.21.15.121:0/1646816398 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557f355317a0 0x557f355aeac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:58.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a648b640 1 -- 172.21.15.121:0/1646816398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557f355cbee0 msgr2=0x557f355af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:18:58.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a648b640 1 --2- 172.21.15.121:0/1646816398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557f355cbee0 0x557f355af540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:18:58.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a648b640 10 monclient: _finish_hunting 0 2024-09-17T18:18:58.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a648b640 1 monclient: found mon.noname-a 2024-09-17T18:18:58.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a648b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:18:58.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a648b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:18:58.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a648b640 1 -- 172.21.15.121:0/1646816398 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f3537c720 con 0x557f35531b80 2024-09-17T18:18:58.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a7eee640 10 monclient: get_auth_request con 0x557f355cbee0 auth_method 0 2024-09-17T18:18:58.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a6c8c640 1 --2- 172.21.15.121:0/1646816398 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557f355317a0 0x557f355aeac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:18:58.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.614+0000 7ff0a7eee640 1 --2- 172.21.15.121:0/1646816398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557f355cbee0 0x557f355af540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T18:18:58.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.616+0000 7ff0a648b640 1 -- 172.21.15.121:0/1646816398 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557f35531b80 msgr2=0x557f355af000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:18:58.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.616+0000 7ff0a648b640 1 --2- 172.21.15.121:0/1646816398 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557f35531b80 0x557f355af000 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff09400bb10 tx=0x7ff094009160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:18:58.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.616+0000 7ff09effd640 10 client.?.objecter ms_handle_connect 0x557f35531b80 2024-09-17T18:18:58.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.616+0000 7ff09effd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:18:58.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.616+0000 7ff09effd640 1 -- 172.21.15.121:0/1646816398 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff094013070 con 0x557f35531b80 2024-09-17T18:18:58.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.616+0000 7ff09effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:58.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.616+0000 7ff09effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:18:58.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.616+0000 7ff09effd640 10 monclient: dump: 2024-09-17T18:18:58.629 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:18:58.629 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:58.629 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:58.629 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:58.630 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:58.630 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:18:58.630 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:58.630 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:58.630 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:58.630 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:18:58.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.616+0000 7ff09effd640 10 monclient: _finish_auth 0 2024-09-17T18:18:58.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.616+0000 7ff09effd640 10 monclient: _check_auth_tickets 2024-09-17T18:18:58.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.616+0000 7ff09effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:18:58.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.616+0000 7ff09effd640 1 -- 172.21.15.121:0/1646816398 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff09400f400 con 0x557f35531b80 2024-09-17T18:18:58.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.616+0000 7ff0a8c3c140 5 monclient: authenticate success, global_id 4200 2024-09-17T18:18:58.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.616+0000 7ff09effd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:18:58.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.616+0000 7ff09effd640 1 -- 172.21.15.121:0/1646816398 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff09400f7f0 con 0x557f35531b80 2024-09-17T18:18:58.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.616+0000 7ff0a8c3c140 10 monclient: _renew_subs 2024-09-17T18:18:58.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.616+0000 7ff0a8c3c140 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:18:58.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.616+0000 7ff0a8c3c140 1 -- 172.21.15.121:0/1646816398 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x557f355b2890 con 0x557f35531b80 2024-09-17T18:18:58.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.616+0000 7ff09effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:18:58.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.617+0000 7ff09effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:18:58.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.617+0000 7ff09effd640 10 monclient: dump: 2024-09-17T18:18:58.631 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:18:58.631 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:18:58.631 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:58.631 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:18:58.631 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:18:58.631 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:18:58.631 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:18:58.631 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:18:58.631 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:18:58.632 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:18:58.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.616+0000 7ff0a5489640 4 set_mon_vals no callback set 2024-09-17T18:18:58.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.617+0000 7ff0a8c3c140 10 client.4200.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:18:58.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.617+0000 7ff0a8c3c140 10 monclient: _renew_subs 2024-09-17T18:18:58.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.617+0000 7ff0a8c3c140 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:18:58.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.617+0000 7ff0a8c3c140 1 -- 172.21.15.121:0/1646816398 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f3552d060 con 0x557f35531b80 2024-09-17T18:18:58.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.617+0000 7ff0a8c3c140 1 librados: init done 2024-09-17T18:18:58.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.617+0000 7ff0a8c3c140 10 librados: wait_for_osdmap waiting 2024-09-17T18:18:58.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.617+0000 7ff09effd640 1 -- 172.21.15.121:0/1646816398 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 4) ==== 50940+0+0 (secure 0 0 0) 0x7ff094016070 con 0x557f35531b80 2024-09-17T18:18:58.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.618+0000 7ff09effd640 1 --2- 172.21.15.121:0/1646816398 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff08c03e260 0x7ff08c040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:58.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.618+0000 7ff09effd640 1 -- 172.21.15.121:0/1646816398 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 3484+0+0 (secure 0 0 0) 0x7ff094051620 con 0x557f35531b80 2024-09-17T18:18:58.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.618+0000 7ff09effd640 10 client.4200.objecter ms_dispatch 0x557f35415ff0 osd_map(13..13 src has 1..13) 2024-09-17T18:18:58.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.618+0000 7ff09effd640 3 client.4200.objecter handle_osd_map got epochs [13,13] > 0 2024-09-17T18:18:58.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.618+0000 7ff09effd640 3 client.4200.objecter handle_osd_map decoding full epoch 13 2024-09-17T18:18:58.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.618+0000 7ff09effd640 20 client.4200.objecter dump_active .. 0 homeless 2024-09-17T18:18:58.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.618+0000 7ff0a8c3c140 10 librados: wait_for_osdmap done waiting 2024-09-17T18:18:58.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.618+0000 7ff0a8c3c140 10 start_mon_command cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-17T18:18:58.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.618+0000 7ff0a8c3c140 10 monclient: _send_command 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-17T18:18:58.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.618+0000 7ff0a8c3c140 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:18:58.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.618+0000 7ff0a8c3c140 1 -- 172.21.15.121:0/1646816398 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) -- 0x557f355b1350 con 0x557f35531b80 2024-09-17T18:18:58.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.618+0000 7ff0a6c8c640 1 --2- 172.21.15.121:0/1646816398 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff08c03e260 0x7ff08c040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:58.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.618+0000 7ff0a6c8c640 10 monclient: get_auth_request con 0x7ff08c03e260 auth_method 0 2024-09-17T18:18:58.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.621+0000 7ff09effd640 10 client.4200.objecter ms_handle_connect 0x7ff08c03e260 2024-09-17T18:18:58.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:58.621+0000 7ff0a6c8c640 1 --2- 172.21.15.121:0/1646816398 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff08c03e260 0x7ff08c040720 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff09000a190 tx=0x7ff090007030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:18:59.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.517+0000 7ff09effd640 1 -- 172.21.15.121:0/1646816398 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v14) ==== 141+0+0 (secure 0 0 0) 0x7ff094017cb0 con 0x557f35531b80 2024-09-17T18:18:59.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.517+0000 7ff09effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-17T18:18:59.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.517+0000 7ff09effd640 10 monclient: _finish_command 1 = system:0 enabled application 'rbd' on pool 'rbd' 2024-09-17T18:18:59.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.517+0000 7ff0a8c3c140 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:18:59.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.517+0000 7ff0a8c3c140 1 -- 172.21.15.121:0/1646816398 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff0900506d0 con 0x557f35531b80 2024-09-17T18:18:59.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.519+0000 7ff09effd640 1 -- 172.21.15.121:0/1646816398 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_get_version_reply(handle=1 version=14) ==== 24+0+0 (secure 0 0 0) 0x7ff09400ea70 con 0x557f35531b80 2024-09-17T18:18:59.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.520+0000 7ff09effd640 10 monclient: handle_get_version_reply finishing 1 version 14 2024-09-17T18:18:59.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.520+0000 7ff0a5489640 10 client.4200.objecter _get_latest_version latest 14, waiting 2024-09-17T18:18:59.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.520+0000 7ff0a5489640 10 client.4200.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:18:59.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.520+0000 7ff0a5489640 10 monclient: _renew_subs 2024-09-17T18:18:59.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.520+0000 7ff0a5489640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:18:59.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.520+0000 7ff0a5489640 1 -- 172.21.15.121:0/1646816398 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=14}) -- 0x7ff080003c90 con 0x557f35531b80 2024-09-17T18:18:59.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.520+0000 7ff09effd640 1 -- 172.21.15.121:0/1646816398 <== mon.0 v2:172.21.15.121:3300/0 8 ==== osd_map(14..14 src has 1..14) ==== 1028+0+0 (secure 0 0 0) 0x7ff09400e6e0 con 0x557f35531b80 2024-09-17T18:18:59.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.520+0000 7ff09effd640 10 client.4200.objecter ms_dispatch 0x557f35415ff0 osd_map(14..14 src has 1..14) 2024-09-17T18:18:59.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.520+0000 7ff09effd640 3 client.4200.objecter handle_osd_map got epochs [14,14] > 13 2024-09-17T18:18:59.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.520+0000 7ff09effd640 3 client.4200.objecter handle_osd_map decoding incremental epoch 14 2024-09-17T18:18:59.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.520+0000 7ff09effd640 20 client.4200.objecter dump_active .. 0 homeless 2024-09-17T18:18:59.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.520+0000 7ff0a8c3c140 10 librados: create oid=rbd_trash nspace= 2024-09-17T18:18:59.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.520+0000 7ff0a8c3c140 10 client.4200.objecter _op_submit op 0x557f355dc390 2024-09-17T18:18:59.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.520+0000 7ff0a8c3c140 20 client.4200.objecter _calc_target epoch 14 base rbd_trash @2 precalc_pgid 0 pgid 0.0 is_write 2024-09-17T18:18:59.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.520+0000 7ff0a8c3c140 20 client.4200.objecter _calc_target target rbd_trash @2 -> pgid 2.5a749323 2024-09-17T18:18:59.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.520+0000 7ff0a8c3c140 10 client.4200.objecter _calc_target raw pgid 2.5a749323 -> actual 2.3 acting [5,2] primary 5 2024-09-17T18:18:59.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.520+0000 7ff0a8c3c140 1 --2- 172.21.15.121:0/1646816398 >> v2:172.21.15.173:6808/1801998802 conn(0x557f355dcec0 0x557f355df2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:18:59.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.520+0000 7ff0a8c3c140 20 client.4200.objecter _get_session s=0x557f355dc840 osd=5 3 2024-09-17T18:18:59.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.520+0000 7ff0a8c3c140 10 client.4200.objecter _op_submit oid rbd_trash '@2' '@2' [create] tid 1 osd.5 2024-09-17T18:18:59.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.520+0000 7ff0a8c3c140 20 client.4200.objecter get_session s=0x557f355dc840 osd=5 3 2024-09-17T18:18:59.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.520+0000 7ff0a8c3c140 15 client.4200.objecter _session_op_assign 5 1 2024-09-17T18:18:59.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.520+0000 7ff0a8c3c140 15 client.4200.objecter _send_op 1 to 2.3 on osd.5 2024-09-17T18:18:59.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.520+0000 7ff0a8c3c140 1 -- 172.21.15.121:0/1646816398 --> v2:172.21.15.173:6808/1801998802 -- osd_op(unknown.0.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14) -- 0x557f355df7f0 con 0x557f355dcec0 2024-09-17T18:18:59.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.520+0000 7ff0a8c3c140 20 client.4200.objecter put_session s=0x557f355dc840 osd=5 4 2024-09-17T18:18:59.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.520+0000 7ff0a8c3c140 5 client.4200.objecter 1 in flight 2024-09-17T18:18:59.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.520+0000 7ff0a7eee640 1 --2- 172.21.15.121:0/1646816398 >> v2:172.21.15.173:6808/1801998802 conn(0x557f355dcec0 0x557f355df2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:18:59.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.520+0000 7ff0a7eee640 10 monclient: get_auth_request con 0x557f355dcec0 auth_method 0 2024-09-17T18:18:59.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.525+0000 7ff0a7eee640 1 --2- 172.21.15.121:0/1646816398 >> v2:172.21.15.173:6808/1801998802 conn(0x557f355dcec0 0x557f355df2b0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:18:59.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.525+0000 7ff09effd640 10 client.4200.objecter ms_handle_connect 0x557f355dcec0 2024-09-17T18:18:59.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.529+0000 7ff0a7eee640 1 -- 172.21.15.121:0/1646816398 <== osd.5 v2:172.21.15.173:6808/1801998802 1 ==== osd_op_reply(1 rbd_trash [create] v14'1 uv1 ondisk = 0) ==== 153+0+0 (crc 0 0 0) 0x7ff098009620 con 0x557f355dcec0 2024-09-17T18:18:59.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.529+0000 7ff0a7eee640 10 client.4200.objecter ms_dispatch 0x557f35415ff0 osd_op_reply(1 rbd_trash [create] v14'1 uv1 ondisk = 0) 2024-09-17T18:18:59.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.529+0000 7ff0a7eee640 10 client.4200.objecter in handle_osd_op_reply 2024-09-17T18:18:59.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.529+0000 7ff0a7eee640 7 client.4200.objecter handle_osd_op_reply 1 ondisk uv 1 in 2.3 attempt 0 2024-09-17T18:18:59.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.529+0000 7ff0a7eee640 10 client.4200.objecter op 0 rval 0 len 0 2024-09-17T18:18:59.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.529+0000 7ff0a7eee640 15 client.4200.objecter handle_osd_op_reply completed tid 1 2024-09-17T18:18:59.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.529+0000 7ff0a7eee640 15 client.4200.objecter _finish_op 1 2024-09-17T18:18:59.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.529+0000 7ff0a7eee640 20 client.4200.objecter put_session s=0x557f355dc840 osd=5 4 2024-09-17T18:18:59.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.529+0000 7ff0a7eee640 15 client.4200.objecter _session_op_remove 5 1 2024-09-17T18:18:59.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.529+0000 7ff0a7eee640 5 client.4200.objecter 0 in flight 2024-09-17T18:18:59.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.529+0000 7ff0a8c3c140 10 librados: Objecter returned from create r=0 2024-09-17T18:18:59.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.531+0000 7ff0a8c3c140 10 client.4200.objecter _op_submit op 0x557f35632550 2024-09-17T18:18:59.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.531+0000 7ff0a8c3c140 20 client.4200.objecter _calc_target epoch 14 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_read 2024-09-17T18:18:59.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.531+0000 7ff0a8c3c140 20 client.4200.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-17T18:18:59.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a8c3c140 10 client.4200.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-09-17T18:18:59.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a8c3c140 20 client.4200.objecter _get_session s=0x557f355dc840 osd=5 3 2024-09-17T18:18:59.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a8c3c140 10 client.4200.objecter _op_submit oid rbd_info '@2' '@2' [call rbd.metadata_list in=33b] tid 2 osd.5 2024-09-17T18:18:59.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a8c3c140 20 client.4200.objecter get_session s=0x557f355dc840 osd=5 3 2024-09-17T18:18:59.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a8c3c140 15 client.4200.objecter _session_op_assign 5 2 2024-09-17T18:18:59.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a8c3c140 15 client.4200.objecter _send_op 2 to 2.2 on osd.5 2024-09-17T18:18:59.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a8c3c140 1 -- 172.21.15.121:0/1646816398 --> v2:172.21.15.173:6808/1801998802 -- osd_op(unknown.0.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e14) -- 0x557f35632aa0 con 0x557f355dcec0 2024-09-17T18:18:59.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a8c3c140 20 client.4200.objecter put_session s=0x557f355dc840 osd=5 4 2024-09-17T18:18:59.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a8c3c140 5 client.4200.objecter 1 in flight 2024-09-17T18:18:59.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a7eee640 1 -- 172.21.15.121:0/1646816398 <== osd.5 v2:172.21.15.173:6808/1801998802 2 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7ff098009620 con 0x557f355dcec0 2024-09-17T18:18:59.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a7eee640 10 client.4200.objecter ms_dispatch 0x557f35415ff0 osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-17T18:18:59.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a7eee640 10 client.4200.objecter in handle_osd_op_reply 2024-09-17T18:18:59.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a7eee640 7 client.4200.objecter handle_osd_op_reply 2 ondisk uv 0 in 2.2 attempt 0 2024-09-17T18:18:59.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a7eee640 10 client.4200.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-17T18:18:59.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a7eee640 10 client.4200.objecter op 0 rval 0 len 0 2024-09-17T18:18:59.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a7eee640 15 client.4200.objecter handle_osd_op_reply completed tid 2 2024-09-17T18:18:59.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a7eee640 15 client.4200.objecter _finish_op 2 2024-09-17T18:18:59.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a7eee640 20 client.4200.objecter put_session s=0x557f355dc840 osd=5 4 2024-09-17T18:18:59.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a7eee640 15 client.4200.objecter _session_op_remove 5 2 2024-09-17T18:18:59.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a7eee640 5 client.4200.objecter 0 in flight 2024-09-17T18:18:59.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a8c3c140 10 client.4200.objecter _op_submit op 0x557f356316e0 2024-09-17T18:18:59.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a8c3c140 20 client.4200.objecter _calc_target epoch 14 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_read 2024-09-17T18:18:59.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a8c3c140 20 client.4200.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-17T18:18:59.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a8c3c140 10 client.4200.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-09-17T18:18:59.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a8c3c140 20 client.4200.objecter _get_session s=0x557f355dc840 osd=5 3 2024-09-17T18:18:59.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a8c3c140 10 client.4200.objecter _op_submit oid rbd_info '@2' '@2' [read 0~0] tid 3 osd.5 2024-09-17T18:18:59.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a8c3c140 20 client.4200.objecter get_session s=0x557f355dc840 osd=5 3 2024-09-17T18:18:59.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a8c3c140 15 client.4200.objecter _session_op_assign 5 3 2024-09-17T18:18:59.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a8c3c140 15 client.4200.objecter _send_op 3 to 2.2 on osd.5 2024-09-17T18:18:59.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a8c3c140 1 -- 172.21.15.121:0/1646816398 --> v2:172.21.15.173:6808/1801998802 -- osd_op(unknown.0.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e14) -- 0x557f35631b90 con 0x557f355dcec0 2024-09-17T18:18:59.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a8c3c140 20 client.4200.objecter put_session s=0x557f355dc840 osd=5 4 2024-09-17T18:18:59.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a8c3c140 5 client.4200.objecter 1 in flight 2024-09-17T18:18:59.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a7eee640 1 -- 172.21.15.121:0/1646816398 <== osd.5 v2:172.21.15.173:6808/1801998802 3 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7ff098009620 con 0x557f355dcec0 2024-09-17T18:18:59.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a7eee640 10 client.4200.objecter ms_dispatch 0x557f35415ff0 osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-17T18:18:59.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a7eee640 10 client.4200.objecter in handle_osd_op_reply 2024-09-17T18:18:59.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a7eee640 7 client.4200.objecter handle_osd_op_reply 3 ondisk uv 0 in 2.2 attempt 0 2024-09-17T18:18:59.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a7eee640 10 client.4200.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-17T18:18:59.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a7eee640 10 client.4200.objecter op 0 rval 0 len 0 2024-09-17T18:18:59.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a7eee640 15 client.4200.objecter handle_osd_op_reply completed tid 3 2024-09-17T18:18:59.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a7eee640 15 client.4200.objecter _finish_op 3 2024-09-17T18:18:59.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a7eee640 20 client.4200.objecter put_session s=0x557f355dc840 osd=5 4 2024-09-17T18:18:59.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a7eee640 15 client.4200.objecter _session_op_remove 5 3 2024-09-17T18:18:59.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a7eee640 5 client.4200.objecter 0 in flight 2024-09-17T18:18:59.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a4c88640 10 client.4200.objecter allocate_selfmanaged_snap; pool: 2 2024-09-17T18:18:59.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a4c88640 10 client.4200.objecter pool_op_submit 4 2024-09-17T18:18:59.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a4c88640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:18:59.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:18:59.532+0000 7ff0a4c88640 1 -- 172.21.15.121:0/1646816398 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- pool_op(create unmanaged snap pool 2 tid 4 name v0) -- 0x7ff0840030d0 con 0x557f35531b80 2024-09-17T18:19:00.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.522+0000 7ff09effd640 1 -- 172.21.15.121:0/1646816398 <== mon.0 v2:172.21.15.121:3300/0 9 ==== pool_op_reply(tid 4 (0) Success v15) ==== 55+0+0 (secure 0 0 0) 0x7ff094020c00 con 0x557f35531b80 2024-09-17T18:19:00.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.522+0000 7ff09effd640 10 client.4200.objecter ms_dispatch 0x557f35415ff0 pool_op_reply(tid 4 (0) Success v15) 2024-09-17T18:19:00.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.522+0000 7ff09effd640 10 client.4200.objecter handle_pool_op_reply pool_op_reply(tid 4 (0) Success v15) 2024-09-17T18:19:00.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.522+0000 7ff09effd640 10 client.4200.objecter have request 4 at 0x7ff084003060 Op: create unmanaged snap 2024-09-17T18:19:00.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.522+0000 7ff09effd640 20 client.4200.objecter waiting for client to reach epoch 15 before calling back 2024-09-17T18:19:00.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.522+0000 7ff09effd640 10 client.4200.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:00.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.522+0000 7ff09effd640 10 monclient: _renew_subs 2024-09-17T18:19:00.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.522+0000 7ff09effd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:00.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.522+0000 7ff09effd640 1 -- 172.21.15.121:0/1646816398 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=15}) -- 0x7ff08c0443b0 con 0x557f35531b80 2024-09-17T18:19:00.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.522+0000 7ff09effd640 10 client.4200.objecter done 2024-09-17T18:19:00.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.522+0000 7ff09effd640 1 -- 172.21.15.121:0/1646816398 <== mon.0 v2:172.21.15.121:3300/0 10 ==== osd_map(15..15 src has 1..15) ==== 996+0+0 (secure 0 0 0) 0x7ff094020de0 con 0x557f35531b80 2024-09-17T18:19:00.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.522+0000 7ff09effd640 10 client.4200.objecter ms_dispatch 0x557f35415ff0 osd_map(15..15 src has 1..15) 2024-09-17T18:19:00.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.522+0000 7ff09effd640 3 client.4200.objecter handle_osd_map got epochs [15,15] > 14 2024-09-17T18:19:00.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.522+0000 7ff09effd640 3 client.4200.objecter handle_osd_map decoding incremental epoch 15 2024-09-17T18:19:00.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.522+0000 7ff09effd640 20 client.4200.objecter dump_active .. 0 homeless 2024-09-17T18:19:00.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.522+0000 7ff0a5489640 20 librados: queue_aio_write 0x557f3562dae0 completion 0x557f35383b50 write_seq 1 2024-09-17T18:19:00.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.522+0000 7ff0a5489640 10 client.4200.objecter _op_submit op 0x7ff080006380 2024-09-17T18:19:00.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.522+0000 7ff0a5489640 20 client.4200.objecter _calc_target epoch 15 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_write 2024-09-17T18:19:00.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.522+0000 7ff0a5489640 20 client.4200.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-17T18:19:00.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.522+0000 7ff0a5489640 10 client.4200.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-09-17T18:19:00.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.522+0000 7ff0a5489640 20 client.4200.objecter _get_session s=0x557f355dc840 osd=5 3 2024-09-17T18:19:00.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.522+0000 7ff0a5489640 10 client.4200.objecter _op_submit oid rbd_info '@2' '@2' [create,write 0~8 in=8b] tid 5 osd.5 2024-09-17T18:19:00.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.522+0000 7ff0a5489640 20 client.4200.objecter get_session s=0x557f355dc840 osd=5 3 2024-09-17T18:19:00.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.522+0000 7ff0a5489640 15 client.4200.objecter _session_op_assign 5 5 2024-09-17T18:19:00.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.522+0000 7ff0a5489640 15 client.4200.objecter _send_op 5 to 2.2 on osd.5 2024-09-17T18:19:00.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.522+0000 7ff0a5489640 1 -- 172.21.15.121:0/1646816398 --> v2:172.21.15.173:6808/1801998802 -- osd_op(unknown.0.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15) -- 0x7ff080006960 con 0x557f355dcec0 2024-09-17T18:19:00.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.522+0000 7ff0a5489640 20 client.4200.objecter put_session s=0x557f355dc840 osd=5 4 2024-09-17T18:19:00.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.522+0000 7ff0a5489640 5 client.4200.objecter 1 in flight 2024-09-17T18:19:00.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.528+0000 7ff0a7eee640 1 -- 172.21.15.121:0/1646816398 <== osd.5 v2:172.21.15.173:6808/1801998802 4 ==== osd_op_reply(5 rbd_info [create,write 0~8] v15'1 uv1 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7ff098009620 con 0x557f355dcec0 2024-09-17T18:19:00.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.528+0000 7ff0a7eee640 10 client.4200.objecter ms_dispatch 0x557f35415ff0 osd_op_reply(5 rbd_info [create,write 0~8] v15'1 uv1 ondisk = 0) 2024-09-17T18:19:00.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.528+0000 7ff0a7eee640 10 client.4200.objecter in handle_osd_op_reply 2024-09-17T18:19:00.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.528+0000 7ff0a7eee640 7 client.4200.objecter handle_osd_op_reply 5 ondisk uv 1 in 2.2 attempt 0 2024-09-17T18:19:00.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.528+0000 7ff0a7eee640 10 client.4200.objecter op 0 rval 0 len 0 2024-09-17T18:19:00.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.528+0000 7ff0a7eee640 10 client.4200.objecter op 1 rval 0 len 0 2024-09-17T18:19:00.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.528+0000 7ff0a7eee640 15 client.4200.objecter handle_osd_op_reply completed tid 5 2024-09-17T18:19:00.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.528+0000 7ff0a7eee640 15 client.4200.objecter _finish_op 5 2024-09-17T18:19:00.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.528+0000 7ff0a7eee640 20 client.4200.objecter put_session s=0x557f355dc840 osd=5 4 2024-09-17T18:19:00.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.528+0000 7ff0a7eee640 15 client.4200.objecter _session_op_remove 5 5 2024-09-17T18:19:00.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.528+0000 7ff0a7eee640 5 client.4200.objecter 0 in flight 2024-09-17T18:19:00.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.528+0000 7ff0a7eee640 20 librados: complete_aio_write 0x557f35383b50 2024-09-17T18:19:00.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.528+0000 7ff0a4c88640 10 client.4200.objecter delete_selfmanaged_snap; pool: 2; snap: 2 2024-09-17T18:19:00.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.528+0000 7ff0a4c88640 10 client.4200.objecter pool_op_submit 6 2024-09-17T18:19:00.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.528+0000 7ff0a4c88640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:00.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:00.528+0000 7ff0a4c88640 1 -- 172.21.15.121:0/1646816398 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- pool_op(delete unmanaged snap pool 2 tid 6 name v15) -- 0x7ff084004520 con 0x557f35531b80 2024-09-17T18:19:01.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.527+0000 7ff09effd640 1 -- 172.21.15.121:0/1646816398 <== mon.0 v2:172.21.15.121:3300/0 11 ==== pool_op_reply(tid 6 (0) Success v16) ==== 43+0+0 (secure 0 0 0) 0x7ff09404e070 con 0x557f35531b80 2024-09-17T18:19:01.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.527+0000 7ff09effd640 10 client.4200.objecter ms_dispatch 0x557f35415ff0 pool_op_reply(tid 6 (0) Success v16) 2024-09-17T18:19:01.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.527+0000 7ff09effd640 10 client.4200.objecter handle_pool_op_reply pool_op_reply(tid 6 (0) Success v16) 2024-09-17T18:19:01.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.527+0000 7ff09effd640 10 client.4200.objecter have request 6 at 0x7ff084003060 Op: delete unmanaged snap 2024-09-17T18:19:01.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.527+0000 7ff09effd640 20 client.4200.objecter waiting for client to reach epoch 16 before calling back 2024-09-17T18:19:01.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.527+0000 7ff09effd640 10 client.4200.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:01.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.527+0000 7ff09effd640 10 monclient: _renew_subs 2024-09-17T18:19:01.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.527+0000 7ff09effd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:01.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.527+0000 7ff09effd640 1 -- 172.21.15.121:0/1646816398 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=16}) -- 0x7ff08c044630 con 0x557f35531b80 2024-09-17T18:19:01.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.527+0000 7ff09effd640 10 client.4200.objecter done 2024-09-17T18:19:01.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.527+0000 7ff09effd640 1 -- 172.21.15.121:0/1646816398 <== mon.0 v2:172.21.15.121:3300/0 12 ==== osd_map(16..16 src has 1..16) ==== 654+0+0 (secure 0 0 0) 0x7ff09401f4e0 con 0x557f35531b80 2024-09-17T18:19:01.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.527+0000 7ff09effd640 10 client.4200.objecter ms_dispatch 0x557f35415ff0 osd_map(16..16 src has 1..16) 2024-09-17T18:19:01.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.527+0000 7ff09effd640 3 client.4200.objecter handle_osd_map got epochs [16,16] > 15 2024-09-17T18:19:01.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.527+0000 7ff09effd640 3 client.4200.objecter handle_osd_map decoding incremental epoch 16 2024-09-17T18:19:01.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.527+0000 7ff09effd640 20 client.4200.objecter dump_active .. 0 homeless 2024-09-17T18:19:01.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.527+0000 7ff0a5489640 20 librados: queue_aio_write 0x557f3562dae0 completion 0x7ff084001d60 write_seq 2 2024-09-17T18:19:01.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.527+0000 7ff0a5489640 10 client.4200.objecter _op_submit op 0x7ff080006380 2024-09-17T18:19:01.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.527+0000 7ff0a5489640 20 client.4200.objecter _calc_target epoch 16 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_write 2024-09-17T18:19:01.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.527+0000 7ff0a5489640 20 client.4200.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-17T18:19:01.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.527+0000 7ff0a5489640 10 client.4200.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-09-17T18:19:01.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.527+0000 7ff0a5489640 20 client.4200.objecter _get_session s=0x557f355dc840 osd=5 3 2024-09-17T18:19:01.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.527+0000 7ff0a5489640 10 client.4200.objecter _op_submit oid rbd_info '@2' '@2' [write 0~19 in=19b] tid 7 osd.5 2024-09-17T18:19:01.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.527+0000 7ff0a5489640 20 client.4200.objecter get_session s=0x557f355dc840 osd=5 3 2024-09-17T18:19:01.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.527+0000 7ff0a5489640 15 client.4200.objecter _session_op_assign 5 7 2024-09-17T18:19:01.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.527+0000 7ff0a5489640 15 client.4200.objecter _send_op 7 to 2.2 on osd.5 2024-09-17T18:19:01.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.527+0000 7ff0a5489640 1 -- 172.21.15.121:0/1646816398 --> v2:172.21.15.173:6808/1801998802 -- osd_op(unknown.0.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16) -- 0x7ff080006d00 con 0x557f355dcec0 2024-09-17T18:19:01.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.527+0000 7ff0a5489640 20 client.4200.objecter put_session s=0x557f355dc840 osd=5 4 2024-09-17T18:19:01.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.527+0000 7ff0a5489640 5 client.4200.objecter 1 in flight 2024-09-17T18:19:01.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.532+0000 7ff0a7eee640 1 -- 172.21.15.121:0/1646816398 <== osd.5 v2:172.21.15.173:6808/1801998802 5 ==== osd_op_reply(7 rbd_info [write 0~19] v16'2 uv2 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff098009620 con 0x557f355dcec0 2024-09-17T18:19:01.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.532+0000 7ff0a7eee640 10 client.4200.objecter ms_dispatch 0x557f35415ff0 osd_op_reply(7 rbd_info [write 0~19] v16'2 uv2 ondisk = 0) 2024-09-17T18:19:01.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.532+0000 7ff0a7eee640 10 client.4200.objecter in handle_osd_op_reply 2024-09-17T18:19:01.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.532+0000 7ff0a7eee640 7 client.4200.objecter handle_osd_op_reply 7 ondisk uv 2 in 2.2 attempt 0 2024-09-17T18:19:01.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.532+0000 7ff0a7eee640 10 client.4200.objecter op 0 rval 0 len 0 2024-09-17T18:19:01.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.532+0000 7ff0a7eee640 15 client.4200.objecter handle_osd_op_reply completed tid 7 2024-09-17T18:19:01.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.532+0000 7ff0a7eee640 15 client.4200.objecter _finish_op 7 2024-09-17T18:19:01.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.532+0000 7ff0a7eee640 20 client.4200.objecter put_session s=0x557f355dc840 osd=5 4 2024-09-17T18:19:01.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.532+0000 7ff0a7eee640 15 client.4200.objecter _session_op_remove 5 7 2024-09-17T18:19:01.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.532+0000 7ff0a7eee640 5 client.4200.objecter 0 in flight 2024-09-17T18:19:01.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.532+0000 7ff0a7eee640 20 librados: complete_aio_write 0x7ff084001d60 2024-09-17T18:19:01.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.532+0000 7ff0a8c3c140 10 librados: watch_flush enter 2024-09-17T18:19:01.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.532+0000 7ff0a8c3c140 10 librados: watch_flush exit 2024-09-17T18:19:01.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.532+0000 7ff0a8c3c140 10 client.4200.objecter close_session for osd.5 2024-09-17T18:19:01.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.532+0000 7ff0a8c3c140 1 -- 172.21.15.121:0/1646816398 >> v2:172.21.15.173:6808/1801998802 conn(0x557f355dcec0 msgr2=0x557f355df2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:01.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.532+0000 7ff0a8c3c140 1 --2- 172.21.15.121:0/1646816398 >> v2:172.21.15.173:6808/1801998802 conn(0x557f355dcec0 0x557f355df2b0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:01.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.532+0000 7ff0a8c3c140 20 client.4200.objecter put_session s=0x557f355dc840 osd=5 1 2024-09-17T18:19:01.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.532+0000 7ff0a8c3c140 20 client.4200.objecter shutdown clearing up homeless session... 2024-09-17T18:19:01.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.532+0000 7ff0a8c3c140 10 client.4200.objecter successfully canceled tick 2024-09-17T18:19:01.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.532+0000 7ff0a8c3c140 1 -- 172.21.15.121:0/1646816398 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff08c03e260 msgr2=0x7ff08c040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:01.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.532+0000 7ff0a8c3c140 1 --2- 172.21.15.121:0/1646816398 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff08c03e260 0x7ff08c040720 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff09000a190 tx=0x7ff090007030 comp rx=0 tx=0).stop 2024-09-17T18:19:01.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.532+0000 7ff0a8c3c140 10 monclient: shutdown 2024-09-17T18:19:01.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.532+0000 7ff0a8c3c140 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:01.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.532+0000 7ff0a8c3c140 1 -- 172.21.15.121:0/1646816398 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557f35531b80 msgr2=0x557f355af000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:01.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.532+0000 7ff0a8c3c140 1 --2- 172.21.15.121:0/1646816398 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557f35531b80 0x557f355af000 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff09400bb10 tx=0x7ff094009160 comp rx=0 tx=0).stop 2024-09-17T18:19:01.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.532+0000 7ff0a7eee640 1 -- 172.21.15.121:0/1646816398 reap_dead start 2024-09-17T18:19:01.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.532+0000 7ff0a8c3c140 1 -- 172.21.15.121:0/1646816398 shutdown_connections 2024-09-17T18:19:01.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.532+0000 7ff0a8c3c140 1 -- 172.21.15.121:0/1646816398 >> 172.21.15.121:0/1646816398 conn(0x557f35533100 msgr2=0x557f3552d810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:01.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.532+0000 7ff0a8c3c140 1 -- 172.21.15.121:0/1646816398 shutdown_connections 2024-09-17T18:19:01.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.532+0000 7ff0a8c3c140 1 -- 172.21.15.121:0/1646816398 wait complete. 2024-09-17T18:19:01.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.533+0000 7ff0a8c3c140 1 librados: shutdown 2024-09-17T18:19:01.540 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-17T18:19:01.541 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-17T18:19:01.541 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-17T18:19:01.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.639+0000 7f4051a7f640 10 monclient: get_monmap_and_config 2024-09-17T18:19:01.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.639+0000 7f4051a7f640 10 monclient: build_initial_monmap 2024-09-17T18:19:01.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.639+0000 7f4051a7f640 10 monclient: monmap: 2024-09-17T18:19:01.641 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:01.641 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:01.642 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:19:01.640240+0000 2024-09-17T18:19:01.642 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:19:01.640240+0000 2024-09-17T18:19:01.642 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:01.642 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:01.642 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:19:01.642 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:19:01.642 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:19:01.642 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:19:01.642 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:19:01.642 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:01.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.640+0000 7f4051a7f640 1 Processor -- start 2024-09-17T18:19:01.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.640+0000 7f4051a7f640 1 -- start start 2024-09-17T18:19:01.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.640+0000 7f4051a7f640 10 monclient: init 2024-09-17T18:19:01.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.640+0000 7f4051a7f640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:01.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.640+0000 7f4051a7f640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-17T18:19:01.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.640+0000 7f4051a7f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f404c135440 addr v1:172.21.15.173:6789/0 2024-09-17T18:19:01.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.640+0000 7f4051a7f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f404c138880 addr v1:172.21.15.121:6789/0 2024-09-17T18:19:01.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.640+0000 7f4051a7f640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f404c13ede0 0x7f404c1411d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:01.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.640+0000 7f4051a7f640 10 monclient(hunting): picked mon.noname-a con 0x7f404c13ede0 addr v2:172.21.15.121:3300/0 2024-09-17T18:19:01.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.640+0000 7f4051a7f640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f404c0a13a0 con 0x7f404c138880 2024-09-17T18:19:01.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.640+0000 7f4051a7f640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f404c077300 con 0x7f404c135440 2024-09-17T18:19:01.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.640+0000 7f4051a7f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:01.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.640+0000 7f4051a7f640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f404c06fc80 con 0x7f404c13ede0 2024-09-17T18:19:01.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.640+0000 7f4051a7f640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:01.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.640+0000 7f4051a7f640 10 monclient(hunting): authenticate will time out at 745.356906s 2024-09-17T18:19:01.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.641+0000 7f404affd640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f404c138880 0x7f404c13d6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:49494/0 (socket says 172.21.15.121:49494) 2024-09-17T18:19:01.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.641+0000 7f404affd640 1 -- 172.21.15.121:0/256367162 learned_addr learned my addr 172.21.15.121:0/256367162 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:19:01.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.641+0000 7f404bfff640 1 --2- 172.21.15.121:0/256367162 >> v2:172.21.15.121:3300/0 conn(0x7f404c13ede0 0x7f404c1411d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:01.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.641+0000 7f404bfff640 10 monclient(hunting): get_auth_request con 0x7f404c13ede0 auth_method 0 2024-09-17T18:19:01.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.641+0000 7f404bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:01.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.641+0000 7f404bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:01.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.641+0000 7f404bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:01.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.641+0000 7f404bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:01.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.641+0000 7f404bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:01.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.641+0000 7f404bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:01.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.641+0000 7f404bfff640 10 monclient(hunting): handle_auth_done global_id 4212 payload 274 2024-09-17T18:19:01.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.641+0000 7f404bfff640 1 -- 172.21.15.121:0/256367162 >> v1:172.21.15.173:6789/0 conn(0x7f404c135440 legacy=0x7f404c135820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:01.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.641+0000 7f404bfff640 1 -- 172.21.15.121:0/256367162 >> v1:172.21.15.121:6789/0 conn(0x7f404c138880 legacy=0x7f404c13d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:01.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.641+0000 7f404bfff640 10 monclient: _finish_hunting 0 2024-09-17T18:19:01.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.641+0000 7f404bfff640 1 monclient: found mon.noname-a 2024-09-17T18:19:01.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.641+0000 7f404bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:01.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.641+0000 7f404bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:19:01.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.641+0000 7f404bfff640 1 -- 172.21.15.121:0/256367162 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f404c06ff60 con 0x7f404c13ede0 2024-09-17T18:19:01.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.644+0000 7f404bfff640 1 -- 172.21.15.121:0/256367162 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f404c13ede0 msgr2=0x7f404c1411d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:01.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.644+0000 7f404bfff640 1 --2- 172.21.15.121:0/256367162 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f404c13ede0 0x7f404c1411d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f404c06d9d0 tx=0x7f404402a460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3401d36ff7ba2705 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:01.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.644+0000 7f404a7fc640 1 -- 172.21.15.121:0/256367162 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4044007a30 con 0x7f404c13ede0 2024-09-17T18:19:01.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.644+0000 7f404a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:01.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.644+0000 7f404a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:19:01.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.644+0000 7f404a7fc640 10 monclient: dump: 2024-09-17T18:19:01.654 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:01.654 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:01.654 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:01.654 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:01.654 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:01.655 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:01.655 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:01.655 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:01.655 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:01.655 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:01.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.644+0000 7f404a7fc640 10 monclient: _finish_auth 0 2024-09-17T18:19:01.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.644+0000 7f404a7fc640 10 monclient: _check_auth_tickets 2024-09-17T18:19:01.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.644+0000 7f404a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:01.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.644+0000 7f404a7fc640 1 -- 172.21.15.121:0/256367162 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4044007bd0 con 0x7f404c13ede0 2024-09-17T18:19:01.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.644+0000 7f4051a7f640 5 monclient: authenticate success, global_id 4212 2024-09-17T18:19:01.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.644+0000 7f4051a7f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:19:01.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.644+0000 7f404a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:01.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.644+0000 7f4051a7f640 10 monclient: get_monmap_and_config success 2024-09-17T18:19:01.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.644+0000 7f404a7fc640 1 -- 172.21.15.121:0/256367162 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4044031400 con 0x7f404c13ede0 2024-09-17T18:19:01.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.644+0000 7f404a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:01.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.644+0000 7f4051a7f640 4 set_mon_vals no callback set 2024-09-17T18:19:01.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.644+0000 7f404a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:19:01.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.644+0000 7f404a7fc640 10 monclient: dump: 2024-09-17T18:19:01.656 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:01.656 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:01.656 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:01.656 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:01.656 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:01.656 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:01.656 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:01.656 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:01.656 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:01.657 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:01.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.644+0000 7f4051a7f640 10 monclient: shutdown 2024-09-17T18:19:01.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.644+0000 7f4051a7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:01.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.644+0000 7f4051a7f640 1 -- 172.21.15.121:0/256367162 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f404c13ede0 msgr2=0x7f404c1411d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:01.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.644+0000 7f4051a7f640 1 --2- 172.21.15.121:0/256367162 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f404c13ede0 0x7f404c1411d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f404c06d9d0 tx=0x7f404402a460 comp rx=0 tx=0).stop 2024-09-17T18:19:01.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.644+0000 7f4051a7f640 1 -- 172.21.15.121:0/256367162 shutdown_connections 2024-09-17T18:19:01.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.644+0000 7f4051a7f640 1 --2- 172.21.15.121:0/256367162 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f404c13ede0 0x7f404c1411d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:01.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.644+0000 7f4051a7f640 1 -- 172.21.15.121:0/256367162 >> 172.21.15.121:0/256367162 conn(0x7f404c0a49c0 msgr2=0x7f404c0768d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:01.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.645+0000 7f4051a7f640 1 -- 172.21.15.121:0/256367162 shutdown_connections 2024-09-17T18:19:01.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.645+0000 7f4051a7f640 1 -- 172.21.15.121:0/256367162 wait complete. 2024-09-17T18:19:01.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.645+0000 7f4051a7f640 10 monclient: build_initial_monmap 2024-09-17T18:19:01.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.645+0000 7f4051a7f640 10 monclient: monmap: 2024-09-17T18:19:01.657 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:01.658 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:19:01.658 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:19:01.658 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:19:01.658 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:01.658 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:01.658 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:19:01.658 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:19:01.658 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:19:01.658 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:01.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.645+0000 7f4051a7f640 1 librados: starting msgr at 2024-09-17T18:19:01.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.645+0000 7f4051a7f640 1 librados: starting objecter 2024-09-17T18:19:01.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.645+0000 7f4051a7f640 1 Processor -- start 2024-09-17T18:19:01.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f4051a7f640 1 -- start start 2024-09-17T18:19:01.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f4051a7f640 1 librados: setting wanted keys 2024-09-17T18:19:01.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f4051a7f640 1 librados: calling monclient init 2024-09-17T18:19:01.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f4051a7f640 10 monclient: init 2024-09-17T18:19:01.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f4051a7f640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:01.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f4051a7f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:19:01.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f4051a7f640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f404c135440 0x7f404c071740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:01.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f4051a7f640 10 monclient(hunting): picked mon.noname-a con 0x7f404c135440 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:01.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f4051a7f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f404c138880 0x7f404c071c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:01.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f4051a7f640 10 monclient(hunting): picked mon.noname-b con 0x7f404c138880 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:01.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f4051a7f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f404c13ede0 0x7f404c0721c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:01.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f4051a7f640 10 monclient(hunting): picked mon.noname-c con 0x7f404c13ede0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:01.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f4051a7f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:01.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f4051a7f640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f404c06d6a0 con 0x7f404c135440 2024-09-17T18:19:01.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f4051a7f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:01.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f4051a7f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f404c0a1e60 con 0x7f404c138880 2024-09-17T18:19:01.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f4051a7f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:01.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f4051a7f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f404c06d820 con 0x7f404c13ede0 2024-09-17T18:19:01.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f4051a7f640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:01.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f4051a7f640 10 monclient(hunting): authenticate will time out at 745.362492s 2024-09-17T18:19:01.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f404b7fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f404c135440 0x7f404c071740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:01.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f404b7fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f404c135440 0x7f404c071740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40808/0 (socket says 172.21.15.121:40808) 2024-09-17T18:19:01.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f404b7fe640 1 -- 172.21.15.121:0/3145551960 learned_addr learned my addr 172.21.15.121:0/3145551960 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:01.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f404b7fe640 10 monclient(hunting): get_auth_request con 0x7f404c135440 auth_method 0 2024-09-17T18:19:01.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f404b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:01.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f404b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:01.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f404b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:01.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f404bfff640 1 --2- 172.21.15.121:0/3145551960 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f404c13ede0 0x7f404c0721c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:01.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f404bfff640 10 monclient(hunting): get_auth_request con 0x7f404c13ede0 auth_method 0 2024-09-17T18:19:01.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f404bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:01.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f404bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:01.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f404bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:01.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f404b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:01.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f404b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:01.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f404b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:01.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f404bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:01.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f404bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:01.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f404bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:01.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f404affd640 1 --2- 172.21.15.121:0/3145551960 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f404c138880 0x7f404c071c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:01.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f404affd640 10 monclient(hunting): get_auth_request con 0x7f404c138880 auth_method 0 2024-09-17T18:19:01.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f404affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:01.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f404affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:01.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f404affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:01.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f404affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:01.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f404affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:01.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f404affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:01.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f404bfff640 10 monclient(hunting): handle_auth_done global_id 4196 payload 824 2024-09-17T18:19:01.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f404bfff640 1 -- 172.21.15.121:0/3145551960 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f404c138880 msgr2=0x7f404c071c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:01.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f404bfff640 1 --2- 172.21.15.121:0/3145551960 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f404c138880 0x7f404c071c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:01.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f404bfff640 1 -- 172.21.15.121:0/3145551960 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f404c135440 msgr2=0x7f404c071740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:01.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f404bfff640 1 --2- 172.21.15.121:0/3145551960 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f404c135440 0x7f404c071740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:01.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f404bfff640 10 monclient: _finish_hunting 0 2024-09-17T18:19:01.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f404bfff640 1 monclient: found mon.noname-c 2024-09-17T18:19:01.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f404bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:01.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f404bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:19:01.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f404bfff640 1 -- 172.21.15.121:0/3145551960 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f404c13a0b0 con 0x7f404c13ede0 2024-09-17T18:19:01.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.646+0000 7f404b7fe640 1 --2- 172.21.15.121:0/3145551960 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f404c135440 0x7f404c071740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:19:01.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.649+0000 7f404bfff640 1 -- 172.21.15.121:0/3145551960 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f404c13ede0 msgr2=0x7f404c0721c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:01.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.649+0000 7f404bfff640 1 --2- 172.21.15.121:0/3145551960 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f404c13ede0 0x7f404c0721c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4044031000 tx=0x7f4044004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:01.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.649+0000 7f4043fff640 10 client.?.objecter ms_handle_connect 0x7f404c13ede0 2024-09-17T18:19:01.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.649+0000 7f4043fff640 10 client.?.objecter resend_mon_ops 2024-09-17T18:19:01.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.649+0000 7f4043fff640 1 -- 172.21.15.121:0/3145551960 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4044040070 con 0x7f404c13ede0 2024-09-17T18:19:01.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.649+0000 7f4043fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:01.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.649+0000 7f4043fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:19:01.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.649+0000 7f4043fff640 10 monclient: dump: 2024-09-17T18:19:01.664 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:01.664 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:01.664 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:01.664 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:01.664 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:01.664 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:01.664 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:01.664 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:01.664 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:01.664 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:01.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.649+0000 7f4043fff640 10 monclient: _finish_auth 0 2024-09-17T18:19:01.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.649+0000 7f4043fff640 10 monclient: _check_auth_tickets 2024-09-17T18:19:01.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.649+0000 7f4043fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:01.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.649+0000 7f4043fff640 1 -- 172.21.15.121:0/3145551960 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4044045660 con 0x7f404c13ede0 2024-09-17T18:19:01.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.649+0000 7f4051a7f640 5 monclient: authenticate success, global_id 4196 2024-09-17T18:19:01.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.649+0000 7f4043fff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:01.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.649+0000 7f4051a7f640 10 monclient: _renew_subs 2024-09-17T18:19:01.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.649+0000 7f4043fff640 1 -- 172.21.15.121:0/3145551960 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4044045a30 con 0x7f404c13ede0 2024-09-17T18:19:01.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.649+0000 7f4051a7f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:01.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.649+0000 7f4051a7f640 1 -- 172.21.15.121:0/3145551960 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f404c13e980 con 0x7f404c13ede0 2024-09-17T18:19:01.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.649+0000 7f4043fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:01.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.649+0000 7f4043fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:19:01.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.649+0000 7f4043fff640 10 monclient: dump: 2024-09-17T18:19:01.665 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:01.666 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:01.666 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:01.666 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:01.666 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:01.666 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:01.666 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:01.666 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:01.666 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:01.666 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:01.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.649+0000 7f40497fa640 4 set_mon_vals no callback set 2024-09-17T18:19:01.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.649+0000 7f4051a7f640 10 client.4196.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:01.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.649+0000 7f4051a7f640 10 monclient: _renew_subs 2024-09-17T18:19:01.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.649+0000 7f4051a7f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:01.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.649+0000 7f4051a7f640 1 -- 172.21.15.121:0/3145551960 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f404c139dd0 con 0x7f404c13ede0 2024-09-17T18:19:01.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.649+0000 7f4051a7f640 1 librados: init done 2024-09-17T18:19:01.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.650+0000 7f4051a7f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:01.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.650+0000 7f4043fff640 1 -- 172.21.15.121:0/3145551960 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 4) ==== 50940+0+0 (secure 0 0 0) 0x7f4044039040 con 0x7f404c13ede0 2024-09-17T18:19:01.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.650+0000 7f4051a7f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:01.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.650+0000 7f4051a7f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:01.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.650+0000 7f4051a7f640 1 -- 172.21.15.121:0/3145551960 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f404c1358a0 con 0x7f404c13ede0 2024-09-17T18:19:01.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.650+0000 7f4043fff640 1 --2- 172.21.15.121:0/3145551960 >> v2:172.21.15.173:6812/2423786037 conn(0x7f402c040110 0x7f402c0425d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:01.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.650+0000 7f4043fff640 1 -- 172.21.15.121:0/3145551960 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 3562+0+0 (secure 0 0 0) 0x7f4044074930 con 0x7f404c13ede0 2024-09-17T18:19:01.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.650+0000 7f4043fff640 10 client.4196.objecter ms_dispatch 0x7f404c000b90 osd_map(16..16 src has 1..16) 2024-09-17T18:19:01.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.650+0000 7f4043fff640 3 client.4196.objecter handle_osd_map got epochs [16,16] > 0 2024-09-17T18:19:01.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.650+0000 7f4043fff640 3 client.4196.objecter handle_osd_map decoding full epoch 16 2024-09-17T18:19:01.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.650+0000 7f4043fff640 20 client.4196.objecter dump_active .. 0 homeless 2024-09-17T18:19:01.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.650+0000 7f404b7fe640 1 --2- 172.21.15.121:0/3145551960 >> v2:172.21.15.173:6812/2423786037 conn(0x7f402c040110 0x7f402c0425d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:01.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.650+0000 7f404b7fe640 10 monclient: get_auth_request con 0x7f402c040110 auth_method 0 2024-09-17T18:19:01.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.653+0000 7f4043fff640 10 client.4196.objecter ms_handle_connect 0x7f402c040110 2024-09-17T18:19:01.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.653+0000 7f404b7fe640 1 --2- 172.21.15.121:0/3145551960 >> v2:172.21.15.173:6812/2423786037 conn(0x7f402c040110 0x7f402c0425d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f404c13a2c0 tx=0x7f4034007ef0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:01.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.655+0000 7f4043fff640 1 -- 172.21.15.121:0/3145551960 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4044073800 con 0x7f404c13ede0 2024-09-17T18:19:01.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.655+0000 7f4043fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:01.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.655+0000 7f4043fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:19:01.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.815+0000 7f4043fff640 1 -- 172.21.15.121:0/3145551960 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f404403ccd0 con 0x7f404c13ede0 2024-09-17T18:19:01.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.831+0000 7f4051a7f640 10 start_mon_command cmd=[{"prefix": "config log", "num": 1, "format": "json"}] 2024-09-17T18:19:01.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.831+0000 7f4051a7f640 10 monclient: _send_command 2 [{"prefix": "config log", "num": 1, "format": "json"}] 2024-09-17T18:19:01.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.831+0000 7f4051a7f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:01.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.831+0000 7f4051a7f640 1 -- 172.21.15.121:0/3145551960 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7f404c06cd40 con 0x7f404c13ede0 2024-09-17T18:19:01.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.832+0000 7f4043fff640 1 -- 172.21.15.121:0/3145551960 <== mon.2 v2:172.21.15.121:3301/0 8 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v5) ==== 86+0+159 (secure 0 0 0) 0x7f404404acd0 con 0x7f404c13ede0 2024-09-17T18:19:01.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.832+0000 7f4043fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config log", "num": 1, "format": "json"}] 2024-09-17T18:19:01.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.832+0000 7f4043fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:19:01.834 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:19:01.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.834+0000 7f4051a7f640 10 librados: watch_flush enter 2024-09-17T18:19:01.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.834+0000 7f4051a7f640 10 librados: watch_flush exit 2024-09-17T18:19:01.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.834+0000 7f4051a7f640 20 client.4196.objecter shutdown clearing up homeless session... 2024-09-17T18:19:01.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.834+0000 7f4051a7f640 10 client.4196.objecter successfully canceled tick 2024-09-17T18:19:01.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.834+0000 7f4051a7f640 1 -- 172.21.15.121:0/3145551960 >> v2:172.21.15.173:6812/2423786037 conn(0x7f402c040110 msgr2=0x7f402c0425d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:01.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.834+0000 7f4051a7f640 1 --2- 172.21.15.121:0/3145551960 >> v2:172.21.15.173:6812/2423786037 conn(0x7f402c040110 0x7f402c0425d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f404c13a2c0 tx=0x7f4034007ef0 comp rx=0 tx=0).stop 2024-09-17T18:19:01.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.834+0000 7f4051a7f640 10 monclient: shutdown 2024-09-17T18:19:01.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.834+0000 7f4051a7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:01.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.834+0000 7f4051a7f640 1 -- 172.21.15.121:0/3145551960 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f404c13ede0 msgr2=0x7f404c0721c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:01.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.834+0000 7f4051a7f640 1 --2- 172.21.15.121:0/3145551960 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f404c13ede0 0x7f404c0721c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4044031000 tx=0x7f4044004a40 comp rx=0 tx=0).stop 2024-09-17T18:19:01.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.834+0000 7f4051a7f640 1 -- 172.21.15.121:0/3145551960 shutdown_connections 2024-09-17T18:19:01.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.834+0000 7f4051a7f640 1 --2- 172.21.15.121:0/3145551960 >> v2:172.21.15.173:6812/2423786037 conn(0x7f402c040110 0x7f402c0425d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:01.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.834+0000 7f4051a7f640 1 --2- 172.21.15.121:0/3145551960 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f404c138880 0x7f404c071c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:01.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.834+0000 7f4051a7f640 1 --2- 172.21.15.121:0/3145551960 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f404c13ede0 0x7f404c0721c0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:01.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.834+0000 7f4051a7f640 1 --2- 172.21.15.121:0/3145551960 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f404c135440 0x7f404c071740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:01.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.834+0000 7f4051a7f640 1 -- 172.21.15.121:0/3145551960 >> 172.21.15.121:0/3145551960 conn(0x7f404c0a49c0 msgr2=0x7f404c1377c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:01.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.834+0000 7f4051a7f640 1 -- 172.21.15.121:0/3145551960 shutdown_connections 2024-09-17T18:19:01.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.834+0000 7f4051a7f640 1 -- 172.21.15.121:0/3145551960 wait complete. 2024-09-17T18:19:01.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.834+0000 7f4051a7f640 1 librados: shutdown 2024-09-17T18:19:01.846 INFO:teuthology.orchestra.run.smithi121.stdout:[{"version":5,"timestamp":"2024-09-17T18:18:57.083480+0000","name":"","changes":[{"name":"osd.1/osd_mclock_max_capacity_iops_ssd","new_value":"9304.681729"}]}] 2024-09-17T18:19:01.846 INFO:tasks.ceph_manager:config epoch is 5 2024-09-17T18:19:01.847 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-17T18:19:01.847 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-17T18:19:01.847 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-17T18:19:01.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.941+0000 7fcd65d00640 10 monclient: get_monmap_and_config 2024-09-17T18:19:01.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.941+0000 7fcd65d00640 10 monclient: build_initial_monmap 2024-09-17T18:19:01.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.941+0000 7fcd65d00640 10 monclient: monmap: 2024-09-17T18:19:01.943 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:01.944 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:01.944 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:19:01.942399+0000 2024-09-17T18:19:01.944 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:19:01.942399+0000 2024-09-17T18:19:01.944 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:01.944 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:01.944 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:19:01.944 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:19:01.944 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:19:01.944 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:19:01.944 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:19:01.944 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:01.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.942+0000 7fcd65d00640 1 Processor -- start 2024-09-17T18:19:01.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.942+0000 7fcd65d00640 1 -- start start 2024-09-17T18:19:01.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.942+0000 7fcd65d00640 10 monclient: init 2024-09-17T18:19:01.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.943+0000 7fcd65d00640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:01.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.943+0000 7fcd65d00640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-17T18:19:01.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.943+0000 7fcd65d00640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcd6006fd20 addr v1:172.21.15.121:6789/0 2024-09-17T18:19:01.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.943+0000 7fcd65d00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fcd600a53a0 0x7fcd6013de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:01.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.943+0000 7fcd65d00640 10 monclient(hunting): picked mon.noname-c con 0x7fcd600a53a0 addr v2:172.21.15.173:3300/0 2024-09-17T18:19:01.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.943+0000 7fcd65d00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd600a57b0 0x7fcd601403b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:01.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.943+0000 7fcd65d00640 10 monclient(hunting): picked mon.noname-b con 0x7fcd600a57b0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:01.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.943+0000 7fcd65d00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd600705e0 con 0x7fcd6006fd20 2024-09-17T18:19:01.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.943+0000 7fcd65d00640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:01.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.943+0000 7fcd65d00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fcd60070100 con 0x7fcd600a53a0 2024-09-17T18:19:01.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.943+0000 7fcd65d00640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:01.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.943+0000 7fcd65d00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcd6006f540 con 0x7fcd600a57b0 2024-09-17T18:19:01.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.943+0000 7fcd65d00640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:01.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.943+0000 7fcd65d00640 10 monclient(hunting): authenticate will time out at 745.659117s 2024-09-17T18:19:01.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.943+0000 7fcd5f7fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fcd6006fd20 0x7fcd600a4c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:49504/0 (socket says 172.21.15.121:49504) 2024-09-17T18:19:01.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.943+0000 7fcd5ffff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd600a57b0 0x7fcd601403b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:01.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.943+0000 7fcd5f7fe640 1 -- 172.21.15.121:0/640335143 learned_addr learned my addr 172.21.15.121:0/640335143 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:19:01.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.943+0000 7fcd5ffff640 10 monclient(hunting): get_auth_request con 0x7fcd600a57b0 auth_method 0 2024-09-17T18:19:01.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.943+0000 7fcd5ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:01.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.943+0000 7fcd5ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:01.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.943+0000 7fcd5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:01.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.943+0000 7fcd5ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:01.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.943+0000 7fcd5ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:01.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.943+0000 7fcd5ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:01.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.943+0000 7fcd56ffd640 1 --2- 172.21.15.121:0/640335143 >> v2:172.21.15.173:3300/0 conn(0x7fcd600a53a0 0x7fcd6013de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:01.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.943+0000 7fcd56ffd640 10 monclient(hunting): get_auth_request con 0x7fcd600a53a0 auth_method 0 2024-09-17T18:19:01.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.943+0000 7fcd56ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:01.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.943+0000 7fcd56ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:01.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.943+0000 7fcd56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:01.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.943+0000 7fcd5ffff640 10 monclient(hunting): handle_auth_done global_id 4199 payload 274 2024-09-17T18:19:01.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.943+0000 7fcd5ffff640 1 -- 172.21.15.121:0/640335143 >> v2:172.21.15.173:3300/0 conn(0x7fcd600a53a0 msgr2=0x7fcd6013de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:01.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.943+0000 7fcd5ffff640 1 --2- 172.21.15.121:0/640335143 >> v2:172.21.15.173:3300/0 conn(0x7fcd600a53a0 0x7fcd6013de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:01.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.943+0000 7fcd5ffff640 1 -- 172.21.15.121:0/640335143 >> v1:172.21.15.121:6789/0 conn(0x7fcd6006fd20 legacy=0x7fcd600a4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:01.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.943+0000 7fcd5ffff640 10 monclient: _finish_hunting 0 2024-09-17T18:19:01.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.943+0000 7fcd5ffff640 1 monclient: found mon.noname-b 2024-09-17T18:19:01.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.943+0000 7fcd5ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:01.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.943+0000 7fcd5ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:19:01.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.943+0000 7fcd5ffff640 1 -- 172.21.15.121:0/640335143 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd600708c0 con 0x7fcd600a57b0 2024-09-17T18:19:01.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.943+0000 7fcd56ffd640 1 --2- 172.21.15.121:0/640335143 >> v2:172.21.15.173:3300/0 conn(0x7fcd600a53a0 0x7fcd6013de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T18:19:01.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.946+0000 7fcd5ffff640 1 -- 172.21.15.121:0/640335143 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd600a57b0 msgr2=0x7fcd601403b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:01.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.946+0000 7fcd5ffff640 1 --2- 172.21.15.121:0/640335143 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd600a57b0 0x7fcd601403b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fcd58009d90 tx=0x7fcd5802a480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d54ba2295be7954 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:01.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.946+0000 7fcd5effd640 1 -- 172.21.15.121:0/640335143 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd58007950 con 0x7fcd600a57b0 2024-09-17T18:19:01.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.946+0000 7fcd5effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:01.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.946+0000 7fcd5effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:19:01.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.946+0000 7fcd5effd640 10 monclient: dump: 2024-09-17T18:19:01.959 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:01.959 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:01.959 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:01.959 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:01.959 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:01.959 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:01.959 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:01.959 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:01.959 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:01.960 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:01.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.946+0000 7fcd5effd640 10 monclient: _finish_auth 0 2024-09-17T18:19:01.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.946+0000 7fcd5effd640 10 monclient: _check_auth_tickets 2024-09-17T18:19:01.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.946+0000 7fcd5effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:01.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.946+0000 7fcd5effd640 1 -- 172.21.15.121:0/640335143 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd58007af0 con 0x7fcd600a57b0 2024-09-17T18:19:01.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.946+0000 7fcd65d00640 5 monclient: authenticate success, global_id 4199 2024-09-17T18:19:01.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.946+0000 7fcd65d00640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:19:01.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.946+0000 7fcd5effd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:01.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.946+0000 7fcd65d00640 10 monclient: get_monmap_and_config success 2024-09-17T18:19:01.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.946+0000 7fcd5effd640 1 -- 172.21.15.121:0/640335143 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd58031400 con 0x7fcd600a57b0 2024-09-17T18:19:01.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.946+0000 7fcd5effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:01.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.946+0000 7fcd65d00640 4 set_mon_vals no callback set 2024-09-17T18:19:01.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.946+0000 7fcd5effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:19:01.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.946+0000 7fcd5effd640 10 monclient: dump: 2024-09-17T18:19:01.961 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:01.961 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:01.961 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:01.961 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:01.961 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:01.961 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:01.961 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:01.961 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:01.961 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:01.961 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:01.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.946+0000 7fcd65d00640 10 monclient: shutdown 2024-09-17T18:19:01.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.946+0000 7fcd65d00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:01.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.946+0000 7fcd65d00640 1 -- 172.21.15.121:0/640335143 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd600a57b0 msgr2=0x7fcd601403b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:01.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.946+0000 7fcd65d00640 1 --2- 172.21.15.121:0/640335143 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd600a57b0 0x7fcd601403b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fcd58009d90 tx=0x7fcd5802a480 comp rx=0 tx=0).stop 2024-09-17T18:19:01.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.947+0000 7fcd65d00640 1 -- 172.21.15.121:0/640335143 shutdown_connections 2024-09-17T18:19:01.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.947+0000 7fcd65d00640 1 --2- 172.21.15.121:0/640335143 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd600a57b0 0x7fcd601403b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:01.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.947+0000 7fcd65d00640 1 --2- 172.21.15.121:0/640335143 >> v2:172.21.15.173:3300/0 conn(0x7fcd600a53a0 0x7fcd6013de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:01.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.947+0000 7fcd65d00640 1 -- 172.21.15.121:0/640335143 >> 172.21.15.121:0/640335143 conn(0x7fcd600a9a60 msgr2=0x7fcd600a5d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:01.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.947+0000 7fcd65d00640 1 -- 172.21.15.121:0/640335143 shutdown_connections 2024-09-17T18:19:01.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.947+0000 7fcd65d00640 1 -- 172.21.15.121:0/640335143 wait complete. 2024-09-17T18:19:01.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.947+0000 7fcd65d00640 10 monclient: build_initial_monmap 2024-09-17T18:19:01.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.947+0000 7fcd65d00640 10 monclient: monmap: 2024-09-17T18:19:01.962 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:01.962 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:19:01.963 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:19:01.963 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:19:01.963 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:01.963 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:01.963 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:19:01.963 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:19:01.963 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:19:01.963 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:01.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.947+0000 7fcd65d00640 1 librados: starting msgr at 2024-09-17T18:19:01.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.947+0000 7fcd65d00640 1 librados: starting objecter 2024-09-17T18:19:01.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.947+0000 7fcd65d00640 1 Processor -- start 2024-09-17T18:19:01.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd65d00640 1 -- start start 2024-09-17T18:19:01.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd65d00640 1 librados: setting wanted keys 2024-09-17T18:19:01.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd65d00640 1 librados: calling monclient init 2024-09-17T18:19:01.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd65d00640 10 monclient: init 2024-09-17T18:19:01.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd65d00640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:01.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd65d00640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T18:19:01.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd65d00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd6006fd20 0x7fcd601ada70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:01.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd65d00640 10 monclient(hunting): picked mon.noname-b con 0x7fcd6006fd20 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:01.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd65d00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd600a53a0 0x7fcd601cde30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:01.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd65d00640 10 monclient(hunting): picked mon.noname-a con 0x7fcd600a53a0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:01.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd65d00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd600a57b0 0x7fcd601c7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:01.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd65d00640 10 monclient(hunting): picked mon.noname-c con 0x7fcd600a57b0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:01.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd65d00640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:01.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd65d00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcd600a0dc0 con 0x7fcd600a53a0 2024-09-17T18:19:01.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd65d00640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:01.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd65d00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcd600a0c40 con 0x7fcd6006fd20 2024-09-17T18:19:01.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd65d00640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:01.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd65d00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcd60074720 con 0x7fcd600a57b0 2024-09-17T18:19:01.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd65d00640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:01.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd65d00640 10 monclient(hunting): authenticate will time out at 745.664580s 2024-09-17T18:19:01.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd56ffd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd600a53a0 0x7fcd601cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:01.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd5ffff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd600a57b0 0x7fcd601c7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:01.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd56ffd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd600a53a0 0x7fcd601cde30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40814/0 (socket says 172.21.15.121:40814) 2024-09-17T18:19:01.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd56ffd640 1 -- 172.21.15.121:0/2612303244 learned_addr learned my addr 172.21.15.121:0/2612303244 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:01.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd56ffd640 10 monclient(hunting): get_auth_request con 0x7fcd600a53a0 auth_method 0 2024-09-17T18:19:01.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd56ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:01.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd56ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:01.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:01.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd5ffff640 10 monclient(hunting): get_auth_request con 0x7fcd600a57b0 auth_method 0 2024-09-17T18:19:01.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd5ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:01.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd5ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:01.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:01.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd56ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:01.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd56ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:01.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd5f7fe640 1 --2- 172.21.15.121:0/2612303244 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd6006fd20 0x7fcd601ada70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:01.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd56ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:01.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd5ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:01.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd5ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:01.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd5ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:01.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd5f7fe640 10 monclient(hunting): get_auth_request con 0x7fcd6006fd20 auth_method 0 2024-09-17T18:19:01.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd5f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:01.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd5f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:01.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:01.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd5f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:01.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd5f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:01.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd5f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:01.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd56ffd640 10 monclient(hunting): handle_auth_done global_id 4218 payload 824 2024-09-17T18:19:01.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd56ffd640 1 -- 172.21.15.121:0/2612303244 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd600a57b0 msgr2=0x7fcd601c7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:01.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd56ffd640 1 --2- 172.21.15.121:0/2612303244 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd600a57b0 0x7fcd601c7810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:01.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd56ffd640 1 -- 172.21.15.121:0/2612303244 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd6006fd20 msgr2=0x7fcd601ada70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:01.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd56ffd640 1 --2- 172.21.15.121:0/2612303244 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd6006fd20 0x7fcd601ada70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:01.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd56ffd640 10 monclient: _finish_hunting 0 2024-09-17T18:19:01.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd56ffd640 1 monclient: found mon.noname-a 2024-09-17T18:19:01.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd56ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:01.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd56ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:19:01.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.948+0000 7fcd56ffd640 1 -- 172.21.15.121:0/2612303244 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd601ae400 con 0x7fcd600a53a0 2024-09-17T18:19:01.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.951+0000 7fcd56ffd640 1 -- 172.21.15.121:0/2612303244 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd600a53a0 msgr2=0x7fcd601cde30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:01.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.951+0000 7fcd56ffd640 1 --2- 172.21.15.121:0/2612303244 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd600a53a0 0x7fcd601cde30 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fcd4800e730 tx=0x7fcd4800ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:01.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.951+0000 7fcd5cff9640 10 client.?.objecter ms_handle_connect 0x7fcd600a53a0 2024-09-17T18:19:01.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.951+0000 7fcd5cff9640 10 client.?.objecter resend_mon_ops 2024-09-17T18:19:01.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.951+0000 7fcd5cff9640 1 -- 172.21.15.121:0/2612303244 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd4800f040 con 0x7fcd600a53a0 2024-09-17T18:19:01.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.951+0000 7fcd5cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:01.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.951+0000 7fcd5cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:19:01.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.951+0000 7fcd5cff9640 10 monclient: dump: 2024-09-17T18:19:01.968 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:01.968 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:01.968 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:01.968 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:01.968 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:01.968 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:01.969 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:01.969 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:01.969 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:01.969 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:01.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.951+0000 7fcd5cff9640 10 monclient: _finish_auth 0 2024-09-17T18:19:01.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.951+0000 7fcd5cff9640 10 monclient: _check_auth_tickets 2024-09-17T18:19:01.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.951+0000 7fcd5cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:01.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.951+0000 7fcd5cff9640 1 -- 172.21.15.121:0/2612303244 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd48004230 con 0x7fcd600a53a0 2024-09-17T18:19:01.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.951+0000 7fcd65d00640 5 monclient: authenticate success, global_id 4218 2024-09-17T18:19:01.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.951+0000 7fcd5cff9640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:01.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.951+0000 7fcd5cff9640 1 -- 172.21.15.121:0/2612303244 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd4800ee40 con 0x7fcd600a53a0 2024-09-17T18:19:01.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.951+0000 7fcd65d00640 10 monclient: _renew_subs 2024-09-17T18:19:01.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.951+0000 7fcd65d00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:01.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.951+0000 7fcd65d00640 1 -- 172.21.15.121:0/2612303244 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd600a0470 con 0x7fcd600a53a0 2024-09-17T18:19:01.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.951+0000 7fcd5cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:01.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.951+0000 7fcd5cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:19:01.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.951+0000 7fcd5cff9640 10 monclient: dump: 2024-09-17T18:19:01.970 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:01.970 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:01.970 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:01.970 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:01.970 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:01.970 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:01.970 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:01.970 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:01.970 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:01.971 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:01.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.951+0000 7fcd5e7fc640 4 set_mon_vals no callback set 2024-09-17T18:19:01.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.951+0000 7fcd65d00640 10 client.4218.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:01.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.951+0000 7fcd65d00640 10 monclient: _renew_subs 2024-09-17T18:19:01.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.951+0000 7fcd65d00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:01.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.951+0000 7fcd65d00640 1 -- 172.21.15.121:0/2612303244 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd601af270 con 0x7fcd600a53a0 2024-09-17T18:19:01.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.951+0000 7fcd65d00640 1 librados: init done 2024-09-17T18:19:01.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.952+0000 7fcd65d00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:01.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.952+0000 7fcd65d00640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:01.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.952+0000 7fcd65d00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:01.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.952+0000 7fcd65d00640 1 -- 172.21.15.121:0/2612303244 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd600a4c90 con 0x7fcd600a53a0 2024-09-17T18:19:01.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.952+0000 7fcd5cff9640 1 -- 172.21.15.121:0/2612303244 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fcd4800c8e0 con 0x7fcd600a53a0 2024-09-17T18:19:01.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.952+0000 7fcd5cff9640 1 --2- 172.21.15.121:0/2612303244 >> v2:172.21.15.173:6812/2423786037 conn(0x7fcd40040170 0x7fcd40042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:01.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.952+0000 7fcd5cff9640 1 -- 172.21.15.121:0/2612303244 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 3562+0+0 (secure 0 0 0) 0x7fcd48052e90 con 0x7fcd600a53a0 2024-09-17T18:19:01.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.952+0000 7fcd5cff9640 10 client.4218.objecter ms_dispatch 0x7fcd60000b90 osd_map(16..16 src has 1..16) 2024-09-17T18:19:01.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.952+0000 7fcd5cff9640 3 client.4218.objecter handle_osd_map got epochs [16,16] > 0 2024-09-17T18:19:01.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.952+0000 7fcd5cff9640 3 client.4218.objecter handle_osd_map decoding full epoch 16 2024-09-17T18:19:01.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.952+0000 7fcd5cff9640 20 client.4218.objecter dump_active .. 0 homeless 2024-09-17T18:19:01.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.952+0000 7fcd5f7fe640 1 --2- 172.21.15.121:0/2612303244 >> v2:172.21.15.173:6812/2423786037 conn(0x7fcd40040170 0x7fcd40042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:01.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.952+0000 7fcd5f7fe640 10 monclient: get_auth_request con 0x7fcd40040170 auth_method 0 2024-09-17T18:19:01.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.955+0000 7fcd5f7fe640 1 --2- 172.21.15.121:0/2612303244 >> v2:172.21.15.173:6812/2423786037 conn(0x7fcd40040170 0x7fcd40042630 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcd50007c90 tx=0x7fcd50007be0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:01.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.955+0000 7fcd5cff9640 10 client.4218.objecter ms_handle_connect 0x7fcd40040170 2024-09-17T18:19:01.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.957+0000 7fcd5cff9640 1 -- 172.21.15.121:0/2612303244 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fcd48017020 con 0x7fcd600a53a0 2024-09-17T18:19:01.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.957+0000 7fcd5cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:01.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:01.957+0000 7fcd5cff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:19:02.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.180+0000 7fcd65d00640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-09-17T18:19:02.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.180+0000 7fcd65d00640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-17T18:19:02.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.180+0000 7fcd65d00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:02.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.180+0000 7fcd65d00640 1 -- 172.21.15.121:0/2612303244 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fcd6006cd40 con 0x7fcd600a53a0 2024-09-17T18:19:02.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.181+0000 7fcd5cff9640 1 -- 172.21.15.121:0/2612303244 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+98032 (secure 0 0 0) 0x7fcd48023350 con 0x7fcd600a53a0 2024-09-17T18:19:02.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.181+0000 7fcd5cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-17T18:19:02.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.181+0000 7fcd5cff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:19:02.185 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:19:02.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.184+0000 7fcd65d00640 10 librados: watch_flush enter 2024-09-17T18:19:02.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.184+0000 7fcd65d00640 10 librados: watch_flush exit 2024-09-17T18:19:02.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.184+0000 7fcd65d00640 20 client.4218.objecter shutdown clearing up homeless session... 2024-09-17T18:19:02.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.184+0000 7fcd65d00640 10 client.4218.objecter successfully canceled tick 2024-09-17T18:19:02.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.184+0000 7fcd65d00640 1 -- 172.21.15.121:0/2612303244 >> v2:172.21.15.173:6812/2423786037 conn(0x7fcd40040170 msgr2=0x7fcd40042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:02.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.184+0000 7fcd65d00640 1 --2- 172.21.15.121:0/2612303244 >> v2:172.21.15.173:6812/2423786037 conn(0x7fcd40040170 0x7fcd40042630 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcd50007c90 tx=0x7fcd50007be0 comp rx=0 tx=0).stop 2024-09-17T18:19:02.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.184+0000 7fcd65d00640 10 monclient: shutdown 2024-09-17T18:19:02.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.184+0000 7fcd65d00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:02.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.184+0000 7fcd65d00640 1 -- 172.21.15.121:0/2612303244 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd600a53a0 msgr2=0x7fcd601cde30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:02.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.184+0000 7fcd65d00640 1 --2- 172.21.15.121:0/2612303244 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd600a53a0 0x7fcd601cde30 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fcd4800e730 tx=0x7fcd4800ebf0 comp rx=0 tx=0).stop 2024-09-17T18:19:02.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.184+0000 7fcd65d00640 1 -- 172.21.15.121:0/2612303244 shutdown_connections 2024-09-17T18:19:02.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.184+0000 7fcd65d00640 1 --2- 172.21.15.121:0/2612303244 >> v2:172.21.15.173:6812/2423786037 conn(0x7fcd40040170 0x7fcd40042630 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:02.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.184+0000 7fcd65d00640 1 --2- 172.21.15.121:0/2612303244 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd600a57b0 0x7fcd601c7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:02.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.184+0000 7fcd65d00640 1 --2- 172.21.15.121:0/2612303244 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd600a53a0 0x7fcd601cde30 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:02.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.184+0000 7fcd65d00640 1 --2- 172.21.15.121:0/2612303244 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd6006fd20 0x7fcd601ada70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:02.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.184+0000 7fcd65d00640 1 -- 172.21.15.121:0/2612303244 >> 172.21.15.121:0/2612303244 conn(0x7fcd600a9a60 msgr2=0x7fcd60072e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:02.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.184+0000 7fcd65d00640 1 -- 172.21.15.121:0/2612303244 shutdown_connections 2024-09-17T18:19:02.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.184+0000 7fcd65d00640 1 -- 172.21.15.121:0/2612303244 wait complete. 2024-09-17T18:19:02.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.184+0000 7fcd65d00640 1 librados: shutdown 2024-09-17T18:19:02.196 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":5,"flags":0,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2423786037}]},"active_addr":"172.21.15.173:6812/2423786037","active_change":"2024-09-17T18:18:51.487408+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.173:0","nonce":343207865}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.173:0","nonce":2066292415}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.173:0","nonce":2945468150}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.173:0","nonce":418582752}]}]} 2024-09-17T18:19:02.198 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-17T18:19:02.198 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-17T18:19:02.198 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T18:19:02.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.293+0000 7f6e782fe640 10 monclient: get_monmap_and_config 2024-09-17T18:19:02.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.293+0000 7f6e782fe640 10 monclient: build_initial_monmap 2024-09-17T18:19:02.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.293+0000 7f6e782fe640 10 monclient: monmap: 2024-09-17T18:19:02.296 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:02.296 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:02.296 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:19:02.294338+0000 2024-09-17T18:19:02.296 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:19:02.294338+0000 2024-09-17T18:19:02.296 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:02.296 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:02.296 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:19:02.296 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:19:02.296 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:19:02.297 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:19:02.297 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:19:02.297 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:02.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.294+0000 7f6e782fe640 1 Processor -- start 2024-09-17T18:19:02.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.294+0000 7f6e782fe640 1 -- start start 2024-09-17T18:19:02.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.294+0000 7f6e782fe640 10 monclient: init 2024-09-17T18:19:02.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.294+0000 7f6e782fe640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:02.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.294+0000 7f6e782fe640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-17T18:19:02.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.294+0000 7f6e782fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6e7006fd20 addr v1:172.21.15.121:6789/0 2024-09-17T18:19:02.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.294+0000 7f6e782fe640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f6e700a53a0 0x7f6e7013de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:02.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.294+0000 7f6e782fe640 10 monclient(hunting): picked mon.noname-c con 0x7f6e700a53a0 addr v2:172.21.15.173:3300/0 2024-09-17T18:19:02.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.294+0000 7f6e782fe640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6e700a57b0 0x7f6e701403b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:02.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.294+0000 7f6e782fe640 10 monclient(hunting): picked mon.noname-a con 0x7f6e700a57b0 addr v2:172.21.15.121:3300/0 2024-09-17T18:19:02.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.294+0000 7f6e782fe640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e700705e0 con 0x7f6e7006fd20 2024-09-17T18:19:02.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.294+0000 7f6e782fe640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:02.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.294+0000 7f6e782fe640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f6e70070140 con 0x7f6e700a57b0 2024-09-17T18:19:02.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.294+0000 7f6e782fe640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:02.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.294+0000 7f6e782fe640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f6e7006f540 con 0x7f6e700a53a0 2024-09-17T18:19:02.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.294+0000 7f6e782fe640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:02.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.294+0000 7f6e782fe640 10 monclient(hunting): authenticate will time out at 746.010996s 2024-09-17T18:19:02.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.295+0000 7f6e7609a640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f6e7006fd20 0x7f6e700a4c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:49510/0 (socket says 172.21.15.121:49510) 2024-09-17T18:19:02.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.295+0000 7f6e7609a640 1 -- 172.21.15.121:0/1812710815 learned_addr learned my addr 172.21.15.121:0/1812710815 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:19:02.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.295+0000 7f6e7689b640 1 --2- 172.21.15.121:0/1812710815 >> v2:172.21.15.121:3300/0 conn(0x7f6e700a57b0 0x7f6e701403b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:02.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.295+0000 7f6e7689b640 10 monclient(hunting): get_auth_request con 0x7f6e700a57b0 auth_method 0 2024-09-17T18:19:02.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.295+0000 7f6e7689b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:02.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.295+0000 7f6e7689b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:02.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.295+0000 7f6e7689b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:02.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.295+0000 7f6e75899640 1 --2- 172.21.15.121:0/1812710815 >> v2:172.21.15.173:3300/0 conn(0x7f6e700a53a0 0x7f6e7013de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:02.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.295+0000 7f6e7689b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:02.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.295+0000 7f6e7689b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:02.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.295+0000 7f6e7689b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:02.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.295+0000 7f6e75899640 10 monclient(hunting): get_auth_request con 0x7f6e700a53a0 auth_method 0 2024-09-17T18:19:02.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.295+0000 7f6e75899640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:02.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.295+0000 7f6e75899640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:02.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.295+0000 7f6e75899640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:02.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.295+0000 7f6e75899640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:02.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.295+0000 7f6e75899640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:02.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.295+0000 7f6e75899640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:02.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.295+0000 7f6e7689b640 10 monclient(hunting): handle_auth_done global_id 4221 payload 274 2024-09-17T18:19:02.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.295+0000 7f6e7689b640 1 -- 172.21.15.121:0/1812710815 >> v2:172.21.15.173:3300/0 conn(0x7f6e700a53a0 msgr2=0x7f6e7013de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:02.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.295+0000 7f6e7689b640 1 --2- 172.21.15.121:0/1812710815 >> v2:172.21.15.173:3300/0 conn(0x7f6e700a53a0 0x7f6e7013de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:02.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.295+0000 7f6e7689b640 1 -- 172.21.15.121:0/1812710815 >> v1:172.21.15.121:6789/0 conn(0x7f6e7006fd20 legacy=0x7f6e700a4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:02.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.295+0000 7f6e7689b640 10 monclient: _finish_hunting 0 2024-09-17T18:19:02.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.295+0000 7f6e7689b640 1 monclient: found mon.noname-a 2024-09-17T18:19:02.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.295+0000 7f6e7689b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:02.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.295+0000 7f6e7689b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:19:02.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.295+0000 7f6e7689b640 1 -- 172.21.15.121:0/1812710815 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e700708c0 con 0x7f6e700a57b0 2024-09-17T18:19:02.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.298+0000 7f6e7689b640 1 -- 172.21.15.121:0/1812710815 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e700a57b0 msgr2=0x7f6e701403b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:02.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.298+0000 7f6e7689b640 1 --2- 172.21.15.121:0/1812710815 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e700a57b0 0x7f6e701403b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6e7009fc40 tx=0x7f6e6802a4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8adeaa9615d5c1f server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:02.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.298+0000 7f6e75098640 1 -- 172.21.15.121:0/1812710815 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e68007950 con 0x7f6e700a57b0 2024-09-17T18:19:02.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.298+0000 7f6e75098640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:02.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.298+0000 7f6e75098640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:19:02.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.298+0000 7f6e75098640 10 monclient: dump: 2024-09-17T18:19:02.308 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:02.308 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:02.309 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:02.309 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:02.309 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:02.309 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:02.309 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:02.309 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:02.309 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:02.309 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:02.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.298+0000 7f6e75098640 10 monclient: _finish_auth 0 2024-09-17T18:19:02.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.298+0000 7f6e75098640 10 monclient: _check_auth_tickets 2024-09-17T18:19:02.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.298+0000 7f6e75098640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:02.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.298+0000 7f6e782fe640 5 monclient: authenticate success, global_id 4221 2024-09-17T18:19:02.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.298+0000 7f6e75098640 1 -- 172.21.15.121:0/1812710815 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e68007af0 con 0x7f6e700a57b0 2024-09-17T18:19:02.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.298+0000 7f6e782fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:19:02.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.298+0000 7f6e75098640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:02.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.298+0000 7f6e782fe640 10 monclient: get_monmap_and_config success 2024-09-17T18:19:02.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.298+0000 7f6e782fe640 4 set_mon_vals no callback set 2024-09-17T18:19:02.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.298+0000 7f6e75098640 1 -- 172.21.15.121:0/1812710815 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e68031400 con 0x7f6e700a57b0 2024-09-17T18:19:02.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.298+0000 7f6e75098640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:02.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.298+0000 7f6e75098640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:19:02.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.298+0000 7f6e75098640 10 monclient: dump: 2024-09-17T18:19:02.310 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:02.310 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:02.310 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:02.310 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:02.310 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:02.310 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:02.311 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:02.311 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:02.311 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:02.311 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:02.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.298+0000 7f6e782fe640 10 monclient: shutdown 2024-09-17T18:19:02.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.298+0000 7f6e782fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:02.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.298+0000 7f6e782fe640 1 -- 172.21.15.121:0/1812710815 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e700a57b0 msgr2=0x7f6e701403b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:02.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.298+0000 7f6e782fe640 1 --2- 172.21.15.121:0/1812710815 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e700a57b0 0x7f6e701403b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6e7009fc40 tx=0x7f6e6802a4b0 comp rx=0 tx=0).stop 2024-09-17T18:19:02.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.299+0000 7f6e782fe640 1 -- 172.21.15.121:0/1812710815 shutdown_connections 2024-09-17T18:19:02.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.299+0000 7f6e782fe640 1 --2- 172.21.15.121:0/1812710815 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e700a57b0 0x7f6e701403b0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:02.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.299+0000 7f6e782fe640 1 --2- 172.21.15.121:0/1812710815 >> v2:172.21.15.173:3300/0 conn(0x7f6e700a53a0 0x7f6e7013de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:02.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.299+0000 7f6e782fe640 1 -- 172.21.15.121:0/1812710815 >> 172.21.15.121:0/1812710815 conn(0x7f6e700a9a60 msgr2=0x7f6e700a5d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:02.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.299+0000 7f6e782fe640 1 -- 172.21.15.121:0/1812710815 shutdown_connections 2024-09-17T18:19:02.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.299+0000 7f6e782fe640 1 -- 172.21.15.121:0/1812710815 wait complete. 2024-09-17T18:19:02.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.299+0000 7f6e782fe640 10 monclient: build_initial_monmap 2024-09-17T18:19:02.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.299+0000 7f6e782fe640 10 monclient: monmap: 2024-09-17T18:19:02.312 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:02.312 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:19:02.312 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:19:02.312 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:19:02.312 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:02.312 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:02.312 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:19:02.312 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:19:02.312 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:19:02.313 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:02.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.299+0000 7f6e782fe640 1 librados: starting msgr at 2024-09-17T18:19:02.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.299+0000 7f6e782fe640 1 librados: starting objecter 2024-09-17T18:19:02.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.299+0000 7f6e782fe640 1 Processor -- start 2024-09-17T18:19:02.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e782fe640 1 -- start start 2024-09-17T18:19:02.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e782fe640 1 librados: setting wanted keys 2024-09-17T18:19:02.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e782fe640 1 librados: calling monclient init 2024-09-17T18:19:02.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e782fe640 10 monclient: init 2024-09-17T18:19:02.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e782fe640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:02.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e782fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:19:02.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e782fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e7006fd20 0x7f6e701ad9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:02.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e782fe640 10 monclient(hunting): picked mon.noname-a con 0x7f6e7006fd20 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:02.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e782fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e700a53a0 0x7f6e701cdd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:02.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e782fe640 10 monclient(hunting): picked mon.noname-b con 0x7f6e700a53a0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:02.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e782fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6e700a57b0 0x7f6e701c7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:02.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e782fe640 10 monclient(hunting): picked mon.noname-c con 0x7f6e700a57b0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:02.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e782fe640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:02.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e782fe640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6e700a0dc0 con 0x7f6e7006fd20 2024-09-17T18:19:02.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e782fe640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:02.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e782fe640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6e700a0c40 con 0x7f6e700a53a0 2024-09-17T18:19:02.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e782fe640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:02.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e782fe640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6e70074720 con 0x7f6e700a57b0 2024-09-17T18:19:02.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e782fe640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:02.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e782fe640 10 monclient(hunting): authenticate will time out at 746.016451s 2024-09-17T18:19:02.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e7609a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e7006fd20 0x7f6e701ad9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:02.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e7609a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e7006fd20 0x7f6e701ad9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40832/0 (socket says 172.21.15.121:40832) 2024-09-17T18:19:02.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e7609a640 1 -- 172.21.15.121:0/4039125409 learned_addr learned my addr 172.21.15.121:0/4039125409 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:02.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e7609a640 10 monclient(hunting): get_auth_request con 0x7f6e7006fd20 auth_method 0 2024-09-17T18:19:02.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e7609a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:02.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e7609a640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:02.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e7609a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:02.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e7689b640 1 --2- 172.21.15.121:0/4039125409 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6e700a57b0 0x7f6e701c7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:02.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e7689b640 10 monclient(hunting): get_auth_request con 0x7f6e700a57b0 auth_method 0 2024-09-17T18:19:02.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e7689b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:02.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e7689b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:02.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e7689b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:02.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e7609a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:02.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e7609a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:02.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e7609a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:02.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e7689b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:02.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e7689b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:02.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e7689b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:02.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e75899640 1 --2- 172.21.15.121:0/4039125409 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e700a53a0 0x7f6e701cdd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:02.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e75899640 10 monclient(hunting): get_auth_request con 0x7f6e700a53a0 auth_method 0 2024-09-17T18:19:02.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e75899640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:02.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e75899640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:02.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e75899640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:02.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e7609a640 10 monclient(hunting): handle_auth_done global_id 4224 payload 824 2024-09-17T18:19:02.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e7609a640 1 -- 172.21.15.121:0/4039125409 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6e700a57b0 msgr2=0x7f6e701c7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:02.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e7609a640 1 --2- 172.21.15.121:0/4039125409 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6e700a57b0 0x7f6e701c7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:02.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e7609a640 1 -- 172.21.15.121:0/4039125409 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e700a53a0 msgr2=0x7f6e701cdd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:02.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e7609a640 1 --2- 172.21.15.121:0/4039125409 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e700a53a0 0x7f6e701cdd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:02.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e7609a640 10 monclient: _finish_hunting 0 2024-09-17T18:19:02.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e7609a640 1 monclient: found mon.noname-a 2024-09-17T18:19:02.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e7609a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:02.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e7609a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:19:02.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e7609a640 1 -- 172.21.15.121:0/4039125409 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e701ae350 con 0x7f6e7006fd20 2024-09-17T18:19:02.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.300+0000 7f6e75899640 1 --2- 172.21.15.121:0/4039125409 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e700a53a0 0x7f6e701cdd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T18:19:02.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.303+0000 7f6e7609a640 1 -- 172.21.15.121:0/4039125409 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e7006fd20 msgr2=0x7f6e701ad9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:02.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.303+0000 7f6e7609a640 1 --2- 172.21.15.121:0/4039125409 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e7006fd20 0x7f6e701ad9c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c00c460 tx=0x7f6e6c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:02.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.303+0000 7f6e66ffd640 10 client.?.objecter ms_handle_connect 0x7f6e7006fd20 2024-09-17T18:19:02.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.303+0000 7f6e66ffd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:19:02.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.303+0000 7f6e66ffd640 1 -- 172.21.15.121:0/4039125409 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c007070 con 0x7f6e7006fd20 2024-09-17T18:19:02.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.303+0000 7f6e66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:02.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.303+0000 7f6e66ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:19:02.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.303+0000 7f6e66ffd640 10 monclient: dump: 2024-09-17T18:19:02.317 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:02.317 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:02.318 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:02.318 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:02.318 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:02.318 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:02.318 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:02.318 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:02.318 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:02.318 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:02.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.303+0000 7f6e66ffd640 10 monclient: _finish_auth 0 2024-09-17T18:19:02.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.303+0000 7f6e66ffd640 10 monclient: _check_auth_tickets 2024-09-17T18:19:02.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.303+0000 7f6e66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:02.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.303+0000 7f6e66ffd640 1 -- 172.21.15.121:0/4039125409 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e6c007230 con 0x7f6e7006fd20 2024-09-17T18:19:02.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.303+0000 7f6e782fe640 5 monclient: authenticate success, global_id 4224 2024-09-17T18:19:02.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.303+0000 7f6e782fe640 10 monclient: _renew_subs 2024-09-17T18:19:02.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.303+0000 7f6e782fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:02.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.303+0000 7f6e782fe640 1 -- 172.21.15.121:0/4039125409 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e700a0470 con 0x7f6e7006fd20 2024-09-17T18:19:02.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.303+0000 7f6e66ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:02.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.303+0000 7f6e66ffd640 1 -- 172.21.15.121:0/4039125409 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c007620 con 0x7f6e7006fd20 2024-09-17T18:19:02.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.303+0000 7f6e66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:02.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.303+0000 7f6e66ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:19:02.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.303+0000 7f6e66ffd640 10 monclient: dump: 2024-09-17T18:19:02.319 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:02.319 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:02.319 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:02.319 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:02.319 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:02.319 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:02.319 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:02.320 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:02.320 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:02.320 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:02.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.303+0000 7f6e782fe640 10 client.4224.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:02.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.303+0000 7f6e782fe640 10 monclient: _renew_subs 2024-09-17T18:19:02.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.303+0000 7f6e74897640 4 set_mon_vals no callback set 2024-09-17T18:19:02.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.303+0000 7f6e782fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:02.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.303+0000 7f6e782fe640 1 -- 172.21.15.121:0/4039125409 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e701af1c0 con 0x7f6e7006fd20 2024-09-17T18:19:02.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.303+0000 7f6e782fe640 1 librados: init done 2024-09-17T18:19:02.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.304+0000 7f6e66ffd640 1 -- 172.21.15.121:0/4039125409 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f6e6c01b400 con 0x7f6e7006fd20 2024-09-17T18:19:02.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.304+0000 7f6e782fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:02.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.304+0000 7f6e782fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:02.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.304+0000 7f6e782fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:02.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.304+0000 7f6e782fe640 1 -- 172.21.15.121:0/4039125409 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e701aee40 con 0x7f6e7006fd20 2024-09-17T18:19:02.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.304+0000 7f6e66ffd640 1 --2- 172.21.15.121:0/4039125409 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6e540401a0 0x7f6e54042660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:02.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.304+0000 7f6e66ffd640 1 -- 172.21.15.121:0/4039125409 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 3562+0+0 (secure 0 0 0) 0x7f6e6c052c90 con 0x7f6e7006fd20 2024-09-17T18:19:02.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.304+0000 7f6e66ffd640 10 client.4224.objecter ms_dispatch 0x7f6e70000b90 osd_map(16..16 src has 1..16) 2024-09-17T18:19:02.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.304+0000 7f6e66ffd640 3 client.4224.objecter handle_osd_map got epochs [16,16] > 0 2024-09-17T18:19:02.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.304+0000 7f6e66ffd640 3 client.4224.objecter handle_osd_map decoding full epoch 16 2024-09-17T18:19:02.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.304+0000 7f6e66ffd640 20 client.4224.objecter dump_active .. 0 homeless 2024-09-17T18:19:02.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.304+0000 7f6e75899640 1 --2- 172.21.15.121:0/4039125409 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6e540401a0 0x7f6e54042660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:02.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.304+0000 7f6e75899640 10 monclient: get_auth_request con 0x7f6e540401a0 auth_method 0 2024-09-17T18:19:02.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.307+0000 7f6e75899640 1 --2- 172.21.15.121:0/4039125409 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6e540401a0 0x7f6e54042660 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6e70072a70 tx=0x7f6e60008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:02.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.307+0000 7f6e66ffd640 10 client.4224.objecter ms_handle_connect 0x7f6e540401a0 2024-09-17T18:19:02.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.309+0000 7f6e66ffd640 1 -- 172.21.15.121:0/4039125409 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6e701aee40 con 0x7f6e7006fd20 2024-09-17T18:19:02.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.309+0000 7f6e66ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:02.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.309+0000 7f6e66ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:19:02.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.481+0000 7f6e782fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-17T18:19:02.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.481+0000 7f6e782fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T18:19:02.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.481+0000 7f6e782fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:02.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.481+0000 7f6e782fe640 1 -- 172.21.15.121:0/4039125409 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6e7006cd40 con 0x7f6e7006fd20 2024-09-17T18:19:02.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.482+0000 7f6e66ffd640 1 -- 172.21.15.121:0/4039125409 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+10977 (secure 0 0 0) 0x7f6e6c059080 con 0x7f6e7006fd20 2024-09-17T18:19:02.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.482+0000 7f6e66ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T18:19:02.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.482+0000 7f6e66ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:19:02.484 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:19:02.484 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":16,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","created":"2024-09-17T18:18:47.237753+0000","modified":"2024-09-17T18:19:01.521005+0000","last_up_change":"2024-09-17T18:18:57.511756+0000","last_in_change":"2024-09-17T18:18:49.165275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T18:18:57.521369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-17T18:18:58.173538+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7bf586d1-05ac-4c7e-8b88-313f07a76801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":1526257129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6801","nonce":1526257129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":1526257129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":1526257129}]},"public_addr":"172.21.15.121:6800/1526257129","cluster_addr":"172.21.15.121:6801/1526257129","heartbeat_back_addr":"172.21.15.121:6803/1526257129","heartbeat_front_addr":"172.21.15.121:6802/1526257129","state":["exists","up"]},{"osd":1,"uuid":"e26c6163-1eda-4200-82e1-57b752fd8a46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":196419920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6809","nonce":196419920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":196419920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":196419920}]},"public_addr":"172.21.15.121:6808/196419920","cluster_addr":"172.21.15.121:6809/196419920","heartbeat_back_addr":"172.21.15.121:6811/196419920","heartbeat_front_addr":"172.21.15.121:6810/196419920","state":["exists","up"]},{"osd":2,"uuid":"6fb45798-21d4-449e-9f68-7210d984e036","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2545987420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6805","nonce":2545987420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":2545987420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2545987420}]},"public_addr":"172.21.15.121:6804/2545987420","cluster_addr":"172.21.15.121:6805/2545987420","heartbeat_back_addr":"172.21.15.121:6807/2545987420","heartbeat_front_addr":"172.21.15.121:6806/2545987420","state":["exists","up"]},{"osd":3,"uuid":"3de6630d-4fb7-4f64-b4f9-e5c304b1be63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":3493964597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6801","nonce":3493964597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6803","nonce":3493964597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":3493964597}]},"public_addr":"172.21.15.173:6800/3493964597","cluster_addr":"172.21.15.173:6801/3493964597","heartbeat_back_addr":"172.21.15.173:6803/3493964597","heartbeat_front_addr":"172.21.15.173:6802/3493964597","state":["exists","up"]},{"osd":4,"uuid":"cfa74383-b7c9-4ecd-8258-3792fbccc237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":1961801317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6805","nonce":1961801317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6807","nonce":1961801317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":1961801317}]},"public_addr":"172.21.15.173:6804/1961801317","cluster_addr":"172.21.15.173:6805/1961801317","heartbeat_back_addr":"172.21.15.173:6807/1961801317","heartbeat_front_addr":"172.21.15.173:6806/1961801317","state":["exists","up"]},{"osd":5,"uuid":"8bdb7768-a603-4f14-9650-47b503621215","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1801998802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6809","nonce":1801998802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6811","nonce":1801998802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1801998802}]},"public_addr":"172.21.15.173:6808/1801998802","cluster_addr":"172.21.15.173:6809/1801998802","heartbeat_back_addr":"172.21.15.173:6811/1801998802","heartbeat_front_addr":"172.21.15.173:6810/1801998802","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T18:18:55.231353+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T18:18:56.130097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T18:18:56.242909+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T18:18:55.454088+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T18:18:55.965469+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T18:18:55.389354+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T18:19:02.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.484+0000 7f6e782fe640 10 librados: watch_flush enter 2024-09-17T18:19:02.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.484+0000 7f6e782fe640 10 librados: watch_flush exit 2024-09-17T18:19:02.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.484+0000 7f6e782fe640 20 client.4224.objecter shutdown clearing up homeless session... 2024-09-17T18:19:02.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.484+0000 7f6e782fe640 10 client.4224.objecter successfully canceled tick 2024-09-17T18:19:02.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.484+0000 7f6e782fe640 1 -- 172.21.15.121:0/4039125409 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6e540401a0 msgr2=0x7f6e54042660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:02.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.484+0000 7f6e782fe640 1 --2- 172.21.15.121:0/4039125409 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6e540401a0 0x7f6e54042660 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6e70072a70 tx=0x7f6e60008040 comp rx=0 tx=0).stop 2024-09-17T18:19:02.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.484+0000 7f6e782fe640 10 monclient: shutdown 2024-09-17T18:19:02.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.484+0000 7f6e782fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:02.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.484+0000 7f6e782fe640 1 -- 172.21.15.121:0/4039125409 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e7006fd20 msgr2=0x7f6e701ad9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:02.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.484+0000 7f6e782fe640 1 --2- 172.21.15.121:0/4039125409 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e7006fd20 0x7f6e701ad9c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c00c460 tx=0x7f6e6c00c920 comp rx=0 tx=0).stop 2024-09-17T18:19:02.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.484+0000 7f6e782fe640 1 -- 172.21.15.121:0/4039125409 shutdown_connections 2024-09-17T18:19:02.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.484+0000 7f6e782fe640 1 --2- 172.21.15.121:0/4039125409 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6e540401a0 0x7f6e54042660 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:02.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.484+0000 7f6e782fe640 1 --2- 172.21.15.121:0/4039125409 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e700a53a0 0x7f6e701cdd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:02.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.484+0000 7f6e782fe640 1 --2- 172.21.15.121:0/4039125409 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6e700a57b0 0x7f6e701c7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:02.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.484+0000 7f6e782fe640 1 --2- 172.21.15.121:0/4039125409 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e7006fd20 0x7f6e701ad9c0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:02.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.485+0000 7f6e782fe640 1 -- 172.21.15.121:0/4039125409 >> 172.21.15.121:0/4039125409 conn(0x7f6e700a9a60 msgr2=0x7f6e700a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:02.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.485+0000 7f6e782fe640 1 -- 172.21.15.121:0/4039125409 shutdown_connections 2024-09-17T18:19:02.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.485+0000 7f6e782fe640 1 -- 172.21.15.121:0/4039125409 wait complete. 2024-09-17T18:19:02.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.485+0000 7f6e782fe640 1 librados: shutdown 2024-09-17T18:19:02.497 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-17T18:19:02.497 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T18:19:02.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.592+0000 7fe24bb4b640 10 monclient: get_monmap_and_config 2024-09-17T18:19:02.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.592+0000 7fe24bb4b640 10 monclient: build_initial_monmap 2024-09-17T18:19:02.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.593+0000 7fe24bb4b640 10 monclient: monmap: 2024-09-17T18:19:02.595 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:02.595 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:02.595 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:19:02.593580+0000 2024-09-17T18:19:02.595 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:19:02.593580+0000 2024-09-17T18:19:02.595 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:02.595 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:02.595 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:19:02.595 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:19:02.595 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:19:02.595 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:19:02.595 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:19:02.595 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:02.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.593+0000 7fe24bb4b640 1 Processor -- start 2024-09-17T18:19:02.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.593+0000 7fe24bb4b640 1 -- start start 2024-09-17T18:19:02.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.593+0000 7fe24bb4b640 10 monclient: init 2024-09-17T18:19:02.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.594+0000 7fe24bb4b640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:02.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.594+0000 7fe24bb4b640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-17T18:19:02.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.594+0000 7fe24bb4b640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fe2441386b0 0x7fe2440707d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:02.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.594+0000 7fe24bb4b640 10 monclient(hunting): picked mon.noname-c con 0x7fe2441386b0 addr v2:172.21.15.173:3300/0 2024-09-17T18:19:02.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.594+0000 7fe24bb4b640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe24406ec00 0x7fe244138a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:02.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.594+0000 7fe24bb4b640 10 monclient(hunting): picked mon.noname-a con 0x7fe24406ec00 addr v2:172.21.15.121:3300/0 2024-09-17T18:19:02.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.594+0000 7fe24bb4b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe244138fd0 addr v1:172.21.15.121:6789/0 2024-09-17T18:19:02.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.594+0000 7fe24bb4b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2440a5570 con 0x7fe244138fd0 2024-09-17T18:19:02.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.594+0000 7fe24bb4b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:02.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.594+0000 7fe24bb4b640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fe24406f830 con 0x7fe24406ec00 2024-09-17T18:19:02.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.594+0000 7fe24bb4b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:02.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.594+0000 7fe24bb4b640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fe24406fb10 con 0x7fe2441386b0 2024-09-17T18:19:02.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.594+0000 7fe24bb4b640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:02.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.594+0000 7fe24bb4b640 10 monclient(hunting): authenticate will time out at 746.310270s 2024-09-17T18:19:02.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.594+0000 7fe2490e6640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe24406ec00 0x7fe244138a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:02.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.594+0000 7fe2490e6640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe24406ec00 0x7fe244138a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40842/0 (socket says 172.21.15.121:40842) 2024-09-17T18:19:02.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.594+0000 7fe2490e6640 1 -- 172.21.15.121:0/2376622315 learned_addr learned my addr 172.21.15.121:0/2376622315 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:02.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.594+0000 7fe24a0e8640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fe244138fd0 0x7fe244131d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:49518/0 (socket says 172.21.15.121:49518) 2024-09-17T18:19:02.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.594+0000 7fe2490e6640 10 monclient(hunting): get_auth_request con 0x7fe24406ec00 auth_method 0 2024-09-17T18:19:02.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.594+0000 7fe2490e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:02.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.594+0000 7fe2490e6640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:02.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.594+0000 7fe2490e6640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:02.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.594+0000 7fe2490e6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:02.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.594+0000 7fe2490e6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:02.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.594+0000 7fe2490e6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:02.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.594+0000 7fe2498e7640 1 --2- 172.21.15.121:0/2376622315 >> v2:172.21.15.173:3300/0 conn(0x7fe2441386b0 0x7fe2440707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:02.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.594+0000 7fe2490e6640 10 monclient(hunting): handle_auth_done global_id 4227 payload 274 2024-09-17T18:19:02.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.594+0000 7fe2490e6640 1 -- 172.21.15.121:0/2376622315 >> v2:172.21.15.173:3300/0 conn(0x7fe2441386b0 msgr2=0x7fe2440707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:02.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.594+0000 7fe2490e6640 1 --2- 172.21.15.121:0/2376622315 >> v2:172.21.15.173:3300/0 conn(0x7fe2441386b0 0x7fe2440707d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:02.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.594+0000 7fe2490e6640 1 -- 172.21.15.121:0/2376622315 >> v1:172.21.15.121:6789/0 conn(0x7fe244138fd0 legacy=0x7fe244131d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:02.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.594+0000 7fe2490e6640 10 monclient: _finish_hunting 0 2024-09-17T18:19:02.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.594+0000 7fe2490e6640 1 monclient: found mon.noname-a 2024-09-17T18:19:02.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.594+0000 7fe2490e6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:02.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.594+0000 7fe2490e6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:19:02.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.594+0000 7fe2490e6640 1 -- 172.21.15.121:0/2376622315 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe244139460 con 0x7fe24406ec00 2024-09-17T18:19:02.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.594+0000 7fe2498e7640 10 monclient: get_auth_request con 0x7fe2441386b0 auth_method 0 2024-09-17T18:19:02.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.594+0000 7fe2498e7640 1 --2- 172.21.15.121:0/2376622315 >> v2:172.21.15.173:3300/0 conn(0x7fe2441386b0 0x7fe2440707d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T18:19:02.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.597+0000 7fe2490e6640 1 -- 172.21.15.121:0/2376622315 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe24406ec00 msgr2=0x7fe244138a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:02.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.597+0000 7fe2490e6640 1 --2- 172.21.15.121:0/2376622315 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe24406ec00 0x7fe244138a90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe234009870 tx=0x7fe23402a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a2a63d8415d1d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:02.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.597+0000 7fe2488e5640 1 -- 172.21.15.121:0/2376622315 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe234037070 con 0x7fe24406ec00 2024-09-17T18:19:02.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.597+0000 7fe2488e5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:02.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.597+0000 7fe2488e5640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:19:02.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.597+0000 7fe2488e5640 10 monclient: dump: 2024-09-17T18:19:02.606 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:02.607 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:02.607 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:02.607 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:02.607 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:02.607 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:02.607 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:02.607 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:02.607 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:02.607 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:02.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.597+0000 7fe2488e5640 10 monclient: _finish_auth 0 2024-09-17T18:19:02.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.597+0000 7fe2488e5640 10 monclient: _check_auth_tickets 2024-09-17T18:19:02.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.597+0000 7fe2488e5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:02.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.597+0000 7fe24bb4b640 5 monclient: authenticate success, global_id 4227 2024-09-17T18:19:02.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.597+0000 7fe24bb4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:19:02.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.597+0000 7fe2488e5640 1 -- 172.21.15.121:0/2376622315 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe234032640 con 0x7fe24406ec00 2024-09-17T18:19:02.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.597+0000 7fe2488e5640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:02.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.597+0000 7fe24bb4b640 10 monclient: get_monmap_and_config success 2024-09-17T18:19:02.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.597+0000 7fe24bb4b640 4 set_mon_vals no callback set 2024-09-17T18:19:02.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.597+0000 7fe2488e5640 1 -- 172.21.15.121:0/2376622315 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe234032a10 con 0x7fe24406ec00 2024-09-17T18:19:02.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.597+0000 7fe2488e5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:02.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.597+0000 7fe2488e5640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:19:02.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.597+0000 7fe2488e5640 10 monclient: dump: 2024-09-17T18:19:02.608 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:02.608 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:02.608 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:02.608 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:02.609 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:02.609 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:02.609 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:02.609 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:02.609 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:02.609 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:02.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.598+0000 7fe24bb4b640 10 monclient: shutdown 2024-09-17T18:19:02.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.598+0000 7fe24bb4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:02.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.598+0000 7fe24bb4b640 1 -- 172.21.15.121:0/2376622315 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe24406ec00 msgr2=0x7fe244138a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:02.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.598+0000 7fe24bb4b640 1 --2- 172.21.15.121:0/2376622315 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe24406ec00 0x7fe244138a90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe234009870 tx=0x7fe23402a100 comp rx=0 tx=0).stop 2024-09-17T18:19:02.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.598+0000 7fe24bb4b640 1 -- 172.21.15.121:0/2376622315 shutdown_connections 2024-09-17T18:19:02.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.598+0000 7fe24bb4b640 1 --2- 172.21.15.121:0/2376622315 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe24406ec00 0x7fe244138a90 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:02.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.598+0000 7fe24bb4b640 1 --2- 172.21.15.121:0/2376622315 >> v2:172.21.15.173:3300/0 conn(0x7fe2441386b0 0x7fe2440707d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:02.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.598+0000 7fe24bb4b640 1 -- 172.21.15.121:0/2376622315 >> 172.21.15.121:0/2376622315 conn(0x7fe2440a91e0 msgr2=0x7fe2440a95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:02.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.598+0000 7fe24bb4b640 1 -- 172.21.15.121:0/2376622315 shutdown_connections 2024-09-17T18:19:02.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.598+0000 7fe24bb4b640 1 -- 172.21.15.121:0/2376622315 wait complete. 2024-09-17T18:19:02.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.598+0000 7fe24bb4b640 10 monclient: build_initial_monmap 2024-09-17T18:19:02.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.598+0000 7fe24bb4b640 10 monclient: monmap: 2024-09-17T18:19:02.610 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:02.610 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:19:02.610 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:19:02.610 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:19:02.610 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:02.610 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:02.610 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:19:02.610 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:19:02.610 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:19:02.610 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:02.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.598+0000 7fe24bb4b640 1 librados: starting msgr at 2024-09-17T18:19:02.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.598+0000 7fe24bb4b640 1 librados: starting objecter 2024-09-17T18:19:02.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe24bb4b640 1 Processor -- start 2024-09-17T18:19:02.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe24bb4b640 1 -- start start 2024-09-17T18:19:02.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe24bb4b640 1 librados: setting wanted keys 2024-09-17T18:19:02.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe24bb4b640 1 librados: calling monclient init 2024-09-17T18:19:02.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe24bb4b640 10 monclient: init 2024-09-17T18:19:02.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe24bb4b640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:02.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe24bb4b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:19:02.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe24bb4b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe24406ec00 0x7fe244072ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:02.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe24bb4b640 10 monclient(hunting): picked mon.noname-a con 0x7fe24406ec00 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:02.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe24bb4b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2441386b0 0x7fe2440730e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:02.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe24bb4b640 10 monclient(hunting): picked mon.noname-b con 0x7fe2441386b0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:02.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe24bb4b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe244138fd0 0x7fe244073620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:02.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe24bb4b640 10 monclient(hunting): picked mon.noname-c con 0x7fe244138fd0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:02.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe24bb4b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:02.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe24bb4b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe2441325b0 con 0x7fe24406ec00 2024-09-17T18:19:02.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe24bb4b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:02.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe24bb4b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe244132430 con 0x7fe2441386b0 2024-09-17T18:19:02.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe24bb4b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:02.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe24bb4b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe244132730 con 0x7fe244138fd0 2024-09-17T18:19:02.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe24bb4b640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:02.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe24bb4b640 10 monclient(hunting): authenticate will time out at 746.315579s 2024-09-17T18:19:02.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe2498e7640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe24406ec00 0x7fe244072ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:02.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe2498e7640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe24406ec00 0x7fe244072ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40852/0 (socket says 172.21.15.121:40852) 2024-09-17T18:19:02.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe2498e7640 1 -- 172.21.15.121:0/2295938652 learned_addr learned my addr 172.21.15.121:0/2295938652 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:02.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe2498e7640 10 monclient(hunting): get_auth_request con 0x7fe24406ec00 auth_method 0 2024-09-17T18:19:02.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe2498e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:02.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe2498e7640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:02.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe2498e7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:02.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe24a0e8640 1 --2- 172.21.15.121:0/2295938652 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe244138fd0 0x7fe244073620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:02.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe24a0e8640 10 monclient(hunting): get_auth_request con 0x7fe244138fd0 auth_method 0 2024-09-17T18:19:02.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe24a0e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:02.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe24a0e8640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:02.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe24a0e8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:02.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe2498e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:02.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe2498e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:02.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe2498e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:02.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe24a0e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:02.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe24a0e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:02.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe24a0e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:02.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe2490e6640 1 --2- 172.21.15.121:0/2295938652 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2441386b0 0x7fe2440730e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:02.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe2490e6640 10 monclient(hunting): get_auth_request con 0x7fe2441386b0 auth_method 0 2024-09-17T18:19:02.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe2490e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:02.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe2490e6640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:02.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe2490e6640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:02.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.599+0000 7fe2498e7640 10 monclient(hunting): handle_auth_done global_id 4230 payload 824 2024-09-17T18:19:02.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.600+0000 7fe2498e7640 1 -- 172.21.15.121:0/2295938652 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe244138fd0 msgr2=0x7fe244073620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:02.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.600+0000 7fe2498e7640 1 --2- 172.21.15.121:0/2295938652 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe244138fd0 0x7fe244073620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:02.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.600+0000 7fe2498e7640 1 -- 172.21.15.121:0/2295938652 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2441386b0 msgr2=0x7fe2440730e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:02.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.600+0000 7fe2498e7640 1 --2- 172.21.15.121:0/2295938652 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2441386b0 0x7fe2440730e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:02.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.600+0000 7fe2498e7640 10 monclient: _finish_hunting 0 2024-09-17T18:19:02.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.600+0000 7fe2498e7640 1 monclient: found mon.noname-a 2024-09-17T18:19:02.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.600+0000 7fe2498e7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:02.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.600+0000 7fe2498e7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:19:02.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.600+0000 7fe2498e7640 1 -- 172.21.15.121:0/2295938652 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2440a03d0 con 0x7fe24406ec00 2024-09-17T18:19:02.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.600+0000 7fe24a0e8640 1 --2- 172.21.15.121:0/2295938652 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe244138fd0 0x7fe244073620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:19:02.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.602+0000 7fe2498e7640 1 -- 172.21.15.121:0/2295938652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe24406ec00 msgr2=0x7fe244072ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:02.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.602+0000 7fe2498e7640 1 --2- 172.21.15.121:0/2295938652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe24406ec00 0x7fe244072ba0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe24000c7f0 tx=0x7fe24000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:02.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.602+0000 7fe23a7fc640 10 client.?.objecter ms_handle_connect 0x7fe24406ec00 2024-09-17T18:19:02.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.602+0000 7fe23a7fc640 10 client.?.objecter resend_mon_ops 2024-09-17T18:19:02.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.602+0000 7fe23a7fc640 1 -- 172.21.15.121:0/2295938652 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe240015070 con 0x7fe24406ec00 2024-09-17T18:19:02.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.602+0000 7fe23a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:02.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.602+0000 7fe23a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:19:02.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.602+0000 7fe23a7fc640 10 monclient: dump: 2024-09-17T18:19:02.615 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:02.615 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:02.615 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:02.616 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:02.616 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:02.616 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:02.616 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:02.616 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:02.616 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:02.616 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:02.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.602+0000 7fe23a7fc640 10 monclient: _finish_auth 0 2024-09-17T18:19:02.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.602+0000 7fe23a7fc640 10 monclient: _check_auth_tickets 2024-09-17T18:19:02.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.602+0000 7fe23a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:02.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.602+0000 7fe24bb4b640 5 monclient: authenticate success, global_id 4230 2024-09-17T18:19:02.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.602+0000 7fe23a7fc640 1 -- 172.21.15.121:0/2295938652 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe240007c40 con 0x7fe24406ec00 2024-09-17T18:19:02.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.602+0000 7fe24bb4b640 10 monclient: _renew_subs 2024-09-17T18:19:02.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.602+0000 7fe24bb4b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:02.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.602+0000 7fe24bb4b640 1 -- 172.21.15.121:0/2295938652 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2440a1d80 con 0x7fe24406ec00 2024-09-17T18:19:02.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.602+0000 7fe23a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:02.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.602+0000 7fe23a7fc640 1 -- 172.21.15.121:0/2295938652 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe24000ce40 con 0x7fe24406ec00 2024-09-17T18:19:02.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.602+0000 7fe23a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:02.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.602+0000 7fe23a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:19:02.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.602+0000 7fe23a7fc640 10 monclient: dump: 2024-09-17T18:19:02.617 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:02.617 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:02.617 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:02.617 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:02.617 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:02.617 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:02.617 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:02.617 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:02.617 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:02.617 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:02.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.602+0000 7fe24bb4b640 10 client.4230.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:02.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.602+0000 7fe23bfff640 4 set_mon_vals no callback set 2024-09-17T18:19:02.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.602+0000 7fe24bb4b640 10 monclient: _renew_subs 2024-09-17T18:19:02.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.602+0000 7fe24bb4b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:02.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.602+0000 7fe24bb4b640 1 -- 172.21.15.121:0/2295938652 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2440744d0 con 0x7fe24406ec00 2024-09-17T18:19:02.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.602+0000 7fe24bb4b640 1 librados: init done 2024-09-17T18:19:02.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.603+0000 7fe24bb4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:02.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.603+0000 7fe24bb4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:02.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.603+0000 7fe23a7fc640 1 -- 172.21.15.121:0/2295938652 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fe240018070 con 0x7fe24406ec00 2024-09-17T18:19:02.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.603+0000 7fe24bb4b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:02.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.603+0000 7fe24bb4b640 1 -- 172.21.15.121:0/2295938652 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe244132aa0 con 0x7fe24406ec00 2024-09-17T18:19:02.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.603+0000 7fe23a7fc640 1 --2- 172.21.15.121:0/2295938652 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe2280401c0 0x7fe228042680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:02.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.603+0000 7fe23a7fc640 1 -- 172.21.15.121:0/2295938652 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 3562+0+0 (secure 0 0 0) 0x7fe240053090 con 0x7fe24406ec00 2024-09-17T18:19:02.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.603+0000 7fe23a7fc640 10 client.4230.objecter ms_dispatch 0x7fe244000b90 osd_map(16..16 src has 1..16) 2024-09-17T18:19:02.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.603+0000 7fe23a7fc640 3 client.4230.objecter handle_osd_map got epochs [16,16] > 0 2024-09-17T18:19:02.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.603+0000 7fe23a7fc640 3 client.4230.objecter handle_osd_map decoding full epoch 16 2024-09-17T18:19:02.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.603+0000 7fe23a7fc640 20 client.4230.objecter dump_active .. 0 homeless 2024-09-17T18:19:02.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.603+0000 7fe2490e6640 1 --2- 172.21.15.121:0/2295938652 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe2280401c0 0x7fe228042680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:02.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.603+0000 7fe2490e6640 10 monclient: get_auth_request con 0x7fe2280401c0 auth_method 0 2024-09-17T18:19:02.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.606+0000 7fe2490e6640 1 --2- 172.21.15.121:0/2295938652 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe2280401c0 0x7fe228042680 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe24406f4c0 tx=0x7fe23402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:02.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.606+0000 7fe23a7fc640 10 client.4230.objecter ms_handle_connect 0x7fe2280401c0 2024-09-17T18:19:02.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.608+0000 7fe23a7fc640 1 -- 172.21.15.121:0/2295938652 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe2440a03d0 con 0x7fe24406ec00 2024-09-17T18:19:02.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.608+0000 7fe23a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:02.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.608+0000 7fe23a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:19:02.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.780+0000 7fe24bb4b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-17T18:19:02.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.780+0000 7fe24bb4b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T18:19:02.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.780+0000 7fe24bb4b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:02.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.780+0000 7fe24bb4b640 1 -- 172.21.15.121:0/2295938652 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe24406cd40 con 0x7fe24406ec00 2024-09-17T18:19:02.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.781+0000 7fe23a7fc640 1 -- 172.21.15.121:0/2295938652 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+10977 (secure 0 0 0) 0x7fe240051f50 con 0x7fe24406ec00 2024-09-17T18:19:02.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.781+0000 7fe23a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T18:19:02.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.781+0000 7fe23a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:19:02.783 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:19:02.784 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":16,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","created":"2024-09-17T18:18:47.237753+0000","modified":"2024-09-17T18:19:01.521005+0000","last_up_change":"2024-09-17T18:18:57.511756+0000","last_in_change":"2024-09-17T18:18:49.165275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T18:18:57.521369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-17T18:18:58.173538+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7bf586d1-05ac-4c7e-8b88-313f07a76801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":1526257129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6801","nonce":1526257129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":1526257129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":1526257129}]},"public_addr":"172.21.15.121:6800/1526257129","cluster_addr":"172.21.15.121:6801/1526257129","heartbeat_back_addr":"172.21.15.121:6803/1526257129","heartbeat_front_addr":"172.21.15.121:6802/1526257129","state":["exists","up"]},{"osd":1,"uuid":"e26c6163-1eda-4200-82e1-57b752fd8a46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":196419920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6809","nonce":196419920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":196419920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":196419920}]},"public_addr":"172.21.15.121:6808/196419920","cluster_addr":"172.21.15.121:6809/196419920","heartbeat_back_addr":"172.21.15.121:6811/196419920","heartbeat_front_addr":"172.21.15.121:6810/196419920","state":["exists","up"]},{"osd":2,"uuid":"6fb45798-21d4-449e-9f68-7210d984e036","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2545987420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6805","nonce":2545987420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":2545987420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2545987420}]},"public_addr":"172.21.15.121:6804/2545987420","cluster_addr":"172.21.15.121:6805/2545987420","heartbeat_back_addr":"172.21.15.121:6807/2545987420","heartbeat_front_addr":"172.21.15.121:6806/2545987420","state":["exists","up"]},{"osd":3,"uuid":"3de6630d-4fb7-4f64-b4f9-e5c304b1be63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":3493964597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6801","nonce":3493964597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6803","nonce":3493964597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":3493964597}]},"public_addr":"172.21.15.173:6800/3493964597","cluster_addr":"172.21.15.173:6801/3493964597","heartbeat_back_addr":"172.21.15.173:6803/3493964597","heartbeat_front_addr":"172.21.15.173:6802/3493964597","state":["exists","up"]},{"osd":4,"uuid":"cfa74383-b7c9-4ecd-8258-3792fbccc237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":1961801317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6805","nonce":1961801317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6807","nonce":1961801317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":1961801317}]},"public_addr":"172.21.15.173:6804/1961801317","cluster_addr":"172.21.15.173:6805/1961801317","heartbeat_back_addr":"172.21.15.173:6807/1961801317","heartbeat_front_addr":"172.21.15.173:6806/1961801317","state":["exists","up"]},{"osd":5,"uuid":"8bdb7768-a603-4f14-9650-47b503621215","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1801998802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6809","nonce":1801998802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6811","nonce":1801998802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1801998802}]},"public_addr":"172.21.15.173:6808/1801998802","cluster_addr":"172.21.15.173:6809/1801998802","heartbeat_back_addr":"172.21.15.173:6811/1801998802","heartbeat_front_addr":"172.21.15.173:6810/1801998802","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T18:18:55.231353+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T18:18:56.130097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T18:18:56.242909+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T18:18:55.454088+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T18:18:55.965469+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T18:18:55.389354+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T18:19:02.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.783+0000 7fe24bb4b640 10 librados: watch_flush enter 2024-09-17T18:19:02.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.783+0000 7fe24bb4b640 10 librados: watch_flush exit 2024-09-17T18:19:02.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.783+0000 7fe24bb4b640 20 client.4230.objecter shutdown clearing up homeless session... 2024-09-17T18:19:02.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.783+0000 7fe24bb4b640 10 client.4230.objecter successfully canceled tick 2024-09-17T18:19:02.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.783+0000 7fe24bb4b640 1 -- 172.21.15.121:0/2295938652 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe2280401c0 msgr2=0x7fe228042680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:02.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.783+0000 7fe24bb4b640 1 --2- 172.21.15.121:0/2295938652 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe2280401c0 0x7fe228042680 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe24406f4c0 tx=0x7fe23402e000 comp rx=0 tx=0).stop 2024-09-17T18:19:02.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.783+0000 7fe24bb4b640 10 monclient: shutdown 2024-09-17T18:19:02.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.783+0000 7fe24bb4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:02.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.783+0000 7fe24bb4b640 1 -- 172.21.15.121:0/2295938652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe24406ec00 msgr2=0x7fe244072ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:02.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.783+0000 7fe24bb4b640 1 --2- 172.21.15.121:0/2295938652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe24406ec00 0x7fe244072ba0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe24000c7f0 tx=0x7fe24000cbf0 comp rx=0 tx=0).stop 2024-09-17T18:19:02.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.784+0000 7fe24bb4b640 1 -- 172.21.15.121:0/2295938652 shutdown_connections 2024-09-17T18:19:02.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.784+0000 7fe24bb4b640 1 --2- 172.21.15.121:0/2295938652 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe2280401c0 0x7fe228042680 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:02.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.784+0000 7fe24bb4b640 1 --2- 172.21.15.121:0/2295938652 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2441386b0 0x7fe2440730e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:02.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.784+0000 7fe24bb4b640 1 --2- 172.21.15.121:0/2295938652 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe244138fd0 0x7fe244073620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:02.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.784+0000 7fe24bb4b640 1 --2- 172.21.15.121:0/2295938652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe24406ec00 0x7fe244072ba0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:02.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.784+0000 7fe24bb4b640 1 -- 172.21.15.121:0/2295938652 >> 172.21.15.121:0/2295938652 conn(0x7fe2440a91e0 msgr2=0x7fe2441375e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:02.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.784+0000 7fe24bb4b640 1 -- 172.21.15.121:0/2295938652 shutdown_connections 2024-09-17T18:19:02.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.784+0000 7fe24bb4b640 1 -- 172.21.15.121:0/2295938652 wait complete. 2024-09-17T18:19:02.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.784+0000 7fe24bb4b640 1 librados: shutdown 2024-09-17T18:19:02.796 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-17T18:19:02.796 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-17T18:19:02.797 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-17T18:19:02.797 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-17T18:19:02.798 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-17T18:19:02.798 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-17T18:19:02.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.908+0000 7ff1137a0640 10 monclient: get_monmap_and_config 2024-09-17T18:19:02.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.908+0000 7f00a1ccd640 10 monclient: get_monmap_and_config 2024-09-17T18:19:02.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.908+0000 7f00a1ccd640 10 monclient: build_initial_monmap 2024-09-17T18:19:02.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.908+0000 7f00a1ccd640 10 monclient: monmap: 2024-09-17T18:19:02.911 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:02.911 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:02.911 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:19:02.909359+0000 2024-09-17T18:19:02.911 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:19:02.909359+0000 2024-09-17T18:19:02.911 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:02.911 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:02.911 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:19:02.912 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:19:02.912 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:19:02.912 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:19:02.912 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:19:02.912 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:02.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.908+0000 7ff1137a0640 10 monclient: build_initial_monmap 2024-09-17T18:19:02.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.908+0000 7ff1137a0640 10 monclient: monmap: 2024-09-17T18:19:02.913 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:02.913 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:02.913 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:19:02.909359+0000 2024-09-17T18:19:02.913 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:19:02.909359+0000 2024-09-17T18:19:02.913 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:02.913 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:02.913 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:19:02.913 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:19:02.913 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:19:02.913 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:19:02.913 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:19:02.913 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:02.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.909+0000 7ff1137a0640 1 Processor -- start 2024-09-17T18:19:02.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.909+0000 7f00a1ccd640 1 Processor -- start 2024-09-17T18:19:02.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.909+0000 7f00a1ccd640 1 -- start start 2024-09-17T18:19:02.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.909+0000 7f00a1ccd640 10 monclient: init 2024-09-17T18:19:02.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.909+0000 7ff1137a0640 1 -- start start 2024-09-17T18:19:02.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.909+0000 7ff1137a0640 10 monclient: init 2024-09-17T18:19:02.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7f00a1ccd640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:02.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7f00a1ccd640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-17T18:19:02.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7f00a1ccd640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f009c0a5b90 0x7f009c0a5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:02.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7ff1137a0640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:02.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7f00a1ccd640 10 monclient(hunting): picked mon.noname-c con 0x7f009c0a5b90 addr v2:172.21.15.173:3300/0 2024-09-17T18:19:02.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7f00a1ccd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f009c0a7080 addr v1:172.21.15.173:6789/0 2024-09-17T18:19:02.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7ff1137a0640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-17T18:19:02.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7ff1137a0640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7ff10c0a5ca0 0x7ff10c0a6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:02.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7ff1137a0640 10 monclient(hunting): picked mon.noname-c con 0x7ff10c0a5ca0 addr v2:172.21.15.173:3300/0 2024-09-17T18:19:02.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7ff1137a0640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff10c0a65c0 0x7ff10c13ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:02.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7f00a1ccd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f009c0a64b0 addr v1:172.21.15.121:6789/0 2024-09-17T18:19:02.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7f00a1ccd640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f009c139770 con 0x7f009c0a64b0 2024-09-17T18:19:02.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7f00a1ccd640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f009c0a8ab0 con 0x7f009c0a7080 2024-09-17T18:19:02.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7f00a1ccd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:02.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7f00a1ccd640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f009c09dff0 con 0x7f009c0a5b90 2024-09-17T18:19:02.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7f00a1ccd640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:02.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7f00a1ccd640 10 monclient(hunting): authenticate will time out at 746.626349s 2024-09-17T18:19:02.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7f009bfff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f009c0a64b0 0x7f009c139060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:49530/0 (socket says 172.21.15.121:49530) 2024-09-17T18:19:02.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7f009bfff640 1 -- 172.21.15.121:0/1137695313 learned_addr learned my addr 172.21.15.121:0/1137695313 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:19:02.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7f009b7fe640 1 --2- 172.21.15.121:0/1137695313 >> v2:172.21.15.173:3300/0 conn(0x7f009c0a5b90 0x7f009c0a5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:02.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7f009b7fe640 10 monclient(hunting): get_auth_request con 0x7f009c0a5b90 auth_method 0 2024-09-17T18:19:02.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7f009b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:02.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7f009b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:02.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7ff1137a0640 10 monclient(hunting): picked mon.noname-a con 0x7ff10c0a65c0 addr v2:172.21.15.121:3300/0 2024-09-17T18:19:02.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7ff1137a0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff10c13cfc0 addr v1:172.21.15.121:6789/0 2024-09-17T18:19:02.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7ff1137a0640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff10c0a7b50 con 0x7ff10c13cfc0 2024-09-17T18:19:02.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7ff1137a0640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:02.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7f009b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:02.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7ff1137a0640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7ff10c0704c0 con 0x7ff10c0a65c0 2024-09-17T18:19:02.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7ff1137a0640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:02.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7ff1137a0640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7ff10c0a5490 con 0x7ff10c0a5ca0 2024-09-17T18:19:02.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7ff110d3b640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff10c0a65c0 0x7ff10c13ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:02.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7ff110d3b640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff10c0a65c0 0x7ff10c13ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40858/0 (socket says 172.21.15.121:40858) 2024-09-17T18:19:02.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7ff110d3b640 1 -- 172.21.15.121:0/4044864637 learned_addr learned my addr 172.21.15.121:0/4044864637 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:02.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7ff1137a0640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:02.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7ff1137a0640 10 monclient(hunting): authenticate will time out at 746.626687s 2024-09-17T18:19:02.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7ff110d3b640 10 monclient(hunting): get_auth_request con 0x7ff10c0a65c0 auth_method 0 2024-09-17T18:19:02.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7ff11153c640 1 --2- 172.21.15.121:0/4044864637 >> v2:172.21.15.173:3300/0 conn(0x7ff10c0a5ca0 0x7ff10c0a6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:02.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7ff110d3b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:02.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7ff110d3b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:02.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7ff110d3b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:02.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7ff11153c640 10 monclient(hunting): get_auth_request con 0x7ff10c0a5ca0 auth_method 0 2024-09-17T18:19:02.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7f009b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:02.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7f009b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:02.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7f009b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:02.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7ff11153c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:02.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7ff11153c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:02.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7ff11153c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:02.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7ff110d3b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:02.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7ff110d3b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:02.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7ff110d3b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:02.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7ff11153c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:02.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7ff11153c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:02.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7ff11153c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:02.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.911+0000 7ff11153c640 10 monclient(hunting): handle_auth_done global_id 4228 payload 274 2024-09-17T18:19:02.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7f009b7fe640 10 monclient(hunting): handle_auth_done global_id 4225 payload 274 2024-09-17T18:19:02.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7f009b7fe640 1 -- 172.21.15.121:0/1137695313 >> v1:172.21.15.173:6789/0 conn(0x7f009c0a7080 legacy=0x7f009c138950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:02.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7f009b7fe640 1 -- 172.21.15.121:0/1137695313 >> v1:172.21.15.121:6789/0 conn(0x7f009c0a64b0 legacy=0x7f009c139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:02.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7f009b7fe640 10 monclient: _finish_hunting 0 2024-09-17T18:19:02.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.910+0000 7f009b7fe640 1 monclient: found mon.noname-c 2024-09-17T18:19:02.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.911+0000 7f009b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:02.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.911+0000 7f009b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.173:3300/0 2024-09-17T18:19:02.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.911+0000 7f009b7fe640 1 -- 172.21.15.121:0/1137695313 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f009c09ca60 con 0x7f009c0a5b90 2024-09-17T18:19:02.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.911+0000 7ff11153c640 1 -- 172.21.15.121:0/4044864637 >> v2:172.21.15.121:3300/0 conn(0x7ff10c0a65c0 msgr2=0x7ff10c13ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:02.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.911+0000 7ff11153c640 1 --2- 172.21.15.121:0/4044864637 >> v2:172.21.15.121:3300/0 conn(0x7ff10c0a65c0 0x7ff10c13ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:02.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.911+0000 7ff11153c640 1 -- 172.21.15.121:0/4044864637 >> v1:172.21.15.121:6789/0 conn(0x7ff10c13cfc0 legacy=0x7ff10c13f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:02.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.911+0000 7ff11153c640 10 monclient: _finish_hunting 0 2024-09-17T18:19:02.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.911+0000 7ff11153c640 1 monclient: found mon.noname-c 2024-09-17T18:19:02.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.911+0000 7ff11153c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:02.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.911+0000 7ff11153c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.173:3300/0 2024-09-17T18:19:02.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.911+0000 7ff11153c640 1 -- 172.21.15.121:0/4044864637 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff10c0a2a50 con 0x7ff10c0a5ca0 2024-09-17T18:19:02.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.914+0000 7f009b7fe640 1 -- 172.21.15.121:0/1137695313 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f009c0a5b90 msgr2=0x7f009c0a5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:02.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.914+0000 7ff11153c640 1 -- 172.21.15.121:0/4044864637 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff10c0a5ca0 msgr2=0x7ff10c0a6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:02.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.914+0000 7ff11153c640 1 --2- 172.21.15.121:0/4044864637 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff10c0a5ca0 0x7ff10c0a6080 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff10c06c260 tx=0x7ff108029fe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b6500f03aadc52f server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:02.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.914+0000 7f009b7fe640 1 --2- 172.21.15.121:0/1137695313 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f009c0a5b90 0x7f009c0a5f70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f008c0097a0 tx=0x7f008c029fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b83373fa933b800b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:02.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.914+0000 7f009a7fc640 1 -- 172.21.15.121:0/1137695313 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008c037070 con 0x7f009c0a5b90 2024-09-17T18:19:02.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.914+0000 7f009a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:02.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.914+0000 7ff103fff640 1 -- 172.21.15.121:0/4044864637 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff108037070 con 0x7ff10c0a5ca0 2024-09-17T18:19:02.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.914+0000 7ff103fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:02.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.914+0000 7ff103fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:19:02.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.914+0000 7ff103fff640 10 monclient: dump: 2024-09-17T18:19:02.927 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:02.927 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:02.927 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:02.928 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:02.928 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:02.928 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:02.928 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:02.928 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:02.928 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:02.928 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:02.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.914+0000 7ff103fff640 10 monclient: _finish_auth 0 2024-09-17T18:19:02.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.914+0000 7ff103fff640 10 monclient: _check_auth_tickets 2024-09-17T18:19:02.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.914+0000 7ff103fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:02.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.914+0000 7f009a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:19:02.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.914+0000 7f009a7fc640 10 monclient: dump: 2024-09-17T18:19:02.929 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:02.929 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:02.929 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:02.929 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:02.929 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:02.929 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:02.929 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:02.929 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:02.929 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:02.929 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:02.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.914+0000 7f009a7fc640 10 monclient: _finish_auth 0 2024-09-17T18:19:02.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.914+0000 7f009a7fc640 10 monclient: _check_auth_tickets 2024-09-17T18:19:02.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.914+0000 7f009a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:02.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.914+0000 7ff1137a0640 5 monclient: authenticate success, global_id 4228 2024-09-17T18:19:02.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.914+0000 7ff1137a0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:19:02.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.914+0000 7ff103fff640 1 -- 172.21.15.121:0/4044864637 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff108032440 con 0x7ff10c0a5ca0 2024-09-17T18:19:02.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.914+0000 7f009a7fc640 1 -- 172.21.15.121:0/1137695313 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f008c032640 con 0x7f009c0a5b90 2024-09-17T18:19:02.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.914+0000 7f009a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:02.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.914+0000 7f009a7fc640 1 -- 172.21.15.121:0/1137695313 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008c032a10 con 0x7f009c0a5b90 2024-09-17T18:19:02.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.914+0000 7f009a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:02.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.914+0000 7f009a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:19:02.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.914+0000 7f009a7fc640 10 monclient: dump: 2024-09-17T18:19:02.931 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:02.931 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:02.931 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:02.931 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:02.931 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:02.931 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:02.931 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:02.931 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:02.931 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:02.931 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:02.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.914+0000 7ff103fff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:02.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.914+0000 7ff103fff640 1 -- 172.21.15.121:0/4044864637 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff108032810 con 0x7ff10c0a5ca0 2024-09-17T18:19:02.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.914+0000 7ff103fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:02.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.914+0000 7ff103fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:19:02.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.914+0000 7ff103fff640 10 monclient: dump: 2024-09-17T18:19:02.932 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:02.932 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:02.932 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:02.932 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:02.932 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:02.932 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:02.933 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:02.933 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:02.933 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:02.933 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:02.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.914+0000 7f00a1ccd640 5 monclient: authenticate success, global_id 4225 2024-09-17T18:19:02.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.914+0000 7ff1137a0640 10 monclient: get_monmap_and_config success 2024-09-17T18:19:02.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.914+0000 7ff1137a0640 4 set_mon_vals no callback set 2024-09-17T18:19:02.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.914+0000 7f00a1ccd640 10 monclient: get_monmap_and_config success 2024-09-17T18:19:02.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.914+0000 7f00a1ccd640 4 set_mon_vals no callback set 2024-09-17T18:19:02.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.915+0000 7ff1137a0640 10 monclient: shutdown 2024-09-17T18:19:02.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.915+0000 7ff1137a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:02.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.915+0000 7ff1137a0640 1 -- 172.21.15.121:0/4044864637 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff10c0a5ca0 msgr2=0x7ff10c0a6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:02.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.915+0000 7ff1137a0640 1 --2- 172.21.15.121:0/4044864637 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff10c0a5ca0 0x7ff10c0a6080 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff10c06c260 tx=0x7ff108029fe0 comp rx=0 tx=0).stop 2024-09-17T18:19:02.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.915+0000 7f00a1ccd640 10 monclient: shutdown 2024-09-17T18:19:02.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.915+0000 7f00a1ccd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:02.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.915+0000 7f00a1ccd640 1 -- 172.21.15.121:0/1137695313 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f009c0a5b90 msgr2=0x7f009c0a5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:02.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.915+0000 7ff1137a0640 1 -- 172.21.15.121:0/4044864637 shutdown_connections 2024-09-17T18:19:02.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.915+0000 7ff1137a0640 1 --2- 172.21.15.121:0/4044864637 >> v2:172.21.15.121:3300/0 conn(0x7ff10c0a65c0 0x7ff10c13ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:02.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.915+0000 7ff1137a0640 1 --2- 172.21.15.121:0/4044864637 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff10c0a5ca0 0x7ff10c0a6080 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:02.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.915+0000 7f00a1ccd640 1 --2- 172.21.15.121:0/1137695313 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f009c0a5b90 0x7f009c0a5f70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f008c0097a0 tx=0x7f008c029fd0 comp rx=0 tx=0).stop 2024-09-17T18:19:02.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.915+0000 7ff1137a0640 1 -- 172.21.15.121:0/4044864637 >> 172.21.15.121:0/4044864637 conn(0x7ff10c06dd10 msgr2=0x7ff10c0a7da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:02.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.915+0000 7f00a1ccd640 1 -- 172.21.15.121:0/1137695313 shutdown_connections 2024-09-17T18:19:02.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.915+0000 7f00a1ccd640 1 --2- 172.21.15.121:0/1137695313 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f009c0a5b90 0x7f009c0a5f70 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:02.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.915+0000 7ff1137a0640 1 -- 172.21.15.121:0/4044864637 shutdown_connections 2024-09-17T18:19:02.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.915+0000 7f00a1ccd640 1 -- 172.21.15.121:0/1137695313 >> 172.21.15.121:0/1137695313 conn(0x7f009c0ab880 msgr2=0x7f009c0a7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:02.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.915+0000 7ff1137a0640 1 -- 172.21.15.121:0/4044864637 wait complete. 2024-09-17T18:19:02.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.915+0000 7f00a1ccd640 1 -- 172.21.15.121:0/1137695313 shutdown_connections 2024-09-17T18:19:02.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.915+0000 7f00a1ccd640 1 -- 172.21.15.121:0/1137695313 wait complete. 2024-09-17T18:19:02.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.916+0000 7ff1137a0640 10 monclient: build_initial_monmap 2024-09-17T18:19:02.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.916+0000 7ff1137a0640 10 monclient: monmap: 2024-09-17T18:19:02.938 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:02.938 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:19:02.938 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:19:02.938 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:19:02.938 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:02.938 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:02.938 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:19:02.938 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:19:02.938 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:19:02.939 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:02.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.916+0000 7f00a1ccd640 10 monclient: build_initial_monmap 2024-09-17T18:19:02.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.916+0000 7f00a1ccd640 10 monclient: monmap: 2024-09-17T18:19:02.939 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:02.939 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:19:02.939 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:19:02.939 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:19:02.939 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:02.939 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:02.939 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:19:02.940 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:19:02.940 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:19:02.940 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:02.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.916+0000 7ff1137a0640 1 librados: starting msgr at 2024-09-17T18:19:02.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.916+0000 7ff1137a0640 1 librados: starting objecter 2024-09-17T18:19:02.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.916+0000 7ff1137a0640 1 Processor -- start 2024-09-17T18:19:02.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.916+0000 7f00a1ccd640 1 librados: starting msgr at 2024-09-17T18:19:02.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.916+0000 7ff1137a0640 1 -- start start 2024-09-17T18:19:02.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.916+0000 7ff1137a0640 1 librados: setting wanted keys 2024-09-17T18:19:02.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.916+0000 7ff1137a0640 1 librados: calling monclient init 2024-09-17T18:19:02.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.916+0000 7ff1137a0640 10 monclient: init 2024-09-17T18:19:02.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.916+0000 7f00a1ccd640 1 librados: starting objecter 2024-09-17T18:19:02.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.916+0000 7f00a1ccd640 1 Processor -- start 2024-09-17T18:19:02.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.916+0000 7f00a1ccd640 1 -- start start 2024-09-17T18:19:02.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.916+0000 7f00a1ccd640 1 librados: setting wanted keys 2024-09-17T18:19:02.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.916+0000 7f00a1ccd640 1 librados: calling monclient init 2024-09-17T18:19:02.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff1137a0640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:02.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff1137a0640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T18:19:02.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff1137a0640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff10c0a5ca0 0x7ff10c139c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:02.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff1137a0640 10 monclient(hunting): picked mon.noname-b con 0x7ff10c0a5ca0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:02.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff1137a0640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff10c0a65c0 0x7ff10c1238f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:02.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.916+0000 7f00a1ccd640 10 monclient: init 2024-09-17T18:19:02.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff1137a0640 10 monclient(hunting): picked mon.noname-a con 0x7ff10c0a65c0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:02.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff1137a0640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff10c13cfc0 0x7ff10c123e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:02.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff1137a0640 10 monclient(hunting): picked mon.noname-c con 0x7ff10c13cfc0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:02.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff1137a0640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:02.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff1137a0640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff10c0a5670 con 0x7ff10c0a65c0 2024-09-17T18:19:02.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff1137a0640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:02.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff11153c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff10c0a5ca0 0x7ff10c139c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:02.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff11153c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff10c0a5ca0 0x7ff10c139c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.121:42008/0 (socket says 172.21.15.121:42008) 2024-09-17T18:19:02.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff11153c640 1 -- 172.21.15.121:0/1593538498 learned_addr learned my addr 172.21.15.121:0/1593538498 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:02.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff110d3b640 1 --2- 172.21.15.121:0/1593538498 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff10c0a65c0 0x7ff10c1238f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:02.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff1137a0640 1 -- 172.21.15.121:0/1593538498 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff10c0a6eb0 con 0x7ff10c0a5ca0 2024-09-17T18:19:02.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff1137a0640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:02.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff1137a0640 1 -- 172.21.15.121:0/1593538498 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff10c0a69d0 con 0x7ff10c13cfc0 2024-09-17T18:19:02.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff1137a0640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:02.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff1137a0640 10 monclient(hunting): authenticate will time out at 746.633458s 2024-09-17T18:19:02.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff11153c640 10 monclient(hunting): get_auth_request con 0x7ff10c0a5ca0 auth_method 0 2024-09-17T18:19:02.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff11153c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:02.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff11153c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:02.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff11153c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:02.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff110d3b640 10 monclient(hunting): get_auth_request con 0x7ff10c0a65c0 auth_method 0 2024-09-17T18:19:02.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff110d3b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:02.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff110d3b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:02.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff110d3b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:02.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7f00a1ccd640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:02.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7f00a1ccd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T18:19:02.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7f00a1ccd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f009c0a5b90 0x7f009c1cdbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:02.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff111d3d640 1 --2- 172.21.15.121:0/1593538498 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff10c13cfc0 0x7ff10c123e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:02.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff11153c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:02.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff11153c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:02.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff11153c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:02.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff111d3d640 10 monclient(hunting): get_auth_request con 0x7ff10c13cfc0 auth_method 0 2024-09-17T18:19:02.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff111d3d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:02.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff111d3d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:02.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff111d3d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:02.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff110d3b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:02.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff110d3b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:02.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff110d3b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:02.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7f00a1ccd640 10 monclient(hunting): picked mon.noname-b con 0x7f009c0a5b90 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:02.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7f00a1ccd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f009c0a64b0 0x7f009c1ce120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:02.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7f00a1ccd640 10 monclient(hunting): picked mon.noname-c con 0x7f009c0a64b0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:02.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7f00a1ccd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f009c0a7080 0x7f009c125240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:02.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7f00a1ccd640 10 monclient(hunting): picked mon.noname-a con 0x7f009c0a7080 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:02.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7f00a1ccd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:02.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7f00a1ccd640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f009c09cd40 con 0x7f009c0a7080 2024-09-17T18:19:02.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f00a1ccd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:02.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f00a1ccd640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f009c0702d0 con 0x7f009c0a5b90 2024-09-17T18:19:02.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f00a1ccd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:02.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f00a1ccd640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f009c070450 con 0x7f009c0a64b0 2024-09-17T18:19:02.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f00a1ccd640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:02.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff11153c640 10 monclient(hunting): handle_auth_done global_id 4231 payload 824 2024-09-17T18:19:02.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff11153c640 1 -- 172.21.15.121:0/1593538498 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff10c13cfc0 msgr2=0x7ff10c123e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:02.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff11153c640 1 --2- 172.21.15.121:0/1593538498 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff10c13cfc0 0x7ff10c123e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:02.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff11153c640 1 -- 172.21.15.121:0/1593538498 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff10c0a65c0 msgr2=0x7ff10c1238f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:02.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff11153c640 1 --2- 172.21.15.121:0/1593538498 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff10c0a65c0 0x7ff10c1238f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:02.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff11153c640 10 monclient: _finish_hunting 0 2024-09-17T18:19:02.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff11153c640 1 monclient: found mon.noname-b 2024-09-17T18:19:02.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff11153c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:02.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff11153c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.173:3300/0 2024-09-17T18:19:02.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff11153c640 1 -- 172.21.15.121:0/1593538498 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff10c13a600 con 0x7ff10c0a5ca0 2024-09-17T18:19:02.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.917+0000 7ff111d3d640 1 --2- 172.21.15.121:0/1593538498 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff10c13cfc0 0x7ff10c123e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T18:19:02.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f00a1ccd640 10 monclient(hunting): authenticate will time out at 746.634117s 2024-09-17T18:19:02.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f009bfff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f009c0a7080 0x7f009c125240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:02.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f009affd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f009c0a64b0 0x7f009c1ce120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:02.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f009b7fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f009c0a5b90 0x7f009c1cdbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:02.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f009affd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f009c0a64b0 0x7f009c1ce120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:59306/0 (socket says 172.21.15.121:59306) 2024-09-17T18:19:02.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f009bfff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f009c0a7080 0x7f009c125240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40862/0 (socket says 172.21.15.121:40862) 2024-09-17T18:19:02.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f009affd640 1 -- 172.21.15.121:0/4210506051 learned_addr learned my addr 172.21.15.121:0/4210506051 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:02.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f009affd640 10 monclient(hunting): get_auth_request con 0x7f009c0a64b0 auth_method 0 2024-09-17T18:19:02.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f009affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:02.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f009affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:02.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f009affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:02.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f009bfff640 10 monclient(hunting): get_auth_request con 0x7f009c0a7080 auth_method 0 2024-09-17T18:19:02.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f009bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:02.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f009bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:02.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f009bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:02.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f009b7fe640 10 monclient(hunting): get_auth_request con 0x7f009c0a5b90 auth_method 0 2024-09-17T18:19:02.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f009b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:02.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f009b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:02.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f009b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:02.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f009affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:02.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f009affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:02.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f009affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:02.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f009bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:02.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f009bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:02.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f009bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:02.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f009b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:02.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f009b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:02.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f009b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:02.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f009affd640 10 monclient(hunting): handle_auth_done global_id 4214 payload 824 2024-09-17T18:19:02.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f009affd640 1 -- 172.21.15.121:0/4210506051 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f009c0a5b90 msgr2=0x7f009c1cdbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:02.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f009affd640 1 --2- 172.21.15.121:0/4210506051 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f009c0a5b90 0x7f009c1cdbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:02.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f009affd640 1 -- 172.21.15.121:0/4210506051 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f009c0a7080 msgr2=0x7f009c125240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:02.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f009affd640 1 --2- 172.21.15.121:0/4210506051 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f009c0a7080 0x7f009c125240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:02.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f009affd640 10 monclient: _finish_hunting 0 2024-09-17T18:19:02.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f009affd640 1 monclient: found mon.noname-c 2024-09-17T18:19:02.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f009affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:02.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f009affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:19:02.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.918+0000 7f009affd640 1 -- 172.21.15.121:0/4210506051 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f009c138620 con 0x7f009c0a64b0 2024-09-17T18:19:02.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.920+0000 7ff11153c640 1 -- 172.21.15.121:0/1593538498 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff10c0a5ca0 msgr2=0x7ff10c139c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:02.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.920+0000 7ff11153c640 1 --2- 172.21.15.121:0/1593538498 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff10c0a5ca0 0x7ff10c139c00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff108000c00 tx=0x7ff1080046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:02.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.920+0000 7ff101ffb640 10 client.?.objecter ms_handle_connect 0x7ff10c0a5ca0 2024-09-17T18:19:02.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.920+0000 7ff101ffb640 10 client.?.objecter resend_mon_ops 2024-09-17T18:19:02.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.920+0000 7ff101ffb640 1 -- 172.21.15.121:0/1593538498 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff108037040 con 0x7ff10c0a5ca0 2024-09-17T18:19:02.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.920+0000 7ff101ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:02.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.920+0000 7ff101ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:19:02.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.920+0000 7ff101ffb640 10 monclient: dump: 2024-09-17T18:19:02.956 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:02.956 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:02.956 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:02.956 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:02.956 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:02.957 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:02.957 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:02.957 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:02.957 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:02.957 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:02.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.920+0000 7ff101ffb640 10 monclient: _finish_auth 0 2024-09-17T18:19:02.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.920+0000 7ff101ffb640 10 monclient: _check_auth_tickets 2024-09-17T18:19:02.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.920+0000 7ff101ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:02.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.920+0000 7ff101ffb640 1 -- 172.21.15.121:0/1593538498 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff108042070 con 0x7ff10c0a5ca0 2024-09-17T18:19:02.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.920+0000 7ff101ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:02.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.920+0000 7ff101ffb640 1 -- 172.21.15.121:0/1593538498 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff108030b70 con 0x7ff10c0a5ca0 2024-09-17T18:19:02.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.920+0000 7ff101ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:02.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.920+0000 7ff101ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:19:02.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.920+0000 7ff101ffb640 10 monclient: dump: 2024-09-17T18:19:02.958 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:02.958 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:02.958 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:02.958 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:02.958 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:02.958 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:02.958 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:02.958 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:02.958 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:02.958 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:02.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.920+0000 7ff1137a0640 5 monclient: authenticate success, global_id 4231 2024-09-17T18:19:02.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.920+0000 7ff1137a0640 10 monclient: _renew_subs 2024-09-17T18:19:02.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.920+0000 7ff1137a0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:19:02.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.920+0000 7ff1137a0640 1 -- 172.21.15.121:0/1593538498 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff10c1276c0 con 0x7ff10c0a5ca0 2024-09-17T18:19:02.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.920+0000 7ff1037fe640 4 set_mon_vals no callback set 2024-09-17T18:19:02.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.920+0000 7ff1137a0640 10 client.4231.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:02.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.920+0000 7ff1137a0640 10 monclient: _renew_subs 2024-09-17T18:19:02.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.920+0000 7ff1137a0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:19:02.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.920+0000 7f009affd640 1 -- 172.21.15.121:0/4210506051 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f009c0a64b0 msgr2=0x7f009c1ce120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:02.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.920+0000 7f009affd640 1 --2- 172.21.15.121:0/4210506051 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f009c0a64b0 0x7f009c1ce120 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f009000c430 tx=0x7f009000c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:02.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.921+0000 7f0083fff640 10 client.?.objecter ms_handle_connect 0x7f009c0a64b0 2024-09-17T18:19:02.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.921+0000 7f0083fff640 10 client.?.objecter resend_mon_ops 2024-09-17T18:19:02.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.921+0000 7f0083fff640 1 -- 172.21.15.121:0/4210506051 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00900119a0 con 0x7f009c0a64b0 2024-09-17T18:19:02.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.921+0000 7f0083fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:02.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.921+0000 7f0083fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:19:02.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.921+0000 7f0083fff640 10 monclient: dump: 2024-09-17T18:19:02.961 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:02.961 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:02.961 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:02.961 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:02.962 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:02.962 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:02.962 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:02.962 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:02.962 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:02.962 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:02.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.921+0000 7f0083fff640 10 monclient: _finish_auth 0 2024-09-17T18:19:02.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.921+0000 7f0083fff640 10 monclient: _check_auth_tickets 2024-09-17T18:19:02.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.921+0000 7f0083fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:02.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.921+0000 7f0083fff640 1 -- 172.21.15.121:0/4210506051 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0090017070 con 0x7f009c0a64b0 2024-09-17T18:19:02.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.921+0000 7f0083fff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:02.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.921+0000 7f0083fff640 1 -- 172.21.15.121:0/4210506051 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0090011d80 con 0x7f009c0a64b0 2024-09-17T18:19:02.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.921+0000 7f0083fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:02.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.921+0000 7f0083fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:19:02.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.921+0000 7f0083fff640 10 monclient: dump: 2024-09-17T18:19:02.963 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:02.963 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:02.963 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:02.963 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:02.963 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:02.963 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:02.963 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:02.963 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:02.963 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:02.963 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:02.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.921+0000 7f0099ffb640 4 set_mon_vals no callback set 2024-09-17T18:19:02.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.920+0000 7ff1137a0640 1 -- 172.21.15.121:0/1593538498 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff10c125360 con 0x7ff10c0a5ca0 2024-09-17T18:19:02.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.921+0000 7f00a1ccd640 5 monclient: authenticate success, global_id 4214 2024-09-17T18:19:02.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.921+0000 7f00a1ccd640 10 monclient: _renew_subs 2024-09-17T18:19:02.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.921+0000 7f00a1ccd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:02.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.921+0000 7f00a1ccd640 1 -- 172.21.15.121:0/4210506051 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f009c09e6e0 con 0x7f009c0a64b0 2024-09-17T18:19:02.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.920+0000 7ff1137a0640 1 librados: init done 2024-09-17T18:19:02.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.921+0000 7f00a1ccd640 10 client.4214.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:02.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.921+0000 7f00a1ccd640 10 monclient: _renew_subs 2024-09-17T18:19:02.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.921+0000 7f00a1ccd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:02.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.921+0000 7f00a1ccd640 1 -- 172.21.15.121:0/4210506051 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f009c129060 con 0x7f009c0a64b0 2024-09-17T18:19:02.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.921+0000 7f00a1ccd640 1 librados: init done 2024-09-17T18:19:02.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.921+0000 7ff101ffb640 1 -- 172.21.15.121:0/1593538498 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7ff108030d10 con 0x7ff10c0a5ca0 2024-09-17T18:19:02.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.922+0000 7ff101ffb640 1 --2- 172.21.15.121:0/1593538498 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff0f00401c0 0x7ff0f0042680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:02.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.922+0000 7ff101ffb640 1 -- 172.21.15.121:0/1593538498 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 3562+0+0 (secure 0 0 0) 0x7ff108073940 con 0x7ff10c0a5ca0 2024-09-17T18:19:02.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.922+0000 7ff1137a0640 10 client.4231.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:02.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.922+0000 7ff1137a0640 15 client.4231.objecter _session_command_op_assign -1 1 2024-09-17T18:19:02.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.922+0000 7ff1137a0640 20 client.4231.objecter _get_session osd=-1 returning homeless 2024-09-17T18:19:02.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.922+0000 7ff1137a0640 10 client.4231.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:02.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.922+0000 7ff1137a0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:19:02.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.922+0000 7ff1137a0640 1 -- 172.21.15.121:0/1593538498 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff10c0a8e90 con 0x7ff10c0a5ca0 2024-09-17T18:19:02.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.922+0000 7ff110d3b640 1 --2- 172.21.15.121:0/1593538498 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff0f00401c0 0x7ff0f0042680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:02.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.922+0000 7ff110d3b640 10 monclient: get_auth_request con 0x7ff0f00401c0 auth_method 0 2024-09-17T18:19:02.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.922+0000 7ff101ffb640 10 client.4231.objecter ms_dispatch 0x7ff10c000b90 osd_map(16..16 src has 1..16) 2024-09-17T18:19:02.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.922+0000 7ff101ffb640 3 client.4231.objecter handle_osd_map got epochs [16,16] > 0 2024-09-17T18:19:02.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.922+0000 7ff101ffb640 3 client.4231.objecter handle_osd_map decoding full epoch 16 2024-09-17T18:19:02.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.923+0000 7f00a1ccd640 10 client.4214.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:02.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.922+0000 7ff101ffb640 10 client.4231.objecter checking command 1 2024-09-17T18:19:02.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.923+0000 7ff101ffb640 1 --2- 172.21.15.121:0/1593538498 >> v2:172.21.15.173:6808/1801998802 conn(0x7ff0f0045cd0 0x7ff0f0048110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:02.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.923+0000 7ff101ffb640 20 client.4231.objecter _get_session s=0x7ff0f0045620 osd=5 3 2024-09-17T18:19:02.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.923+0000 7ff101ffb640 20 client.4231.objecter put_session s=0x7ff0f0045620 osd=5 3 2024-09-17T18:19:02.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.923+0000 7ff101ffb640 15 client.4231.objecter _session_command_op_remove -1 1 2024-09-17T18:19:02.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.923+0000 7ff101ffb640 20 client.4231.objecter _get_session s=0x7ff0f0045620 osd=5 3 2024-09-17T18:19:02.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.923+0000 7ff101ffb640 20 client.4231.objecter get_session s=0x7ff0f0045620 osd=5 3 2024-09-17T18:19:02.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.923+0000 7ff101ffb640 15 client.4231.objecter _session_command_op_assign 5 1 2024-09-17T18:19:02.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.923+0000 7f00a1ccd640 15 client.4214.objecter _session_command_op_assign -1 1 2024-09-17T18:19:02.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.923+0000 7f00a1ccd640 20 client.4214.objecter _get_session osd=-1 returning homeless 2024-09-17T18:19:02.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.923+0000 7f00a1ccd640 10 client.4214.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:02.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.923+0000 7ff101ffb640 20 client.4231.objecter put_session s=0x7ff0f0045620 osd=5 4 2024-09-17T18:19:02.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.923+0000 7ff101ffb640 10 client.4231.objecter _send_command 1 2024-09-17T18:19:02.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.923+0000 7ff101ffb640 1 -- 172.21.15.121:0/1593538498 --> v2:172.21.15.173:6808/1801998802 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff10803ee50 con 0x7ff0f0045cd0 2024-09-17T18:19:02.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.923+0000 7ff101ffb640 20 client.4231.objecter dump_active .. 0 homeless 2024-09-17T18:19:02.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.923+0000 7ff101ffb640 1 -- 172.21.15.121:0/1593538498 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7ff10803b430 con 0x7ff10c0a5ca0 2024-09-17T18:19:02.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.923+0000 7ff101ffb640 10 monclient: handle_get_version_reply finishing 1 version 16 2024-09-17T18:19:02.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.923+0000 7f00a1ccd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:02.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.923+0000 7f00a1ccd640 1 -- 172.21.15.121:0/4210506051 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f009c0a8d80 con 0x7f009c0a64b0 2024-09-17T18:19:02.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.923+0000 7ff111d3d640 1 --2- 172.21.15.121:0/1593538498 >> v2:172.21.15.173:6808/1801998802 conn(0x7ff0f0045cd0 0x7ff0f0048110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:02.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.923+0000 7ff111d3d640 10 monclient: get_auth_request con 0x7ff0f0045cd0 auth_method 0 2024-09-17T18:19:02.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.923+0000 7f0083fff640 1 -- 172.21.15.121:0/4210506051 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f009001c460 con 0x7f009c0a64b0 2024-09-17T18:19:02.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.923+0000 7f0083fff640 1 --2- 172.21.15.121:0/4210506051 >> v2:172.21.15.173:6812/2423786037 conn(0x7f00880401f0 0x7f00880426b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:02.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.923+0000 7f0083fff640 1 -- 172.21.15.121:0/4210506051 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 3562+0+0 (secure 0 0 0) 0x7f0090052e60 con 0x7f009c0a64b0 2024-09-17T18:19:02.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.923+0000 7f0083fff640 10 client.4214.objecter ms_dispatch 0x7f009c000b90 osd_map(16..16 src has 1..16) 2024-09-17T18:19:02.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.923+0000 7f0083fff640 3 client.4214.objecter handle_osd_map got epochs [16,16] > 0 2024-09-17T18:19:02.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.923+0000 7f0083fff640 3 client.4214.objecter handle_osd_map decoding full epoch 16 2024-09-17T18:19:02.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.924+0000 7f0083fff640 10 client.4214.objecter checking command 1 2024-09-17T18:19:02.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.924+0000 7f0083fff640 1 --2- 172.21.15.121:0/4210506051 >> v2:172.21.15.173:6800/3493964597 conn(0x7f0088045d80 0x7f00880481c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:02.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.924+0000 7f0083fff640 20 client.4214.objecter _get_session s=0x7f00880456d0 osd=3 3 2024-09-17T18:19:02.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.924+0000 7f0083fff640 20 client.4214.objecter put_session s=0x7f00880456d0 osd=3 3 2024-09-17T18:19:02.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.924+0000 7f0083fff640 15 client.4214.objecter _session_command_op_remove -1 1 2024-09-17T18:19:02.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.924+0000 7f0083fff640 20 client.4214.objecter _get_session s=0x7f00880456d0 osd=3 3 2024-09-17T18:19:02.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.924+0000 7f0083fff640 20 client.4214.objecter get_session s=0x7f00880456d0 osd=3 3 2024-09-17T18:19:02.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.924+0000 7f0083fff640 15 client.4214.objecter _session_command_op_assign 3 1 2024-09-17T18:19:02.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.924+0000 7f0083fff640 20 client.4214.objecter put_session s=0x7f00880456d0 osd=3 4 2024-09-17T18:19:02.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.924+0000 7f0083fff640 10 client.4214.objecter _send_command 1 2024-09-17T18:19:02.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.924+0000 7f0083fff640 1 -- 172.21.15.121:0/4210506051 --> v2:172.21.15.173:6800/3493964597 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0090013ec0 con 0x7f0088045d80 2024-09-17T18:19:02.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.924+0000 7f0083fff640 20 client.4214.objecter dump_active .. 0 homeless 2024-09-17T18:19:02.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.924+0000 7f0083fff640 1 -- 172.21.15.121:0/4210506051 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f0090055430 con 0x7f009c0a64b0 2024-09-17T18:19:02.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.924+0000 7f0083fff640 10 monclient: handle_get_version_reply finishing 1 version 16 2024-09-17T18:19:02.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.924+0000 7f009bfff640 1 --2- 172.21.15.121:0/4210506051 >> v2:172.21.15.173:6800/3493964597 conn(0x7f0088045d80 0x7f00880481c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:02.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.924+0000 7f009bfff640 10 monclient: get_auth_request con 0x7f0088045d80 auth_method 0 2024-09-17T18:19:02.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.924+0000 7f009bfff640 1 -- 172.21.15.121:0/4210506051 >> v2:172.21.15.173:6800/3493964597 conn(0x7f0088045d80 msgr2=0x7f00880481c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T18:19:02.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.924+0000 7f009bfff640 1 -- 172.21.15.121:0/4210506051 >> v2:172.21.15.173:6800/3493964597 conn(0x7f0088045d80 msgr2=0x7f00880481c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:19:02.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.924+0000 7f009bfff640 1 --2- 172.21.15.121:0/4210506051 >> v2:172.21.15.173:6800/3493964597 conn(0x7f0088045d80 0x7f00880481c0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:19:02.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.924+0000 7f009bfff640 1 --2- 172.21.15.121:0/4210506051 >> v2:172.21.15.173:6800/3493964597 conn(0x7f0088045d80 0x7f00880481c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:19:02.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.925+0000 7ff101ffb640 10 client.4231.objecter ms_handle_connect 0x7ff0f00401c0 2024-09-17T18:19:02.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.925+0000 7ff110d3b640 1 --2- 172.21.15.121:0/1593538498 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff0f00401c0 0x7ff0f0042680 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff10c0a1700 tx=0x7ff0fc009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:02.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.925+0000 7f009b7fe640 1 --2- 172.21.15.121:0/4210506051 >> v2:172.21.15.173:6812/2423786037 conn(0x7f00880401f0 0x7f00880426b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:02.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.925+0000 7f009b7fe640 10 monclient: get_auth_request con 0x7f00880401f0 auth_method 0 2024-09-17T18:19:02.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.927+0000 7f0083fff640 10 client.4214.objecter ms_handle_connect 0x7f00880401f0 2024-09-17T18:19:02.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.927+0000 7f009b7fe640 1 --2- 172.21.15.121:0/4210506051 >> v2:172.21.15.173:6812/2423786037 conn(0x7f00880401f0 0x7f00880426b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f009c06eea0 tx=0x7f008c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:02.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.928+0000 7ff101ffb640 10 client.4231.objecter ms_handle_connect 0x7ff0f0045cd0 2024-09-17T18:19:02.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.928+0000 7ff111d3d640 1 --2- 172.21.15.121:0/1593538498 >> v2:172.21.15.173:6808/1801998802 conn(0x7ff0f0045cd0 0x7ff0f0048110 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:02.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.928+0000 7ff101ffb640 1 -- 172.21.15.121:0/1593538498 <== osd.5 v2:172.21.15.173:6808/1801998802 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff10803ee50 con 0x7ff0f0045cd0 2024-09-17T18:19:02.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.928+0000 7ff101ffb640 10 client.4231.objecter ms_dispatch 0x7ff10c000b90 command_reply(tid 1: 0 ) 2024-09-17T18:19:02.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.928+0000 7ff101ffb640 10 client.4231.objecter _finish_command 1 = system:0 2024-09-17T18:19:02.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.929+0000 7ff101ffb640 20 client.4231.objecter put_session s=0x7ff0f0045620 osd=5 4 2024-09-17T18:19:02.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.929+0000 7ff101ffb640 15 client.4231.objecter _session_command_op_remove 5 1 2024-09-17T18:19:02.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.945+0000 7ff1137a0640 10 client.4231.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-17T18:19:02.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.945+0000 7ff1137a0640 15 client.4231.objecter _session_command_op_assign -1 2 2024-09-17T18:19:02.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.945+0000 7ff1137a0640 20 client.4231.objecter _get_session s=0x7ff0f0045620 osd=5 3 2024-09-17T18:19:02.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.945+0000 7ff1137a0640 20 client.4231.objecter put_session s=0x7ff0f0045620 osd=5 3 2024-09-17T18:19:02.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.945+0000 7ff1137a0640 20 client.4231.objecter _get_session s=0x7ff0f0045620 osd=5 3 2024-09-17T18:19:02.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.945+0000 7ff1137a0640 15 client.4231.objecter _session_command_op_remove -1 2 2024-09-17T18:19:02.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.945+0000 7ff1137a0640 20 client.4231.objecter get_session s=0x7ff0f0045620 osd=5 3 2024-09-17T18:19:02.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.945+0000 7ff1137a0640 15 client.4231.objecter _session_command_op_assign 5 2 2024-09-17T18:19:02.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.945+0000 7ff1137a0640 20 client.4231.objecter put_session s=0x7ff0f0045620 osd=5 4 2024-09-17T18:19:02.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.945+0000 7ff1137a0640 10 client.4231.objecter _send_command 2 2024-09-17T18:19:02.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.945+0000 7ff1137a0640 1 -- 172.21.15.121:0/1593538498 --> v2:172.21.15.173:6808/1801998802 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff10c0a8e90 con 0x7ff0f0045cd0 2024-09-17T18:19:02.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.946+0000 7ff101ffb640 1 -- 172.21.15.121:0/1593538498 <== osd.5 v2:172.21.15.173:6808/1801998802 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff10c0a8e90 con 0x7ff0f0045cd0 2024-09-17T18:19:02.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.946+0000 7ff101ffb640 10 client.4231.objecter ms_dispatch 0x7ff10c000b90 command_reply(tid 2: 0 ) 2024-09-17T18:19:02.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.946+0000 7ff101ffb640 10 client.4231.objecter _finish_command 2 = system:0 2024-09-17T18:19:02.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.946+0000 7ff101ffb640 20 client.4231.objecter put_session s=0x7ff0f0045620 osd=5 4 2024-09-17T18:19:02.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.946+0000 7ff101ffb640 15 client.4231.objecter _session_command_op_remove 5 2 2024-09-17T18:19:02.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.946+0000 7ff1137a0640 10 librados: watch_flush enter 2024-09-17T18:19:02.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.946+0000 7ff1137a0640 10 librados: watch_flush exit 2024-09-17T18:19:02.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.946+0000 7ff1137a0640 10 client.4231.objecter close_session for osd.5 2024-09-17T18:19:02.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.946+0000 7ff1137a0640 1 -- 172.21.15.121:0/1593538498 >> v2:172.21.15.173:6808/1801998802 conn(0x7ff0f0045cd0 msgr2=0x7ff0f0048110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:02.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.946+0000 7ff1137a0640 1 --2- 172.21.15.121:0/1593538498 >> v2:172.21.15.173:6808/1801998802 conn(0x7ff0f0045cd0 0x7ff0f0048110 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:02.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.946+0000 7ff1137a0640 20 client.4231.objecter put_session s=0x7ff0f0045620 osd=5 1 2024-09-17T18:19:02.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.946+0000 7ff1137a0640 20 client.4231.objecter shutdown clearing up homeless session... 2024-09-17T18:19:02.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.946+0000 7ff1137a0640 10 client.4231.objecter successfully canceled tick 2024-09-17T18:19:02.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.947+0000 7ff1137a0640 1 -- 172.21.15.121:0/1593538498 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff0f00401c0 msgr2=0x7ff0f0042680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:02.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.947+0000 7ff1137a0640 1 --2- 172.21.15.121:0/1593538498 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff0f00401c0 0x7ff0f0042680 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff10c0a1700 tx=0x7ff0fc009240 comp rx=0 tx=0).stop 2024-09-17T18:19:02.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.947+0000 7ff1137a0640 10 monclient: shutdown 2024-09-17T18:19:02.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.947+0000 7ff1137a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:02.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.947+0000 7ff1137a0640 1 -- 172.21.15.121:0/1593538498 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff10c0a5ca0 msgr2=0x7ff10c139c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:02.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.947+0000 7ff1137a0640 1 --2- 172.21.15.121:0/1593538498 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff10c0a5ca0 0x7ff10c139c00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff108000c00 tx=0x7ff1080046c0 comp rx=0 tx=0).stop 2024-09-17T18:19:02.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.947+0000 7ff111d3d640 1 -- 172.21.15.121:0/1593538498 reap_dead start 2024-09-17T18:19:02.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.947+0000 7ff1137a0640 1 -- 172.21.15.121:0/1593538498 shutdown_connections 2024-09-17T18:19:02.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.947+0000 7ff1137a0640 1 -- 172.21.15.121:0/1593538498 >> 172.21.15.121:0/1593538498 conn(0x7ff10c06dd10 msgr2=0x7ff10c13ae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:02.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.947+0000 7ff1137a0640 1 -- 172.21.15.121:0/1593538498 shutdown_connections 2024-09-17T18:19:02.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.947+0000 7ff1137a0640 1 -- 172.21.15.121:0/1593538498 wait complete. 2024-09-17T18:19:02.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.947+0000 7ff1137a0640 1 librados: shutdown 2024-09-17T18:19:02.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.947+0000 7f2cf78b0640 10 monclient: get_monmap_and_config 2024-09-17T18:19:02.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.947+0000 7f51b39d0640 10 monclient: get_monmap_and_config 2024-09-17T18:19:02.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.947+0000 7f51b39d0640 10 monclient: build_initial_monmap 2024-09-17T18:19:02.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.947+0000 7f51b39d0640 10 monclient: monmap: 2024-09-17T18:19:02.983 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:02.983 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:02.983 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:19:02.948555+0000 2024-09-17T18:19:02.983 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:19:02.948555+0000 2024-09-17T18:19:02.983 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:02.983 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:02.983 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:19:02.983 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:19:02.983 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:19:02.983 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:19:02.983 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:19:02.983 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:02.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.947+0000 7f2cf78b0640 10 monclient: build_initial_monmap 2024-09-17T18:19:02.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.947+0000 7f2cf78b0640 10 monclient: monmap: 2024-09-17T18:19:02.984 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:02.984 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:02.984 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:19:02.948541+0000 2024-09-17T18:19:02.984 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:19:02.948541+0000 2024-09-17T18:19:02.985 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:02.985 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:02.985 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:19:02.985 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:19:02.985 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:19:02.985 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:19:02.985 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:19:02.985 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:02.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.948+0000 7f2cf78b0640 1 Processor -- start 2024-09-17T18:19:02.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.948+0000 7f51b39d0640 1 Processor -- start 2024-09-17T18:19:02.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.948+0000 7f2cf78b0640 1 -- start start 2024-09-17T18:19:02.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.948+0000 7f2cf78b0640 10 monclient: init 2024-09-17T18:19:02.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.948+0000 7f51b39d0640 1 -- start start 2024-09-17T18:19:02.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.948+0000 7f51b39d0640 10 monclient: init 2024-09-17T18:19:02.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f2cf78b0640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:02.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f2cf78b0640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-17T18:19:02.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f2cf78b0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2cf00a5ca0 addr v1:172.21.15.121:6789/0 2024-09-17T18:19:02.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f2cf78b0640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2cf00a6790 0x7f2cf013da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:02.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f2cf78b0640 10 monclient(hunting): picked mon.noname-a con 0x7f2cf00a6790 addr v2:172.21.15.121:3300/0 2024-09-17T18:19:02.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f2cf78b0640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2cf013dfd0 0x7f2cf01403c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:02.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f2cf78b0640 10 monclient(hunting): picked mon.noname-b con 0x7f2cf013dfd0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:02.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f2cf78b0640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2cf00a7b50 con 0x7f2cf00a5ca0 2024-09-17T18:19:02.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f2cf78b0640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:02.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f2cf78b0640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f2cf00704c0 con 0x7f2cf00a6790 2024-09-17T18:19:02.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f2cf78b0640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:02.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f2cf78b0640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2cf00a5490 con 0x7f2cf013dfd0 2024-09-17T18:19:02.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f2cf78b0640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:02.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f2cf78b0640 10 monclient(hunting): authenticate will time out at 746.665511s 2024-09-17T18:19:02.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7fc2f4ad4640 10 monclient: get_monmap_and_config 2024-09-17T18:19:02.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7fc2f4ad4640 10 monclient: build_initial_monmap 2024-09-17T18:19:02.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f2cf564c640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f2cf00a5ca0 0x7f2cf00a6080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:49562/0 (socket says 172.21.15.121:49562) 2024-09-17T18:19:02.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f2cf564c640 1 -- 172.21.15.121:0/3897580968 learned_addr learned my addr 172.21.15.121:0/3897580968 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:19:02.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f2cf5e4d640 1 --2- 172.21.15.121:0/3897580968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2cf013dfd0 0x7f2cf01403c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:02.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f2cf4e4b640 1 --2- 172.21.15.121:0/3897580968 >> v2:172.21.15.121:3300/0 conn(0x7f2cf00a6790 0x7f2cf013da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:02.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7fc2f4ad4640 10 monclient: monmap: 2024-09-17T18:19:02.989 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:02.989 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:02.989 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:19:02.950306+0000 2024-09-17T18:19:02.990 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:19:02.950306+0000 2024-09-17T18:19:02.990 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:02.990 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:02.990 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:19:02.990 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:19:02.990 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:19:02.990 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:19:02.990 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:19:02.990 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:02.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f2cf4e4b640 10 monclient(hunting): get_auth_request con 0x7f2cf00a6790 auth_method 0 2024-09-17T18:19:02.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f2cf4e4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:02.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f2cf4e4b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:02.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f2cf4e4b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:02.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f2cf5e4d640 10 monclient(hunting): get_auth_request con 0x7f2cf013dfd0 auth_method 0 2024-09-17T18:19:02.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f2cf5e4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:02.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f2cf5e4d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:02.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f2cf5e4d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:02.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f51b39d0640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:02.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f51b39d0640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-17T18:19:02.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f51b39d0640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f51ac0700f0 0x7f51ac134e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:02.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f51b39d0640 10 monclient(hunting): picked mon.noname-c con 0x7f51ac0700f0 addr v2:172.21.15.173:3300/0 2024-09-17T18:19:02.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f51b39d0640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f51ac09e810 0x7f51ac138360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:02.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f51b39d0640 10 monclient(hunting): picked mon.noname-b con 0x7f51ac09e810 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:02.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f51b39d0640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f51ac09ec60 0x7f51ac1388a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:02.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f51b39d0640 10 monclient(hunting): picked mon.noname-a con 0x7f51ac09ec60 addr v2:172.21.15.121:3300/0 2024-09-17T18:19:02.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f51b39d0640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:02.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f51b39d0640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f51ac0704d0 con 0x7f51ac09ec60 2024-09-17T18:19:02.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f51b39d0640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:02.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f51b39d0640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f51ac09ca60 con 0x7f51ac0700f0 2024-09-17T18:19:02.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f51b39d0640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:02.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f51b39d0640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f51ac09cd20 con 0x7f51ac09e810 2024-09-17T18:19:02.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f51b39d0640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:02.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f51b39d0640 10 monclient(hunting): authenticate will time out at 746.665515s 2024-09-17T18:19:02.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f51b1f6d640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f51ac09ec60 0x7f51ac1388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:02.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f51b1f6d640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f51ac09ec60 0x7f51ac1388a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40876/0 (socket says 172.21.15.121:40876) 2024-09-17T18:19:02.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f51b1f6d640 1 -- 172.21.15.121:0/465882080 learned_addr learned my addr 172.21.15.121:0/465882080 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:02.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f51b1f6d640 10 monclient(hunting): get_auth_request con 0x7f51ac09ec60 auth_method 0 2024-09-17T18:19:02.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f51b1f6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:02.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f51b1f6d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:02.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f51b1f6d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:02.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f51b1f6d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:02.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f51b1f6d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:02.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f51b1f6d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:02.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f51b176c640 1 --2- 172.21.15.121:0/465882080 >> v2:172.21.15.173:3300/0 conn(0x7f51ac0700f0 0x7f51ac134e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:02.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f51b0f6b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f51ac09e810 0x7f51ac138360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:02.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f51b0f6b640 10 monclient(hunting): get_auth_request con 0x7f51ac09e810 auth_method 0 2024-09-17T18:19:02.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f51b0f6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:02.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f51b0f6b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:02.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f51b0f6b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:02.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.950+0000 7f51b176c640 10 monclient(hunting): get_auth_request con 0x7f51ac0700f0 auth_method 0 2024-09-17T18:19:02.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.950+0000 7f51b176c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:02.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.950+0000 7f51b176c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:02.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.950+0000 7f51b176c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:02.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f2cf4e4b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:02.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f2cf4e4b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:02.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f2cf4e4b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:02.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f2cf5e4d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:02.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f2cf5e4d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:02.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.949+0000 7f2cf5e4d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:02.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.950+0000 7f2cf5e4d640 10 monclient(hunting): handle_auth_done global_id 4217 payload 274 2024-09-17T18:19:02.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.950+0000 7f2cf5e4d640 1 -- 172.21.15.121:0/3897580968 >> v2:172.21.15.121:3300/0 conn(0x7f2cf00a6790 msgr2=0x7f2cf013da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:02.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.950+0000 7f2cf5e4d640 1 --2- 172.21.15.121:0/3897580968 >> v2:172.21.15.121:3300/0 conn(0x7f2cf00a6790 0x7f2cf013da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:02.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.950+0000 7f2cf5e4d640 1 -- 172.21.15.121:0/3897580968 >> v1:172.21.15.121:6789/0 conn(0x7f2cf00a5ca0 legacy=0x7f2cf00a6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:02.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.950+0000 7f2cf5e4d640 10 monclient: _finish_hunting 0 2024-09-17T18:19:02.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.950+0000 7f2cf5e4d640 1 monclient: found mon.noname-b 2024-09-17T18:19:02.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.950+0000 7f2cf5e4d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:02.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.950+0000 7f2cf5e4d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:19:02.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.950+0000 7f2cf5e4d640 1 -- 172.21.15.121:0/3897580968 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cf00a2a50 con 0x7f2cf013dfd0 2024-09-17T18:19:02.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.950+0000 7f51b176c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:02.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.950+0000 7f51b176c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:02.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.950+0000 7f51b176c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:02.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.950+0000 7f51b0f6b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:02.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.950+0000 7f51b0f6b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:02.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.950+0000 7f51b0f6b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:02.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.950+0000 7f51b1f6d640 10 monclient(hunting): handle_auth_done global_id 4245 payload 274 2024-09-17T18:19:02.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.950+0000 7f51b1f6d640 1 -- 172.21.15.121:0/465882080 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f51ac09e810 msgr2=0x7f51ac138360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:02.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.950+0000 7f51b1f6d640 1 --2- 172.21.15.121:0/465882080 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f51ac09e810 0x7f51ac138360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:02.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.950+0000 7f51b1f6d640 1 -- 172.21.15.121:0/465882080 >> v2:172.21.15.173:3300/0 conn(0x7f51ac0700f0 msgr2=0x7f51ac134e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:02.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.950+0000 7f51b1f6d640 1 --2- 172.21.15.121:0/465882080 >> v2:172.21.15.173:3300/0 conn(0x7f51ac0700f0 0x7f51ac134e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:02.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.950+0000 7f51b1f6d640 10 monclient: _finish_hunting 0 2024-09-17T18:19:02.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.950+0000 7f51b1f6d640 1 monclient: found mon.noname-a 2024-09-17T18:19:02.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.950+0000 7f51b1f6d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:02.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.950+0000 7f51b1f6d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:19:02.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.950+0000 7f51b1f6d640 1 -- 172.21.15.121:0/465882080 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51ac09d000 con 0x7f51ac09ec60 2024-09-17T18:19:02.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.950+0000 7fc2f4ad4640 1 Processor -- start 2024-09-17T18:19:02.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.950+0000 7fc2f4ad4640 1 -- start start 2024-09-17T18:19:02.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.950+0000 7fc2f4ad4640 10 monclient: init 2024-09-17T18:19:02.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.951+0000 7fc2f4ad4640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:02.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.951+0000 7fc2f4ad4640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-17T18:19:02.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.951+0000 7fc2f4ad4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc2f00a5ca0 addr v1:172.21.15.121:6789/0 2024-09-17T18:19:02.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.951+0000 7fc2f4ad4640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc2f00a6790 0x7fc2f013da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:02.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.951+0000 7fc2f4ad4640 10 monclient(hunting): picked mon.noname-b con 0x7fc2f00a6790 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:03.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.951+0000 7fc2f4ad4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc2f013dfd0 addr v1:172.21.15.173:6789/0 2024-09-17T18:19:03.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.951+0000 7fc2f4ad4640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2f00a7b50 con 0x7fc2f00a5ca0 2024-09-17T18:19:03.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.951+0000 7fc2f4ad4640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2f00a2a50 con 0x7fc2f013dfd0 2024-09-17T18:19:03.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.951+0000 7fc2f4ad4640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.951+0000 7fc2f4ad4640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc2f00704c0 con 0x7fc2f00a6790 2024-09-17T18:19:03.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.951+0000 7fc2f4ad4640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:03.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.951+0000 7fc2f4ad4640 10 monclient(hunting): authenticate will time out at 746.667461s 2024-09-17T18:19:03.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.951+0000 7fc2e5d9b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc2f00a6790 0x7fc2f013da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.951+0000 7fc2e5d9b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc2f00a6790 0x7fc2f013da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:59314/0 (socket says 172.21.15.121:59314) 2024-09-17T18:19:03.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.951+0000 7fc2eed9d640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fc2f013dfd0 0x7fc2f01403c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.121:40252/0 (socket says 172.21.15.121:40252) 2024-09-17T18:19:03.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.951+0000 7fc2e5d9b640 1 -- 172.21.15.121:0/1835347570 learned_addr learned my addr 172.21.15.121:0/1835347570 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:03.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.951+0000 7fc2e5d9b640 10 monclient(hunting): get_auth_request con 0x7fc2f00a6790 auth_method 0 2024-09-17T18:19:03.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.951+0000 7fc2e5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:03.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.951+0000 7fc2e5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:03.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.951+0000 7fc2e5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:03.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.951+0000 7fc2e5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:03.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.951+0000 7fc2e5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:03.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.951+0000 7fc2e5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:03.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.951+0000 7fc2e5d9b640 10 monclient(hunting): handle_auth_done global_id 4223 payload 274 2024-09-17T18:19:03.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.951+0000 7fc2e5d9b640 1 -- 172.21.15.121:0/1835347570 >> v1:172.21.15.173:6789/0 conn(0x7fc2f013dfd0 legacy=0x7fc2f01403c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:03.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.951+0000 7fc2e5d9b640 1 -- 172.21.15.121:0/1835347570 >> v1:172.21.15.121:6789/0 conn(0x7fc2f00a5ca0 legacy=0x7fc2f00a6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:03.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.951+0000 7fc2e5d9b640 10 monclient: _finish_hunting 0 2024-09-17T18:19:03.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.951+0000 7fc2e5d9b640 1 monclient: found mon.noname-b 2024-09-17T18:19:03.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.951+0000 7fc2e5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:03.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.951+0000 7fc2e5d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:19:03.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.951+0000 7fc2e5d9b640 1 -- 172.21.15.121:0/1835347570 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2f00a2d30 con 0x7fc2f00a6790 2024-09-17T18:19:03.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.953+0000 7f2cf5e4d640 1 -- 172.21.15.121:0/3897580968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2cf013dfd0 msgr2=0x7f2cf01403c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:03.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.953+0000 7f51b1f6d640 1 -- 172.21.15.121:0/465882080 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f51ac09ec60 msgr2=0x7f51ac1388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:03.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.953+0000 7f51b1f6d640 1 --2- 172.21.15.121:0/465882080 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f51ac09ec60 0x7f51ac1388a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f51ac077b10 tx=0x7f51a4032c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c16dbd3d11924188 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:03.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.953+0000 7f2cf5e4d640 1 --2- 172.21.15.121:0/3897580968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2cf013dfd0 0x7f2cf01403c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2ce8009d90 tx=0x7f2ce802a520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=767a02582e79b8db server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:03.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.953+0000 7f51a3fff640 1 -- 172.21.15.121:0/465882080 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51a4003b90 con 0x7f51ac09ec60 2024-09-17T18:19:03.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.953+0000 7f51a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:03.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.953+0000 7f51a3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:19:03.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.953+0000 7f51a3fff640 10 monclient: dump: 2024-09-17T18:19:03.003 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:03.003 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.003 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.003 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.003 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:03.003 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.003 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:03.003 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:03.003 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:03.004 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.953+0000 7f51a3fff640 10 monclient: _finish_auth 0 2024-09-17T18:19:03.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.953+0000 7f51a3fff640 10 monclient: _check_auth_tickets 2024-09-17T18:19:03.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.953+0000 7f51a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:03.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.953+0000 7f2ce7fff640 1 -- 172.21.15.121:0/3897580968 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce8007950 con 0x7f2cf013dfd0 2024-09-17T18:19:03.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.953+0000 7f2ce7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:03.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.953+0000 7f2ce7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:19:03.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.953+0000 7f2ce7fff640 10 monclient: dump: 2024-09-17T18:19:03.005 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:03.005 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.005 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.005 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.005 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:03.005 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.005 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:03.005 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:03.005 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:03.005 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.953+0000 7f2ce7fff640 10 monclient: _finish_auth 0 2024-09-17T18:19:03.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.953+0000 7f2ce7fff640 10 monclient: _check_auth_tickets 2024-09-17T18:19:03.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.953+0000 7f2ce7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:03.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.953+0000 7f2ce7fff640 1 -- 172.21.15.121:0/3897580968 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ce8007af0 con 0x7f2cf013dfd0 2024-09-17T18:19:03.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.953+0000 7f2ce7fff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:03.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.953+0000 7f2ce7fff640 1 -- 172.21.15.121:0/3897580968 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce8031400 con 0x7f2cf013dfd0 2024-09-17T18:19:03.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.953+0000 7f2ce7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:03.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.953+0000 7f2ce7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:19:03.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.953+0000 7f2ce7fff640 10 monclient: dump: 2024-09-17T18:19:03.006 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:03.006 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.006 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.006 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.006 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:03.006 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.006 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:03.006 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:03.007 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:03.007 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.953+0000 7f2cf78b0640 5 monclient: authenticate success, global_id 4217 2024-09-17T18:19:03.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.953+0000 7f2cf78b0640 10 monclient: get_monmap_and_config success 2024-09-17T18:19:03.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.953+0000 7f2cf78b0640 4 set_mon_vals no callback set 2024-09-17T18:19:03.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.953+0000 7f51a3fff640 1 -- 172.21.15.121:0/465882080 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51a4003d30 con 0x7f51ac09ec60 2024-09-17T18:19:03.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.953+0000 7f51a3fff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:03.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.953+0000 7f51b39d0640 5 monclient: authenticate success, global_id 4245 2024-09-17T18:19:03.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.953+0000 7f51b39d0640 10 monclient: get_monmap_and_config success 2024-09-17T18:19:03.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.953+0000 7f51b39d0640 4 set_mon_vals no callback set 2024-09-17T18:19:03.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.953+0000 7f51a3fff640 1 -- 172.21.15.121:0/465882080 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51a4033b90 con 0x7f51ac09ec60 2024-09-17T18:19:03.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.953+0000 7f51a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:03.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.953+0000 7f51a3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:19:03.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.953+0000 7f51a3fff640 10 monclient: dump: 2024-09-17T18:19:03.008 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:03.008 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.008 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.008 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.008 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:03.008 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.009 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:03.009 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:03.009 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:03.009 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.953+0000 7f3b239f8640 10 monclient: get_monmap_and_config 2024-09-17T18:19:03.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.953+0000 7f3b239f8640 10 monclient: build_initial_monmap 2024-09-17T18:19:03.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.953+0000 7f3b239f8640 10 monclient: monmap: 2024-09-17T18:19:03.009 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:03.009 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.009 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:19:02.954224+0000 2024-09-17T18:19:03.009 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:19:02.954224+0000 2024-09-17T18:19:03.010 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:03.010 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.010 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:19:03.010 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:19:03.010 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:19:03.010 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:19:03.010 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:19:03.010 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.954+0000 7f51b39d0640 10 monclient: shutdown 2024-09-17T18:19:03.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.954+0000 7f51b39d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:03.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.954+0000 7f51b39d0640 1 -- 172.21.15.121:0/465882080 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f51ac09ec60 msgr2=0x7f51ac1388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.954+0000 7f51b39d0640 1 --2- 172.21.15.121:0/465882080 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f51ac09ec60 0x7f51ac1388a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f51ac077b10 tx=0x7f51a4032c30 comp rx=0 tx=0).stop 2024-09-17T18:19:03.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.954+0000 7f2cf78b0640 10 monclient: shutdown 2024-09-17T18:19:03.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.954+0000 7f2cf78b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:03.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.954+0000 7f2cf78b0640 1 -- 172.21.15.121:0/3897580968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2cf013dfd0 msgr2=0x7f2cf01403c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.954+0000 7f51b39d0640 1 -- 172.21.15.121:0/465882080 shutdown_connections 2024-09-17T18:19:03.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.954+0000 7f51b39d0640 1 --2- 172.21.15.121:0/465882080 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f51ac09ec60 0x7f51ac1388a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.954+0000 7f51b39d0640 1 --2- 172.21.15.121:0/465882080 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f51ac09e810 0x7f51ac138360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.954+0000 7f51b39d0640 1 --2- 172.21.15.121:0/465882080 >> v2:172.21.15.173:3300/0 conn(0x7f51ac0700f0 0x7f51ac134e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.954+0000 7f51b39d0640 1 -- 172.21.15.121:0/465882080 >> 172.21.15.121:0/465882080 conn(0x7f51ac0ab830 msgr2=0x7f51ac0a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:03.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.954+0000 7f2cf78b0640 1 --2- 172.21.15.121:0/3897580968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2cf013dfd0 0x7f2cf01403c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2ce8009d90 tx=0x7f2ce802a520 comp rx=0 tx=0).stop 2024-09-17T18:19:03.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.954+0000 7f2cf78b0640 1 -- 172.21.15.121:0/3897580968 shutdown_connections 2024-09-17T18:19:03.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.954+0000 7f51b39d0640 1 -- 172.21.15.121:0/465882080 shutdown_connections 2024-09-17T18:19:03.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.954+0000 7f2cf78b0640 1 --2- 172.21.15.121:0/3897580968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2cf013dfd0 0x7f2cf01403c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.954+0000 7f2cf78b0640 1 --2- 172.21.15.121:0/3897580968 >> v2:172.21.15.121:3300/0 conn(0x7f2cf00a6790 0x7f2cf013da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.954+0000 7f3b239f8640 1 Processor -- start 2024-09-17T18:19:03.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.954+0000 7f51b39d0640 1 -- 172.21.15.121:0/465882080 wait complete. 2024-09-17T18:19:03.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.954+0000 7f2cf78b0640 1 -- 172.21.15.121:0/3897580968 >> 172.21.15.121:0/3897580968 conn(0x7f2cf006dd10 msgr2=0x7f2cf00a7da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:03.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.954+0000 7fc2e5d9b640 1 -- 172.21.15.121:0/1835347570 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc2f00a6790 msgr2=0x7fc2f013da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:03.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.954+0000 7fc2e5d9b640 1 --2- 172.21.15.121:0/1835347570 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc2f00a6790 0x7fc2f013da90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc2d80098a0 tx=0x7fc2d802a080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9e722a3438d1dc4 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:03.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.954+0000 7f2cf78b0640 1 -- 172.21.15.121:0/3897580968 shutdown_connections 2024-09-17T18:19:03.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.954+0000 7fc2edd9b640 1 -- 172.21.15.121:0/1835347570 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d8037070 con 0x7fc2f00a6790 2024-09-17T18:19:03.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.954+0000 7fc2edd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:03.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.954+0000 7fc2edd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:19:03.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.954+0000 7fc2edd9b640 10 monclient: dump: 2024-09-17T18:19:03.015 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:03.015 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.015 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.015 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.016 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:03.016 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.016 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:03.016 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:03.016 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:03.016 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.954+0000 7fc2edd9b640 10 monclient: _finish_auth 0 2024-09-17T18:19:03.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.954+0000 7fc2edd9b640 10 monclient: _check_auth_tickets 2024-09-17T18:19:03.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.954+0000 7fc2edd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:03.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.954+0000 7fc2edd9b640 1 -- 172.21.15.121:0/1835347570 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2d8032640 con 0x7fc2f00a6790 2024-09-17T18:19:03.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.954+0000 7fc2edd9b640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:03.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.954+0000 7fc2edd9b640 1 -- 172.21.15.121:0/1835347570 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d8032a10 con 0x7fc2f00a6790 2024-09-17T18:19:03.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.954+0000 7fc2edd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:03.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.954+0000 7f2cf78b0640 1 -- 172.21.15.121:0/3897580968 wait complete. 2024-09-17T18:19:03.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.954+0000 7fc2edd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:19:03.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.954+0000 7fc2edd9b640 10 monclient: dump: 2024-09-17T18:19:03.017 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:03.017 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.017 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.017 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.017 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:03.018 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.018 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:03.018 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:03.018 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:03.018 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.954+0000 7fc2f4ad4640 5 monclient: authenticate success, global_id 4223 2024-09-17T18:19:03.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.954+0000 7fc2f4ad4640 10 monclient: get_monmap_and_config success 2024-09-17T18:19:03.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.954+0000 7fc2f4ad4640 4 set_mon_vals no callback set 2024-09-17T18:19:03.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.954+0000 7f3b239f8640 1 -- start start 2024-09-17T18:19:03.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.954+0000 7f3b239f8640 10 monclient: init 2024-09-17T18:19:03.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.955+0000 7f51b39d0640 10 monclient: build_initial_monmap 2024-09-17T18:19:03.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.955+0000 7f3b239f8640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:03.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.955+0000 7f51b39d0640 10 monclient: monmap: 2024-09-17T18:19:03.019 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:03.019 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:19:03.019 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:19:03.019 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:19:03.020 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:03.020 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.020 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:19:03.020 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:19:03.020 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:19:03.020 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.955+0000 7f3b239f8640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-17T18:19:03.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.955+0000 7f3b239f8640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f3b1c0a5ca0 0x7f3b1c0a6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.955+0000 7f3b239f8640 10 monclient(hunting): picked mon.noname-a con 0x7f3b1c0a5ca0 addr v2:172.21.15.121:3300/0 2024-09-17T18:19:03.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.955+0000 7f3b239f8640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b1c0a65c0 0x7f3b1c13ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.955+0000 7f3b239f8640 10 monclient(hunting): picked mon.noname-b con 0x7f3b1c0a65c0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:03.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.955+0000 7f3b239f8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3b1c13cfc0 addr v1:172.21.15.173:6789/0 2024-09-17T18:19:03.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.955+0000 7f2cf78b0640 10 monclient: build_initial_monmap 2024-09-17T18:19:03.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.955+0000 7f3b239f8640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b1c0a7b50 con 0x7f3b1c13cfc0 2024-09-17T18:19:03.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.955+0000 7f3b239f8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.955+0000 7f2cf78b0640 10 monclient: monmap: 2024-09-17T18:19:03.022 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:03.022 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:19:03.022 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:19:03.022 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:19:03.022 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:03.022 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.022 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:19:03.023 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:19:03.023 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:19:03.023 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.955+0000 7fc2f4ad4640 10 monclient: shutdown 2024-09-17T18:19:03.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.955+0000 7f3b239f8640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f3b1c0704c0 con 0x7f3b1c0a5ca0 2024-09-17T18:19:03.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.955+0000 7fc2f4ad4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:03.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.955+0000 7fc2f4ad4640 1 -- 172.21.15.121:0/1835347570 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc2f00a6790 msgr2=0x7fc2f013da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.955+0000 7fc2f4ad4640 1 --2- 172.21.15.121:0/1835347570 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc2f00a6790 0x7fc2f013da90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc2d80098a0 tx=0x7fc2d802a080 comp rx=0 tx=0).stop 2024-09-17T18:19:03.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.955+0000 7f3b239f8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.955+0000 7f3b21794640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f3b1c0a5ca0 0x7f3b1c0a6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.955+0000 7f3b239f8640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3b1c0a5490 con 0x7f3b1c0a65c0 2024-09-17T18:19:03.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.955+0000 7fc2f4ad4640 1 -- 172.21.15.121:0/1835347570 shutdown_connections 2024-09-17T18:19:03.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.955+0000 7fc2f4ad4640 1 --2- 172.21.15.121:0/1835347570 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc2f00a6790 0x7fc2f013da90 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.955+0000 7fc2f4ad4640 1 -- 172.21.15.121:0/1835347570 >> 172.21.15.121:0/1835347570 conn(0x7fc2f006dd10 msgr2=0x7fc2f00a7da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:03.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.955+0000 7f3b21794640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f3b1c0a5ca0 0x7f3b1c0a6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40890/0 (socket says 172.21.15.121:40890) 2024-09-17T18:19:03.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.955+0000 7f3b21794640 1 -- 172.21.15.121:0/1828442591 learned_addr learned my addr 172.21.15.121:0/1828442591 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:03.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.955+0000 7f3b21f95640 1 --1- 172.21.15.121:0/1828442591 >> v1:172.21.15.173:6789/0 conn(0x7f3b1c13cfc0 0x7f3b1c13f3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.121:40262/0 (socket says 172.21.15.121:40262) 2024-09-17T18:19:03.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.955+0000 7f51b39d0640 1 librados: starting msgr at 2024-09-17T18:19:03.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.955+0000 7f3b239f8640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:03.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.955+0000 7f3b20f93640 1 --2- 172.21.15.121:0/1828442591 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b1c0a65c0 0x7f3b1c13ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.955+0000 7f3b239f8640 10 monclient(hunting): authenticate will time out at 746.671875s 2024-09-17T18:19:03.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.955+0000 7f51b39d0640 1 librados: starting objecter 2024-09-17T18:19:03.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.955+0000 7f3b21794640 10 monclient(hunting): get_auth_request con 0x7f3b1c0a5ca0 auth_method 0 2024-09-17T18:19:03.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.955+0000 7fc2f4ad4640 1 -- 172.21.15.121:0/1835347570 shutdown_connections 2024-09-17T18:19:03.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.955+0000 7f3b21794640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:03.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.955+0000 7f3b21794640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:03.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.955+0000 7f3b21794640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:03.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.955+0000 7f3b20f93640 10 monclient(hunting): get_auth_request con 0x7f3b1c0a65c0 auth_method 0 2024-09-17T18:19:03.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.955+0000 7f3b20f93640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:03.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.955+0000 7f3b20f93640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:03.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.955+0000 7f3b20f93640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:03.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.955+0000 7fc2f4ad4640 1 -- 172.21.15.121:0/1835347570 wait complete. 2024-09-17T18:19:03.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.956+0000 7f51b39d0640 1 Processor -- start 2024-09-17T18:19:03.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.956+0000 7f2cf78b0640 1 librados: starting msgr at 2024-09-17T18:19:03.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.956+0000 7f3b21794640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:03.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.956+0000 7f3b21794640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:03.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.956+0000 7f3b21794640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:03.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.956+0000 7f3b20f93640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:03.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.956+0000 7f3b20f93640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:03.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.956+0000 7f3b20f93640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:03.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.956+0000 7f3b20f93640 10 monclient(hunting): handle_auth_done global_id 4226 payload 274 2024-09-17T18:19:03.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.956+0000 7f2cf78b0640 1 librados: starting objecter 2024-09-17T18:19:03.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.956+0000 7f3b20f93640 1 -- 172.21.15.121:0/1828442591 >> v2:172.21.15.121:3300/0 conn(0x7f3b1c0a5ca0 msgr2=0x7f3b1c0a6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:03.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.956+0000 7f2cf78b0640 1 Processor -- start 2024-09-17T18:19:03.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.956+0000 7f51b39d0640 1 -- start start 2024-09-17T18:19:03.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.956+0000 7f51b39d0640 1 librados: setting wanted keys 2024-09-17T18:19:03.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.956+0000 7f51b39d0640 1 librados: calling monclient init 2024-09-17T18:19:03.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.956+0000 7f51b39d0640 10 monclient: init 2024-09-17T18:19:03.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.956+0000 7f2cf78b0640 1 -- start start 2024-09-17T18:19:03.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.956+0000 7f2cf78b0640 1 librados: setting wanted keys 2024-09-17T18:19:03.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.956+0000 7f2cf78b0640 1 librados: calling monclient init 2024-09-17T18:19:03.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.956+0000 7f2cf78b0640 10 monclient: init 2024-09-17T18:19:03.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.956+0000 7fc2f4ad4640 10 monclient: build_initial_monmap 2024-09-17T18:19:03.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.956+0000 7fc2f4ad4640 10 monclient: monmap: 2024-09-17T18:19:03.031 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:03.032 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:19:03.032 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:19:03.032 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:19:03.032 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:03.032 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.032 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:19:03.032 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:19:03.032 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:19:03.032 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.956+0000 7f3b20f93640 1 --2- 172.21.15.121:0/1828442591 >> v2:172.21.15.121:3300/0 conn(0x7f3b1c0a5ca0 0x7f3b1c0a6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.956+0000 7f3b20f93640 1 -- 172.21.15.121:0/1828442591 >> v1:172.21.15.173:6789/0 conn(0x7f3b1c13cfc0 legacy=0x7f3b1c13f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:03.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.956+0000 7f3b20f93640 10 monclient: _finish_hunting 0 2024-09-17T18:19:03.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.956+0000 7f3b20f93640 1 monclient: found mon.noname-b 2024-09-17T18:19:03.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.956+0000 7f3b20f93640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:03.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.956+0000 7f3b20f93640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:19:03.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.956+0000 7f3b20f93640 1 -- 172.21.15.121:0/1828442591 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b1c0a2a50 con 0x7f3b1c0a65c0 2024-09-17T18:19:03.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.956+0000 7f3b21794640 1 --2- 172.21.15.121:0/1828442591 >> v2:172.21.15.121:3300/0 conn(0x7f3b1c0a5ca0 0x7f3b1c0a6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:19:03.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.956+0000 7f2cf78b0640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:03.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.956+0000 7f2cf78b0640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T18:19:03.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.956+0000 7f2cf78b0640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2cf00a5ca0 0x7f2cf00a1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf78b0640 10 monclient(hunting): picked mon.noname-b con 0x7f2cf00a5ca0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:03.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf78b0640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2cf00a6790 0x7f2cf0137f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf78b0640 10 monclient(hunting): picked mon.noname-c con 0x7f2cf00a6790 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:03.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf78b0640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2cf013dfd0 0x7f2cf0138470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f51b39d0640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:03.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf4e4b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2cf00a6790 0x7f2cf0137f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf78b0640 10 monclient(hunting): picked mon.noname-a con 0x7f2cf013dfd0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:03.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf78b0640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf4e4b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2cf00a6790 0x7f2cf0137f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:59344/0 (socket says 172.21.15.121:59344) 2024-09-17T18:19:03.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf78b0640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf00a5670 con 0x7f2cf013dfd0 2024-09-17T18:19:03.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf78b0640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf4e4b640 1 -- 172.21.15.121:0/2538836062 learned_addr learned my addr 172.21.15.121:0/2538836062 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:03.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf78b0640 1 -- 172.21.15.121:0/2538836062 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf00a6eb0 con 0x7f2cf00a5ca0 2024-09-17T18:19:03.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf78b0640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f51b39d0640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T18:19:03.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf78b0640 1 -- 172.21.15.121:0/2538836062 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2cf00a8720 con 0x7f2cf00a6790 2024-09-17T18:19:03.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf78b0640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:03.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf78b0640 10 monclient(hunting): authenticate will time out at 746.673243s 2024-09-17T18:19:03.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf5e4d640 1 --2- 172.21.15.121:0/2538836062 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2cf013dfd0 0x7f2cf0138470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f51b39d0640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f51ac0700f0 0x7f51ac1c94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf5e4d640 10 monclient(hunting): get_auth_request con 0x7f2cf013dfd0 auth_method 0 2024-09-17T18:19:03.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf5e4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:03.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf5e4d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:03.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf5e4d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:03.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf564c640 1 --2- 172.21.15.121:0/2538836062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2cf00a5ca0 0x7f2cf00a1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf4e4b640 10 monclient(hunting): get_auth_request con 0x7f2cf00a6790 auth_method 0 2024-09-17T18:19:03.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf4e4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:03.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf4e4b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:03.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf4e4b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:03.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf564c640 10 monclient(hunting): get_auth_request con 0x7f2cf00a5ca0 auth_method 0 2024-09-17T18:19:03.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf564c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:03.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf564c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:03.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf564c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:03.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf5e4d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:03.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f51b39d0640 10 monclient(hunting): picked mon.noname-b con 0x7f51ac0700f0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:03.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f51b39d0640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f51ac09e810 0x7f51ac1c9a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf5e4d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:03.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf5e4d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:03.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf4e4b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:03.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf4e4b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:03.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf4e4b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:03.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf564c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:03.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf564c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:03.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf564c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:03.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7fc2f4ad4640 1 librados: starting msgr at 2024-09-17T18:19:03.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7fc2f4ad4640 1 librados: starting objecter 2024-09-17T18:19:03.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7fc2f4ad4640 1 Processor -- start 2024-09-17T18:19:03.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f51b176c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f51ac0700f0 0x7f51ac1c94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f51b176c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f51ac0700f0 0x7f51ac1c94c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.121:42070/0 (socket says 172.21.15.121:42070) 2024-09-17T18:19:03.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f51b176c640 1 -- 172.21.15.121:0/731052961 learned_addr learned my addr 172.21.15.121:0/731052961 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:03.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf5e4d640 10 monclient(hunting): handle_auth_done global_id 4251 payload 824 2024-09-17T18:19:03.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f51b0f6b640 1 --2- 172.21.15.121:0/731052961 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f51ac09e810 0x7f51ac1c9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf5e4d640 1 -- 172.21.15.121:0/2538836062 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2cf00a6790 msgr2=0x7f2cf0137f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf5e4d640 1 --2- 172.21.15.121:0/2538836062 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2cf00a6790 0x7f2cf0137f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf5e4d640 1 -- 172.21.15.121:0/2538836062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2cf00a5ca0 msgr2=0x7f2cf00a1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf5e4d640 1 --2- 172.21.15.121:0/2538836062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2cf00a5ca0 0x7f2cf00a1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf5e4d640 10 monclient: _finish_hunting 0 2024-09-17T18:19:03.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf5e4d640 1 monclient: found mon.noname-a 2024-09-17T18:19:03.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf5e4d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:03.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf5e4d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:19:03.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf5e4d640 1 -- 172.21.15.121:0/2538836062 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cf0123870 con 0x7f2cf013dfd0 2024-09-17T18:19:03.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf4e4b640 1 --2- 172.21.15.121:0/2538836062 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2cf00a6790 0x7f2cf0137f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:19:03.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f2cf564c640 1 --2- 172.21.15.121:0/2538836062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2cf00a5ca0 0x7f2cf00a1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:19:03.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f51b39d0640 10 monclient(hunting): picked mon.noname-c con 0x7f51ac09e810 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:03.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f51b39d0640 1 --2- 172.21.15.121:0/731052961 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f51ac09ec60 0x7f51ac09fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7f51b39d0640 10 monclient(hunting): picked mon.noname-a con 0x7f51ac09ec60 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:03.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7f51b1f6d640 1 --2- 172.21.15.121:0/731052961 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f51ac09ec60 0x7f51ac09fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7f51b39d0640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7fc2f4ad4640 1 -- start start 2024-09-17T18:19:03.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7fc2f4ad4640 1 librados: setting wanted keys 2024-09-17T18:19:03.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7fc2f4ad4640 1 librados: calling monclient init 2024-09-17T18:19:03.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.957+0000 7fc2f4ad4640 10 monclient: init 2024-09-17T18:19:03.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7f51b39d0640 1 -- 172.21.15.121:0/731052961 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f51ac135510 con 0x7f51ac09ec60 2024-09-17T18:19:03.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7f51b39d0640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7f51b39d0640 1 -- 172.21.15.121:0/731052961 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f51ac135390 con 0x7f51ac0700f0 2024-09-17T18:19:03.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7f51b39d0640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7f51b39d0640 1 -- 172.21.15.121:0/731052961 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f51ac138de0 con 0x7f51ac09e810 2024-09-17T18:19:03.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7f51b39d0640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:03.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7f51b39d0640 10 monclient(hunting): authenticate will time out at 746.674278s 2024-09-17T18:19:03.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7fc2f4ad4640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:03.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7fc2f4ad4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T18:19:03.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7fc2f4ad4640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc2f00a5ca0 0x7fc2f0138260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7f51b176c640 10 monclient(hunting): get_auth_request con 0x7f51ac0700f0 auth_method 0 2024-09-17T18:19:03.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7f51b176c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:03.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7f51b176c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:03.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7f51b176c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:03.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7f51b0f6b640 10 monclient(hunting): get_auth_request con 0x7f51ac09e810 auth_method 0 2024-09-17T18:19:03.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7f51b0f6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:03.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7f51b0f6b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:03.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7f51b0f6b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:03.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7fc2f4ad4640 10 monclient(hunting): picked mon.noname-c con 0x7fc2f00a5ca0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:03.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7f51b0f6b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:03.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7f51b0f6b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:03.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7f51b0f6b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:03.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7f51b176c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:03.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7f51b176c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:03.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7f51b176c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:03.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7fc2f4ad4640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2f00a6790 0x7fc2f01ca1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7f51b1f6d640 10 monclient(hunting): get_auth_request con 0x7f51ac09ec60 auth_method 0 2024-09-17T18:19:03.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7f51b1f6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:03.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7f51b1f6d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:03.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7f51b1f6d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:03.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7fc2f4ad4640 10 monclient(hunting): picked mon.noname-a con 0x7fc2f00a6790 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:03.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7fc2e5d9b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2f00a6790 0x7fc2f01ca1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7fc2e5d9b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2f00a6790 0x7fc2f01ca1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40914/0 (socket says 172.21.15.121:40914) 2024-09-17T18:19:03.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7fc2e5d9b640 1 -- 172.21.15.121:0/1365029666 learned_addr learned my addr 172.21.15.121:0/1365029666 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:03.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7fc2ee59c640 1 --2- 172.21.15.121:0/1365029666 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc2f00a5ca0 0x7fc2f0138260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7f51b0f6b640 10 monclient(hunting): handle_auth_done global_id 4232 payload 824 2024-09-17T18:19:03.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7f51b0f6b640 1 -- 172.21.15.121:0/731052961 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f51ac0700f0 msgr2=0x7f51ac1c94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7f51b0f6b640 1 --2- 172.21.15.121:0/731052961 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f51ac0700f0 0x7f51ac1c94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7f51b0f6b640 1 -- 172.21.15.121:0/731052961 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f51ac09ec60 msgr2=0x7f51ac09fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7f51b0f6b640 1 --2- 172.21.15.121:0/731052961 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f51ac09ec60 0x7f51ac09fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7fc2f4ad4640 1 --2- 172.21.15.121:0/1365029666 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2f013dfd0 0x7fc2f01ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7f51b0f6b640 10 monclient: _finish_hunting 0 2024-09-17T18:19:03.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7f51b0f6b640 1 monclient: found mon.noname-c 2024-09-17T18:19:03.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7f51b0f6b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:03.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7f51b0f6b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:19:03.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7f51b0f6b640 1 -- 172.21.15.121:0/731052961 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51ac1343b0 con 0x7f51ac09e810 2024-09-17T18:19:03.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7fc2f4ad4640 10 monclient(hunting): picked mon.noname-b con 0x7fc2f013dfd0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:03.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7fc2f4ad4640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7fc2f4ad4640 1 -- 172.21.15.121:0/1365029666 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f00a5670 con 0x7fc2f00a6790 2024-09-17T18:19:03.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7fc2f4ad4640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7f51b1f6d640 1 --2- 172.21.15.121:0/731052961 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f51ac09ec60 0x7f51ac09fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T18:19:03.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7fc2f4ad4640 1 -- 172.21.15.121:0/1365029666 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f00a5490 con 0x7fc2f013dfd0 2024-09-17T18:19:03.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7fc2f4ad4640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.958+0000 7fc2f4ad4640 1 -- 172.21.15.121:0/1365029666 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc2f00a8720 con 0x7fc2f00a5ca0 2024-09-17T18:19:03.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7fc2f4ad4640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:03.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7fc2f4ad4640 10 monclient(hunting): authenticate will time out at 746.675028s 2024-09-17T18:19:03.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7fc2e5d9b640 10 monclient(hunting): get_auth_request con 0x7fc2f00a6790 auth_method 0 2024-09-17T18:19:03.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7fc2e5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:03.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7fc2e5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:03.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7fc2e5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:03.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7fc2eed9d640 1 --2- 172.21.15.121:0/1365029666 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2f013dfd0 0x7fc2f01ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7fc2eed9d640 10 monclient(hunting): get_auth_request con 0x7fc2f013dfd0 auth_method 0 2024-09-17T18:19:03.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7fc2eed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:03.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7fc2eed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:03.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7fc2eed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:03.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7fc2ee59c640 10 monclient(hunting): get_auth_request con 0x7fc2f00a5ca0 auth_method 0 2024-09-17T18:19:03.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7fc2ee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:03.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7fc2ee59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:03.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7fc2ee59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:03.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7f3b20f93640 1 -- 172.21.15.121:0/1828442591 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b1c0a65c0 msgr2=0x7f3b1c13ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:03.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7f3b20f93640 1 --2- 172.21.15.121:0/1828442591 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b1c0a65c0 0x7f3b1c13ca80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3b180097a0 tx=0x7f3b18029fc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=333e1d1b18d9f2aa server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:03.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7fc2e5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:03.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7fc2e5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:03.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7fc2e5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:03.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7fc2eed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:03.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7fc2eed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:03.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7fc2eed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:03.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7fc2e5d9b640 10 monclient(hunting): handle_auth_done global_id 4254 payload 824 2024-09-17T18:19:03.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7fc2e5d9b640 1 -- 172.21.15.121:0/1365029666 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc2f00a5ca0 msgr2=0x7fc2f0138260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7fc2e5d9b640 1 --2- 172.21.15.121:0/1365029666 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc2f00a5ca0 0x7fc2f0138260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7fc2e5d9b640 1 -- 172.21.15.121:0/1365029666 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2f013dfd0 msgr2=0x7fc2f01ca6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7fc2e5d9b640 1 --2- 172.21.15.121:0/1365029666 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2f013dfd0 0x7fc2f01ca6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7fc2e5d9b640 10 monclient: _finish_hunting 0 2024-09-17T18:19:03.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7fc2e5d9b640 1 monclient: found mon.noname-a 2024-09-17T18:19:03.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7fc2e5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:03.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7fc2e5d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:19:03.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7f3b0ffff640 1 -- 172.21.15.121:0/1828442591 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b18037070 con 0x7f3b1c0a65c0 2024-09-17T18:19:03.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7f3b0ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:03.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7fc2e5d9b640 1 -- 172.21.15.121:0/1365029666 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2f0124140 con 0x7fc2f00a6790 2024-09-17T18:19:03.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7f3b0ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:19:03.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7f3b0ffff640 10 monclient: dump: 2024-09-17T18:19:03.059 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:03.059 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.059 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.059 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.059 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:03.059 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.059 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:03.059 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:03.059 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:03.059 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7f3b0ffff640 10 monclient: _finish_auth 0 2024-09-17T18:19:03.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7f3b0ffff640 10 monclient: _check_auth_tickets 2024-09-17T18:19:03.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7f3b0ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:03.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7f3b0ffff640 1 -- 172.21.15.121:0/1828442591 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b18032440 con 0x7f3b1c0a65c0 2024-09-17T18:19:03.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7f3b0ffff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:03.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7f3b0ffff640 1 -- 172.21.15.121:0/1828442591 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b18037070 con 0x7f3b1c0a65c0 2024-09-17T18:19:03.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7f3b239f8640 5 monclient: authenticate success, global_id 4226 2024-09-17T18:19:03.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7f3b0ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:03.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7f3b0ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:19:03.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7f3b0ffff640 10 monclient: dump: 2024-09-17T18:19:03.060 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:03.060 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.060 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.060 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.060 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:03.061 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.061 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:03.061 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:03.061 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:03.061 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7f3b239f8640 10 monclient: get_monmap_and_config success 2024-09-17T18:19:03.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7f3b239f8640 4 set_mon_vals no callback set 2024-09-17T18:19:03.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.959+0000 7fc2ee59c640 1 --2- 172.21.15.121:0/1365029666 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc2f00a5ca0 0x7fc2f0138260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T18:19:03.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.960+0000 7f2cf5e4d640 1 -- 172.21.15.121:0/2538836062 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2cf013dfd0 msgr2=0x7f2cf0138470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:03.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.960+0000 7f2cf5e4d640 1 --2- 172.21.15.121:0/2538836062 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2cf013dfd0 0x7f2cf0138470 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2ce8034000 tx=0x7f2ce8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:03.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.960+0000 7f2ce5ffb640 10 client.?.objecter ms_handle_connect 0x7f2cf013dfd0 2024-09-17T18:19:03.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.960+0000 7f2ce5ffb640 10 client.?.objecter resend_mon_ops 2024-09-17T18:19:03.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.960+0000 7f2ce5ffb640 1 -- 172.21.15.121:0/2538836062 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce8040070 con 0x7f2cf013dfd0 2024-09-17T18:19:03.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.960+0000 7f2ce5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:03.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.960+0000 7f2ce5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:19:03.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.960+0000 7f2ce5ffb640 10 monclient: dump: 2024-09-17T18:19:03.063 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:03.063 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.063 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.063 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.063 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:03.063 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.063 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:03.063 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:03.063 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:03.063 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.960+0000 7f2ce5ffb640 10 monclient: _finish_auth 0 2024-09-17T18:19:03.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.960+0000 7f2ce5ffb640 10 monclient: _check_auth_tickets 2024-09-17T18:19:03.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.960+0000 7f2ce5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:03.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.960+0000 7f2ce5ffb640 1 -- 172.21.15.121:0/2538836062 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ce8031e50 con 0x7f2cf013dfd0 2024-09-17T18:19:03.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.960+0000 7f2ce5ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:03.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.960+0000 7f2ce5ffb640 1 -- 172.21.15.121:0/2538836062 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce80458a0 con 0x7f2cf013dfd0 2024-09-17T18:19:03.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.960+0000 7f2ce5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:03.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.960+0000 7f2ce5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:19:03.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.960+0000 7f2ce5ffb640 10 monclient: dump: 2024-09-17T18:19:03.064 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:03.064 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.064 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.064 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.065 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:03.065 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.065 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:03.065 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:03.065 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:03.065 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.960+0000 7f2cf78b0640 5 monclient: authenticate success, global_id 4251 2024-09-17T18:19:03.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.960+0000 7f2ce77fe640 4 set_mon_vals no callback set 2024-09-17T18:19:03.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.960+0000 7f2cf78b0640 10 monclient: _renew_subs 2024-09-17T18:19:03.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.960+0000 7f2cf78b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:03.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.960+0000 7f2cf78b0640 1 -- 172.21.15.121:0/2538836062 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cf00a1ad0 con 0x7f2cf013dfd0 2024-09-17T18:19:03.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.960+0000 7f3b239f8640 10 monclient: shutdown 2024-09-17T18:19:03.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.960+0000 7f3b239f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:03.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.960+0000 7f3b239f8640 1 -- 172.21.15.121:0/1828442591 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b1c0a65c0 msgr2=0x7f3b1c13ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.960+0000 7f3b239f8640 1 --2- 172.21.15.121:0/1828442591 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b1c0a65c0 0x7f3b1c13ca80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3b180097a0 tx=0x7f3b18029fc0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.960+0000 7f3b239f8640 1 -- 172.21.15.121:0/1828442591 shutdown_connections 2024-09-17T18:19:03.067 INFO:teuthology.orchestra.run.smithi121.stdout:47244640259 2024-09-17T18:19:03.067 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-17T18:19:03.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.960+0000 7f3b239f8640 1 --2- 172.21.15.121:0/1828442591 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b1c0a65c0 0x7f3b1c13ca80 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.960+0000 7f3b239f8640 1 --2- 172.21.15.121:0/1828442591 >> v2:172.21.15.121:3300/0 conn(0x7f3b1c0a5ca0 0x7f3b1c0a6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.960+0000 7f3b239f8640 1 -- 172.21.15.121:0/1828442591 >> 172.21.15.121:0/1828442591 conn(0x7f3b1c06dd10 msgr2=0x7f3b1c0a7da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:03.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.960+0000 7f2cf78b0640 10 client.4251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:03.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.960+0000 7f2cf78b0640 10 monclient: _renew_subs 2024-09-17T18:19:03.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.960+0000 7f2cf78b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:03.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.960+0000 7f2cf78b0640 1 -- 172.21.15.121:0/2538836062 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cf0138a30 con 0x7f2cf013dfd0 2024-09-17T18:19:03.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.960+0000 7f2cf78b0640 1 librados: init done 2024-09-17T18:19:03.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.961+0000 7f51b0f6b640 1 -- 172.21.15.121:0/731052961 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f51ac09e810 msgr2=0x7f51ac1c9a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:03.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.961+0000 7f3b239f8640 1 -- 172.21.15.121:0/1828442591 shutdown_connections 2024-09-17T18:19:03.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.961+0000 7f3b239f8640 1 -- 172.21.15.121:0/1828442591 wait complete. 2024-09-17T18:19:03.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.961+0000 7f51b0f6b640 1 --2- 172.21.15.121:0/731052961 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f51ac09e810 0x7f51ac1c9a00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f519c00e850 tx=0x7f519c00ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:03.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.961+0000 7f51a1ffb640 10 client.?.objecter ms_handle_connect 0x7f51ac09e810 2024-09-17T18:19:03.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.961+0000 7f51a1ffb640 10 client.?.objecter resend_mon_ops 2024-09-17T18:19:03.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.961+0000 7f51a1ffb640 1 -- 172.21.15.121:0/731052961 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f519c017070 con 0x7f51ac09e810 2024-09-17T18:19:03.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.961+0000 7f51a1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:03.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.961+0000 7f51a1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:19:03.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.961+0000 7f51a1ffb640 10 monclient: dump: 2024-09-17T18:19:03.071 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:03.071 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.071 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.071 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.071 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:03.071 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.071 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:03.071 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:03.071 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:03.071 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.961+0000 7f51a1ffb640 10 monclient: _finish_auth 0 2024-09-17T18:19:03.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.961+0000 7f2ce5ffb640 1 -- 172.21.15.121:0/2538836062 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f2ce8039040 con 0x7f2cf013dfd0 2024-09-17T18:19:03.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.961+0000 7f2cf78b0640 10 client.4251.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:03.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.961+0000 7f2cf78b0640 15 client.4251.objecter _session_command_op_assign -1 1 2024-09-17T18:19:03.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.961+0000 7f2cf78b0640 20 client.4251.objecter _get_session osd=-1 returning homeless 2024-09-17T18:19:03.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.961+0000 7f2cf78b0640 10 client.4251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:03.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.961+0000 7f2cf78b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:03.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.961+0000 7f2cf78b0640 1 -- 172.21.15.121:0/2538836062 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2cf00a6080 con 0x7f2cf013dfd0 2024-09-17T18:19:03.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.961+0000 7fc2e5d9b640 1 -- 172.21.15.121:0/1365029666 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2f00a6790 msgr2=0x7fc2f01ca1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:03.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.961+0000 7fc2e5d9b640 1 --2- 172.21.15.121:0/1365029666 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2f00a6790 0x7fc2f01ca1a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc2d80098a0 tx=0x7fc2d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:03.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.961+0000 7f51a1ffb640 10 monclient: _check_auth_tickets 2024-09-17T18:19:03.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.961+0000 7f51a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:03.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.961+0000 7f51a1ffb640 1 -- 172.21.15.121:0/731052961 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f519c00cb50 con 0x7f51ac09e810 2024-09-17T18:19:03.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.961+0000 7f51a1ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:03.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.961+0000 7f2ce5ffb640 1 --2- 172.21.15.121:0/2538836062 >> v2:172.21.15.173:6812/2423786037 conn(0x7f2cd40401c0 0x7f2cd4042680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.961+0000 7f51a1ffb640 1 -- 172.21.15.121:0/731052961 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f519c013470 con 0x7f51ac09e810 2024-09-17T18:19:03.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.961+0000 7f51a1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:03.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.961+0000 7f51a1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:19:03.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.961+0000 7f51a1ffb640 10 monclient: dump: 2024-09-17T18:19:03.074 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:03.074 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.074 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.074 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.075 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:03.075 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.075 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:03.075 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:03.075 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:03.075 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.961+0000 7f51a37fe640 4 set_mon_vals no callback set 2024-09-17T18:19:03.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.961+0000 7f51b39d0640 5 monclient: authenticate success, global_id 4232 2024-09-17T18:19:03.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.961+0000 7f51b39d0640 10 monclient: _renew_subs 2024-09-17T18:19:03.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.961+0000 7f51b39d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:03.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.961+0000 7f51b39d0640 1 -- 172.21.15.121:0/731052961 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51ac071680 con 0x7f51ac09e810 2024-09-17T18:19:03.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.961+0000 7f51b39d0640 10 client.4232.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:03.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.961+0000 7f51b39d0640 10 monclient: _renew_subs 2024-09-17T18:19:03.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.961+0000 7f51b39d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:03.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.961+0000 7f51b39d0640 1 -- 172.21.15.121:0/731052961 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f51ac074d80 con 0x7f51ac09e810 2024-09-17T18:19:03.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.961+0000 7f51b39d0640 1 librados: init done 2024-09-17T18:19:03.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.962+0000 7f3b239f8640 10 monclient: build_initial_monmap 2024-09-17T18:19:03.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.962+0000 7f2ce5ffb640 1 -- 172.21.15.121:0/2538836062 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 3562+0+0 (secure 0 0 0) 0x7f2ce80747f0 con 0x7f2cf013dfd0 2024-09-17T18:19:03.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.962+0000 7fc2e77fe640 10 client.?.objecter ms_handle_connect 0x7fc2f00a6790 2024-09-17T18:19:03.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.962+0000 7fc2e77fe640 10 client.?.objecter resend_mon_ops 2024-09-17T18:19:03.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.962+0000 7f2ce5ffb640 10 client.4251.objecter ms_dispatch 0x7f2cf0000b90 osd_map(16..16 src has 1..16) 2024-09-17T18:19:03.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.962+0000 7f2ce5ffb640 3 client.4251.objecter handle_osd_map got epochs [16,16] > 0 2024-09-17T18:19:03.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.962+0000 7f2ce5ffb640 3 client.4251.objecter handle_osd_map decoding full epoch 16 2024-09-17T18:19:03.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.962+0000 7f3b239f8640 10 monclient: monmap: 2024-09-17T18:19:03.077 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:03.077 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:19:03.078 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:19:03.078 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:19:03.078 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:03.078 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.078 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:19:03.078 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:19:03.078 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:19:03.078 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.962+0000 7f2cf564c640 1 --2- 172.21.15.121:0/2538836062 >> v2:172.21.15.173:6812/2423786037 conn(0x7f2cd40401c0 0x7f2cd4042680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.962+0000 7f2cf564c640 10 monclient: get_auth_request con 0x7f2cd40401c0 auth_method 0 2024-09-17T18:19:03.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.962+0000 7f2ce5ffb640 10 client.4251.objecter checking command 1 2024-09-17T18:19:03.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.962+0000 7f3b239f8640 1 librados: starting msgr at 2024-09-17T18:19:03.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.962+0000 7f3b239f8640 1 librados: starting objecter 2024-09-17T18:19:03.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.962+0000 7f2ce5ffb640 1 --2- 172.21.15.121:0/2538836062 >> v2:172.21.15.121:6808/196419920 conn(0x7f2cd4045cd0 0x7f2cd4048110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.962+0000 7f3b239f8640 1 Processor -- start 2024-09-17T18:19:03.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.962+0000 7f2ce5ffb640 20 client.4251.objecter _get_session s=0x7f2cd4045620 osd=1 3 2024-09-17T18:19:03.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.962+0000 7f3b239f8640 1 -- start start 2024-09-17T18:19:03.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.962+0000 7f3b239f8640 1 librados: setting wanted keys 2024-09-17T18:19:03.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.962+0000 7f3b239f8640 1 librados: calling monclient init 2024-09-17T18:19:03.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.962+0000 7f3b239f8640 10 monclient: init 2024-09-17T18:19:03.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.962+0000 7f2ce5ffb640 20 client.4251.objecter put_session s=0x7f2cd4045620 osd=1 3 2024-09-17T18:19:03.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.962+0000 7f2ce5ffb640 15 client.4251.objecter _session_command_op_remove -1 1 2024-09-17T18:19:03.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.962+0000 7f2ce5ffb640 20 client.4251.objecter _get_session s=0x7f2cd4045620 osd=1 3 2024-09-17T18:19:03.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.962+0000 7f2ce5ffb640 20 client.4251.objecter get_session s=0x7f2cd4045620 osd=1 3 2024-09-17T18:19:03.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.962+0000 7f2ce5ffb640 15 client.4251.objecter _session_command_op_assign 1 1 2024-09-17T18:19:03.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.962+0000 7f2ce5ffb640 20 client.4251.objecter put_session s=0x7f2cd4045620 osd=1 4 2024-09-17T18:19:03.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.962+0000 7f2ce5ffb640 10 client.4251.objecter _send_command 1 2024-09-17T18:19:03.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.962+0000 7f2ce5ffb640 1 -- 172.21.15.121:0/2538836062 --> v2:172.21.15.121:6808/196419920 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2cd40487a0 con 0x7f2cd4045cd0 2024-09-17T18:19:03.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.962+0000 7f2ce5ffb640 20 client.4251.objecter dump_active .. 0 homeless 2024-09-17T18:19:03.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.962+0000 7f51a1ffb640 1 -- 172.21.15.121:0/731052961 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f519c01a070 con 0x7f51ac09e810 2024-09-17T18:19:03.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.962+0000 7f51b39d0640 10 client.4232.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:03.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.962+0000 7f51b39d0640 15 client.4232.objecter _session_command_op_assign -1 1 2024-09-17T18:19:03.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.962+0000 7f51b39d0640 20 client.4232.objecter _get_session osd=-1 returning homeless 2024-09-17T18:19:03.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.962+0000 7f51b39d0640 10 client.4232.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:03.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.962+0000 7f51b39d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:03.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.962+0000 7f51b39d0640 1 -- 172.21.15.121:0/731052961 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f51ac134e50 con 0x7f51ac09e810 2024-09-17T18:19:03.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.962+0000 7f2cf4e4b640 1 --2- 172.21.15.121:0/2538836062 >> v2:172.21.15.121:6808/196419920 conn(0x7f2cd4045cd0 0x7f2cd4048110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.962+0000 7f2cf4e4b640 10 monclient: get_auth_request con 0x7f2cd4045cd0 auth_method 0 2024-09-17T18:19:03.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f51a1ffb640 1 --2- 172.21.15.121:0/731052961 >> v2:172.21.15.173:6812/2423786037 conn(0x7f51900401c0 0x7f5190042680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f51a1ffb640 1 -- 172.21.15.121:0/731052961 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 3562+0+0 (secure 0 0 0) 0x7f519c053360 con 0x7f51ac09e810 2024-09-17T18:19:03.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f51a1ffb640 10 client.4232.objecter ms_dispatch 0x7f51ac000b90 osd_map(16..16 src has 1..16) 2024-09-17T18:19:03.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f51a1ffb640 3 client.4232.objecter handle_osd_map got epochs [16,16] > 0 2024-09-17T18:19:03.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f51a1ffb640 3 client.4232.objecter handle_osd_map decoding full epoch 16 2024-09-17T18:19:03.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f51a1ffb640 10 client.4232.objecter checking command 1 2024-09-17T18:19:03.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f51a1ffb640 1 --2- 172.21.15.121:0/731052961 >> v2:172.21.15.121:6804/2545987420 conn(0x7f5190045d50 0x7f5190048190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f51a1ffb640 20 client.4232.objecter _get_session s=0x7f51900456a0 osd=2 3 2024-09-17T18:19:03.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f51a1ffb640 20 client.4232.objecter put_session s=0x7f51900456a0 osd=2 3 2024-09-17T18:19:03.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f51a1ffb640 15 client.4232.objecter _session_command_op_remove -1 1 2024-09-17T18:19:03.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f51a1ffb640 20 client.4232.objecter _get_session s=0x7f51900456a0 osd=2 3 2024-09-17T18:19:03.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f51a1ffb640 20 client.4232.objecter get_session s=0x7f51900456a0 osd=2 3 2024-09-17T18:19:03.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f51a1ffb640 15 client.4232.objecter _session_command_op_assign 2 1 2024-09-17T18:19:03.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f51a1ffb640 20 client.4232.objecter put_session s=0x7f51900456a0 osd=2 4 2024-09-17T18:19:03.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f51a1ffb640 10 client.4232.objecter _send_command 1 2024-09-17T18:19:03.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f51a1ffb640 1 -- 172.21.15.121:0/731052961 --> v2:172.21.15.121:6804/2545987420 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f519c01de50 con 0x7f5190045d50 2024-09-17T18:19:03.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f51a1ffb640 20 client.4232.objecter dump_active .. 0 homeless 2024-09-17T18:19:03.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f51a1ffb640 1 -- 172.21.15.121:0/731052961 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f519c05a050 con 0x7f51ac09e810 2024-09-17T18:19:03.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f51a1ffb640 10 monclient: handle_get_version_reply finishing 1 version 16 2024-09-17T18:19:03.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f3b239f8640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:03.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f3b239f8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T18:19:03.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f51b1f6d640 1 --2- 172.21.15.121:0/731052961 >> v2:172.21.15.121:6804/2545987420 conn(0x7f5190045d50 0x7f5190048190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f51b1f6d640 10 monclient: get_auth_request con 0x7f5190045d50 auth_method 0 2024-09-17T18:19:03.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f3b239f8640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b1c0a5ca0 0x7f3b1c1cd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f3b239f8640 10 monclient(hunting): picked mon.noname-c con 0x7f3b1c0a5ca0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:03.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f3b239f8640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3b1c0a65c0 0x7f3b1c137f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f3b21794640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b1c0a5ca0 0x7f3b1c1cd8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f3b239f8640 10 monclient(hunting): picked mon.noname-a con 0x7f3b1c0a65c0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:03.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f3b21794640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b1c0a5ca0 0x7f3b1c1cd8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:59378/0 (socket says 172.21.15.121:59378) 2024-09-17T18:19:03.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f3b239f8640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b1c13cfc0 0x7f3b1c1384c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f3b239f8640 10 monclient(hunting): picked mon.noname-b con 0x7f3b1c13cfc0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:03.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f3b21794640 1 -- 172.21.15.121:0/1189801351 learned_addr learned my addr 172.21.15.121:0/1189801351 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:03.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f3b239f8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f3b239f8640 1 -- 172.21.15.121:0/1189801351 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3b1c0a5670 con 0x7f3b1c0a65c0 2024-09-17T18:19:03.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f3b239f8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f3b239f8640 1 -- 172.21.15.121:0/1189801351 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3b1c0a6eb0 con 0x7f3b1c13cfc0 2024-09-17T18:19:03.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f3b239f8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f3b239f8640 1 -- 172.21.15.121:0/1189801351 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3b1c0a69d0 con 0x7f3b1c0a5ca0 2024-09-17T18:19:03.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f3b239f8640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:03.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f3b20f93640 1 --2- 172.21.15.121:0/1189801351 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3b1c0a65c0 0x7f3b1c137f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f3b239f8640 10 monclient(hunting): authenticate will time out at 746.679777s 2024-09-17T18:19:03.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f3b21794640 10 monclient(hunting): get_auth_request con 0x7f3b1c0a5ca0 auth_method 0 2024-09-17T18:19:03.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f3b21794640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:03.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f3b21794640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:03.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f3b21794640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:03.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f3b21794640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:03.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f3b21794640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:03.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f3b21794640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:03.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f3b20f93640 10 monclient(hunting): get_auth_request con 0x7f3b1c0a65c0 auth_method 0 2024-09-17T18:19:03.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f3b20f93640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:03.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f3b20f93640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:03.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f3b20f93640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:03.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f2ce5ffb640 1 -- 172.21.15.121:0/2538836062 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f2ce807b050 con 0x7f2cf013dfd0 2024-09-17T18:19:03.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f2ce5ffb640 10 monclient: handle_get_version_reply finishing 1 version 16 2024-09-17T18:19:03.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.963+0000 7f3b21f95640 1 --2- 172.21.15.121:0/1189801351 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b1c13cfc0 0x7f3b1c1384c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7f3b21f95640 10 monclient(hunting): get_auth_request con 0x7f3b1c13cfc0 auth_method 0 2024-09-17T18:19:03.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7f3b21f95640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:03.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7f3b21f95640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:03.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7f3b21f95640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:03.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7f3b20f93640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:03.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7f3b20f93640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:03.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7fc2e77fe640 1 -- 172.21.15.121:0/1365029666 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d8041070 con 0x7fc2f00a6790 2024-09-17T18:19:03.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7fc2e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:03.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7fc2e77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:19:03.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7fc2e77fe640 10 monclient: dump: 2024-09-17T18:19:03.091 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:03.091 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.091 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.091 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.091 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:03.091 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.091 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:03.091 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:03.091 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:03.091 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7f3b20f93640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:03.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7fc2e77fe640 10 monclient: _finish_auth 0 2024-09-17T18:19:03.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7fc2e77fe640 10 monclient: _check_auth_tickets 2024-09-17T18:19:03.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7fc2e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:03.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7fc2e77fe640 1 -- 172.21.15.121:0/1365029666 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2d803c880 con 0x7fc2f00a6790 2024-09-17T18:19:03.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7fc2e77fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:03.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7fc2f4ad4640 5 monclient: authenticate success, global_id 4254 2024-09-17T18:19:03.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7fc2f4ad4640 10 monclient: _renew_subs 2024-09-17T18:19:03.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7fc2e77fe640 1 -- 172.21.15.121:0/1365029666 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d8037040 con 0x7fc2f00a6790 2024-09-17T18:19:03.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7fc2f4ad4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:03.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7fc2f4ad4640 1 -- 172.21.15.121:0/1365029666 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2f0124420 con 0x7fc2f00a6790 2024-09-17T18:19:03.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7f3b21794640 10 monclient(hunting): handle_auth_done global_id 4235 payload 824 2024-09-17T18:19:03.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7f3b21794640 1 -- 172.21.15.121:0/1189801351 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b1c13cfc0 msgr2=0x7f3b1c1384c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7f3b21794640 1 --2- 172.21.15.121:0/1189801351 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b1c13cfc0 0x7f3b1c1384c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7f3b21794640 1 -- 172.21.15.121:0/1189801351 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3b1c0a65c0 msgr2=0x7f3b1c137f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7f3b21794640 1 --2- 172.21.15.121:0/1189801351 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3b1c0a65c0 0x7f3b1c137f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7f3b21794640 10 monclient: _finish_hunting 0 2024-09-17T18:19:03.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7f3b21794640 1 monclient: found mon.noname-c 2024-09-17T18:19:03.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7f3b21794640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:03.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7fc2ed59a640 4 set_mon_vals no callback set 2024-09-17T18:19:03.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7f3b21794640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:19:03.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7f3b21794640 1 -- 172.21.15.121:0/1189801351 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b1c1235e0 con 0x7f3b1c0a5ca0 2024-09-17T18:19:03.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7fc2e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:03.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7fc2e77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:19:03.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7fc2e77fe640 10 monclient: dump: 2024-09-17T18:19:03.095 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:03.095 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.095 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.095 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.095 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:03.095 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.095 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:03.095 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:03.095 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:03.095 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7fc2f4ad4640 10 client.4254.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:03.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7fc2f4ad4640 10 monclient: _renew_subs 2024-09-17T18:19:03.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7fc2f4ad4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:03.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7fc2f4ad4640 1 -- 172.21.15.121:0/1365029666 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2f00a0790 con 0x7fc2f00a6790 2024-09-17T18:19:03.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7fc2f4ad4640 1 librados: init done 2024-09-17T18:19:03.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7f2ce5ffb640 10 client.4251.objecter ms_handle_connect 0x7f2cd40401c0 2024-09-17T18:19:03.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7f2cf564c640 1 --2- 172.21.15.121:0/2538836062 >> v2:172.21.15.173:6812/2423786037 conn(0x7f2cd40401c0 0x7f2cd4042680 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2cec007c90 tx=0x7f2cec007be0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:03.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7f51b176c640 1 --2- 172.21.15.121:0/731052961 >> v2:172.21.15.173:6812/2423786037 conn(0x7f51900401c0 0x7f5190042680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.964+0000 7f51b176c640 10 monclient: get_auth_request con 0x7f51900401c0 auth_method 0 2024-09-17T18:19:03.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.965+0000 7fc2e4d99640 10 client.4254.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:03.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.965+0000 7fc2e4d99640 15 client.4254.objecter _session_command_op_assign -1 1 2024-09-17T18:19:03.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.965+0000 7fc2e4d99640 20 client.4254.objecter _get_session osd=-1 returning homeless 2024-09-17T18:19:03.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.965+0000 7fc2e4d99640 10 client.4254.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:03.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.965+0000 7fc2e4d99640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:03.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.965+0000 7fc2e4d99640 1 -- 172.21.15.121:0/1365029666 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc2c8002910 con 0x7fc2f00a6790 2024-09-17T18:19:03.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.965+0000 7fc2e77fe640 1 -- 172.21.15.121:0/1365029666 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fc2d8044070 con 0x7fc2f00a6790 2024-09-17T18:19:03.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.965+0000 7fc2e77fe640 1 --2- 172.21.15.121:0/1365029666 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc2d0040170 0x7fc2d0042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.965+0000 7fc2e77fe640 1 -- 172.21.15.121:0/1365029666 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 3562+0+0 (secure 0 0 0) 0x7fc2d80738f0 con 0x7fc2f00a6790 2024-09-17T18:19:03.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.965+0000 7fc2e77fe640 10 client.4254.objecter ms_dispatch 0x7fc2f0000b90 osd_map(16..16 src has 1..16) 2024-09-17T18:19:03.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.965+0000 7fc2e77fe640 3 client.4254.objecter handle_osd_map got epochs [16,16] > 0 2024-09-17T18:19:03.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.965+0000 7fc2e77fe640 3 client.4254.objecter handle_osd_map decoding full epoch 16 2024-09-17T18:19:03.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.965+0000 7fc2e77fe640 10 client.4254.objecter checking command 1 2024-09-17T18:19:03.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.965+0000 7fc2e77fe640 1 --2- 172.21.15.121:0/1365029666 >> v2:172.21.15.121:6800/1526257129 conn(0x7fc2d0045c80 0x7fc2d00480c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.965+0000 7fc2e77fe640 20 client.4254.objecter _get_session s=0x7fc2d00455d0 osd=0 3 2024-09-17T18:19:03.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.965+0000 7fc2e77fe640 20 client.4254.objecter put_session s=0x7fc2d00455d0 osd=0 3 2024-09-17T18:19:03.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.965+0000 7fc2e77fe640 15 client.4254.objecter _session_command_op_remove -1 1 2024-09-17T18:19:03.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.965+0000 7fc2e77fe640 20 client.4254.objecter _get_session s=0x7fc2d00455d0 osd=0 3 2024-09-17T18:19:03.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.965+0000 7fc2e77fe640 20 client.4254.objecter get_session s=0x7fc2d00455d0 osd=0 3 2024-09-17T18:19:03.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.965+0000 7fc2e77fe640 15 client.4254.objecter _session_command_op_assign 0 1 2024-09-17T18:19:03.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.965+0000 7fc2e77fe640 20 client.4254.objecter put_session s=0x7fc2d00455d0 osd=0 4 2024-09-17T18:19:03.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.965+0000 7fc2e77fe640 10 client.4254.objecter _send_command 1 2024-09-17T18:19:03.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.965+0000 7fc2e77fe640 1 -- 172.21.15.121:0/1365029666 --> v2:172.21.15.121:6800/1526257129 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc2d8032640 con 0x7fc2d0045c80 2024-09-17T18:19:03.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.965+0000 7fc2e77fe640 20 client.4254.objecter dump_active .. 0 homeless 2024-09-17T18:19:03.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.965+0000 7fc2e77fe640 1 -- 172.21.15.121:0/1365029666 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7fc2d8078320 con 0x7fc2f00a6790 2024-09-17T18:19:03.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.965+0000 7fc2e77fe640 10 monclient: handle_get_version_reply finishing 1 version 16 2024-09-17T18:19:03.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.965+0000 7fc2eed9d640 1 --2- 172.21.15.121:0/1365029666 >> v2:172.21.15.121:6800/1526257129 conn(0x7fc2d0045c80 0x7fc2d00480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.965+0000 7fc2eed9d640 10 monclient: get_auth_request con 0x7fc2d0045c80 auth_method 0 2024-09-17T18:19:03.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.966+0000 7f3b21794640 1 -- 172.21.15.121:0/1189801351 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b1c0a5ca0 msgr2=0x7f3b1c1cd8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:03.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.966+0000 7f3b21794640 1 --2- 172.21.15.121:0/1189801351 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b1c0a5ca0 0x7f3b1c1cd8e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3b1000ea30 tx=0x7f3b1000eef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:03.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.966+0000 7f3b0dffb640 10 client.?.objecter ms_handle_connect 0x7f3b1c0a5ca0 2024-09-17T18:19:03.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.966+0000 7f3b0dffb640 10 client.?.objecter resend_mon_ops 2024-09-17T18:19:03.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.966+0000 7f3b0dffb640 1 -- 172.21.15.121:0/1189801351 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b10017070 con 0x7f3b1c0a5ca0 2024-09-17T18:19:03.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.966+0000 7f3b0dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:03.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.966+0000 7f3b0dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:19:03.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.966+0000 7f3b0dffb640 10 monclient: dump: 2024-09-17T18:19:03.102 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:03.102 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.102 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.102 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.102 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:03.102 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.102 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:03.102 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:03.102 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:03.102 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.966+0000 7f3b0dffb640 10 monclient: _finish_auth 0 2024-09-17T18:19:03.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.966+0000 7f3b0dffb640 10 monclient: _check_auth_tickets 2024-09-17T18:19:03.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.966+0000 7f3b0dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:03.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.966+0000 7f3b0dffb640 1 -- 172.21.15.121:0/1189801351 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b1000cc70 con 0x7f3b1c0a5ca0 2024-09-17T18:19:03.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.966+0000 7f3b0dffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:03.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.966+0000 7f3b0dffb640 1 -- 172.21.15.121:0/1189801351 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b10013470 con 0x7f3b1c0a5ca0 2024-09-17T18:19:03.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.966+0000 7f3b239f8640 5 monclient: authenticate success, global_id 4235 2024-09-17T18:19:03.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.966+0000 7f3b239f8640 10 monclient: _renew_subs 2024-09-17T18:19:03.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.966+0000 7f3b239f8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:03.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.966+0000 7f3b239f8640 1 -- 172.21.15.121:0/1189801351 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b1c0a2490 con 0x7f3b1c0a5ca0 2024-09-17T18:19:03.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.966+0000 7f3b0dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:03.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.966+0000 7f3b0dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:19:03.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.966+0000 7f3b0dffb640 10 monclient: dump: 2024-09-17T18:19:03.104 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:03.104 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.104 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.104 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.104 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:03.104 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.104 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:03.104 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:03.104 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:03.104 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.966+0000 7f3b0f7fe640 4 set_mon_vals no callback set 2024-09-17T18:19:03.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.966+0000 7f3b239f8640 10 client.4235.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:03.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.966+0000 7f3b239f8640 10 monclient: _renew_subs 2024-09-17T18:19:03.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.966+0000 7f3b239f8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:03.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.966+0000 7f3b239f8640 1 -- 172.21.15.121:0/1189801351 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b1c138a00 con 0x7f3b1c0a5ca0 2024-09-17T18:19:03.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.967+0000 7f3b239f8640 1 librados: init done 2024-09-17T18:19:03.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.967+0000 7f2cf4e4b640 1 --2- 172.21.15.121:0/2538836062 >> v2:172.21.15.121:6808/196419920 conn(0x7f2cd4045cd0 0x7f2cd4048110 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:03.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.967+0000 7f2ce5ffb640 10 client.4251.objecter ms_handle_connect 0x7f2cd4045cd0 2024-09-17T18:19:03.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.967+0000 7f51b176c640 1 --2- 172.21.15.121:0/731052961 >> v2:172.21.15.173:6812/2423786037 conn(0x7f51900401c0 0x7f5190042680 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f51a8009c40 tx=0x7f51a8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:03.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.967+0000 7f51a1ffb640 10 client.4232.objecter ms_handle_connect 0x7f51900401c0 2024-09-17T18:19:03.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.967+0000 7fc2ee59c640 1 --2- 172.21.15.121:0/1365029666 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc2d0040170 0x7fc2d0042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.967+0000 7fc2ee59c640 10 monclient: get_auth_request con 0x7fc2d0040170 auth_method 0 2024-09-17T18:19:03.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.967+0000 7f3b239f8640 10 client.4235.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:03.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.967+0000 7f3b239f8640 15 client.4235.objecter _session_command_op_assign -1 1 2024-09-17T18:19:03.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.967+0000 7f3b239f8640 20 client.4235.objecter _get_session osd=-1 returning homeless 2024-09-17T18:19:03.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.967+0000 7f3b239f8640 10 client.4235.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:03.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.967+0000 7f3b239f8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:03.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.967+0000 7f3b239f8640 1 -- 172.21.15.121:0/1189801351 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3b1c0a8720 con 0x7f3b1c0a5ca0 2024-09-17T18:19:03.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.968+0000 7f51a1ffb640 10 client.4232.objecter ms_handle_connect 0x7f5190045d50 2024-09-17T18:19:03.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.968+0000 7f51b1f6d640 1 --2- 172.21.15.121:0/731052961 >> v2:172.21.15.121:6804/2545987420 conn(0x7f5190045d50 0x7f5190048190 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:03.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.968+0000 7f3b0dffb640 1 -- 172.21.15.121:0/1189801351 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f3b1001a070 con 0x7f3b1c0a5ca0 2024-09-17T18:19:03.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.968+0000 7f2ce5ffb640 1 -- 172.21.15.121:0/2538836062 <== osd.1 v2:172.21.15.121:6808/196419920 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2cd40487a0 con 0x7f2cd4045cd0 2024-09-17T18:19:03.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.968+0000 7f2ce5ffb640 10 client.4251.objecter ms_dispatch 0x7f2cf0000b90 command_reply(tid 1: 0 ) 2024-09-17T18:19:03.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.968+0000 7f2ce5ffb640 10 client.4251.objecter _finish_command 1 = system:0 2024-09-17T18:19:03.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.968+0000 7f2ce5ffb640 20 client.4251.objecter put_session s=0x7f2cd4045620 osd=1 4 2024-09-17T18:19:03.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.968+0000 7f2ce5ffb640 15 client.4251.objecter _session_command_op_remove 1 1 2024-09-17T18:19:03.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.968+0000 7f3b0dffb640 1 --2- 172.21.15.121:0/1189801351 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3b000401c0 0x7f3b00042680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.968+0000 7f3b0dffb640 1 -- 172.21.15.121:0/1189801351 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 3562+0+0 (secure 0 0 0) 0x7f3b10052f70 con 0x7f3b1c0a5ca0 2024-09-17T18:19:03.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.968+0000 7f3b0dffb640 10 client.4235.objecter ms_dispatch 0x7f3b1c000b90 osd_map(16..16 src has 1..16) 2024-09-17T18:19:03.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.968+0000 7f3b0dffb640 3 client.4235.objecter handle_osd_map got epochs [16,16] > 0 2024-09-17T18:19:03.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.968+0000 7f3b0dffb640 3 client.4235.objecter handle_osd_map decoding full epoch 16 2024-09-17T18:19:03.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.968+0000 7f3b0dffb640 10 client.4235.objecter checking command 1 2024-09-17T18:19:03.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.968+0000 7f3b0dffb640 1 --2- 172.21.15.121:0/1189801351 >> v2:172.21.15.173:6804/1961801317 conn(0x7f3b00045d50 0x7f3b00048190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.968+0000 7f3b0dffb640 20 client.4235.objecter _get_session s=0x7f3b000456a0 osd=4 3 2024-09-17T18:19:03.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.968+0000 7f3b0dffb640 20 client.4235.objecter put_session s=0x7f3b000456a0 osd=4 3 2024-09-17T18:19:03.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.968+0000 7f3b0dffb640 15 client.4235.objecter _session_command_op_remove -1 1 2024-09-17T18:19:03.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.968+0000 7f3b0dffb640 20 client.4235.objecter _get_session s=0x7f3b000456a0 osd=4 3 2024-09-17T18:19:03.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.968+0000 7f3b0dffb640 20 client.4235.objecter get_session s=0x7f3b000456a0 osd=4 3 2024-09-17T18:19:03.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.968+0000 7f3b0dffb640 15 client.4235.objecter _session_command_op_assign 4 1 2024-09-17T18:19:03.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.968+0000 7f3b0dffb640 20 client.4235.objecter put_session s=0x7f3b000456a0 osd=4 4 2024-09-17T18:19:03.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.968+0000 7f3b0dffb640 10 client.4235.objecter _send_command 1 2024-09-17T18:19:03.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.968+0000 7f3b0dffb640 1 -- 172.21.15.121:0/1189801351 --> v2:172.21.15.173:6804/1961801317 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3b10008eb0 con 0x7f3b00045d50 2024-09-17T18:19:03.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.968+0000 7f3b0dffb640 20 client.4235.objecter dump_active .. 0 homeless 2024-09-17T18:19:03.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.968+0000 7f3b0dffb640 1 -- 172.21.15.121:0/1189801351 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f3b10055540 con 0x7f3b1c0a5ca0 2024-09-17T18:19:03.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.968+0000 7f3b0dffb640 10 monclient: handle_get_version_reply finishing 1 version 16 2024-09-17T18:19:03.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.968+0000 7f51a1ffb640 1 -- 172.21.15.121:0/731052961 <== osd.2 v2:172.21.15.121:6804/2545987420 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f519c01de50 con 0x7f5190045d50 2024-09-17T18:19:03.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.968+0000 7f51a1ffb640 10 client.4232.objecter ms_dispatch 0x7f51ac000b90 command_reply(tid 1: 0 ) 2024-09-17T18:19:03.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.968+0000 7f51a1ffb640 10 client.4232.objecter _finish_command 1 = system:0 2024-09-17T18:19:03.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.968+0000 7f51a1ffb640 20 client.4232.objecter put_session s=0x7f51900456a0 osd=2 4 2024-09-17T18:19:03.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.968+0000 7f51a1ffb640 15 client.4232.objecter _session_command_op_remove 2 1 2024-09-17T18:19:03.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.968+0000 7f3b21f95640 1 --2- 172.21.15.121:0/1189801351 >> v2:172.21.15.173:6804/1961801317 conn(0x7f3b00045d50 0x7f3b00048190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.968+0000 7f3b21f95640 10 monclient: get_auth_request con 0x7f3b00045d50 auth_method 0 2024-09-17T18:19:03.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.970+0000 7fc2ee59c640 1 --2- 172.21.15.121:0/1365029666 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc2d0040170 0x7fc2d0042630 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc2e00017a0 tx=0x7fc2e0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:03.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.970+0000 7fc2e77fe640 10 client.4254.objecter ms_handle_connect 0x7fc2d0040170 2024-09-17T18:19:03.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.970+0000 7fc2eed9d640 1 --2- 172.21.15.121:0/1365029666 >> v2:172.21.15.121:6800/1526257129 conn(0x7fc2d0045c80 0x7fc2d00480c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:03.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.970+0000 7fc2e77fe640 10 client.4254.objecter ms_handle_connect 0x7fc2d0045c80 2024-09-17T18:19:03.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.970+0000 7fc2e77fe640 1 -- 172.21.15.121:0/1365029666 <== osd.0 v2:172.21.15.121:6800/1526257129 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc2d8032640 con 0x7fc2d0045c80 2024-09-17T18:19:03.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.970+0000 7fc2e77fe640 10 client.4254.objecter ms_dispatch 0x7fc2f0000b90 command_reply(tid 1: 0 ) 2024-09-17T18:19:03.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.970+0000 7fc2e77fe640 10 client.4254.objecter _finish_command 1 = system:0 2024-09-17T18:19:03.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.970+0000 7fc2e77fe640 20 client.4254.objecter put_session s=0x7fc2d00455d0 osd=0 4 2024-09-17T18:19:03.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.970+0000 7fc2e77fe640 15 client.4254.objecter _session_command_op_remove 0 1 2024-09-17T18:19:03.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.973+0000 7f3b21f95640 1 --2- 172.21.15.121:0/1189801351 >> v2:172.21.15.173:6804/1961801317 conn(0x7f3b00045d50 0x7f3b00048190 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:03.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.973+0000 7f3b0dffb640 10 client.4235.objecter ms_handle_connect 0x7f3b00045d50 2024-09-17T18:19:03.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.974+0000 7f3b0dffb640 1 -- 172.21.15.121:0/1189801351 <== osd.4 v2:172.21.15.173:6804/1961801317 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3b10008eb0 con 0x7f3b00045d50 2024-09-17T18:19:03.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.974+0000 7f3b0dffb640 10 client.4235.objecter ms_dispatch 0x7f3b1c000b90 command_reply(tid 1: 0 ) 2024-09-17T18:19:03.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.974+0000 7f3b0dffb640 10 client.4235.objecter _finish_command 1 = system:0 2024-09-17T18:19:03.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.974+0000 7f3b0dffb640 20 client.4235.objecter put_session s=0x7f3b000456a0 osd=4 4 2024-09-17T18:19:03.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.974+0000 7f3b0dffb640 15 client.4235.objecter _session_command_op_remove 4 1 2024-09-17T18:19:03.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.976+0000 7f3b20f93640 1 --2- 172.21.15.121:0/1189801351 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3b000401c0 0x7f3b00042680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.976+0000 7f3b20f93640 10 monclient: get_auth_request con 0x7f3b000401c0 auth_method 0 2024-09-17T18:19:03.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.979+0000 7f3b0dffb640 10 client.4235.objecter ms_handle_connect 0x7f3b000401c0 2024-09-17T18:19:03.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.979+0000 7f3b20f93640 1 --2- 172.21.15.121:0/1189801351 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3b000401c0 0x7f3b00042680 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c0a3d20 tx=0x7f3b1802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:03.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.982+0000 7f2cf78b0640 10 client.4251.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-17T18:19:03.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.982+0000 7f2cf78b0640 15 client.4251.objecter _session_command_op_assign -1 2 2024-09-17T18:19:03.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.982+0000 7f2cf78b0640 20 client.4251.objecter _get_session s=0x7f2cd4045620 osd=1 3 2024-09-17T18:19:03.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.982+0000 7f2cf78b0640 20 client.4251.objecter put_session s=0x7f2cd4045620 osd=1 3 2024-09-17T18:19:03.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.982+0000 7f2cf78b0640 20 client.4251.objecter _get_session s=0x7f2cd4045620 osd=1 3 2024-09-17T18:19:03.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.982+0000 7f2cf78b0640 15 client.4251.objecter _session_command_op_remove -1 2 2024-09-17T18:19:03.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.982+0000 7f2cf78b0640 20 client.4251.objecter get_session s=0x7f2cd4045620 osd=1 3 2024-09-17T18:19:03.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.982+0000 7f2cf78b0640 15 client.4251.objecter _session_command_op_assign 1 2 2024-09-17T18:19:03.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.982+0000 7f2cf78b0640 20 client.4251.objecter put_session s=0x7f2cd4045620 osd=1 4 2024-09-17T18:19:03.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.982+0000 7f2cf78b0640 10 client.4251.objecter _send_command 2 2024-09-17T18:19:03.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.982+0000 7f2cf78b0640 1 -- 172.21.15.121:0/2538836062 --> v2:172.21.15.121:6808/196419920 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2cf0124040 con 0x7f2cd4045cd0 2024-09-17T18:19:03.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.982+0000 7f2ce5ffb640 1 -- 172.21.15.121:0/2538836062 <== osd.1 v2:172.21.15.121:6808/196419920 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f2cf0124040 con 0x7f2cd4045cd0 2024-09-17T18:19:03.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.982+0000 7f2ce5ffb640 10 client.4251.objecter ms_dispatch 0x7f2cf0000b90 command_reply(tid 2: 0 ) 2024-09-17T18:19:03.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.982+0000 7f2ce5ffb640 10 client.4251.objecter _finish_command 2 = system:0 2024-09-17T18:19:03.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.982+0000 7f2ce5ffb640 20 client.4251.objecter put_session s=0x7f2cd4045620 osd=1 4 2024-09-17T18:19:03.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.982+0000 7f51b39d0640 10 client.4232.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-17T18:19:03.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.982+0000 7f51b39d0640 15 client.4232.objecter _session_command_op_assign -1 2 2024-09-17T18:19:03.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.982+0000 7f51b39d0640 20 client.4232.objecter _get_session s=0x7f51900456a0 osd=2 3 2024-09-17T18:19:03.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.982+0000 7f51b39d0640 20 client.4232.objecter put_session s=0x7f51900456a0 osd=2 3 2024-09-17T18:19:03.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.982+0000 7f51b39d0640 20 client.4232.objecter _get_session s=0x7f51900456a0 osd=2 3 2024-09-17T18:19:03.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.982+0000 7f2ce5ffb640 15 client.4251.objecter _session_command_op_remove 1 2 2024-09-17T18:19:03.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.982+0000 7f51b39d0640 15 client.4232.objecter _session_command_op_remove -1 2 2024-09-17T18:19:03.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.982+0000 7f51b39d0640 20 client.4232.objecter get_session s=0x7f51900456a0 osd=2 3 2024-09-17T18:19:03.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.982+0000 7f51b39d0640 15 client.4232.objecter _session_command_op_assign 2 2 2024-09-17T18:19:03.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.982+0000 7f51b39d0640 20 client.4232.objecter put_session s=0x7f51900456a0 osd=2 4 2024-09-17T18:19:03.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.982+0000 7f51b39d0640 10 client.4232.objecter _send_command 2 2024-09-17T18:19:03.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.982+0000 7f51b39d0640 1 -- 172.21.15.121:0/731052961 --> v2:172.21.15.121:6804/2545987420 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f51ac1c9f40 con 0x7f5190045d50 2024-09-17T18:19:03.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.982+0000 7f51a1ffb640 1 -- 172.21.15.121:0/731052961 <== osd.2 v2:172.21.15.121:6804/2545987420 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f51ac1c9f40 con 0x7f5190045d50 2024-09-17T18:19:03.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.982+0000 7f51a1ffb640 10 client.4232.objecter ms_dispatch 0x7f51ac000b90 command_reply(tid 2: 0 ) 2024-09-17T18:19:03.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.982+0000 7f2cf78b0640 10 librados: watch_flush enter 2024-09-17T18:19:03.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.982+0000 7f51a1ffb640 10 client.4232.objecter _finish_command 2 = system:0 2024-09-17T18:19:03.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.982+0000 7f51a1ffb640 20 client.4232.objecter put_session s=0x7f51900456a0 osd=2 4 2024-09-17T18:19:03.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.982+0000 7f51a1ffb640 15 client.4232.objecter _session_command_op_remove 2 2 2024-09-17T18:19:03.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.982+0000 7f2cf78b0640 10 librados: watch_flush exit 2024-09-17T18:19:03.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.982+0000 7f2cf78b0640 10 client.4251.objecter close_session for osd.1 2024-09-17T18:19:03.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.982+0000 7f2cf78b0640 1 -- 172.21.15.121:0/2538836062 >> v2:172.21.15.121:6808/196419920 conn(0x7f2cd4045cd0 msgr2=0x7f2cd4048110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.982+0000 7f2cf78b0640 1 --2- 172.21.15.121:0/2538836062 >> v2:172.21.15.121:6808/196419920 conn(0x7f2cd4045cd0 0x7f2cd4048110 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.982+0000 7f2cf78b0640 20 client.4251.objecter put_session s=0x7f2cd4045620 osd=1 1 2024-09-17T18:19:03.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.982+0000 7f2cf78b0640 20 client.4251.objecter shutdown clearing up homeless session... 2024-09-17T18:19:03.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.982+0000 7f2cf78b0640 10 client.4251.objecter successfully canceled tick 2024-09-17T18:19:03.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.983+0000 7f51b39d0640 10 librados: watch_flush enter 2024-09-17T18:19:03.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.983+0000 7f51b39d0640 10 librados: watch_flush exit 2024-09-17T18:19:03.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.983+0000 7f51b39d0640 10 client.4232.objecter close_session for osd.2 2024-09-17T18:19:03.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.983+0000 7f51b39d0640 1 -- 172.21.15.121:0/731052961 >> v2:172.21.15.121:6804/2545987420 conn(0x7f5190045d50 msgr2=0x7f5190048190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.983+0000 7f51b39d0640 1 --2- 172.21.15.121:0/731052961 >> v2:172.21.15.121:6804/2545987420 conn(0x7f5190045d50 0x7f5190048190 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.983+0000 7f51b39d0640 20 client.4232.objecter put_session s=0x7f51900456a0 osd=2 1 2024-09-17T18:19:03.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.983+0000 7f51b39d0640 20 client.4232.objecter shutdown clearing up homeless session... 2024-09-17T18:19:03.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.983+0000 7f51b39d0640 10 client.4232.objecter successfully canceled tick 2024-09-17T18:19:03.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.983+0000 7f2cf78b0640 1 -- 172.21.15.121:0/2538836062 >> v2:172.21.15.173:6812/2423786037 conn(0x7f2cd40401c0 msgr2=0x7f2cd4042680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.983+0000 7f2cf78b0640 1 --2- 172.21.15.121:0/2538836062 >> v2:172.21.15.173:6812/2423786037 conn(0x7f2cd40401c0 0x7f2cd4042680 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2cec007c90 tx=0x7f2cec007be0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.983+0000 7f2cf78b0640 10 monclient: shutdown 2024-09-17T18:19:03.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.983+0000 7f2cf78b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:03.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.983+0000 7f2cf78b0640 1 -- 172.21.15.121:0/2538836062 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2cf013dfd0 msgr2=0x7f2cf0138470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.983+0000 7f2cf78b0640 1 --2- 172.21.15.121:0/2538836062 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2cf013dfd0 0x7f2cf0138470 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2ce8034000 tx=0x7f2ce8004a40 comp rx=0 tx=0).stop 2024-09-17T18:19:03.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.983+0000 7f2cf5e4d640 1 -- 172.21.15.121:0/2538836062 reap_dead start 2024-09-17T18:19:03.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.983+0000 7f2cf78b0640 1 -- 172.21.15.121:0/2538836062 shutdown_connections 2024-09-17T18:19:03.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.983+0000 7f2cf78b0640 1 -- 172.21.15.121:0/2538836062 >> 172.21.15.121:0/2538836062 conn(0x7f2cf006dd10 msgr2=0x7f2cf013ac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:03.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.983+0000 7f51b39d0640 1 -- 172.21.15.121:0/731052961 >> v2:172.21.15.173:6812/2423786037 conn(0x7f51900401c0 msgr2=0x7f5190042680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.983+0000 7f51b39d0640 1 --2- 172.21.15.121:0/731052961 >> v2:172.21.15.173:6812/2423786037 conn(0x7f51900401c0 0x7f5190042680 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f51a8009c40 tx=0x7f51a8009190 comp rx=0 tx=0).stop 2024-09-17T18:19:03.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.983+0000 7f51b39d0640 10 monclient: shutdown 2024-09-17T18:19:03.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.983+0000 7f51b39d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:03.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.983+0000 7f51b39d0640 1 -- 172.21.15.121:0/731052961 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f51ac09e810 msgr2=0x7f51ac1c9a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.983+0000 7f51b39d0640 1 --2- 172.21.15.121:0/731052961 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f51ac09e810 0x7f51ac1c9a00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f519c00e850 tx=0x7f519c00ed10 comp rx=0 tx=0).stop 2024-09-17T18:19:03.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.983+0000 7f51b1f6d640 1 -- 172.21.15.121:0/731052961 reap_dead start 2024-09-17T18:19:03.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.983+0000 7f2cf78b0640 1 -- 172.21.15.121:0/2538836062 shutdown_connections 2024-09-17T18:19:03.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.983+0000 7f51b39d0640 1 -- 172.21.15.121:0/731052961 shutdown_connections 2024-09-17T18:19:03.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.983+0000 7f2cf78b0640 1 -- 172.21.15.121:0/2538836062 wait complete. 2024-09-17T18:19:03.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.983+0000 7f51b39d0640 1 -- 172.21.15.121:0/731052961 >> 172.21.15.121:0/731052961 conn(0x7f51ac0ab830 msgr2=0x7f51ac133260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:03.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.983+0000 7f2cf78b0640 1 librados: shutdown 2024-09-17T18:19:03.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.983+0000 7f51b39d0640 1 -- 172.21.15.121:0/731052961 shutdown_connections 2024-09-17T18:19:03.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.983+0000 7f51b39d0640 1 -- 172.21.15.121:0/731052961 wait complete. 2024-09-17T18:19:03.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.983+0000 7f51b39d0640 1 librados: shutdown 2024-09-17T18:19:03.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.985+0000 7fc2e4d99640 10 client.4254.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-17T18:19:03.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.985+0000 7fc2e4d99640 15 client.4254.objecter _session_command_op_assign -1 2 2024-09-17T18:19:03.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.985+0000 7fc2e4d99640 20 client.4254.objecter _get_session s=0x7fc2d00455d0 osd=0 3 2024-09-17T18:19:03.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.985+0000 7fc2e4d99640 20 client.4254.objecter put_session s=0x7fc2d00455d0 osd=0 3 2024-09-17T18:19:03.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.985+0000 7fc2e4d99640 20 client.4254.objecter _get_session s=0x7fc2d00455d0 osd=0 3 2024-09-17T18:19:03.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.985+0000 7fc2e4d99640 15 client.4254.objecter _session_command_op_remove -1 2 2024-09-17T18:19:03.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.985+0000 7fc2e4d99640 20 client.4254.objecter get_session s=0x7fc2d00455d0 osd=0 3 2024-09-17T18:19:03.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.985+0000 7fc2e4d99640 15 client.4254.objecter _session_command_op_assign 0 2 2024-09-17T18:19:03.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.985+0000 7fc2e4d99640 20 client.4254.objecter put_session s=0x7fc2d00455d0 osd=0 4 2024-09-17T18:19:03.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.985+0000 7fc2e4d99640 10 client.4254.objecter _send_command 2 2024-09-17T18:19:03.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.985+0000 7fc2e4d99640 1 -- 172.21.15.121:0/1365029666 --> v2:172.21.15.121:6800/1526257129 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc2c8001220 con 0x7fc2d0045c80 2024-09-17T18:19:03.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.985+0000 7fc2e77fe640 1 -- 172.21.15.121:0/1365029666 <== osd.0 v2:172.21.15.121:6800/1526257129 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fc2c8001220 con 0x7fc2d0045c80 2024-09-17T18:19:03.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.985+0000 7fc2e77fe640 10 client.4254.objecter ms_dispatch 0x7fc2f0000b90 command_reply(tid 2: 0 ) 2024-09-17T18:19:03.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.985+0000 7fc2e77fe640 10 client.4254.objecter _finish_command 2 = system:0 2024-09-17T18:19:03.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.985+0000 7fc2e77fe640 20 client.4254.objecter put_session s=0x7fc2d00455d0 osd=0 4 2024-09-17T18:19:03.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.985+0000 7fc2e77fe640 15 client.4254.objecter _session_command_op_remove 0 2 2024-09-17T18:19:03.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.986+0000 7fc2e4d99640 10 librados: watch_flush enter 2024-09-17T18:19:03.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.986+0000 7fc2e4d99640 10 librados: watch_flush exit 2024-09-17T18:19:03.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.986+0000 7fc2e4d99640 10 client.4254.objecter close_session for osd.0 2024-09-17T18:19:03.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.986+0000 7fc2e4d99640 1 -- 172.21.15.121:0/1365029666 >> v2:172.21.15.121:6800/1526257129 conn(0x7fc2d0045c80 msgr2=0x7fc2d00480c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.986+0000 7fc2e4d99640 1 --2- 172.21.15.121:0/1365029666 >> v2:172.21.15.121:6800/1526257129 conn(0x7fc2d0045c80 0x7fc2d00480c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.986+0000 7fc2e4d99640 20 client.4254.objecter put_session s=0x7fc2d00455d0 osd=0 1 2024-09-17T18:19:03.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.986+0000 7fc2e4d99640 20 client.4254.objecter shutdown clearing up homeless session... 2024-09-17T18:19:03.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.986+0000 7fc2e4d99640 10 client.4254.objecter successfully canceled tick 2024-09-17T18:19:03.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.986+0000 7fc2e4d99640 1 -- 172.21.15.121:0/1365029666 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc2d0040170 msgr2=0x7fc2d0042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.986+0000 7fc2e4d99640 1 --2- 172.21.15.121:0/1365029666 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc2d0040170 0x7fc2d0042630 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc2e00017a0 tx=0x7fc2e0009040 comp rx=0 tx=0).stop 2024-09-17T18:19:03.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.986+0000 7fc2e4d99640 10 monclient: shutdown 2024-09-17T18:19:03.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.986+0000 7fc2e4d99640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:03.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.986+0000 7fc2e4d99640 1 -- 172.21.15.121:0/1365029666 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2f00a6790 msgr2=0x7fc2f01ca1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.986+0000 7fc2e4d99640 1 --2- 172.21.15.121:0/1365029666 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2f00a6790 0x7fc2f01ca1a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc2d80098a0 tx=0x7fc2d80046c0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.986+0000 7fc2eed9d640 1 -- 172.21.15.121:0/1365029666 reap_dead start 2024-09-17T18:19:03.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.986+0000 7fc2e4d99640 1 -- 172.21.15.121:0/1365029666 shutdown_connections 2024-09-17T18:19:03.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.986+0000 7fc2e4d99640 1 -- 172.21.15.121:0/1365029666 >> 172.21.15.121:0/1365029666 conn(0x7fc2f006dd10 msgr2=0x7fc2f013bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:03.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.986+0000 7fc2e4d99640 1 -- 172.21.15.121:0/1365029666 shutdown_connections 2024-09-17T18:19:03.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.986+0000 7fc2e4d99640 1 -- 172.21.15.121:0/1365029666 wait complete. 2024-09-17T18:19:03.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.987+0000 7fc2e4d99640 1 librados: shutdown 2024-09-17T18:19:03.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.989+0000 7f3b239f8640 10 client.4235.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-17T18:19:03.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.989+0000 7f3b239f8640 15 client.4235.objecter _session_command_op_assign -1 2 2024-09-17T18:19:03.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.989+0000 7f3b239f8640 20 client.4235.objecter _get_session s=0x7f3b000456a0 osd=4 3 2024-09-17T18:19:03.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.989+0000 7f3b239f8640 20 client.4235.objecter put_session s=0x7f3b000456a0 osd=4 3 2024-09-17T18:19:03.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.989+0000 7f3b239f8640 20 client.4235.objecter _get_session s=0x7f3b000456a0 osd=4 3 2024-09-17T18:19:03.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.989+0000 7f3b239f8640 15 client.4235.objecter _session_command_op_remove -1 2 2024-09-17T18:19:03.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.989+0000 7f3b239f8640 20 client.4235.objecter get_session s=0x7f3b000456a0 osd=4 3 2024-09-17T18:19:03.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.989+0000 7f3b239f8640 15 client.4235.objecter _session_command_op_assign 4 2 2024-09-17T18:19:03.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.989+0000 7f3b239f8640 20 client.4235.objecter put_session s=0x7f3b000456a0 osd=4 4 2024-09-17T18:19:03.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.989+0000 7f3b239f8640 10 client.4235.objecter _send_command 2 2024-09-17T18:19:03.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.989+0000 7f3b239f8640 1 -- 172.21.15.121:0/1189801351 --> v2:172.21.15.173:6804/1961801317 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3b1c1257f0 con 0x7f3b00045d50 2024-09-17T18:19:03.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.989+0000 7f3b0dffb640 1 -- 172.21.15.121:0/1189801351 <== osd.4 v2:172.21.15.173:6804/1961801317 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3b140072e0 con 0x7f3b00045d50 2024-09-17T18:19:03.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.989+0000 7f3b0dffb640 10 client.4235.objecter ms_dispatch 0x7f3b1c000b90 command_reply(tid 2: 0 ) 2024-09-17T18:19:03.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.989+0000 7f3b0dffb640 10 client.4235.objecter _finish_command 2 = system:0 2024-09-17T18:19:03.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.989+0000 7f3b0dffb640 20 client.4235.objecter put_session s=0x7f3b000456a0 osd=4 4 2024-09-17T18:19:03.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.989+0000 7f3b0dffb640 15 client.4235.objecter _session_command_op_remove 4 2 2024-09-17T18:19:03.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.990+0000 7f3b239f8640 10 librados: watch_flush enter 2024-09-17T18:19:03.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.990+0000 7f3b239f8640 10 librados: watch_flush exit 2024-09-17T18:19:03.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.990+0000 7f3b239f8640 10 client.4235.objecter close_session for osd.4 2024-09-17T18:19:03.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.990+0000 7f3b239f8640 1 -- 172.21.15.121:0/1189801351 >> v2:172.21.15.173:6804/1961801317 conn(0x7f3b00045d50 msgr2=0x7f3b00048190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.990+0000 7f3b239f8640 1 --2- 172.21.15.121:0/1189801351 >> v2:172.21.15.173:6804/1961801317 conn(0x7f3b00045d50 0x7f3b00048190 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.990+0000 7f3b239f8640 20 client.4235.objecter put_session s=0x7f3b000456a0 osd=4 1 2024-09-17T18:19:03.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.990+0000 7f3b239f8640 20 client.4235.objecter shutdown clearing up homeless session... 2024-09-17T18:19:03.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.990+0000 7f3b239f8640 10 client.4235.objecter successfully canceled tick 2024-09-17T18:19:03.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.990+0000 7f3b239f8640 1 -- 172.21.15.121:0/1189801351 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3b000401c0 msgr2=0x7f3b00042680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.990+0000 7f3b239f8640 1 --2- 172.21.15.121:0/1189801351 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3b000401c0 0x7f3b00042680 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c0a3d20 tx=0x7f3b1802e000 comp rx=0 tx=0).stop 2024-09-17T18:19:03.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.990+0000 7f3b239f8640 10 monclient: shutdown 2024-09-17T18:19:03.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.990+0000 7f3b239f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:03.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.990+0000 7f3b239f8640 1 -- 172.21.15.121:0/1189801351 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b1c0a5ca0 msgr2=0x7f3b1c1cd8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.990+0000 7f3b239f8640 1 --2- 172.21.15.121:0/1189801351 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b1c0a5ca0 0x7f3b1c1cd8e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3b1000ea30 tx=0x7f3b1000eef0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.990+0000 7f3b21f95640 1 -- 172.21.15.121:0/1189801351 reap_dead start 2024-09-17T18:19:03.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.990+0000 7f3b239f8640 1 -- 172.21.15.121:0/1189801351 shutdown_connections 2024-09-17T18:19:03.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.990+0000 7f3b239f8640 1 -- 172.21.15.121:0/1189801351 >> 172.21.15.121:0/1189801351 conn(0x7f3b1c06dd10 msgr2=0x7f3b1c0a90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:03.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.990+0000 7f3b239f8640 1 -- 172.21.15.121:0/1189801351 shutdown_connections 2024-09-17T18:19:03.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.990+0000 7f3b239f8640 1 -- 172.21.15.121:0/1189801351 wait complete. 2024-09-17T18:19:03.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:02.991+0000 7f3b239f8640 1 librados: shutdown 2024-09-17T18:19:03.134 INFO:teuthology.orchestra.run.smithi121.stdout:51539607555 2024-09-17T18:19:03.134 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-17T18:19:03.135 INFO:teuthology.orchestra.run.smithi121.stdout:47244640260 2024-09-17T18:19:03.135 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-17T18:19:03.136 INFO:teuthology.orchestra.run.smithi121.stdout:47244640260 2024-09-17T18:19:03.136 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-17T18:19:03.137 INFO:teuthology.orchestra.run.smithi121.stdout:51539607556 2024-09-17T18:19:03.137 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-17T18:19:03.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.125+0000 7f009bfff640 1 --2- 172.21.15.121:0/4210506051 >> v2:172.21.15.173:6800/3493964597 conn(0x7f0088045d80 0x7f00880481c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.125+0000 7f009bfff640 10 monclient: get_auth_request con 0x7f0088045d80 auth_method 0 2024-09-17T18:19:03.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.130+0000 7f0083fff640 10 client.4214.objecter ms_handle_connect 0x7f0088045d80 2024-09-17T18:19:03.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.130+0000 7f009bfff640 1 --2- 172.21.15.121:0/4210506051 >> v2:172.21.15.173:6800/3493964597 conn(0x7f0088045d80 0x7f00880481c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:03.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.131+0000 7f0083fff640 1 -- 172.21.15.121:0/4210506051 <== osd.3 v2:172.21.15.173:6800/3493964597 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0090013ec0 con 0x7f0088045d80 2024-09-17T18:19:03.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.131+0000 7f0083fff640 10 client.4214.objecter ms_dispatch 0x7f009c000b90 command_reply(tid 1: 0 ) 2024-09-17T18:19:03.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.131+0000 7f0083fff640 10 client.4214.objecter _finish_command 1 = system:0 2024-09-17T18:19:03.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.131+0000 7f0083fff640 20 client.4214.objecter put_session s=0x7f00880456d0 osd=3 4 2024-09-17T18:19:03.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.131+0000 7f0083fff640 15 client.4214.objecter _session_command_op_remove 3 1 2024-09-17T18:19:03.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.144+0000 7f00a1ccd640 10 client.4214.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-17T18:19:03.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.144+0000 7f00a1ccd640 15 client.4214.objecter _session_command_op_assign -1 2 2024-09-17T18:19:03.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.144+0000 7f00a1ccd640 20 client.4214.objecter _get_session s=0x7f00880456d0 osd=3 3 2024-09-17T18:19:03.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.144+0000 7f00a1ccd640 20 client.4214.objecter put_session s=0x7f00880456d0 osd=3 3 2024-09-17T18:19:03.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.144+0000 7f00a1ccd640 20 client.4214.objecter _get_session s=0x7f00880456d0 osd=3 3 2024-09-17T18:19:03.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.144+0000 7f00a1ccd640 15 client.4214.objecter _session_command_op_remove -1 2 2024-09-17T18:19:03.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.144+0000 7f00a1ccd640 20 client.4214.objecter get_session s=0x7f00880456d0 osd=3 3 2024-09-17T18:19:03.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.144+0000 7f00a1ccd640 15 client.4214.objecter _session_command_op_assign 3 2 2024-09-17T18:19:03.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.144+0000 7f00a1ccd640 20 client.4214.objecter put_session s=0x7f00880456d0 osd=3 4 2024-09-17T18:19:03.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.144+0000 7f00a1ccd640 10 client.4214.objecter _send_command 2 2024-09-17T18:19:03.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.144+0000 7f00a1ccd640 1 -- 172.21.15.121:0/4210506051 --> v2:172.21.15.173:6800/3493964597 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f009c09fba0 con 0x7f0088045d80 2024-09-17T18:19:03.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.145+0000 7f0083fff640 1 -- 172.21.15.121:0/4210506051 <== osd.3 v2:172.21.15.173:6800/3493964597 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f009c09fba0 con 0x7f0088045d80 2024-09-17T18:19:03.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.145+0000 7f0083fff640 10 client.4214.objecter ms_dispatch 0x7f009c000b90 command_reply(tid 2: 0 ) 2024-09-17T18:19:03.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.145+0000 7f0083fff640 10 client.4214.objecter _finish_command 2 = system:0 2024-09-17T18:19:03.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.145+0000 7f0083fff640 20 client.4214.objecter put_session s=0x7f00880456d0 osd=3 4 2024-09-17T18:19:03.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.145+0000 7f0083fff640 15 client.4214.objecter _session_command_op_remove 3 2 2024-09-17T18:19:03.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.145+0000 7f00a1ccd640 10 librados: watch_flush enter 2024-09-17T18:19:03.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.145+0000 7f00a1ccd640 10 librados: watch_flush exit 2024-09-17T18:19:03.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.145+0000 7f00a1ccd640 10 client.4214.objecter close_session for osd.3 2024-09-17T18:19:03.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.145+0000 7f00a1ccd640 1 -- 172.21.15.121:0/4210506051 >> v2:172.21.15.173:6800/3493964597 conn(0x7f0088045d80 msgr2=0x7f00880481c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.145+0000 7f00a1ccd640 1 --2- 172.21.15.121:0/4210506051 >> v2:172.21.15.173:6800/3493964597 conn(0x7f0088045d80 0x7f00880481c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.145+0000 7f00a1ccd640 20 client.4214.objecter put_session s=0x7f00880456d0 osd=3 1 2024-09-17T18:19:03.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.145+0000 7f00a1ccd640 20 client.4214.objecter shutdown clearing up homeless session... 2024-09-17T18:19:03.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.145+0000 7f00a1ccd640 10 client.4214.objecter successfully canceled tick 2024-09-17T18:19:03.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.145+0000 7f00a1ccd640 1 -- 172.21.15.121:0/4210506051 >> v2:172.21.15.173:6812/2423786037 conn(0x7f00880401f0 msgr2=0x7f00880426b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.145+0000 7f00a1ccd640 1 --2- 172.21.15.121:0/4210506051 >> v2:172.21.15.173:6812/2423786037 conn(0x7f00880401f0 0x7f00880426b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f009c06eea0 tx=0x7f008c02e000 comp rx=0 tx=0).stop 2024-09-17T18:19:03.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.145+0000 7f00a1ccd640 10 monclient: shutdown 2024-09-17T18:19:03.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.145+0000 7f00a1ccd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:03.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.145+0000 7f00a1ccd640 1 -- 172.21.15.121:0/4210506051 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f009c0a64b0 msgr2=0x7f009c1ce120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.145+0000 7f00a1ccd640 1 --2- 172.21.15.121:0/4210506051 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f009c0a64b0 0x7f009c1ce120 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f009000c430 tx=0x7f009000c8f0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.145+0000 7f009bfff640 1 -- 172.21.15.121:0/4210506051 reap_dead start 2024-09-17T18:19:03.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.145+0000 7f00a1ccd640 1 -- 172.21.15.121:0/4210506051 shutdown_connections 2024-09-17T18:19:03.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.145+0000 7f00a1ccd640 1 -- 172.21.15.121:0/4210506051 >> 172.21.15.121:0/4210506051 conn(0x7f009c0ab880 msgr2=0x7f009c09fdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:03.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.146+0000 7f00a1ccd640 1 -- 172.21.15.121:0/4210506051 shutdown_connections 2024-09-17T18:19:03.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.146+0000 7f00a1ccd640 1 -- 172.21.15.121:0/4210506051 wait complete. 2024-09-17T18:19:03.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.146+0000 7f00a1ccd640 1 librados: shutdown 2024-09-17T18:19:03.161 INFO:teuthology.orchestra.run.smithi121.stdout:47244640260 2024-09-17T18:19:03.161 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-17T18:19:03.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.258+0000 7f37cb0b0640 10 monclient: get_monmap_and_config 2024-09-17T18:19:03.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.258+0000 7f37cb0b0640 10 monclient: build_initial_monmap 2024-09-17T18:19:03.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.258+0000 7f37cb0b0640 10 monclient: monmap: 2024-09-17T18:19:03.261 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:03.261 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.261 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:19:03.259536+0000 2024-09-17T18:19:03.261 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:19:03.259536+0000 2024-09-17T18:19:03.261 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:03.261 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.261 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:19:03.261 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:19:03.261 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:19:03.262 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:19:03.262 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:19:03.262 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.259+0000 7f37cb0b0640 1 Processor -- start 2024-09-17T18:19:03.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.259+0000 7f37cb0b0640 1 -- start start 2024-09-17T18:19:03.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.259+0000 7f37cb0b0640 10 monclient: init 2024-09-17T18:19:03.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.260+0000 7f37cb0b0640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:03.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.260+0000 7f37cb0b0640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-17T18:19:03.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.260+0000 7f37cb0b0640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f37c40700f0 0x7f37c4134e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.260+0000 7f37cb0b0640 10 monclient(hunting): picked mon.noname-c con 0x7f37c40700f0 addr v2:172.21.15.173:3300/0 2024-09-17T18:19:03.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.260+0000 7f37cb0b0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f37c409e810 addr v1:172.21.15.173:6789/0 2024-09-17T18:19:03.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.260+0000 7f37cb0b0640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37c409ec60 0x7f37c4138a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.260+0000 7f37cb0b0640 10 monclient(hunting): picked mon.noname-b con 0x7f37c409ec60 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:03.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.260+0000 7f37cb0b0640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37c409ca60 con 0x7f37c409e810 2024-09-17T18:19:03.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.260+0000 7f37cb0b0640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.260+0000 7f37cb0b0640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f37c40704d0 con 0x7f37c40700f0 2024-09-17T18:19:03.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.260+0000 7f37cb0b0640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.260+0000 7f37cb0b0640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f37c409cd20 con 0x7f37c409ec60 2024-09-17T18:19:03.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.260+0000 7f37cb0b0640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:03.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.260+0000 7f37cb0b0640 10 monclient(hunting): authenticate will time out at 746.976579s 2024-09-17T18:19:03.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.260+0000 7f37c8e4c640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f37c40700f0 0x7f37c4134e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.260+0000 7f37c8e4c640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f37c40700f0 0x7f37c4134e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.121:42104/0 (socket says 172.21.15.121:42104) 2024-09-17T18:19:03.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.260+0000 7f37c8e4c640 1 -- 172.21.15.121:0/1951265070 learned_addr learned my addr 172.21.15.121:0/1951265070 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:03.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.260+0000 7f37c8e4c640 10 monclient(hunting): get_auth_request con 0x7f37c40700f0 auth_method 0 2024-09-17T18:19:03.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.260+0000 7f37c8e4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:03.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.260+0000 7f37c8e4c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:03.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.260+0000 7f37c8e4c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:03.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.260+0000 7f37c964d640 1 --2- 172.21.15.121:0/1951265070 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37c409ec60 0x7f37c4138a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.260+0000 7f37c964d640 10 monclient(hunting): get_auth_request con 0x7f37c409ec60 auth_method 0 2024-09-17T18:19:03.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.260+0000 7f37c964d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:03.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.260+0000 7f37c964d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:03.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.260+0000 7f37c964d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:03.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.260+0000 7f37c8e4c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:03.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.260+0000 7f37c8e4c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:03.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.260+0000 7f37c8e4c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:03.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.260+0000 7f37c964d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:03.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.260+0000 7f37c964d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:03.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.260+0000 7f37c964d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:03.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.261+0000 7f37c8e4c640 10 monclient(hunting): handle_auth_done global_id 4252 payload 274 2024-09-17T18:19:03.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.261+0000 7f37c8e4c640 1 -- 172.21.15.121:0/1951265070 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37c409ec60 msgr2=0x7f37c4138a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:03.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.261+0000 7f37c8e4c640 1 --2- 172.21.15.121:0/1951265070 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37c409ec60 0x7f37c4138a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.261+0000 7f37c8e4c640 1 -- 172.21.15.121:0/1951265070 >> v1:172.21.15.173:6789/0 conn(0x7f37c409e810 legacy=0x7f37c4138360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:03.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.261+0000 7f37c8e4c640 10 monclient: _finish_hunting 0 2024-09-17T18:19:03.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.261+0000 7f37c8e4c640 1 monclient: found mon.noname-c 2024-09-17T18:19:03.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.261+0000 7f37c8e4c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:03.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.261+0000 7f37c8e4c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.173:3300/0 2024-09-17T18:19:03.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.261+0000 7f37c8e4c640 1 -- 172.21.15.121:0/1951265070 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37c409d000 con 0x7f37c40700f0 2024-09-17T18:19:03.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.261+0000 7f37c964d640 1 --2- 172.21.15.121:0/1951265070 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37c409ec60 0x7f37c4138a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:19:03.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.264+0000 7ff050d3d640 10 monclient: get_monmap_and_config 2024-09-17T18:19:03.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.264+0000 7ff050d3d640 10 monclient: build_initial_monmap 2024-09-17T18:19:03.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.264+0000 7ff050d3d640 10 monclient: monmap: 2024-09-17T18:19:03.270 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:03.270 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.270 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:19:03.264728+0000 2024-09-17T18:19:03.270 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:19:03.264728+0000 2024-09-17T18:19:03.270 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:03.270 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.270 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:19:03.271 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:19:03.271 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:19:03.271 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:19:03.271 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:19:03.271 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.264+0000 7f37c8e4c640 1 -- 172.21.15.121:0/1951265070 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37c40700f0 msgr2=0x7f37c4134e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:03.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.264+0000 7f37c8e4c640 1 --2- 172.21.15.121:0/1951265070 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37c40700f0 0x7f37c4134e50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f37b8009870 tx=0x7f37b802a0b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3df882d714cadba9 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:03.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.264+0000 7ff050d3d640 1 Processor -- start 2024-09-17T18:19:03.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.264+0000 7ff050d3d640 1 -- start start 2024-09-17T18:19:03.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.264+0000 7ff050d3d640 10 monclient: init 2024-09-17T18:19:03.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.264+0000 7f37c37fe640 1 -- 172.21.15.121:0/1951265070 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b8037070 con 0x7f37c40700f0 2024-09-17T18:19:03.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.264+0000 7f37c37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:03.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.264+0000 7f37c37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:19:03.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.264+0000 7f37c37fe640 10 monclient: dump: 2024-09-17T18:19:03.274 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:03.274 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.274 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.274 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.274 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:03.274 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.274 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:03.274 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:03.274 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:03.274 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.265+0000 7f37c37fe640 10 monclient: _finish_auth 0 2024-09-17T18:19:03.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.265+0000 7f37c37fe640 10 monclient: _check_auth_tickets 2024-09-17T18:19:03.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.265+0000 7f37c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:03.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.265+0000 7f37cb0b0640 5 monclient: authenticate success, global_id 4252 2024-09-17T18:19:03.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.265+0000 7f37cb0b0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:19:03.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.265+0000 7f37c37fe640 1 -- 172.21.15.121:0/1951265070 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37b8032640 con 0x7f37c40700f0 2024-09-17T18:19:03.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.265+0000 7f37c37fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:03.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.265+0000 7f37c37fe640 1 -- 172.21.15.121:0/1951265070 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b8032a10 con 0x7f37c40700f0 2024-09-17T18:19:03.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.265+0000 7f37cb0b0640 10 monclient: get_monmap_and_config success 2024-09-17T18:19:03.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.265+0000 7f37cb0b0640 4 set_mon_vals no callback set 2024-09-17T18:19:03.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.265+0000 7f37c37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:03.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.265+0000 7f37c37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:19:03.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.265+0000 7f37c37fe640 10 monclient: dump: 2024-09-17T18:19:03.276 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:03.276 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.276 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.276 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.276 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:03.276 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.276 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:03.276 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:03.277 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:03.277 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.265+0000 7ff050d3d640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:03.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.265+0000 7ff050d3d640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-17T18:19:03.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.265+0000 7ff050d3d640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7ff04c0a5ca0 0x7ff04c0a6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.265+0000 7ff050d3d640 10 monclient(hunting): picked mon.noname-c con 0x7ff04c0a5ca0 addr v2:172.21.15.173:3300/0 2024-09-17T18:19:03.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.265+0000 7ff050d3d640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff04c0a65c0 0x7ff04c13ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.265+0000 7ff050d3d640 10 monclient(hunting): picked mon.noname-a con 0x7ff04c0a65c0 addr v2:172.21.15.121:3300/0 2024-09-17T18:19:03.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.265+0000 7ff050d3d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff04c13cfc0 addr v1:172.21.15.121:6789/0 2024-09-17T18:19:03.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.265+0000 7ff050d3d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff04c0a7b50 con 0x7ff04c13cfc0 2024-09-17T18:19:03.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.265+0000 7ff050d3d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.265+0000 7ff050d3d640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7ff04c0704c0 con 0x7ff04c0a65c0 2024-09-17T18:19:03.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.265+0000 7ff050d3d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.265+0000 7f37cb0b0640 10 monclient: shutdown 2024-09-17T18:19:03.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.265+0000 7ff050d3d640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7ff04c0a5490 con 0x7ff04c0a5ca0 2024-09-17T18:19:03.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.265+0000 7ff050d3d640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:03.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.265+0000 7ff050d3d640 10 monclient(hunting): authenticate will time out at 746.981603s 2024-09-17T18:19:03.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.265+0000 7ff04ad9d640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7ff04c13cfc0 0x7ff04c13f3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:49582/0 (socket says 172.21.15.121:49582) 2024-09-17T18:19:03.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.265+0000 7ff04ad9d640 1 -- 172.21.15.121:0/3385996437 learned_addr learned my addr 172.21.15.121:0/3385996437 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:19:03.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.265+0000 7ff049d9b640 1 --2- 172.21.15.121:0/3385996437 >> v2:172.21.15.121:3300/0 conn(0x7ff04c0a65c0 0x7ff04c13ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.265+0000 7ff049d9b640 10 monclient(hunting): get_auth_request con 0x7ff04c0a65c0 auth_method 0 2024-09-17T18:19:03.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.265+0000 7ff049d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:03.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.265+0000 7ff049d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:03.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.265+0000 7ff049d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:03.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.265+0000 7ff04a59c640 1 --2- 172.21.15.121:0/3385996437 >> v2:172.21.15.173:3300/0 conn(0x7ff04c0a5ca0 0x7ff04c0a6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.265+0000 7ff04a59c640 10 monclient(hunting): get_auth_request con 0x7ff04c0a5ca0 auth_method 0 2024-09-17T18:19:03.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.265+0000 7ff04a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:03.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.265+0000 7ff04a59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:03.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.265+0000 7ff04a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:03.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.266+0000 7ff049d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:03.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.266+0000 7ff049d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:03.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.266+0000 7ff049d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:03.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.266+0000 7ff04a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:03.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.266+0000 7f37cb0b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:03.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.266+0000 7f37cb0b0640 1 -- 172.21.15.121:0/1951265070 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37c40700f0 msgr2=0x7f37c4134e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.266+0000 7f37cb0b0640 1 --2- 172.21.15.121:0/1951265070 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37c40700f0 0x7f37c4134e50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f37b8009870 tx=0x7f37b802a0b0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.266+0000 7f37cb0b0640 1 -- 172.21.15.121:0/1951265070 shutdown_connections 2024-09-17T18:19:03.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.266+0000 7f37cb0b0640 1 --2- 172.21.15.121:0/1951265070 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37c409ec60 0x7f37c4138a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.266+0000 7f37cb0b0640 1 --2- 172.21.15.121:0/1951265070 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37c40700f0 0x7f37c4134e50 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.266+0000 7f37cb0b0640 1 -- 172.21.15.121:0/1951265070 >> 172.21.15.121:0/1951265070 conn(0x7f37c40ab830 msgr2=0x7f37c40a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:03.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.266+0000 7ff04a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:03.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.266+0000 7ff04a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:03.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.266+0000 7f37cb0b0640 1 -- 172.21.15.121:0/1951265070 shutdown_connections 2024-09-17T18:19:03.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.266+0000 7ff049d9b640 10 monclient(hunting): handle_auth_done global_id 4260 payload 274 2024-09-17T18:19:03.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.266+0000 7ff049d9b640 1 -- 172.21.15.121:0/3385996437 >> v2:172.21.15.173:3300/0 conn(0x7ff04c0a5ca0 msgr2=0x7ff04c0a6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:03.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.266+0000 7ff049d9b640 1 --2- 172.21.15.121:0/3385996437 >> v2:172.21.15.173:3300/0 conn(0x7ff04c0a5ca0 0x7ff04c0a6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.266+0000 7ff049d9b640 1 -- 172.21.15.121:0/3385996437 >> v1:172.21.15.121:6789/0 conn(0x7ff04c13cfc0 legacy=0x7ff04c13f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:03.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.266+0000 7ff049d9b640 10 monclient: _finish_hunting 0 2024-09-17T18:19:03.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.266+0000 7ff049d9b640 1 monclient: found mon.noname-a 2024-09-17T18:19:03.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.266+0000 7ff049d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:03.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.266+0000 7ff049d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:19:03.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.266+0000 7ff049d9b640 1 -- 172.21.15.121:0/3385996437 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff04c0a2a50 con 0x7ff04c0a65c0 2024-09-17T18:19:03.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.266+0000 7f37cb0b0640 1 -- 172.21.15.121:0/1951265070 wait complete. 2024-09-17T18:19:03.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.267+0000 7f53bb19b640 10 monclient: get_monmap_and_config 2024-09-17T18:19:03.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.267+0000 7f53bb19b640 10 monclient: build_initial_monmap 2024-09-17T18:19:03.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.267+0000 7f53bb19b640 10 monclient: monmap: 2024-09-17T18:19:03.285 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:03.285 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.285 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:19:03.267838+0000 2024-09-17T18:19:03.285 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:19:03.267838+0000 2024-09-17T18:19:03.285 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:03.285 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.286 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:19:03.286 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:19:03.286 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:19:03.286 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:19:03.286 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:19:03.286 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.267+0000 7f37cb0b0640 10 monclient: build_initial_monmap 2024-09-17T18:19:03.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.267+0000 7f37cb0b0640 10 monclient: monmap: 2024-09-17T18:19:03.287 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:03.287 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:19:03.287 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:19:03.287 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:19:03.287 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:03.287 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.287 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:19:03.287 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:19:03.287 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:19:03.287 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.267+0000 7f37cb0b0640 1 librados: starting msgr at 2024-09-17T18:19:03.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.267+0000 7f37cb0b0640 1 librados: starting objecter 2024-09-17T18:19:03.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.267+0000 7f53bb19b640 1 Processor -- start 2024-09-17T18:19:03.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.268+0000 7f37cb0b0640 1 Processor -- start 2024-09-17T18:19:03.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.268+0000 7f53bb19b640 1 -- start start 2024-09-17T18:19:03.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.268+0000 7f53bb19b640 10 monclient: init 2024-09-17T18:19:03.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.268+0000 7f37cb0b0640 1 -- start start 2024-09-17T18:19:03.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.268+0000 7f37cb0b0640 1 librados: setting wanted keys 2024-09-17T18:19:03.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.268+0000 7f37cb0b0640 1 librados: calling monclient init 2024-09-17T18:19:03.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.268+0000 7f37cb0b0640 10 monclient: init 2024-09-17T18:19:03.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.268+0000 7f53bb19b640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:03.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.268+0000 7f53bb19b640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-17T18:19:03.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.268+0000 7f53bb19b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53b40a5ca0 0x7f53b40a6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.268+0000 7f53bb19b640 10 monclient(hunting): picked mon.noname-b con 0x7f53b40a5ca0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:03.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.268+0000 7f53b8f37640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53b40a5ca0 0x7f53b40a6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.268+0000 7f53b8f37640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53b40a5ca0 0x7f53b40a6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:59396/0 (socket says 172.21.15.121:59396) 2024-09-17T18:19:03.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.268+0000 7f53b8f37640 1 -- 172.21.15.121:0/778619321 learned_addr learned my addr 172.21.15.121:0/778619321 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:03.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.268+0000 7f53bb19b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f53b40a65c0 addr v1:172.21.15.173:6789/0 2024-09-17T18:19:03.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.268+0000 7f53bb19b640 1 --2- 172.21.15.121:0/778619321 >> v2:172.21.15.121:3300/0 conn(0x7f53b413e1a0 0x7f53b4140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.268+0000 7f53bb19b640 10 monclient(hunting): picked mon.noname-a con 0x7f53b413e1a0 addr v2:172.21.15.121:3300/0 2024-09-17T18:19:03.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.268+0000 7f53bb19b640 1 -- 172.21.15.121:0/778619321 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53b40a7b50 con 0x7f53b40a65c0 2024-09-17T18:19:03.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.268+0000 7f53bb19b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.268+0000 7f53bb19b640 1 -- 172.21.15.121:0/778619321 --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f53b40704c0 con 0x7f53b413e1a0 2024-09-17T18:19:03.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.268+0000 7f53bb19b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.268+0000 7f53bb19b640 1 -- 172.21.15.121:0/778619321 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f53b40a5490 con 0x7f53b40a5ca0 2024-09-17T18:19:03.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.268+0000 7f53bb19b640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:03.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.268+0000 7f53bb19b640 10 monclient(hunting): authenticate will time out at 746.984809s 2024-09-17T18:19:03.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.268+0000 7f53b8f37640 10 monclient(hunting): get_auth_request con 0x7f53b40a5ca0 auth_method 0 2024-09-17T18:19:03.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.268+0000 7f53b8f37640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:03.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.268+0000 7f53b8f37640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:03.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.268+0000 7f53b8f37640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:03.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.268+0000 7f53b8f37640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:03.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.268+0000 7f53b8f37640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:03.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.268+0000 7f53b8f37640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:03.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.269+0000 7f53b9738640 1 --2- 172.21.15.121:0/778619321 >> v2:172.21.15.121:3300/0 conn(0x7f53b413e1a0 0x7f53b4140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.269+0000 7f53b9738640 10 monclient(hunting): get_auth_request con 0x7f53b413e1a0 auth_method 0 2024-09-17T18:19:03.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.269+0000 7f53b9738640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:03.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.269+0000 7f53b9738640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:03.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.269+0000 7f53b9738640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:03.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.269+0000 7f53b8f37640 10 monclient(hunting): handle_auth_done global_id 4241 payload 274 2024-09-17T18:19:03.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.269+0000 7f53b8f37640 1 -- 172.21.15.121:0/778619321 >> v2:172.21.15.121:3300/0 conn(0x7f53b413e1a0 msgr2=0x7f53b4140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:03.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.269+0000 7f53b8f37640 1 --2- 172.21.15.121:0/778619321 >> v2:172.21.15.121:3300/0 conn(0x7f53b413e1a0 0x7f53b4140590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.269+0000 7ff049d9b640 1 -- 172.21.15.121:0/3385996437 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff04c0a65c0 msgr2=0x7ff04c13ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:03.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.269+0000 7f53b8f37640 1 -- 172.21.15.121:0/778619321 >> v1:172.21.15.173:6789/0 conn(0x7f53b40a65c0 legacy=0x7f53b413ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:03.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.269+0000 7f53b8f37640 10 monclient: _finish_hunting 0 2024-09-17T18:19:03.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.269+0000 7f53b8f37640 1 monclient: found mon.noname-b 2024-09-17T18:19:03.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.269+0000 7f53b8f37640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:03.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.269+0000 7f53b8f37640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:19:03.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.269+0000 7f37cb0b0640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:03.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.269+0000 7f53b8f37640 1 -- 172.21.15.121:0/778619321 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53b40a2a50 con 0x7f53b40a5ca0 2024-09-17T18:19:03.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.269+0000 7f37cb0b0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T18:19:03.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.269+0000 7f37cb0b0640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37c40700f0 0x7f37c412ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.269+0000 7ff049d9b640 1 --2- 172.21.15.121:0/3385996437 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff04c0a65c0 0x7ff04c13ca80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff04c06c260 tx=0x7ff03c02a010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6551495ae20a33d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:03.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.269+0000 7ff04959a640 1 -- 172.21.15.121:0/3385996437 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff03c037070 con 0x7ff04c0a65c0 2024-09-17T18:19:03.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.269+0000 7f37cb0b0640 10 monclient(hunting): picked mon.noname-c con 0x7f37c40700f0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:03.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.269+0000 7f37c8e4c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37c40700f0 0x7f37c412ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.269+0000 7ff04959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:03.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.269+0000 7f37c8e4c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37c40700f0 0x7f37c412ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:59408/0 (socket says 172.21.15.121:59408) 2024-09-17T18:19:03.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.269+0000 7f37c8e4c640 1 -- 172.21.15.121:0/1226608797 learned_addr learned my addr 172.21.15.121:0/1226608797 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:03.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.269+0000 7ff04959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:19:03.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.269+0000 7f37cb0b0640 1 --2- 172.21.15.121:0/1226608797 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37c409e810 0x7f37c412f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.269+0000 7ff04959a640 10 monclient: dump: 2024-09-17T18:19:03.298 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:03.298 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.298 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.298 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.298 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:03.298 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.298 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:03.299 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:03.299 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:03.299 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.269+0000 7ff04959a640 10 monclient: _finish_auth 0 2024-09-17T18:19:03.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.269+0000 7ff04959a640 10 monclient: _check_auth_tickets 2024-09-17T18:19:03.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.269+0000 7ff04959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:03.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.269+0000 7ff050d3d640 5 monclient: authenticate success, global_id 4260 2024-09-17T18:19:03.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.269+0000 7ff050d3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:19:03.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.269+0000 7ff04959a640 1 -- 172.21.15.121:0/3385996437 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff03c032440 con 0x7ff04c0a65c0 2024-09-17T18:19:03.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.269+0000 7ff04959a640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:03.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.269+0000 7f37c3fff640 1 --2- 172.21.15.121:0/1226608797 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37c409e810 0x7f37c412f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7f37cb0b0640 10 monclient(hunting): picked mon.noname-a con 0x7f37c409e810 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:03.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.269+0000 7ff050d3d640 10 monclient: get_monmap_and_config success 2024-09-17T18:19:03.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7f37cb0b0640 1 --2- 172.21.15.121:0/1226608797 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37c409ec60 0x7f37c41cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7f37cb0b0640 10 monclient(hunting): picked mon.noname-b con 0x7f37c409ec60 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:03.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7f37cb0b0640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7f37cb0b0640 1 -- 172.21.15.121:0/1226608797 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f37c4135510 con 0x7f37c409e810 2024-09-17T18:19:03.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7f37cb0b0640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7f37cb0b0640 1 -- 172.21.15.121:0/1226608797 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f37c4135390 con 0x7f37c409ec60 2024-09-17T18:19:03.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7f37cb0b0640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7f37cb0b0640 1 -- 172.21.15.121:0/1226608797 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f37c4138fb0 con 0x7f37c40700f0 2024-09-17T18:19:03.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7f37cb0b0640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:03.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7f37cb0b0640 10 monclient(hunting): authenticate will time out at 746.986206s 2024-09-17T18:19:03.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7f37c8e4c640 10 monclient(hunting): get_auth_request con 0x7f37c40700f0 auth_method 0 2024-09-17T18:19:03.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7f37c8e4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:03.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7f37c8e4c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:03.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7f37c8e4c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:03.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.269+0000 7ff050d3d640 4 set_mon_vals no callback set 2024-09-17T18:19:03.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7f37c3fff640 10 monclient(hunting): get_auth_request con 0x7f37c409e810 auth_method 0 2024-09-17T18:19:03.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7f37c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:03.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7f37c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:03.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7f37c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:03.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7f37c964d640 1 --2- 172.21.15.121:0/1226608797 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37c409ec60 0x7f37c41cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7f37c964d640 10 monclient(hunting): get_auth_request con 0x7f37c409ec60 auth_method 0 2024-09-17T18:19:03.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7f37c964d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:03.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7f37c964d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:03.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7f37c964d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:03.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7ff04959a640 1 -- 172.21.15.121:0/3385996437 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff03c032810 con 0x7ff04c0a65c0 2024-09-17T18:19:03.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7ff04959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:03.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7ff04959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:19:03.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7ff04959a640 10 monclient: dump: 2024-09-17T18:19:03.305 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:03.305 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.305 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.305 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.305 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:03.305 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.305 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:03.305 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:03.305 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:03.305 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7f37c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:03.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7f37c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:03.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7ff050d3d640 10 monclient: shutdown 2024-09-17T18:19:03.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7ff050d3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:03.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7f37c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:03.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7f37c964d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:03.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7ff050d3d640 1 -- 172.21.15.121:0/3385996437 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff04c0a65c0 msgr2=0x7ff04c13ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7ff050d3d640 1 --2- 172.21.15.121:0/3385996437 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff04c0a65c0 0x7ff04c13ca80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff04c06c260 tx=0x7ff03c02a010 comp rx=0 tx=0).stop 2024-09-17T18:19:03.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7f37c964d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:03.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7f37c964d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:03.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7f37c964d640 10 monclient(hunting): handle_auth_done global_id 4258 payload 824 2024-09-17T18:19:03.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7f37c964d640 1 -- 172.21.15.121:0/1226608797 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37c40700f0 msgr2=0x7f37c412ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7ff050d3d640 1 -- 172.21.15.121:0/3385996437 shutdown_connections 2024-09-17T18:19:03.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7ff050d3d640 1 --2- 172.21.15.121:0/3385996437 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff04c0a65c0 0x7ff04c13ca80 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7ff050d3d640 1 --2- 172.21.15.121:0/3385996437 >> v2:172.21.15.173:3300/0 conn(0x7ff04c0a5ca0 0x7ff04c0a6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7ff050d3d640 1 -- 172.21.15.121:0/3385996437 >> 172.21.15.121:0/3385996437 conn(0x7ff04c06dd10 msgr2=0x7ff04c0a7da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:03.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7ff050d3d640 1 -- 172.21.15.121:0/3385996437 shutdown_connections 2024-09-17T18:19:03.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7ff050d3d640 1 -- 172.21.15.121:0/3385996437 wait complete. 2024-09-17T18:19:03.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7f37c964d640 1 --2- 172.21.15.121:0/1226608797 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37c40700f0 0x7f37c412ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7f37c964d640 1 -- 172.21.15.121:0/1226608797 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37c409e810 msgr2=0x7f37c412f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7f37c964d640 1 --2- 172.21.15.121:0/1226608797 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37c409e810 0x7f37c412f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7f37c964d640 10 monclient: _finish_hunting 0 2024-09-17T18:19:03.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7f37c964d640 1 monclient: found mon.noname-b 2024-09-17T18:19:03.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7f37c964d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:03.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7f37c964d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.173:3300/0 2024-09-17T18:19:03.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7f37c964d640 1 -- 172.21.15.121:0/1226608797 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37c41328d0 con 0x7f37c409ec60 2024-09-17T18:19:03.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7f37c3fff640 1 --2- 172.21.15.121:0/1226608797 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37c409e810 0x7f37c412f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:19:03.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.270+0000 7f37c8e4c640 1 --2- 172.21.15.121:0/1226608797 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37c40700f0 0x7f37c412ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T18:19:03.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.271+0000 7ff050d3d640 10 monclient: build_initial_monmap 2024-09-17T18:19:03.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.271+0000 7ff050d3d640 10 monclient: monmap: 2024-09-17T18:19:03.311 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:03.311 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:19:03.311 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:19:03.311 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:19:03.311 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:03.311 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.311 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:19:03.311 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:19:03.311 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:19:03.311 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.271+0000 7ff050d3d640 1 librados: starting msgr at 2024-09-17T18:19:03.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.271+0000 7ff050d3d640 1 librados: starting objecter 2024-09-17T18:19:03.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.271+0000 7ff050d3d640 1 Processor -- start 2024-09-17T18:19:03.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7ff050d3d640 1 -- start start 2024-09-17T18:19:03.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7ff050d3d640 1 librados: setting wanted keys 2024-09-17T18:19:03.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7ff050d3d640 1 librados: calling monclient init 2024-09-17T18:19:03.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7ff050d3d640 10 monclient: init 2024-09-17T18:19:03.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7f53b8f37640 1 -- 172.21.15.121:0/778619321 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53b40a5ca0 msgr2=0x7f53b40a6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:03.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7f53b8f37640 1 --2- 172.21.15.121:0/778619321 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53b40a5ca0 0x7f53b40a6080 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f53b40a3dd0 tx=0x7f53a802a080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15a6f11eafa4e7d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:03.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7f53b37fe640 1 -- 172.21.15.121:0/778619321 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a8037070 con 0x7f53b40a5ca0 2024-09-17T18:19:03.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7f53b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:03.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7f53b37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:19:03.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7f53b37fe640 10 monclient: dump: 2024-09-17T18:19:03.313 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:03.313 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.313 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.313 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.313 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:03.313 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.313 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:03.313 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:03.314 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:03.314 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7f53b37fe640 10 monclient: _finish_auth 0 2024-09-17T18:19:03.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7f53b37fe640 10 monclient: _check_auth_tickets 2024-09-17T18:19:03.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7f53b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:03.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7f53b37fe640 1 -- 172.21.15.121:0/778619321 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53a8032440 con 0x7f53b40a5ca0 2024-09-17T18:19:03.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7f53b37fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:03.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7f53b37fe640 1 -- 172.21.15.121:0/778619321 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a80327f0 con 0x7f53b40a5ca0 2024-09-17T18:19:03.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7f53b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:03.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7f53b37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:19:03.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7f53b37fe640 10 monclient: dump: 2024-09-17T18:19:03.314 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:03.314 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.314 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.315 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.315 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:03.315 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.315 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:03.315 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:03.315 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:03.315 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7f53bb19b640 5 monclient: authenticate success, global_id 4241 2024-09-17T18:19:03.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7f53bb19b640 10 monclient: get_monmap_and_config success 2024-09-17T18:19:03.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7f53bb19b640 4 set_mon_vals no callback set 2024-09-17T18:19:03.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7ff050d3d640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:03.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7ff050d3d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T18:19:03.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7ff050d3d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff04c0a5ca0 0x7ff04c124290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7ff050d3d640 10 monclient(hunting): picked mon.noname-a con 0x7ff04c0a5ca0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:03.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7ff050d3d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff04c0a65c0 0x7ff04c1380d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7ff050d3d640 10 monclient(hunting): picked mon.noname-c con 0x7ff04c0a65c0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:03.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7ff050d3d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff04c13cfc0 0x7ff04c138610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7ff050d3d640 10 monclient(hunting): picked mon.noname-b con 0x7ff04c13cfc0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:03.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7ff050d3d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7ff050d3d640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff04c0a5670 con 0x7ff04c0a5ca0 2024-09-17T18:19:03.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7ff050d3d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7ff050d3d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff04c0a6eb0 con 0x7ff04c13cfc0 2024-09-17T18:19:03.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7ff050d3d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7ff050d3d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff04c0a69d0 con 0x7ff04c0a65c0 2024-09-17T18:19:03.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7ff050d3d640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:03.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7ff050d3d640 10 monclient(hunting): authenticate will time out at 746.988603s 2024-09-17T18:19:03.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7ff04a59c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff04c0a5ca0 0x7ff04c124290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7ff049d9b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff04c0a65c0 0x7ff04c1380d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7ff04ad9d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff04c13cfc0 0x7ff04c138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7ff049d9b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff04c0a65c0 0x7ff04c1380d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:59422/0 (socket says 172.21.15.121:59422) 2024-09-17T18:19:03.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7f37c964d640 1 -- 172.21.15.121:0/1226608797 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37c409ec60 msgr2=0x7f37c41cd7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:03.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7ff049d9b640 1 -- 172.21.15.121:0/1472694789 learned_addr learned my addr 172.21.15.121:0/1472694789 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:03.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7ff049d9b640 10 monclient(hunting): get_auth_request con 0x7ff04c0a65c0 auth_method 0 2024-09-17T18:19:03.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7ff049d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:03.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7ff049d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:03.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7ff049d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:03.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7ff04a59c640 10 monclient(hunting): get_auth_request con 0x7ff04c0a5ca0 auth_method 0 2024-09-17T18:19:03.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7ff04a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:03.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7f53bb19b640 10 monclient: shutdown 2024-09-17T18:19:03.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7f53bb19b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:03.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7f53bb19b640 1 -- 172.21.15.121:0/778619321 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53b40a5ca0 msgr2=0x7f53b40a6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7ff04a59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:03.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7ff04a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:03.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7ff04ad9d640 10 monclient(hunting): get_auth_request con 0x7ff04c13cfc0 auth_method 0 2024-09-17T18:19:03.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7ff04ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:03.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7ff04ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:03.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7ff04ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:03.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7ff049d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:03.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7ff049d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:03.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.272+0000 7ff049d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:03.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7f53bb19b640 1 --2- 172.21.15.121:0/778619321 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53b40a5ca0 0x7f53b40a6080 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f53b40a3dd0 tx=0x7f53a802a080 comp rx=0 tx=0).stop 2024-09-17T18:19:03.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7ff04ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:03.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7ff04ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:03.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7ff04ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:03.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7ff04a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:03.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7ff04a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:03.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7ff04a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:03.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7ff049d9b640 10 monclient(hunting): handle_auth_done global_id 4247 payload 824 2024-09-17T18:19:03.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7ff049d9b640 1 -- 172.21.15.121:0/1472694789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff04c13cfc0 msgr2=0x7ff04c138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7ff049d9b640 1 --2- 172.21.15.121:0/1472694789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff04c13cfc0 0x7ff04c138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7ff049d9b640 1 -- 172.21.15.121:0/1472694789 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff04c0a5ca0 msgr2=0x7ff04c124290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7f53bb19b640 1 -- 172.21.15.121:0/778619321 shutdown_connections 2024-09-17T18:19:03.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7f53bb19b640 1 --2- 172.21.15.121:0/778619321 >> v2:172.21.15.121:3300/0 conn(0x7f53b413e1a0 0x7f53b4140590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7f53bb19b640 1 --2- 172.21.15.121:0/778619321 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53b40a5ca0 0x7f53b40a6080 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7f53bb19b640 1 -- 172.21.15.121:0/778619321 >> 172.21.15.121:0/778619321 conn(0x7f53b406dd10 msgr2=0x7f53b40a7da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:03.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7f37c964d640 1 --2- 172.21.15.121:0/1226608797 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37c409ec60 0x7f37c41cd7a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f37bc009e50 tx=0x7f37bc00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:03.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7f53bb19b640 1 -- 172.21.15.121:0/778619321 shutdown_connections 2024-09-17T18:19:03.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7f53bb19b640 1 -- 172.21.15.121:0/778619321 wait complete. 2024-09-17T18:19:03.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7f37c17fa640 10 client.?.objecter ms_handle_connect 0x7f37c409ec60 2024-09-17T18:19:03.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7f37c17fa640 10 client.?.objecter resend_mon_ops 2024-09-17T18:19:03.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7f37c17fa640 1 -- 172.21.15.121:0/1226608797 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37bc018030 con 0x7f37c409ec60 2024-09-17T18:19:03.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7ff049d9b640 1 --2- 172.21.15.121:0/1472694789 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff04c0a5ca0 0x7ff04c124290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7ff049d9b640 10 monclient: _finish_hunting 0 2024-09-17T18:19:03.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7ff049d9b640 1 monclient: found mon.noname-c 2024-09-17T18:19:03.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7ff049d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:03.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7ff049d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:19:03.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7ff049d9b640 1 -- 172.21.15.121:0/1472694789 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff04c0a2490 con 0x7ff04c0a65c0 2024-09-17T18:19:03.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7ff04a59c640 1 --2- 172.21.15.121:0/1472694789 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff04c0a5ca0 0x7ff04c124290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:19:03.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7f37c17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:03.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7f37c17fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:19:03.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7f37c17fa640 10 monclient: dump: 2024-09-17T18:19:03.327 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:03.327 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.327 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.327 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.327 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:03.327 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.327 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:03.327 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:03.327 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:03.327 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7f37c17fa640 10 monclient: _finish_auth 0 2024-09-17T18:19:03.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7f37c17fa640 10 monclient: _check_auth_tickets 2024-09-17T18:19:03.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7f37c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:03.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7f37c17fa640 1 -- 172.21.15.121:0/1226608797 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37bc003990 con 0x7f37c409ec60 2024-09-17T18:19:03.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7f37c17fa640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:03.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7f37c17fa640 1 -- 172.21.15.121:0/1226608797 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37bc003d60 con 0x7f37c409ec60 2024-09-17T18:19:03.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7f37c17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:03.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7f37c17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:19:03.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7f37c17fa640 10 monclient: dump: 2024-09-17T18:19:03.328 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:03.328 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.328 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.328 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.328 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:03.328 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.329 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:03.329 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:03.329 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:03.329 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7f37c2ffd640 4 set_mon_vals no callback set 2024-09-17T18:19:03.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7f37cb0b0640 5 monclient: authenticate success, global_id 4258 2024-09-17T18:19:03.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7f37cb0b0640 10 monclient: _renew_subs 2024-09-17T18:19:03.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7f37cb0b0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:19:03.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.274+0000 7f53bb19b640 10 monclient: build_initial_monmap 2024-09-17T18:19:03.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7f37cb0b0640 1 -- 172.21.15.121:0/1226608797 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37c40a4650 con 0x7f37c409ec60 2024-09-17T18:19:03.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.274+0000 7f53bb19b640 10 monclient: monmap: 2024-09-17T18:19:03.330 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:03.330 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:19:03.330 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:19:03.330 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:19:03.330 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:03.330 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.330 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:19:03.331 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:19:03.331 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:19:03.331 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7f37cb0b0640 10 client.4258.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:03.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7f37cb0b0640 10 monclient: _renew_subs 2024-09-17T18:19:03.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7f37cb0b0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:19:03.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7f37cb0b0640 1 -- 172.21.15.121:0/1226608797 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37c4130fd0 con 0x7f37c409ec60 2024-09-17T18:19:03.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.273+0000 7f37cb0b0640 1 librados: init done 2024-09-17T18:19:03.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.274+0000 7f37cb0b0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:03.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.274+0000 7f53bb19b640 1 librados: starting msgr at 2024-09-17T18:19:03.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.274+0000 7f53bb19b640 1 librados: starting objecter 2024-09-17T18:19:03.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.274+0000 7f53bb19b640 1 Processor -- start 2024-09-17T18:19:03.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.274+0000 7f37cb0b0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:03.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.274+0000 7f53bb19b640 1 -- start start 2024-09-17T18:19:03.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.275+0000 7f53bb19b640 1 librados: setting wanted keys 2024-09-17T18:19:03.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.275+0000 7f53bb19b640 1 librados: calling monclient init 2024-09-17T18:19:03.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.275+0000 7f53bb19b640 10 monclient: init 2024-09-17T18:19:03.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.274+0000 7f37cb0b0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:19:03.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.274+0000 7f37cb0b0640 1 -- 172.21.15.121:0/1226608797 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37c4134e50 con 0x7f37c409ec60 2024-09-17T18:19:03.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.275+0000 7f37c17fa640 1 -- 172.21.15.121:0/1226608797 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f37bc005d10 con 0x7f37c409ec60 2024-09-17T18:19:03.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.275+0000 7f37c17fa640 1 --2- 172.21.15.121:0/1226608797 >> v2:172.21.15.173:6812/2423786037 conn(0x7f37a80401c0 0x7f37a8042680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.275+0000 7f37c17fa640 1 -- 172.21.15.121:0/1226608797 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 3562+0+0 (secure 0 0 0) 0x7f37bc053a40 con 0x7f37c409ec60 2024-09-17T18:19:03.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.275+0000 7f37c17fa640 10 client.4258.objecter ms_dispatch 0x7f37c4000b90 osd_map(16..16 src has 1..16) 2024-09-17T18:19:03.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.275+0000 7f37c17fa640 3 client.4258.objecter handle_osd_map got epochs [16,16] > 0 2024-09-17T18:19:03.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.275+0000 7f37c17fa640 3 client.4258.objecter handle_osd_map decoding full epoch 16 2024-09-17T18:19:03.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.275+0000 7f53bb19b640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:03.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.275+0000 7f53bb19b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:19:03.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.275+0000 7f53bb19b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53b40a5ca0 0x7f53b40a0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.275+0000 7f37c8e4c640 1 --2- 172.21.15.121:0/1226608797 >> v2:172.21.15.173:6812/2423786037 conn(0x7f37a80401c0 0x7f37a8042680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.275+0000 7f37c17fa640 20 client.4258.objecter dump_active .. 0 homeless 2024-09-17T18:19:03.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.275+0000 7f37c8e4c640 10 monclient: get_auth_request con 0x7f37a80401c0 auth_method 0 2024-09-17T18:19:03.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.275+0000 7f53bb19b640 10 monclient(hunting): picked mon.noname-a con 0x7f53b40a5ca0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:03.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.275+0000 7f53bb19b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53b40a65c0 0x7f53b4137cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.275+0000 7f53bb19b640 10 monclient(hunting): picked mon.noname-b con 0x7f53b40a65c0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:03.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.275+0000 7f53bb19b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53b413e1a0 0x7f53b4138210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.275+0000 7f53bb19b640 10 monclient(hunting): picked mon.noname-c con 0x7f53b413e1a0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:03.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.275+0000 7f53bb19b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.275+0000 7f53bb19b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f53b40a5670 con 0x7f53b40a5ca0 2024-09-17T18:19:03.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.275+0000 7f53bb19b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.275+0000 7f53bb19b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f53b40a6eb0 con 0x7f53b40a65c0 2024-09-17T18:19:03.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.275+0000 7f53bb19b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.275+0000 7f53bb19b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f53b40a69a0 con 0x7f53b413e1a0 2024-09-17T18:19:03.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.275+0000 7f53bb19b640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:03.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.275+0000 7f53bb19b640 10 monclient(hunting): authenticate will time out at 746.991818s 2024-09-17T18:19:03.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.275+0000 7f53b9738640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53b413e1a0 0x7f53b4138210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7f53b9738640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53b413e1a0 0x7f53b4138210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:59436/0 (socket says 172.21.15.121:59436) 2024-09-17T18:19:03.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7f53b8f37640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53b40a5ca0 0x7f53b40a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7f53b9738640 1 -- 172.21.15.121:0/811428999 learned_addr learned my addr 172.21.15.121:0/811428999 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:03.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7f53b8f37640 10 monclient(hunting): get_auth_request con 0x7f53b40a5ca0 auth_method 0 2024-09-17T18:19:03.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7ff049d9b640 1 -- 172.21.15.121:0/1472694789 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff04c0a65c0 msgr2=0x7ff04c1380d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:03.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7ff049d9b640 1 --2- 172.21.15.121:0/1472694789 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff04c0a65c0 0x7ff04c1380d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff03c02a600 tx=0x7ff03c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:03.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7ff042ffd640 10 client.?.objecter ms_handle_connect 0x7ff04c0a65c0 2024-09-17T18:19:03.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7ff042ffd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:19:03.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7f53b8f37640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:03.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7f53b8f37640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:03.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7f53b8f37640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:03.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7f53b3fff640 1 --2- 172.21.15.121:0/811428999 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53b40a65c0 0x7f53b4137cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7f53b3fff640 10 monclient(hunting): get_auth_request con 0x7f53b40a65c0 auth_method 0 2024-09-17T18:19:03.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7f53b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:03.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7f53b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:03.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7f53b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:03.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7f53b8f37640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:03.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7f53b8f37640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:03.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7f53b8f37640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:03.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7f53b9738640 10 monclient(hunting): get_auth_request con 0x7f53b413e1a0 auth_method 0 2024-09-17T18:19:03.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7f53b9738640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:03.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7f53b9738640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:03.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7f53b9738640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:03.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7f53b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:03.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7f53b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:03.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7f53b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:03.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7ff042ffd640 1 -- 172.21.15.121:0/1472694789 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff03c037070 con 0x7ff04c0a65c0 2024-09-17T18:19:03.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7ff042ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:03.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7f53b9738640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:03.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7f53b9738640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:03.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7f53b9738640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:03.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7f53b8f37640 10 monclient(hunting): handle_auth_done global_id 4272 payload 824 2024-09-17T18:19:03.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7f53b8f37640 1 -- 172.21.15.121:0/811428999 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53b413e1a0 msgr2=0x7f53b4138210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7f53b8f37640 1 --2- 172.21.15.121:0/811428999 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53b413e1a0 0x7f53b4138210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7f53b8f37640 1 -- 172.21.15.121:0/811428999 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53b40a65c0 msgr2=0x7f53b4137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7f53b8f37640 1 --2- 172.21.15.121:0/811428999 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53b40a65c0 0x7f53b4137cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7f53b8f37640 10 monclient: _finish_hunting 0 2024-09-17T18:19:03.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7f53b8f37640 1 monclient: found mon.noname-a 2024-09-17T18:19:03.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7ff042ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:19:03.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7ff042ffd640 10 monclient: dump: 2024-09-17T18:19:03.341 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:03.341 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.341 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.341 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.341 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:03.341 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.342 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:03.342 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:03.342 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:03.342 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7ff042ffd640 10 monclient: _finish_auth 0 2024-09-17T18:19:03.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7ff042ffd640 10 monclient: _check_auth_tickets 2024-09-17T18:19:03.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7ff042ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:03.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7ff050d3d640 5 monclient: authenticate success, global_id 4247 2024-09-17T18:19:03.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7f53b8f37640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:03.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7f53b8f37640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:19:03.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7f53b8f37640 1 -- 172.21.15.121:0/811428999 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53b41246a0 con 0x7f53b40a5ca0 2024-09-17T18:19:03.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7ff050d3d640 10 monclient: _renew_subs 2024-09-17T18:19:03.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7ff050d3d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:03.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7ff050d3d640 1 -- 172.21.15.121:0/1472694789 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff04c1ce240 con 0x7ff04c0a65c0 2024-09-17T18:19:03.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7ff050d3d640 10 client.4247.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:03.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7ff050d3d640 10 monclient: _renew_subs 2024-09-17T18:19:03.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7ff050d3d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:03.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7ff050d3d640 1 -- 172.21.15.121:0/1472694789 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff04c1254d0 con 0x7ff04c0a65c0 2024-09-17T18:19:03.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7ff050d3d640 1 librados: init done 2024-09-17T18:19:03.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7ff042ffd640 1 -- 172.21.15.121:0/1472694789 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff03c038040 con 0x7ff04c0a65c0 2024-09-17T18:19:03.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7ff042ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:03.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7ff042ffd640 1 -- 172.21.15.121:0/1472694789 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff03c004490 con 0x7ff04c0a65c0 2024-09-17T18:19:03.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7ff042ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:03.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7ff042ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:19:03.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7ff042ffd640 10 monclient: dump: 2024-09-17T18:19:03.346 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:03.346 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.346 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.346 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.346 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:03.346 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.346 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:03.346 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:03.346 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:03.346 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.276+0000 7ff048d99640 4 set_mon_vals no callback set 2024-09-17T18:19:03.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.277+0000 7ff042ffd640 1 -- 172.21.15.121:0/1472694789 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7ff03c03c8e0 con 0x7ff04c0a65c0 2024-09-17T18:19:03.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.277+0000 7ff050d3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:03.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.277+0000 7ff050d3d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:03.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.277+0000 7ff050d3d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:03.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.277+0000 7ff042ffd640 1 --2- 172.21.15.121:0/1472694789 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff02c0401a0 0x7ff02c042660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.277+0000 7ff050d3d640 1 -- 172.21.15.121:0/1472694789 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff04c0a8720 con 0x7ff04c0a65c0 2024-09-17T18:19:03.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.277+0000 7ff042ffd640 1 -- 172.21.15.121:0/1472694789 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 3562+0+0 (secure 0 0 0) 0x7ff03c073840 con 0x7ff04c0a65c0 2024-09-17T18:19:03.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.277+0000 7ff042ffd640 10 client.4247.objecter ms_dispatch 0x7ff04c000b90 osd_map(16..16 src has 1..16) 2024-09-17T18:19:03.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.277+0000 7ff042ffd640 3 client.4247.objecter handle_osd_map got epochs [16,16] > 0 2024-09-17T18:19:03.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.277+0000 7ff042ffd640 3 client.4247.objecter handle_osd_map decoding full epoch 16 2024-09-17T18:19:03.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.277+0000 7ff042ffd640 20 client.4247.objecter dump_active .. 0 homeless 2024-09-17T18:19:03.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.278+0000 7ff04a59c640 1 --2- 172.21.15.121:0/1472694789 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff02c0401a0 0x7ff02c042660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.278+0000 7ff04a59c640 10 monclient: get_auth_request con 0x7ff02c0401a0 auth_method 0 2024-09-17T18:19:03.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.278+0000 7f37c8e4c640 1 --2- 172.21.15.121:0/1226608797 >> v2:172.21.15.173:6812/2423786037 conn(0x7f37a80401c0 0x7f37a8042680 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f37b80090b0 tx=0x7f37b802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:03.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.278+0000 7f37c17fa640 10 client.4258.objecter ms_handle_connect 0x7f37a80401c0 2024-09-17T18:19:03.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.278+0000 7f53b8f37640 1 -- 172.21.15.121:0/811428999 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53b40a5ca0 msgr2=0x7f53b40a0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:03.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.278+0000 7f53b8f37640 1 --2- 172.21.15.121:0/811428999 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53b40a5ca0 0x7f53b40a0160 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f53b40706d0 tx=0x7f53a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:03.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.278+0000 7f53b17fa640 10 client.?.objecter ms_handle_connect 0x7f53b40a5ca0 2024-09-17T18:19:03.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.278+0000 7f53b17fa640 10 client.?.objecter resend_mon_ops 2024-09-17T18:19:03.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.279+0000 7f53b17fa640 1 -- 172.21.15.121:0/811428999 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a8041020 con 0x7f53b40a5ca0 2024-09-17T18:19:03.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.279+0000 7f53b17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:03.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.279+0000 7f53b17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:19:03.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.279+0000 7f53b17fa640 10 monclient: dump: 2024-09-17T18:19:03.350 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:03.350 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.350 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.350 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.350 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:03.350 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.350 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:03.350 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:03.350 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:03.350 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.279+0000 7f53b17fa640 10 monclient: _finish_auth 0 2024-09-17T18:19:03.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.279+0000 7f53b17fa640 10 monclient: _check_auth_tickets 2024-09-17T18:19:03.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.279+0000 7f53b17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:03.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.279+0000 7f53b17fa640 1 -- 172.21.15.121:0/811428999 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53a803d850 con 0x7f53b40a5ca0 2024-09-17T18:19:03.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.279+0000 7f53b17fa640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:03.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.279+0000 7f53b17fa640 1 -- 172.21.15.121:0/811428999 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a8037040 con 0x7f53b40a5ca0 2024-09-17T18:19:03.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.279+0000 7f53b17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:03.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.279+0000 7f53b17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:19:03.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.279+0000 7f53b17fa640 10 monclient: dump: 2024-09-17T18:19:03.351 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:03.351 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.351 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.351 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.351 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:03.351 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.351 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:03.352 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:03.352 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:03.352 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.279+0000 7f53bb19b640 5 monclient: authenticate success, global_id 4272 2024-09-17T18:19:03.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.279+0000 7f53bb19b640 10 monclient: _renew_subs 2024-09-17T18:19:03.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.279+0000 7f53bb19b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:03.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.279+0000 7f53bb19b640 1 -- 172.21.15.121:0/811428999 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53b40a2490 con 0x7f53b40a5ca0 2024-09-17T18:19:03.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.279+0000 7f53b2ffd640 4 set_mon_vals no callback set 2024-09-17T18:19:03.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.279+0000 7f53bb19b640 10 client.4272.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:03.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.279+0000 7f53bb19b640 10 monclient: _renew_subs 2024-09-17T18:19:03.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.279+0000 7f53bb19b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:03.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.279+0000 7f53bb19b640 1 -- 172.21.15.121:0/811428999 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53b41243c0 con 0x7f53b40a5ca0 2024-09-17T18:19:03.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.279+0000 7f53bb19b640 1 librados: init done 2024-09-17T18:19:03.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.280+0000 7f53bb19b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:03.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.280+0000 7f53bb19b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:03.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.280+0000 7f53bb19b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:03.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.280+0000 7f53bb19b640 1 -- 172.21.15.121:0/811428999 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53b41241c0 con 0x7f53b40a5ca0 2024-09-17T18:19:03.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.280+0000 7f37c17fa640 1 -- 172.21.15.121:0/1226608797 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f37bc022070 con 0x7f37c409ec60 2024-09-17T18:19:03.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.280+0000 7f37c17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:03.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.280+0000 7f37c17fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:19:03.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.280+0000 7ff04a59c640 1 --2- 172.21.15.121:0/1472694789 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff02c0401a0 0x7ff02c042660 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff034009f30 tx=0x7ff034009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:03.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.281+0000 7ff042ffd640 10 client.4247.objecter ms_handle_connect 0x7ff02c0401a0 2024-09-17T18:19:03.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.282+0000 7f53b17fa640 1 -- 172.21.15.121:0/811428999 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f53a8044020 con 0x7f53b40a5ca0 2024-09-17T18:19:03.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.282+0000 7f53b17fa640 1 --2- 172.21.15.121:0/811428999 >> v2:172.21.15.173:6812/2423786037 conn(0x7f53980401c0 0x7f5398042680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.282+0000 7f53b17fa640 1 -- 172.21.15.121:0/811428999 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 3562+0+0 (secure 0 0 0) 0x7f53a8073450 con 0x7f53b40a5ca0 2024-09-17T18:19:03.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.282+0000 7f53b17fa640 10 client.4272.objecter ms_dispatch 0x7f53b4000b90 osd_map(16..16 src has 1..16) 2024-09-17T18:19:03.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.282+0000 7f53b17fa640 3 client.4272.objecter handle_osd_map got epochs [16,16] > 0 2024-09-17T18:19:03.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.282+0000 7f53b17fa640 3 client.4272.objecter handle_osd_map decoding full epoch 16 2024-09-17T18:19:03.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.282+0000 7f53b17fa640 20 client.4272.objecter dump_active .. 0 homeless 2024-09-17T18:19:03.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.282+0000 7f53b3fff640 1 --2- 172.21.15.121:0/811428999 >> v2:172.21.15.173:6812/2423786037 conn(0x7f53980401c0 0x7f5398042680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.282+0000 7f53b3fff640 10 monclient: get_auth_request con 0x7f53980401c0 auth_method 0 2024-09-17T18:19:03.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.285+0000 7f53b3fff640 1 --2- 172.21.15.121:0/811428999 >> v2:172.21.15.173:6812/2423786037 conn(0x7f53980401c0 0x7f5398042680 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f53b4138c40 tx=0x7f53a0007e30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:03.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.285+0000 7f53b17fa640 10 client.4272.objecter ms_handle_connect 0x7f53980401c0 2024-09-17T18:19:03.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.286+0000 7ff042ffd640 1 -- 172.21.15.121:0/1472694789 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff03c03b360 con 0x7ff04c0a65c0 2024-09-17T18:19:03.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.286+0000 7ff042ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:03.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.286+0000 7ff042ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:19:03.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.289+0000 7f53b17fa640 1 -- 172.21.15.121:0/811428999 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f53a804b030 con 0x7f53b40a5ca0 2024-09-17T18:19:03.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.289+0000 7f53b17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:03.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.289+0000 7f53b17fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:19:03.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.293+0000 7fe5e1764640 10 monclient: get_monmap_and_config 2024-09-17T18:19:03.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.293+0000 7fe5e1764640 10 monclient: build_initial_monmap 2024-09-17T18:19:03.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.293+0000 7fe5e1764640 10 monclient: monmap: 2024-09-17T18:19:03.360 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:03.360 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.360 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:19:03.293937+0000 2024-09-17T18:19:03.360 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:19:03.293937+0000 2024-09-17T18:19:03.360 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:03.360 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.360 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:19:03.360 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:19:03.360 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:19:03.360 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:19:03.360 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:19:03.360 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.294+0000 7fe5e1764640 1 Processor -- start 2024-09-17T18:19:03.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.294+0000 7fe5e1764640 1 -- start start 2024-09-17T18:19:03.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.294+0000 7fe5e1764640 10 monclient: init 2024-09-17T18:19:03.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.294+0000 7fe5e1764640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:03.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.294+0000 7fe5e1764640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-17T18:19:03.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.294+0000 7fe5e1764640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5dc0a5ca0 0x7fe5dc0a6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.294+0000 7fe5e1764640 10 monclient(hunting): picked mon.noname-b con 0x7fe5dc0a5ca0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:03.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.294+0000 7fe5e1764640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe5dc0a65c0 addr v1:172.21.15.173:6789/0 2024-09-17T18:19:03.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.294+0000 7fe5e1764640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fe5dc13e1a0 0x7fe5dc140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.294+0000 7fe5e1764640 10 monclient(hunting): picked mon.noname-c con 0x7fe5dc13e1a0 addr v2:172.21.15.173:3300/0 2024-09-17T18:19:03.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.294+0000 7fe5e1764640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5dc0a7b50 con 0x7fe5dc0a65c0 2024-09-17T18:19:03.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.294+0000 7fe5e1764640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.294+0000 7fe5e1764640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fe5dc0704c0 con 0x7fe5dc13e1a0 2024-09-17T18:19:03.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.294+0000 7fe5e1764640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.294+0000 7fe5e1764640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe5dc0a5490 con 0x7fe5dc0a5ca0 2024-09-17T18:19:03.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.294+0000 7fe5e1764640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:03.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.294+0000 7fe5e1764640 10 monclient(hunting): authenticate will time out at 747.010785s 2024-09-17T18:19:03.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.294+0000 7fe5daffd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5dc0a5ca0 0x7fe5dc0a6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.294+0000 7fe5daffd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5dc0a5ca0 0x7fe5dc0a6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:59448/0 (socket says 172.21.15.121:59448) 2024-09-17T18:19:03.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.294+0000 7fe5daffd640 1 -- 172.21.15.121:0/188328392 learned_addr learned my addr 172.21.15.121:0/188328392 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:03.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.294+0000 7fe5daffd640 10 monclient(hunting): get_auth_request con 0x7fe5dc0a5ca0 auth_method 0 2024-09-17T18:19:03.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.294+0000 7fe5daffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:03.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.294+0000 7fe5daffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:03.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.294+0000 7fe5daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:03.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.295+0000 7fe5daffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:03.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.295+0000 7fe5daffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:03.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.295+0000 7fe5daffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:03.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.295+0000 7fe5db7fe640 1 --2- 172.21.15.121:0/188328392 >> v2:172.21.15.173:3300/0 conn(0x7fe5dc13e1a0 0x7fe5dc140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.295+0000 7fe5db7fe640 10 monclient(hunting): get_auth_request con 0x7fe5dc13e1a0 auth_method 0 2024-09-17T18:19:03.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.295+0000 7fe5db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:03.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.295+0000 7fe5db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:03.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.295+0000 7fe5db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:03.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.295+0000 7fe5db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:03.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.295+0000 7fe5db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:03.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.295+0000 7fe5db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:03.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.295+0000 7fe5daffd640 10 monclient(hunting): handle_auth_done global_id 4253 payload 274 2024-09-17T18:19:03.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.295+0000 7fe5daffd640 1 -- 172.21.15.121:0/188328392 >> v2:172.21.15.173:3300/0 conn(0x7fe5dc13e1a0 msgr2=0x7fe5dc140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:03.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.295+0000 7fe5daffd640 1 --2- 172.21.15.121:0/188328392 >> v2:172.21.15.173:3300/0 conn(0x7fe5dc13e1a0 0x7fe5dc140590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.295+0000 7fe5daffd640 1 -- 172.21.15.121:0/188328392 >> v1:172.21.15.173:6789/0 conn(0x7fe5dc0a65c0 legacy=0x7fe5dc13ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:03.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.295+0000 7fe5daffd640 10 monclient: _finish_hunting 0 2024-09-17T18:19:03.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.295+0000 7fe5daffd640 1 monclient: found mon.noname-b 2024-09-17T18:19:03.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.295+0000 7fe5daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:03.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.295+0000 7fe5daffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:19:03.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.295+0000 7fe5daffd640 1 -- 172.21.15.121:0/188328392 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5dc0a2a50 con 0x7fe5dc0a5ca0 2024-09-17T18:19:03.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.296+0000 7f072b9d3640 10 monclient: get_monmap_and_config 2024-09-17T18:19:03.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.296+0000 7f072b9d3640 10 monclient: build_initial_monmap 2024-09-17T18:19:03.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.296+0000 7f072b9d3640 10 monclient: monmap: 2024-09-17T18:19:03.365 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:03.365 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.365 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:19:03.297414+0000 2024-09-17T18:19:03.365 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:19:03.297414+0000 2024-09-17T18:19:03.365 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:03.366 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.366 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:19:03.366 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:19:03.366 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:19:03.366 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:19:03.366 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:19:03.366 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.297+0000 7f072b9d3640 1 Processor -- start 2024-09-17T18:19:03.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.297+0000 7f072b9d3640 1 -- start start 2024-09-17T18:19:03.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.297+0000 7f072b9d3640 10 monclient: init 2024-09-17T18:19:03.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7f072b9d3640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:03.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7f072b9d3640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-17T18:19:03.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7f072b9d3640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f07240a5ca0 0x7f07240a6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7f072b9d3640 10 monclient(hunting): picked mon.noname-c con 0x7f07240a5ca0 addr v2:172.21.15.173:3300/0 2024-09-17T18:19:03.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7f072b9d3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f07240a65c0 addr v1:172.21.15.173:6789/0 2024-09-17T18:19:03.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7f072b9d3640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f072413e1a0 0x7f0724140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7f072b9d3640 10 monclient(hunting): picked mon.noname-a con 0x7f072413e1a0 addr v2:172.21.15.121:3300/0 2024-09-17T18:19:03.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7f072b9d3640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07240a7b50 con 0x7f07240a65c0 2024-09-17T18:19:03.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7f072b9d3640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7f072b9d3640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f07240704c0 con 0x7f072413e1a0 2024-09-17T18:19:03.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7f072b9d3640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7f072b9d3640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f07240a5490 con 0x7f07240a5ca0 2024-09-17T18:19:03.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7f072b9d3640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:03.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7f072b9d3640 10 monclient(hunting): authenticate will time out at 747.014392s 2024-09-17T18:19:03.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7f0729f70640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f072413e1a0 0x7f0724140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7f072976f640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f07240a5ca0 0x7f07240a6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7f0729f70640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f072413e1a0 0x7f0724140590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40984/0 (socket says 172.21.15.121:40984) 2024-09-17T18:19:03.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7f0729f70640 1 -- 172.21.15.121:0/3127541977 learned_addr learned my addr 172.21.15.121:0/3127541977 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:03.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7f0728f6e640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f07240a65c0 0x7f072413ca80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.121:40298/0 (socket says 172.21.15.121:40298) 2024-09-17T18:19:03.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7f0729f70640 10 monclient(hunting): get_auth_request con 0x7f072413e1a0 auth_method 0 2024-09-17T18:19:03.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7f0729f70640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:03.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7f0729f70640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:03.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7f0729f70640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:03.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7f0729f70640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:03.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7f0729f70640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:03.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7f0729f70640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:03.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7f072976f640 10 monclient(hunting): get_auth_request con 0x7f07240a5ca0 auth_method 0 2024-09-17T18:19:03.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7f072976f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:03.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7f072976f640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:03.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7f072976f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:03.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7fe5daffd640 1 -- 172.21.15.121:0/188328392 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5dc0a5ca0 msgr2=0x7fe5dc0a6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:03.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7f072976f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:03.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7f072976f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:03.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7f072976f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:03.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7fe5daffd640 1 --2- 172.21.15.121:0/188328392 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5dc0a5ca0 0x7fe5dc0a6080 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc0a3dd0 tx=0x7fe5cc02a080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e0b14a9bac80c25 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:03.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7f0729f70640 10 monclient(hunting): handle_auth_done global_id 4275 payload 274 2024-09-17T18:19:03.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7f0729f70640 1 -- 172.21.15.121:0/3127541977 >> v2:172.21.15.173:3300/0 conn(0x7f07240a5ca0 msgr2=0x7f07240a6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:03.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7f0729f70640 1 --2- 172.21.15.121:0/3127541977 >> v2:172.21.15.173:3300/0 conn(0x7f07240a5ca0 0x7f07240a6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7f0729f70640 1 -- 172.21.15.121:0/3127541977 >> v1:172.21.15.173:6789/0 conn(0x7f07240a65c0 legacy=0x7f072413ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:03.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7f0729f70640 10 monclient: _finish_hunting 0 2024-09-17T18:19:03.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7f0729f70640 1 monclient: found mon.noname-a 2024-09-17T18:19:03.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7f0729f70640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:03.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7f0729f70640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:19:03.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.298+0000 7f0729f70640 1 -- 172.21.15.121:0/3127541977 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07240a2a50 con 0x7f072413e1a0 2024-09-17T18:19:03.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.299+0000 7fe5d9ffb640 1 -- 172.21.15.121:0/188328392 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5cc037070 con 0x7fe5dc0a5ca0 2024-09-17T18:19:03.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.299+0000 7fe5d9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:03.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.299+0000 7fe5d9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:19:03.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.299+0000 7fe5d9ffb640 10 monclient: dump: 2024-09-17T18:19:03.374 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:03.374 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.374 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.374 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.374 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:03.374 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.374 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:03.374 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:03.374 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:03.374 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.299+0000 7fe5d9ffb640 10 monclient: _finish_auth 0 2024-09-17T18:19:03.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.299+0000 7fe5d9ffb640 10 monclient: _check_auth_tickets 2024-09-17T18:19:03.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.299+0000 7fe5d9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:03.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.299+0000 7fe5d9ffb640 1 -- 172.21.15.121:0/188328392 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5cc032440 con 0x7fe5dc0a5ca0 2024-09-17T18:19:03.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.299+0000 7fe5d9ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:03.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.299+0000 7fe5d9ffb640 1 -- 172.21.15.121:0/188328392 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5cc0327f0 con 0x7fe5dc0a5ca0 2024-09-17T18:19:03.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.299+0000 7fe5d9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:03.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.299+0000 7fe5d9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:19:03.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.299+0000 7fe5d9ffb640 10 monclient: dump: 2024-09-17T18:19:03.375 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:03.375 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.375 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.375 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.375 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:03.375 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.375 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:03.376 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:03.376 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:03.376 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.299+0000 7fe5e1764640 5 monclient: authenticate success, global_id 4253 2024-09-17T18:19:03.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.299+0000 7fe5e1764640 10 monclient: get_monmap_and_config success 2024-09-17T18:19:03.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.299+0000 7fe5e1764640 4 set_mon_vals no callback set 2024-09-17T18:19:03.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.300+0000 7fe5e1764640 10 monclient: shutdown 2024-09-17T18:19:03.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.300+0000 7fe5e1764640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:03.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.300+0000 7fe5e1764640 1 -- 172.21.15.121:0/188328392 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5dc0a5ca0 msgr2=0x7fe5dc0a6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.300+0000 7fe5e1764640 1 --2- 172.21.15.121:0/188328392 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5dc0a5ca0 0x7fe5dc0a6080 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc0a3dd0 tx=0x7fe5cc02a080 comp rx=0 tx=0).stop 2024-09-17T18:19:03.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.300+0000 7fe5e1764640 1 -- 172.21.15.121:0/188328392 shutdown_connections 2024-09-17T18:19:03.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.300+0000 7fe5e1764640 1 --2- 172.21.15.121:0/188328392 >> v2:172.21.15.173:3300/0 conn(0x7fe5dc13e1a0 0x7fe5dc140590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.300+0000 7fe5e1764640 1 --2- 172.21.15.121:0/188328392 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5dc0a5ca0 0x7fe5dc0a6080 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.300+0000 7fe5e1764640 1 -- 172.21.15.121:0/188328392 >> 172.21.15.121:0/188328392 conn(0x7fe5dc06dd10 msgr2=0x7fe5dc0a7da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:03.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.300+0000 7fe5e1764640 1 -- 172.21.15.121:0/188328392 shutdown_connections 2024-09-17T18:19:03.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.300+0000 7fe5e1764640 1 -- 172.21.15.121:0/188328392 wait complete. 2024-09-17T18:19:03.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.301+0000 7fe5e1764640 10 monclient: build_initial_monmap 2024-09-17T18:19:03.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.301+0000 7fe5e1764640 10 monclient: monmap: 2024-09-17T18:19:03.377 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:03.377 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:19:03.377 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:19:03.377 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:19:03.377 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:03.377 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.377 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:19:03.377 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:19:03.381 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:19:03.381 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.301+0000 7fe5e1764640 1 librados: starting msgr at 2024-09-17T18:19:03.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.301+0000 7fe5e1764640 1 librados: starting objecter 2024-09-17T18:19:03.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.302+0000 7fe5e1764640 1 Processor -- start 2024-09-17T18:19:03.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.302+0000 7f0729f70640 1 -- 172.21.15.121:0/3127541977 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f072413e1a0 msgr2=0x7f0724140590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:03.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.302+0000 7f0729f70640 1 --2- 172.21.15.121:0/3127541977 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f072413e1a0 0x7f0724140590 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f072406c260 tx=0x7f071c02a5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a6919400a6e0c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:03.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.302+0000 7fe5e1764640 1 -- start start 2024-09-17T18:19:03.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.302+0000 7fe5e1764640 1 librados: setting wanted keys 2024-09-17T18:19:03.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.302+0000 7fe5e1764640 1 librados: calling monclient init 2024-09-17T18:19:03.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.302+0000 7fe5e1764640 10 monclient: init 2024-09-17T18:19:03.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.302+0000 7f071bfff640 1 -- 172.21.15.121:0/3127541977 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f071c036070 con 0x7f072413e1a0 2024-09-17T18:19:03.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.302+0000 7f071bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:03.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.302+0000 7f071bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:19:03.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.302+0000 7f071bfff640 10 monclient: dump: 2024-09-17T18:19:03.383 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:03.384 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.384 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.384 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.384 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:03.384 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.384 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:03.384 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:03.384 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:03.384 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.302+0000 7f071bfff640 10 monclient: _finish_auth 0 2024-09-17T18:19:03.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.302+0000 7f071bfff640 10 monclient: _check_auth_tickets 2024-09-17T18:19:03.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.302+0000 7f071bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:03.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.302+0000 7f071bfff640 1 -- 172.21.15.121:0/3127541977 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f071c007800 con 0x7f072413e1a0 2024-09-17T18:19:03.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.302+0000 7f072b9d3640 5 monclient: authenticate success, global_id 4275 2024-09-17T18:19:03.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.302+0000 7f071bfff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:03.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.302+0000 7f071bfff640 1 -- 172.21.15.121:0/3127541977 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f071c007bd0 con 0x7f072413e1a0 2024-09-17T18:19:03.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.302+0000 7f071bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:03.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.302+0000 7f071bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:19:03.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.302+0000 7f071bfff640 10 monclient: dump: 2024-09-17T18:19:03.385 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:03.385 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.385 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.385 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.385 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:03.385 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.385 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:03.385 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:03.386 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:03.386 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.302+0000 7f072b9d3640 10 monclient: get_monmap_and_config success 2024-09-17T18:19:03.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.302+0000 7f072b9d3640 4 set_mon_vals no callback set 2024-09-17T18:19:03.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.303+0000 7fe5e1764640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:03.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.303+0000 7fe5e1764640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:19:03.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.303+0000 7f072b9d3640 10 monclient: shutdown 2024-09-17T18:19:03.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.303+0000 7fe5e1764640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5dc0a65c0 0x7fe5dc0a0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.303+0000 7f072b9d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:03.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.303+0000 7f072b9d3640 1 -- 172.21.15.121:0/3127541977 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f072413e1a0 msgr2=0x7f0724140590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.303+0000 7f072b9d3640 1 --2- 172.21.15.121:0/3127541977 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f072413e1a0 0x7f0724140590 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f072406c260 tx=0x7f071c02a5b0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.303+0000 7f072b9d3640 1 -- 172.21.15.121:0/3127541977 shutdown_connections 2024-09-17T18:19:03.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.303+0000 7f072b9d3640 1 --2- 172.21.15.121:0/3127541977 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f072413e1a0 0x7f0724140590 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.303+0000 7f072b9d3640 1 --2- 172.21.15.121:0/3127541977 >> v2:172.21.15.173:3300/0 conn(0x7f07240a5ca0 0x7f07240a6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.303+0000 7f072b9d3640 1 -- 172.21.15.121:0/3127541977 >> 172.21.15.121:0/3127541977 conn(0x7f072406dd10 msgr2=0x7f07240a7da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:03.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.303+0000 7f072b9d3640 1 -- 172.21.15.121:0/3127541977 shutdown_connections 2024-09-17T18:19:03.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.303+0000 7f072b9d3640 1 -- 172.21.15.121:0/3127541977 wait complete. 2024-09-17T18:19:03.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.303+0000 7fe5daffd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5dc0a65c0 0x7fe5dc0a0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.303+0000 7fe5daffd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5dc0a65c0 0x7fe5dc0a0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40992/0 (socket says 172.21.15.121:40992) 2024-09-17T18:19:03.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.303+0000 7fe5daffd640 1 -- 172.21.15.121:0/3826817118 learned_addr learned my addr 172.21.15.121:0/3826817118 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:03.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.303+0000 7fe5e1764640 10 monclient(hunting): picked mon.noname-a con 0x7fe5dc0a65c0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:03.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.303+0000 7fe5e1764640 1 --2- 172.21.15.121:0/3826817118 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5dc13e1a0 0x7fe5dc137f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.303+0000 7fe5e1764640 10 monclient(hunting): picked mon.noname-b con 0x7fe5dc13e1a0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:03.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.303+0000 7fe5e1764640 1 --2- 172.21.15.121:0/3826817118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5dc1384c0 0x7fe5dc1cd790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.303+0000 7fe5e1764640 10 monclient(hunting): picked mon.noname-c con 0x7fe5dc1384c0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:03.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.303+0000 7fe5e1764640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.303+0000 7fe5e1764640 1 -- 172.21.15.121:0/3826817118 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe5dc0a5670 con 0x7fe5dc0a65c0 2024-09-17T18:19:03.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.303+0000 7fe5db7fe640 1 --2- 172.21.15.121:0/3826817118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5dc1384c0 0x7fe5dc1cd790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.304+0000 7fe5e1764640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.304+0000 7fe5e1764640 1 -- 172.21.15.121:0/3826817118 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe5dc0a6eb0 con 0x7fe5dc13e1a0 2024-09-17T18:19:03.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.304+0000 7fe5da7fc640 1 --2- 172.21.15.121:0/3826817118 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5dc13e1a0 0x7fe5dc137f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.304+0000 7fe5e1764640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.304+0000 7fe5e1764640 1 -- 172.21.15.121:0/3826817118 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe5dc0a69a0 con 0x7fe5dc1384c0 2024-09-17T18:19:03.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.304+0000 7fe5e1764640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:03.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.304+0000 7fe5e1764640 10 monclient(hunting): authenticate will time out at 747.020110s 2024-09-17T18:19:03.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.304+0000 7f072b9d3640 10 monclient: build_initial_monmap 2024-09-17T18:19:03.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.304+0000 7f072b9d3640 10 monclient: monmap: 2024-09-17T18:19:03.392 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:03.392 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:19:03.392 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:19:03.392 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:19:03.392 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:03.392 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.392 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:19:03.392 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:19:03.392 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:19:03.392 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.304+0000 7fe5daffd640 10 monclient(hunting): get_auth_request con 0x7fe5dc0a65c0 auth_method 0 2024-09-17T18:19:03.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.304+0000 7fe5daffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:03.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.304+0000 7fe5daffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:03.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.304+0000 7fe5daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:03.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.304+0000 7fe5db7fe640 10 monclient(hunting): get_auth_request con 0x7fe5dc1384c0 auth_method 0 2024-09-17T18:19:03.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.304+0000 7fe5db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:03.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.304+0000 7fe5db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:03.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.304+0000 7fe5db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:03.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.304+0000 7fe5da7fc640 10 monclient(hunting): get_auth_request con 0x7fe5dc13e1a0 auth_method 0 2024-09-17T18:19:03.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.304+0000 7fe5da7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:03.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.304+0000 7fe5da7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:03.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.304+0000 7fe5da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:03.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.304+0000 7fe5db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:03.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.304+0000 7fe5db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:03.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.304+0000 7fe5db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:03.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.304+0000 7fe5db7fe640 10 monclient(hunting): handle_auth_done global_id 4256 payload 824 2024-09-17T18:19:03.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.304+0000 7fe5db7fe640 1 -- 172.21.15.121:0/3826817118 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5dc13e1a0 msgr2=0x7fe5dc137f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.304+0000 7fe5db7fe640 1 --2- 172.21.15.121:0/3826817118 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5dc13e1a0 0x7fe5dc137f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.304+0000 7fe5db7fe640 1 -- 172.21.15.121:0/3826817118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5dc0a65c0 msgr2=0x7fe5dc0a0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.304+0000 7fe5db7fe640 1 --2- 172.21.15.121:0/3826817118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5dc0a65c0 0x7fe5dc0a0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.304+0000 7fe5db7fe640 10 monclient: _finish_hunting 0 2024-09-17T18:19:03.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.304+0000 7fe5db7fe640 1 monclient: found mon.noname-c 2024-09-17T18:19:03.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.304+0000 7fe5db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:03.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.304+0000 7fe5db7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:19:03.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.304+0000 7fe5db7fe640 1 -- 172.21.15.121:0/3826817118 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5dc124950 con 0x7fe5dc1384c0 2024-09-17T18:19:03.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.304+0000 7fe5daffd640 1 --2- 172.21.15.121:0/3826817118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5dc0a65c0 0x7fe5dc0a0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T18:19:03.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.305+0000 7f072b9d3640 1 librados: starting msgr at 2024-09-17T18:19:03.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.305+0000 7f072b9d3640 1 librados: starting objecter 2024-09-17T18:19:03.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.305+0000 7f072b9d3640 1 Processor -- start 2024-09-17T18:19:03.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.305+0000 7f072b9d3640 1 -- start start 2024-09-17T18:19:03.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.305+0000 7f072b9d3640 1 librados: setting wanted keys 2024-09-17T18:19:03.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.305+0000 7f072b9d3640 1 librados: calling monclient init 2024-09-17T18:19:03.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.305+0000 7f072b9d3640 10 monclient: init 2024-09-17T18:19:03.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.306+0000 7f072b9d3640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:03.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.306+0000 7f072b9d3640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T18:19:03.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.306+0000 7f072b9d3640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f07240a5ca0 0x7f0724122dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.306+0000 7f072b9d3640 10 monclient(hunting): picked mon.noname-c con 0x7f07240a5ca0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:03.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.306+0000 7f072b9d3640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07240a65c0 0x7f07241218f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.306+0000 7f072b9d3640 10 monclient(hunting): picked mon.noname-b con 0x7f07240a65c0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:03.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.306+0000 7f072b9d3640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f072413e1a0 0x7f0724137c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.306+0000 7f072b9d3640 10 monclient(hunting): picked mon.noname-a con 0x7f072413e1a0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:03.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.306+0000 7f072b9d3640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.306+0000 7f072b9d3640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f07240a5670 con 0x7f072413e1a0 2024-09-17T18:19:03.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.306+0000 7f072b9d3640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.306+0000 7f072b9d3640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f07240a6eb0 con 0x7f07240a65c0 2024-09-17T18:19:03.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.306+0000 7f0728f6e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07240a65c0 0x7f07241218f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.306+0000 7f0729f70640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f072413e1a0 0x7f0724137c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.306+0000 7f072b9d3640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.306+0000 7f0728f6e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07240a65c0 0x7f07241218f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.121:42190/0 (socket says 172.21.15.121:42190) 2024-09-17T18:19:03.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.306+0000 7f0729f70640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f072413e1a0 0x7f0724137c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41002/0 (socket says 172.21.15.121:41002) 2024-09-17T18:19:03.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.306+0000 7f0728f6e640 1 -- 172.21.15.121:0/2404947717 learned_addr learned my addr 172.21.15.121:0/2404947717 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:03.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.306+0000 7f072b9d3640 1 -- 172.21.15.121:0/2404947717 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f07240a69d0 con 0x7f07240a5ca0 2024-09-17T18:19:03.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.306+0000 7f072b9d3640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:03.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.306+0000 7f072b9d3640 10 monclient(hunting): authenticate will time out at 747.022789s 2024-09-17T18:19:03.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.306+0000 7f0728f6e640 10 monclient(hunting): get_auth_request con 0x7f07240a65c0 auth_method 0 2024-09-17T18:19:03.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.306+0000 7f0728f6e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:03.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.306+0000 7f0728f6e640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:03.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.306+0000 7f0728f6e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:03.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.306+0000 7f072976f640 1 --2- 172.21.15.121:0/2404947717 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f07240a5ca0 0x7f0724122dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.306+0000 7fe5db7fe640 1 -- 172.21.15.121:0/3826817118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5dc1384c0 msgr2=0x7fe5dc1cd790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:03.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.306+0000 7f0728f6e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:03.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.306+0000 7f0728f6e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:03.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.306+0000 7f0728f6e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:03.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.306+0000 7fe5db7fe640 1 --2- 172.21.15.121:0/3826817118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5dc1384c0 0x7fe5dc1cd790 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe5d4009d90 tx=0x7fe5d400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:03.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.306+0000 7fe5c37fe640 10 client.?.objecter ms_handle_connect 0x7fe5dc1384c0 2024-09-17T18:19:03.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.306+0000 7fe5c37fe640 10 client.?.objecter resend_mon_ops 2024-09-17T18:19:03.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.307+0000 7f072976f640 10 monclient(hunting): get_auth_request con 0x7f07240a5ca0 auth_method 0 2024-09-17T18:19:03.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.307+0000 7f072976f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:03.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.307+0000 7f072976f640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:03.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.307+0000 7f072976f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:03.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.307+0000 7f072976f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:03.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.307+0000 7f072976f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:03.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.307+0000 7f072976f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:03.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.307+0000 7f072976f640 10 monclient(hunting): handle_auth_done global_id 4259 payload 824 2024-09-17T18:19:03.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.307+0000 7f072976f640 1 -- 172.21.15.121:0/2404947717 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07240a65c0 msgr2=0x7f07241218f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.307+0000 7fe5c37fe640 1 -- 172.21.15.121:0/3826817118 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d4018020 con 0x7fe5dc1384c0 2024-09-17T18:19:03.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.307+0000 7f072976f640 1 --2- 172.21.15.121:0/2404947717 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07240a65c0 0x7f07241218f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.307+0000 7fe5c37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:03.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.307+0000 7fe5c37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:19:03.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.307+0000 7fe5c37fe640 10 monclient: dump: 2024-09-17T18:19:03.403 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:03.403 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.403 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.403 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.403 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:03.403 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.404 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:03.404 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:03.404 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:03.404 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.307+0000 7f072976f640 1 -- 172.21.15.121:0/2404947717 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f072413e1a0 msgr2=0x7f0724137c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.307+0000 7fe5c37fe640 10 monclient: _finish_auth 0 2024-09-17T18:19:03.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.307+0000 7fe5c37fe640 10 monclient: _check_auth_tickets 2024-09-17T18:19:03.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.307+0000 7fe5c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:03.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.307+0000 7fe5c37fe640 1 -- 172.21.15.121:0/3826817118 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5d40146a0 con 0x7fe5dc1384c0 2024-09-17T18:19:03.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.307+0000 7fe5c37fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:03.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.307+0000 7fe5c37fe640 1 -- 172.21.15.121:0/3826817118 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d4014a90 con 0x7fe5dc1384c0 2024-09-17T18:19:03.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.307+0000 7f072976f640 1 --2- 172.21.15.121:0/2404947717 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f072413e1a0 0x7f0724137c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.307+0000 7f072976f640 10 monclient: _finish_hunting 0 2024-09-17T18:19:03.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.307+0000 7fe5c37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:03.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.307+0000 7fe5c37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:19:03.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.307+0000 7fe5c37fe640 10 monclient: dump: 2024-09-17T18:19:03.406 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:03.406 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.406 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.406 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.406 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:03.406 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.406 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:03.406 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:03.406 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:03.406 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.307+0000 7f072976f640 1 monclient: found mon.noname-c 2024-09-17T18:19:03.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.307+0000 7fe5e1764640 5 monclient: authenticate success, global_id 4256 2024-09-17T18:19:03.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.307+0000 7fe5e1764640 10 monclient: _renew_subs 2024-09-17T18:19:03.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.307+0000 7fe5e1764640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:03.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.307+0000 7fe5e1764640 1 -- 172.21.15.121:0/3826817118 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5dc0a0ea0 con 0x7fe5dc1384c0 2024-09-17T18:19:03.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.307+0000 7f072976f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:03.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.307+0000 7f072976f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:19:03.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.307+0000 7f072976f640 1 -- 172.21.15.121:0/2404947717 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07240a0a00 con 0x7f07240a5ca0 2024-09-17T18:19:03.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.307+0000 7f0728f6e640 1 --2- 172.21.15.121:0/2404947717 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07240a65c0 0x7f07241218f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:19:03.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.308+0000 7f0729f70640 10 monclient: get_auth_request con 0x7f072413e1a0 auth_method 0 2024-09-17T18:19:03.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.308+0000 7f0729f70640 1 --2- 172.21.15.121:0/2404947717 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f072413e1a0 0x7f0724137c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T18:19:03.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.307+0000 7fe5e1764640 10 client.4256.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:03.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.307+0000 7fe5e1764640 10 monclient: _renew_subs 2024-09-17T18:19:03.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.307+0000 7fe5e1764640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:03.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.307+0000 7fe5e1764640 1 -- 172.21.15.121:0/3826817118 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5dc123150 con 0x7fe5dc1384c0 2024-09-17T18:19:03.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.307+0000 7fe5e1764640 1 librados: init done 2024-09-17T18:19:03.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.307+0000 7fe5d97fa640 4 set_mon_vals no callback set 2024-09-17T18:19:03.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.308+0000 7fe5e1764640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:03.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.308+0000 7fe5e1764640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:03.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.308+0000 7fe5e1764640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:03.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.308+0000 7fe5e1764640 1 -- 172.21.15.121:0/3826817118 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5dc0a8720 con 0x7fe5dc1384c0 2024-09-17T18:19:03.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.309+0000 7fe5c37fe640 1 -- 172.21.15.121:0/3826817118 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fe5d4005d10 con 0x7fe5dc1384c0 2024-09-17T18:19:03.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.309+0000 7fe5c37fe640 1 --2- 172.21.15.121:0/3826817118 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe5c80401c0 0x7fe5c8042680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.309+0000 7fe5c37fe640 1 -- 172.21.15.121:0/3826817118 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 3562+0+0 (secure 0 0 0) 0x7fe5d4053ef0 con 0x7fe5dc1384c0 2024-09-17T18:19:03.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.309+0000 7fe5c37fe640 10 client.4256.objecter ms_dispatch 0x7fe5dc000b90 osd_map(16..16 src has 1..16) 2024-09-17T18:19:03.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.309+0000 7fe5c37fe640 3 client.4256.objecter handle_osd_map got epochs [16,16] > 0 2024-09-17T18:19:03.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.309+0000 7fe5c37fe640 3 client.4256.objecter handle_osd_map decoding full epoch 16 2024-09-17T18:19:03.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.309+0000 7fe5c37fe640 20 client.4256.objecter dump_active .. 0 homeless 2024-09-17T18:19:03.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.310+0000 7fe5daffd640 1 --2- 172.21.15.121:0/3826817118 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe5c80401c0 0x7fe5c8042680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.310+0000 7f0719ffb640 10 client.?.objecter ms_handle_connect 0x7f07240a5ca0 2024-09-17T18:19:03.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.310+0000 7fe5daffd640 10 monclient: get_auth_request con 0x7fe5c80401c0 auth_method 0 2024-09-17T18:19:03.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.310+0000 7f0719ffb640 10 client.?.objecter resend_mon_ops 2024-09-17T18:19:03.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.310+0000 7f072976f640 1 -- 172.21.15.121:0/2404947717 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f07240a5ca0 msgr2=0x7f0724122dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:03.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.310+0000 7f072976f640 1 --2- 172.21.15.121:0/2404947717 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f07240a5ca0 0x7f0724122dc0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f072000e8a0 tx=0x7f072000ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:03.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.313+0000 7fe5c37fe640 10 client.4256.objecter ms_handle_connect 0x7fe5c80401c0 2024-09-17T18:19:03.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.313+0000 7fe5daffd640 1 --2- 172.21.15.121:0/3826817118 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe5c80401c0 0x7fe5c8042680 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc0090b0 tx=0x7fe5cc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:03.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.318+0000 7fe5c37fe640 1 -- 172.21.15.121:0/3826817118 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe5d4022030 con 0x7fe5dc1384c0 2024-09-17T18:19:03.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.318+0000 7fe5c37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:03.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.318+0000 7fe5c37fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:19:03.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.318+0000 7f0719ffb640 1 -- 172.21.15.121:0/2404947717 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0720017070 con 0x7f07240a5ca0 2024-09-17T18:19:03.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.318+0000 7f0719ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:03.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.318+0000 7f0719ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:19:03.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.318+0000 7f0719ffb640 10 monclient: dump: 2024-09-17T18:19:03.415 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:03.415 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.415 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.415 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.415 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:03.415 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.415 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:03.415 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:03.415 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:03.416 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.318+0000 7f0719ffb640 10 monclient: _finish_auth 0 2024-09-17T18:19:03.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.318+0000 7f0719ffb640 10 monclient: _check_auth_tickets 2024-09-17T18:19:03.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.318+0000 7f0719ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:03.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.318+0000 7f0719ffb640 1 -- 172.21.15.121:0/2404947717 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f072000cb50 con 0x7f07240a5ca0 2024-09-17T18:19:03.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.318+0000 7f0719ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:03.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.318+0000 7f0719ffb640 1 -- 172.21.15.121:0/2404947717 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0720013470 con 0x7f07240a5ca0 2024-09-17T18:19:03.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.318+0000 7f0719ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:03.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.318+0000 7f0719ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:19:03.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.318+0000 7f0719ffb640 10 monclient: dump: 2024-09-17T18:19:03.416 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:03.416 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.416 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.416 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.417 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:03.417 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.417 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:03.417 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:03.417 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:03.417 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.318+0000 7f072b9d3640 5 monclient: authenticate success, global_id 4259 2024-09-17T18:19:03.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.318+0000 7f072b9d3640 10 monclient: _renew_subs 2024-09-17T18:19:03.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.318+0000 7f072b9d3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:03.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.318+0000 7f072b9d3640 1 -- 172.21.15.121:0/2404947717 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07241245c0 con 0x7f07240a5ca0 2024-09-17T18:19:03.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.318+0000 7f071b7fe640 4 set_mon_vals no callback set 2024-09-17T18:19:03.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.318+0000 7f072b9d3640 10 client.4259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:03.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.318+0000 7f072b9d3640 10 monclient: _renew_subs 2024-09-17T18:19:03.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.318+0000 7f072b9d3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:03.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.318+0000 7f072b9d3640 1 -- 172.21.15.121:0/2404947717 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0724138150 con 0x7f07240a5ca0 2024-09-17T18:19:03.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.318+0000 7f072b9d3640 1 librados: init done 2024-09-17T18:19:03.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.319+0000 7f072b9d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:03.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.319+0000 7f072b9d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:03.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.320+0000 7f072b9d3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:03.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.320+0000 7f072b9d3640 1 -- 172.21.15.121:0/2404947717 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07240a8720 con 0x7f07240a5ca0 2024-09-17T18:19:03.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.320+0000 7f0719ffb640 1 -- 172.21.15.121:0/2404947717 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f072001a070 con 0x7f07240a5ca0 2024-09-17T18:19:03.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.320+0000 7f0719ffb640 1 --2- 172.21.15.121:0/2404947717 >> v2:172.21.15.173:6812/2423786037 conn(0x7f07080401c0 0x7f0708042680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.320+0000 7f0719ffb640 1 -- 172.21.15.121:0/2404947717 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 3562+0+0 (secure 0 0 0) 0x7f0720010cc0 con 0x7f07240a5ca0 2024-09-17T18:19:03.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.320+0000 7f0719ffb640 10 client.4259.objecter ms_dispatch 0x7f0724000b90 osd_map(16..16 src has 1..16) 2024-09-17T18:19:03.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.320+0000 7f0719ffb640 3 client.4259.objecter handle_osd_map got epochs [16,16] > 0 2024-09-17T18:19:03.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.320+0000 7f0719ffb640 3 client.4259.objecter handle_osd_map decoding full epoch 16 2024-09-17T18:19:03.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.321+0000 7f0728f6e640 1 --2- 172.21.15.121:0/2404947717 >> v2:172.21.15.173:6812/2423786037 conn(0x7f07080401c0 0x7f0708042680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.321+0000 7f0728f6e640 10 monclient: get_auth_request con 0x7f07080401c0 auth_method 0 2024-09-17T18:19:03.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.321+0000 7f0719ffb640 20 client.4259.objecter dump_active .. 0 homeless 2024-09-17T18:19:03.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.323+0000 7f0728f6e640 1 --2- 172.21.15.121:0/2404947717 >> v2:172.21.15.173:6812/2423786037 conn(0x7f07080401c0 0x7f0708042680 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f072406fd00 tx=0x7f0714007f20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:03.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.323+0000 7f0719ffb640 10 client.4259.objecter ms_handle_connect 0x7f07080401c0 2024-09-17T18:19:03.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.329+0000 7f0719ffb640 1 -- 172.21.15.121:0/2404947717 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f072001b3b0 con 0x7f07240a5ca0 2024-09-17T18:19:03.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.329+0000 7f0719ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:03.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.329+0000 7f0719ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:19:03.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.357+0000 7f1fff5fe640 10 monclient: get_monmap_and_config 2024-09-17T18:19:03.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.357+0000 7f1fff5fe640 10 monclient: build_initial_monmap 2024-09-17T18:19:03.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.357+0000 7f1fff5fe640 10 monclient: monmap: 2024-09-17T18:19:03.432 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:03.432 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.432 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:19:03.357734+0000 2024-09-17T18:19:03.432 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:19:03.357734+0000 2024-09-17T18:19:03.433 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:03.433 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.433 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:19:03.433 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:19:03.433 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:19:03.433 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:19:03.433 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:19:03.433 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.357+0000 7f1fff5fe640 1 Processor -- start 2024-09-17T18:19:03.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.358+0000 7f1fff5fe640 1 -- start start 2024-09-17T18:19:03.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.358+0000 7f1fff5fe640 10 monclient: init 2024-09-17T18:19:03.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.359+0000 7f1fff5fe640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:03.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.359+0000 7f1fff5fe640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-17T18:19:03.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.359+0000 7f1fff5fe640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1ff80a5ca0 0x7f1ff80a6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.359+0000 7f1fff5fe640 10 monclient(hunting): picked mon.noname-a con 0x7f1ff80a5ca0 addr v2:172.21.15.121:3300/0 2024-09-17T18:19:03.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.359+0000 7f1fff5fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1ff80a65c0 addr v1:172.21.15.121:6789/0 2024-09-17T18:19:03.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.359+0000 7f1ffcb99640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f1ff80a65c0 0x7f1ff813ca80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:49594/0 (socket says 172.21.15.121:49594) 2024-09-17T18:19:03.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.359+0000 7f1ffcb99640 1 -- 172.21.15.121:0/1660330181 learned_addr learned my addr 172.21.15.121:0/1660330181 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:19:03.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.359+0000 7f1fff5fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1ff813e1a0 addr v1:172.21.15.173:6789/0 2024-09-17T18:19:03.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.359+0000 7f1fff5fe640 1 -- 172.21.15.121:0/1660330181 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ff80a7b50 con 0x7f1ff80a65c0 2024-09-17T18:19:03.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.359+0000 7f1fff5fe640 1 -- 172.21.15.121:0/1660330181 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ff80a2a50 con 0x7f1ff813e1a0 2024-09-17T18:19:03.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.359+0000 7f1ffd39a640 1 --2- 172.21.15.121:0/1660330181 >> v2:172.21.15.121:3300/0 conn(0x7f1ff80a5ca0 0x7f1ff80a6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.359+0000 7f1fff5fe640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.359+0000 7f1fff5fe640 1 -- 172.21.15.121:0/1660330181 --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f1ff80704c0 con 0x7f1ff80a5ca0 2024-09-17T18:19:03.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.359+0000 7f1fff5fe640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:03.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.359+0000 7f1fff5fe640 10 monclient(hunting): authenticate will time out at 747.075886s 2024-09-17T18:19:03.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.359+0000 7f1ffd39a640 10 monclient(hunting): get_auth_request con 0x7f1ff80a5ca0 auth_method 0 2024-09-17T18:19:03.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.359+0000 7f1ffd39a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:03.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.359+0000 7f1ffd39a640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:03.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.359+0000 7f1ffd39a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:03.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.360+0000 7f1ffd39a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:03.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.360+0000 7f1ffd39a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:03.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.360+0000 7f1ffd39a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:03.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.360+0000 7f1ffd39a640 10 monclient(hunting): handle_auth_done global_id 4281 payload 274 2024-09-17T18:19:03.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.360+0000 7f1ffd39a640 1 -- 172.21.15.121:0/1660330181 >> v1:172.21.15.173:6789/0 conn(0x7f1ff813e1a0 legacy=0x7f1ff8140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:03.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.360+0000 7f1ffd39a640 1 -- 172.21.15.121:0/1660330181 >> v1:172.21.15.121:6789/0 conn(0x7f1ff80a65c0 legacy=0x7f1ff813ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:03.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.360+0000 7f1ffd39a640 10 monclient: _finish_hunting 0 2024-09-17T18:19:03.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.360+0000 7f1ffd39a640 1 monclient: found mon.noname-a 2024-09-17T18:19:03.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.360+0000 7f1ffd39a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:03.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.360+0000 7f1ffd39a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:19:03.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.360+0000 7f1ffd39a640 1 -- 172.21.15.121:0/1660330181 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ff80a2d30 con 0x7f1ff80a5ca0 2024-09-17T18:19:03.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.363+0000 7f1ffd39a640 1 -- 172.21.15.121:0/1660330181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ff80a5ca0 msgr2=0x7f1ff80a6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:03.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.363+0000 7f1ffd39a640 1 --2- 172.21.15.121:0/1660330181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ff80a5ca0 0x7f1ff80a6080 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1ff40097a0 tx=0x7f1ff4029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52873cc90f33c85a server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:03.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.364+0000 7f1feffff640 1 -- 172.21.15.121:0/1660330181 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff4037070 con 0x7f1ff80a5ca0 2024-09-17T18:19:03.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.364+0000 7f1feffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:03.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.364+0000 7f1feffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:19:03.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.364+0000 7f1feffff640 10 monclient: dump: 2024-09-17T18:19:03.436 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:03.436 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.436 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.436 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.436 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:03.436 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.437 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:03.437 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:03.437 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:03.437 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.364+0000 7f1feffff640 10 monclient: _finish_auth 0 2024-09-17T18:19:03.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.364+0000 7f1feffff640 10 monclient: _check_auth_tickets 2024-09-17T18:19:03.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.364+0000 7f1feffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:03.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.364+0000 7f1feffff640 1 -- 172.21.15.121:0/1660330181 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ff4032440 con 0x7f1ff80a5ca0 2024-09-17T18:19:03.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.364+0000 7f1feffff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:03.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.364+0000 7f1feffff640 1 -- 172.21.15.121:0/1660330181 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff4032810 con 0x7f1ff80a5ca0 2024-09-17T18:19:03.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.364+0000 7f1feffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:03.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.364+0000 7f1feffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:19:03.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.364+0000 7f1feffff640 10 monclient: dump: 2024-09-17T18:19:03.438 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:03.438 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.438 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.438 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.438 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:03.438 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.438 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:03.438 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:03.438 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:03.438 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.364+0000 7f1fff5fe640 5 monclient: authenticate success, global_id 4281 2024-09-17T18:19:03.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.364+0000 7f1fff5fe640 10 monclient: get_monmap_and_config success 2024-09-17T18:19:03.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.364+0000 7f1fff5fe640 4 set_mon_vals no callback set 2024-09-17T18:19:03.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.365+0000 7f1fff5fe640 10 monclient: shutdown 2024-09-17T18:19:03.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.365+0000 7f1fff5fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:03.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.365+0000 7f1fff5fe640 1 -- 172.21.15.121:0/1660330181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ff80a5ca0 msgr2=0x7f1ff80a6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.365+0000 7f1fff5fe640 1 --2- 172.21.15.121:0/1660330181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ff80a5ca0 0x7f1ff80a6080 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1ff40097a0 tx=0x7f1ff4029fd0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.365+0000 7f1fff5fe640 1 -- 172.21.15.121:0/1660330181 shutdown_connections 2024-09-17T18:19:03.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.365+0000 7f1fff5fe640 1 --2- 172.21.15.121:0/1660330181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ff80a5ca0 0x7f1ff80a6080 secure :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1ff40097a0 tx=0x7f1ff4029fd0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.365+0000 7f1fff5fe640 1 -- 172.21.15.121:0/1660330181 >> 172.21.15.121:0/1660330181 conn(0x7f1ff806dd10 msgr2=0x7f1ff80a7da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:03.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.365+0000 7f1fff5fe640 1 -- 172.21.15.121:0/1660330181 shutdown_connections 2024-09-17T18:19:03.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.365+0000 7f1fff5fe640 1 -- 172.21.15.121:0/1660330181 wait complete. 2024-09-17T18:19:03.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.366+0000 7f1fff5fe640 10 monclient: build_initial_monmap 2024-09-17T18:19:03.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.366+0000 7f1fff5fe640 10 monclient: monmap: 2024-09-17T18:19:03.439 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:03.439 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:19:03.439 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:19:03.439 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:19:03.439 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:03.439 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.439 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:19:03.440 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:19:03.440 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:19:03.440 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.366+0000 7f1fff5fe640 1 librados: starting msgr at 2024-09-17T18:19:03.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.366+0000 7f1fff5fe640 1 librados: starting objecter 2024-09-17T18:19:03.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.366+0000 7f1fff5fe640 1 Processor -- start 2024-09-17T18:19:03.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.366+0000 7f1fff5fe640 1 -- start start 2024-09-17T18:19:03.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.366+0000 7f1fff5fe640 1 librados: setting wanted keys 2024-09-17T18:19:03.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.366+0000 7f1fff5fe640 1 librados: calling monclient init 2024-09-17T18:19:03.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.366+0000 7f1fff5fe640 10 monclient: init 2024-09-17T18:19:03.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.367+0000 7f1fff5fe640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:03.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.367+0000 7f1fff5fe640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T18:19:03.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.367+0000 7f1fff5fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ff80a65c0 0x7f1ff80a0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.367+0000 7f1fff5fe640 10 monclient(hunting): picked mon.noname-b con 0x7f1ff80a65c0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:03.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.367+0000 7f1fff5fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ff813e1a0 0x7f1ff8137df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.367+0000 7f1fff5fe640 10 monclient(hunting): picked mon.noname-a con 0x7f1ff813e1a0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:03.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.367+0000 7f1ffcb99640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ff813e1a0 0x7f1ff8137df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.367+0000 7f1ffcb99640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ff813e1a0 0x7f1ff8137df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41022/0 (socket says 172.21.15.121:41022) 2024-09-17T18:19:03.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.367+0000 7f1fff5fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ff8122fa0 0x7f1ff8138330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.367+0000 7f1ffcb99640 1 -- 172.21.15.121:0/268682807 learned_addr learned my addr 172.21.15.121:0/268682807 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:03.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.367+0000 7f1fff5fe640 10 monclient(hunting): picked mon.noname-c con 0x7f1ff8122fa0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:03.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.367+0000 7f1fff5fe640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.367+0000 7f1fff5fe640 1 -- 172.21.15.121:0/268682807 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1ff80a5670 con 0x7f1ff813e1a0 2024-09-17T18:19:03.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.367+0000 7f1fff5fe640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.367+0000 7f1fff5fe640 1 -- 172.21.15.121:0/268682807 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1ff80a5490 con 0x7f1ff80a65c0 2024-09-17T18:19:03.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.367+0000 7f1fff5fe640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:03.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.367+0000 7f1fff5fe640 1 -- 172.21.15.121:0/268682807 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1ff80a69d0 con 0x7f1ff8122fa0 2024-09-17T18:19:03.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.367+0000 7f1fff5fe640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:03.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.367+0000 7f1fff5fe640 10 monclient(hunting): authenticate will time out at 747.083987s 2024-09-17T18:19:03.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.367+0000 7f1ffcb99640 10 monclient(hunting): get_auth_request con 0x7f1ff813e1a0 auth_method 0 2024-09-17T18:19:03.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.367+0000 7f1ffcb99640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:03.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.368+0000 7f1ffcb99640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:03.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.368+0000 7f1ffcb99640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:03.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.368+0000 7f1ffcb99640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:03.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.368+0000 7f1ffcb99640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:03.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.368+0000 7f1ffdb9b640 1 --2- 172.21.15.121:0/268682807 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ff8122fa0 0x7f1ff8138330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.368+0000 7f1ffcb99640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:03.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.368+0000 7f1ffdb9b640 10 monclient(hunting): get_auth_request con 0x7f1ff8122fa0 auth_method 0 2024-09-17T18:19:03.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.368+0000 7f1ffdb9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:03.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.368+0000 7f1ffdb9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:03.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.368+0000 7f1ffdb9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:03.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.368+0000 7f1ffdb9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:03.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.368+0000 7f1ffdb9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:03.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.368+0000 7f1ffdb9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:03.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.368+0000 7f1ffcb99640 10 monclient(hunting): handle_auth_done global_id 4284 payload 824 2024-09-17T18:19:03.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.368+0000 7f1ffcb99640 1 -- 172.21.15.121:0/268682807 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ff8122fa0 msgr2=0x7f1ff8138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.368+0000 7f1ffcb99640 1 --2- 172.21.15.121:0/268682807 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ff8122fa0 0x7f1ff8138330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.368+0000 7f1ffcb99640 1 -- 172.21.15.121:0/268682807 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ff80a65c0 msgr2=0x7f1ff80a0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.368+0000 7f1ffcb99640 1 --2- 172.21.15.121:0/268682807 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ff80a65c0 0x7f1ff80a0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.368+0000 7f1ffcb99640 10 monclient: _finish_hunting 0 2024-09-17T18:19:03.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.368+0000 7f1ffcb99640 1 monclient: found mon.noname-a 2024-09-17T18:19:03.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.368+0000 7f1ffcb99640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:03.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.368+0000 7f1ffcb99640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:19:03.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.368+0000 7f1ffcb99640 1 -- 172.21.15.121:0/268682807 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ff80a0d10 con 0x7f1ff813e1a0 2024-09-17T18:19:03.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.368+0000 7f1ffdb9b640 1 --2- 172.21.15.121:0/268682807 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ff8122fa0 0x7f1ff8138330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:19:03.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.370+0000 7f1ffcb99640 1 -- 172.21.15.121:0/268682807 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ff813e1a0 msgr2=0x7f1ff8137df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:03.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.370+0000 7f1ffcb99640 1 --2- 172.21.15.121:0/268682807 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ff813e1a0 0x7f1ff8137df0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1fe800c460 tx=0x7f1fe800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:03.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.370+0000 7f1fedffb640 10 client.?.objecter ms_handle_connect 0x7f1ff813e1a0 2024-09-17T18:19:03.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.370+0000 7f1fedffb640 10 client.?.objecter resend_mon_ops 2024-09-17T18:19:03.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.371+0000 7f1fedffb640 1 -- 172.21.15.121:0/268682807 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe8007070 con 0x7f1ff813e1a0 2024-09-17T18:19:03.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.371+0000 7f1fedffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:03.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.371+0000 7f1fedffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:19:03.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.371+0000 7f1fedffb640 10 monclient: dump: 2024-09-17T18:19:03.444 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:03.444 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.444 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.444 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.444 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:03.444 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.444 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:03.444 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:03.444 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:03.444 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.371+0000 7f1fedffb640 10 monclient: _finish_auth 0 2024-09-17T18:19:03.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.371+0000 7f1fedffb640 10 monclient: _check_auth_tickets 2024-09-17T18:19:03.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.371+0000 7f1fedffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:03.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.371+0000 7f1fedffb640 1 -- 172.21.15.121:0/268682807 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fe8007230 con 0x7f1ff813e1a0 2024-09-17T18:19:03.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.371+0000 7f1fff5fe640 5 monclient: authenticate success, global_id 4284 2024-09-17T18:19:03.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.371+0000 7f1fff5fe640 10 monclient: _renew_subs 2024-09-17T18:19:03.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.371+0000 7f1fff5fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:03.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.371+0000 7f1fff5fe640 1 -- 172.21.15.121:0/268682807 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ff8121e10 con 0x7f1ff813e1a0 2024-09-17T18:19:03.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.371+0000 7f1fedffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:03.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.371+0000 7f1fedffb640 1 -- 172.21.15.121:0/268682807 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe8007620 con 0x7f1ff813e1a0 2024-09-17T18:19:03.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.371+0000 7f1fedffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:03.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.371+0000 7f1fedffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:19:03.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.371+0000 7f1fedffb640 10 monclient: dump: 2024-09-17T18:19:03.445 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:03.445 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:03.445 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.445 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:03.446 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:03.446 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:03.446 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:03.446 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:03.446 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:03.446 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:03.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.371+0000 7f1fef7fe640 4 set_mon_vals no callback set 2024-09-17T18:19:03.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.371+0000 7f1fff5fe640 10 client.4284.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:03.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.371+0000 7f1fff5fe640 10 monclient: _renew_subs 2024-09-17T18:19:03.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.371+0000 7f1fff5fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:03.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.371+0000 7f1fff5fe640 1 -- 172.21.15.121:0/268682807 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ff81389d0 con 0x7f1ff813e1a0 2024-09-17T18:19:03.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.371+0000 7f1fff5fe640 1 librados: init done 2024-09-17T18:19:03.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.372+0000 7f1fedffb640 1 -- 172.21.15.121:0/268682807 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f1fe801b400 con 0x7f1ff813e1a0 2024-09-17T18:19:03.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.372+0000 7f1fff5fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:03.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.372+0000 7f1fff5fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:03.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.372+0000 7f1fff5fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:03.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.372+0000 7f1fff5fe640 1 -- 172.21.15.121:0/268682807 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ff80a87a0 con 0x7f1ff813e1a0 2024-09-17T18:19:03.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.373+0000 7f1fedffb640 1 --2- 172.21.15.121:0/268682807 >> v2:172.21.15.173:6812/2423786037 conn(0x7f1fdc0401c0 0x7f1fdc042680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:03.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.373+0000 7f1fedffb640 1 -- 172.21.15.121:0/268682807 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 3562+0+0 (secure 0 0 0) 0x7f1fe8052bb0 con 0x7f1ff813e1a0 2024-09-17T18:19:03.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.373+0000 7f1fedffb640 10 client.4284.objecter ms_dispatch 0x7f1ff8000b90 osd_map(16..16 src has 1..16) 2024-09-17T18:19:03.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.373+0000 7f1fedffb640 3 client.4284.objecter handle_osd_map got epochs [16,16] > 0 2024-09-17T18:19:03.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.373+0000 7f1fedffb640 3 client.4284.objecter handle_osd_map decoding full epoch 16 2024-09-17T18:19:03.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.373+0000 7f1fedffb640 20 client.4284.objecter dump_active .. 0 homeless 2024-09-17T18:19:03.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.373+0000 7f1ffd39a640 1 --2- 172.21.15.121:0/268682807 >> v2:172.21.15.173:6812/2423786037 conn(0x7f1fdc0401c0 0x7f1fdc042680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:03.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.373+0000 7f1ffd39a640 10 monclient: get_auth_request con 0x7f1fdc0401c0 auth_method 0 2024-09-17T18:19:03.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.376+0000 7f1fedffb640 10 client.4284.objecter ms_handle_connect 0x7f1fdc0401c0 2024-09-17T18:19:03.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.376+0000 7f1ffd39a640 1 --2- 172.21.15.121:0/268682807 >> v2:172.21.15.173:6812/2423786037 conn(0x7f1fdc0401c0 0x7f1fdc042680 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1ff4031000 tx=0x7f1ff402c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:03.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.382+0000 7f1fedffb640 1 -- 172.21.15.121:0/268682807 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f1fe801b8d0 con 0x7f1ff813e1a0 2024-09-17T18:19:03.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.382+0000 7f1fedffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:03.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.382+0000 7f1fedffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:19:03.489 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.487+0000 7ff050d3d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-17T18:19:03.489 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.487+0000 7ff050d3d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-17T18:19:03.489 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.487+0000 7ff050d3d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:03.489 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.487+0000 7ff050d3d640 1 -- 172.21.15.121:0/1472694789 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7ff04c1cdce0 con 0x7ff04c0a65c0 2024-09-17T18:19:03.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.488+0000 7ff042ffd640 1 -- 172.21.15.121:0/1472694789 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff03c003b30 con 0x7ff04c0a65c0 2024-09-17T18:19:03.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.488+0000 7ff042ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-17T18:19:03.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.488+0000 7ff042ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:19:03.490 INFO:teuthology.orchestra.run.smithi121.stdout:51539607554 2024-09-17T18:19:03.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.490+0000 7ff050d3d640 10 librados: watch_flush enter 2024-09-17T18:19:03.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.490+0000 7ff050d3d640 10 librados: watch_flush exit 2024-09-17T18:19:03.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.490+0000 7ff050d3d640 20 client.4247.objecter shutdown clearing up homeless session... 2024-09-17T18:19:03.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.490+0000 7ff050d3d640 10 client.4247.objecter successfully canceled tick 2024-09-17T18:19:03.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.490+0000 7ff050d3d640 1 -- 172.21.15.121:0/1472694789 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff02c0401a0 msgr2=0x7ff02c042660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.490+0000 7ff050d3d640 1 --2- 172.21.15.121:0/1472694789 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff02c0401a0 0x7ff02c042660 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff034009f30 tx=0x7ff034009190 comp rx=0 tx=0).stop 2024-09-17T18:19:03.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.490+0000 7ff050d3d640 10 monclient: shutdown 2024-09-17T18:19:03.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.490+0000 7ff050d3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:03.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.490+0000 7ff050d3d640 1 -- 172.21.15.121:0/1472694789 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff04c0a65c0 msgr2=0x7ff04c1380d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.490+0000 7ff050d3d640 1 --2- 172.21.15.121:0/1472694789 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff04c0a65c0 0x7ff04c1380d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff03c02a600 tx=0x7ff03c0046c0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.490+0000 7ff050d3d640 1 -- 172.21.15.121:0/1472694789 shutdown_connections 2024-09-17T18:19:03.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.490+0000 7ff050d3d640 1 --2- 172.21.15.121:0/1472694789 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff02c0401a0 0x7ff02c042660 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.490+0000 7ff050d3d640 1 --2- 172.21.15.121:0/1472694789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff04c13cfc0 0x7ff04c138610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.490+0000 7ff050d3d640 1 --2- 172.21.15.121:0/1472694789 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff04c0a65c0 0x7ff04c1380d0 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff03c02a600 tx=0x7ff03c0046c0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.490+0000 7ff050d3d640 1 --2- 172.21.15.121:0/1472694789 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff04c0a5ca0 0x7ff04c124290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.490+0000 7ff050d3d640 1 -- 172.21.15.121:0/1472694789 >> 172.21.15.121:0/1472694789 conn(0x7ff04c06dd10 msgr2=0x7ff04c0a12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:03.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.491+0000 7ff050d3d640 1 -- 172.21.15.121:0/1472694789 shutdown_connections 2024-09-17T18:19:03.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.491+0000 7ff050d3d640 1 -- 172.21.15.121:0/1472694789 wait complete. 2024-09-17T18:19:03.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.491+0000 7ff050d3d640 1 librados: shutdown 2024-09-17T18:19:03.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.496+0000 7f37cb0b0640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-17T18:19:03.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.496+0000 7f37cb0b0640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-17T18:19:03.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.496+0000 7f37cb0b0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:19:03.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.496+0000 7f37cb0b0640 1 -- 172.21.15.121:0/1226608797 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f37c4134750 con 0x7f37c409ec60 2024-09-17T18:19:03.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.497+0000 7f37c17fa640 1 -- 172.21.15.121:0/1226608797 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f37bc010040 con 0x7f37c409ec60 2024-09-17T18:19:03.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.497+0000 7f37c17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-17T18:19:03.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.497+0000 7f37c17fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:19:03.499 INFO:teuthology.orchestra.run.smithi121.stdout:47244640258 2024-09-17T18:19:03.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.499+0000 7f37cb0b0640 10 librados: watch_flush enter 2024-09-17T18:19:03.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.499+0000 7f37cb0b0640 10 librados: watch_flush exit 2024-09-17T18:19:03.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.499+0000 7f37cb0b0640 20 client.4258.objecter shutdown clearing up homeless session... 2024-09-17T18:19:03.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.499+0000 7f37cb0b0640 10 client.4258.objecter successfully canceled tick 2024-09-17T18:19:03.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.499+0000 7f37cb0b0640 1 -- 172.21.15.121:0/1226608797 >> v2:172.21.15.173:6812/2423786037 conn(0x7f37a80401c0 msgr2=0x7f37a8042680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.499+0000 7f37cb0b0640 1 --2- 172.21.15.121:0/1226608797 >> v2:172.21.15.173:6812/2423786037 conn(0x7f37a80401c0 0x7f37a8042680 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f37b80090b0 tx=0x7f37b802e000 comp rx=0 tx=0).stop 2024-09-17T18:19:03.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.499+0000 7f37cb0b0640 10 monclient: shutdown 2024-09-17T18:19:03.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.499+0000 7f37cb0b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:03.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.499+0000 7f37cb0b0640 1 -- 172.21.15.121:0/1226608797 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37c409ec60 msgr2=0x7f37c41cd7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.499+0000 7f37cb0b0640 1 --2- 172.21.15.121:0/1226608797 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37c409ec60 0x7f37c41cd7a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f37bc009e50 tx=0x7f37bc00c490 comp rx=0 tx=0).stop 2024-09-17T18:19:03.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.499+0000 7f37cb0b0640 1 -- 172.21.15.121:0/1226608797 shutdown_connections 2024-09-17T18:19:03.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.499+0000 7f37cb0b0640 1 --2- 172.21.15.121:0/1226608797 >> v2:172.21.15.173:6812/2423786037 conn(0x7f37a80401c0 0x7f37a8042680 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.499+0000 7f37cb0b0640 1 --2- 172.21.15.121:0/1226608797 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37c409ec60 0x7f37c41cd7a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.499+0000 7f37cb0b0640 1 --2- 172.21.15.121:0/1226608797 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37c409e810 0x7f37c412f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.499+0000 7f37cb0b0640 1 --2- 172.21.15.121:0/1226608797 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37c40700f0 0x7f37c412ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.499+0000 7f37cb0b0640 1 -- 172.21.15.121:0/1226608797 >> 172.21.15.121:0/1226608797 conn(0x7f37c40ab830 msgr2=0x7f37c4137f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:03.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.499+0000 7f37cb0b0640 1 -- 172.21.15.121:0/1226608797 shutdown_connections 2024-09-17T18:19:03.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.499+0000 7f37cb0b0640 1 -- 172.21.15.121:0/1226608797 wait complete. 2024-09-17T18:19:03.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.499+0000 7f37cb0b0640 1 librados: shutdown 2024-09-17T18:19:03.505 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607555 got 51539607554 for osd.1 2024-09-17T18:19:03.512 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640258 for osd.0 2024-09-17T18:19:03.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.595+0000 7f53bb19b640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-17T18:19:03.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.595+0000 7f53bb19b640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-17T18:19:03.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.595+0000 7f53bb19b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:03.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.595+0000 7f53bb19b640 1 -- 172.21.15.121:0/811428999 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f53b40a8720 con 0x7f53b40a5ca0 2024-09-17T18:19:03.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.595+0000 7f53b17fa640 1 -- 172.21.15.121:0/811428999 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f53a8045dd0 con 0x7f53b40a5ca0 2024-09-17T18:19:03.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.595+0000 7f53b17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-17T18:19:03.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.595+0000 7f53b17fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:19:03.598 INFO:teuthology.orchestra.run.smithi121.stdout:51539607555 2024-09-17T18:19:03.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.597+0000 7f53bb19b640 10 librados: watch_flush enter 2024-09-17T18:19:03.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.598+0000 7f53bb19b640 10 librados: watch_flush exit 2024-09-17T18:19:03.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.598+0000 7f53bb19b640 20 client.4272.objecter shutdown clearing up homeless session... 2024-09-17T18:19:03.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.598+0000 7f53bb19b640 10 client.4272.objecter successfully canceled tick 2024-09-17T18:19:03.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.598+0000 7f53bb19b640 1 -- 172.21.15.121:0/811428999 >> v2:172.21.15.173:6812/2423786037 conn(0x7f53980401c0 msgr2=0x7f5398042680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.598+0000 7f53bb19b640 1 --2- 172.21.15.121:0/811428999 >> v2:172.21.15.173:6812/2423786037 conn(0x7f53980401c0 0x7f5398042680 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f53b4138c40 tx=0x7f53a0007e30 comp rx=0 tx=0).stop 2024-09-17T18:19:03.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.598+0000 7f53bb19b640 10 monclient: shutdown 2024-09-17T18:19:03.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.598+0000 7f53bb19b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:03.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.598+0000 7f53bb19b640 1 -- 172.21.15.121:0/811428999 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53b40a5ca0 msgr2=0x7f53b40a0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.598+0000 7f53bb19b640 1 --2- 172.21.15.121:0/811428999 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53b40a5ca0 0x7f53b40a0160 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f53b40706d0 tx=0x7f53a80046c0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.598+0000 7f53bb19b640 1 -- 172.21.15.121:0/811428999 shutdown_connections 2024-09-17T18:19:03.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.598+0000 7f53bb19b640 1 --2- 172.21.15.121:0/811428999 >> v2:172.21.15.173:6812/2423786037 conn(0x7f53980401c0 0x7f5398042680 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.598+0000 7f53bb19b640 1 --2- 172.21.15.121:0/811428999 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53b40a65c0 0x7f53b4137cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.598+0000 7f53bb19b640 1 --2- 172.21.15.121:0/811428999 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53b413e1a0 0x7f53b4138210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.598+0000 7f53bb19b640 1 --2- 172.21.15.121:0/811428999 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53b40a5ca0 0x7f53b40a0160 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.598+0000 7f53bb19b640 1 -- 172.21.15.121:0/811428999 >> 172.21.15.121:0/811428999 conn(0x7f53b406dd10 msgr2=0x7f53b4138c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:03.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.598+0000 7f53bb19b640 1 -- 172.21.15.121:0/811428999 shutdown_connections 2024-09-17T18:19:03.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.598+0000 7f53bb19b640 1 -- 172.21.15.121:0/811428999 wait complete. 2024-09-17T18:19:03.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.598+0000 7f53bb19b640 1 librados: shutdown 2024-09-17T18:19:03.614 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607555 for osd.4 2024-09-17T18:19:03.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.625+0000 7fe5e1764640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-17T18:19:03.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.625+0000 7fe5e1764640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-17T18:19:03.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.625+0000 7fe5e1764640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:03.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.625+0000 7fe5e1764640 1 -- 172.21.15.121:0/3826817118 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fe5dc0a2490 con 0x7fe5dc1384c0 2024-09-17T18:19:03.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.626+0000 7fe5c37fe640 1 -- 172.21.15.121:0/3826817118 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe5d4010040 con 0x7fe5dc1384c0 2024-09-17T18:19:03.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.626+0000 7fe5c37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-17T18:19:03.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.626+0000 7fe5c37fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:19:03.628 INFO:teuthology.orchestra.run.smithi121.stdout:47244640258 2024-09-17T18:19:03.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.628+0000 7fe5e1764640 10 librados: watch_flush enter 2024-09-17T18:19:03.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.628+0000 7fe5e1764640 10 librados: watch_flush exit 2024-09-17T18:19:03.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.628+0000 7fe5e1764640 20 client.4256.objecter shutdown clearing up homeless session... 2024-09-17T18:19:03.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.628+0000 7fe5e1764640 10 client.4256.objecter successfully canceled tick 2024-09-17T18:19:03.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.628+0000 7fe5e1764640 1 -- 172.21.15.121:0/3826817118 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe5c80401c0 msgr2=0x7fe5c8042680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.628+0000 7fe5e1764640 1 --2- 172.21.15.121:0/3826817118 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe5c80401c0 0x7fe5c8042680 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc0090b0 tx=0x7fe5cc02e000 comp rx=0 tx=0).stop 2024-09-17T18:19:03.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.628+0000 7fe5e1764640 10 monclient: shutdown 2024-09-17T18:19:03.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.628+0000 7fe5e1764640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:03.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.628+0000 7fe5e1764640 1 -- 172.21.15.121:0/3826817118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5dc1384c0 msgr2=0x7fe5dc1cd790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.628+0000 7fe5e1764640 1 --2- 172.21.15.121:0/3826817118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5dc1384c0 0x7fe5dc1cd790 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe5d4009d90 tx=0x7fe5d400c490 comp rx=0 tx=0).stop 2024-09-17T18:19:03.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.628+0000 7fe5e1764640 1 -- 172.21.15.121:0/3826817118 shutdown_connections 2024-09-17T18:19:03.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.628+0000 7fe5e1764640 1 --2- 172.21.15.121:0/3826817118 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe5c80401c0 0x7fe5c8042680 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.628+0000 7fe5e1764640 1 --2- 172.21.15.121:0/3826817118 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5dc13e1a0 0x7fe5dc137f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.628+0000 7fe5e1764640 1 --2- 172.21.15.121:0/3826817118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5dc1384c0 0x7fe5dc1cd790 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.628+0000 7fe5e1764640 1 --2- 172.21.15.121:0/3826817118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5dc0a65c0 0x7fe5dc0a0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.628+0000 7fe5e1764640 1 -- 172.21.15.121:0/3826817118 >> 172.21.15.121:0/3826817118 conn(0x7fe5dc06dd10 msgr2=0x7fe5dc0a91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:03.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.628+0000 7fe5e1764640 1 -- 172.21.15.121:0/3826817118 shutdown_connections 2024-09-17T18:19:03.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.628+0000 7fe5e1764640 1 -- 172.21.15.121:0/3826817118 wait complete. 2024-09-17T18:19:03.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.628+0000 7fe5e1764640 1 librados: shutdown 2024-09-17T18:19:03.640 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640258 for osd.2 2024-09-17T18:19:03.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.641+0000 7f072b9d3640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-17T18:19:03.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.641+0000 7f072b9d3640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-17T18:19:03.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.641+0000 7f072b9d3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:03.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.641+0000 7f072b9d3640 1 -- 172.21.15.121:0/2404947717 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f072406cd40 con 0x7f07240a5ca0 2024-09-17T18:19:03.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.642+0000 7f0719ffb640 1 -- 172.21.15.121:0/2404947717 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f072001bcd0 con 0x7f07240a5ca0 2024-09-17T18:19:03.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.642+0000 7f0719ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-17T18:19:03.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.642+0000 7f0719ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:19:03.644 INFO:teuthology.orchestra.run.smithi121.stdout:47244640257 2024-09-17T18:19:03.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.644+0000 7f072b9d3640 10 librados: watch_flush enter 2024-09-17T18:19:03.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.644+0000 7f072b9d3640 10 librados: watch_flush exit 2024-09-17T18:19:03.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.644+0000 7f072b9d3640 20 client.4259.objecter shutdown clearing up homeless session... 2024-09-17T18:19:03.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.644+0000 7f072b9d3640 10 client.4259.objecter successfully canceled tick 2024-09-17T18:19:03.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.644+0000 7f072b9d3640 1 -- 172.21.15.121:0/2404947717 >> v2:172.21.15.173:6812/2423786037 conn(0x7f07080401c0 msgr2=0x7f0708042680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.644+0000 7f072b9d3640 1 --2- 172.21.15.121:0/2404947717 >> v2:172.21.15.173:6812/2423786037 conn(0x7f07080401c0 0x7f0708042680 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f072406fd00 tx=0x7f0714007f20 comp rx=0 tx=0).stop 2024-09-17T18:19:03.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.644+0000 7f072b9d3640 10 monclient: shutdown 2024-09-17T18:19:03.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.644+0000 7f072b9d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:03.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.644+0000 7f072b9d3640 1 -- 172.21.15.121:0/2404947717 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f07240a5ca0 msgr2=0x7f0724122dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.644+0000 7f072b9d3640 1 --2- 172.21.15.121:0/2404947717 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f07240a5ca0 0x7f0724122dc0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f072000e8a0 tx=0x7f072000ed60 comp rx=0 tx=0).stop 2024-09-17T18:19:03.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.644+0000 7f072b9d3640 1 -- 172.21.15.121:0/2404947717 shutdown_connections 2024-09-17T18:19:03.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.644+0000 7f072b9d3640 1 --2- 172.21.15.121:0/2404947717 >> v2:172.21.15.173:6812/2423786037 conn(0x7f07080401c0 0x7f0708042680 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.644+0000 7f072b9d3640 1 --2- 172.21.15.121:0/2404947717 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07240a65c0 0x7f07241218f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.644+0000 7f072b9d3640 1 --2- 172.21.15.121:0/2404947717 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f072413e1a0 0x7f0724137c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.644+0000 7f072b9d3640 1 --2- 172.21.15.121:0/2404947717 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f07240a5ca0 0x7f0724122dc0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.644+0000 7f072b9d3640 1 -- 172.21.15.121:0/2404947717 >> 172.21.15.121:0/2404947717 conn(0x7f072406dd10 msgr2=0x7f0724138c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:03.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.644+0000 7f072b9d3640 1 -- 172.21.15.121:0/2404947717 shutdown_connections 2024-09-17T18:19:03.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.644+0000 7f072b9d3640 1 -- 172.21.15.121:0/2404947717 wait complete. 2024-09-17T18:19:03.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.644+0000 7f072b9d3640 1 librados: shutdown 2024-09-17T18:19:03.657 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640257 for osd.5 2024-09-17T18:19:03.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.661+0000 7f1fff5fe640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-17T18:19:03.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.661+0000 7f1fff5fe640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-17T18:19:03.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.661+0000 7f1fff5fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:03.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.661+0000 7f1fff5fe640 1 -- 172.21.15.121:0/268682807 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f1ff81246b0 con 0x7f1ff813e1a0 2024-09-17T18:19:03.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.661+0000 7f1fedffb640 1 -- 172.21.15.121:0/268682807 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1fe8051a70 con 0x7f1ff813e1a0 2024-09-17T18:19:03.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.661+0000 7f1fedffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-17T18:19:03.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.661+0000 7f1fedffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:19:03.664 INFO:teuthology.orchestra.run.smithi121.stdout:47244640258 2024-09-17T18:19:03.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.663+0000 7f1fff5fe640 10 librados: watch_flush enter 2024-09-17T18:19:03.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.663+0000 7f1fff5fe640 10 librados: watch_flush exit 2024-09-17T18:19:03.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.663+0000 7f1fff5fe640 20 client.4284.objecter shutdown clearing up homeless session... 2024-09-17T18:19:03.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.663+0000 7f1fff5fe640 10 client.4284.objecter successfully canceled tick 2024-09-17T18:19:03.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.664+0000 7f1fff5fe640 1 -- 172.21.15.121:0/268682807 >> v2:172.21.15.173:6812/2423786037 conn(0x7f1fdc0401c0 msgr2=0x7f1fdc042680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.664+0000 7f1fff5fe640 1 --2- 172.21.15.121:0/268682807 >> v2:172.21.15.173:6812/2423786037 conn(0x7f1fdc0401c0 0x7f1fdc042680 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1ff4031000 tx=0x7f1ff402c000 comp rx=0 tx=0).stop 2024-09-17T18:19:03.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.664+0000 7f1fff5fe640 10 monclient: shutdown 2024-09-17T18:19:03.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.664+0000 7f1fff5fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:03.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.664+0000 7f1fff5fe640 1 -- 172.21.15.121:0/268682807 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ff813e1a0 msgr2=0x7f1ff8137df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:03.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.664+0000 7f1fff5fe640 1 --2- 172.21.15.121:0/268682807 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ff813e1a0 0x7f1ff8137df0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1fe800c460 tx=0x7f1fe800c920 comp rx=0 tx=0).stop 2024-09-17T18:19:03.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.664+0000 7f1fff5fe640 1 -- 172.21.15.121:0/268682807 shutdown_connections 2024-09-17T18:19:03.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.664+0000 7f1fff5fe640 1 --2- 172.21.15.121:0/268682807 >> v2:172.21.15.173:6812/2423786037 conn(0x7f1fdc0401c0 0x7f1fdc042680 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.664+0000 7f1fff5fe640 1 --2- 172.21.15.121:0/268682807 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ff8122fa0 0x7f1ff8138330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.664+0000 7f1fff5fe640 1 --2- 172.21.15.121:0/268682807 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ff813e1a0 0x7f1ff8137df0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.664+0000 7f1fff5fe640 1 --2- 172.21.15.121:0/268682807 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ff80a65c0 0x7f1ff80a0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:03.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.664+0000 7f1fff5fe640 1 -- 172.21.15.121:0/268682807 >> 172.21.15.121:0/268682807 conn(0x7f1ff806dd10 msgr2=0x7f1ff80a91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:03.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.664+0000 7f1fff5fe640 1 -- 172.21.15.121:0/268682807 shutdown_connections 2024-09-17T18:19:03.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.664+0000 7f1fff5fe640 1 -- 172.21.15.121:0/268682807 wait complete. 2024-09-17T18:19:03.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:03.664+0000 7f1fff5fe640 1 librados: shutdown 2024-09-17T18:19:03.676 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640258 for osd.3 2024-09-17T18:19:04.505 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-17T18:19:04.512 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-17T18:19:04.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.610+0000 7fac51e72640 10 monclient: get_monmap_and_config 2024-09-17T18:19:04.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.610+0000 7fac51e72640 10 monclient: build_initial_monmap 2024-09-17T18:19:04.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.610+0000 7fac51e72640 10 monclient: monmap: 2024-09-17T18:19:04.612 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:04.612 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:04.612 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:19:04.610734+0000 2024-09-17T18:19:04.612 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:19:04.610734+0000 2024-09-17T18:19:04.612 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:04.612 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:04.612 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:19:04.612 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:19:04.612 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:19:04.612 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:19:04.612 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:19:04.612 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:04.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.611+0000 7fac51e72640 1 Processor -- start 2024-09-17T18:19:04.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.611+0000 7fac51e72640 1 -- start start 2024-09-17T18:19:04.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.611+0000 7fac51e72640 10 monclient: init 2024-09-17T18:19:04.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.611+0000 7fac51e72640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:04.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.611+0000 7fac51e72640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-17T18:19:04.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.611+0000 7fac51e72640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fac4c0700f0 0x7fac4c134e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:04.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.611+0000 7fac51e72640 10 monclient(hunting): picked mon.noname-b con 0x7fac4c0700f0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:04.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.611+0000 7fac51e72640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fac4c09e810 addr v1:172.21.15.173:6789/0 2024-09-17T18:19:04.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.611+0000 7fac51e72640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fac4c09ec30 0x7fac4c138a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:04.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.611+0000 7fac51e72640 10 monclient(hunting): picked mon.noname-a con 0x7fac4c09ec30 addr v2:172.21.15.121:3300/0 2024-09-17T18:19:04.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.611+0000 7fac51e72640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac4c09ca60 con 0x7fac4c09e810 2024-09-17T18:19:04.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.611+0000 7fac51e72640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:04.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.611+0000 7fac51e72640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fac4c0704d0 con 0x7fac4c09ec30 2024-09-17T18:19:04.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.611+0000 7fac51e72640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:04.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.611+0000 7fac51e72640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fac4c09cd20 con 0x7fac4c0700f0 2024-09-17T18:19:04.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.611+0000 7fac51e72640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:04.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.611+0000 7fac51e72640 10 monclient(hunting): authenticate will time out at 748.327661s 2024-09-17T18:19:04.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.611+0000 7fac4bfff640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fac4c09ec30 0x7fac4c138a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:04.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.611+0000 7fac4b7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fac4c0700f0 0x7fac4c134e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:04.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.611+0000 7fac4b7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fac4c0700f0 0x7fac4c134e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:59486/0 (socket says 172.21.15.121:59486) 2024-09-17T18:19:04.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.611+0000 7fac4bfff640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fac4c09ec30 0x7fac4c138a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41038/0 (socket says 172.21.15.121:41038) 2024-09-17T18:19:04.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.611+0000 7fac4b7fe640 1 -- 172.21.15.121:0/3008607703 learned_addr learned my addr 172.21.15.121:0/3008607703 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:04.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.611+0000 7fac4b7fe640 10 monclient(hunting): get_auth_request con 0x7fac4c0700f0 auth_method 0 2024-09-17T18:19:04.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.611+0000 7fac4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:04.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.611+0000 7fac4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:04.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.611+0000 7fac4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:04.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.611+0000 7fac4bfff640 10 monclient(hunting): get_auth_request con 0x7fac4c09ec30 auth_method 0 2024-09-17T18:19:04.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.611+0000 7fac4bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:04.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.611+0000 7fac4bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:04.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.611+0000 7fac4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:04.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.611+0000 7fac4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:04.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.611+0000 7fac4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:04.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.611+0000 7fac4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:04.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.611+0000 7fac4bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:04.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.611+0000 7fac4bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:04.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.611+0000 7fac4bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:04.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.612+0000 7fac4b7fe640 10 monclient(hunting): handle_auth_done global_id 4265 payload 274 2024-09-17T18:19:04.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.612+0000 7fac4b7fe640 1 -- 172.21.15.121:0/3008607703 >> v2:172.21.15.121:3300/0 conn(0x7fac4c09ec30 msgr2=0x7fac4c138a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:04.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.612+0000 7fac4b7fe640 1 --2- 172.21.15.121:0/3008607703 >> v2:172.21.15.121:3300/0 conn(0x7fac4c09ec30 0x7fac4c138a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:04.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.612+0000 7fac4b7fe640 1 -- 172.21.15.121:0/3008607703 >> v1:172.21.15.173:6789/0 conn(0x7fac4c09e810 legacy=0x7fac4c138360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:04.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.612+0000 7fac4b7fe640 10 monclient: _finish_hunting 0 2024-09-17T18:19:04.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.612+0000 7fac4b7fe640 1 monclient: found mon.noname-b 2024-09-17T18:19:04.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.612+0000 7fac4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:04.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.612+0000 7fac4b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:19:04.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.612+0000 7fac4b7fe640 1 -- 172.21.15.121:0/3008607703 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac4c09d000 con 0x7fac4c0700f0 2024-09-17T18:19:04.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.612+0000 7fac4bfff640 1 --2- 172.21.15.121:0/3008607703 >> v2:172.21.15.121:3300/0 conn(0x7fac4c09ec30 0x7fac4c138a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:19:04.619 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-17T18:19:04.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.614+0000 7fac4b7fe640 1 -- 172.21.15.121:0/3008607703 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fac4c0700f0 msgr2=0x7fac4c134e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:04.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.614+0000 7fac4b7fe640 1 --2- 172.21.15.121:0/3008607703 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fac4c0700f0 0x7fac4c134e50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fac3c009870 tx=0x7fac3c02a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5a255a672c3acb8 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:04.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.615+0000 7fac4affd640 1 -- 172.21.15.121:0/3008607703 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac3c037070 con 0x7fac4c0700f0 2024-09-17T18:19:04.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.615+0000 7fac4affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:04.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.615+0000 7fac4affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:19:04.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.615+0000 7fac4affd640 10 monclient: dump: 2024-09-17T18:19:04.625 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:04.625 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:04.625 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.625 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.625 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:04.625 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:04.625 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:04.625 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:04.625 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:04.625 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:04.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.615+0000 7fac4affd640 10 monclient: _finish_auth 0 2024-09-17T18:19:04.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.615+0000 7fac4affd640 10 monclient: _check_auth_tickets 2024-09-17T18:19:04.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.615+0000 7fac4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:04.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.615+0000 7fac4affd640 1 -- 172.21.15.121:0/3008607703 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac3c032640 con 0x7fac4c0700f0 2024-09-17T18:19:04.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.615+0000 7fac51e72640 5 monclient: authenticate success, global_id 4265 2024-09-17T18:19:04.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.615+0000 7fac51e72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:19:04.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.615+0000 7fac4affd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:04.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.615+0000 7fac51e72640 10 monclient: get_monmap_and_config success 2024-09-17T18:19:04.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.615+0000 7fac4affd640 1 -- 172.21.15.121:0/3008607703 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac3c032a10 con 0x7fac4c0700f0 2024-09-17T18:19:04.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.615+0000 7fac4affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:04.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.615+0000 7fac51e72640 4 set_mon_vals no callback set 2024-09-17T18:19:04.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.615+0000 7fac4affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:19:04.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.615+0000 7fac4affd640 10 monclient: dump: 2024-09-17T18:19:04.627 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:04.627 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:04.627 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.627 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.627 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:04.627 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:04.627 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:04.628 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:04.628 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:04.628 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:04.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.615+0000 7fac51e72640 10 monclient: shutdown 2024-09-17T18:19:04.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.615+0000 7fac51e72640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:04.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.615+0000 7fac51e72640 1 -- 172.21.15.121:0/3008607703 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fac4c0700f0 msgr2=0x7fac4c134e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:04.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.615+0000 7fac51e72640 1 --2- 172.21.15.121:0/3008607703 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fac4c0700f0 0x7fac4c134e50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fac3c009870 tx=0x7fac3c02a130 comp rx=0 tx=0).stop 2024-09-17T18:19:04.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.615+0000 7fac51e72640 1 -- 172.21.15.121:0/3008607703 shutdown_connections 2024-09-17T18:19:04.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.615+0000 7fac51e72640 1 --2- 172.21.15.121:0/3008607703 >> v2:172.21.15.121:3300/0 conn(0x7fac4c09ec30 0x7fac4c138a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:04.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.615+0000 7fac51e72640 1 --2- 172.21.15.121:0/3008607703 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fac4c0700f0 0x7fac4c134e50 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:04.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.615+0000 7fac51e72640 1 -- 172.21.15.121:0/3008607703 >> 172.21.15.121:0/3008607703 conn(0x7fac4c0ab830 msgr2=0x7fac4c0a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:04.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.615+0000 7fac51e72640 1 -- 172.21.15.121:0/3008607703 shutdown_connections 2024-09-17T18:19:04.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.615+0000 7fac51e72640 1 -- 172.21.15.121:0/3008607703 wait complete. 2024-09-17T18:19:04.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.616+0000 7fac51e72640 10 monclient: build_initial_monmap 2024-09-17T18:19:04.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.616+0000 7fac51e72640 10 monclient: monmap: 2024-09-17T18:19:04.630 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:04.630 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:19:04.630 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:19:04.630 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:19:04.630 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:04.630 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:04.630 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:19:04.630 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:19:04.630 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:19:04.630 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:04.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.616+0000 7fac51e72640 1 librados: starting msgr at 2024-09-17T18:19:04.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.616+0000 7fac51e72640 1 librados: starting objecter 2024-09-17T18:19:04.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.616+0000 7fac51e72640 1 Processor -- start 2024-09-17T18:19:04.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.616+0000 7fac51e72640 1 -- start start 2024-09-17T18:19:04.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.616+0000 7fac51e72640 1 librados: setting wanted keys 2024-09-17T18:19:04.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.616+0000 7fac51e72640 1 librados: calling monclient init 2024-09-17T18:19:04.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.616+0000 7fac51e72640 10 monclient: init 2024-09-17T18:19:04.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac51e72640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:04.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac51e72640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T18:19:04.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac51e72640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fac4c0700f0 0x7fac4c131bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:04.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac51e72640 10 monclient(hunting): picked mon.noname-c con 0x7fac4c0700f0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:04.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac51e72640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fac4c09e810 0x7fac4c132100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:04.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac51e72640 10 monclient(hunting): picked mon.noname-b con 0x7fac4c09e810 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:04.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac51e72640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac4c09ec30 0x7fac4c0a0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:04.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac51e72640 10 monclient(hunting): picked mon.noname-a con 0x7fac4c09ec30 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:04.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac51e72640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:04.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac51e72640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fac4c135550 con 0x7fac4c09ec30 2024-09-17T18:19:04.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac51e72640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:04.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac51e72640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fac4c1353d0 con 0x7fac4c09e810 2024-09-17T18:19:04.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac51e72640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:04.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac51e72640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fac4c138fb0 con 0x7fac4c0700f0 2024-09-17T18:19:04.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac51e72640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:04.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac51e72640 10 monclient(hunting): authenticate will time out at 748.333255s 2024-09-17T18:19:04.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac4b7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fac4c0700f0 0x7fac4c131bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:04.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac4b7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fac4c0700f0 0x7fac4c131bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:59496/0 (socket says 172.21.15.121:59496) 2024-09-17T18:19:04.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac4b7fe640 1 -- 172.21.15.121:0/1490544094 learned_addr learned my addr 172.21.15.121:0/1490544094 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:04.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac4b7fe640 10 monclient(hunting): get_auth_request con 0x7fac4c0700f0 auth_method 0 2024-09-17T18:19:04.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:04.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:04.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:04.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac4bfff640 1 --2- 172.21.15.121:0/1490544094 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac4c09ec30 0x7fac4c0a0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:04.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac4bfff640 10 monclient(hunting): get_auth_request con 0x7fac4c09ec30 auth_method 0 2024-09-17T18:19:04.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac4bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:04.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac4bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:04.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:04.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:04.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:04.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:04.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac43fff640 1 --2- 172.21.15.121:0/1490544094 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fac4c09e810 0x7fac4c132100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:04.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac4bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:04.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac4bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:04.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac4bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:04.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac43fff640 10 monclient(hunting): get_auth_request con 0x7fac4c09e810 auth_method 0 2024-09-17T18:19:04.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:04.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac43fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:04.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:04.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac43fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:04.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac43fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:04.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac43fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:04.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac4b7fe640 10 monclient(hunting): handle_auth_done global_id 4268 payload 824 2024-09-17T18:19:04.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac4b7fe640 1 -- 172.21.15.121:0/1490544094 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fac4c09e810 msgr2=0x7fac4c132100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:04.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac4b7fe640 1 --2- 172.21.15.121:0/1490544094 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fac4c09e810 0x7fac4c132100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:04.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac4b7fe640 1 -- 172.21.15.121:0/1490544094 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac4c09ec30 msgr2=0x7fac4c0a0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:04.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac4b7fe640 1 --2- 172.21.15.121:0/1490544094 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac4c09ec30 0x7fac4c0a0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:04.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac4b7fe640 10 monclient: _finish_hunting 0 2024-09-17T18:19:04.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac4b7fe640 1 monclient: found mon.noname-c 2024-09-17T18:19:04.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:04.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac4b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:19:04.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac4b7fe640 1 -- 172.21.15.121:0/1490544094 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac4c139450 con 0x7fac4c0700f0 2024-09-17T18:19:04.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.617+0000 7fac4bfff640 1 --2- 172.21.15.121:0/1490544094 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac4c09ec30 0x7fac4c0a0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:19:04.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.619+0000 7fac4b7fe640 1 -- 172.21.15.121:0/1490544094 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fac4c0700f0 msgr2=0x7fac4c131bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:04.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.619+0000 7fac4b7fe640 1 --2- 172.21.15.121:0/1490544094 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fac4c0700f0 0x7fac4c131bc0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fac3c0090b0 tx=0x7fac3c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:04.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.619+0000 7fac48ff9640 10 client.?.objecter ms_handle_connect 0x7fac4c0700f0 2024-09-17T18:19:04.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.619+0000 7fac48ff9640 10 client.?.objecter resend_mon_ops 2024-09-17T18:19:04.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.620+0000 7fac48ff9640 1 -- 172.21.15.121:0/1490544094 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac3c041020 con 0x7fac4c0700f0 2024-09-17T18:19:04.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.620+0000 7fac48ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:04.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.620+0000 7fac48ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:19:04.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.620+0000 7fac48ff9640 10 monclient: dump: 2024-09-17T18:19:04.635 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:04.635 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:04.635 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.635 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.635 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:04.636 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:04.636 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:04.636 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:04.636 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:04.636 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:04.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.620+0000 7fac48ff9640 10 monclient: _finish_auth 0 2024-09-17T18:19:04.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.620+0000 7fac48ff9640 10 monclient: _check_auth_tickets 2024-09-17T18:19:04.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.620+0000 7fac48ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:04.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.620+0000 7fac48ff9640 1 -- 172.21.15.121:0/1490544094 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac3c03d630 con 0x7fac4c0700f0 2024-09-17T18:19:04.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.620+0000 7fac51e72640 5 monclient: authenticate success, global_id 4268 2024-09-17T18:19:04.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.620+0000 7fac48ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:04.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.620+0000 7fac48ff9640 1 -- 172.21.15.121:0/1490544094 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac3c037040 con 0x7fac4c0700f0 2024-09-17T18:19:04.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.620+0000 7fac48ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:04.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.620+0000 7fac48ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:19:04.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.620+0000 7fac48ff9640 10 monclient: dump: 2024-09-17T18:19:04.637 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:04.637 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:04.637 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.637 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.637 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:04.637 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:04.637 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:04.637 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:04.637 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:04.637 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:04.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.620+0000 7fac51e72640 10 monclient: _renew_subs 2024-09-17T18:19:04.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.620+0000 7fac4a7fc640 4 set_mon_vals no callback set 2024-09-17T18:19:04.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.620+0000 7fac51e72640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:04.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.620+0000 7fac51e72640 1 -- 172.21.15.121:0/1490544094 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac4c0a1be0 con 0x7fac4c0700f0 2024-09-17T18:19:04.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.620+0000 7fac51e72640 10 client.4268.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:04.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.620+0000 7fac51e72640 10 monclient: _renew_subs 2024-09-17T18:19:04.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.620+0000 7fac51e72640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:04.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.620+0000 7fac51e72640 1 -- 172.21.15.121:0/1490544094 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fac4c132640 con 0x7fac4c0700f0 2024-09-17T18:19:04.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.620+0000 7fac51e72640 1 librados: init done 2024-09-17T18:19:04.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.621+0000 7fac4b7fe640 1 --2- 172.21.15.121:0/1490544094 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fac4c0700f0 0x7fac4c131bc0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fac3c0090b0 tx=0x7fac3c0046c0 comp rx=0 tx=0).queue_received will delay after 1.7266e+09 on 0x7fac3c0733a0 osd_map(16..16 src has 1..16) 2024-09-17T18:19:04.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.621+0000 7fac48ff9640 1 -- 172.21.15.121:0/1490544094 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fac3c044020 con 0x7fac4c0700f0 2024-09-17T18:19:04.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.621+0000 7fac51e72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:04.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.621+0000 7fac51e72640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:04.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.621+0000 7fac51e72640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:04.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.621+0000 7fac51e72640 1 -- 172.21.15.121:0/1490544094 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac4c134e50 con 0x7fac4c0700f0 2024-09-17T18:19:04.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.621+0000 7fac48ff9640 1 --2- 172.21.15.121:0/1490544094 >> v2:172.21.15.173:6812/2423786037 conn(0x7fac2c040170 0x7fac2c042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:04.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.621+0000 7fac43fff640 1 --2- 172.21.15.121:0/1490544094 >> v2:172.21.15.173:6812/2423786037 conn(0x7fac2c040170 0x7fac2c042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:04.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.621+0000 7fac43fff640 10 monclient: get_auth_request con 0x7fac2c040170 auth_method 0 2024-09-17T18:19:04.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.624+0000 7fac48ff9640 10 client.4268.objecter ms_handle_connect 0x7fac2c040170 2024-09-17T18:19:04.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.624+0000 7fac43fff640 1 --2- 172.21.15.121:0/1490544094 >> v2:172.21.15.173:6812/2423786037 conn(0x7fac2c040170 0x7fac2c042630 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fac4c1329c0 tx=0x7fac34007e30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:04.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.626+0000 7fac48ff9640 1 -- 172.21.15.121:0/1490544094 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 3562+0+0 (secure 0 0 0) 0x7fac3c0733a0 con 0x7fac4c0700f0 2024-09-17T18:19:04.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.626+0000 7fac48ff9640 10 client.4268.objecter ms_dispatch 0x7fac4c000b90 osd_map(16..16 src has 1..16) 2024-09-17T18:19:04.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.626+0000 7fac48ff9640 3 client.4268.objecter handle_osd_map got epochs [16,16] > 0 2024-09-17T18:19:04.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.626+0000 7fac48ff9640 3 client.4268.objecter handle_osd_map decoding full epoch 16 2024-09-17T18:19:04.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.626+0000 7fac48ff9640 20 client.4268.objecter dump_active .. 0 homeless 2024-09-17T18:19:04.641 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-17T18:19:04.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.650+0000 7f0ceaa7f640 10 monclient: get_monmap_and_config 2024-09-17T18:19:04.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.650+0000 7f0ceaa7f640 10 monclient: build_initial_monmap 2024-09-17T18:19:04.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.650+0000 7f0ceaa7f640 10 monclient: monmap: 2024-09-17T18:19:04.652 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:04.652 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:04.652 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:19:04.651163+0000 2024-09-17T18:19:04.652 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:19:04.651163+0000 2024-09-17T18:19:04.652 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:04.653 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:04.653 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:19:04.653 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:19:04.653 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:19:04.653 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:19:04.653 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:19:04.653 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:04.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.651+0000 7f0ceaa7f640 1 Processor -- start 2024-09-17T18:19:04.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.651+0000 7f0ceaa7f640 1 -- start start 2024-09-17T18:19:04.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.651+0000 7f0ceaa7f640 10 monclient: init 2024-09-17T18:19:04.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.651+0000 7f0ceaa7f640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:04.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.651+0000 7f0ceaa7f640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-17T18:19:04.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.651+0000 7f0ceaa7f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ce40a5ca0 0x7f0ce40a6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:04.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.651+0000 7f0ceaa7f640 10 monclient(hunting): picked mon.noname-b con 0x7f0ce40a5ca0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:04.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.651+0000 7f0ceaa7f640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0ce40a65c0 0x7f0ce413ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:04.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.651+0000 7f0ceaa7f640 10 monclient(hunting): picked mon.noname-a con 0x7f0ce40a65c0 addr v2:172.21.15.121:3300/0 2024-09-17T18:19:04.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.651+0000 7f0ceaa7f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0ce413cfc0 addr v1:172.21.15.121:6789/0 2024-09-17T18:19:04.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.651+0000 7f0ceaa7f640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ce40a7b50 con 0x7f0ce413cfc0 2024-09-17T18:19:04.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.651+0000 7f0ceaa7f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:04.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.651+0000 7f0ceaa7f640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f0ce40704c0 con 0x7f0ce40a65c0 2024-09-17T18:19:04.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.651+0000 7f0ceaa7f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:04.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.651+0000 7f0ceaa7f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0ce40a5490 con 0x7f0ce40a5ca0 2024-09-17T18:19:04.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.652+0000 7f0ceaa7f640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:04.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.652+0000 7f0ceaa7f640 10 monclient(hunting): authenticate will time out at 748.368021s 2024-09-17T18:19:04.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.652+0000 7f0ce881b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ce40a5ca0 0x7f0ce40a6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:04.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.652+0000 7f0ce3fff640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0ce40a65c0 0x7f0ce413ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:04.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.652+0000 7f0ce881b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ce40a5ca0 0x7f0ce40a6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:59504/0 (socket says 172.21.15.121:59504) 2024-09-17T18:19:04.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.652+0000 7f0ce881b640 1 -- 172.21.15.121:0/2123298263 learned_addr learned my addr 172.21.15.121:0/2123298263 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:04.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.652+0000 7f0ce881b640 10 monclient(hunting): get_auth_request con 0x7f0ce40a5ca0 auth_method 0 2024-09-17T18:19:04.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.652+0000 7f0ce881b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:04.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.652+0000 7f0ce881b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:04.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.652+0000 7f0ce881b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:04.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.652+0000 7f0ce3fff640 10 monclient(hunting): get_auth_request con 0x7f0ce40a65c0 auth_method 0 2024-09-17T18:19:04.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.652+0000 7f0ce3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:04.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.652+0000 7f0ce3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:04.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.652+0000 7f0ce3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:04.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.652+0000 7f0ce881b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:04.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.652+0000 7f0ce881b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:04.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.652+0000 7f0ce881b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:04.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.652+0000 7f0ce3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:04.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.652+0000 7f0ce3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:04.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.652+0000 7f0ce3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:04.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.652+0000 7f0ce881b640 10 monclient(hunting): handle_auth_done global_id 4271 payload 274 2024-09-17T18:19:04.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.652+0000 7f0ce881b640 1 -- 172.21.15.121:0/2123298263 >> v2:172.21.15.121:3300/0 conn(0x7f0ce40a65c0 msgr2=0x7f0ce413ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:04.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.652+0000 7f0ce881b640 1 --2- 172.21.15.121:0/2123298263 >> v2:172.21.15.121:3300/0 conn(0x7f0ce40a65c0 0x7f0ce413ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:04.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.652+0000 7f0ce881b640 1 -- 172.21.15.121:0/2123298263 >> v1:172.21.15.121:6789/0 conn(0x7f0ce413cfc0 legacy=0x7f0ce413f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:04.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.652+0000 7f0ce881b640 10 monclient: _finish_hunting 0 2024-09-17T18:19:04.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.652+0000 7f0ce881b640 1 monclient: found mon.noname-b 2024-09-17T18:19:04.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.652+0000 7f0ce881b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:04.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.652+0000 7f0ce881b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:19:04.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.652+0000 7f0ce881b640 1 -- 172.21.15.121:0/2123298263 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ce40a2a50 con 0x7f0ce40a5ca0 2024-09-17T18:19:04.659 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-17T18:19:04.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.655+0000 7f0ce881b640 1 -- 172.21.15.121:0/2123298263 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ce40a5ca0 msgr2=0x7f0ce40a6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:04.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.655+0000 7f0ce881b640 1 --2- 172.21.15.121:0/2123298263 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ce40a5ca0 0x7f0ce40a6080 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0ce40a3dd0 tx=0x7f0cd802a080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a7092453f66428d server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:04.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.655+0000 7f0ce37fe640 1 -- 172.21.15.121:0/2123298263 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd8037070 con 0x7f0ce40a5ca0 2024-09-17T18:19:04.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.655+0000 7f0ce37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:04.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.655+0000 7f0ce37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:19:04.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.655+0000 7f0ce37fe640 10 monclient: dump: 2024-09-17T18:19:04.665 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:04.666 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:04.666 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.666 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.666 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:04.666 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:04.666 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:04.666 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:04.666 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:04.666 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:04.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.655+0000 7f0ce37fe640 10 monclient: _finish_auth 0 2024-09-17T18:19:04.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.655+0000 7f0ce37fe640 10 monclient: _check_auth_tickets 2024-09-17T18:19:04.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.655+0000 7f0ce37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:04.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.655+0000 7f0ceaa7f640 5 monclient: authenticate success, global_id 4271 2024-09-17T18:19:04.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.655+0000 7f0ceaa7f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:19:04.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.655+0000 7f0ce37fe640 1 -- 172.21.15.121:0/2123298263 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cd8032440 con 0x7f0ce40a5ca0 2024-09-17T18:19:04.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.655+0000 7f0ce37fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:04.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.655+0000 7f0ceaa7f640 10 monclient: get_monmap_and_config success 2024-09-17T18:19:04.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.655+0000 7f0ceaa7f640 4 set_mon_vals no callback set 2024-09-17T18:19:04.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.655+0000 7f0ce37fe640 1 -- 172.21.15.121:0/2123298263 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd80327f0 con 0x7f0ce40a5ca0 2024-09-17T18:19:04.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.655+0000 7f0ce37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:04.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.655+0000 7f0ce37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:19:04.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.655+0000 7f0ce37fe640 10 monclient: dump: 2024-09-17T18:19:04.667 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:04.667 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:04.667 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.667 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.667 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:04.667 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:04.667 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:04.668 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:04.668 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:04.668 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:04.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.656+0000 7f0ceaa7f640 10 monclient: shutdown 2024-09-17T18:19:04.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.656+0000 7f0ceaa7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:04.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.656+0000 7f0ceaa7f640 1 -- 172.21.15.121:0/2123298263 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ce40a5ca0 msgr2=0x7f0ce40a6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:04.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.656+0000 7f0ceaa7f640 1 --2- 172.21.15.121:0/2123298263 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ce40a5ca0 0x7f0ce40a6080 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0ce40a3dd0 tx=0x7f0cd802a080 comp rx=0 tx=0).stop 2024-09-17T18:19:04.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.656+0000 7f0ceaa7f640 1 -- 172.21.15.121:0/2123298263 shutdown_connections 2024-09-17T18:19:04.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.656+0000 7f0ceaa7f640 1 --2- 172.21.15.121:0/2123298263 >> v2:172.21.15.121:3300/0 conn(0x7f0ce40a65c0 0x7f0ce413ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:04.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.656+0000 7f0ceaa7f640 1 --2- 172.21.15.121:0/2123298263 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ce40a5ca0 0x7f0ce40a6080 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:04.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.656+0000 7f0ceaa7f640 1 -- 172.21.15.121:0/2123298263 >> 172.21.15.121:0/2123298263 conn(0x7f0ce406dd10 msgr2=0x7f0ce40a7da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:04.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.656+0000 7f0ceaa7f640 1 -- 172.21.15.121:0/2123298263 shutdown_connections 2024-09-17T18:19:04.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.656+0000 7f0ceaa7f640 1 -- 172.21.15.121:0/2123298263 wait complete. 2024-09-17T18:19:04.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.656+0000 7f0ceaa7f640 10 monclient: build_initial_monmap 2024-09-17T18:19:04.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.656+0000 7f0ceaa7f640 10 monclient: monmap: 2024-09-17T18:19:04.669 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:04.669 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:19:04.669 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:19:04.669 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:19:04.669 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:04.669 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:04.669 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:19:04.669 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:19:04.669 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:19:04.669 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:04.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.656+0000 7f0ceaa7f640 1 librados: starting msgr at 2024-09-17T18:19:04.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.656+0000 7f0ceaa7f640 1 librados: starting objecter 2024-09-17T18:19:04.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.657+0000 7f0ceaa7f640 1 Processor -- start 2024-09-17T18:19:04.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.657+0000 7f0ceaa7f640 1 -- start start 2024-09-17T18:19:04.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.657+0000 7f0ceaa7f640 1 librados: setting wanted keys 2024-09-17T18:19:04.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.657+0000 7f0ceaa7f640 1 librados: calling monclient init 2024-09-17T18:19:04.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.657+0000 7f0ceaa7f640 10 monclient: init 2024-09-17T18:19:04.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.657+0000 7f0ceaa7f640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:04.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.657+0000 7f0ceaa7f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T18:19:04.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.657+0000 7f0ceaa7f640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ce40a5ca0 0x7f0ce41cd450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:04.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.657+0000 7f0ceaa7f640 10 monclient(hunting): picked mon.noname-a con 0x7f0ce40a5ca0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:04.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.657+0000 7f0ceaa7f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ce40a65c0 0x7f0ce4137d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:04.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.657+0000 7f0ceaa7f640 10 monclient(hunting): picked mon.noname-c con 0x7f0ce40a65c0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:04.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.657+0000 7f0ceaa7f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ce413cfc0 0x7f0ce4138260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:04.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.657+0000 7f0ceaa7f640 10 monclient(hunting): picked mon.noname-b con 0x7f0ce413cfc0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:04.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.657+0000 7f0ceaa7f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:04.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.657+0000 7f0ceaa7f640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce40a5670 con 0x7f0ce40a5ca0 2024-09-17T18:19:04.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.657+0000 7f0ceaa7f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:04.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.657+0000 7f0ceaa7f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce40a6eb0 con 0x7f0ce413cfc0 2024-09-17T18:19:04.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.657+0000 7f0ceaa7f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:04.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.657+0000 7f0ceaa7f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0ce40a69a0 con 0x7f0ce40a65c0 2024-09-17T18:19:04.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.657+0000 7f0ceaa7f640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:04.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.657+0000 7f0ceaa7f640 10 monclient(hunting): authenticate will time out at 748.373738s 2024-09-17T18:19:04.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.657+0000 7f0ce3fff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ce40a65c0 0x7f0ce4137d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:04.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.657+0000 7f0ce881b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ce40a5ca0 0x7f0ce41cd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:04.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.657+0000 7f0ce881b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ce40a5ca0 0x7f0ce41cd450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41058/0 (socket says 172.21.15.121:41058) 2024-09-17T18:19:04.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.657+0000 7f0ce881b640 1 -- 172.21.15.121:0/3857287722 learned_addr learned my addr 172.21.15.121:0/3857287722 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:04.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.657+0000 7f0ce3fff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ce40a65c0 0x7f0ce4137d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:59506/0 (socket says 172.21.15.121:59506) 2024-09-17T18:19:04.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.657+0000 7f0ce3fff640 10 monclient(hunting): get_auth_request con 0x7f0ce40a65c0 auth_method 0 2024-09-17T18:19:04.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.657+0000 7f0ce3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:04.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.657+0000 7f0ce3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:04.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.657+0000 7f0ce3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:04.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.657+0000 7f0ce881b640 10 monclient(hunting): get_auth_request con 0x7f0ce40a5ca0 auth_method 0 2024-09-17T18:19:04.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.657+0000 7f0ce881b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:04.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.657+0000 7f0ce881b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:04.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.657+0000 7f0ce881b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:04.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.657+0000 7f0ce3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:04.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.657+0000 7f0ce3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:04.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.657+0000 7f0ce3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:04.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.657+0000 7f0ce881b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:04.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.658+0000 7f0ce881b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:04.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.658+0000 7f0ce881b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:04.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.658+0000 7f0ce901c640 1 --2- 172.21.15.121:0/3857287722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ce413cfc0 0x7f0ce4138260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:04.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.658+0000 7f0ce901c640 10 monclient(hunting): get_auth_request con 0x7f0ce413cfc0 auth_method 0 2024-09-17T18:19:04.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.658+0000 7f0ce901c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:04.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.658+0000 7f0ce901c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:04.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.658+0000 7f0ce901c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:04.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.658+0000 7f0ce901c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:04.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.658+0000 7f0ce901c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:04.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.658+0000 7f0ce901c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:04.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.658+0000 7f0ce881b640 10 monclient(hunting): handle_auth_done global_id 4296 payload 824 2024-09-17T18:19:04.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.658+0000 7f0ce881b640 1 -- 172.21.15.121:0/3857287722 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ce40a65c0 msgr2=0x7f0ce4137d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:04.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.658+0000 7f0ce881b640 1 --2- 172.21.15.121:0/3857287722 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ce40a65c0 0x7f0ce4137d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:04.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.658+0000 7f0ce881b640 1 -- 172.21.15.121:0/3857287722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ce413cfc0 msgr2=0x7f0ce4138260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:04.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.658+0000 7f0ce881b640 1 --2- 172.21.15.121:0/3857287722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ce413cfc0 0x7f0ce4138260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:04.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.658+0000 7f0ce881b640 10 monclient: _finish_hunting 0 2024-09-17T18:19:04.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.658+0000 7f0ce881b640 1 monclient: found mon.noname-a 2024-09-17T18:19:04.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.658+0000 7f0ce881b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:04.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.658+0000 7f0ce881b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:19:04.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.658+0000 7f0ce881b640 1 -- 172.21.15.121:0/3857287722 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ce4123380 con 0x7f0ce40a5ca0 2024-09-17T18:19:04.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.658+0000 7f0ce3fff640 1 --2- 172.21.15.121:0/3857287722 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ce40a65c0 0x7f0ce4137d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:19:04.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.660+0000 7f0ce881b640 1 -- 172.21.15.121:0/3857287722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ce40a5ca0 msgr2=0x7f0ce41cd450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:04.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.660+0000 7f0ce881b640 1 --2- 172.21.15.121:0/3857287722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ce40a5ca0 0x7f0ce41cd450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0cd80090b0 tx=0x7f0cd80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:04.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.660+0000 7f0ce17fa640 10 client.?.objecter ms_handle_connect 0x7f0ce40a5ca0 2024-09-17T18:19:04.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.660+0000 7f0ce17fa640 10 client.?.objecter resend_mon_ops 2024-09-17T18:19:04.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.660+0000 7f0ce17fa640 1 -- 172.21.15.121:0/3857287722 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd8041020 con 0x7f0ce40a5ca0 2024-09-17T18:19:04.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.660+0000 7f0ce17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:04.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.660+0000 7f0ce17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:19:04.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.660+0000 7f0ce17fa640 10 monclient: dump: 2024-09-17T18:19:04.674 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:04.674 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:04.674 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.674 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.674 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:04.675 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:04.675 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:04.675 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:04.675 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:04.675 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:04.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.660+0000 7f0ce17fa640 10 monclient: _finish_auth 0 2024-09-17T18:19:04.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.660+0000 7f0ce17fa640 10 monclient: _check_auth_tickets 2024-09-17T18:19:04.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.660+0000 7f0ce17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:04.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.660+0000 7f0ce17fa640 1 -- 172.21.15.121:0/3857287722 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cd803d830 con 0x7f0ce40a5ca0 2024-09-17T18:19:04.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.660+0000 7f0ce17fa640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:04.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.660+0000 7f0ce17fa640 1 -- 172.21.15.121:0/3857287722 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd8037040 con 0x7f0ce40a5ca0 2024-09-17T18:19:04.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.660+0000 7f0ce17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:04.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.660+0000 7f0ce17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:19:04.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.660+0000 7f0ce17fa640 10 monclient: dump: 2024-09-17T18:19:04.675 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:04.676 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:04.676 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.676 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.676 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:04.676 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:04.676 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:04.676 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:04.676 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:04.676 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:04.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.660+0000 7f0ceaa7f640 5 monclient: authenticate success, global_id 4296 2024-09-17T18:19:04.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.660+0000 7f0ceaa7f640 10 monclient: _renew_subs 2024-09-17T18:19:04.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.660+0000 7f0ceaa7f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:04.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.660+0000 7f0ceaa7f640 1 -- 172.21.15.121:0/3857287722 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ce41cd990 con 0x7f0ce40a5ca0 2024-09-17T18:19:04.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.660+0000 7f0ce2ffd640 4 set_mon_vals no callback set 2024-09-17T18:19:04.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.661+0000 7f0ceaa7f640 10 client.4296.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:04.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.661+0000 7f0ceaa7f640 10 monclient: _renew_subs 2024-09-17T18:19:04.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.661+0000 7f0ceaa7f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:04.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.661+0000 7f0ceaa7f640 1 -- 172.21.15.121:0/3857287722 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ce4123ee0 con 0x7f0ce40a5ca0 2024-09-17T18:19:04.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.661+0000 7f0ceaa7f640 1 librados: init done 2024-09-17T18:19:04.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.661+0000 7f0ce17fa640 1 -- 172.21.15.121:0/3857287722 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f0cd8044020 con 0x7f0ce40a5ca0 2024-09-17T18:19:04.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.661+0000 7f0ce881b640 1 --2- 172.21.15.121:0/3857287722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ce40a5ca0 0x7f0ce41cd450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0cd80090b0 tx=0x7f0cd80046c0 comp rx=0 tx=0).queue_received will delay after 1.7266e+09 on 0x7f0cd8078060 osd_map(16..16 src has 1..16) 2024-09-17T18:19:04.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.662+0000 7f0ce17fa640 1 --2- 172.21.15.121:0/3857287722 >> v2:172.21.15.173:6812/2423786037 conn(0x7f0cc80401a0 0x7f0cc8042660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:04.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.662+0000 7f0ce3fff640 1 --2- 172.21.15.121:0/3857287722 >> v2:172.21.15.173:6812/2423786037 conn(0x7f0cc80401a0 0x7f0cc8042660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:04.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.662+0000 7f0ce3fff640 10 monclient: get_auth_request con 0x7f0cc80401a0 auth_method 0 2024-09-17T18:19:04.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.662+0000 7f0ceaa7f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:04.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.662+0000 7f0ceaa7f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:04.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.662+0000 7f0ceaa7f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:04.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.662+0000 7f0ceaa7f640 1 -- 172.21.15.121:0/3857287722 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ce40a8720 con 0x7f0ce40a5ca0 2024-09-17T18:19:04.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.665+0000 7f0ce17fa640 10 client.4296.objecter ms_handle_connect 0x7f0cc80401a0 2024-09-17T18:19:04.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.665+0000 7f0ce3fff640 1 --2- 172.21.15.121:0/3857287722 >> v2:172.21.15.173:6812/2423786037 conn(0x7f0cc80401a0 0x7f0cc8042660 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4124d30 tx=0x7f0cd0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:04.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.667+0000 7f0ce17fa640 1 -- 172.21.15.121:0/3857287722 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 3562+0+0 (secure 0 0 0) 0x7f0cd8078060 con 0x7f0ce40a5ca0 2024-09-17T18:19:04.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.667+0000 7f0ce17fa640 10 client.4296.objecter ms_dispatch 0x7f0ce4000b90 osd_map(16..16 src has 1..16) 2024-09-17T18:19:04.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.667+0000 7f0ce17fa640 3 client.4296.objecter handle_osd_map got epochs [16,16] > 0 2024-09-17T18:19:04.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.667+0000 7f0ce17fa640 3 client.4296.objecter handle_osd_map decoding full epoch 16 2024-09-17T18:19:04.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.667+0000 7f0ce17fa640 20 client.4296.objecter dump_active .. 0 homeless 2024-09-17T18:19:04.679 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-17T18:19:04.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.724+0000 7f8667f7d640 10 monclient: get_monmap_and_config 2024-09-17T18:19:04.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.724+0000 7f8667f7d640 10 monclient: build_initial_monmap 2024-09-17T18:19:04.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.724+0000 7f8667f7d640 10 monclient: monmap: 2024-09-17T18:19:04.726 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:04.726 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:04.726 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:19:04.725161+0000 2024-09-17T18:19:04.726 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:19:04.725161+0000 2024-09-17T18:19:04.726 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:04.726 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:04.726 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:19:04.727 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:19:04.727 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:19:04.727 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:19:04.727 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:19:04.727 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:04.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.725+0000 7f8667f7d640 1 Processor -- start 2024-09-17T18:19:04.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.725+0000 7f8667f7d640 1 -- start start 2024-09-17T18:19:04.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.725+0000 7f8667f7d640 10 monclient: init 2024-09-17T18:19:04.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.725+0000 7f8667f7d640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:04.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.725+0000 7f8667f7d640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-17T18:19:04.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.726+0000 7f8667f7d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f86600a5ca0 addr v1:172.21.15.121:6789/0 2024-09-17T18:19:04.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.726+0000 7f8667f7d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f86600a6790 addr v1:172.21.15.173:6789/0 2024-09-17T18:19:04.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.726+0000 7f8667f7d640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f866013f1b0 0x7f86601415a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:04.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.726+0000 7f8667f7d640 10 monclient(hunting): picked mon.noname-c con 0x7f866013f1b0 addr v2:172.21.15.173:3300/0 2024-09-17T18:19:04.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.726+0000 7f8667f7d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86600a7b50 con 0x7f86600a5ca0 2024-09-17T18:19:04.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.726+0000 7f8667f7d640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86600a2a50 con 0x7f86600a6790 2024-09-17T18:19:04.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.726+0000 7f8667f7d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:04.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.726+0000 7f8667f7d640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f86600704c0 con 0x7f866013f1b0 2024-09-17T18:19:04.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.726+0000 7f8665d19640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f86600a5ca0 0x7f86600a6080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:49618/0 (socket says 172.21.15.121:49618) 2024-09-17T18:19:04.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.726+0000 7f8665d19640 1 -- 172.21.15.121:0/467143415 learned_addr learned my addr 172.21.15.121:0/467143415 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:19:04.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.726+0000 7f8667f7d640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:04.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.726+0000 7f8667f7d640 10 monclient(hunting): authenticate will time out at 748.442289s 2024-09-17T18:19:04.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.726+0000 7f866651a640 1 --2- 172.21.15.121:0/467143415 >> v2:172.21.15.173:3300/0 conn(0x7f866013f1b0 0x7f86601415a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:04.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.726+0000 7f866651a640 10 monclient(hunting): get_auth_request con 0x7f866013f1b0 auth_method 0 2024-09-17T18:19:04.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.726+0000 7f866651a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:04.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.726+0000 7f866651a640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:04.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.726+0000 7f866651a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:04.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.726+0000 7f866651a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:04.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.726+0000 7f866651a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:04.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.726+0000 7f866651a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:04.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.726+0000 7f866651a640 10 monclient(hunting): handle_auth_done global_id 4285 payload 274 2024-09-17T18:19:04.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.726+0000 7f866651a640 1 -- 172.21.15.121:0/467143415 >> v1:172.21.15.173:6789/0 conn(0x7f86600a6790 legacy=0x7f866013da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:04.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.726+0000 7f866651a640 1 -- 172.21.15.121:0/467143415 >> v1:172.21.15.121:6789/0 conn(0x7f86600a5ca0 legacy=0x7f86600a6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:04.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.726+0000 7f866651a640 10 monclient: _finish_hunting 0 2024-09-17T18:19:04.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.726+0000 7f866651a640 1 monclient: found mon.noname-c 2024-09-17T18:19:04.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.726+0000 7f866651a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:04.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.726+0000 7f866651a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.173:3300/0 2024-09-17T18:19:04.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.726+0000 7f866651a640 1 -- 172.21.15.121:0/467143415 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86600a2d30 con 0x7f866013f1b0 2024-09-17T18:19:04.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.729+0000 7f866651a640 1 -- 172.21.15.121:0/467143415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f866013f1b0 msgr2=0x7f86601415a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:04.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.729+0000 7f866651a640 1 --2- 172.21.15.121:0/467143415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f866013f1b0 0x7f86601415a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8658009dc0 tx=0x7f865802a4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7712194c2e2f37ff server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:04.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.730+0000 7f8664d17640 1 -- 172.21.15.121:0/467143415 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8658007a30 con 0x7f866013f1b0 2024-09-17T18:19:04.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.730+0000 7f8664d17640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:04.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.730+0000 7f8664d17640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:19:04.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.730+0000 7f8664d17640 10 monclient: dump: 2024-09-17T18:19:04.737 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:04.737 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:04.737 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.737 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.737 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:04.737 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:04.738 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:04.738 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:04.738 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:04.738 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:04.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.730+0000 7f8664d17640 10 monclient: _finish_auth 0 2024-09-17T18:19:04.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.730+0000 7f8664d17640 10 monclient: _check_auth_tickets 2024-09-17T18:19:04.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.730+0000 7f8664d17640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:04.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.730+0000 7f8664d17640 1 -- 172.21.15.121:0/467143415 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8658007bd0 con 0x7f866013f1b0 2024-09-17T18:19:04.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.730+0000 7f8664d17640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:04.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.730+0000 7f8664d17640 1 -- 172.21.15.121:0/467143415 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8658031400 con 0x7f866013f1b0 2024-09-17T18:19:04.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.730+0000 7f8664d17640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:04.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.730+0000 7f8664d17640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:19:04.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.730+0000 7f8664d17640 10 monclient: dump: 2024-09-17T18:19:04.738 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:04.738 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:04.739 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.739 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.739 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:04.739 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:04.739 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:04.739 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:04.739 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:04.739 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:04.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.730+0000 7f8667f7d640 5 monclient: authenticate success, global_id 4285 2024-09-17T18:19:04.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.730+0000 7f8667f7d640 10 monclient: get_monmap_and_config success 2024-09-17T18:19:04.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.730+0000 7f8667f7d640 4 set_mon_vals no callback set 2024-09-17T18:19:04.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.730+0000 7f8667f7d640 10 monclient: shutdown 2024-09-17T18:19:04.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.730+0000 7f8667f7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:04.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.730+0000 7f8667f7d640 1 -- 172.21.15.121:0/467143415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f866013f1b0 msgr2=0x7f86601415a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:04.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.730+0000 7f8667f7d640 1 --2- 172.21.15.121:0/467143415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f866013f1b0 0x7f86601415a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8658009dc0 tx=0x7f865802a4d0 comp rx=0 tx=0).stop 2024-09-17T18:19:04.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.731+0000 7f8667f7d640 1 -- 172.21.15.121:0/467143415 shutdown_connections 2024-09-17T18:19:04.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.731+0000 7f8667f7d640 1 --2- 172.21.15.121:0/467143415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f866013f1b0 0x7f86601415a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:04.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.731+0000 7f8667f7d640 1 -- 172.21.15.121:0/467143415 >> 172.21.15.121:0/467143415 conn(0x7f866006dd10 msgr2=0x7f86600a7da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:04.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.731+0000 7f8667f7d640 1 -- 172.21.15.121:0/467143415 shutdown_connections 2024-09-17T18:19:04.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.731+0000 7f8667f7d640 1 -- 172.21.15.121:0/467143415 wait complete. 2024-09-17T18:19:04.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.732+0000 7f8667f7d640 10 monclient: build_initial_monmap 2024-09-17T18:19:04.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.732+0000 7f8667f7d640 10 monclient: monmap: 2024-09-17T18:19:04.740 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:04.740 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:19:04.740 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:19:04.740 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:19:04.740 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:04.740 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:04.741 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:19:04.741 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:19:04.741 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:19:04.741 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:04.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.732+0000 7f8667f7d640 1 librados: starting msgr at 2024-09-17T18:19:04.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.732+0000 7f8667f7d640 1 librados: starting objecter 2024-09-17T18:19:04.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.732+0000 7f8667f7d640 1 Processor -- start 2024-09-17T18:19:04.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.732+0000 7f8667f7d640 1 -- start start 2024-09-17T18:19:04.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.732+0000 7f8667f7d640 1 librados: setting wanted keys 2024-09-17T18:19:04.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.732+0000 7f8667f7d640 1 librados: calling monclient init 2024-09-17T18:19:04.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.732+0000 7f8667f7d640 10 monclient: init 2024-09-17T18:19:04.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f8667f7d640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:04.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f8667f7d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T18:19:04.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f8667f7d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f86600a5ca0 0x7f86600a1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:04.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f8667f7d640 10 monclient(hunting): picked mon.noname-c con 0x7f86600a5ca0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:04.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f8667f7d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86600a6790 0x7f8660139140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:04.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f8667f7d640 10 monclient(hunting): picked mon.noname-a con 0x7f86600a6790 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:04.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f8667f7d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f866013f1b0 0x7f86601218f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:04.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f8667f7d640 10 monclient(hunting): picked mon.noname-b con 0x7f866013f1b0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:04.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f8667f7d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:04.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f8667f7d640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f86600a5670 con 0x7f86600a6790 2024-09-17T18:19:04.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f8667f7d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:04.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f8667f7d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f86600a5490 con 0x7f866013f1b0 2024-09-17T18:19:04.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f8667f7d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:04.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f8667f7d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f86600a8720 con 0x7f86600a5ca0 2024-09-17T18:19:04.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f8667f7d640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:04.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f8667f7d640 10 monclient(hunting): authenticate will time out at 748.449244s 2024-09-17T18:19:04.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f8665518640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86600a6790 0x7f8660139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:04.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f8665518640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86600a6790 0x7f8660139140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41072/0 (socket says 172.21.15.121:41072) 2024-09-17T18:19:04.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f8665518640 1 -- 172.21.15.121:0/2246933401 learned_addr learned my addr 172.21.15.121:0/2246933401 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:04.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f8665518640 10 monclient(hunting): get_auth_request con 0x7f86600a6790 auth_method 0 2024-09-17T18:19:04.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f8665518640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:04.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f8665518640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:04.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f8665518640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:04.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f8665d19640 1 --2- 172.21.15.121:0/2246933401 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f86600a5ca0 0x7f86600a1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:04.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f8665d19640 10 monclient(hunting): get_auth_request con 0x7f86600a5ca0 auth_method 0 2024-09-17T18:19:04.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f8665d19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:04.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f8665d19640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:04.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f8665d19640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:04.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f866651a640 1 --2- 172.21.15.121:0/2246933401 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f866013f1b0 0x7f86601218f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:04.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f866651a640 10 monclient(hunting): get_auth_request con 0x7f866013f1b0 auth_method 0 2024-09-17T18:19:04.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f866651a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:04.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f866651a640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:04.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f866651a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:04.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f8665518640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:04.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f8665518640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:04.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f8665518640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:04.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f8665d19640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:04.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f8665d19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:04.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f8665d19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:04.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f866651a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:04.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f866651a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:04.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f866651a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:04.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f8665518640 10 monclient(hunting): handle_auth_done global_id 4299 payload 824 2024-09-17T18:19:04.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f8665518640 1 -- 172.21.15.121:0/2246933401 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f86600a5ca0 msgr2=0x7f86600a1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:04.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f8665518640 1 --2- 172.21.15.121:0/2246933401 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f86600a5ca0 0x7f86600a1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:04.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f8665518640 1 -- 172.21.15.121:0/2246933401 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f866013f1b0 msgr2=0x7f86601218f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:04.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f8665518640 1 --2- 172.21.15.121:0/2246933401 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f866013f1b0 0x7f86601218f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:04.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.733+0000 7f8665518640 10 monclient: _finish_hunting 0 2024-09-17T18:19:04.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.734+0000 7f8665518640 1 monclient: found mon.noname-a 2024-09-17T18:19:04.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.734+0000 7f8665518640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:04.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.734+0000 7f8665518640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:19:04.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.734+0000 7f8665518640 1 -- 172.21.15.121:0/2246933401 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8660139ee0 con 0x7f86600a6790 2024-09-17T18:19:04.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.736+0000 7f8665518640 1 -- 172.21.15.121:0/2246933401 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86600a6790 msgr2=0x7f8660139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:04.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.736+0000 7f8665518640 1 --2- 172.21.15.121:0/2246933401 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86600a6790 0x7f8660139140 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f865000c430 tx=0x7f865000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:04.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.736+0000 7f86567fc640 10 client.?.objecter ms_handle_connect 0x7f86600a6790 2024-09-17T18:19:04.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.736+0000 7f86567fc640 10 client.?.objecter resend_mon_ops 2024-09-17T18:19:04.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.736+0000 7f86567fc640 1 -- 172.21.15.121:0/2246933401 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86500119a0 con 0x7f86600a6790 2024-09-17T18:19:04.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.736+0000 7f86567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:04.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.736+0000 7f86567fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:19:04.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.736+0000 7f86567fc640 10 monclient: dump: 2024-09-17T18:19:04.747 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:04.748 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:04.748 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.748 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.748 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:04.748 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:04.748 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:04.748 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:04.748 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:04.748 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:04.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.736+0000 7f86567fc640 10 monclient: _finish_auth 0 2024-09-17T18:19:04.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.736+0000 7f86567fc640 10 monclient: _check_auth_tickets 2024-09-17T18:19:04.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.736+0000 7f86567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:04.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.736+0000 7f86567fc640 1 -- 172.21.15.121:0/2246933401 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8650017070 con 0x7f86600a6790 2024-09-17T18:19:04.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.736+0000 7f86567fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:04.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.736+0000 7f86567fc640 1 -- 172.21.15.121:0/2246933401 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8650011d80 con 0x7f86600a6790 2024-09-17T18:19:04.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.736+0000 7f8667f7d640 5 monclient: authenticate success, global_id 4299 2024-09-17T18:19:04.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.736+0000 7f8667f7d640 10 monclient: _renew_subs 2024-09-17T18:19:04.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.736+0000 7f8667f7d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:04.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.736+0000 7f8667f7d640 1 -- 172.21.15.121:0/2246933401 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8660124240 con 0x7f86600a6790 2024-09-17T18:19:04.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.736+0000 7f86567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:04.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.736+0000 7f86567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:19:04.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.736+0000 7f86567fc640 10 monclient: dump: 2024-09-17T18:19:04.749 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:04.749 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:04.749 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.749 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.749 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:04.750 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:04.750 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:04.750 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:04.750 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:04.750 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:04.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.736+0000 7f8667f7d640 10 client.4299.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:04.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.736+0000 7f8667f7d640 10 monclient: _renew_subs 2024-09-17T18:19:04.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.736+0000 7f8667f7d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:04.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.736+0000 7f8667f7d640 1 -- 172.21.15.121:0/2246933401 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8660124800 con 0x7f86600a6790 2024-09-17T18:19:04.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.736+0000 7f8657fff640 4 set_mon_vals no callback set 2024-09-17T18:19:04.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.736+0000 7f8667f7d640 1 librados: init done 2024-09-17T18:19:04.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.737+0000 7f86567fc640 1 -- 172.21.15.121:0/2246933401 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f865001c460 con 0x7f86600a6790 2024-09-17T18:19:04.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.738+0000 7f86567fc640 1 --2- 172.21.15.121:0/2246933401 >> v2:172.21.15.173:6812/2423786037 conn(0x7f8644040170 0x7f8644042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:04.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.738+0000 7f86567fc640 1 -- 172.21.15.121:0/2246933401 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 3562+0+0 (secure 0 0 0) 0x7f8650052e50 con 0x7f86600a6790 2024-09-17T18:19:04.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.738+0000 7f86567fc640 10 client.4299.objecter ms_dispatch 0x7f8660000b90 osd_map(16..16 src has 1..16) 2024-09-17T18:19:04.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.738+0000 7f86567fc640 3 client.4299.objecter handle_osd_map got epochs [16,16] > 0 2024-09-17T18:19:04.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.738+0000 7f86567fc640 3 client.4299.objecter handle_osd_map decoding full epoch 16 2024-09-17T18:19:04.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.738+0000 7f8667f7d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:04.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.738+0000 7f8667f7d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:04.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.738+0000 7f8667f7d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:04.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.738+0000 7f86567fc640 20 client.4299.objecter dump_active .. 0 homeless 2024-09-17T18:19:04.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.738+0000 7f8667f7d640 1 -- 172.21.15.121:0/2246933401 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86600008d0 con 0x7f86600a6790 2024-09-17T18:19:04.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.738+0000 7f8665d19640 1 --2- 172.21.15.121:0/2246933401 >> v2:172.21.15.173:6812/2423786037 conn(0x7f8644040170 0x7f8644042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:04.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.738+0000 7f8665d19640 10 monclient: get_auth_request con 0x7f8644040170 auth_method 0 2024-09-17T18:19:04.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.741+0000 7f8665d19640 1 --2- 172.21.15.121:0/2246933401 >> v2:172.21.15.173:6812/2423786037 conn(0x7f8644040170 0x7f8644042630 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f865c007c90 tx=0x7f865c007be0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:04.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.741+0000 7f86567fc640 10 client.4299.objecter ms_handle_connect 0x7f8644040170 2024-09-17T18:19:04.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.747+0000 7f86567fc640 1 -- 172.21.15.121:0/2246933401 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8650012420 con 0x7f86600a6790 2024-09-17T18:19:04.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.747+0000 7f86567fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:04.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.747+0000 7f86567fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:19:04.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.756+0000 7f3d6d29b640 10 monclient: get_monmap_and_config 2024-09-17T18:19:04.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.756+0000 7f3d6d29b640 10 monclient: build_initial_monmap 2024-09-17T18:19:04.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.756+0000 7f3d6d29b640 10 monclient: monmap: 2024-09-17T18:19:04.758 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:04.758 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:04.758 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:19:04.757069+0000 2024-09-17T18:19:04.758 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:19:04.757069+0000 2024-09-17T18:19:04.759 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:04.759 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:04.759 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:19:04.759 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:19:04.759 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:19:04.759 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:19:04.759 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:19:04.759 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:04.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.757+0000 7f3d6d29b640 1 Processor -- start 2024-09-17T18:19:04.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.757+0000 7f3d6d29b640 1 -- start start 2024-09-17T18:19:04.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.757+0000 7f3d6d29b640 10 monclient: init 2024-09-17T18:19:04.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.757+0000 7f3d6d29b640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:04.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.757+0000 7f3d6d29b640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-17T18:19:04.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.757+0000 7f3d6d29b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3d680a5ca0 addr v1:172.21.15.173:6789/0 2024-09-17T18:19:04.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.757+0000 7f3d6d29b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d680a6790 0x7f3d6813da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:04.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.757+0000 7f3d6d29b640 10 monclient(hunting): picked mon.noname-b con 0x7f3d680a6790 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:04.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.757+0000 7f3d6d29b640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f3d6813dfd0 0x7f3d681403c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:04.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.757+0000 7f3d6d29b640 10 monclient(hunting): picked mon.noname-a con 0x7f3d6813dfd0 addr v2:172.21.15.121:3300/0 2024-09-17T18:19:04.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.757+0000 7f3d6d29b640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d680a7b50 con 0x7f3d680a5ca0 2024-09-17T18:19:04.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.757+0000 7f3d6d29b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:04.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.757+0000 7f3d6d29b640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f3d680704c0 con 0x7f3d6813dfd0 2024-09-17T18:19:04.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.757+0000 7f3d6d29b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:04.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.757+0000 7f3d6d29b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3d680a5490 con 0x7f3d680a6790 2024-09-17T18:19:04.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.757+0000 7f3d6d29b640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:04.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.757+0000 7f3d6d29b640 10 monclient(hunting): authenticate will time out at 748.473675s 2024-09-17T18:19:04.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.757+0000 7f3d677fe640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f3d6813dfd0 0x7f3d681403c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:04.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.757+0000 7f3d677fe640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f3d6813dfd0 0x7f3d681403c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41084/0 (socket says 172.21.15.121:41084) 2024-09-17T18:19:04.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.757+0000 7f3d677fe640 1 -- 172.21.15.121:0/3981506169 learned_addr learned my addr 172.21.15.121:0/3981506169 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:04.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.757+0000 7f3d677fe640 10 monclient(hunting): get_auth_request con 0x7f3d6813dfd0 auth_method 0 2024-09-17T18:19:04.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.757+0000 7f3d677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:04.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.757+0000 7f3d677fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:04.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.757+0000 7f3d677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:04.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.757+0000 7f3d677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:04.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.757+0000 7f3d677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:04.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.758+0000 7f3d677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:04.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.758+0000 7f3d667fc640 1 --2- 172.21.15.121:0/3981506169 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d680a6790 0x7f3d6813da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:04.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.758+0000 7f3d667fc640 10 monclient(hunting): get_auth_request con 0x7f3d680a6790 auth_method 0 2024-09-17T18:19:04.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.758+0000 7f3d667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:04.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.758+0000 7f3d667fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:04.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.758+0000 7f3d667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:04.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.758+0000 7f3d667fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:04.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.758+0000 7f3d667fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:04.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.758+0000 7f3d667fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:04.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.758+0000 7f3d677fe640 10 monclient(hunting): handle_auth_done global_id 4302 payload 274 2024-09-17T18:19:04.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.758+0000 7f3d677fe640 1 -- 172.21.15.121:0/3981506169 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d680a6790 msgr2=0x7f3d6813da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:04.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.758+0000 7f3d677fe640 1 --2- 172.21.15.121:0/3981506169 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d680a6790 0x7f3d6813da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:04.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.758+0000 7f3d677fe640 1 -- 172.21.15.121:0/3981506169 >> v1:172.21.15.173:6789/0 conn(0x7f3d680a5ca0 legacy=0x7f3d680a6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:04.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.758+0000 7f3d677fe640 10 monclient: _finish_hunting 0 2024-09-17T18:19:04.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.758+0000 7f3d677fe640 1 monclient: found mon.noname-a 2024-09-17T18:19:04.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.758+0000 7f3d677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:04.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.758+0000 7f3d677fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:19:04.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.758+0000 7f3d677fe640 1 -- 172.21.15.121:0/3981506169 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d680a2a50 con 0x7f3d6813dfd0 2024-09-17T18:19:04.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.761+0000 7f3d677fe640 1 -- 172.21.15.121:0/3981506169 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d6813dfd0 msgr2=0x7f3d681403c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:04.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.761+0000 7f3d677fe640 1 --2- 172.21.15.121:0/3981506169 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d6813dfd0 0x7f3d681403c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3d60009e50 tx=0x7f3d6002a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f84dfdf5af841bf2 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:04.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.761+0000 7f3d65ffb640 1 -- 172.21.15.121:0/3981506169 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d60036070 con 0x7f3d6813dfd0 2024-09-17T18:19:04.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.761+0000 7f3d65ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:04.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.761+0000 7f3d65ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:19:04.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.761+0000 7f3d65ffb640 10 monclient: dump: 2024-09-17T18:19:04.775 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:04.775 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:04.776 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.776 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.776 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:04.776 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:04.776 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:04.776 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:04.776 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:04.776 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:04.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.761+0000 7f3d65ffb640 10 monclient: _finish_auth 0 2024-09-17T18:19:04.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.761+0000 7f3d65ffb640 10 monclient: _check_auth_tickets 2024-09-17T18:19:04.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.761+0000 7f3d65ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:04.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.761+0000 7f3d6d29b640 5 monclient: authenticate success, global_id 4302 2024-09-17T18:19:04.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.761+0000 7f3d6d29b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:19:04.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.761+0000 7f3d65ffb640 1 -- 172.21.15.121:0/3981506169 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d60007800 con 0x7f3d6813dfd0 2024-09-17T18:19:04.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.761+0000 7f3d65ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:04.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.761+0000 7f3d65ffb640 1 -- 172.21.15.121:0/3981506169 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d60007bd0 con 0x7f3d6813dfd0 2024-09-17T18:19:04.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.761+0000 7f3d65ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:04.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.761+0000 7f3d65ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:19:04.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.761+0000 7f3d65ffb640 10 monclient: dump: 2024-09-17T18:19:04.777 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:04.777 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:04.777 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.777 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.777 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:04.777 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:04.777 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:04.777 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:04.777 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:04.778 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:04.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.761+0000 7f3d6d29b640 10 monclient: get_monmap_and_config success 2024-09-17T18:19:04.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.761+0000 7f3d6d29b640 4 set_mon_vals no callback set 2024-09-17T18:19:04.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.762+0000 7f3d6d29b640 10 monclient: shutdown 2024-09-17T18:19:04.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.762+0000 7f3d6d29b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:04.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.762+0000 7f3d6d29b640 1 -- 172.21.15.121:0/3981506169 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d6813dfd0 msgr2=0x7f3d681403c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:04.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.762+0000 7f3d6d29b640 1 --2- 172.21.15.121:0/3981506169 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d6813dfd0 0x7f3d681403c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3d60009e50 tx=0x7f3d6002a560 comp rx=0 tx=0).stop 2024-09-17T18:19:04.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.762+0000 7f3d6d29b640 1 -- 172.21.15.121:0/3981506169 shutdown_connections 2024-09-17T18:19:04.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.762+0000 7f3d6d29b640 1 --2- 172.21.15.121:0/3981506169 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d6813dfd0 0x7f3d681403c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:04.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.762+0000 7f3d6d29b640 1 --2- 172.21.15.121:0/3981506169 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d680a6790 0x7f3d6813da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:04.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.762+0000 7f3d6d29b640 1 -- 172.21.15.121:0/3981506169 >> 172.21.15.121:0/3981506169 conn(0x7f3d6806dd10 msgr2=0x7f3d680a7da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:04.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.762+0000 7f3d6d29b640 1 -- 172.21.15.121:0/3981506169 shutdown_connections 2024-09-17T18:19:04.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.762+0000 7f3d6d29b640 1 -- 172.21.15.121:0/3981506169 wait complete. 2024-09-17T18:19:04.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.763+0000 7f3d6d29b640 10 monclient: build_initial_monmap 2024-09-17T18:19:04.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.763+0000 7f3d6d29b640 10 monclient: monmap: 2024-09-17T18:19:04.779 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:04.779 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:19:04.779 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:19:04.779 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:19:04.779 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:04.779 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:04.779 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:19:04.779 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:19:04.779 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:19:04.779 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:04.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.763+0000 7f3d6d29b640 1 librados: starting msgr at 2024-09-17T18:19:04.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.763+0000 7f3d6d29b640 1 librados: starting objecter 2024-09-17T18:19:04.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.763+0000 7f3d6d29b640 1 Processor -- start 2024-09-17T18:19:04.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.763+0000 7f3d6d29b640 1 -- start start 2024-09-17T18:19:04.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.763+0000 7f3d6d29b640 1 librados: setting wanted keys 2024-09-17T18:19:04.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.763+0000 7f3d6d29b640 1 librados: calling monclient init 2024-09-17T18:19:04.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.763+0000 7f3d6d29b640 10 monclient: init 2024-09-17T18:19:04.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.764+0000 7f3d6d29b640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:04.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.764+0000 7f3d6d29b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:19:04.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.764+0000 7f3d6d29b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d680a5ca0 0x7f3d6809d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:04.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.764+0000 7f3d6d29b640 10 monclient(hunting): picked mon.noname-a con 0x7f3d680a5ca0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:04.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.764+0000 7f3d6d29b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d680a6790 0x7f3d6809d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:04.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.764+0000 7f3d6d29b640 10 monclient(hunting): picked mon.noname-b con 0x7f3d680a6790 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:04.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.764+0000 7f3d6d29b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d6813dfd0 0x7f3d6809db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:04.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.764+0000 7f3d6d29b640 10 monclient(hunting): picked mon.noname-c con 0x7f3d6813dfd0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:04.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.764+0000 7f3d6d29b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:04.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.764+0000 7f3d66ffd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d680a5ca0 0x7f3d6809d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:04.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.764+0000 7f3d6d29b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3d680a5670 con 0x7f3d680a5ca0 2024-09-17T18:19:04.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.764+0000 7f3d6d29b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:04.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.764+0000 7f3d6d29b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3d680a6eb0 con 0x7f3d680a6790 2024-09-17T18:19:04.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.764+0000 7f3d6d29b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:04.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.764+0000 7f3d6d29b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3d680a8720 con 0x7f3d6813dfd0 2024-09-17T18:19:04.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.764+0000 7f3d6d29b640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:04.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.764+0000 7f3d6d29b640 10 monclient(hunting): authenticate will time out at 748.480867s 2024-09-17T18:19:04.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.764+0000 7f3d66ffd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d680a5ca0 0x7f3d6809d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41100/0 (socket says 172.21.15.121:41100) 2024-09-17T18:19:04.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.764+0000 7f3d66ffd640 1 -- 172.21.15.121:0/4130684975 learned_addr learned my addr 172.21.15.121:0/4130684975 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:04.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.764+0000 7f3d66ffd640 10 monclient(hunting): get_auth_request con 0x7f3d680a5ca0 auth_method 0 2024-09-17T18:19:04.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.764+0000 7f3d66ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:04.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.764+0000 7f3d66ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:04.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.764+0000 7f3d66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:04.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.765+0000 7f3d66ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:04.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.765+0000 7f3d66ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:04.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.765+0000 7f3d677fe640 1 --2- 172.21.15.121:0/4130684975 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d6813dfd0 0x7f3d6809db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:04.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.765+0000 7f3d66ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:04.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.765+0000 7f3d677fe640 10 monclient(hunting): get_auth_request con 0x7f3d6813dfd0 auth_method 0 2024-09-17T18:19:04.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.765+0000 7f3d677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:04.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.765+0000 7f3d677fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:04.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.765+0000 7f3d677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:04.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.765+0000 7f3d667fc640 1 --2- 172.21.15.121:0/4130684975 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d680a6790 0x7f3d6809d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:04.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.765+0000 7f3d667fc640 10 monclient(hunting): get_auth_request con 0x7f3d680a6790 auth_method 0 2024-09-17T18:19:04.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.765+0000 7f3d667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:04.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.765+0000 7f3d667fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:04.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.765+0000 7f3d667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:04.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.765+0000 7f3d66ffd640 10 monclient(hunting): handle_auth_done global_id 4305 payload 824 2024-09-17T18:19:04.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.765+0000 7f3d66ffd640 1 -- 172.21.15.121:0/4130684975 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d6813dfd0 msgr2=0x7f3d6809db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:04.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.765+0000 7f3d66ffd640 1 --2- 172.21.15.121:0/4130684975 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d6813dfd0 0x7f3d6809db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:04.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.765+0000 7f3d66ffd640 1 -- 172.21.15.121:0/4130684975 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d680a6790 msgr2=0x7f3d6809d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:04.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.765+0000 7f3d66ffd640 1 --2- 172.21.15.121:0/4130684975 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d680a6790 0x7f3d6809d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:04.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.765+0000 7f3d66ffd640 10 monclient: _finish_hunting 0 2024-09-17T18:19:04.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.765+0000 7f3d66ffd640 1 monclient: found mon.noname-a 2024-09-17T18:19:04.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.765+0000 7f3d66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:04.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.765+0000 7f3d66ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:19:04.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.765+0000 7f3d66ffd640 1 -- 172.21.15.121:0/4130684975 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d680a0aa0 con 0x7f3d680a5ca0 2024-09-17T18:19:04.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.765+0000 7f3d677fe640 1 --2- 172.21.15.121:0/4130684975 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d6813dfd0 0x7f3d6809db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T18:19:04.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.767+0000 7f3d4f7fe640 10 client.?.objecter ms_handle_connect 0x7f3d680a5ca0 2024-09-17T18:19:04.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.767+0000 7f3d4f7fe640 10 client.?.objecter resend_mon_ops 2024-09-17T18:19:04.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.767+0000 7f3d66ffd640 1 -- 172.21.15.121:0/4130684975 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d680a5ca0 msgr2=0x7f3d6809d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:04.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.767+0000 7f3d66ffd640 1 --2- 172.21.15.121:0/4130684975 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d680a5ca0 0x7f3d6809d0b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3d5800c460 tx=0x7f3d5800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:04.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.767+0000 7f3d4f7fe640 1 -- 172.21.15.121:0/4130684975 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d58016020 con 0x7f3d680a5ca0 2024-09-17T18:19:04.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.768+0000 7f3d4f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:04.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.768+0000 7f3d4f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:19:04.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.768+0000 7f3d4f7fe640 10 monclient: dump: 2024-09-17T18:19:04.784 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:04.784 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:04.784 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.784 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.784 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:04.784 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:04.784 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:04.784 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:04.784 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:04.784 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:04.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.768+0000 7f3d4f7fe640 10 monclient: _finish_auth 0 2024-09-17T18:19:04.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.768+0000 7f3d4f7fe640 10 monclient: _check_auth_tickets 2024-09-17T18:19:04.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.768+0000 7f3d4f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:04.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.768+0000 7f3d4f7fe640 1 -- 172.21.15.121:0/4130684975 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d58012a80 con 0x7f3d680a5ca0 2024-09-17T18:19:04.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.768+0000 7f3d4f7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:04.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.768+0000 7f3d4f7fe640 1 -- 172.21.15.121:0/4130684975 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d58011380 con 0x7f3d680a5ca0 2024-09-17T18:19:04.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.768+0000 7f3d4f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:04.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.768+0000 7f3d4f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:19:04.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.768+0000 7f3d4f7fe640 10 monclient: dump: 2024-09-17T18:19:04.785 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:04.785 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:04.785 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.785 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.785 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:04.785 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:04.785 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:04.786 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:04.786 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:04.786 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:04.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.768+0000 7f3d657fa640 4 set_mon_vals no callback set 2024-09-17T18:19:04.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.768+0000 7f3d6d29b640 5 monclient: authenticate success, global_id 4305 2024-09-17T18:19:04.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.768+0000 7f3d6d29b640 10 monclient: _renew_subs 2024-09-17T18:19:04.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.768+0000 7f3d6d29b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:04.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.768+0000 7f3d6d29b640 1 -- 172.21.15.121:0/4130684975 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d680a0d80 con 0x7f3d680a5ca0 2024-09-17T18:19:04.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.768+0000 7f3d6d29b640 10 client.4305.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:04.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.768+0000 7f3d6d29b640 10 monclient: _renew_subs 2024-09-17T18:19:04.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.768+0000 7f3d6d29b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:04.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.768+0000 7f3d6d29b640 1 -- 172.21.15.121:0/4130684975 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d6809ea20 con 0x7f3d680a5ca0 2024-09-17T18:19:04.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.768+0000 7f3d6d29b640 1 librados: init done 2024-09-17T18:19:04.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.770+0000 7f3d4f7fe640 1 -- 172.21.15.121:0/4130684975 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f3d58019020 con 0x7f3d680a5ca0 2024-09-17T18:19:04.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.770+0000 7f3d6d29b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:04.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.770+0000 7f3d6d29b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:04.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.770+0000 7f3d4f7fe640 1 --2- 172.21.15.121:0/4130684975 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3d54040170 0x7f3d54042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:04.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.770+0000 7f3d4f7fe640 1 -- 172.21.15.121:0/4130684975 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 3562+0+0 (secure 0 0 0) 0x7f3d58052e30 con 0x7f3d680a5ca0 2024-09-17T18:19:04.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.770+0000 7f3d4f7fe640 10 client.4305.objecter ms_dispatch 0x7f3d68000b90 osd_map(16..16 src has 1..16) 2024-09-17T18:19:04.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.770+0000 7f3d4f7fe640 3 client.4305.objecter handle_osd_map got epochs [16,16] > 0 2024-09-17T18:19:04.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.770+0000 7f3d4f7fe640 3 client.4305.objecter handle_osd_map decoding full epoch 16 2024-09-17T18:19:04.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.770+0000 7f3d4f7fe640 20 client.4305.objecter dump_active .. 0 homeless 2024-09-17T18:19:04.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.770+0000 7f3d6d29b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:04.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.770+0000 7f3d667fc640 1 --2- 172.21.15.121:0/4130684975 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3d54040170 0x7f3d54042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:04.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.770+0000 7f3d6d29b640 1 -- 172.21.15.121:0/4130684975 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d6809cb90 con 0x7f3d680a5ca0 2024-09-17T18:19:04.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.770+0000 7f3d667fc640 10 monclient: get_auth_request con 0x7f3d54040170 auth_method 0 2024-09-17T18:19:04.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.773+0000 7f3d667fc640 1 --2- 172.21.15.121:0/4130684975 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3d54040170 0x7f3d54042630 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3d6809f090 tx=0x7f3d5c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:04.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.773+0000 7f3d4f7fe640 10 client.4305.objecter ms_handle_connect 0x7f3d54040170 2024-09-17T18:19:04.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.774+0000 7fd7abc73640 10 monclient: get_monmap_and_config 2024-09-17T18:19:04.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.774+0000 7fd7abc73640 10 monclient: build_initial_monmap 2024-09-17T18:19:04.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.774+0000 7fd7abc73640 10 monclient: monmap: 2024-09-17T18:19:04.789 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:04.789 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:04.789 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:19:04.775347+0000 2024-09-17T18:19:04.789 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:19:04.775347+0000 2024-09-17T18:19:04.789 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:04.789 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:04.789 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:19:04.789 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:19:04.789 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:19:04.789 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:19:04.789 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:19:04.790 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:04.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.775+0000 7fd7abc73640 1 Processor -- start 2024-09-17T18:19:04.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.775+0000 7f3d4f7fe640 1 -- 172.21.15.121:0/4130684975 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3d6809cb90 con 0x7f3d680a5ca0 2024-09-17T18:19:04.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.775+0000 7f3d4f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:04.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.775+0000 7f3d4f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:19:04.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.775+0000 7fd7abc73640 1 -- start start 2024-09-17T18:19:04.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.775+0000 7fd7abc73640 10 monclient: init 2024-09-17T18:19:04.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.776+0000 7fd7abc73640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:04.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.776+0000 7fd7abc73640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-17T18:19:04.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.776+0000 7fd7abc73640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7a40a5ca0 0x7fd7a40a6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:04.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.776+0000 7fd7abc73640 10 monclient(hunting): picked mon.noname-b con 0x7fd7a40a5ca0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:04.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.776+0000 7fd7abc73640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fd7a40a65c0 0x7fd7a413ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:04.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.776+0000 7fd7abc73640 10 monclient(hunting): picked mon.noname-c con 0x7fd7a40a65c0 addr v2:172.21.15.173:3300/0 2024-09-17T18:19:04.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.776+0000 7fd7abc73640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd7a413cfc0 addr v1:172.21.15.121:6789/0 2024-09-17T18:19:04.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.776+0000 7fd7abc73640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7a40a7b50 con 0x7fd7a413cfc0 2024-09-17T18:19:04.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.776+0000 7fd7abc73640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:04.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.776+0000 7fd7abc73640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fd7a40704c0 con 0x7fd7a40a65c0 2024-09-17T18:19:04.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.776+0000 7fd7abc73640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:04.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.776+0000 7fd7abc73640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd7a40a5490 con 0x7fd7a40a5ca0 2024-09-17T18:19:04.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.776+0000 7fd7abc73640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:04.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.776+0000 7fd7abc73640 10 monclient(hunting): authenticate will time out at 748.492187s 2024-09-17T18:19:04.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.776+0000 7fd7aa210640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fd7a413cfc0 0x7fd7a413f3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:49626/0 (socket says 172.21.15.121:49626) 2024-09-17T18:19:04.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.776+0000 7fd7aa210640 1 -- 172.21.15.121:0/3230154184 learned_addr learned my addr 172.21.15.121:0/3230154184 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:19:04.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.776+0000 7fd7a9a0f640 1 --2- 172.21.15.121:0/3230154184 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7a40a5ca0 0x7fd7a40a6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:04.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.776+0000 7fd7a9a0f640 10 monclient(hunting): get_auth_request con 0x7fd7a40a5ca0 auth_method 0 2024-09-17T18:19:04.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.776+0000 7fd7a9a0f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:04.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.776+0000 7fd7a9a0f640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:04.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.776+0000 7fd7a9a0f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:04.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.776+0000 7fd7a920e640 1 --2- 172.21.15.121:0/3230154184 >> v2:172.21.15.173:3300/0 conn(0x7fd7a40a65c0 0x7fd7a413ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:04.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.776+0000 7fd7a920e640 10 monclient(hunting): get_auth_request con 0x7fd7a40a65c0 auth_method 0 2024-09-17T18:19:04.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.776+0000 7fd7a920e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:04.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.776+0000 7fd7a920e640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:04.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.776+0000 7fd7a920e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:04.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.776+0000 7fd7a9a0f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:04.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.776+0000 7fd7a9a0f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:04.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.776+0000 7fd7a9a0f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:04.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.776+0000 7fd7a920e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:04.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.776+0000 7fd7a920e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:04.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.776+0000 7fd7a920e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:04.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.776+0000 7fd7a9a0f640 10 monclient(hunting): handle_auth_done global_id 4286 payload 274 2024-09-17T18:19:04.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.776+0000 7fd7a9a0f640 1 -- 172.21.15.121:0/3230154184 >> v2:172.21.15.173:3300/0 conn(0x7fd7a40a65c0 msgr2=0x7fd7a413ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:04.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.776+0000 7fd7a9a0f640 1 --2- 172.21.15.121:0/3230154184 >> v2:172.21.15.173:3300/0 conn(0x7fd7a40a65c0 0x7fd7a413ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:04.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.776+0000 7fd7a9a0f640 1 -- 172.21.15.121:0/3230154184 >> v1:172.21.15.121:6789/0 conn(0x7fd7a413cfc0 legacy=0x7fd7a413f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:04.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.776+0000 7fd7a9a0f640 10 monclient: _finish_hunting 0 2024-09-17T18:19:04.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.776+0000 7fd7a9a0f640 1 monclient: found mon.noname-b 2024-09-17T18:19:04.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.776+0000 7fd7a9a0f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:04.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.776+0000 7fd7a9a0f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:19:04.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.776+0000 7fd7a9a0f640 1 -- 172.21.15.121:0/3230154184 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7a40a2a50 con 0x7fd7a40a5ca0 2024-09-17T18:19:04.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.779+0000 7fd7a9a0f640 1 -- 172.21.15.121:0/3230154184 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7a40a5ca0 msgr2=0x7fd7a40a6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:04.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.779+0000 7fd7a9a0f640 1 --2- 172.21.15.121:0/3230154184 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7a40a5ca0 0x7fd7a40a6080 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd7a40a3dd0 tx=0x7fd7a0029fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec2a1864ceaa3eee server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:04.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.780+0000 7fd7a8a0d640 1 -- 172.21.15.121:0/3230154184 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a0037070 con 0x7fd7a40a5ca0 2024-09-17T18:19:04.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.780+0000 7fd7a8a0d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:04.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.780+0000 7fd7a8a0d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:19:04.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.780+0000 7fd7a8a0d640 10 monclient: dump: 2024-09-17T18:19:04.805 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:04.805 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:04.805 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.806 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.806 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:04.806 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:04.806 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:04.806 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:04.806 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:04.806 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:04.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.780+0000 7fd7a8a0d640 10 monclient: _finish_auth 0 2024-09-17T18:19:04.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.780+0000 7fd7a8a0d640 10 monclient: _check_auth_tickets 2024-09-17T18:19:04.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.780+0000 7fd7a8a0d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:04.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.780+0000 7fd7a8a0d640 1 -- 172.21.15.121:0/3230154184 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7a0032440 con 0x7fd7a40a5ca0 2024-09-17T18:19:04.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.780+0000 7fd7abc73640 5 monclient: authenticate success, global_id 4286 2024-09-17T18:19:04.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.780+0000 7fd7abc73640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:19:04.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.780+0000 7fd7a8a0d640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:04.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.780+0000 7fd7a8a0d640 1 -- 172.21.15.121:0/3230154184 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a0032810 con 0x7fd7a40a5ca0 2024-09-17T18:19:04.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.780+0000 7fd7abc73640 10 monclient: get_monmap_and_config success 2024-09-17T18:19:04.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.780+0000 7fd7abc73640 4 set_mon_vals no callback set 2024-09-17T18:19:04.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.780+0000 7fd7a8a0d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:04.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.780+0000 7fd7a8a0d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:19:04.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.780+0000 7fd7a8a0d640 10 monclient: dump: 2024-09-17T18:19:04.808 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:04.808 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:04.808 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.809 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.809 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:04.809 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:04.809 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:04.809 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:04.809 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:04.809 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:04.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.781+0000 7fd7abc73640 10 monclient: shutdown 2024-09-17T18:19:04.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.781+0000 7fd7abc73640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:04.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.781+0000 7fd7abc73640 1 -- 172.21.15.121:0/3230154184 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7a40a5ca0 msgr2=0x7fd7a40a6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:04.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.781+0000 7fd7abc73640 1 --2- 172.21.15.121:0/3230154184 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7a40a5ca0 0x7fd7a40a6080 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd7a40a3dd0 tx=0x7fd7a0029fb0 comp rx=0 tx=0).stop 2024-09-17T18:19:04.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.781+0000 7fd7abc73640 1 -- 172.21.15.121:0/3230154184 shutdown_connections 2024-09-17T18:19:04.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.781+0000 7fd7abc73640 1 --2- 172.21.15.121:0/3230154184 >> v2:172.21.15.173:3300/0 conn(0x7fd7a40a65c0 0x7fd7a413ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:04.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.781+0000 7fd7abc73640 1 --2- 172.21.15.121:0/3230154184 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7a40a5ca0 0x7fd7a40a6080 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:04.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.781+0000 7fd7abc73640 1 -- 172.21.15.121:0/3230154184 >> 172.21.15.121:0/3230154184 conn(0x7fd7a406dd10 msgr2=0x7fd7a40a7da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:04.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.781+0000 7fd7abc73640 1 -- 172.21.15.121:0/3230154184 shutdown_connections 2024-09-17T18:19:04.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.781+0000 7fd7abc73640 1 -- 172.21.15.121:0/3230154184 wait complete. 2024-09-17T18:19:04.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.781+0000 7fd7abc73640 10 monclient: build_initial_monmap 2024-09-17T18:19:04.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.781+0000 7fd7abc73640 10 monclient: monmap: 2024-09-17T18:19:04.810 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:04.810 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:19:04.810 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:19:04.810 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:19:04.810 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:04.810 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:04.810 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:19:04.810 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:19:04.810 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:19:04.810 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:04.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.782+0000 7fd7abc73640 1 librados: starting msgr at 2024-09-17T18:19:04.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.782+0000 7fd7abc73640 1 librados: starting objecter 2024-09-17T18:19:04.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.782+0000 7fd7abc73640 1 Processor -- start 2024-09-17T18:19:04.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.782+0000 7fd7abc73640 1 -- start start 2024-09-17T18:19:04.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.782+0000 7fd7abc73640 1 librados: setting wanted keys 2024-09-17T18:19:04.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.782+0000 7fd7abc73640 1 librados: calling monclient init 2024-09-17T18:19:04.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.782+0000 7fd7abc73640 10 monclient: init 2024-09-17T18:19:04.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.782+0000 7fd7abc73640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:04.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.782+0000 7fd7abc73640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T18:19:04.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.782+0000 7fd7abc73640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7a40a5ca0 0x7fd7a4138040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:04.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.782+0000 7fd7abc73640 10 monclient(hunting): picked mon.noname-c con 0x7fd7a40a5ca0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:04.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.782+0000 7fd7abc73640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7a40a65c0 0x7fd7a4138580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:04.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.782+0000 7fd7abc73640 10 monclient(hunting): picked mon.noname-b con 0x7fd7a40a65c0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:04.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.782+0000 7fd7abc73640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7a413cfc0 0x7fd7a41cd730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:04.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.782+0000 7fd7a9a0f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7a40a5ca0 0x7fd7a4138040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:04.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.782+0000 7fd7a9a0f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7a40a5ca0 0x7fd7a4138040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:59546/0 (socket says 172.21.15.121:59546) 2024-09-17T18:19:04.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.782+0000 7fd7abc73640 10 monclient(hunting): picked mon.noname-a con 0x7fd7a413cfc0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:04.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.782+0000 7fd7a9a0f640 1 -- 172.21.15.121:0/1303253733 learned_addr learned my addr 172.21.15.121:0/1303253733 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:04.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.782+0000 7fd7abc73640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:04.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.782+0000 7fd7abc73640 1 -- 172.21.15.121:0/1303253733 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a40a5670 con 0x7fd7a413cfc0 2024-09-17T18:19:04.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.782+0000 7fd7abc73640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:04.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.782+0000 7fd7abc73640 1 -- 172.21.15.121:0/1303253733 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a40a6eb0 con 0x7fd7a40a65c0 2024-09-17T18:19:04.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.782+0000 7fd7abc73640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:04.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.782+0000 7fd7abc73640 1 -- 172.21.15.121:0/1303253733 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd7a40a69a0 con 0x7fd7a40a5ca0 2024-09-17T18:19:04.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.782+0000 7fd7abc73640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:04.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.782+0000 7fd7abc73640 10 monclient(hunting): authenticate will time out at 748.498910s 2024-09-17T18:19:04.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.782+0000 7fd7a9a0f640 10 monclient(hunting): get_auth_request con 0x7fd7a40a5ca0 auth_method 0 2024-09-17T18:19:04.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.782+0000 7fd7a9a0f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:04.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.782+0000 7fd7a9a0f640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:04.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.782+0000 7fd7a9a0f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:04.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.782+0000 7fd7a9a0f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:04.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.782+0000 7fd7a9a0f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:04.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.782+0000 7fd7a9a0f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:04.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.783+0000 7fd7a920e640 1 --2- 172.21.15.121:0/1303253733 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7a40a65c0 0x7fd7a4138580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:04.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.783+0000 7fd7aa210640 1 --2- 172.21.15.121:0/1303253733 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7a413cfc0 0x7fd7a41cd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:04.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.783+0000 7fd7a9a0f640 10 monclient(hunting): handle_auth_done global_id 4289 payload 824 2024-09-17T18:19:04.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.783+0000 7fd7a9a0f640 1 -- 172.21.15.121:0/1303253733 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7a40a65c0 msgr2=0x7fd7a4138580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:04.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.783+0000 7fd7a9a0f640 1 --2- 172.21.15.121:0/1303253733 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7a40a65c0 0x7fd7a4138580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:04.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.783+0000 7fd7a9a0f640 1 -- 172.21.15.121:0/1303253733 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7a413cfc0 msgr2=0x7fd7a41cd730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:04.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.783+0000 7fd7a9a0f640 1 --2- 172.21.15.121:0/1303253733 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7a413cfc0 0x7fd7a41cd730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:04.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.783+0000 7fd7a9a0f640 10 monclient: _finish_hunting 0 2024-09-17T18:19:04.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.783+0000 7fd7a9a0f640 1 monclient: found mon.noname-c 2024-09-17T18:19:04.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.783+0000 7fd7a9a0f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:04.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.783+0000 7fd7a9a0f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:19:04.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.783+0000 7fd7a9a0f640 1 -- 172.21.15.121:0/1303253733 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7a41236a0 con 0x7fd7a40a5ca0 2024-09-17T18:19:04.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.783+0000 7fd7a920e640 10 monclient: get_auth_request con 0x7fd7a40a65c0 auth_method 0 2024-09-17T18:19:04.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.783+0000 7fd7aa210640 10 monclient: get_auth_request con 0x7fd7a413cfc0 auth_method 0 2024-09-17T18:19:04.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.783+0000 7fd7a920e640 1 --2- 172.21.15.121:0/1303253733 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7a40a65c0 0x7fd7a4138580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T18:19:04.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.783+0000 7fd7aa210640 1 --2- 172.21.15.121:0/1303253733 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7a413cfc0 0x7fd7a41cd730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T18:19:04.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.785+0000 7fd7a9a0f640 1 -- 172.21.15.121:0/1303253733 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7a40a5ca0 msgr2=0x7fd7a4138040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:04.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.785+0000 7fd7a9a0f640 1 --2- 172.21.15.121:0/1303253733 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7a40a5ca0 0x7fd7a4138040 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd7a0000c00 tx=0x7fd7a00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:04.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.785+0000 7fd79a7fc640 10 client.?.objecter ms_handle_connect 0x7fd7a40a5ca0 2024-09-17T18:19:04.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.785+0000 7fd79a7fc640 10 client.?.objecter resend_mon_ops 2024-09-17T18:19:04.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.785+0000 7fd79a7fc640 1 -- 172.21.15.121:0/1303253733 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a0037040 con 0x7fd7a40a5ca0 2024-09-17T18:19:04.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.785+0000 7fd79a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:04.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.785+0000 7fd79a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:19:04.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.785+0000 7fd79a7fc640 10 monclient: dump: 2024-09-17T18:19:04.814 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:04.815 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:04.815 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.815 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.815 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:04.815 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:04.815 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:04.815 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:04.815 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:04.815 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:04.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.785+0000 7fd79a7fc640 10 monclient: _finish_auth 0 2024-09-17T18:19:04.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.785+0000 7fd79a7fc640 10 monclient: _check_auth_tickets 2024-09-17T18:19:04.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.785+0000 7fd79a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:04.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.785+0000 7fd7abc73640 5 monclient: authenticate success, global_id 4289 2024-09-17T18:19:04.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.785+0000 7fd7abc73640 10 monclient: _renew_subs 2024-09-17T18:19:04.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.785+0000 7fd7abc73640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:04.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.785+0000 7fd7abc73640 1 -- 172.21.15.121:0/1303253733 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7a4125160 con 0x7fd7a40a5ca0 2024-09-17T18:19:04.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.785+0000 7fd7abc73640 10 client.4289.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:04.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.785+0000 7fd7abc73640 10 monclient: _renew_subs 2024-09-17T18:19:04.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.785+0000 7fd7abc73640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:04.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.785+0000 7fd7abc73640 1 -- 172.21.15.121:0/1303253733 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7a41cac70 con 0x7fd7a40a5ca0 2024-09-17T18:19:04.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.785+0000 7fd7abc73640 1 librados: init done 2024-09-17T18:19:04.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.785+0000 7fd79a7fc640 1 -- 172.21.15.121:0/1303253733 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7a0042070 con 0x7fd7a40a5ca0 2024-09-17T18:19:04.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.785+0000 7fd79a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:04.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.785+0000 7fd79a7fc640 1 -- 172.21.15.121:0/1303253733 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a003bbc0 con 0x7fd7a40a5ca0 2024-09-17T18:19:04.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.785+0000 7fd79a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:04.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.785+0000 7fd79a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:19:04.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.785+0000 7fd79bfff640 4 set_mon_vals no callback set 2024-09-17T18:19:04.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.785+0000 7fd79a7fc640 10 monclient: dump: 2024-09-17T18:19:04.817 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:04.817 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:04.817 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.817 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.817 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:04.817 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:04.817 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:04.817 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:04.817 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:04.817 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:04.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.786+0000 7fd79a7fc640 1 -- 172.21.15.121:0/1303253733 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fd7a002ad10 con 0x7fd7a40a5ca0 2024-09-17T18:19:04.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.787+0000 7fd7abc73640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:04.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.787+0000 7fd7abc73640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:04.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.787+0000 7fd79a7fc640 1 --2- 172.21.15.121:0/1303253733 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd788040170 0x7fd788042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:04.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.787+0000 7fd7abc73640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:04.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.787+0000 7fd7abc73640 1 -- 172.21.15.121:0/1303253733 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd78c005080 con 0x7fd7a40a5ca0 2024-09-17T18:19:04.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.787+0000 7fd79a7fc640 1 -- 172.21.15.121:0/1303253733 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 3562+0+0 (secure 0 0 0) 0x7fd7a0073650 con 0x7fd7a40a5ca0 2024-09-17T18:19:04.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.787+0000 7fd79a7fc640 10 client.4289.objecter ms_dispatch 0x7fd7a4000b90 osd_map(16..16 src has 1..16) 2024-09-17T18:19:04.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.787+0000 7fd79a7fc640 3 client.4289.objecter handle_osd_map got epochs [16,16] > 0 2024-09-17T18:19:04.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.787+0000 7fd79a7fc640 3 client.4289.objecter handle_osd_map decoding full epoch 16 2024-09-17T18:19:04.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.787+0000 7fd79a7fc640 20 client.4289.objecter dump_active .. 0 homeless 2024-09-17T18:19:04.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.787+0000 7fd7a920e640 1 --2- 172.21.15.121:0/1303253733 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd788040170 0x7fd788042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:04.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.787+0000 7fd7a920e640 10 monclient: get_auth_request con 0x7fd788040170 auth_method 0 2024-09-17T18:19:04.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.790+0000 7fd79a7fc640 10 client.4289.objecter ms_handle_connect 0x7fd788040170 2024-09-17T18:19:04.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.790+0000 7fd7a920e640 1 --2- 172.21.15.121:0/1303253733 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd788040170 0x7fd788042630 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd7a40aa260 tx=0x7fd794008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:04.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.796+0000 7fd79a7fc640 1 -- 172.21.15.121:0/1303253733 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd7a003c3a0 con 0x7fd7a40a5ca0 2024-09-17T18:19:04.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.796+0000 7fd79a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:04.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.796+0000 7fd79a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:19:04.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.823+0000 7fb9a90fe640 10 monclient: get_monmap_and_config 2024-09-17T18:19:04.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.823+0000 7fb9a90fe640 10 monclient: build_initial_monmap 2024-09-17T18:19:04.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.823+0000 7fb9a90fe640 10 monclient: monmap: 2024-09-17T18:19:04.825 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:04.825 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:04.825 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:19:04.823722+0000 2024-09-17T18:19:04.825 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:19:04.823722+0000 2024-09-17T18:19:04.825 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:04.825 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:04.825 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:19:04.825 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:19:04.825 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:19:04.825 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:19:04.826 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:19:04.826 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:04.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.823+0000 7fb9a90fe640 1 Processor -- start 2024-09-17T18:19:04.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.823+0000 7fb9a90fe640 1 -- start start 2024-09-17T18:19:04.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.823+0000 7fb9a90fe640 10 monclient: init 2024-09-17T18:19:04.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.824+0000 7fb9a90fe640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:04.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.824+0000 7fb9a90fe640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-17T18:19:04.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.824+0000 7fb9a90fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb9a40a5ca0 addr v1:172.21.15.173:6789/0 2024-09-17T18:19:04.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.824+0000 7fb9a90fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9a40a6790 0x7fb9a413da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:04.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.824+0000 7fb9a90fe640 10 monclient(hunting): picked mon.noname-b con 0x7fb9a40a6790 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:04.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.824+0000 7fb9a90fe640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fb9a413dfd0 0x7fb9a41403c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:04.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.824+0000 7fb9a90fe640 10 monclient(hunting): picked mon.noname-c con 0x7fb9a413dfd0 addr v2:172.21.15.173:3300/0 2024-09-17T18:19:04.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.824+0000 7fb9a90fe640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9a40a7b50 con 0x7fb9a40a5ca0 2024-09-17T18:19:04.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.824+0000 7fb9a90fe640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:04.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.824+0000 7fb9a90fe640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fb9a40704c0 con 0x7fb9a413dfd0 2024-09-17T18:19:04.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.824+0000 7fb9a90fe640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:04.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.824+0000 7fb9a90fe640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb9a40a5490 con 0x7fb9a40a6790 2024-09-17T18:19:04.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.824+0000 7fb9a90fe640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:04.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.824+0000 7fb9a90fe640 10 monclient(hunting): authenticate will time out at 748.540305s 2024-09-17T18:19:04.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.824+0000 7fb9a259c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9a40a6790 0x7fb9a413da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:04.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.824+0000 7fb9a259c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9a40a6790 0x7fb9a413da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:59548/0 (socket says 172.21.15.121:59548) 2024-09-17T18:19:04.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.824+0000 7fb9a259c640 1 -- 172.21.15.121:0/3857737895 learned_addr learned my addr 172.21.15.121:0/3857737895 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:04.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.824+0000 7fb9a259c640 10 monclient(hunting): get_auth_request con 0x7fb9a40a6790 auth_method 0 2024-09-17T18:19:04.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.824+0000 7fb9a259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:04.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.824+0000 7fb9a259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:04.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.824+0000 7fb9a259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:04.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.824+0000 7fb9a359e640 1 --2- 172.21.15.121:0/3857737895 >> v2:172.21.15.173:3300/0 conn(0x7fb9a413dfd0 0x7fb9a41403c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:04.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.824+0000 7fb9a259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:04.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.824+0000 7fb9a259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:04.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.824+0000 7fb9a259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:04.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.824+0000 7fb9a359e640 10 monclient(hunting): get_auth_request con 0x7fb9a413dfd0 auth_method 0 2024-09-17T18:19:04.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.824+0000 7fb9a359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:04.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.824+0000 7fb9a359e640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:04.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.824+0000 7fb9a359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:04.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.824+0000 7fb9a359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:04.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.824+0000 7fb9a359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:04.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.824+0000 7fb9a359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:04.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.824+0000 7fb9a259c640 10 monclient(hunting): handle_auth_done global_id 4292 payload 274 2024-09-17T18:19:04.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.824+0000 7fb9a259c640 1 -- 172.21.15.121:0/3857737895 >> v2:172.21.15.173:3300/0 conn(0x7fb9a413dfd0 msgr2=0x7fb9a41403c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:04.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.824+0000 7fb9a259c640 1 --2- 172.21.15.121:0/3857737895 >> v2:172.21.15.173:3300/0 conn(0x7fb9a413dfd0 0x7fb9a41403c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:04.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.824+0000 7fb9a259c640 1 -- 172.21.15.121:0/3857737895 >> v1:172.21.15.173:6789/0 conn(0x7fb9a40a5ca0 legacy=0x7fb9a40a6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:04.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.824+0000 7fb9a259c640 10 monclient: _finish_hunting 0 2024-09-17T18:19:04.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.824+0000 7fb9a259c640 1 monclient: found mon.noname-b 2024-09-17T18:19:04.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.824+0000 7fb9a259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:04.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.824+0000 7fb9a259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:19:04.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.824+0000 7fb9a259c640 1 -- 172.21.15.121:0/3857737895 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9a40a2a50 con 0x7fb9a40a6790 2024-09-17T18:19:04.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.827+0000 7fb9a259c640 1 -- 172.21.15.121:0/3857737895 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9a40a6790 msgr2=0x7fb9a413da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:04.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.827+0000 7fb9a259c640 1 --2- 172.21.15.121:0/3857737895 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9a40a6790 0x7fb9a413da90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb998009870 tx=0x7fb99802a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ef72884cd2f1d0b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:04.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.827+0000 7fb9a1d9b640 1 -- 172.21.15.121:0/3857737895 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb998037070 con 0x7fb9a40a6790 2024-09-17T18:19:04.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.827+0000 7fb9a1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:04.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.827+0000 7fb9a1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:19:04.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.827+0000 7fb9a1d9b640 10 monclient: dump: 2024-09-17T18:19:04.837 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:04.837 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:04.837 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.837 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.837 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:04.837 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:04.837 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:04.837 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:04.837 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:04.837 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:04.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.827+0000 7fb9a1d9b640 10 monclient: _finish_auth 0 2024-09-17T18:19:04.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.827+0000 7fb9a1d9b640 10 monclient: _check_auth_tickets 2024-09-17T18:19:04.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.827+0000 7fb9a1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:04.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.827+0000 7fb9a90fe640 5 monclient: authenticate success, global_id 4292 2024-09-17T18:19:04.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.827+0000 7fb9a90fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:19:04.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.827+0000 7fb9a1d9b640 1 -- 172.21.15.121:0/3857737895 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb998032440 con 0x7fb9a40a6790 2024-09-17T18:19:04.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.827+0000 7fb9a1d9b640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:04.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.827+0000 7fb9a90fe640 10 monclient: get_monmap_and_config success 2024-09-17T18:19:04.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.827+0000 7fb9a90fe640 4 set_mon_vals no callback set 2024-09-17T18:19:04.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.827+0000 7fb9a1d9b640 1 -- 172.21.15.121:0/3857737895 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb998032810 con 0x7fb9a40a6790 2024-09-17T18:19:04.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.827+0000 7fb9a1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:04.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.827+0000 7fb9a1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:19:04.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.827+0000 7fb9a1d9b640 10 monclient: dump: 2024-09-17T18:19:04.838 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:04.838 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:04.839 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.839 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.839 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:04.839 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:04.839 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:04.839 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:04.839 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:04.839 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:04.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.828+0000 7fb9a90fe640 10 monclient: shutdown 2024-09-17T18:19:04.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.828+0000 7fb9a90fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:04.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.828+0000 7fb9a90fe640 1 -- 172.21.15.121:0/3857737895 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9a40a6790 msgr2=0x7fb9a413da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:04.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.828+0000 7fb9a90fe640 1 --2- 172.21.15.121:0/3857737895 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9a40a6790 0x7fb9a413da90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb998009870 tx=0x7fb99802a0a0 comp rx=0 tx=0).stop 2024-09-17T18:19:04.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.828+0000 7fb9a90fe640 1 -- 172.21.15.121:0/3857737895 shutdown_connections 2024-09-17T18:19:04.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.828+0000 7fb9a90fe640 1 --2- 172.21.15.121:0/3857737895 >> v2:172.21.15.173:3300/0 conn(0x7fb9a413dfd0 0x7fb9a41403c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:04.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.828+0000 7fb9a90fe640 1 --2- 172.21.15.121:0/3857737895 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9a40a6790 0x7fb9a413da90 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:04.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.828+0000 7fb9a90fe640 1 -- 172.21.15.121:0/3857737895 >> 172.21.15.121:0/3857737895 conn(0x7fb9a406dd10 msgr2=0x7fb9a40a7da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:04.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.828+0000 7fb9a90fe640 1 -- 172.21.15.121:0/3857737895 shutdown_connections 2024-09-17T18:19:04.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.828+0000 7fb9a90fe640 1 -- 172.21.15.121:0/3857737895 wait complete. 2024-09-17T18:19:04.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.829+0000 7fb9a90fe640 10 monclient: build_initial_monmap 2024-09-17T18:19:04.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.829+0000 7fb9a90fe640 10 monclient: monmap: 2024-09-17T18:19:04.840 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:04.840 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:19:04.840 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:19:04.840 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:19:04.840 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:04.840 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:04.840 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:19:04.840 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:19:04.840 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:19:04.841 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:04.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.829+0000 7fb9a90fe640 1 librados: starting msgr at 2024-09-17T18:19:04.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.829+0000 7fb9a90fe640 1 librados: starting objecter 2024-09-17T18:19:04.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.829+0000 7fb9a90fe640 1 Processor -- start 2024-09-17T18:19:04.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.829+0000 7fb9a90fe640 1 -- start start 2024-09-17T18:19:04.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.829+0000 7fb9a90fe640 1 librados: setting wanted keys 2024-09-17T18:19:04.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.829+0000 7fb9a90fe640 1 librados: calling monclient init 2024-09-17T18:19:04.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.829+0000 7fb9a90fe640 10 monclient: init 2024-09-17T18:19:04.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a90fe640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:04.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a90fe640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T18:19:04.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a90fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9a40a5ca0 0x7fb9a4138030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:04.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a90fe640 10 monclient(hunting): picked mon.noname-b con 0x7fb9a40a5ca0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:04.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a90fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9a40a6790 0x7fb9a4138570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:04.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a90fe640 10 monclient(hunting): picked mon.noname-c con 0x7fb9a40a6790 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:04.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a90fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb9a413dfd0 0x7fb9a41ca1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:04.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a90fe640 10 monclient(hunting): picked mon.noname-a con 0x7fb9a413dfd0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:04.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a90fe640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:04.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a90fe640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a40a5670 con 0x7fb9a413dfd0 2024-09-17T18:19:04.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a90fe640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:04.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a90fe640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a40a6eb0 con 0x7fb9a40a5ca0 2024-09-17T18:19:04.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a90fe640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:04.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a90fe640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb9a40a8720 con 0x7fb9a40a6790 2024-09-17T18:19:04.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a90fe640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:04.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a90fe640 10 monclient(hunting): authenticate will time out at 748.546512s 2024-09-17T18:19:04.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a259c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9a40a6790 0x7fb9a4138570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:04.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a2d9d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9a40a5ca0 0x7fb9a4138030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:04.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a259c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9a40a6790 0x7fb9a4138570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:59552/0 (socket says 172.21.15.121:59552) 2024-09-17T18:19:04.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a259c640 1 -- 172.21.15.121:0/1110227036 learned_addr learned my addr 172.21.15.121:0/1110227036 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:04.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a259c640 10 monclient(hunting): get_auth_request con 0x7fb9a40a6790 auth_method 0 2024-09-17T18:19:04.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:04.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:04.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:04.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a2d9d640 10 monclient(hunting): get_auth_request con 0x7fb9a40a5ca0 auth_method 0 2024-09-17T18:19:04.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:04.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:04.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:04.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:04.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:04.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:04.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a359e640 1 --2- 172.21.15.121:0/1110227036 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb9a413dfd0 0x7fb9a41ca1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:04.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a359e640 10 monclient(hunting): get_auth_request con 0x7fb9a413dfd0 auth_method 0 2024-09-17T18:19:04.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:04.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a359e640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:04.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:04.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:04.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:04.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:04.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:04.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:04.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:04.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a259c640 10 monclient(hunting): handle_auth_done global_id 4295 payload 824 2024-09-17T18:19:04.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a259c640 1 -- 172.21.15.121:0/1110227036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9a40a5ca0 msgr2=0x7fb9a4138030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:04.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a259c640 1 --2- 172.21.15.121:0/1110227036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9a40a5ca0 0x7fb9a4138030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:04.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a259c640 1 -- 172.21.15.121:0/1110227036 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb9a413dfd0 msgr2=0x7fb9a41ca1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:04.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.830+0000 7fb9a259c640 1 --2- 172.21.15.121:0/1110227036 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb9a413dfd0 0x7fb9a41ca1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:04.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.831+0000 7fb9a259c640 10 monclient: _finish_hunting 0 2024-09-17T18:19:04.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.831+0000 7fb9a259c640 1 monclient: found mon.noname-c 2024-09-17T18:19:04.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.831+0000 7fb9a259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:04.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.831+0000 7fb9a259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:19:04.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.831+0000 7fb9a259c640 1 -- 172.21.15.121:0/1110227036 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9a40a0eb0 con 0x7fb9a40a6790 2024-09-17T18:19:04.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.831+0000 7fb9a2d9d640 1 --2- 172.21.15.121:0/1110227036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9a40a5ca0 0x7fb9a4138030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:19:04.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.831+0000 7fb9a359e640 1 --2- 172.21.15.121:0/1110227036 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb9a413dfd0 0x7fb9a41ca1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:19:04.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.833+0000 7fb9a259c640 1 -- 172.21.15.121:0/1110227036 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9a40a6790 msgr2=0x7fb9a4138570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:04.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.833+0000 7fb98b7fe640 10 client.?.objecter ms_handle_connect 0x7fb9a40a6790 2024-09-17T18:19:04.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.833+0000 7fb98b7fe640 10 client.?.objecter resend_mon_ops 2024-09-17T18:19:04.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.833+0000 7fb9a259c640 1 --2- 172.21.15.121:0/1110227036 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9a40a6790 0x7fb9a4138570 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb998009870 tx=0x7fb9980046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:04.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.833+0000 7fb98b7fe640 1 -- 172.21.15.121:0/1110227036 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb998037040 con 0x7fb9a40a6790 2024-09-17T18:19:04.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.833+0000 7fb98b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:04.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.833+0000 7fb98b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:19:04.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.833+0000 7fb98b7fe640 10 monclient: dump: 2024-09-17T18:19:04.846 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:04.846 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:04.846 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.846 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.846 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:04.846 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:04.846 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:04.846 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:04.846 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:04.846 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:04.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.833+0000 7fb98b7fe640 10 monclient: _finish_auth 0 2024-09-17T18:19:04.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.833+0000 7fb98b7fe640 10 monclient: _check_auth_tickets 2024-09-17T18:19:04.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.833+0000 7fb98b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:04.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.833+0000 7fb9a90fe640 5 monclient: authenticate success, global_id 4295 2024-09-17T18:19:04.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.833+0000 7fb9a90fe640 10 monclient: _renew_subs 2024-09-17T18:19:04.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.833+0000 7fb9a90fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:04.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.833+0000 7fb9a90fe640 1 -- 172.21.15.121:0/1110227036 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9a4124140 con 0x7fb9a40a6790 2024-09-17T18:19:04.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.833+0000 7fb98b7fe640 1 -- 172.21.15.121:0/1110227036 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb998042070 con 0x7fb9a40a6790 2024-09-17T18:19:04.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.833+0000 7fb98b7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:04.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.833+0000 7fb98b7fe640 1 -- 172.21.15.121:0/1110227036 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb99803b820 con 0x7fb9a40a6790 2024-09-17T18:19:04.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.833+0000 7fb98b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:04.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.833+0000 7fb9a90fe640 10 client.4295.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:04.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.833+0000 7fb98b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:19:04.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.833+0000 7fb98b7fe640 10 monclient: dump: 2024-09-17T18:19:04.848 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:04.848 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:04.848 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.848 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:04.848 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:04.848 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:04.848 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:04.848 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:04.848 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:04.848 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:04.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.833+0000 7fb9a159a640 4 set_mon_vals no callback set 2024-09-17T18:19:04.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.833+0000 7fb9a90fe640 10 monclient: _renew_subs 2024-09-17T18:19:04.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.833+0000 7fb9a90fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:04.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.833+0000 7fb9a90fe640 1 -- 172.21.15.121:0/1110227036 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9a41cdbc0 con 0x7fb9a40a6790 2024-09-17T18:19:04.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.833+0000 7fb9a90fe640 1 librados: init done 2024-09-17T18:19:04.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.834+0000 7fb9a90fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:04.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.834+0000 7fb9a90fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:04.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.834+0000 7fb9a90fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:04.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.834+0000 7fb9a90fe640 1 -- 172.21.15.121:0/1110227036 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9a40a73f0 con 0x7fb9a40a6790 2024-09-17T18:19:04.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.835+0000 7fb98b7fe640 1 -- 172.21.15.121:0/1110227036 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fb99803b9c0 con 0x7fb9a40a6790 2024-09-17T18:19:04.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.835+0000 7fb98b7fe640 1 --2- 172.21.15.121:0/1110227036 >> v2:172.21.15.173:6812/2423786037 conn(0x7fb9900401a0 0x7fb990042660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:04.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.835+0000 7fb98b7fe640 1 -- 172.21.15.121:0/1110227036 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 3562+0+0 (secure 0 0 0) 0x7fb998073970 con 0x7fb9a40a6790 2024-09-17T18:19:04.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.835+0000 7fb98b7fe640 10 client.4295.objecter ms_dispatch 0x7fb9a4000b90 osd_map(16..16 src has 1..16) 2024-09-17T18:19:04.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.835+0000 7fb98b7fe640 3 client.4295.objecter handle_osd_map got epochs [16,16] > 0 2024-09-17T18:19:04.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.835+0000 7fb98b7fe640 3 client.4295.objecter handle_osd_map decoding full epoch 16 2024-09-17T18:19:04.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.835+0000 7fb98b7fe640 20 client.4295.objecter dump_active .. 0 homeless 2024-09-17T18:19:04.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.836+0000 7fb9a2d9d640 1 --2- 172.21.15.121:0/1110227036 >> v2:172.21.15.173:6812/2423786037 conn(0x7fb9900401a0 0x7fb990042660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:04.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.836+0000 7fb9a2d9d640 10 monclient: get_auth_request con 0x7fb9900401a0 auth_method 0 2024-09-17T18:19:04.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.838+0000 7fb98b7fe640 10 client.4295.objecter ms_handle_connect 0x7fb9900401a0 2024-09-17T18:19:04.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.838+0000 7fb9a2d9d640 1 --2- 172.21.15.121:0/1110227036 >> v2:172.21.15.173:6812/2423786037 conn(0x7fb9900401a0 0x7fb990042660 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb994004320 tx=0x7fb994007e30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:04.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.843+0000 7fb98b7fe640 1 -- 172.21.15.121:0/1110227036 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb9a40a73f0 con 0x7fb9a40a6790 2024-09-17T18:19:04.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.843+0000 7fb98b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:04.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.843+0000 7fb98b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:19:04.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.952+0000 7f8667f7d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-17T18:19:04.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.952+0000 7f8667f7d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-17T18:19:04.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.952+0000 7f8667f7d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:04.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.952+0000 7f8667f7d640 1 -- 172.21.15.121:0/2246933401 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f8660139680 con 0x7f86600a6790 2024-09-17T18:19:04.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.953+0000 7f86567fc640 1 -- 172.21.15.121:0/2246933401 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f865001cbd0 con 0x7f86600a6790 2024-09-17T18:19:04.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.953+0000 7f86567fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-17T18:19:04.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.953+0000 7f86567fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:19:04.955 INFO:teuthology.orchestra.run.smithi121.stdout:51539607556 2024-09-17T18:19:04.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.954+0000 7f8667f7d640 10 librados: watch_flush enter 2024-09-17T18:19:04.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.955+0000 7f8667f7d640 10 librados: watch_flush exit 2024-09-17T18:19:04.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.955+0000 7f8667f7d640 20 client.4299.objecter shutdown clearing up homeless session... 2024-09-17T18:19:04.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.955+0000 7f8667f7d640 10 client.4299.objecter successfully canceled tick 2024-09-17T18:19:04.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.955+0000 7f8667f7d640 1 -- 172.21.15.121:0/2246933401 >> v2:172.21.15.173:6812/2423786037 conn(0x7f8644040170 msgr2=0x7f8644042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:04.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.955+0000 7f8667f7d640 1 --2- 172.21.15.121:0/2246933401 >> v2:172.21.15.173:6812/2423786037 conn(0x7f8644040170 0x7f8644042630 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f865c007c90 tx=0x7f865c007be0 comp rx=0 tx=0).stop 2024-09-17T18:19:04.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.955+0000 7f8667f7d640 10 monclient: shutdown 2024-09-17T18:19:04.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.955+0000 7f8667f7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:04.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.955+0000 7f8667f7d640 1 -- 172.21.15.121:0/2246933401 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86600a6790 msgr2=0x7f8660139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:04.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.955+0000 7f8667f7d640 1 --2- 172.21.15.121:0/2246933401 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86600a6790 0x7f8660139140 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f865000c430 tx=0x7f865000c8f0 comp rx=0 tx=0).stop 2024-09-17T18:19:04.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.955+0000 7f8667f7d640 1 -- 172.21.15.121:0/2246933401 shutdown_connections 2024-09-17T18:19:04.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.955+0000 7f8667f7d640 1 --2- 172.21.15.121:0/2246933401 >> v2:172.21.15.173:6812/2423786037 conn(0x7f8644040170 0x7f8644042630 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:04.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.955+0000 7f8667f7d640 1 --2- 172.21.15.121:0/2246933401 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f866013f1b0 0x7f86601218f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:04.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.955+0000 7f8667f7d640 1 --2- 172.21.15.121:0/2246933401 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86600a6790 0x7f8660139140 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:04.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.955+0000 7f8667f7d640 1 --2- 172.21.15.121:0/2246933401 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f86600a5ca0 0x7f86600a1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:04.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.955+0000 7f8667f7d640 1 -- 172.21.15.121:0/2246933401 >> 172.21.15.121:0/2246933401 conn(0x7f866006dd10 msgr2=0x7f866013a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:04.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.955+0000 7f8667f7d640 1 -- 172.21.15.121:0/2246933401 shutdown_connections 2024-09-17T18:19:04.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.955+0000 7f8667f7d640 1 -- 172.21.15.121:0/2246933401 wait complete. 2024-09-17T18:19:04.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:04.955+0000 7f8667f7d640 1 librados: shutdown 2024-09-17T18:19:04.967 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607556 for osd.4 2024-09-17T18:19:04.967 DEBUG:teuthology.parallel:result is None 2024-09-17T18:19:05.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.000+0000 7f3d6d29b640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-17T18:19:05.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.000+0000 7f3d6d29b640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-17T18:19:05.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.000+0000 7f3d6d29b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:05.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.000+0000 7f3d6d29b640 1 -- 172.21.15.121:0/4130684975 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f3d680a4750 con 0x7f3d680a5ca0 2024-09-17T18:19:05.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.000+0000 7f3d4f7fe640 1 -- 172.21.15.121:0/4130684975 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3d680a4750 con 0x7f3d680a5ca0 2024-09-17T18:19:05.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.000+0000 7f3d4f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-17T18:19:05.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.000+0000 7f3d4f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:19:05.003 INFO:teuthology.orchestra.run.smithi121.stdout:47244640260 2024-09-17T18:19:05.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.002+0000 7f3d6d29b640 10 librados: watch_flush enter 2024-09-17T18:19:05.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.002+0000 7f3d6d29b640 10 librados: watch_flush exit 2024-09-17T18:19:05.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.002+0000 7f3d6d29b640 20 client.4305.objecter shutdown clearing up homeless session... 2024-09-17T18:19:05.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.002+0000 7f3d6d29b640 10 client.4305.objecter successfully canceled tick 2024-09-17T18:19:05.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.002+0000 7f3d6d29b640 1 -- 172.21.15.121:0/4130684975 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3d54040170 msgr2=0x7f3d54042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:05.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.002+0000 7f3d6d29b640 1 --2- 172.21.15.121:0/4130684975 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3d54040170 0x7f3d54042630 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3d6809f090 tx=0x7f3d5c008040 comp rx=0 tx=0).stop 2024-09-17T18:19:05.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.002+0000 7f3d6d29b640 10 monclient: shutdown 2024-09-17T18:19:05.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.002+0000 7f3d6d29b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:05.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.002+0000 7f3d6d29b640 1 -- 172.21.15.121:0/4130684975 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d680a5ca0 msgr2=0x7f3d6809d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:05.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.002+0000 7f3d6d29b640 1 --2- 172.21.15.121:0/4130684975 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d680a5ca0 0x7f3d6809d0b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3d5800c460 tx=0x7f3d5800c920 comp rx=0 tx=0).stop 2024-09-17T18:19:05.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.003+0000 7f3d6d29b640 1 -- 172.21.15.121:0/4130684975 shutdown_connections 2024-09-17T18:19:05.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.003+0000 7f3d6d29b640 1 --2- 172.21.15.121:0/4130684975 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3d54040170 0x7f3d54042630 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:05.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.003+0000 7f3d6d29b640 1 --2- 172.21.15.121:0/4130684975 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d680a6790 0x7f3d6809d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:05.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.003+0000 7f3d6d29b640 1 --2- 172.21.15.121:0/4130684975 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d6813dfd0 0x7f3d6809db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:05.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.003+0000 7f3d6d29b640 1 --2- 172.21.15.121:0/4130684975 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d680a5ca0 0x7f3d6809d0b0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:05.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.003+0000 7f3d6d29b640 1 -- 172.21.15.121:0/4130684975 >> 172.21.15.121:0/4130684975 conn(0x7f3d6806dd10 msgr2=0x7f3d6813c870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:05.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.003+0000 7f3d6d29b640 1 -- 172.21.15.121:0/4130684975 shutdown_connections 2024-09-17T18:19:05.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.003+0000 7f3d6d29b640 1 -- 172.21.15.121:0/4130684975 wait complete. 2024-09-17T18:19:05.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.003+0000 7f3d6d29b640 1 librados: shutdown 2024-09-17T18:19:05.015 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.2 2024-09-17T18:19:05.015 DEBUG:teuthology.parallel:result is None 2024-09-17T18:19:05.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.020+0000 7fd7abc73640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-17T18:19:05.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.020+0000 7fd7abc73640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-17T18:19:05.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.020+0000 7fd7abc73640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:05.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.020+0000 7fd7abc73640 1 -- 172.21.15.121:0/1303253733 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fd78c005370 con 0x7fd7a40a5ca0 2024-09-17T18:19:05.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.020+0000 7fd79a7fc640 1 -- 172.21.15.121:0/1303253733 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd7a003bde0 con 0x7fd7a40a5ca0 2024-09-17T18:19:05.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.020+0000 7fd79a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-17T18:19:05.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.020+0000 7fd79a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:19:05.022 INFO:teuthology.orchestra.run.smithi121.stdout:47244640259 2024-09-17T18:19:05.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.022+0000 7fd7abc73640 10 librados: watch_flush enter 2024-09-17T18:19:05.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.022+0000 7fd7abc73640 10 librados: watch_flush exit 2024-09-17T18:19:05.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.022+0000 7fd7abc73640 20 client.4289.objecter shutdown clearing up homeless session... 2024-09-17T18:19:05.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.022+0000 7fd7abc73640 10 client.4289.objecter successfully canceled tick 2024-09-17T18:19:05.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.022+0000 7fd7abc73640 1 -- 172.21.15.121:0/1303253733 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd788040170 msgr2=0x7fd788042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:05.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.022+0000 7fd7abc73640 1 --2- 172.21.15.121:0/1303253733 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd788040170 0x7fd788042630 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd7a40aa260 tx=0x7fd794008040 comp rx=0 tx=0).stop 2024-09-17T18:19:05.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.022+0000 7fd7abc73640 10 monclient: shutdown 2024-09-17T18:19:05.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.022+0000 7fd7abc73640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:05.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.022+0000 7fd7abc73640 1 -- 172.21.15.121:0/1303253733 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7a40a5ca0 msgr2=0x7fd7a4138040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:05.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.022+0000 7fd7abc73640 1 --2- 172.21.15.121:0/1303253733 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7a40a5ca0 0x7fd7a4138040 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd7a0000c00 tx=0x7fd7a00046c0 comp rx=0 tx=0).stop 2024-09-17T18:19:05.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.022+0000 7fd7abc73640 1 -- 172.21.15.121:0/1303253733 shutdown_connections 2024-09-17T18:19:05.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.022+0000 7fd7abc73640 1 --2- 172.21.15.121:0/1303253733 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd788040170 0x7fd788042630 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:05.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.022+0000 7fd7abc73640 1 --2- 172.21.15.121:0/1303253733 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7a40a65c0 0x7fd7a4138580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:05.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.022+0000 7fd7abc73640 1 --2- 172.21.15.121:0/1303253733 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7a413cfc0 0x7fd7a41cd730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:05.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.022+0000 7fd7abc73640 1 --2- 172.21.15.121:0/1303253733 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7a40a5ca0 0x7fd7a4138040 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:05.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.022+0000 7fd7abc73640 1 -- 172.21.15.121:0/1303253733 >> 172.21.15.121:0/1303253733 conn(0x7fd7a406dd10 msgr2=0x7fd7a413ad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:05.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.022+0000 7fd7abc73640 1 -- 172.21.15.121:0/1303253733 shutdown_connections 2024-09-17T18:19:05.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.022+0000 7fd7abc73640 1 -- 172.21.15.121:0/1303253733 wait complete. 2024-09-17T18:19:05.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.022+0000 7fd7abc73640 1 librados: shutdown 2024-09-17T18:19:05.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.026+0000 7fb9a90fe640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-17T18:19:05.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.026+0000 7fb9a90fe640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-17T18:19:05.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.026+0000 7fb9a90fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:05.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.026+0000 7fb9a90fe640 1 -- 172.21.15.121:0/1110227036 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fb9a40a6080 con 0x7fb9a40a6790 2024-09-17T18:19:05.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.026+0000 7fb98b7fe640 1 -- 172.21.15.121:0/1110227036 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb99804b020 con 0x7fb9a40a6790 2024-09-17T18:19:05.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.026+0000 7fb98b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-17T18:19:05.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.026+0000 7fb98b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:19:05.029 INFO:teuthology.orchestra.run.smithi121.stdout:47244640260 2024-09-17T18:19:05.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.028+0000 7fb9a90fe640 10 librados: watch_flush enter 2024-09-17T18:19:05.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.028+0000 7fb9a90fe640 10 librados: watch_flush exit 2024-09-17T18:19:05.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.029+0000 7fb9a90fe640 20 client.4295.objecter shutdown clearing up homeless session... 2024-09-17T18:19:05.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.029+0000 7fb9a90fe640 10 client.4295.objecter successfully canceled tick 2024-09-17T18:19:05.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.029+0000 7fb9a90fe640 1 -- 172.21.15.121:0/1110227036 >> v2:172.21.15.173:6812/2423786037 conn(0x7fb9900401a0 msgr2=0x7fb990042660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:05.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.029+0000 7fb9a90fe640 1 --2- 172.21.15.121:0/1110227036 >> v2:172.21.15.173:6812/2423786037 conn(0x7fb9900401a0 0x7fb990042660 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb994004320 tx=0x7fb994007e30 comp rx=0 tx=0).stop 2024-09-17T18:19:05.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.029+0000 7fb9a90fe640 10 monclient: shutdown 2024-09-17T18:19:05.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.029+0000 7fb9a90fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:05.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.029+0000 7fb9a90fe640 1 -- 172.21.15.121:0/1110227036 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9a40a6790 msgr2=0x7fb9a4138570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:05.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.029+0000 7fb9a90fe640 1 --2- 172.21.15.121:0/1110227036 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9a40a6790 0x7fb9a4138570 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb998009870 tx=0x7fb9980046c0 comp rx=0 tx=0).stop 2024-09-17T18:19:05.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.029+0000 7fb9a90fe640 1 -- 172.21.15.121:0/1110227036 shutdown_connections 2024-09-17T18:19:05.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.029+0000 7fb9a90fe640 1 --2- 172.21.15.121:0/1110227036 >> v2:172.21.15.173:6812/2423786037 conn(0x7fb9900401a0 0x7fb990042660 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:05.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.029+0000 7fb9a90fe640 1 --2- 172.21.15.121:0/1110227036 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb9a413dfd0 0x7fb9a41ca1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:05.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.029+0000 7fb9a90fe640 1 --2- 172.21.15.121:0/1110227036 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9a40a6790 0x7fb9a4138570 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:05.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.029+0000 7fb9a90fe640 1 --2- 172.21.15.121:0/1110227036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9a40a5ca0 0x7fb9a4138030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:05.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.029+0000 7fb9a90fe640 1 -- 172.21.15.121:0/1110227036 >> 172.21.15.121:0/1110227036 conn(0x7fb9a406dd10 msgr2=0x7fb9a4138c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:05.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.029+0000 7fb9a90fe640 1 -- 172.21.15.121:0/1110227036 shutdown_connections 2024-09-17T18:19:05.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.029+0000 7fb9a90fe640 1 -- 172.21.15.121:0/1110227036 wait complete. 2024-09-17T18:19:05.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.029+0000 7fb9a90fe640 1 librados: shutdown 2024-09-17T18:19:05.035 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.5 2024-09-17T18:19:05.035 DEBUG:teuthology.parallel:result is None 2024-09-17T18:19:05.041 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.3 2024-09-17T18:19:05.041 DEBUG:teuthology.parallel:result is None 2024-09-17T18:19:05.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.449+0000 7fac48ff9640 1 -- 172.21.15.121:0/1490544094 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fac3c04b030 con 0x7fac4c0700f0 2024-09-17T18:19:05.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.449+0000 7fac48ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:05.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.449+0000 7fac48ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:19:05.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.489+0000 7f0ce17fa640 1 -- 172.21.15.121:0/3857287722 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0cd803a350 con 0x7f0ce40a5ca0 2024-09-17T18:19:05.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.489+0000 7f0ce17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:05.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.489+0000 7f0ce17fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:19:05.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.629+0000 7fac51e72640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-17T18:19:05.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.629+0000 7fac51e72640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-17T18:19:05.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.629+0000 7fac51e72640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:05.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.629+0000 7fac51e72640 1 -- 172.21.15.121:0/1490544094 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fac4c06a660 con 0x7fac4c0700f0 2024-09-17T18:19:05.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.630+0000 7fac48ff9640 1 -- 172.21.15.121:0/1490544094 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fac3c072260 con 0x7fac4c0700f0 2024-09-17T18:19:05.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.630+0000 7fac48ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-17T18:19:05.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.630+0000 7fac48ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:19:05.632 INFO:teuthology.orchestra.run.smithi121.stdout:51539607555 2024-09-17T18:19:05.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.632+0000 7fac51e72640 10 librados: watch_flush enter 2024-09-17T18:19:05.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.632+0000 7fac51e72640 10 librados: watch_flush exit 2024-09-17T18:19:05.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.632+0000 7fac51e72640 20 client.4268.objecter shutdown clearing up homeless session... 2024-09-17T18:19:05.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.632+0000 7fac51e72640 10 client.4268.objecter successfully canceled tick 2024-09-17T18:19:05.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.632+0000 7fac51e72640 1 -- 172.21.15.121:0/1490544094 >> v2:172.21.15.173:6812/2423786037 conn(0x7fac2c040170 msgr2=0x7fac2c042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:05.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.632+0000 7fac51e72640 1 --2- 172.21.15.121:0/1490544094 >> v2:172.21.15.173:6812/2423786037 conn(0x7fac2c040170 0x7fac2c042630 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fac4c1329c0 tx=0x7fac34007e30 comp rx=0 tx=0).stop 2024-09-17T18:19:05.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.632+0000 7fac51e72640 10 monclient: shutdown 2024-09-17T18:19:05.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.632+0000 7fac51e72640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:05.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.632+0000 7fac51e72640 1 -- 172.21.15.121:0/1490544094 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fac4c0700f0 msgr2=0x7fac4c131bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:05.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.632+0000 7fac51e72640 1 --2- 172.21.15.121:0/1490544094 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fac4c0700f0 0x7fac4c131bc0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fac3c0090b0 tx=0x7fac3c0046c0 comp rx=0 tx=0).stop 2024-09-17T18:19:05.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.632+0000 7fac51e72640 1 -- 172.21.15.121:0/1490544094 shutdown_connections 2024-09-17T18:19:05.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.632+0000 7fac51e72640 1 --2- 172.21.15.121:0/1490544094 >> v2:172.21.15.173:6812/2423786037 conn(0x7fac2c040170 0x7fac2c042630 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:05.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.632+0000 7fac51e72640 1 --2- 172.21.15.121:0/1490544094 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fac4c09e810 0x7fac4c132100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:05.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.632+0000 7fac51e72640 1 --2- 172.21.15.121:0/1490544094 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac4c09ec30 0x7fac4c0a0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:05.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.632+0000 7fac51e72640 1 --2- 172.21.15.121:0/1490544094 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fac4c0700f0 0x7fac4c131bc0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:05.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.632+0000 7fac51e72640 1 -- 172.21.15.121:0/1490544094 >> 172.21.15.121:0/1490544094 conn(0x7fac4c0ab830 msgr2=0x7fac4c133ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:05.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.632+0000 7fac51e72640 1 -- 172.21.15.121:0/1490544094 shutdown_connections 2024-09-17T18:19:05.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.632+0000 7fac51e72640 1 -- 172.21.15.121:0/1490544094 wait complete. 2024-09-17T18:19:05.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.632+0000 7fac51e72640 1 librados: shutdown 2024-09-17T18:19:05.644 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607555 got 51539607555 for osd.1 2024-09-17T18:19:05.644 DEBUG:teuthology.parallel:result is None 2024-09-17T18:19:05.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.670+0000 7f0ceaa7f640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-17T18:19:05.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.670+0000 7f0ceaa7f640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-17T18:19:05.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.670+0000 7f0ceaa7f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:05.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.670+0000 7f0ceaa7f640 1 -- 172.21.15.121:0/3857287722 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f0ce4125020 con 0x7f0ce40a5ca0 2024-09-17T18:19:05.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.671+0000 7f0ce17fa640 1 -- 172.21.15.121:0/3857287722 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0cd8045dd0 con 0x7f0ce40a5ca0 2024-09-17T18:19:05.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.671+0000 7f0ce17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-17T18:19:05.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.671+0000 7f0ce17fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:19:05.673 INFO:teuthology.orchestra.run.smithi121.stdout:47244640260 2024-09-17T18:19:05.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.673+0000 7f0ceaa7f640 10 librados: watch_flush enter 2024-09-17T18:19:05.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.673+0000 7f0ceaa7f640 10 librados: watch_flush exit 2024-09-17T18:19:05.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.673+0000 7f0ceaa7f640 20 client.4296.objecter shutdown clearing up homeless session... 2024-09-17T18:19:05.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.673+0000 7f0ceaa7f640 10 client.4296.objecter successfully canceled tick 2024-09-17T18:19:05.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.673+0000 7f0ceaa7f640 1 -- 172.21.15.121:0/3857287722 >> v2:172.21.15.173:6812/2423786037 conn(0x7f0cc80401a0 msgr2=0x7f0cc8042660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:05.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.673+0000 7f0ceaa7f640 1 --2- 172.21.15.121:0/3857287722 >> v2:172.21.15.173:6812/2423786037 conn(0x7f0cc80401a0 0x7f0cc8042660 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4124d30 tx=0x7f0cd0009190 comp rx=0 tx=0).stop 2024-09-17T18:19:05.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.673+0000 7f0ceaa7f640 10 monclient: shutdown 2024-09-17T18:19:05.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.673+0000 7f0ceaa7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:05.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.673+0000 7f0ceaa7f640 1 -- 172.21.15.121:0/3857287722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ce40a5ca0 msgr2=0x7f0ce41cd450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:05.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.673+0000 7f0ceaa7f640 1 --2- 172.21.15.121:0/3857287722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ce40a5ca0 0x7f0ce41cd450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0cd80090b0 tx=0x7f0cd80046c0 comp rx=0 tx=0).stop 2024-09-17T18:19:05.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.673+0000 7f0ceaa7f640 1 -- 172.21.15.121:0/3857287722 shutdown_connections 2024-09-17T18:19:05.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.673+0000 7f0ceaa7f640 1 --2- 172.21.15.121:0/3857287722 >> v2:172.21.15.173:6812/2423786037 conn(0x7f0cc80401a0 0x7f0cc8042660 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:05.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.673+0000 7f0ceaa7f640 1 --2- 172.21.15.121:0/3857287722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ce413cfc0 0x7f0ce4138260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:05.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.673+0000 7f0ceaa7f640 1 --2- 172.21.15.121:0/3857287722 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ce40a65c0 0x7f0ce4137d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:05.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.673+0000 7f0ceaa7f640 1 --2- 172.21.15.121:0/3857287722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ce40a5ca0 0x7f0ce41cd450 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:05.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.673+0000 7f0ceaa7f640 1 -- 172.21.15.121:0/3857287722 >> 172.21.15.121:0/3857287722 conn(0x7f0ce406dd10 msgr2=0x7f0ce40a90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:05.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.673+0000 7f0ceaa7f640 1 -- 172.21.15.121:0/3857287722 shutdown_connections 2024-09-17T18:19:05.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.673+0000 7f0ceaa7f640 1 -- 172.21.15.121:0/3857287722 wait complete. 2024-09-17T18:19:05.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.673+0000 7f0ceaa7f640 1 librados: shutdown 2024-09-17T18:19:05.685 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.0 2024-09-17T18:19:05.685 DEBUG:teuthology.parallel:result is None 2024-09-17T18:19:05.686 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-17T18:19:05.686 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T18:19:05.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.821+0000 7fc2e5964640 10 monclient: get_monmap_and_config 2024-09-17T18:19:05.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.821+0000 7fc2e5964640 10 monclient: build_initial_monmap 2024-09-17T18:19:05.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.821+0000 7fc2e5964640 10 monclient: monmap: 2024-09-17T18:19:05.823 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:05.823 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:05.823 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:19:05.822002+0000 2024-09-17T18:19:05.823 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:19:05.822002+0000 2024-09-17T18:19:05.823 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:05.823 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:05.823 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:19:05.824 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:19:05.824 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:19:05.824 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:19:05.824 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:19:05.824 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:05.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.822+0000 7fc2e5964640 1 Processor -- start 2024-09-17T18:19:05.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.822+0000 7fc2e5964640 1 -- start start 2024-09-17T18:19:05.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.822+0000 7fc2e5964640 10 monclient: init 2024-09-17T18:19:05.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.822+0000 7fc2e5964640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:05.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.822+0000 7fc2e5964640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-17T18:19:05.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.822+0000 7fc2e5964640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fc2e00a8620 0x7fc2e00a5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:05.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.822+0000 7fc2e5964640 10 monclient(hunting): picked mon.noname-c con 0x7fc2e00a8620 addr v2:172.21.15.173:3300/0 2024-09-17T18:19:05.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.822+0000 7fc2e5964640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc2e00a6350 addr v1:172.21.15.121:6789/0 2024-09-17T18:19:05.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.822+0000 7fc2e5964640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc2e00a6730 0x7fc2e0140320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:05.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.822+0000 7fc2e5964640 10 monclient(hunting): picked mon.noname-a con 0x7fc2e00a6730 addr v2:172.21.15.121:3300/0 2024-09-17T18:19:05.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.822+0000 7fc2e5964640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2e009f0e0 con 0x7fc2e00a6350 2024-09-17T18:19:05.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.822+0000 7fc2e5964640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:05.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.822+0000 7fc2e5964640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fc2e009cf50 con 0x7fc2e00a6730 2024-09-17T18:19:05.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.822+0000 7fc2e5964640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:05.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.822+0000 7fc2e5964640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fc2e0070900 con 0x7fc2e00a8620 2024-09-17T18:19:05.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.822+0000 7fc2e5964640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:05.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.822+0000 7fc2e5964640 10 monclient(hunting): authenticate will time out at 749.538831s 2024-09-17T18:19:05.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.822+0000 7fc2d7fff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fc2e00a6350 0x7fc2e013cbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:49630/0 (socket says 172.21.15.121:49630) 2024-09-17T18:19:05.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.822+0000 7fc2df7fe640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc2e00a6730 0x7fc2e0140320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:05.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.822+0000 7fc2d7fff640 1 -- 172.21.15.121:0/568391812 learned_addr learned my addr 172.21.15.121:0/568391812 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:19:05.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.822+0000 7fc2df7fe640 10 monclient(hunting): get_auth_request con 0x7fc2e00a6730 auth_method 0 2024-09-17T18:19:05.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.822+0000 7fc2df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:05.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.822+0000 7fc2df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:05.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.822+0000 7fc2df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:05.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.823+0000 7fc2deffd640 1 --2- 172.21.15.121:0/568391812 >> v2:172.21.15.173:3300/0 conn(0x7fc2e00a8620 0x7fc2e00a5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:05.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.823+0000 7fc2deffd640 10 monclient(hunting): get_auth_request con 0x7fc2e00a8620 auth_method 0 2024-09-17T18:19:05.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.823+0000 7fc2deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:05.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.823+0000 7fc2deffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:05.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.823+0000 7fc2deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:05.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.823+0000 7fc2df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:05.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.823+0000 7fc2df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:05.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.823+0000 7fc2df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:05.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.823+0000 7fc2deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:05.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.823+0000 7fc2deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:05.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.823+0000 7fc2deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:05.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.823+0000 7fc2df7fe640 10 monclient(hunting): handle_auth_done global_id 4311 payload 274 2024-09-17T18:19:05.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.823+0000 7fc2df7fe640 1 -- 172.21.15.121:0/568391812 >> v2:172.21.15.173:3300/0 conn(0x7fc2e00a8620 msgr2=0x7fc2e00a5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:05.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.823+0000 7fc2df7fe640 1 --2- 172.21.15.121:0/568391812 >> v2:172.21.15.173:3300/0 conn(0x7fc2e00a8620 0x7fc2e00a5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:05.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.823+0000 7fc2df7fe640 1 -- 172.21.15.121:0/568391812 >> v1:172.21.15.121:6789/0 conn(0x7fc2e00a6350 legacy=0x7fc2e013cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:05.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.823+0000 7fc2df7fe640 10 monclient: _finish_hunting 0 2024-09-17T18:19:05.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.823+0000 7fc2df7fe640 1 monclient: found mon.noname-a 2024-09-17T18:19:05.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.823+0000 7fc2df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:05.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.823+0000 7fc2df7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:19:05.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.823+0000 7fc2df7fe640 1 -- 172.21.15.121:0/568391812 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2e009cc70 con 0x7fc2e00a6730 2024-09-17T18:19:05.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.825+0000 7fc2df7fe640 1 -- 172.21.15.121:0/568391812 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2e00a6730 msgr2=0x7fc2e0140320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:05.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.826+0000 7fc2df7fe640 1 --2- 172.21.15.121:0/568391812 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2e00a6730 0x7fc2e0140320 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc2e0077c30 tx=0x7fc2d802a4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60f4a658b7d90ac7 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:05.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.826+0000 7fc2de7fc640 1 -- 172.21.15.121:0/568391812 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d8007950 con 0x7fc2e00a6730 2024-09-17T18:19:05.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.826+0000 7fc2de7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:05.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.826+0000 7fc2de7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:19:05.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.826+0000 7fc2de7fc640 10 monclient: dump: 2024-09-17T18:19:05.838 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:05.838 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:05.838 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:05.838 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:05.838 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:05.838 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:05.838 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:05.838 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:05.839 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:05.839 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:05.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.826+0000 7fc2de7fc640 10 monclient: _finish_auth 0 2024-09-17T18:19:05.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.826+0000 7fc2de7fc640 10 monclient: _check_auth_tickets 2024-09-17T18:19:05.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.826+0000 7fc2de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:05.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.826+0000 7fc2e5964640 5 monclient: authenticate success, global_id 4311 2024-09-17T18:19:05.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.826+0000 7fc2e5964640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:19:05.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.826+0000 7fc2de7fc640 1 -- 172.21.15.121:0/568391812 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2d8007af0 con 0x7fc2e00a6730 2024-09-17T18:19:05.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.826+0000 7fc2de7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:05.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.826+0000 7fc2e5964640 10 monclient: get_monmap_and_config success 2024-09-17T18:19:05.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.826+0000 7fc2e5964640 4 set_mon_vals no callback set 2024-09-17T18:19:05.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.826+0000 7fc2de7fc640 1 -- 172.21.15.121:0/568391812 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d8031400 con 0x7fc2e00a6730 2024-09-17T18:19:05.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.826+0000 7fc2de7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:05.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.826+0000 7fc2de7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:19:05.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.826+0000 7fc2de7fc640 10 monclient: dump: 2024-09-17T18:19:05.840 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:05.840 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:05.840 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:05.840 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:05.840 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:05.840 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:05.840 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:05.840 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:05.840 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:05.840 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:05.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.826+0000 7fc2e5964640 10 monclient: shutdown 2024-09-17T18:19:05.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.826+0000 7fc2e5964640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:05.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.826+0000 7fc2e5964640 1 -- 172.21.15.121:0/568391812 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2e00a6730 msgr2=0x7fc2e0140320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:05.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.826+0000 7fc2e5964640 1 --2- 172.21.15.121:0/568391812 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2e00a6730 0x7fc2e0140320 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc2e0077c30 tx=0x7fc2d802a4b0 comp rx=0 tx=0).stop 2024-09-17T18:19:05.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.826+0000 7fc2e5964640 1 -- 172.21.15.121:0/568391812 shutdown_connections 2024-09-17T18:19:05.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.826+0000 7fc2e5964640 1 --2- 172.21.15.121:0/568391812 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2e00a6730 0x7fc2e0140320 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:05.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.826+0000 7fc2e5964640 1 --2- 172.21.15.121:0/568391812 >> v2:172.21.15.173:3300/0 conn(0x7fc2e00a8620 0x7fc2e00a5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:05.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.826+0000 7fc2e5964640 1 -- 172.21.15.121:0/568391812 >> 172.21.15.121:0/568391812 conn(0x7fc2e00abcb0 msgr2=0x7fc2e00a7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:05.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.826+0000 7fc2e5964640 1 -- 172.21.15.121:0/568391812 shutdown_connections 2024-09-17T18:19:05.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.826+0000 7fc2e5964640 1 -- 172.21.15.121:0/568391812 wait complete. 2024-09-17T18:19:05.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.827+0000 7fc2e5964640 10 monclient: build_initial_monmap 2024-09-17T18:19:05.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.827+0000 7fc2e5964640 10 monclient: monmap: 2024-09-17T18:19:05.841 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:05.841 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:19:05.841 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:19:05.841 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:19:05.841 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:05.841 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:05.842 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:19:05.842 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:19:05.842 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:19:05.842 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:05.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.827+0000 7fc2e5964640 1 librados: starting msgr at 2024-09-17T18:19:05.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.827+0000 7fc2e5964640 1 librados: starting objecter 2024-09-17T18:19:05.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.827+0000 7fc2e5964640 1 Processor -- start 2024-09-17T18:19:05.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.827+0000 7fc2e5964640 1 -- start start 2024-09-17T18:19:05.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.827+0000 7fc2e5964640 1 librados: setting wanted keys 2024-09-17T18:19:05.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.827+0000 7fc2e5964640 1 librados: calling monclient init 2024-09-17T18:19:05.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.827+0000 7fc2e5964640 10 monclient: init 2024-09-17T18:19:05.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2e5964640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:05.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2e5964640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T18:19:05.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2e5964640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2e00a6350 0x7fc2e01ad5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:05.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2e5964640 10 monclient(hunting): picked mon.noname-b con 0x7fc2e00a6350 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:05.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2e5964640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2e00a6730 0x7fc2e01c7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:05.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2e5964640 10 monclient(hunting): picked mon.noname-a con 0x7fc2e00a6730 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:05.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2e5964640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc2e00a8620 0x7fc2e01c9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:05.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2e5964640 10 monclient(hunting): picked mon.noname-c con 0x7fc2e00a8620 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:05.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2e5964640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:05.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2e5964640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc2e00a6b80 con 0x7fc2e00a6730 2024-09-17T18:19:05.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2e5964640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:05.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2e5964640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc2e00706c0 con 0x7fc2e00a6350 2024-09-17T18:19:05.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2e5964640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:05.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2e5964640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc2e00a7490 con 0x7fc2e00a8620 2024-09-17T18:19:05.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2e5964640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:05.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2e5964640 10 monclient(hunting): authenticate will time out at 749.544370s 2024-09-17T18:19:05.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2d7fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2e00a6730 0x7fc2e01c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:05.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2df7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc2e00a8620 0x7fc2e01c9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:05.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2d7fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2e00a6730 0x7fc2e01c7380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41132/0 (socket says 172.21.15.121:41132) 2024-09-17T18:19:05.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2d7fff640 1 -- 172.21.15.121:0/1563514368 learned_addr learned my addr 172.21.15.121:0/1563514368 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:05.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2d7fff640 10 monclient(hunting): get_auth_request con 0x7fc2e00a6730 auth_method 0 2024-09-17T18:19:05.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:05.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:05.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:05.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2df7fe640 10 monclient(hunting): get_auth_request con 0x7fc2e00a8620 auth_method 0 2024-09-17T18:19:05.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:05.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:05.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:05.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2deffd640 1 --2- 172.21.15.121:0/1563514368 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2e00a6350 0x7fc2e01ad5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:05.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2deffd640 10 monclient(hunting): get_auth_request con 0x7fc2e00a6350 auth_method 0 2024-09-17T18:19:05.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:05.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2deffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:05.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:05.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:05.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:05.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:05.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:05.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:05.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:05.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:05.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:05.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:05.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2d7fff640 10 monclient(hunting): handle_auth_done global_id 4314 payload 824 2024-09-17T18:19:05.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2d7fff640 1 -- 172.21.15.121:0/1563514368 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc2e00a8620 msgr2=0x7fc2e01c9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:05.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2d7fff640 1 --2- 172.21.15.121:0/1563514368 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc2e00a8620 0x7fc2e01c9f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:05.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2d7fff640 1 -- 172.21.15.121:0/1563514368 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2e00a6350 msgr2=0x7fc2e01ad5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:05.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2d7fff640 1 --2- 172.21.15.121:0/1563514368 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2e00a6350 0x7fc2e01ad5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:05.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2d7fff640 10 monclient: _finish_hunting 0 2024-09-17T18:19:05.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2d7fff640 1 monclient: found mon.noname-a 2024-09-17T18:19:05.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:05.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2d7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:19:05.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2d7fff640 1 -- 172.21.15.121:0/1563514368 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2e01adf70 con 0x7fc2e00a6730 2024-09-17T18:19:05.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2df7fe640 1 --2- 172.21.15.121:0/1563514368 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc2e00a8620 0x7fc2e01c9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:19:05.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.828+0000 7fc2deffd640 1 --2- 172.21.15.121:0/1563514368 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2e00a6350 0x7fc2e01ad5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:19:05.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.831+0000 7fc2d7fff640 1 -- 172.21.15.121:0/1563514368 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2e00a6730 msgr2=0x7fc2e01c7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:05.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.831+0000 7fc2d7fff640 1 --2- 172.21.15.121:0/1563514368 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2e00a6730 0x7fc2e01c7380 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc2c800b460 tx=0x7fc2c800b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:05.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.831+0000 7fc2dcff9640 10 client.?.objecter ms_handle_connect 0x7fc2e00a6730 2024-09-17T18:19:05.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.831+0000 7fc2dcff9640 10 client.?.objecter resend_mon_ops 2024-09-17T18:19:05.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.831+0000 7fc2dcff9640 1 -- 172.21.15.121:0/1563514368 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c8015020 con 0x7fc2e00a6730 2024-09-17T18:19:05.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.831+0000 7fc2dcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:05.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.831+0000 7fc2dcff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:19:05.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.831+0000 7fc2dcff9640 10 monclient: dump: 2024-09-17T18:19:05.847 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:05.847 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:05.847 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:05.847 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:05.847 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:05.847 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:05.847 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:05.847 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:05.847 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:05.847 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:05.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.831+0000 7fc2dcff9640 10 monclient: _finish_auth 0 2024-09-17T18:19:05.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.831+0000 7fc2dcff9640 10 monclient: _check_auth_tickets 2024-09-17T18:19:05.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.831+0000 7fc2dcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:05.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.831+0000 7fc2dcff9640 1 -- 172.21.15.121:0/1563514368 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2c8011ac0 con 0x7fc2e00a6730 2024-09-17T18:19:05.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.831+0000 7fc2e5964640 5 monclient: authenticate success, global_id 4314 2024-09-17T18:19:05.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.831+0000 7fc2e5964640 10 monclient: _renew_subs 2024-09-17T18:19:05.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.831+0000 7fc2e5964640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:05.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.831+0000 7fc2e5964640 1 -- 172.21.15.121:0/1563514368 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2e01cdd40 con 0x7fc2e00a6730 2024-09-17T18:19:05.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.831+0000 7fc2dcff9640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:05.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.831+0000 7fc2dcff9640 1 -- 172.21.15.121:0/1563514368 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c80103e0 con 0x7fc2e00a6730 2024-09-17T18:19:05.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.831+0000 7fc2dcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:05.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.831+0000 7fc2dcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:19:05.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.831+0000 7fc2dcff9640 10 monclient: dump: 2024-09-17T18:19:05.849 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:05.849 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:05.849 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:05.849 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:05.849 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:05.849 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:05.849 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:05.849 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:05.849 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:05.849 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:05.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.831+0000 7fc2e5964640 10 client.4314.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:05.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.831+0000 7fc2e5964640 10 monclient: _renew_subs 2024-09-17T18:19:05.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.831+0000 7fc2ddffb640 4 set_mon_vals no callback set 2024-09-17T18:19:05.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.831+0000 7fc2e5964640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:05.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.831+0000 7fc2e5964640 1 -- 172.21.15.121:0/1563514368 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2e01aede0 con 0x7fc2e00a6730 2024-09-17T18:19:05.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.831+0000 7fc2e5964640 1 librados: init done 2024-09-17T18:19:05.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.832+0000 7fc2dcff9640 1 -- 172.21.15.121:0/1563514368 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fc2c8018020 con 0x7fc2e00a6730 2024-09-17T18:19:05.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.832+0000 7fc2e5964640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:05.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.832+0000 7fc2e5964640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:05.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.832+0000 7fc2e5964640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:05.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.832+0000 7fc2e5964640 1 -- 172.21.15.121:0/1563514368 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2e00a5e10 con 0x7fc2e00a6730 2024-09-17T18:19:05.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.832+0000 7fc2dcff9640 1 --2- 172.21.15.121:0/1563514368 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc2c40401c0 0x7fc2c4042680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:05.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.832+0000 7fc2dcff9640 1 -- 172.21.15.121:0/1563514368 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 3562+0+0 (secure 0 0 0) 0x7fc2c8052e90 con 0x7fc2e00a6730 2024-09-17T18:19:05.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.832+0000 7fc2dcff9640 10 client.4314.objecter ms_dispatch 0x7fc2e0000b90 osd_map(16..16 src has 1..16) 2024-09-17T18:19:05.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.832+0000 7fc2dcff9640 3 client.4314.objecter handle_osd_map got epochs [16,16] > 0 2024-09-17T18:19:05.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.832+0000 7fc2dcff9640 3 client.4314.objecter handle_osd_map decoding full epoch 16 2024-09-17T18:19:05.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.832+0000 7fc2dcff9640 20 client.4314.objecter dump_active .. 0 homeless 2024-09-17T18:19:05.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.832+0000 7fc2deffd640 1 --2- 172.21.15.121:0/1563514368 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc2c40401c0 0x7fc2c4042680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:05.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.832+0000 7fc2deffd640 10 monclient: get_auth_request con 0x7fc2c40401c0 auth_method 0 2024-09-17T18:19:05.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.835+0000 7fc2dcff9640 10 client.4314.objecter ms_handle_connect 0x7fc2c40401c0 2024-09-17T18:19:05.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.835+0000 7fc2deffd640 1 --2- 172.21.15.121:0/1563514368 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc2c40401c0 0x7fc2c4042680 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc2d0009eb0 tx=0x7fc2d0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:05.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.836+0000 7fc2dcff9640 1 -- 172.21.15.121:0/1563514368 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc2c801f030 con 0x7fc2e00a6730 2024-09-17T18:19:05.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.836+0000 7fc2dcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:05.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:05.836+0000 7fc2dcff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:19:06.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.007+0000 7fc2e5964640 1 -- 172.21.15.121:0/1563514368 --> v2:172.21.15.173:6812/2423786037 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc2e01ae5a0 con 0x7fc2c40401c0 2024-09-17T18:19:06.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.009+0000 7fc2dcff9640 1 -- 172.21.15.121:0/1563514368 <== mgr.4105 v2:172.21.15.173:6812/2423786037 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34209 (secure 0 0 0) 0x7fc2e01ae5a0 con 0x7fc2c40401c0 2024-09-17T18:19:06.011 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:19:06.011 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-09-17T18:19:06.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.011+0000 7fc2e5964640 10 librados: watch_flush enter 2024-09-17T18:19:06.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.011+0000 7fc2e5964640 10 librados: watch_flush exit 2024-09-17T18:19:06.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.011+0000 7fc2e5964640 20 client.4314.objecter shutdown clearing up homeless session... 2024-09-17T18:19:06.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.011+0000 7fc2e5964640 10 client.4314.objecter successfully canceled tick 2024-09-17T18:19:06.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.011+0000 7fc2e5964640 1 -- 172.21.15.121:0/1563514368 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc2c40401c0 msgr2=0x7fc2c4042680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:06.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.011+0000 7fc2e5964640 1 --2- 172.21.15.121:0/1563514368 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc2c40401c0 0x7fc2c4042680 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc2d0009eb0 tx=0x7fc2d0009190 comp rx=0 tx=0).stop 2024-09-17T18:19:06.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.011+0000 7fc2e5964640 10 monclient: shutdown 2024-09-17T18:19:06.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.011+0000 7fc2e5964640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:06.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.011+0000 7fc2e5964640 1 -- 172.21.15.121:0/1563514368 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2e00a6730 msgr2=0x7fc2e01c7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:06.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.011+0000 7fc2e5964640 1 --2- 172.21.15.121:0/1563514368 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2e00a6730 0x7fc2e01c7380 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc2c800b460 tx=0x7fc2c800b920 comp rx=0 tx=0).stop 2024-09-17T18:19:06.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.011+0000 7fc2e5964640 1 -- 172.21.15.121:0/1563514368 shutdown_connections 2024-09-17T18:19:06.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.011+0000 7fc2e5964640 1 --2- 172.21.15.121:0/1563514368 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc2c40401c0 0x7fc2c4042680 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:06.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.011+0000 7fc2e5964640 1 --2- 172.21.15.121:0/1563514368 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc2e00a8620 0x7fc2e01c9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:06.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.011+0000 7fc2e5964640 1 --2- 172.21.15.121:0/1563514368 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2e00a6730 0x7fc2e01c7380 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:06.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.011+0000 7fc2e5964640 1 --2- 172.21.15.121:0/1563514368 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2e00a6350 0x7fc2e01ad5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:06.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.011+0000 7fc2e5964640 1 -- 172.21.15.121:0/1563514368 >> 172.21.15.121:0/1563514368 conn(0x7fc2e00abcb0 msgr2=0x7fc2e00a36e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:06.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.011+0000 7fc2e5964640 1 -- 172.21.15.121:0/1563514368 shutdown_connections 2024-09-17T18:19:06.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.011+0000 7fc2e5964640 1 -- 172.21.15.121:0/1563514368 wait complete. 2024-09-17T18:19:06.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.011+0000 7fc2e5964640 1 librados: shutdown 2024-09-17T18:19:06.023 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2024-09-17T18:19:05.496474+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163136,"kb_used_data":944,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067904,"statfs":{"total":579820584960,"available":579653533696,"internally_reserved":0,"allocated":966656,"data_stored":1313332,"data_compressed":59128,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":165927585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.970505"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T18:19:01.527478+0000","last_change":"2024-09-17T18:19:01.527658+0000","last_active":"2024-09-17T18:19:01.527478+0000","last_peered":"2024-09-17T18:19:01.527478+0000","last_clean":"2024-09-17T18:19:01.527478+0000","last_became_active":"2024-09-17T18:19:00.056752+0000","last_became_peered":"2024-09-17T18:19:00.056752+0000","last_unstale":"2024-09-17T18:19:01.527478+0000","last_undegraded":"2024-09-17T18:19:01.527478+0000","last_fullsized":"2024-09-17T18:19:01.527478+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:57:47.271964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049856399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T18:19:01.527858+0000","last_change":"2024-09-17T18:19:01.527858+0000","last_active":"2024-09-17T18:19:01.527858+0000","last_peered":"2024-09-17T18:19:01.527858+0000","last_clean":"2024-09-17T18:19:01.527858+0000","last_became_active":"2024-09-17T18:18:59.525295+0000","last_became_peered":"2024-09-17T18:18:59.525295+0000","last_unstale":"2024-09-17T18:19:01.527858+0000","last_undegraded":"2024-09-17T18:19:01.527858+0000","last_fullsized":"2024-09-17T18:19:01.527858+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:41:21.181263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062121300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T18:19:01.527332+0000","last_change":"2024-09-17T18:19:01.527563+0000","last_active":"2024-09-17T18:19:01.527332+0000","last_peered":"2024-09-17T18:19:01.527332+0000","last_clean":"2024-09-17T18:19:01.527332+0000","last_became_active":"2024-09-17T18:19:00.244851+0000","last_became_peered":"2024-09-17T18:19:00.244851+0000","last_unstale":"2024-09-17T18:19:01.527332+0000","last_undegraded":"2024-09-17T18:19:01.527332+0000","last_fullsized":"2024-09-17T18:19:01.527332+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:19:57.036107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041282100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T18:19:01.527784+0000","last_change":"2024-09-17T18:19:01.527784+0000","last_active":"2024-09-17T18:19:01.527784+0000","last_peered":"2024-09-17T18:19:01.527784+0000","last_clean":"2024-09-17T18:19:01.527784+0000","last_became_active":"2024-09-17T18:19:00.244613+0000","last_became_peered":"2024-09-17T18:19:00.244613+0000","last_unstale":"2024-09-17T18:19:01.527784+0000","last_undegraded":"2024-09-17T18:19:01.527784+0000","last_fullsized":"2024-09-17T18:19:01.527784+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:31:24.599660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063646500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":22,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T18:19:01.532623+0000","last_change":"2024-09-17T18:19:01.532623+0000","last_active":"2024-09-17T18:19:01.532623+0000","last_peered":"2024-09-17T18:19:01.532623+0000","last_clean":"2024-09-17T18:19:01.532623+0000","last_became_active":"2024-09-17T18:18:59.526791+0000","last_became_peered":"2024-09-17T18:18:59.526791+0000","last_unstale":"2024-09-17T18:19:01.532623+0000","last_undegraded":"2024-09-17T18:19:01.532623+0000","last_fullsized":"2024-09-17T18:19:01.532623+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T21:46:24.217486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010088289999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T18:19:01.527597+0000","last_change":"2024-09-17T18:19:01.527848+0000","last_active":"2024-09-17T18:19:01.527597+0000","last_peered":"2024-09-17T18:19:01.527597+0000","last_clean":"2024-09-17T18:19:01.527597+0000","last_became_active":"2024-09-17T18:18:59.529390+0000","last_became_peered":"2024-09-17T18:18:59.529390+0000","last_unstale":"2024-09-17T18:19:01.527597+0000","last_undegraded":"2024-09-17T18:19:01.527597+0000","last_fullsized":"2024-09-17T18:19:01.527597+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:40:45.376247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065986199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T18:19:01.527258+0000","last_change":"2024-09-17T18:19:01.527425+0000","last_active":"2024-09-17T18:19:01.527258+0000","last_peered":"2024-09-17T18:19:01.527258+0000","last_clean":"2024-09-17T18:19:01.527258+0000","last_became_active":"2024-09-17T18:18:59.526855+0000","last_became_peered":"2024-09-17T18:18:59.526855+0000","last_unstale":"2024-09-17T18:19:01.527258+0000","last_undegraded":"2024-09-17T18:19:01.527258+0000","last_fullsized":"2024-09-17T18:19:01.527258+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:05:38.754621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032532500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T18:19:01.531845+0000","last_change":"2024-09-17T18:19:01.532053+0000","last_active":"2024-09-17T18:19:01.531845+0000","last_peered":"2024-09-17T18:19:01.531845+0000","last_clean":"2024-09-17T18:19:01.531845+0000","last_became_active":"2024-09-17T18:18:59.526797+0000","last_became_peered":"2024-09-17T18:18:59.526797+0000","last_unstale":"2024-09-17T18:19:01.531845+0000","last_undegraded":"2024-09-17T18:19:01.531845+0000","last_fullsized":"2024-09-17T18:19:01.531845+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:19:28.196155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000425759,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"15'72","reported_seq":105,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T18:19:01.527033+0000","last_change":"2024-09-17T18:19:00.245854+0000","last_active":"2024-09-17T18:19:01.527033+0000","last_peered":"2024-09-17T18:19:01.527033+0000","last_clean":"2024-09-17T18:19:01.527033+0000","last_became_active":"2024-09-17T18:19:00.244903+0000","last_became_peered":"2024-09-17T18:19:00.244903+0000","last_unstale":"2024-09-17T18:19:01.527033+0000","last_undegraded":"2024-09-17T18:19:01.527033+0000","last_fullsized":"2024-09-17T18:19:01.527033+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:55:38.336150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59128,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607557,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":139264,"data_stored":42573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607556,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":143360,"data_stored":42592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":143360,"data_stored":42592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":200704,"data_stored":571501,"data_compressed":29564,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":139264,"data_stored":42573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":200704,"data_stored":571501,"data_compressed":29564,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29564,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29564,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T18:19:06.024 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T18:19:06.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.119+0000 7fea14924640 10 monclient: get_monmap_and_config 2024-09-17T18:19:06.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.119+0000 7fea14924640 10 monclient: build_initial_monmap 2024-09-17T18:19:06.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.119+0000 7fea14924640 10 monclient: monmap: 2024-09-17T18:19:06.138 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:06.138 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:06.138 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:19:06.120239+0000 2024-09-17T18:19:06.138 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:19:06.120239+0000 2024-09-17T18:19:06.138 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:06.138 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:06.138 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:19:06.138 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:19:06.138 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:19:06.138 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:19:06.139 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:19:06.139 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:06.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.120+0000 7fea14924640 1 Processor -- start 2024-09-17T18:19:06.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.120+0000 7fea14924640 1 -- start start 2024-09-17T18:19:06.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.120+0000 7fea14924640 10 monclient: init 2024-09-17T18:19:06.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.120+0000 7fea14924640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:06.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.120+0000 7fea14924640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-17T18:19:06.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.120+0000 7fea14924640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fea1009ccd0 0x7fea1009d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:06.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.120+0000 7fea14924640 10 monclient(hunting): picked mon.noname-c con 0x7fea1009ccd0 addr v2:172.21.15.173:3300/0 2024-09-17T18:19:06.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.120+0000 7fea14924640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fea1009d5f0 addr v1:172.21.15.173:6789/0 2024-09-17T18:19:06.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.120+0000 7fea14924640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fea1009da00 addr v1:172.21.15.121:6789/0 2024-09-17T18:19:06.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.120+0000 7fea14924640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea100a1c50 con 0x7fea1009da00 2024-09-17T18:19:06.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.120+0000 7fea14924640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea100a21d0 con 0x7fea1009d5f0 2024-09-17T18:19:06.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.120+0000 7fea14924640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:06.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.120+0000 7fea14924640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fea1009de90 con 0x7fea1009ccd0 2024-09-17T18:19:06.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.120+0000 7fea14924640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:06.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.120+0000 7fea14924640 10 monclient(hunting): authenticate will time out at 749.836886s 2024-09-17T18:19:06.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.121+0000 7fea0ed9d640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fea1009da00 0x7fea1014ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:49646/0 (socket says 172.21.15.121:49646) 2024-09-17T18:19:06.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.121+0000 7fea0ed9d640 1 -- 172.21.15.121:0/824778251 learned_addr learned my addr 172.21.15.121:0/824778251 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:19:06.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.121+0000 7fea0e59c640 1 --2- 172.21.15.121:0/824778251 >> v2:172.21.15.173:3300/0 conn(0x7fea1009ccd0 0x7fea1009d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:06.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.121+0000 7fea0e59c640 10 monclient(hunting): get_auth_request con 0x7fea1009ccd0 auth_method 0 2024-09-17T18:19:06.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.121+0000 7fea0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:06.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.121+0000 7fea0e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:06.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.123+0000 7fea0ed9d640 1 -- 172.21.15.121:0/824778251 >> v1:172.21.15.121:6789/0 conn(0x7fea1009da00 legacy=0x7fea1014ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:06.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.121+0000 7fea0e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:06.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0dd9b640 1 -- 172.21.15.121:0/824778251 >> v1:172.21.15.173:6789/0 conn(0x7fea1009d5f0 legacy=0x7fea10149730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:06.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0d59a640 1 -- 172.21.15.121:0/824778251 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2844359420 0 0) 0x7fea100a1c50 con 0x7fea1009da00 2024-09-17T18:19:06.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0d59a640 10 monclient(hunting): _negotiate 2024-09-17T18:19:06.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0d59a640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:06.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:06.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0d59a640 10 monclient(hunting): my global_id is 4317 2024-09-17T18:19:06.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0d59a640 1 -- 172.21.15.121:0/824778251 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe9f8003520 con 0x7fea1009da00 2024-09-17T18:19:06.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:06.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:06.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:06.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0d59a640 1 -- 172.21.15.121:0/824778251 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 876477663 0 0) 0x7fe9f8003520 con 0x7fea1009da00 2024-09-17T18:19:06.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0d59a640 1 -- 172.21.15.121:0/824778251 >> v2:172.21.15.173:3300/0 conn(0x7fea1009ccd0 msgr2=0x7fea1009d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:06.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0d59a640 1 --2- 172.21.15.121:0/824778251 >> v2:172.21.15.173:3300/0 conn(0x7fea1009ccd0 0x7fea1009d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:06.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0d59a640 1 -- 172.21.15.121:0/824778251 >> v1:172.21.15.173:6789/0 conn(0x7fea1009d5f0 legacy=0x7fea10149730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:06.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0d59a640 10 monclient: _finish_hunting 0 2024-09-17T18:19:06.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0d59a640 1 monclient: found mon.noname-a-legacy 2024-09-17T18:19:06.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:06.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0d59a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.121:6789/0 2024-09-17T18:19:06.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0d59a640 1 -- 172.21.15.121:0/824778251 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea100a1e50 con 0x7fea1009da00 2024-09-17T18:19:06.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0d59a640 10 monclient: _finish_auth 0 2024-09-17T18:19:06.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0d59a640 10 monclient: _check_auth_tickets 2024-09-17T18:19:06.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:06.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0d59a640 1 -- 172.21.15.121:0/824778251 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4002019445 0 0) 0x7fea08001d20 con 0x7fea1009da00 2024-09-17T18:19:06.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea14924640 5 monclient: authenticate success, global_id 4317 2024-09-17T18:19:06.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea14924640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:19:06.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:06.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0d59a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T18:19:06.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0d59a640 10 monclient: dump: 2024-09-17T18:19:06.142 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:06.142 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:06.142 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:06.142 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:06.143 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:06.143 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:06.143 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:06.143 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:06.143 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:06.143 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:06.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0d59a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] but i'm connected to v1:172.21.15.121:6789/0, reconnecting 2024-09-17T18:19:06.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0d59a640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:06.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0d59a640 1 -- 172.21.15.121:0/824778251 >> v1:172.21.15.121:6789/0 conn(0x7fea1009da00 legacy=0x7fea1014ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:06.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0d59a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:19:06.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0d59a640 1 --2- 172.21.15.121:0/824778251 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9f8003ad0 0x7fe9f8003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:06.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0d59a640 10 monclient(hunting): picked mon.a con 0x7fe9f8003ad0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:06.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0d59a640 1 --2- 172.21.15.121:0/824778251 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9f8004800 0x7fe9f8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:06.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0d59a640 10 monclient(hunting): picked mon.b con 0x7fe9f8004800 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:06.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0d59a640 1 --2- 172.21.15.121:0/824778251 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe9f8005290 0x7fe9f800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:06.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0d59a640 10 monclient(hunting): picked mon.c con 0x7fe9f8005290 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:06.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0d59a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:06.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0d59a640 1 -- 172.21.15.121:0/824778251 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fea1009de90 con 0x7fe9f8003ad0 2024-09-17T18:19:06.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0d59a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:06.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0d59a640 1 -- 172.21.15.121:0/824778251 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f800c1e0 con 0x7fe9f8004800 2024-09-17T18:19:06.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0d59a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:06.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0d59a640 1 -- 172.21.15.121:0/824778251 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe9f800c3e0 con 0x7fe9f8005290 2024-09-17T18:19:06.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0d59a640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:06.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0d59a640 10 monclient(hunting): _finish_auth 0 2024-09-17T18:19:06.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea14924640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-17T18:19:06.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0e59c640 1 --2- 172.21.15.121:0/824778251 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9f8003ad0 0x7fe9f8003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:06.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0e59c640 10 monclient(hunting): get_auth_request con 0x7fe9f8003ad0 auth_method 0 2024-09-17T18:19:06.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:06.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:06.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:19:06.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0ed9d640 1 --2- 172.21.15.121:0/824778251 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe9f8005290 0x7fe9f800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:06.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0ed9d640 10 monclient(hunting): get_auth_request con 0x7fe9f8005290 auth_method 0 2024-09-17T18:19:06.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:06.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:06.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:19:06.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0dd9b640 1 --2- 172.21.15.121:0/824778251 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9f8004800 0x7fe9f8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:06.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:06.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:06.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0e59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:19:06.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0dd9b640 10 monclient(hunting): get_auth_request con 0x7fe9f8004800 auth_method 0 2024-09-17T18:19:06.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:06.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:06.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0dd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:19:06.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:06.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:06.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.133+0000 7fea0ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:19:06.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.134+0000 7fea0dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:06.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.134+0000 7fea0dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:06.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.134+0000 7fea0dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:19:06.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.134+0000 7fea0e59c640 10 monclient(hunting): handle_auth_done global_id 4317 payload 293 2024-09-17T18:19:06.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.134+0000 7fea0e59c640 1 -- 172.21.15.121:0/824778251 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe9f8005290 msgr2=0x7fe9f800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:06.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.134+0000 7fea0e59c640 1 --2- 172.21.15.121:0/824778251 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe9f8005290 0x7fe9f800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:06.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.134+0000 7fea0e59c640 1 -- 172.21.15.121:0/824778251 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9f8004800 msgr2=0x7fe9f8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:06.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.134+0000 7fea0e59c640 1 --2- 172.21.15.121:0/824778251 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9f8004800 0x7fe9f8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:06.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.134+0000 7fea0e59c640 10 monclient: _finish_hunting 0 2024-09-17T18:19:06.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.134+0000 7fea0e59c640 1 monclient: found mon.a 2024-09-17T18:19:06.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.134+0000 7fea0e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:06.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.134+0000 7fea0e59c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:06.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.134+0000 7fea0e59c640 1 -- 172.21.15.121:0/824778251 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9f8003520 con 0x7fe9f8003ad0 2024-09-17T18:19:06.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.134+0000 7fea0e59c640 10 monclient: _finish_auth 0 2024-09-17T18:19:06.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.134+0000 7fea0e59c640 10 monclient: _check_auth_tickets 2024-09-17T18:19:06.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.134+0000 7fea0e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:06.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.134+0000 7fea0ed9d640 1 -- 172.21.15.121:0/824778251 reap_dead start 2024-09-17T18:19:06.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.136+0000 7fea0e59c640 1 -- 172.21.15.121:0/824778251 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9f8003ad0 msgr2=0x7fe9f8003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:06.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.136+0000 7fea0e59c640 1 --2- 172.21.15.121:0/824778251 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9f8003ad0 0x7fe9f8003eb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fea00009410 tx=0x7fe9f8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4556d375f94fcfd5 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:06.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.137+0000 7fea0d59a640 1 -- 172.21.15.121:0/824778251 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea00037070 con 0x7fe9f8003ad0 2024-09-17T18:19:06.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.137+0000 7fea0d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:06.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.137+0000 7fea0d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:19:06.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.137+0000 7fea0d59a640 10 monclient: dump: 2024-09-17T18:19:06.153 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:06.153 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:06.153 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:06.153 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:06.153 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:06.153 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:06.153 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:06.153 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:06.153 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:06.153 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:06.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.137+0000 7fea0d59a640 1 -- 172.21.15.121:0/824778251 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea00032640 con 0x7fe9f8003ad0 2024-09-17T18:19:06.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.137+0000 7fea14924640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:19:06.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.137+0000 7fea0d59a640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:06.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.137+0000 7fea0d59a640 1 -- 172.21.15.121:0/824778251 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea00009060 con 0x7fe9f8003ad0 2024-09-17T18:19:06.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.137+0000 7fea14924640 10 monclient: get_monmap_and_config success 2024-09-17T18:19:06.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.137+0000 7fea14924640 4 set_mon_vals no callback set 2024-09-17T18:19:06.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.137+0000 7fea0d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:06.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.137+0000 7fea0d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:19:06.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.137+0000 7fea0d59a640 10 monclient: dump: 2024-09-17T18:19:06.154 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:06.154 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:06.154 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:06.154 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:06.154 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:06.154 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:06.155 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:06.155 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:06.155 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:06.155 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:06.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.137+0000 7fea14924640 10 monclient: shutdown 2024-09-17T18:19:06.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.137+0000 7fea14924640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:06.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.137+0000 7fea14924640 1 -- 172.21.15.121:0/824778251 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9f8003ad0 msgr2=0x7fe9f8003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:06.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.137+0000 7fea14924640 1 --2- 172.21.15.121:0/824778251 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9f8003ad0 0x7fe9f8003eb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fea00009410 tx=0x7fe9f8005830 comp rx=0 tx=0).stop 2024-09-17T18:19:06.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.137+0000 7fea14924640 1 -- 172.21.15.121:0/824778251 shutdown_connections 2024-09-17T18:19:06.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.137+0000 7fea14924640 1 --2- 172.21.15.121:0/824778251 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9f8003ad0 0x7fe9f8003eb0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:06.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.137+0000 7fea14924640 1 -- 172.21.15.121:0/824778251 >> 172.21.15.121:0/824778251 conn(0x7fea100a7230 msgr2=0x7fea100a7640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:06.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.137+0000 7fea14924640 1 -- 172.21.15.121:0/824778251 shutdown_connections 2024-09-17T18:19:06.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.137+0000 7fea14924640 1 -- 172.21.15.121:0/824778251 wait complete. 2024-09-17T18:19:06.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.138+0000 7fea14924640 10 monclient: build_initial_monmap 2024-09-17T18:19:06.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.138+0000 7fea14924640 10 monclient: monmap: 2024-09-17T18:19:06.156 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:06.156 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:19:06.156 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:19:06.156 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:19:06.156 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:06.156 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:06.156 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:19:06.156 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:19:06.156 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:19:06.156 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:06.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.138+0000 7fea14924640 1 librados: starting msgr at 2024-09-17T18:19:06.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.138+0000 7fea14924640 1 librados: starting objecter 2024-09-17T18:19:06.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.138+0000 7fea14924640 1 Processor -- start 2024-09-17T18:19:06.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.138+0000 7fea14924640 1 -- start start 2024-09-17T18:19:06.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.138+0000 7fea14924640 1 librados: setting wanted keys 2024-09-17T18:19:06.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.138+0000 7fea14924640 1 librados: calling monclient init 2024-09-17T18:19:06.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.138+0000 7fea14924640 10 monclient: init 2024-09-17T18:19:06.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea14924640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:06.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea14924640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:19:06.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea14924640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9f8003ad0 0x7fea100733b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:06.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea14924640 10 monclient(hunting): picked mon.noname-a con 0x7fe9f8003ad0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:06.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea14924640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea100738f0 0x7fea10073d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:06.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea14924640 10 monclient(hunting): picked mon.noname-b con 0x7fea100738f0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:06.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea14924640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea101de970 0x7fea101dedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:06.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea14924640 10 monclient(hunting): picked mon.noname-c con 0x7fea101de970 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:06.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea14924640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:06.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea14924640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fea1008dc90 con 0x7fe9f8003ad0 2024-09-17T18:19:06.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea14924640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:06.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea14924640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fea100a2410 con 0x7fea100738f0 2024-09-17T18:19:06.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea14924640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:06.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea14924640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fea101c9c80 con 0x7fea101de970 2024-09-17T18:19:06.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea14924640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:06.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea14924640 10 monclient(hunting): authenticate will time out at 749.855413s 2024-09-17T18:19:06.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea0e59c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9f8003ad0 0x7fea100733b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:06.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea0e59c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9f8003ad0 0x7fea100733b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41146/0 (socket says 172.21.15.121:41146) 2024-09-17T18:19:06.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea0e59c640 1 -- 172.21.15.121:0/2857668666 learned_addr learned my addr 172.21.15.121:0/2857668666 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:06.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea0e59c640 10 monclient(hunting): get_auth_request con 0x7fe9f8003ad0 auth_method 0 2024-09-17T18:19:06.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:06.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea0e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:06.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea0e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:06.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea0ed9d640 1 --2- 172.21.15.121:0/2857668666 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea101de970 0x7fea101dedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:06.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea0e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:06.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea0e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:06.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea0e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:06.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea0ed9d640 10 monclient(hunting): get_auth_request con 0x7fea101de970 auth_method 0 2024-09-17T18:19:06.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea0ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:06.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea0ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:06.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea0ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:06.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea0dd9b640 1 --2- 172.21.15.121:0/2857668666 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea100738f0 0x7fea10073d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:06.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea0dd9b640 10 monclient(hunting): get_auth_request con 0x7fea100738f0 auth_method 0 2024-09-17T18:19:06.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea0dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:06.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea0dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:06.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea0dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:06.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea0ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:06.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea0ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:06.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea0ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:06.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea0dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:06.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea0dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:06.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea0dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:06.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea0e59c640 10 monclient(hunting): handle_auth_done global_id 4320 payload 824 2024-09-17T18:19:06.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea0e59c640 1 -- 172.21.15.121:0/2857668666 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea101de970 msgr2=0x7fea101dedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:06.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea0e59c640 1 --2- 172.21.15.121:0/2857668666 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea101de970 0x7fea101dedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:06.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea0e59c640 1 -- 172.21.15.121:0/2857668666 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea100738f0 msgr2=0x7fea10073d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:06.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea0e59c640 1 --2- 172.21.15.121:0/2857668666 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea100738f0 0x7fea10073d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:06.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea0e59c640 10 monclient: _finish_hunting 0 2024-09-17T18:19:06.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea0e59c640 1 monclient: found mon.noname-a 2024-09-17T18:19:06.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea0e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:06.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea0e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:19:06.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.139+0000 7fea0e59c640 1 -- 172.21.15.121:0/2857668666 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea1009e240 con 0x7fe9f8003ad0 2024-09-17T18:19:06.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.141+0000 7fea0e59c640 1 -- 172.21.15.121:0/2857668666 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9f8003ad0 msgr2=0x7fea100733b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:06.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.141+0000 7fea0e59c640 1 --2- 172.21.15.121:0/2857668666 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9f8003ad0 0x7fea100733b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fea00000c00 tx=0x7fea00009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:06.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.141+0000 7fe9feffd640 10 client.?.objecter ms_handle_connect 0x7fe9f8003ad0 2024-09-17T18:19:06.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.141+0000 7fe9feffd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:19:06.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.142+0000 7fe9feffd640 1 -- 172.21.15.121:0/2857668666 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea00037040 con 0x7fe9f8003ad0 2024-09-17T18:19:06.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.142+0000 7fe9feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:06.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.142+0000 7fe9feffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:19:06.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.142+0000 7fe9feffd640 10 monclient: dump: 2024-09-17T18:19:06.161 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:06.161 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:06.161 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:06.161 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:06.161 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:06.162 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:06.162 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:06.162 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:06.162 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:06.162 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:06.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.142+0000 7fe9feffd640 10 monclient: _finish_auth 0 2024-09-17T18:19:06.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.142+0000 7fe9feffd640 10 monclient: _check_auth_tickets 2024-09-17T18:19:06.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.142+0000 7fe9feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:06.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.142+0000 7fe9feffd640 1 -- 172.21.15.121:0/2857668666 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea00033cc0 con 0x7fe9f8003ad0 2024-09-17T18:19:06.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.142+0000 7fea14924640 5 monclient: authenticate success, global_id 4320 2024-09-17T18:19:06.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.142+0000 7fea14924640 10 monclient: _renew_subs 2024-09-17T18:19:06.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.142+0000 7fea14924640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:06.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.142+0000 7fea14924640 1 -- 172.21.15.121:0/2857668666 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea100a55e0 con 0x7fe9f8003ad0 2024-09-17T18:19:06.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.142+0000 7fe9feffd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:06.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.142+0000 7fe9feffd640 1 -- 172.21.15.121:0/2857668666 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea00004520 con 0x7fe9f8003ad0 2024-09-17T18:19:06.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.142+0000 7fe9feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:06.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.142+0000 7fe9feffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:19:06.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.142+0000 7fe9feffd640 10 monclient: dump: 2024-09-17T18:19:06.163 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:06.163 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:06.163 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:06.163 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:06.163 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:06.163 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:06.164 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:06.164 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:06.164 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:06.164 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:06.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.142+0000 7fea0cd99640 4 set_mon_vals no callback set 2024-09-17T18:19:06.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.142+0000 7fea14924640 10 client.4320.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:06.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.142+0000 7fea14924640 10 monclient: _renew_subs 2024-09-17T18:19:06.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.142+0000 7fea14924640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:06.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.142+0000 7fea14924640 1 -- 172.21.15.121:0/2857668666 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea1007d6b0 con 0x7fe9f8003ad0 2024-09-17T18:19:06.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.142+0000 7fea14924640 1 librados: init done 2024-09-17T18:19:06.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.142+0000 7fe9feffd640 1 -- 172.21.15.121:0/2857668666 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fea00003910 con 0x7fe9f8003ad0 2024-09-17T18:19:06.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.143+0000 7fe9feffd640 1 --2- 172.21.15.121:0/2857668666 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe9f00401f0 0x7fe9f00426b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:06.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.143+0000 7fe9feffd640 1 -- 172.21.15.121:0/2857668666 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 3562+0+0 (secure 0 0 0) 0x7fea00072be0 con 0x7fe9f8003ad0 2024-09-17T18:19:06.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.143+0000 7fea14924640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:06.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.143+0000 7fe9feffd640 10 client.4320.objecter ms_dispatch 0x7fea10000b90 osd_map(16..16 src has 1..16) 2024-09-17T18:19:06.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.143+0000 7fe9feffd640 3 client.4320.objecter handle_osd_map got epochs [16,16] > 0 2024-09-17T18:19:06.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.143+0000 7fe9feffd640 3 client.4320.objecter handle_osd_map decoding full epoch 16 2024-09-17T18:19:06.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.143+0000 7fea14924640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:06.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.143+0000 7fea14924640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:06.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.143+0000 7fea14924640 1 -- 172.21.15.121:0/2857668666 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea1009d0b0 con 0x7fe9f8003ad0 2024-09-17T18:19:06.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.143+0000 7fe9feffd640 20 client.4320.objecter dump_active .. 0 homeless 2024-09-17T18:19:06.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.143+0000 7fea0dd9b640 1 --2- 172.21.15.121:0/2857668666 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe9f00401f0 0x7fe9f00426b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:06.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.143+0000 7fea0dd9b640 10 monclient: get_auth_request con 0x7fe9f00401f0 auth_method 0 2024-09-17T18:19:06.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.146+0000 7fea0dd9b640 1 --2- 172.21.15.121:0/2857668666 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe9f00401f0 0x7fe9f00426b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fea10074a60 tx=0x7fea04007a10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:06.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.146+0000 7fe9feffd640 10 client.4320.objecter ms_handle_connect 0x7fe9f00401f0 2024-09-17T18:19:06.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.147+0000 7fe9feffd640 1 -- 172.21.15.121:0/2857668666 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fea0002ae20 con 0x7fe9f8003ad0 2024-09-17T18:19:06.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.147+0000 7fe9feffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:06.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.147+0000 7fe9feffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:19:06.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.322+0000 7fea14924640 1 -- 172.21.15.121:0/2857668666 --> v2:172.21.15.173:6812/2423786037 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fea100008d0 con 0x7fe9f00401f0 2024-09-17T18:19:06.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.323+0000 7fe9feffd640 1 -- 172.21.15.121:0/2857668666 <== mgr.4105 v2:172.21.15.173:6812/2423786037 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34209 (secure 0 0 0) 0x7fea100008d0 con 0x7fe9f00401f0 2024-09-17T18:19:06.325 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:19:06.325 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-09-17T18:19:06.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.325+0000 7fea14924640 10 librados: watch_flush enter 2024-09-17T18:19:06.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.325+0000 7fea14924640 10 librados: watch_flush exit 2024-09-17T18:19:06.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.325+0000 7fea14924640 20 client.4320.objecter shutdown clearing up homeless session... 2024-09-17T18:19:06.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.325+0000 7fea14924640 10 client.4320.objecter successfully canceled tick 2024-09-17T18:19:06.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.325+0000 7fea14924640 1 -- 172.21.15.121:0/2857668666 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe9f00401f0 msgr2=0x7fe9f00426b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:06.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.325+0000 7fea14924640 1 --2- 172.21.15.121:0/2857668666 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe9f00401f0 0x7fe9f00426b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fea10074a60 tx=0x7fea04007a10 comp rx=0 tx=0).stop 2024-09-17T18:19:06.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.325+0000 7fea14924640 10 monclient: shutdown 2024-09-17T18:19:06.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.325+0000 7fea14924640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:06.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.325+0000 7fea14924640 1 -- 172.21.15.121:0/2857668666 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9f8003ad0 msgr2=0x7fea100733b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:06.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.325+0000 7fea14924640 1 --2- 172.21.15.121:0/2857668666 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9f8003ad0 0x7fea100733b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fea00000c00 tx=0x7fea00009a20 comp rx=0 tx=0).stop 2024-09-17T18:19:06.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.325+0000 7fea14924640 1 -- 172.21.15.121:0/2857668666 shutdown_connections 2024-09-17T18:19:06.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.325+0000 7fea14924640 1 --2- 172.21.15.121:0/2857668666 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe9f00401f0 0x7fe9f00426b0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:06.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.325+0000 7fea14924640 1 --2- 172.21.15.121:0/2857668666 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea100738f0 0x7fea10073d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:06.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.325+0000 7fea14924640 1 --2- 172.21.15.121:0/2857668666 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea101de970 0x7fea101dedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:06.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.325+0000 7fea14924640 1 --2- 172.21.15.121:0/2857668666 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9f8003ad0 0x7fea100733b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:06.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.325+0000 7fea14924640 1 -- 172.21.15.121:0/2857668666 >> 172.21.15.121:0/2857668666 conn(0x7fea100a7230 msgr2=0x7fea1008c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:06.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.325+0000 7fea14924640 1 -- 172.21.15.121:0/2857668666 shutdown_connections 2024-09-17T18:19:06.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.325+0000 7fea14924640 1 -- 172.21.15.121:0/2857668666 wait complete. 2024-09-17T18:19:06.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.325+0000 7fea14924640 1 librados: shutdown 2024-09-17T18:19:06.338 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2024-09-17T18:19:05.496474+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163136,"kb_used_data":944,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067904,"statfs":{"total":579820584960,"available":579653533696,"internally_reserved":0,"allocated":966656,"data_stored":1313332,"data_compressed":59128,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":165927585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.970505"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T18:19:01.527478+0000","last_change":"2024-09-17T18:19:01.527658+0000","last_active":"2024-09-17T18:19:01.527478+0000","last_peered":"2024-09-17T18:19:01.527478+0000","last_clean":"2024-09-17T18:19:01.527478+0000","last_became_active":"2024-09-17T18:19:00.056752+0000","last_became_peered":"2024-09-17T18:19:00.056752+0000","last_unstale":"2024-09-17T18:19:01.527478+0000","last_undegraded":"2024-09-17T18:19:01.527478+0000","last_fullsized":"2024-09-17T18:19:01.527478+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:57:47.271964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049856399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T18:19:01.527858+0000","last_change":"2024-09-17T18:19:01.527858+0000","last_active":"2024-09-17T18:19:01.527858+0000","last_peered":"2024-09-17T18:19:01.527858+0000","last_clean":"2024-09-17T18:19:01.527858+0000","last_became_active":"2024-09-17T18:18:59.525295+0000","last_became_peered":"2024-09-17T18:18:59.525295+0000","last_unstale":"2024-09-17T18:19:01.527858+0000","last_undegraded":"2024-09-17T18:19:01.527858+0000","last_fullsized":"2024-09-17T18:19:01.527858+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:41:21.181263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062121300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T18:19:01.527332+0000","last_change":"2024-09-17T18:19:01.527563+0000","last_active":"2024-09-17T18:19:01.527332+0000","last_peered":"2024-09-17T18:19:01.527332+0000","last_clean":"2024-09-17T18:19:01.527332+0000","last_became_active":"2024-09-17T18:19:00.244851+0000","last_became_peered":"2024-09-17T18:19:00.244851+0000","last_unstale":"2024-09-17T18:19:01.527332+0000","last_undegraded":"2024-09-17T18:19:01.527332+0000","last_fullsized":"2024-09-17T18:19:01.527332+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:19:57.036107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041282100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T18:19:01.527784+0000","last_change":"2024-09-17T18:19:01.527784+0000","last_active":"2024-09-17T18:19:01.527784+0000","last_peered":"2024-09-17T18:19:01.527784+0000","last_clean":"2024-09-17T18:19:01.527784+0000","last_became_active":"2024-09-17T18:19:00.244613+0000","last_became_peered":"2024-09-17T18:19:00.244613+0000","last_unstale":"2024-09-17T18:19:01.527784+0000","last_undegraded":"2024-09-17T18:19:01.527784+0000","last_fullsized":"2024-09-17T18:19:01.527784+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:31:24.599660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063646500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":22,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T18:19:01.532623+0000","last_change":"2024-09-17T18:19:01.532623+0000","last_active":"2024-09-17T18:19:01.532623+0000","last_peered":"2024-09-17T18:19:01.532623+0000","last_clean":"2024-09-17T18:19:01.532623+0000","last_became_active":"2024-09-17T18:18:59.526791+0000","last_became_peered":"2024-09-17T18:18:59.526791+0000","last_unstale":"2024-09-17T18:19:01.532623+0000","last_undegraded":"2024-09-17T18:19:01.532623+0000","last_fullsized":"2024-09-17T18:19:01.532623+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T21:46:24.217486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010088289999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T18:19:01.527597+0000","last_change":"2024-09-17T18:19:01.527848+0000","last_active":"2024-09-17T18:19:01.527597+0000","last_peered":"2024-09-17T18:19:01.527597+0000","last_clean":"2024-09-17T18:19:01.527597+0000","last_became_active":"2024-09-17T18:18:59.529390+0000","last_became_peered":"2024-09-17T18:18:59.529390+0000","last_unstale":"2024-09-17T18:19:01.527597+0000","last_undegraded":"2024-09-17T18:19:01.527597+0000","last_fullsized":"2024-09-17T18:19:01.527597+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:40:45.376247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065986199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T18:19:01.527258+0000","last_change":"2024-09-17T18:19:01.527425+0000","last_active":"2024-09-17T18:19:01.527258+0000","last_peered":"2024-09-17T18:19:01.527258+0000","last_clean":"2024-09-17T18:19:01.527258+0000","last_became_active":"2024-09-17T18:18:59.526855+0000","last_became_peered":"2024-09-17T18:18:59.526855+0000","last_unstale":"2024-09-17T18:19:01.527258+0000","last_undegraded":"2024-09-17T18:19:01.527258+0000","last_fullsized":"2024-09-17T18:19:01.527258+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:05:38.754621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032532500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T18:19:01.531845+0000","last_change":"2024-09-17T18:19:01.532053+0000","last_active":"2024-09-17T18:19:01.531845+0000","last_peered":"2024-09-17T18:19:01.531845+0000","last_clean":"2024-09-17T18:19:01.531845+0000","last_became_active":"2024-09-17T18:18:59.526797+0000","last_became_peered":"2024-09-17T18:18:59.526797+0000","last_unstale":"2024-09-17T18:19:01.531845+0000","last_undegraded":"2024-09-17T18:19:01.531845+0000","last_fullsized":"2024-09-17T18:19:01.531845+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:19:28.196155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000425759,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"15'72","reported_seq":105,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T18:19:01.527033+0000","last_change":"2024-09-17T18:19:00.245854+0000","last_active":"2024-09-17T18:19:01.527033+0000","last_peered":"2024-09-17T18:19:01.527033+0000","last_clean":"2024-09-17T18:19:01.527033+0000","last_became_active":"2024-09-17T18:19:00.244903+0000","last_became_peered":"2024-09-17T18:19:00.244903+0000","last_unstale":"2024-09-17T18:19:01.527033+0000","last_undegraded":"2024-09-17T18:19:01.527033+0000","last_fullsized":"2024-09-17T18:19:01.527033+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:55:38.336150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59128,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607557,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":139264,"data_stored":42573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607556,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":143360,"data_stored":42592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":143360,"data_stored":42592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":200704,"data_stored":571501,"data_compressed":29564,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":139264,"data_stored":42573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":200704,"data_stored":571501,"data_compressed":29564,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29564,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29564,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T18:19:06.338 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-17T18:19:06.339 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-17T18:19:06.339 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-17T18:19:06.339 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-17T18:19:06.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.431+0000 7f354c048640 10 monclient: get_monmap_and_config 2024-09-17T18:19:06.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.431+0000 7f354c048640 10 monclient: build_initial_monmap 2024-09-17T18:19:06.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.431+0000 7f354c048640 10 monclient: monmap: 2024-09-17T18:19:06.434 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:06.434 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:06.434 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:19:06.432490+0000 2024-09-17T18:19:06.434 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:19:06.432490+0000 2024-09-17T18:19:06.434 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:06.434 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:06.434 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:19:06.434 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:19:06.434 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:19:06.435 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:19:06.435 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:19:06.435 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:06.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.432+0000 7f354c048640 1 Processor -- start 2024-09-17T18:19:06.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.432+0000 7f354c048640 1 -- start start 2024-09-17T18:19:06.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.432+0000 7f354c048640 10 monclient: init 2024-09-17T18:19:06.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f354c048640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:06.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f354c048640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-17T18:19:06.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f354c048640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35440738a0 0x7f3544138a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:06.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f354c048640 10 monclient(hunting): picked mon.noname-b con 0x7f35440738a0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:06.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f354c048640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f3544073c80 0x7f35440740c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:06.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f354c048640 10 monclient(hunting): picked mon.noname-c con 0x7f3544073c80 addr v2:172.21.15.173:3300/0 2024-09-17T18:19:06.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f354c048640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f35440a0e70 0x7f35440a12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:06.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f354c048640 10 monclient(hunting): picked mon.noname-a con 0x7f35440a0e70 addr v2:172.21.15.121:3300/0 2024-09-17T18:19:06.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f354c048640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:06.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f354c048640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f3544070350 con 0x7f35440a0e70 2024-09-17T18:19:06.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f354c048640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:06.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f354c048640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f354406fd90 con 0x7f3544073c80 2024-09-17T18:19:06.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f354c048640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:06.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f354c048640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3544070070 con 0x7f35440738a0 2024-09-17T18:19:06.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f354c048640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:06.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f354c048640 10 monclient(hunting): authenticate will time out at 750.149252s 2024-09-17T18:19:06.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f3549de4640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35440738a0 0x7f3544138a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:06.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f354a5e5640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f35440a0e70 0x7f35440a12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:06.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f354a5e5640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f35440a0e70 0x7f35440a12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41148/0 (socket says 172.21.15.121:41148) 2024-09-17T18:19:06.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f354a5e5640 1 -- 172.21.15.121:0/1136104821 learned_addr learned my addr 172.21.15.121:0/1136104821 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:06.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f3549de4640 10 monclient(hunting): get_auth_request con 0x7f35440738a0 auth_method 0 2024-09-17T18:19:06.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f3549de4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:06.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f3549de4640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:06.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f3549de4640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:06.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f354a5e5640 10 monclient(hunting): get_auth_request con 0x7f35440a0e70 auth_method 0 2024-09-17T18:19:06.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f354a5e5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:06.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f354a5e5640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:06.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f354a5e5640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:06.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f354a5e5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:06.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f354a5e5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:06.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f354a5e5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:06.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f3549de4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:06.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f3549de4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:06.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f3549de4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:06.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f35495e3640 1 --2- 172.21.15.121:0/1136104821 >> v2:172.21.15.173:3300/0 conn(0x7f3544073c80 0x7f35440740c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:06.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f35495e3640 10 monclient(hunting): get_auth_request con 0x7f3544073c80 auth_method 0 2024-09-17T18:19:06.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f35495e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:06.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f35495e3640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:06.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f35495e3640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:06.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f354a5e5640 10 monclient(hunting): handle_auth_done global_id 4323 payload 274 2024-09-17T18:19:06.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f354a5e5640 1 -- 172.21.15.121:0/1136104821 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35440738a0 msgr2=0x7f3544138a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:06.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f354a5e5640 1 --2- 172.21.15.121:0/1136104821 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35440738a0 0x7f3544138a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:06.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f354a5e5640 1 -- 172.21.15.121:0/1136104821 >> v2:172.21.15.173:3300/0 conn(0x7f3544073c80 msgr2=0x7f35440740c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:06.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f354a5e5640 1 --2- 172.21.15.121:0/1136104821 >> v2:172.21.15.173:3300/0 conn(0x7f3544073c80 0x7f35440740c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:06.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f354a5e5640 10 monclient: _finish_hunting 0 2024-09-17T18:19:06.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f354a5e5640 1 monclient: found mon.noname-a 2024-09-17T18:19:06.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f354a5e5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:06.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f354a5e5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:19:06.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f354a5e5640 1 -- 172.21.15.121:0/1136104821 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35440a4020 con 0x7f35440a0e70 2024-09-17T18:19:06.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f35495e3640 1 --2- 172.21.15.121:0/1136104821 >> v2:172.21.15.173:3300/0 conn(0x7f3544073c80 0x7f35440740c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T18:19:06.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.433+0000 7f3549de4640 1 --2- 172.21.15.121:0/1136104821 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35440738a0 0x7f3544138a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:19:06.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f354a5e5640 1 -- 172.21.15.121:0/1136104821 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35440a0e70 msgr2=0x7f35440a12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:06.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f354a5e5640 1 --2- 172.21.15.121:0/1136104821 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35440a0e70 0x7f35440a12f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f35440a1f40 tx=0x7f353402a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c4cbbf5725f6155 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:06.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f3548de2640 1 -- 172.21.15.121:0/1136104821 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3534036070 con 0x7f35440a0e70 2024-09-17T18:19:06.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f3548de2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:06.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f354a5e5640 1 -- 172.21.15.121:0/1136104821 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35440a0e70 msgr2=0x7f35440a12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-09-17T18:19:06.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f354a5e5640 1 -- 172.21.15.121:0/1136104821 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35440a0e70 msgr2=0x7f35440a12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:19:06.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f354a5e5640 1 --2- 172.21.15.121:0/1136104821 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35440a0e70 0x7f35440a12f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f35440a1f40 tx=0x7f353402a570 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:19:06.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f3548de2640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:19:06.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f354a5e5640 1 --2- 172.21.15.121:0/1136104821 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35440a0e70 0x7f35440a12f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f35440a1f40 tx=0x7f353402a570 comp rx=0 tx=0).stop 2024-09-17T18:19:06.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f3548de2640 10 monclient: dump: 2024-09-17T18:19:06.451 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:06.451 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:06.451 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:06.451 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:06.451 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:06.451 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:06.452 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:06.452 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:06.452 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:06.452 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:06.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f3548de2640 10 monclient: _finish_auth 0 2024-09-17T18:19:06.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f3548de2640 10 monclient: _check_auth_tickets 2024-09-17T18:19:06.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f3548de2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:06.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f3548de2640 10 monclient: ms_handle_reset current mon [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:06.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f3548de2640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:06.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f3548de2640 1 -- 172.21.15.121:0/1136104821 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35440a0e70 msgr2=0x7f35440a12f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:19:06.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f3548de2640 1 --2- 172.21.15.121:0/1136104821 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35440a0e70 0x7f35440a12f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:06.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f3548de2640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T18:19:06.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f3548de2640 1 --2- 172.21.15.121:0/1136104821 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35440738a0 0x7f352c005760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:06.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f3548de2640 10 monclient(hunting): picked mon.c con 0x7f35440738a0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:06.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f3548de2640 1 --2- 172.21.15.121:0/1136104821 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f352c005ec0 0x7f352c008350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:06.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f3548de2640 10 monclient(hunting): picked mon.a con 0x7f352c005ec0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:06.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f3548de2640 1 --2- 172.21.15.121:0/1136104821 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f352c008b90 0x7f352c00b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:06.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f3548de2640 10 monclient(hunting): picked mon.b con 0x7f352c008b90 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:06.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f3548de2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:06.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f3548de2640 1 -- 172.21.15.121:0/1136104821 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f352c00b960 con 0x7f352c005ec0 2024-09-17T18:19:06.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f3548de2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:06.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f3548de2640 1 -- 172.21.15.121:0/1136104821 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f352c00bb10 con 0x7f352c008b90 2024-09-17T18:19:06.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f3548de2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:06.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f3548de2640 1 -- 172.21.15.121:0/1136104821 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f352c00bd10 con 0x7f35440738a0 2024-09-17T18:19:06.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f3548de2640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:06.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f3548de2640 10 monclient(hunting): ms_handle_reset hunted mon [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:06.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f3549de4640 1 --2- 172.21.15.121:0/1136104821 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35440738a0 0x7f352c005760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:06.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f35495e3640 1 --2- 172.21.15.121:0/1136104821 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f352c005ec0 0x7f352c008350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:06.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f35495e3640 10 monclient(hunting): get_auth_request con 0x7f352c005ec0 auth_method 0 2024-09-17T18:19:06.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f35495e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:06.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f35495e3640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:06.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f35495e3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:19:06.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f3549de4640 10 monclient(hunting): get_auth_request con 0x7f35440738a0 auth_method 0 2024-09-17T18:19:06.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f3549de4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:06.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f3549de4640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:06.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f3549de4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:19:06.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f35495e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:06.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f35495e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:06.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f35495e3640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:19:06.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f3549de4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:06.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f3549de4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:06.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f3549de4640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:19:06.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f35495e3640 10 monclient(hunting): handle_auth_done global_id 4323 payload 293 2024-09-17T18:19:06.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f354a5e5640 1 --2- 172.21.15.121:0/1136104821 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f352c008b90 0x7f352c00b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:06.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f35495e3640 1 -- 172.21.15.121:0/1136104821 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35440738a0 msgr2=0x7f352c005760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:06.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f35495e3640 1 --2- 172.21.15.121:0/1136104821 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35440738a0 0x7f352c005760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:06.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f35495e3640 1 -- 172.21.15.121:0/1136104821 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f352c008b90 msgr2=0x7f352c00b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:06.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f35495e3640 1 --2- 172.21.15.121:0/1136104821 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f352c008b90 0x7f352c00b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:06.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f35495e3640 10 monclient: _finish_hunting 0 2024-09-17T18:19:06.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f35495e3640 1 monclient: found mon.a 2024-09-17T18:19:06.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f35495e3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:06.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f35495e3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:06.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f35495e3640 1 -- 172.21.15.121:0/1136104821 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=2+}) -- 0x7f352c00bfd0 con 0x7f352c005ec0 2024-09-17T18:19:06.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f35495e3640 10 monclient: _finish_auth 0 2024-09-17T18:19:06.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f35495e3640 10 monclient: _check_auth_tickets 2024-09-17T18:19:06.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f35495e3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:06.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f354c048640 5 monclient: authenticate success, global_id 4323 2024-09-17T18:19:06.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f354c048640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:19:06.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f354a5e5640 10 monclient: get_auth_request con 0x7f352c008b90 auth_method 0 2024-09-17T18:19:06.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.436+0000 7f354a5e5640 1 --2- 172.21.15.121:0/1136104821 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f352c008b90 0x7f352c00b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T18:19:06.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.439+0000 7f35495e3640 1 -- 172.21.15.121:0/1136104821 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f352c005ec0 msgr2=0x7f352c008350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:06.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.439+0000 7f35495e3640 1 --2- 172.21.15.121:0/1136104821 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f352c005ec0 0x7f352c008350 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f354406fc10 tx=0x7f3534009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cec6fb139802b12 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:06.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.439+0000 7f3548de2640 1 -- 172.21.15.121:0/1136104821 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353800f040 con 0x7f352c005ec0 2024-09-17T18:19:06.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.439+0000 7f3548de2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:06.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.439+0000 7f3548de2640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:19:06.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.439+0000 7f3548de2640 10 monclient: dump: 2024-09-17T18:19:06.457 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:06.457 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:06.457 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:06.457 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:06.457 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:06.457 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:06.458 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:06.458 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:06.458 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:06.458 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:06.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.439+0000 7f354c048640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:19:06.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.439+0000 7f3548de2640 1 -- 172.21.15.121:0/1136104821 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3538004230 con 0x7f352c005ec0 2024-09-17T18:19:06.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.439+0000 7f3548de2640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:06.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.439+0000 7f354c048640 10 monclient: get_monmap_and_config success 2024-09-17T18:19:06.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.439+0000 7f354c048640 4 set_mon_vals no callback set 2024-09-17T18:19:06.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.439+0000 7f354c048640 10 monclient: shutdown 2024-09-17T18:19:06.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.439+0000 7f354c048640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:06.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.439+0000 7f354c048640 1 -- 172.21.15.121:0/1136104821 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f352c005ec0 msgr2=0x7f352c008350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:06.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.439+0000 7f354c048640 1 --2- 172.21.15.121:0/1136104821 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f352c005ec0 0x7f352c008350 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f354406fc10 tx=0x7f3534009810 comp rx=0 tx=0).stop 2024-09-17T18:19:06.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.439+0000 7f354a5e5640 1 -- 172.21.15.121:0/1136104821 reap_dead start 2024-09-17T18:19:06.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.439+0000 7f354c048640 1 -- 172.21.15.121:0/1136104821 shutdown_connections 2024-09-17T18:19:06.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.439+0000 7f354c048640 1 --2- 172.21.15.121:0/1136104821 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35440a0e70 0x7f35440a12f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:06.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.439+0000 7f354c048640 1 -- 172.21.15.121:0/1136104821 >> 172.21.15.121:0/1136104821 conn(0x7f3544076f00 msgr2=0x7f3544077310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:06.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.440+0000 7f354c048640 1 -- 172.21.15.121:0/1136104821 shutdown_connections 2024-09-17T18:19:06.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.440+0000 7f354c048640 1 -- 172.21.15.121:0/1136104821 wait complete. 2024-09-17T18:19:06.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.440+0000 7f354c048640 10 monclient: build_initial_monmap 2024-09-17T18:19:06.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.440+0000 7f354c048640 10 monclient: monmap: 2024-09-17T18:19:06.459 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:06.459 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:19:06.459 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:19:06.459 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:19:06.459 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:06.459 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:06.460 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:19:06.460 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:19:06.460 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:19:06.460 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:06.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.440+0000 7f354c048640 1 librados: starting msgr at 2024-09-17T18:19:06.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.440+0000 7f354c048640 1 librados: starting objecter 2024-09-17T18:19:06.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.440+0000 7f354c048640 1 Processor -- start 2024-09-17T18:19:06.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f354c048640 1 -- start start 2024-09-17T18:19:06.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f354c048640 1 librados: setting wanted keys 2024-09-17T18:19:06.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f354c048640 1 librados: calling monclient init 2024-09-17T18:19:06.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f354c048640 10 monclient: init 2024-09-17T18:19:06.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f354c048640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:06.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f354c048640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:19:06.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f354c048640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35440a0e70 0x7f35441cd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:06.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f354c048640 10 monclient(hunting): picked mon.noname-a con 0x7f35440a0e70 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:06.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f354c048640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f352c005ec0 0x7f35441cdc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:06.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f354c048640 10 monclient(hunting): picked mon.noname-b con 0x7f352c005ec0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:06.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f354c048640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3544123900 0x7f3544123d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:06.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f354c048640 10 monclient(hunting): picked mon.noname-c con 0x7f3544123900 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:06.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f354c048640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:06.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f354c048640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f354406d820 con 0x7f35440a0e70 2024-09-17T18:19:06.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f354c048640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:06.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f354c048640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f354406d6a0 con 0x7f352c005ec0 2024-09-17T18:19:06.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f354c048640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:06.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f354c048640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3544138a40 con 0x7f3544123900 2024-09-17T18:19:06.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f354c048640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:06.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f354c048640 10 monclient(hunting): authenticate will time out at 750.157472s 2024-09-17T18:19:06.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f3549de4640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35440a0e70 0x7f35441cd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:06.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f354a5e5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3544123900 0x7f3544123d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:06.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f3549de4640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35440a0e70 0x7f35441cd6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41166/0 (socket says 172.21.15.121:41166) 2024-09-17T18:19:06.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f3549de4640 1 -- 172.21.15.121:0/2259624366 learned_addr learned my addr 172.21.15.121:0/2259624366 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:06.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f3549de4640 10 monclient(hunting): get_auth_request con 0x7f35440a0e70 auth_method 0 2024-09-17T18:19:06.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f3549de4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:06.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f3549de4640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:06.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f3549de4640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:06.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f354a5e5640 10 monclient(hunting): get_auth_request con 0x7f3544123900 auth_method 0 2024-09-17T18:19:06.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f354a5e5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:06.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f354a5e5640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:06.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f354a5e5640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:06.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f35495e3640 1 --2- 172.21.15.121:0/2259624366 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f352c005ec0 0x7f35441cdc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:06.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f354a5e5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:06.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f354a5e5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:06.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f354a5e5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:06.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f3549de4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:06.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f3549de4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:06.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f3549de4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:06.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f35495e3640 10 monclient(hunting): get_auth_request con 0x7f352c005ec0 auth_method 0 2024-09-17T18:19:06.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f35495e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:06.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f35495e3640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:06.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f35495e3640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:06.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f35495e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:06.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f35495e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:06.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f35495e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:06.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f354a5e5640 10 monclient(hunting): handle_auth_done global_id 4307 payload 824 2024-09-17T18:19:06.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f354a5e5640 1 -- 172.21.15.121:0/2259624366 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f352c005ec0 msgr2=0x7f35441cdc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:06.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f354a5e5640 1 --2- 172.21.15.121:0/2259624366 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f352c005ec0 0x7f35441cdc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:06.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f354a5e5640 1 -- 172.21.15.121:0/2259624366 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35440a0e70 msgr2=0x7f35441cd6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:06.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f354a5e5640 1 --2- 172.21.15.121:0/2259624366 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35440a0e70 0x7f35441cd6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:06.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f354a5e5640 10 monclient: _finish_hunting 0 2024-09-17T18:19:06.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f354a5e5640 1 monclient: found mon.noname-c 2024-09-17T18:19:06.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f354a5e5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:06.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f354a5e5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:19:06.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f354a5e5640 1 -- 172.21.15.121:0/2259624366 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35441253c0 con 0x7f3544123900 2024-09-17T18:19:06.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.441+0000 7f3549de4640 1 --2- 172.21.15.121:0/2259624366 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35440a0e70 0x7f35441cd6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:19:06.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.444+0000 7f354a5e5640 1 -- 172.21.15.121:0/2259624366 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3544123900 msgr2=0x7f3544123d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:06.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.444+0000 7f354a5e5640 1 --2- 172.21.15.121:0/2259624366 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3544123900 0x7f3544123d90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3534030000 tx=0x7f3534004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:06.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.444+0000 7f35427fc640 10 client.?.objecter ms_handle_connect 0x7f3544123900 2024-09-17T18:19:06.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.444+0000 7f35427fc640 10 client.?.objecter resend_mon_ops 2024-09-17T18:19:06.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.444+0000 7f35427fc640 1 -- 172.21.15.121:0/2259624366 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3534041020 con 0x7f3544123900 2024-09-17T18:19:06.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.444+0000 7f35427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:06.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.444+0000 7f35427fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:19:06.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.444+0000 7f35427fc640 10 monclient: dump: 2024-09-17T18:19:06.466 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:06.466 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:06.466 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:06.466 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:06.466 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:06.466 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:06.466 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:06.466 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:06.466 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:06.466 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:06.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.444+0000 7f35427fc640 10 monclient: _finish_auth 0 2024-09-17T18:19:06.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.444+0000 7f35427fc640 10 monclient: _check_auth_tickets 2024-09-17T18:19:06.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.444+0000 7f35427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:06.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.444+0000 7f354c048640 5 monclient: authenticate success, global_id 4307 2024-09-17T18:19:06.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.444+0000 7f35427fc640 1 -- 172.21.15.121:0/2259624366 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3534037040 con 0x7f3544123900 2024-09-17T18:19:06.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.444+0000 7f354c048640 10 monclient: _renew_subs 2024-09-17T18:19:06.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.444+0000 7f354c048640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:06.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.444+0000 7f354c048640 1 -- 172.21.15.121:0/2259624366 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3544128100 con 0x7f3544123900 2024-09-17T18:19:06.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.444+0000 7f35427fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:06.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.444+0000 7f35427fc640 1 -- 172.21.15.121:0/2259624366 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3534036070 con 0x7f3544123900 2024-09-17T18:19:06.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.444+0000 7f35427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:06.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.444+0000 7f35427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:19:06.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.444+0000 7f35427fc640 10 monclient: dump: 2024-09-17T18:19:06.467 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:06.467 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:06.467 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:06.467 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:06.468 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:06.468 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:06.468 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:06.468 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:06.468 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:06.468 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:06.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.444+0000 7f354c048640 10 client.4307.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:06.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.444+0000 7f354c048640 10 monclient: _renew_subs 2024-09-17T18:19:06.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.444+0000 7f354c048640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:06.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.444+0000 7f3543fff640 4 set_mon_vals no callback set 2024-09-17T18:19:06.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.444+0000 7f354c048640 1 -- 172.21.15.121:0/2259624366 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f35441ce140 con 0x7f3544123900 2024-09-17T18:19:06.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.444+0000 7f354c048640 1 librados: init done 2024-09-17T18:19:06.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.445+0000 7f35427fc640 1 -- 172.21.15.121:0/2259624366 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f3534044020 con 0x7f3544123900 2024-09-17T18:19:06.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.445+0000 7f354c048640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:06.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.445+0000 7f354c048640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:06.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.445+0000 7f354c048640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:06.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.445+0000 7f354c048640 1 -- 172.21.15.121:0/2259624366 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35440740c0 con 0x7f3544123900 2024-09-17T18:19:06.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.445+0000 7f35427fc640 1 --2- 172.21.15.121:0/2259624366 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3530040170 0x7f3530042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:06.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.445+0000 7f35427fc640 1 -- 172.21.15.121:0/2259624366 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 3562+0+0 (secure 0 0 0) 0x7f3534074180 con 0x7f3544123900 2024-09-17T18:19:06.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.445+0000 7f35427fc640 10 client.4307.objecter ms_dispatch 0x7f3544000b90 osd_map(16..16 src has 1..16) 2024-09-17T18:19:06.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.445+0000 7f35427fc640 3 client.4307.objecter handle_osd_map got epochs [16,16] > 0 2024-09-17T18:19:06.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.445+0000 7f35427fc640 3 client.4307.objecter handle_osd_map decoding full epoch 16 2024-09-17T18:19:06.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.445+0000 7f35427fc640 20 client.4307.objecter dump_active .. 0 homeless 2024-09-17T18:19:06.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.445+0000 7f3549de4640 1 --2- 172.21.15.121:0/2259624366 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3530040170 0x7f3530042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:06.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.445+0000 7f3549de4640 10 monclient: get_auth_request con 0x7f3530040170 auth_method 0 2024-09-17T18:19:06.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.448+0000 7f35427fc640 10 client.4307.objecter ms_handle_connect 0x7f3530040170 2024-09-17T18:19:06.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.448+0000 7f3549de4640 1 --2- 172.21.15.121:0/2259624366 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3530040170 0x7f3530042630 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f353c014470 tx=0x7f353c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:06.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.450+0000 7f35427fc640 1 -- 172.21.15.121:0/2259624366 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f353404b030 con 0x7f3544123900 2024-09-17T18:19:06.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.450+0000 7f35427fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:06.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.450+0000 7f35427fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:19:06.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.663+0000 7f354c048640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-09-17T18:19:06.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.663+0000 7f354c048640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-09-17T18:19:06.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.663+0000 7f354c048640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:06.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.663+0000 7f354c048640 1 -- 172.21.15.121:0/2259624366 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f354406cd40 con 0x7f3544123900 2024-09-17T18:19:06.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.663+0000 7f35427fc640 1 -- 172.21.15.121:0/2259624366 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f3534073040 con 0x7f3544123900 2024-09-17T18:19:06.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.663+0000 7f35427fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-09-17T18:19:06.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.663+0000 7f35427fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:19:06.666 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:19:06.666 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-17T18:19:06.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.665+0000 7f354c048640 10 librados: watch_flush enter 2024-09-17T18:19:06.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.665+0000 7f354c048640 10 librados: watch_flush exit 2024-09-17T18:19:06.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.665+0000 7f354c048640 20 client.4307.objecter shutdown clearing up homeless session... 2024-09-17T18:19:06.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.665+0000 7f354c048640 10 client.4307.objecter successfully canceled tick 2024-09-17T18:19:06.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.665+0000 7f354c048640 1 -- 172.21.15.121:0/2259624366 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3530040170 msgr2=0x7f3530042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:06.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.665+0000 7f354c048640 1 --2- 172.21.15.121:0/2259624366 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3530040170 0x7f3530042630 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f353c014470 tx=0x7f353c009190 comp rx=0 tx=0).stop 2024-09-17T18:19:06.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.666+0000 7f354c048640 10 monclient: shutdown 2024-09-17T18:19:06.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.666+0000 7f354c048640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:06.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.666+0000 7f354c048640 1 -- 172.21.15.121:0/2259624366 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3544123900 msgr2=0x7f3544123d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:06.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.666+0000 7f354c048640 1 --2- 172.21.15.121:0/2259624366 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3544123900 0x7f3544123d90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3534030000 tx=0x7f3534004a40 comp rx=0 tx=0).stop 2024-09-17T18:19:06.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.666+0000 7f354c048640 1 -- 172.21.15.121:0/2259624366 shutdown_connections 2024-09-17T18:19:06.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.666+0000 7f354c048640 1 --2- 172.21.15.121:0/2259624366 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3530040170 0x7f3530042630 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:06.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.666+0000 7f354c048640 1 --2- 172.21.15.121:0/2259624366 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f352c005ec0 0x7f35441cdc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:06.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.666+0000 7f354c048640 1 --2- 172.21.15.121:0/2259624366 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3544123900 0x7f3544123d90 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:06.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.666+0000 7f354c048640 1 --2- 172.21.15.121:0/2259624366 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35440a0e70 0x7f35441cd6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:06.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.666+0000 7f354c048640 1 -- 172.21.15.121:0/2259624366 >> 172.21.15.121:0/2259624366 conn(0x7f3544076f00 msgr2=0x7f35440a5660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:06.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.666+0000 7f354c048640 1 -- 172.21.15.121:0/2259624366 shutdown_connections 2024-09-17T18:19:06.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.666+0000 7f354c048640 1 -- 172.21.15.121:0/2259624366 wait complete. 2024-09-17T18:19:06.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.666+0000 7f354c048640 1 librados: shutdown 2024-09-17T18:19:06.678 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-17T18:19:06.678 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-09-17T18:19:06.691 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-09-17T18:19:06.691 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T18:19:06.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.786+0000 7f542ce47640 10 monclient: get_monmap_and_config 2024-09-17T18:19:06.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.786+0000 7f542ce47640 10 monclient: build_initial_monmap 2024-09-17T18:19:06.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.786+0000 7f542ce47640 10 monclient: monmap: 2024-09-17T18:19:06.789 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:06.789 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:06.789 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:19:06.787559+0000 2024-09-17T18:19:06.789 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:19:06.787559+0000 2024-09-17T18:19:06.789 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:06.789 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:06.789 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:19:06.789 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:19:06.789 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:19:06.789 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:19:06.789 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:19:06.789 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:06.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.787+0000 7f542ce47640 1 Processor -- start 2024-09-17T18:19:06.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.787+0000 7f542ce47640 1 -- start start 2024-09-17T18:19:06.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.787+0000 7f542ce47640 10 monclient: init 2024-09-17T18:19:06.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.788+0000 7f542ce47640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:06.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.788+0000 7f542ce47640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-17T18:19:06.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.788+0000 7f542ce47640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f54281386b0 0x7f54280707d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:06.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.788+0000 7f542ce47640 10 monclient(hunting): picked mon.noname-b con 0x7f54281386b0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:06.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.788+0000 7f542ce47640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f542806ec00 0x7f5428138a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:06.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.788+0000 7f542ce47640 10 monclient(hunting): picked mon.noname-c con 0x7f542806ec00 addr v2:172.21.15.173:3300/0 2024-09-17T18:19:06.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.788+0000 7f542ce47640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5428138fd0 addr v1:172.21.15.173:6789/0 2024-09-17T18:19:06.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.788+0000 7f542ce47640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54280a5570 con 0x7f5428138fd0 2024-09-17T18:19:06.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.788+0000 7f542ce47640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:06.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.788+0000 7f542ce47640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f542806f830 con 0x7f542806ec00 2024-09-17T18:19:06.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.788+0000 7f542ce47640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:06.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.788+0000 7f542ce47640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f542806fb10 con 0x7f54281386b0 2024-09-17T18:19:06.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.788+0000 7f542ce47640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:06.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.788+0000 7f542ce47640 10 monclient(hunting): authenticate will time out at 750.504361s 2024-09-17T18:19:06.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.788+0000 7f542659c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f54281386b0 0x7f54280707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:06.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.788+0000 7f542659c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f54281386b0 0x7f54280707d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:59634/0 (socket says 172.21.15.121:59634) 2024-09-17T18:19:06.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.788+0000 7f542659c640 1 -- 172.21.15.121:0/2574766685 learned_addr learned my addr 172.21.15.121:0/2574766685 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:06.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.788+0000 7f542659c640 10 monclient(hunting): get_auth_request con 0x7f54281386b0 auth_method 0 2024-09-17T18:19:06.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.788+0000 7f542659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:06.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.788+0000 7f542659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:06.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.788+0000 7f542659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:06.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.788+0000 7f542659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:06.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.788+0000 7f542659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:06.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.788+0000 7f542659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:06.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.788+0000 7f542659c640 10 monclient(hunting): handle_auth_done global_id 4310 payload 274 2024-09-17T18:19:06.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.788+0000 7f542659c640 1 -- 172.21.15.121:0/2574766685 >> v2:172.21.15.173:3300/0 conn(0x7f542806ec00 msgr2=0x7f5428138a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:06.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.788+0000 7f542659c640 1 --2- 172.21.15.121:0/2574766685 >> v2:172.21.15.173:3300/0 conn(0x7f542806ec00 0x7f5428138a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:06.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.788+0000 7f542659c640 1 -- 172.21.15.121:0/2574766685 >> v1:172.21.15.173:6789/0 conn(0x7f5428138fd0 legacy=0x7f5428131d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:06.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.788+0000 7f542659c640 10 monclient: _finish_hunting 0 2024-09-17T18:19:06.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.788+0000 7f542659c640 1 monclient: found mon.noname-b 2024-09-17T18:19:06.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.788+0000 7f542659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:06.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.788+0000 7f542659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:19:06.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.788+0000 7f542659c640 1 -- 172.21.15.121:0/2574766685 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5428070d10 con 0x7f54281386b0 2024-09-17T18:19:06.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.791+0000 7f542659c640 1 -- 172.21.15.121:0/2574766685 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f54281386b0 msgr2=0x7f54280707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:06.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.791+0000 7f542659c640 1 --2- 172.21.15.121:0/2574766685 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f54281386b0 0x7f54280707d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5418009e50 tx=0x7f541802a560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31291e24c0505b6c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:06.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.791+0000 7f542559a640 1 -- 172.21.15.121:0/2574766685 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5418036070 con 0x7f54281386b0 2024-09-17T18:19:06.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.791+0000 7f542559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:06.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.791+0000 7f542559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:19:06.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.791+0000 7f542559a640 10 monclient: dump: 2024-09-17T18:19:06.801 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:06.801 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:06.801 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:06.801 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:06.802 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:06.802 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:06.802 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:06.802 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:06.802 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:06.802 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:06.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.791+0000 7f542559a640 10 monclient: _finish_auth 0 2024-09-17T18:19:06.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.791+0000 7f542559a640 10 monclient: _check_auth_tickets 2024-09-17T18:19:06.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.791+0000 7f542559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:06.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.791+0000 7f542559a640 1 -- 172.21.15.121:0/2574766685 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5418007800 con 0x7f54281386b0 2024-09-17T18:19:06.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.791+0000 7f542ce47640 5 monclient: authenticate success, global_id 4310 2024-09-17T18:19:06.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.791+0000 7f542ce47640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:19:06.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.791+0000 7f542559a640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:06.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.791+0000 7f542ce47640 10 monclient: get_monmap_and_config success 2024-09-17T18:19:06.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.791+0000 7f542559a640 1 -- 172.21.15.121:0/2574766685 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5418007bd0 con 0x7f54281386b0 2024-09-17T18:19:06.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.791+0000 7f542559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:06.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.791+0000 7f542ce47640 4 set_mon_vals no callback set 2024-09-17T18:19:06.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.791+0000 7f542559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:19:06.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.791+0000 7f542559a640 10 monclient: dump: 2024-09-17T18:19:06.803 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:06.803 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:06.803 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:06.803 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:06.803 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:06.803 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:06.803 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:06.803 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:06.804 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:06.804 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:06.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.792+0000 7f542ce47640 10 monclient: shutdown 2024-09-17T18:19:06.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.792+0000 7f542ce47640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:06.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.792+0000 7f542ce47640 1 -- 172.21.15.121:0/2574766685 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f54281386b0 msgr2=0x7f54280707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:06.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.792+0000 7f542ce47640 1 --2- 172.21.15.121:0/2574766685 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f54281386b0 0x7f54280707d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5418009e50 tx=0x7f541802a560 comp rx=0 tx=0).stop 2024-09-17T18:19:06.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.792+0000 7f542ce47640 1 -- 172.21.15.121:0/2574766685 shutdown_connections 2024-09-17T18:19:06.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.792+0000 7f542ce47640 1 --2- 172.21.15.121:0/2574766685 >> v2:172.21.15.173:3300/0 conn(0x7f542806ec00 0x7f5428138a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:06.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.792+0000 7f542ce47640 1 --2- 172.21.15.121:0/2574766685 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f54281386b0 0x7f54280707d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:06.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.792+0000 7f542ce47640 1 -- 172.21.15.121:0/2574766685 >> 172.21.15.121:0/2574766685 conn(0x7f54280a91e0 msgr2=0x7f54280a95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:06.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.792+0000 7f542ce47640 1 -- 172.21.15.121:0/2574766685 shutdown_connections 2024-09-17T18:19:06.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.792+0000 7f542ce47640 1 -- 172.21.15.121:0/2574766685 wait complete. 2024-09-17T18:19:06.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.793+0000 7f542ce47640 10 monclient: build_initial_monmap 2024-09-17T18:19:06.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.793+0000 7f542ce47640 10 monclient: monmap: 2024-09-17T18:19:06.805 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:06.805 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:19:06.805 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:19:06.805 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:19:06.805 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:06.805 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:06.805 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:19:06.805 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:19:06.805 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:19:06.805 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:06.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.793+0000 7f542ce47640 1 librados: starting msgr at 2024-09-17T18:19:06.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.793+0000 7f542ce47640 1 librados: starting objecter 2024-09-17T18:19:06.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.793+0000 7f542ce47640 1 Processor -- start 2024-09-17T18:19:06.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.793+0000 7f542ce47640 1 -- start start 2024-09-17T18:19:06.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.793+0000 7f542ce47640 1 librados: setting wanted keys 2024-09-17T18:19:06.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.793+0000 7f542ce47640 1 librados: calling monclient init 2024-09-17T18:19:06.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.793+0000 7f542ce47640 10 monclient: init 2024-09-17T18:19:06.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.793+0000 7f542ce47640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:06.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.793+0000 7f542ce47640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T18:19:06.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.793+0000 7f542ce47640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f542806ec00 0x7f5428072cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:06.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.793+0000 7f542ce47640 10 monclient(hunting): picked mon.noname-b con 0x7f542806ec00 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:06.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.793+0000 7f542ce47640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54281386b0 0x7f5428073200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:06.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.793+0000 7f542ce47640 10 monclient(hunting): picked mon.noname-a con 0x7f54281386b0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:06.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.793+0000 7f542ce47640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5428138fd0 0x7f5428073740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:06.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.793+0000 7f542ce47640 10 monclient(hunting): picked mon.noname-c con 0x7f5428138fd0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:06.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.793+0000 7f542ce47640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:06.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.793+0000 7f542ce47640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f54281325b0 con 0x7f54281386b0 2024-09-17T18:19:06.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.793+0000 7f542ce47640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:06.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.793+0000 7f542ce47640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5428132430 con 0x7f542806ec00 2024-09-17T18:19:06.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.793+0000 7f542ce47640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:06.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.793+0000 7f542ce47640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5428132730 con 0x7f5428138fd0 2024-09-17T18:19:06.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.793+0000 7f542ce47640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:06.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.793+0000 7f542ce47640 10 monclient(hunting): authenticate will time out at 750.509865s 2024-09-17T18:19:06.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.793+0000 7f5426d9d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5428138fd0 0x7f5428073740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:06.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.793+0000 7f5425d9b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54281386b0 0x7f5428073200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:06.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.793+0000 7f5426d9d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5428138fd0 0x7f5428073740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:59646/0 (socket says 172.21.15.121:59646) 2024-09-17T18:19:06.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.793+0000 7f5426d9d640 1 -- 172.21.15.121:0/2759095762 learned_addr learned my addr 172.21.15.121:0/2759095762 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:06.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.793+0000 7f5426d9d640 10 monclient(hunting): get_auth_request con 0x7f5428138fd0 auth_method 0 2024-09-17T18:19:06.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.794+0000 7f5426d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:06.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.794+0000 7f5426d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:06.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.794+0000 7f5426d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:06.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.794+0000 7f5425d9b640 10 monclient(hunting): get_auth_request con 0x7f54281386b0 auth_method 0 2024-09-17T18:19:06.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.794+0000 7f5425d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:06.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.794+0000 7f5425d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:06.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.794+0000 7f5425d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:06.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.794+0000 7f5426d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:06.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.794+0000 7f5426d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:06.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.794+0000 7f5426d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:06.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.794+0000 7f5425d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:06.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.794+0000 7f5425d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:06.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.794+0000 7f5425d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:06.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.794+0000 7f542659c640 1 --2- 172.21.15.121:0/2759095762 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f542806ec00 0x7f5428072cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:06.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.794+0000 7f5426d9d640 10 monclient(hunting): handle_auth_done global_id 4313 payload 824 2024-09-17T18:19:06.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.794+0000 7f5426d9d640 1 -- 172.21.15.121:0/2759095762 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f542806ec00 msgr2=0x7f5428072cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:06.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.794+0000 7f5426d9d640 1 --2- 172.21.15.121:0/2759095762 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f542806ec00 0x7f5428072cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:06.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.794+0000 7f5426d9d640 1 -- 172.21.15.121:0/2759095762 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54281386b0 msgr2=0x7f5428073200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:06.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.794+0000 7f5426d9d640 1 --2- 172.21.15.121:0/2759095762 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54281386b0 0x7f5428073200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:06.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.794+0000 7f5426d9d640 10 monclient: _finish_hunting 0 2024-09-17T18:19:06.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.794+0000 7f5426d9d640 1 monclient: found mon.noname-c 2024-09-17T18:19:06.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.794+0000 7f5426d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:06.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.794+0000 7f5426d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:19:06.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.794+0000 7f5426d9d640 1 -- 172.21.15.121:0/2759095762 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54281344a0 con 0x7f5428138fd0 2024-09-17T18:19:06.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.796+0000 7f5426d9d640 1 -- 172.21.15.121:0/2759095762 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5428138fd0 msgr2=0x7f5428073740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:06.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.796+0000 7f5426d9d640 1 --2- 172.21.15.121:0/2759095762 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5428138fd0 0x7f5428073740 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f542000c390 tx=0x7f542000c790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:06.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.796+0000 7f5416ffd640 10 client.?.objecter ms_handle_connect 0x7f5428138fd0 2024-09-17T18:19:06.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.796+0000 7f5416ffd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:19:06.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.796+0000 7f5416ffd640 1 -- 172.21.15.121:0/2759095762 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5420017070 con 0x7f5428138fd0 2024-09-17T18:19:06.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.796+0000 7f5416ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:06.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.796+0000 7f5416ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:19:06.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.796+0000 7f5416ffd640 10 monclient: dump: 2024-09-17T18:19:06.811 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:06.811 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:06.811 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:06.811 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:06.811 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:06.811 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:06.811 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:06.811 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:06.811 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:06.811 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:06.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.796+0000 7f5416ffd640 10 monclient: _finish_auth 0 2024-09-17T18:19:06.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.796+0000 7f5416ffd640 10 monclient: _check_auth_tickets 2024-09-17T18:19:06.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.796+0000 7f5416ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:06.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.796+0000 7f5416ffd640 1 -- 172.21.15.121:0/2759095762 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54200037f0 con 0x7f5428138fd0 2024-09-17T18:19:06.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.796+0000 7f542ce47640 5 monclient: authenticate success, global_id 4313 2024-09-17T18:19:06.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.796+0000 7f542ce47640 10 monclient: _renew_subs 2024-09-17T18:19:06.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.796+0000 7f542ce47640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:06.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.796+0000 7f542ce47640 1 -- 172.21.15.121:0/2759095762 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5428131e50 con 0x7f5428138fd0 2024-09-17T18:19:06.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.796+0000 7f5416ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:06.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.796+0000 7f5416ffd640 1 -- 172.21.15.121:0/2759095762 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5420003be0 con 0x7f5428138fd0 2024-09-17T18:19:06.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.796+0000 7f5416ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:06.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.796+0000 7f5416ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:19:06.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.796+0000 7f5416ffd640 10 monclient: dump: 2024-09-17T18:19:06.812 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:06.812 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:06.813 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:06.813 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:06.813 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:06.813 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:06.813 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:06.813 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:06.813 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:06.813 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:06.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.796+0000 7f5424d99640 4 set_mon_vals no callback set 2024-09-17T18:19:06.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.796+0000 7f542ce47640 10 client.4313.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:06.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.796+0000 7f542ce47640 10 monclient: _renew_subs 2024-09-17T18:19:06.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.796+0000 7f542ce47640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:06.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.796+0000 7f542ce47640 1 -- 172.21.15.121:0/2759095762 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f54281358f0 con 0x7f5428138fd0 2024-09-17T18:19:06.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.796+0000 7f542ce47640 1 librados: init done 2024-09-17T18:19:06.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.797+0000 7f5416ffd640 1 -- 172.21.15.121:0/2759095762 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f542001a070 con 0x7f5428138fd0 2024-09-17T18:19:06.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.797+0000 7f542ce47640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:06.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.797+0000 7f542ce47640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:06.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.797+0000 7f542ce47640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:06.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.797+0000 7f542ce47640 1 -- 172.21.15.121:0/2759095762 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5428132af0 con 0x7f5428138fd0 2024-09-17T18:19:06.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.797+0000 7f5416ffd640 1 --2- 172.21.15.121:0/2759095762 >> v2:172.21.15.173:6812/2423786037 conn(0x7f54080401a0 0x7f5408042660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:06.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.797+0000 7f5416ffd640 1 -- 172.21.15.121:0/2759095762 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 3562+0+0 (secure 0 0 0) 0x7f54200522f0 con 0x7f5428138fd0 2024-09-17T18:19:06.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.797+0000 7f5416ffd640 10 client.4313.objecter ms_dispatch 0x7f5428000b90 osd_map(16..16 src has 1..16) 2024-09-17T18:19:06.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.797+0000 7f5416ffd640 3 client.4313.objecter handle_osd_map got epochs [16,16] > 0 2024-09-17T18:19:06.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.797+0000 7f5416ffd640 3 client.4313.objecter handle_osd_map decoding full epoch 16 2024-09-17T18:19:06.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.797+0000 7f5416ffd640 20 client.4313.objecter dump_active .. 0 homeless 2024-09-17T18:19:06.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.798+0000 7f542659c640 1 --2- 172.21.15.121:0/2759095762 >> v2:172.21.15.173:6812/2423786037 conn(0x7f54080401a0 0x7f5408042660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:06.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.798+0000 7f542659c640 10 monclient: get_auth_request con 0x7f54080401a0 auth_method 0 2024-09-17T18:19:06.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.800+0000 7f5416ffd640 10 client.4313.objecter ms_handle_connect 0x7f54080401a0 2024-09-17T18:19:06.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.800+0000 7f542659c640 1 --2- 172.21.15.121:0/2759095762 >> v2:172.21.15.173:6812/2423786037 conn(0x7f54080401a0 0x7f5408042660 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f54280a37a0 tx=0x7f5418009000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:06.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.802+0000 7f5416ffd640 1 -- 172.21.15.121:0/2759095762 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f5420003e00 con 0x7f5428138fd0 2024-09-17T18:19:06.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.802+0000 7f5416ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:06.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.802+0000 7f5416ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:19:06.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.978+0000 7f542ce47640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-17T18:19:06.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.978+0000 7f542ce47640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T18:19:06.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.978+0000 7f542ce47640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:06.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.978+0000 7f542ce47640 1 -- 172.21.15.121:0/2759095762 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f54280744f0 con 0x7f5428138fd0 2024-09-17T18:19:06.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.979+0000 7f5416ffd640 1 -- 172.21.15.121:0/2759095762 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+10977 (secure 0 0 0) 0x7f54200511b0 con 0x7f5428138fd0 2024-09-17T18:19:06.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.979+0000 7f5416ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T18:19:06.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.979+0000 7f5416ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:19:06.982 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:19:06.982 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":16,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","created":"2024-09-17T18:18:47.237753+0000","modified":"2024-09-17T18:19:01.521005+0000","last_up_change":"2024-09-17T18:18:57.511756+0000","last_in_change":"2024-09-17T18:18:49.165275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T18:18:57.521369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-17T18:18:58.173538+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7bf586d1-05ac-4c7e-8b88-313f07a76801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":1526257129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6801","nonce":1526257129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":1526257129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":1526257129}]},"public_addr":"172.21.15.121:6800/1526257129","cluster_addr":"172.21.15.121:6801/1526257129","heartbeat_back_addr":"172.21.15.121:6803/1526257129","heartbeat_front_addr":"172.21.15.121:6802/1526257129","state":["exists","up"]},{"osd":1,"uuid":"e26c6163-1eda-4200-82e1-57b752fd8a46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":196419920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6809","nonce":196419920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":196419920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":196419920}]},"public_addr":"172.21.15.121:6808/196419920","cluster_addr":"172.21.15.121:6809/196419920","heartbeat_back_addr":"172.21.15.121:6811/196419920","heartbeat_front_addr":"172.21.15.121:6810/196419920","state":["exists","up"]},{"osd":2,"uuid":"6fb45798-21d4-449e-9f68-7210d984e036","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2545987420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6805","nonce":2545987420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":2545987420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2545987420}]},"public_addr":"172.21.15.121:6804/2545987420","cluster_addr":"172.21.15.121:6805/2545987420","heartbeat_back_addr":"172.21.15.121:6807/2545987420","heartbeat_front_addr":"172.21.15.121:6806/2545987420","state":["exists","up"]},{"osd":3,"uuid":"3de6630d-4fb7-4f64-b4f9-e5c304b1be63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":3493964597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6801","nonce":3493964597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6803","nonce":3493964597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":3493964597}]},"public_addr":"172.21.15.173:6800/3493964597","cluster_addr":"172.21.15.173:6801/3493964597","heartbeat_back_addr":"172.21.15.173:6803/3493964597","heartbeat_front_addr":"172.21.15.173:6802/3493964597","state":["exists","up"]},{"osd":4,"uuid":"cfa74383-b7c9-4ecd-8258-3792fbccc237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":1961801317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6805","nonce":1961801317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6807","nonce":1961801317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":1961801317}]},"public_addr":"172.21.15.173:6804/1961801317","cluster_addr":"172.21.15.173:6805/1961801317","heartbeat_back_addr":"172.21.15.173:6807/1961801317","heartbeat_front_addr":"172.21.15.173:6806/1961801317","state":["exists","up"]},{"osd":5,"uuid":"8bdb7768-a603-4f14-9650-47b503621215","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1801998802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6809","nonce":1801998802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6811","nonce":1801998802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1801998802}]},"public_addr":"172.21.15.173:6808/1801998802","cluster_addr":"172.21.15.173:6809/1801998802","heartbeat_back_addr":"172.21.15.173:6811/1801998802","heartbeat_front_addr":"172.21.15.173:6810/1801998802","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T18:18:55.231353+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T18:18:56.130097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T18:18:56.242909+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T18:18:55.454088+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T18:18:55.965469+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T18:18:55.389354+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T18:19:06.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.981+0000 7f542ce47640 10 librados: watch_flush enter 2024-09-17T18:19:06.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.981+0000 7f542ce47640 10 librados: watch_flush exit 2024-09-17T18:19:06.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.981+0000 7f542ce47640 20 client.4313.objecter shutdown clearing up homeless session... 2024-09-17T18:19:06.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.981+0000 7f542ce47640 10 client.4313.objecter successfully canceled tick 2024-09-17T18:19:06.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.981+0000 7f542ce47640 1 -- 172.21.15.121:0/2759095762 >> v2:172.21.15.173:6812/2423786037 conn(0x7f54080401a0 msgr2=0x7f5408042660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:06.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.981+0000 7f542ce47640 1 --2- 172.21.15.121:0/2759095762 >> v2:172.21.15.173:6812/2423786037 conn(0x7f54080401a0 0x7f5408042660 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f54280a37a0 tx=0x7f5418009000 comp rx=0 tx=0).stop 2024-09-17T18:19:06.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.981+0000 7f542ce47640 10 monclient: shutdown 2024-09-17T18:19:06.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.981+0000 7f542ce47640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:06.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.981+0000 7f542ce47640 1 -- 172.21.15.121:0/2759095762 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5428138fd0 msgr2=0x7f5428073740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:06.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.981+0000 7f542ce47640 1 --2- 172.21.15.121:0/2759095762 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5428138fd0 0x7f5428073740 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f542000c390 tx=0x7f542000c790 comp rx=0 tx=0).stop 2024-09-17T18:19:06.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.981+0000 7f542ce47640 1 -- 172.21.15.121:0/2759095762 shutdown_connections 2024-09-17T18:19:06.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.981+0000 7f542ce47640 1 --2- 172.21.15.121:0/2759095762 >> v2:172.21.15.173:6812/2423786037 conn(0x7f54080401a0 0x7f5408042660 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:06.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.981+0000 7f542ce47640 1 --2- 172.21.15.121:0/2759095762 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5428138fd0 0x7f5428073740 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:06.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.981+0000 7f542ce47640 1 --2- 172.21.15.121:0/2759095762 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54281386b0 0x7f5428073200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:06.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.982+0000 7f542ce47640 1 --2- 172.21.15.121:0/2759095762 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f542806ec00 0x7f5428072cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:06.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.982+0000 7f542ce47640 1 -- 172.21.15.121:0/2759095762 >> 172.21.15.121:0/2759095762 conn(0x7f54280a91e0 msgr2=0x7f54280a6fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:06.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.982+0000 7f542ce47640 1 -- 172.21.15.121:0/2759095762 shutdown_connections 2024-09-17T18:19:06.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.982+0000 7f542ce47640 1 -- 172.21.15.121:0/2759095762 wait complete. 2024-09-17T18:19:06.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:06.982+0000 7f542ce47640 1 librados: shutdown 2024-09-17T18:19:06.994 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T18:18:57.521369+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-17T18:18:58.173538+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 16, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-17T18:19:06.994 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-17T18:19:07.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.093+0000 7fa37b6f6640 10 monclient: get_monmap_and_config 2024-09-17T18:19:07.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.093+0000 7fa37b6f6640 10 monclient: build_initial_monmap 2024-09-17T18:19:07.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.093+0000 7fa37b6f6640 10 monclient: monmap: 2024-09-17T18:19:07.095 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:07.095 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:07.095 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:19:07.094158+0000 2024-09-17T18:19:07.096 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:19:07.094158+0000 2024-09-17T18:19:07.096 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:07.096 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:07.096 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:19:07.096 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:19:07.096 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:19:07.096 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:19:07.096 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:19:07.096 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:07.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.094+0000 7fa37b6f6640 1 Processor -- start 2024-09-17T18:19:07.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.094+0000 7fa37b6f6640 1 -- start start 2024-09-17T18:19:07.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.094+0000 7fa37b6f6640 10 monclient: init 2024-09-17T18:19:07.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.094+0000 7fa37b6f6640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:07.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.094+0000 7fa37b6f6640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-17T18:19:07.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.094+0000 7fa37b6f6640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fa37409e4e0 0x7fa37409e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:07.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.094+0000 7fa37b6f6640 10 monclient(hunting): picked mon.noname-c con 0x7fa37409e4e0 addr v2:172.21.15.173:3300/0 2024-09-17T18:19:07.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.094+0000 7fa37b6f6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa3740a06e0 addr v1:172.21.15.173:6789/0 2024-09-17T18:19:07.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.094+0000 7fa37b6f6640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa37409ee00 0x7fa37413cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:07.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.094+0000 7fa37b6f6640 10 monclient(hunting): picked mon.noname-b con 0x7fa37409ee00 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:07.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.094+0000 7fa37b6f6640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3740a48a0 con 0x7fa3740a06e0 2024-09-17T18:19:07.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.094+0000 7fa37b6f6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:07.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.094+0000 7fa37b6f6640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fa3740a46c0 con 0x7fa37409e4e0 2024-09-17T18:19:07.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.094+0000 7fa37b6f6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:07.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.094+0000 7fa37b6f6640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa3740a4460 con 0x7fa37409ee00 2024-09-17T18:19:07.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.094+0000 7fa37b6f6640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:07.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.094+0000 7fa37b6f6640 10 monclient(hunting): authenticate will time out at 750.810852s 2024-09-17T18:19:07.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.094+0000 7fa379c93640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa37409ee00 0x7fa37413cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:07.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.094+0000 7fa379c93640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa37409ee00 0x7fa37413cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:59660/0 (socket says 172.21.15.121:59660) 2024-09-17T18:19:07.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.095+0000 7fa379c93640 1 -- 172.21.15.121:0/2302480745 learned_addr learned my addr 172.21.15.121:0/2302480745 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:07.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.095+0000 7fa379c93640 10 monclient(hunting): get_auth_request con 0x7fa37409ee00 auth_method 0 2024-09-17T18:19:07.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.095+0000 7fa379c93640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:07.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.095+0000 7fa379c93640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:07.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.095+0000 7fa379c93640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:07.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.095+0000 7fa379c93640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:07.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.095+0000 7fa379c93640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:07.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.095+0000 7fa379c93640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:07.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.095+0000 7fa379492640 1 --2- 172.21.15.121:0/2302480745 >> v2:172.21.15.173:3300/0 conn(0x7fa37409e4e0 0x7fa37409e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:07.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.095+0000 7fa379492640 10 monclient(hunting): get_auth_request con 0x7fa37409e4e0 auth_method 0 2024-09-17T18:19:07.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.095+0000 7fa379492640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:07.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.095+0000 7fa379492640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:07.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.095+0000 7fa379492640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:07.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.095+0000 7fa379c93640 10 monclient(hunting): handle_auth_done global_id 4316 payload 274 2024-09-17T18:19:07.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.095+0000 7fa379c93640 1 -- 172.21.15.121:0/2302480745 >> v2:172.21.15.173:3300/0 conn(0x7fa37409e4e0 msgr2=0x7fa37409e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:07.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.095+0000 7fa379c93640 1 --2- 172.21.15.121:0/2302480745 >> v2:172.21.15.173:3300/0 conn(0x7fa37409e4e0 0x7fa37409e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:07.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.095+0000 7fa379c93640 1 -- 172.21.15.121:0/2302480745 >> v1:172.21.15.173:6789/0 conn(0x7fa3740a06e0 legacy=0x7fa374139840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:07.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.095+0000 7fa379c93640 10 monclient: _finish_hunting 0 2024-09-17T18:19:07.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.095+0000 7fa379c93640 1 monclient: found mon.noname-b 2024-09-17T18:19:07.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.095+0000 7fa379c93640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:07.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.095+0000 7fa379c93640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:19:07.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.095+0000 7fa379c93640 1 -- 172.21.15.121:0/2302480745 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa37409f7d0 con 0x7fa37409ee00 2024-09-17T18:19:07.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.097+0000 7fa379c93640 1 -- 172.21.15.121:0/2302480745 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa37409ee00 msgr2=0x7fa37413cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:07.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.097+0000 7fa379c93640 1 --2- 172.21.15.121:0/2302480745 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa37409ee00 0x7fa37413cf70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa36c00d150 tx=0x7fa36c02c760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e83538dfdb808f6d server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:07.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.098+0000 7fa36bfff640 1 -- 172.21.15.121:0/2302480745 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa36c038070 con 0x7fa37409ee00 2024-09-17T18:19:07.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.098+0000 7fa36bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:07.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.098+0000 7fa36bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:19:07.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.098+0000 7fa36bfff640 10 monclient: dump: 2024-09-17T18:19:07.107 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:07.107 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:07.107 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:07.107 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:07.108 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:07.108 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:07.108 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:07.108 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:07.108 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:07.108 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:07.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.098+0000 7fa36bfff640 10 monclient: _finish_auth 0 2024-09-17T18:19:07.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.098+0000 7fa36bfff640 10 monclient: _check_auth_tickets 2024-09-17T18:19:07.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.098+0000 7fa36bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:07.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.098+0000 7fa36bfff640 1 -- 172.21.15.121:0/2302480745 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa36c009800 con 0x7fa37409ee00 2024-09-17T18:19:07.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.098+0000 7fa37b6f6640 5 monclient: authenticate success, global_id 4316 2024-09-17T18:19:07.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.098+0000 7fa37b6f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:19:07.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.098+0000 7fa36bfff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:07.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.098+0000 7fa37b6f6640 10 monclient: get_monmap_and_config success 2024-09-17T18:19:07.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.098+0000 7fa37b6f6640 4 set_mon_vals no callback set 2024-09-17T18:19:07.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.098+0000 7fa36bfff640 1 -- 172.21.15.121:0/2302480745 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa36c009bd0 con 0x7fa37409ee00 2024-09-17T18:19:07.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.098+0000 7fa36bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:07.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.098+0000 7fa36bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:19:07.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.098+0000 7fa36bfff640 10 monclient: dump: 2024-09-17T18:19:07.109 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:07.109 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:07.109 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:07.109 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:07.109 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:07.109 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:07.109 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:07.109 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:07.109 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:07.110 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:07.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.098+0000 7fa37b6f6640 10 monclient: shutdown 2024-09-17T18:19:07.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.098+0000 7fa37b6f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:07.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.098+0000 7fa37b6f6640 1 -- 172.21.15.121:0/2302480745 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa37409ee00 msgr2=0x7fa37413cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:07.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.098+0000 7fa37b6f6640 1 --2- 172.21.15.121:0/2302480745 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa37409ee00 0x7fa37413cf70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa36c00d150 tx=0x7fa36c02c760 comp rx=0 tx=0).stop 2024-09-17T18:19:07.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.098+0000 7fa37b6f6640 1 -- 172.21.15.121:0/2302480745 shutdown_connections 2024-09-17T18:19:07.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.098+0000 7fa37b6f6640 1 --2- 172.21.15.121:0/2302480745 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa37409ee00 0x7fa37413cf70 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:07.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.098+0000 7fa37b6f6640 1 --2- 172.21.15.121:0/2302480745 >> v2:172.21.15.173:3300/0 conn(0x7fa37409e4e0 0x7fa37409e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:07.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.098+0000 7fa37b6f6640 1 -- 172.21.15.121:0/2302480745 >> 172.21.15.121:0/2302480745 conn(0x7fa3740a78d0 msgr2=0x7fa3740a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:07.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.098+0000 7fa37b6f6640 1 -- 172.21.15.121:0/2302480745 shutdown_connections 2024-09-17T18:19:07.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.098+0000 7fa37b6f6640 1 -- 172.21.15.121:0/2302480745 wait complete. 2024-09-17T18:19:07.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.099+0000 7fa37b6f6640 10 monclient: build_initial_monmap 2024-09-17T18:19:07.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.099+0000 7fa37b6f6640 10 monclient: monmap: 2024-09-17T18:19:07.110 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:07.111 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:19:07.111 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:19:07.111 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:19:07.111 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:07.111 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:07.111 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:19:07.111 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:19:07.111 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:19:07.111 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:07.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.099+0000 7fa37b6f6640 1 librados: starting msgr at 2024-09-17T18:19:07.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.099+0000 7fa37b6f6640 1 librados: starting objecter 2024-09-17T18:19:07.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.099+0000 7fa37b6f6640 1 Processor -- start 2024-09-17T18:19:07.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.099+0000 7fa37b6f6640 1 -- start start 2024-09-17T18:19:07.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.099+0000 7fa37b6f6640 1 librados: setting wanted keys 2024-09-17T18:19:07.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.099+0000 7fa37b6f6640 1 librados: calling monclient init 2024-09-17T18:19:07.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.099+0000 7fa37b6f6640 10 monclient: init 2024-09-17T18:19:07.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa37b6f6640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:07.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa37b6f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T18:19:07.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa37b6f6640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa37409e4e0 0x7fa3741ad620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:07.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa37b6f6640 10 monclient(hunting): picked mon.noname-b con 0x7fa37409e4e0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:07.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa37b6f6640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa37409ee00 0x7fa3741cd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:07.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa37b6f6640 10 monclient(hunting): picked mon.noname-c con 0x7fa37409ee00 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:07.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa37b6f6640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3740a06e0 0x7fa3741c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:07.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa37b6f6640 10 monclient(hunting): picked mon.noname-a con 0x7fa3740a06e0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:07.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa37b6f6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:07.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa37b6f6640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa37409de60 con 0x7fa3740a06e0 2024-09-17T18:19:07.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa37b6f6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:07.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa37b6f6640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa37409dce0 con 0x7fa37409e4e0 2024-09-17T18:19:07.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa37b6f6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:07.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa37b6f6640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa374074840 con 0x7fa37409ee00 2024-09-17T18:19:07.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa37b6f6640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:07.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa37b6f6640 10 monclient(hunting): authenticate will time out at 750.816259s 2024-09-17T18:19:07.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa378c91640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa37409ee00 0x7fa3741cd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:07.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa379c93640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3740a06e0 0x7fa3741c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:07.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa378c91640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa37409ee00 0x7fa3741cd9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:59676/0 (socket says 172.21.15.121:59676) 2024-09-17T18:19:07.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa378c91640 1 -- 172.21.15.121:0/1277990351 learned_addr learned my addr 172.21.15.121:0/1277990351 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:07.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa378c91640 10 monclient(hunting): get_auth_request con 0x7fa37409ee00 auth_method 0 2024-09-17T18:19:07.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa378c91640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:07.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa378c91640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:07.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa378c91640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:07.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa379c93640 10 monclient(hunting): get_auth_request con 0x7fa3740a06e0 auth_method 0 2024-09-17T18:19:07.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa379c93640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:07.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa379c93640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:07.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa379c93640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:07.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa378c91640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:07.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa378c91640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:07.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa378c91640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:07.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa379c93640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:07.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa379c93640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:07.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa379c93640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:07.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa379492640 1 --2- 172.21.15.121:0/1277990351 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa37409e4e0 0x7fa3741ad620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:07.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa379492640 10 monclient(hunting): get_auth_request con 0x7fa37409e4e0 auth_method 0 2024-09-17T18:19:07.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa379492640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:07.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa379492640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:07.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa379492640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:07.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa378c91640 10 monclient(hunting): handle_auth_done global_id 4319 payload 824 2024-09-17T18:19:07.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa378c91640 1 -- 172.21.15.121:0/1277990351 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa37409e4e0 msgr2=0x7fa3741ad620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:07.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa378c91640 1 --2- 172.21.15.121:0/1277990351 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa37409e4e0 0x7fa3741ad620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:07.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa378c91640 1 -- 172.21.15.121:0/1277990351 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3740a06e0 msgr2=0x7fa3741c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:07.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa378c91640 1 --2- 172.21.15.121:0/1277990351 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3740a06e0 0x7fa3741c73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:07.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa378c91640 10 monclient: _finish_hunting 0 2024-09-17T18:19:07.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa378c91640 1 monclient: found mon.noname-c 2024-09-17T18:19:07.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa378c91640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:07.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa378c91640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:19:07.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa378c91640 1 -- 172.21.15.121:0/1277990351 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3741adfb0 con 0x7fa37409ee00 2024-09-17T18:19:07.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa379c93640 1 --2- 172.21.15.121:0/1277990351 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3740a06e0 0x7fa3741c73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:19:07.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.100+0000 7fa379492640 1 --2- 172.21.15.121:0/1277990351 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa37409e4e0 0x7fa3741ad620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T18:19:07.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.102+0000 7fa378c91640 1 -- 172.21.15.121:0/1277990351 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa37409ee00 msgr2=0x7fa3741cd9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:07.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.102+0000 7fa378c91640 1 --2- 172.21.15.121:0/1277990351 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa37409ee00 0x7fa3741cd9e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa36400c460 tx=0x7fa36400c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:07.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.102+0000 7fa369ffb640 10 client.?.objecter ms_handle_connect 0x7fa37409ee00 2024-09-17T18:19:07.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.102+0000 7fa369ffb640 10 client.?.objecter resend_mon_ops 2024-09-17T18:19:07.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.103+0000 7fa369ffb640 1 -- 172.21.15.121:0/1277990351 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa364016020 con 0x7fa37409ee00 2024-09-17T18:19:07.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.103+0000 7fa369ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:07.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.103+0000 7fa369ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:19:07.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.103+0000 7fa369ffb640 10 monclient: dump: 2024-09-17T18:19:07.116 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:07.116 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:07.116 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:07.116 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:07.116 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:07.116 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:07.116 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:07.116 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:07.116 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:07.117 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:07.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.103+0000 7fa369ffb640 10 monclient: _finish_auth 0 2024-09-17T18:19:07.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.103+0000 7fa369ffb640 10 monclient: _check_auth_tickets 2024-09-17T18:19:07.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.103+0000 7fa369ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:07.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.103+0000 7fa369ffb640 1 -- 172.21.15.121:0/1277990351 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa364012a80 con 0x7fa37409ee00 2024-09-17T18:19:07.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.103+0000 7fa37b6f6640 5 monclient: authenticate success, global_id 4319 2024-09-17T18:19:07.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.103+0000 7fa37b6f6640 10 monclient: _renew_subs 2024-09-17T18:19:07.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.103+0000 7fa37b6f6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:07.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.103+0000 7fa37b6f6640 1 -- 172.21.15.121:0/1277990351 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa37409e1d0 con 0x7fa37409ee00 2024-09-17T18:19:07.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.103+0000 7fa369ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:07.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.103+0000 7fa369ffb640 1 -- 172.21.15.121:0/1277990351 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa364011380 con 0x7fa37409ee00 2024-09-17T18:19:07.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.103+0000 7fa369ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:07.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.103+0000 7fa369ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:19:07.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.103+0000 7fa369ffb640 10 monclient: dump: 2024-09-17T18:19:07.118 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:07.118 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:07.118 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:07.118 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:07.118 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:07.118 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:07.118 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:07.118 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:07.118 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:07.118 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:07.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.103+0000 7fa37b6f6640 10 client.4319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:07.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.103+0000 7fa36b7fe640 4 set_mon_vals no callback set 2024-09-17T18:19:07.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.103+0000 7fa37b6f6640 10 monclient: _renew_subs 2024-09-17T18:19:07.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.103+0000 7fa37b6f6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:07.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.103+0000 7fa37b6f6640 1 -- 172.21.15.121:0/1277990351 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3741aee20 con 0x7fa37409ee00 2024-09-17T18:19:07.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.103+0000 7fa37b6f6640 1 librados: init done 2024-09-17T18:19:07.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.103+0000 7fa369ffb640 1 -- 172.21.15.121:0/1277990351 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fa364019020 con 0x7fa37409ee00 2024-09-17T18:19:07.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.104+0000 7fa369ffb640 1 --2- 172.21.15.121:0/1277990351 >> v2:172.21.15.173:6812/2423786037 conn(0x7fa3580401f0 0x7fa3580426b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:07.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.104+0000 7fa37b6f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:07.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.104+0000 7fa37b6f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:07.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.104+0000 7fa369ffb640 1 -- 172.21.15.121:0/1277990351 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(17..17 src has 1..17) ==== 3534+0+0 (secure 0 0 0) 0x7fa3640529c0 con 0x7fa37409ee00 2024-09-17T18:19:07.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.104+0000 7fa37b6f6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:07.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.104+0000 7fa369ffb640 10 client.4319.objecter ms_dispatch 0x7fa374000b90 osd_map(17..17 src has 1..17) 2024-09-17T18:19:07.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.104+0000 7fa369ffb640 3 client.4319.objecter handle_osd_map got epochs [17,17] > 0 2024-09-17T18:19:07.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.104+0000 7fa369ffb640 3 client.4319.objecter handle_osd_map decoding full epoch 17 2024-09-17T18:19:07.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.104+0000 7fa37b6f6640 1 -- 172.21.15.121:0/1277990351 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa37409e8c0 con 0x7fa37409ee00 2024-09-17T18:19:07.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.104+0000 7fa369ffb640 20 client.4319.objecter dump_active .. 0 homeless 2024-09-17T18:19:07.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.104+0000 7fa379492640 1 --2- 172.21.15.121:0/1277990351 >> v2:172.21.15.173:6812/2423786037 conn(0x7fa3580401f0 0x7fa3580426b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:07.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.104+0000 7fa379492640 10 monclient: get_auth_request con 0x7fa3580401f0 auth_method 0 2024-09-17T18:19:07.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.107+0000 7fa379492640 1 --2- 172.21.15.121:0/1277990351 >> v2:172.21.15.173:6812/2423786037 conn(0x7fa3580401f0 0x7fa3580426b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa370006fd0 tx=0x7fa370008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:07.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.107+0000 7fa369ffb640 10 client.4319.objecter ms_handle_connect 0x7fa3580401f0 2024-09-17T18:19:07.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.109+0000 7fa369ffb640 1 -- 172.21.15.121:0/1277990351 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fa364020070 con 0x7fa37409ee00 2024-09-17T18:19:07.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.109+0000 7fa369ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:07.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.109+0000 7fa369ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:19:07.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.283+0000 7fa37b6f6640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-17T18:19:07.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.283+0000 7fa37b6f6640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-17T18:19:07.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.283+0000 7fa37b6f6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:07.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.283+0000 7fa37b6f6640 1 -- 172.21.15.121:0/1277990351 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa37406cd40 con 0x7fa37409ee00 2024-09-17T18:19:07.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.283+0000 7fa369ffb640 1 -- 172.21.15.121:0/1277990351 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) ==== 93+0+10 (secure 0 0 0) 0x7fa364011d60 con 0x7fa37409ee00 2024-09-17T18:19:07.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.283+0000 7fa369ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-17T18:19:07.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.283+0000 7fa369ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:19:07.286 INFO:teuthology.orchestra.run.smithi121.stdout:pg_num: 1 2024-09-17T18:19:07.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.285+0000 7fa37b6f6640 10 librados: watch_flush enter 2024-09-17T18:19:07.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.285+0000 7fa37b6f6640 10 librados: watch_flush exit 2024-09-17T18:19:07.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.285+0000 7fa37b6f6640 20 client.4319.objecter shutdown clearing up homeless session... 2024-09-17T18:19:07.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.285+0000 7fa37b6f6640 10 client.4319.objecter successfully canceled tick 2024-09-17T18:19:07.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.285+0000 7fa37b6f6640 1 -- 172.21.15.121:0/1277990351 >> v2:172.21.15.173:6812/2423786037 conn(0x7fa3580401f0 msgr2=0x7fa3580426b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:07.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.285+0000 7fa37b6f6640 1 --2- 172.21.15.121:0/1277990351 >> v2:172.21.15.173:6812/2423786037 conn(0x7fa3580401f0 0x7fa3580426b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa370006fd0 tx=0x7fa370008040 comp rx=0 tx=0).stop 2024-09-17T18:19:07.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.285+0000 7fa37b6f6640 10 monclient: shutdown 2024-09-17T18:19:07.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.285+0000 7fa37b6f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:07.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.285+0000 7fa37b6f6640 1 -- 172.21.15.121:0/1277990351 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa37409ee00 msgr2=0x7fa3741cd9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:07.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.285+0000 7fa37b6f6640 1 --2- 172.21.15.121:0/1277990351 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa37409ee00 0x7fa3741cd9e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa36400c460 tx=0x7fa36400c920 comp rx=0 tx=0).stop 2024-09-17T18:19:07.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.285+0000 7fa37b6f6640 1 -- 172.21.15.121:0/1277990351 shutdown_connections 2024-09-17T18:19:07.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.285+0000 7fa37b6f6640 1 --2- 172.21.15.121:0/1277990351 >> v2:172.21.15.173:6812/2423786037 conn(0x7fa3580401f0 0x7fa3580426b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:07.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.285+0000 7fa37b6f6640 1 --2- 172.21.15.121:0/1277990351 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3740a06e0 0x7fa3741c73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:07.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.285+0000 7fa37b6f6640 1 --2- 172.21.15.121:0/1277990351 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa37409ee00 0x7fa3741cd9e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:07.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.285+0000 7fa37b6f6640 1 --2- 172.21.15.121:0/1277990351 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa37409e4e0 0x7fa3741ad620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:07.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.285+0000 7fa37b6f6640 1 -- 172.21.15.121:0/1277990351 >> 172.21.15.121:0/1277990351 conn(0x7fa3740a78d0 msgr2=0x7fa374072dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:07.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.286+0000 7fa37b6f6640 1 -- 172.21.15.121:0/1277990351 shutdown_connections 2024-09-17T18:19:07.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.286+0000 7fa37b6f6640 1 -- 172.21.15.121:0/1277990351 wait complete. 2024-09-17T18:19:07.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.286+0000 7fa37b6f6640 1 librados: shutdown 2024-09-17T18:19:07.298 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-09-17T18:19:07.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.395+0000 7fa393784640 10 monclient: get_monmap_and_config 2024-09-17T18:19:07.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.395+0000 7fa393784640 10 monclient: build_initial_monmap 2024-09-17T18:19:07.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.395+0000 7fa393784640 10 monclient: monmap: 2024-09-17T18:19:07.398 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:07.398 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:07.398 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:19:07.396466+0000 2024-09-17T18:19:07.398 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:19:07.396466+0000 2024-09-17T18:19:07.398 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:07.398 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:07.398 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:19:07.398 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:19:07.398 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:19:07.398 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:19:07.398 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:19:07.398 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:07.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.396+0000 7fa393784640 1 Processor -- start 2024-09-17T18:19:07.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.396+0000 7fa393784640 1 -- start start 2024-09-17T18:19:07.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.396+0000 7fa393784640 10 monclient: init 2024-09-17T18:19:07.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.397+0000 7fa393784640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:07.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.397+0000 7fa393784640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-17T18:19:07.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.397+0000 7fa393784640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fa38c0a3a70 0x7fa38c09dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:07.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.397+0000 7fa393784640 10 monclient(hunting): picked mon.noname-c con 0x7fa38c0a3a70 addr v2:172.21.15.173:3300/0 2024-09-17T18:19:07.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.397+0000 7fa393784640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa38c0a3e50 addr v1:172.21.15.173:6789/0 2024-09-17T18:19:07.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.397+0000 7fa393784640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa38c13dca0 addr v1:172.21.15.121:6789/0 2024-09-17T18:19:07.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.397+0000 7fa393784640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa38c09d420 con 0x7fa38c13dca0 2024-09-17T18:19:07.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.397+0000 7fa393784640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa38c076a90 con 0x7fa38c0a3e50 2024-09-17T18:19:07.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.397+0000 7fa393784640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:07.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.397+0000 7fa393784640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fa38c0768d0 con 0x7fa38c0a3a70 2024-09-17T18:19:07.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.397+0000 7fa393784640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:07.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.397+0000 7fa393784640 10 monclient(hunting): authenticate will time out at 751.113184s 2024-09-17T18:19:07.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.397+0000 7fa391d21640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fa38c13dca0 0x7fa38c140090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:49662/0 (socket says 172.21.15.121:49662) 2024-09-17T18:19:07.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.397+0000 7fa391d21640 1 -- 172.21.15.121:0/1739060698 learned_addr learned my addr 172.21.15.121:0/1739060698 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:19:07.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.397+0000 7fa391520640 1 --2- 172.21.15.121:0/1739060698 >> v2:172.21.15.173:3300/0 conn(0x7fa38c0a3a70 0x7fa38c09dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:07.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.397+0000 7fa391520640 10 monclient(hunting): get_auth_request con 0x7fa38c0a3a70 auth_method 0 2024-09-17T18:19:07.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.397+0000 7fa391520640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:07.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.397+0000 7fa391520640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:07.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.397+0000 7fa391520640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:07.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.397+0000 7fa391520640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:07.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.397+0000 7fa391520640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:07.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.397+0000 7fa391520640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:07.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.397+0000 7fa391520640 10 monclient(hunting): handle_auth_done global_id 4330 payload 274 2024-09-17T18:19:07.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.397+0000 7fa391520640 1 -- 172.21.15.121:0/1739060698 >> v1:172.21.15.173:6789/0 conn(0x7fa38c0a3e50 legacy=0x7fa38c0a4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:07.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.397+0000 7fa391520640 1 -- 172.21.15.121:0/1739060698 >> v1:172.21.15.121:6789/0 conn(0x7fa38c13dca0 legacy=0x7fa38c140090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:07.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.397+0000 7fa391520640 10 monclient: _finish_hunting 0 2024-09-17T18:19:07.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.397+0000 7fa391520640 1 monclient: found mon.noname-c 2024-09-17T18:19:07.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.397+0000 7fa391520640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:07.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.397+0000 7fa391520640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.173:3300/0 2024-09-17T18:19:07.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.397+0000 7fa391520640 1 -- 172.21.15.121:0/1739060698 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa38c0a54e0 con 0x7fa38c0a3a70 2024-09-17T18:19:07.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.400+0000 7fa391520640 1 -- 172.21.15.121:0/1739060698 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa38c0a3a70 msgr2=0x7fa38c09dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:07.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.400+0000 7fa391520640 1 --2- 172.21.15.121:0/1739060698 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa38c0a3a70 0x7fa38c09dc00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa37c0097d0 tx=0x7fa37c02a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bcb969036e6dc07 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:07.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.400+0000 7fa38bfff640 1 -- 172.21.15.121:0/1739060698 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa37c037070 con 0x7fa38c0a3a70 2024-09-17T18:19:07.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.400+0000 7fa38bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:07.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.400+0000 7fa38bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:19:07.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.400+0000 7fa38bfff640 10 monclient: dump: 2024-09-17T18:19:07.410 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:07.410 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:07.411 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:07.411 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:07.411 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:07.411 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:07.411 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:07.411 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:07.411 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:07.411 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:07.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.400+0000 7fa38bfff640 10 monclient: _finish_auth 0 2024-09-17T18:19:07.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.400+0000 7fa38bfff640 10 monclient: _check_auth_tickets 2024-09-17T18:19:07.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.400+0000 7fa38bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:07.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.400+0000 7fa38bfff640 1 -- 172.21.15.121:0/1739060698 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa37c032440 con 0x7fa38c0a3a70 2024-09-17T18:19:07.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.400+0000 7fa393784640 5 monclient: authenticate success, global_id 4330 2024-09-17T18:19:07.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.400+0000 7fa393784640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:19:07.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.400+0000 7fa38bfff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:07.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.400+0000 7fa393784640 10 monclient: get_monmap_and_config success 2024-09-17T18:19:07.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.400+0000 7fa38bfff640 1 -- 172.21.15.121:0/1739060698 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa37c032810 con 0x7fa38c0a3a70 2024-09-17T18:19:07.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.400+0000 7fa38bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:07.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.400+0000 7fa393784640 4 set_mon_vals no callback set 2024-09-17T18:19:07.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.400+0000 7fa38bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:19:07.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.400+0000 7fa38bfff640 10 monclient: dump: 2024-09-17T18:19:07.412 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:07.412 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:07.412 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:07.412 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:07.412 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:07.413 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:07.413 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:07.413 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:07.413 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:07.413 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:07.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.401+0000 7fa393784640 10 monclient: shutdown 2024-09-17T18:19:07.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.401+0000 7fa393784640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:07.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.401+0000 7fa393784640 1 -- 172.21.15.121:0/1739060698 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa38c0a3a70 msgr2=0x7fa38c09dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:07.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.401+0000 7fa393784640 1 --2- 172.21.15.121:0/1739060698 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa38c0a3a70 0x7fa38c09dc00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa37c0097d0 tx=0x7fa37c02a000 comp rx=0 tx=0).stop 2024-09-17T18:19:07.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.401+0000 7fa393784640 1 -- 172.21.15.121:0/1739060698 shutdown_connections 2024-09-17T18:19:07.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.401+0000 7fa393784640 1 --2- 172.21.15.121:0/1739060698 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa38c0a3a70 0x7fa38c09dc00 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:07.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.401+0000 7fa393784640 1 -- 172.21.15.121:0/1739060698 >> 172.21.15.121:0/1739060698 conn(0x7fa38c0a8690 msgr2=0x7fa38c0a4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:07.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.401+0000 7fa393784640 1 -- 172.21.15.121:0/1739060698 shutdown_connections 2024-09-17T18:19:07.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.401+0000 7fa393784640 1 -- 172.21.15.121:0/1739060698 wait complete. 2024-09-17T18:19:07.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.402+0000 7fa393784640 10 monclient: build_initial_monmap 2024-09-17T18:19:07.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.402+0000 7fa393784640 10 monclient: monmap: 2024-09-17T18:19:07.414 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:07.414 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:19:07.414 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:19:07.414 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:19:07.414 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:07.414 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:07.414 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:19:07.414 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:19:07.414 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:19:07.414 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:07.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.402+0000 7fa393784640 1 librados: starting msgr at 2024-09-17T18:19:07.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.402+0000 7fa393784640 1 librados: starting objecter 2024-09-17T18:19:07.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.402+0000 7fa393784640 1 Processor -- start 2024-09-17T18:19:07.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.402+0000 7fa393784640 1 -- start start 2024-09-17T18:19:07.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.402+0000 7fa393784640 1 librados: setting wanted keys 2024-09-17T18:19:07.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.402+0000 7fa393784640 1 librados: calling monclient init 2024-09-17T18:19:07.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.402+0000 7fa393784640 10 monclient: init 2024-09-17T18:19:07.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.402+0000 7fa393784640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:07.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.402+0000 7fa393784640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T18:19:07.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.402+0000 7fa393784640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa38c0a3a70 0x7fa38c120be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:07.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.402+0000 7fa393784640 10 monclient(hunting): picked mon.noname-b con 0x7fa38c0a3a70 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:07.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.402+0000 7fa393784640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa38c0a3e50 0x7fa38c138a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:07.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.402+0000 7fa393784640 10 monclient(hunting): picked mon.noname-c con 0x7fa38c0a3e50 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:07.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.402+0000 7fa393784640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa38c13dca0 0x7fa38c11e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:07.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.402+0000 7fa393784640 10 monclient(hunting): picked mon.noname-a con 0x7fa38c13dca0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:07.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.402+0000 7fa393784640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:07.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.402+0000 7fa393784640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa38c076d10 con 0x7fa38c13dca0 2024-09-17T18:19:07.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa393784640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:07.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa393784640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa38c1419c0 con 0x7fa38c0a3a70 2024-09-17T18:19:07.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa393784640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:07.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa393784640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa38c141b40 con 0x7fa38c0a3e50 2024-09-17T18:19:07.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa393784640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:07.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa393784640 10 monclient(hunting): authenticate will time out at 751.119024s 2024-09-17T18:19:07.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa391d21640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa38c13dca0 0x7fa38c11e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:07.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa390d1f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa38c0a3e50 0x7fa38c138a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:07.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa391d21640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa38c13dca0 0x7fa38c11e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41186/0 (socket says 172.21.15.121:41186) 2024-09-17T18:19:07.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa391d21640 1 -- 172.21.15.121:0/1636797898 learned_addr learned my addr 172.21.15.121:0/1636797898 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:07.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa390d1f640 10 monclient(hunting): get_auth_request con 0x7fa38c0a3e50 auth_method 0 2024-09-17T18:19:07.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa390d1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:07.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa390d1f640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:07.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa390d1f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:07.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa391d21640 10 monclient(hunting): get_auth_request con 0x7fa38c13dca0 auth_method 0 2024-09-17T18:19:07.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa391d21640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:07.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa391d21640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:07.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa391d21640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:07.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa390d1f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:07.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa390d1f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:07.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa390d1f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:07.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa391d21640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:07.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa391d21640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:07.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa391520640 1 --2- 172.21.15.121:0/1636797898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa38c0a3a70 0x7fa38c120be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:07.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa391d21640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:07.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa391520640 10 monclient(hunting): get_auth_request con 0x7fa38c0a3a70 auth_method 0 2024-09-17T18:19:07.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa391520640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:07.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa391520640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:07.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa391520640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:07.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa391520640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:07.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa391520640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:07.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa391520640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:07.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa390d1f640 10 monclient(hunting): handle_auth_done global_id 4322 payload 824 2024-09-17T18:19:07.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa390d1f640 1 -- 172.21.15.121:0/1636797898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa38c0a3a70 msgr2=0x7fa38c120be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:07.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa390d1f640 1 --2- 172.21.15.121:0/1636797898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa38c0a3a70 0x7fa38c120be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:07.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa390d1f640 1 -- 172.21.15.121:0/1636797898 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa38c13dca0 msgr2=0x7fa38c11e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:07.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa390d1f640 1 --2- 172.21.15.121:0/1636797898 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa38c13dca0 0x7fa38c11e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:07.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa390d1f640 10 monclient: _finish_hunting 0 2024-09-17T18:19:07.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa390d1f640 1 monclient: found mon.noname-c 2024-09-17T18:19:07.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa390d1f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:07.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa390d1f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:19:07.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa390d1f640 1 -- 172.21.15.121:0/1636797898 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa38c1cea60 con 0x7fa38c0a3e50 2024-09-17T18:19:07.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.403+0000 7fa391d21640 1 --2- 172.21.15.121:0/1636797898 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa38c13dca0 0x7fa38c11e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:19:07.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.405+0000 7fa390d1f640 1 -- 172.21.15.121:0/1636797898 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa38c0a3e50 msgr2=0x7fa38c138a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:07.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.405+0000 7fa390d1f640 1 --2- 172.21.15.121:0/1636797898 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa38c0a3e50 0x7fa38c138a40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa38000c780 tx=0x7fa38000cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:07.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.405+0000 7fa389ffb640 10 client.?.objecter ms_handle_connect 0x7fa38c0a3e50 2024-09-17T18:19:07.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.405+0000 7fa389ffb640 10 client.?.objecter resend_mon_ops 2024-09-17T18:19:07.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.405+0000 7fa389ffb640 1 -- 172.21.15.121:0/1636797898 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa380016020 con 0x7fa38c0a3e50 2024-09-17T18:19:07.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.405+0000 7fa389ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:07.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.405+0000 7fa389ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:19:07.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.405+0000 7fa389ffb640 10 monclient: dump: 2024-09-17T18:19:07.420 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:07.420 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:07.420 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:07.420 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:07.420 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:07.420 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:07.420 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:07.420 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:07.420 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:07.420 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:07.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.405+0000 7fa389ffb640 10 monclient: _finish_auth 0 2024-09-17T18:19:07.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.405+0000 7fa389ffb640 10 monclient: _check_auth_tickets 2024-09-17T18:19:07.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.405+0000 7fa389ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:07.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.405+0000 7fa389ffb640 1 -- 172.21.15.121:0/1636797898 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa38000aca0 con 0x7fa38c0a3e50 2024-09-17T18:19:07.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.405+0000 7fa393784640 5 monclient: authenticate success, global_id 4322 2024-09-17T18:19:07.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.405+0000 7fa393784640 10 monclient: _renew_subs 2024-09-17T18:19:07.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.405+0000 7fa393784640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:07.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.405+0000 7fa393784640 1 -- 172.21.15.121:0/1636797898 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa38c1223e0 con 0x7fa38c0a3e50 2024-09-17T18:19:07.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.405+0000 7fa389ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:07.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.405+0000 7fa389ffb640 1 -- 172.21.15.121:0/1636797898 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa380012660 con 0x7fa38c0a3e50 2024-09-17T18:19:07.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.405+0000 7fa389ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:07.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.405+0000 7fa389ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:19:07.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.405+0000 7fa389ffb640 10 monclient: dump: 2024-09-17T18:19:07.421 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:07.421 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:07.421 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:07.421 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:07.421 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:07.422 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:07.422 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:07.422 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:07.422 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:07.422 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:07.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.405+0000 7fa38b7fe640 4 set_mon_vals no callback set 2024-09-17T18:19:07.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.406+0000 7fa393784640 10 client.4322.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:07.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.406+0000 7fa393784640 10 monclient: _renew_subs 2024-09-17T18:19:07.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.406+0000 7fa393784640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:07.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.406+0000 7fa393784640 1 -- 172.21.15.121:0/1636797898 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa38c120080 con 0x7fa38c0a3e50 2024-09-17T18:19:07.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.406+0000 7fa393784640 1 librados: init done 2024-09-17T18:19:07.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.406+0000 7fa389ffb640 1 -- 172.21.15.121:0/1636797898 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fa380019020 con 0x7fa38c0a3e50 2024-09-17T18:19:07.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.407+0000 7fa393784640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:07.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.407+0000 7fa389ffb640 1 --2- 172.21.15.121:0/1636797898 >> v2:172.21.15.173:6812/2423786037 conn(0x7fa3780401a0 0x7fa378042660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:07.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.407+0000 7fa393784640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:07.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.407+0000 7fa393784640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:07.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.407+0000 7fa393784640 1 -- 172.21.15.121:0/1636797898 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa38c09dc00 con 0x7fa38c0a3e50 2024-09-17T18:19:07.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.407+0000 7fa389ffb640 1 -- 172.21.15.121:0/1636797898 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(17..17 src has 1..17) ==== 3534+0+0 (secure 0 0 0) 0x7fa380052ec0 con 0x7fa38c0a3e50 2024-09-17T18:19:07.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.407+0000 7fa389ffb640 10 client.4322.objecter ms_dispatch 0x7fa38c000b90 osd_map(17..17 src has 1..17) 2024-09-17T18:19:07.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.407+0000 7fa389ffb640 3 client.4322.objecter handle_osd_map got epochs [17,17] > 0 2024-09-17T18:19:07.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.407+0000 7fa389ffb640 3 client.4322.objecter handle_osd_map decoding full epoch 17 2024-09-17T18:19:07.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.407+0000 7fa389ffb640 20 client.4322.objecter dump_active .. 0 homeless 2024-09-17T18:19:07.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.407+0000 7fa391520640 1 --2- 172.21.15.121:0/1636797898 >> v2:172.21.15.173:6812/2423786037 conn(0x7fa3780401a0 0x7fa378042660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:07.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.407+0000 7fa391520640 10 monclient: get_auth_request con 0x7fa3780401a0 auth_method 0 2024-09-17T18:19:07.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.410+0000 7fa391520640 1 --2- 172.21.15.121:0/1636797898 >> v2:172.21.15.173:6812/2423786037 conn(0x7fa3780401a0 0x7fa378042660 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa37c0097a0 tx=0x7fa37c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:07.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.410+0000 7fa389ffb640 10 client.4322.objecter ms_handle_connect 0x7fa3780401a0 2024-09-17T18:19:07.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.411+0000 7fa389ffb640 1 -- 172.21.15.121:0/1636797898 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fa380020030 con 0x7fa38c0a3e50 2024-09-17T18:19:07.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.411+0000 7fa389ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:07.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.411+0000 7fa389ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:19:07.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.588+0000 7fa393784640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-17T18:19:07.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.588+0000 7fa393784640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-17T18:19:07.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.588+0000 7fa393784640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:07.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.588+0000 7fa393784640 1 -- 172.21.15.121:0/1636797898 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) -- 0x7fa38c09f360 con 0x7fa38c0a3e50 2024-09-17T18:19:07.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.588+0000 7fa389ffb640 1 -- 172.21.15.121:0/1636797898 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v17) ==== 92+0+10 (secure 0 0 0) 0x7fa380051da0 con 0x7fa38c0a3e50 2024-09-17T18:19:07.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.588+0000 7fa389ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-17T18:19:07.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.588+0000 7fa389ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:19:07.591 INFO:teuthology.orchestra.run.smithi121.stdout:pg_num: 8 2024-09-17T18:19:07.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.591+0000 7fa393784640 10 librados: watch_flush enter 2024-09-17T18:19:07.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.591+0000 7fa393784640 10 librados: watch_flush exit 2024-09-17T18:19:07.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.591+0000 7fa393784640 20 client.4322.objecter shutdown clearing up homeless session... 2024-09-17T18:19:07.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.591+0000 7fa393784640 10 client.4322.objecter successfully canceled tick 2024-09-17T18:19:07.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.591+0000 7fa393784640 1 -- 172.21.15.121:0/1636797898 >> v2:172.21.15.173:6812/2423786037 conn(0x7fa3780401a0 msgr2=0x7fa378042660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:07.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.591+0000 7fa393784640 1 --2- 172.21.15.121:0/1636797898 >> v2:172.21.15.173:6812/2423786037 conn(0x7fa3780401a0 0x7fa378042660 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa37c0097a0 tx=0x7fa37c02e000 comp rx=0 tx=0).stop 2024-09-17T18:19:07.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.591+0000 7fa393784640 10 monclient: shutdown 2024-09-17T18:19:07.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.591+0000 7fa393784640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:07.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.591+0000 7fa393784640 1 -- 172.21.15.121:0/1636797898 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa38c0a3e50 msgr2=0x7fa38c138a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:07.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.591+0000 7fa393784640 1 --2- 172.21.15.121:0/1636797898 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa38c0a3e50 0x7fa38c138a40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa38000c780 tx=0x7fa38000cc40 comp rx=0 tx=0).stop 2024-09-17T18:19:07.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.591+0000 7fa393784640 1 -- 172.21.15.121:0/1636797898 shutdown_connections 2024-09-17T18:19:07.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.591+0000 7fa393784640 1 --2- 172.21.15.121:0/1636797898 >> v2:172.21.15.173:6812/2423786037 conn(0x7fa3780401a0 0x7fa378042660 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:07.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.591+0000 7fa393784640 1 --2- 172.21.15.121:0/1636797898 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa38c13dca0 0x7fa38c11e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:07.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.591+0000 7fa393784640 1 --2- 172.21.15.121:0/1636797898 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa38c0a3e50 0x7fa38c138a40 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:07.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.591+0000 7fa393784640 1 --2- 172.21.15.121:0/1636797898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa38c0a3a70 0x7fa38c120be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:07.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.591+0000 7fa393784640 1 -- 172.21.15.121:0/1636797898 >> 172.21.15.121:0/1636797898 conn(0x7fa38c0a8690 msgr2=0x7fa38c0a1360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:07.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.591+0000 7fa393784640 1 -- 172.21.15.121:0/1636797898 shutdown_connections 2024-09-17T18:19:07.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.591+0000 7fa393784640 1 -- 172.21.15.121:0/1636797898 wait complete. 2024-09-17T18:19:07.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.591+0000 7fa393784640 1 librados: shutdown 2024-09-17T18:19:07.603 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-09-17T18:19:07.603 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T18:19:07.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.698+0000 7f667a15a640 10 monclient: get_monmap_and_config 2024-09-17T18:19:07.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.698+0000 7f667a15a640 10 monclient: build_initial_monmap 2024-09-17T18:19:07.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.698+0000 7f667a15a640 10 monclient: monmap: 2024-09-17T18:19:07.700 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:07.700 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:07.700 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:19:07.698809+0000 2024-09-17T18:19:07.700 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:19:07.698809+0000 2024-09-17T18:19:07.700 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:07.700 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:07.700 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:19:07.701 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:19:07.701 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:19:07.701 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:19:07.701 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:19:07.701 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:07.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.699+0000 7f667a15a640 1 Processor -- start 2024-09-17T18:19:07.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.699+0000 7f667a15a640 1 -- start start 2024-09-17T18:19:07.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.699+0000 7f667a15a640 10 monclient: init 2024-09-17T18:19:07.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.699+0000 7f667a15a640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:07.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.699+0000 7f667a15a640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-17T18:19:07.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.699+0000 7f667a15a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66741386b0 0x7f66740707d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:07.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.699+0000 7f667a15a640 10 monclient(hunting): picked mon.noname-b con 0x7f66741386b0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:07.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.699+0000 7f667a15a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f667406ec00 addr v1:172.21.15.173:6789/0 2024-09-17T18:19:07.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.699+0000 7f667a15a640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f66741391a0 0x7f667413fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:07.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.699+0000 7f667a15a640 10 monclient(hunting): picked mon.noname-a con 0x7f66741391a0 addr v2:172.21.15.121:3300/0 2024-09-17T18:19:07.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.699+0000 7f667a15a640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66740a5570 con 0x7f667406ec00 2024-09-17T18:19:07.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.699+0000 7f667a15a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:07.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.699+0000 7f667a15a640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f667406f830 con 0x7f66741391a0 2024-09-17T18:19:07.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.699+0000 7f667a15a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:07.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.699+0000 7f667a15a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f667406fb10 con 0x7f66741386b0 2024-09-17T18:19:07.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.699+0000 7f667a15a640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:07.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.699+0000 7f667a15a640 10 monclient(hunting): authenticate will time out at 751.415662s 2024-09-17T18:19:07.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.699+0000 7f66737fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66741386b0 0x7f66740707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:07.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.699+0000 7f6673fff640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f66741391a0 0x7f667413fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:07.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.699+0000 7f66737fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66741386b0 0x7f66740707d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:59686/0 (socket says 172.21.15.121:59686) 2024-09-17T18:19:07.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.699+0000 7f6673fff640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f66741391a0 0x7f667413fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41194/0 (socket says 172.21.15.121:41194) 2024-09-17T18:19:07.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.699+0000 7f66737fe640 1 -- 172.21.15.121:0/3896279169 learned_addr learned my addr 172.21.15.121:0/3896279169 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:07.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.699+0000 7f66737fe640 10 monclient(hunting): get_auth_request con 0x7f66741386b0 auth_method 0 2024-09-17T18:19:07.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.699+0000 7f66737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:07.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.699+0000 7f66737fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:07.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.699+0000 7f66737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:07.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.699+0000 7f6673fff640 10 monclient(hunting): get_auth_request con 0x7f66741391a0 auth_method 0 2024-09-17T18:19:07.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.699+0000 7f6673fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:07.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.699+0000 7f6673fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:07.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.699+0000 7f6673fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:07.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.699+0000 7f66737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:07.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.699+0000 7f66737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:07.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.699+0000 7f66737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:07.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.699+0000 7f6673fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:07.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.699+0000 7f6673fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:07.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.699+0000 7f6673fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:07.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.700+0000 7f66737fe640 10 monclient(hunting): handle_auth_done global_id 4325 payload 274 2024-09-17T18:19:07.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.700+0000 7f66737fe640 1 -- 172.21.15.121:0/3896279169 >> v2:172.21.15.121:3300/0 conn(0x7f66741391a0 msgr2=0x7f667413fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:07.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.700+0000 7f66737fe640 1 --2- 172.21.15.121:0/3896279169 >> v2:172.21.15.121:3300/0 conn(0x7f66741391a0 0x7f667413fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:07.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.700+0000 7f66737fe640 1 -- 172.21.15.121:0/3896279169 >> v1:172.21.15.173:6789/0 conn(0x7f667406ec00 legacy=0x7f6674138a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:07.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.700+0000 7f66737fe640 10 monclient: _finish_hunting 0 2024-09-17T18:19:07.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.700+0000 7f66737fe640 1 monclient: found mon.noname-b 2024-09-17T18:19:07.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.700+0000 7f66737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:07.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.700+0000 7f66737fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:19:07.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.700+0000 7f66737fe640 1 -- 172.21.15.121:0/3896279169 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6674070d10 con 0x7f66741386b0 2024-09-17T18:19:07.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.700+0000 7f6673fff640 1 --2- 172.21.15.121:0/3896279169 >> v2:172.21.15.121:3300/0 conn(0x7f66741391a0 0x7f667413fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:19:07.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.702+0000 7f66737fe640 1 -- 172.21.15.121:0/3896279169 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66741386b0 msgr2=0x7f66740707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:07.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.702+0000 7f66737fe640 1 --2- 172.21.15.121:0/3896279169 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66741386b0 0x7f66740707d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6664009870 tx=0x7f666402a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad663d1870d3f1a3 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:07.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.703+0000 7f6672ffd640 1 -- 172.21.15.121:0/3896279169 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6664037070 con 0x7f66741386b0 2024-09-17T18:19:07.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.703+0000 7f6672ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:07.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.703+0000 7f6672ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:19:07.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.703+0000 7f6672ffd640 10 monclient: dump: 2024-09-17T18:19:07.714 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:07.714 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:07.714 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:07.714 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:07.714 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:07.714 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:07.714 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:07.714 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:07.714 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:07.714 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:07.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.703+0000 7f6672ffd640 10 monclient: _finish_auth 0 2024-09-17T18:19:07.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.703+0000 7f6672ffd640 10 monclient: _check_auth_tickets 2024-09-17T18:19:07.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.703+0000 7f6672ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:07.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.703+0000 7f6672ffd640 1 -- 172.21.15.121:0/3896279169 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6664032440 con 0x7f66741386b0 2024-09-17T18:19:07.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.703+0000 7f667a15a640 5 monclient: authenticate success, global_id 4325 2024-09-17T18:19:07.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.703+0000 7f667a15a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:19:07.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.703+0000 7f6672ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:07.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.703+0000 7f6672ffd640 1 -- 172.21.15.121:0/3896279169 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6664032810 con 0x7f66741386b0 2024-09-17T18:19:07.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.703+0000 7f667a15a640 10 monclient: get_monmap_and_config success 2024-09-17T18:19:07.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.703+0000 7f667a15a640 4 set_mon_vals no callback set 2024-09-17T18:19:07.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.703+0000 7f6672ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:07.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.703+0000 7f6672ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:19:07.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.703+0000 7f6672ffd640 10 monclient: dump: 2024-09-17T18:19:07.715 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:07.716 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:07.716 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:07.716 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:07.716 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:07.716 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:07.716 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:07.716 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:07.716 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:07.716 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:07.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.703+0000 7f667a15a640 10 monclient: shutdown 2024-09-17T18:19:07.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.703+0000 7f667a15a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:07.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.703+0000 7f667a15a640 1 -- 172.21.15.121:0/3896279169 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66741386b0 msgr2=0x7f66740707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:07.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.703+0000 7f667a15a640 1 --2- 172.21.15.121:0/3896279169 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66741386b0 0x7f66740707d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6664009870 tx=0x7f666402a0a0 comp rx=0 tx=0).stop 2024-09-17T18:19:07.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.703+0000 7f667a15a640 1 -- 172.21.15.121:0/3896279169 shutdown_connections 2024-09-17T18:19:07.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.703+0000 7f667a15a640 1 --2- 172.21.15.121:0/3896279169 >> v2:172.21.15.121:3300/0 conn(0x7f66741391a0 0x7f667413fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:07.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.703+0000 7f667a15a640 1 --2- 172.21.15.121:0/3896279169 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66741386b0 0x7f66740707d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:07.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.703+0000 7f667a15a640 1 -- 172.21.15.121:0/3896279169 >> 172.21.15.121:0/3896279169 conn(0x7f66740a91e0 msgr2=0x7f66740a95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:07.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.703+0000 7f667a15a640 1 -- 172.21.15.121:0/3896279169 shutdown_connections 2024-09-17T18:19:07.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.703+0000 7f667a15a640 1 -- 172.21.15.121:0/3896279169 wait complete. 2024-09-17T18:19:07.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.704+0000 7f667a15a640 10 monclient: build_initial_monmap 2024-09-17T18:19:07.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.704+0000 7f667a15a640 10 monclient: monmap: 2024-09-17T18:19:07.717 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:07.717 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:19:07.717 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:19:07.717 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:19:07.717 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:07.717 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:07.717 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:19:07.717 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:19:07.718 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:19:07.718 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:07.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.704+0000 7f667a15a640 1 librados: starting msgr at 2024-09-17T18:19:07.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.704+0000 7f667a15a640 1 librados: starting objecter 2024-09-17T18:19:07.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.704+0000 7f667a15a640 1 Processor -- start 2024-09-17T18:19:07.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.704+0000 7f667a15a640 1 -- start start 2024-09-17T18:19:07.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.704+0000 7f667a15a640 1 librados: setting wanted keys 2024-09-17T18:19:07.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.704+0000 7f667a15a640 1 librados: calling monclient init 2024-09-17T18:19:07.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.704+0000 7f667a15a640 10 monclient: init 2024-09-17T18:19:07.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.704+0000 7f667a15a640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:07.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.704+0000 7f667a15a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T18:19:07.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f667a15a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f667406ec00 0x7f667412f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:07.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f667a15a640 10 monclient(hunting): picked mon.noname-c con 0x7f667406ec00 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:07.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f667a15a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f66741386b0 0x7f667412f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:07.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f667a15a640 10 monclient(hunting): picked mon.noname-b con 0x7f66741386b0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:07.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f667a15a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66741391a0 0x7f66741cd730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:07.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f667a15a640 10 monclient(hunting): picked mon.noname-a con 0x7f66741391a0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:07.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f667a15a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:07.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f667a15a640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f66741404c0 con 0x7f66741391a0 2024-09-17T18:19:07.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f667a15a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:07.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f667a15a640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6674140340 con 0x7f66741386b0 2024-09-17T18:19:07.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f667a15a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:07.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f667a15a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6674140640 con 0x7f667406ec00 2024-09-17T18:19:07.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f667a15a640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:07.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f667a15a640 10 monclient(hunting): authenticate will time out at 751.421048s 2024-09-17T18:19:07.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f66737fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f667406ec00 0x7f667412f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:07.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f66737fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f667406ec00 0x7f667412f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:59688/0 (socket says 172.21.15.121:59688) 2024-09-17T18:19:07.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f6673fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66741391a0 0x7f66741cd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:07.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f66737fe640 1 -- 172.21.15.121:0/2701437616 learned_addr learned my addr 172.21.15.121:0/2701437616 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:07.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f66737fe640 10 monclient(hunting): get_auth_request con 0x7f667406ec00 auth_method 0 2024-09-17T18:19:07.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f66737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:07.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f66737fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:07.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f66737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:07.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f6673fff640 10 monclient(hunting): get_auth_request con 0x7f66741391a0 auth_method 0 2024-09-17T18:19:07.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f6673fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:07.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f6673fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:07.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f6673fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:07.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f66737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:07.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f66737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:07.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f66737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:07.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f6673fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:07.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f6673fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:07.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f6673fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:07.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f66737fe640 10 monclient(hunting): handle_auth_done global_id 4328 payload 824 2024-09-17T18:19:07.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f66737fe640 1 -- 172.21.15.121:0/2701437616 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f66741386b0 msgr2=0x7f667412f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:07.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f66737fe640 1 --2- 172.21.15.121:0/2701437616 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f66741386b0 0x7f667412f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:07.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f66737fe640 1 -- 172.21.15.121:0/2701437616 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66741391a0 msgr2=0x7f66741cd730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:07.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f66737fe640 1 --2- 172.21.15.121:0/2701437616 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66741391a0 0x7f66741cd730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:07.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f66737fe640 10 monclient: _finish_hunting 0 2024-09-17T18:19:07.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f66737fe640 1 monclient: found mon.noname-c 2024-09-17T18:19:07.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f66737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:07.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f66737fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:19:07.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f66737fe640 1 -- 172.21.15.121:0/2701437616 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66740a8110 con 0x7f667406ec00 2024-09-17T18:19:07.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.705+0000 7f6673fff640 1 --2- 172.21.15.121:0/2701437616 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66741391a0 0x7f66741cd730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:19:07.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.707+0000 7f66737fe640 1 -- 172.21.15.121:0/2701437616 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f667406ec00 msgr2=0x7f667412f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:07.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.707+0000 7f66737fe640 1 --2- 172.21.15.121:0/2701437616 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f667406ec00 0x7f667412f040 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f66640090b0 tx=0x7f66640046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:07.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.707+0000 7f6670ff9640 10 client.?.objecter ms_handle_connect 0x7f667406ec00 2024-09-17T18:19:07.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.707+0000 7f6670ff9640 10 client.?.objecter resend_mon_ops 2024-09-17T18:19:07.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.707+0000 7f6670ff9640 1 -- 172.21.15.121:0/2701437616 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6664041020 con 0x7f667406ec00 2024-09-17T18:19:07.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.707+0000 7f6670ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:07.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.707+0000 7f6670ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:19:07.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.707+0000 7f6670ff9640 10 monclient: dump: 2024-09-17T18:19:07.722 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:07.722 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:07.722 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:07.722 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:07.722 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:07.722 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:07.722 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:07.722 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:07.722 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:07.722 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:07.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.707+0000 7f6670ff9640 10 monclient: _finish_auth 0 2024-09-17T18:19:07.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.707+0000 7f6670ff9640 10 monclient: _check_auth_tickets 2024-09-17T18:19:07.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.707+0000 7f6670ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:07.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.707+0000 7f6670ff9640 1 -- 172.21.15.121:0/2701437616 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f666403d630 con 0x7f667406ec00 2024-09-17T18:19:07.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.707+0000 7f667a15a640 5 monclient: authenticate success, global_id 4328 2024-09-17T18:19:07.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.707+0000 7f6670ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:07.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.707+0000 7f667a15a640 10 monclient: _renew_subs 2024-09-17T18:19:07.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.707+0000 7f6670ff9640 1 -- 172.21.15.121:0/2701437616 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6664037040 con 0x7f667406ec00 2024-09-17T18:19:07.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.707+0000 7f667a15a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:07.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.707+0000 7f667a15a640 1 -- 172.21.15.121:0/2701437616 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66741349f0 con 0x7f667406ec00 2024-09-17T18:19:07.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.707+0000 7f6670ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:07.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.707+0000 7f6670ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:19:07.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.707+0000 7f6670ff9640 10 monclient: dump: 2024-09-17T18:19:07.723 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:07.723 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:07.723 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:07.724 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:07.724 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:07.724 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:07.724 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:07.724 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:07.724 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:07.724 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:07.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.707+0000 7f66727fc640 4 set_mon_vals no callback set 2024-09-17T18:19:07.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.707+0000 7f667a15a640 10 client.4328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:07.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.707+0000 7f667a15a640 10 monclient: _renew_subs 2024-09-17T18:19:07.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.707+0000 7f667a15a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:07.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.707+0000 7f667a15a640 1 -- 172.21.15.121:0/2701437616 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f66741330f0 con 0x7f667406ec00 2024-09-17T18:19:07.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.707+0000 7f667a15a640 1 librados: init done 2024-09-17T18:19:07.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.708+0000 7f6670ff9640 1 -- 172.21.15.121:0/2701437616 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f6664044020 con 0x7f667406ec00 2024-09-17T18:19:07.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.708+0000 7f667a15a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:07.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.708+0000 7f667a15a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:07.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.708+0000 7f667a15a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:07.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.708+0000 7f667a15a640 1 -- 172.21.15.121:0/2701437616 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f667412fac0 con 0x7f667406ec00 2024-09-17T18:19:07.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.708+0000 7f6670ff9640 1 --2- 172.21.15.121:0/2701437616 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6654040170 0x7f6654042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:07.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.708+0000 7f6670ff9640 1 -- 172.21.15.121:0/2701437616 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(17..17 src has 1..17) ==== 3534+0+0 (secure 0 0 0) 0x7f6664073330 con 0x7f667406ec00 2024-09-17T18:19:07.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.708+0000 7f6670ff9640 10 client.4328.objecter ms_dispatch 0x7f6674000b90 osd_map(17..17 src has 1..17) 2024-09-17T18:19:07.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.708+0000 7f6670ff9640 3 client.4328.objecter handle_osd_map got epochs [17,17] > 0 2024-09-17T18:19:07.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.708+0000 7f6670ff9640 3 client.4328.objecter handle_osd_map decoding full epoch 17 2024-09-17T18:19:07.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.709+0000 7f6670ff9640 20 client.4328.objecter dump_active .. 0 homeless 2024-09-17T18:19:07.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.709+0000 7f666bfff640 1 --2- 172.21.15.121:0/2701437616 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6654040170 0x7f6654042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:07.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.709+0000 7f666bfff640 10 monclient: get_auth_request con 0x7f6654040170 auth_method 0 2024-09-17T18:19:07.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.711+0000 7f6670ff9640 10 client.4328.objecter ms_handle_connect 0x7f6654040170 2024-09-17T18:19:07.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.711+0000 7f666bfff640 1 --2- 172.21.15.121:0/2701437616 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6654040170 0x7f6654042630 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6674135f10 tx=0x7f665c0079b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:07.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.713+0000 7f6670ff9640 1 -- 172.21.15.121:0/2701437616 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f666404b030 con 0x7f667406ec00 2024-09-17T18:19:07.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.713+0000 7f6670ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:07.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.713+0000 7f6670ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:19:07.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.884+0000 7f667a15a640 1 -- 172.21.15.121:0/2701437616 --> v2:172.21.15.173:6812/2423786037 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f66740707d0 con 0x7f6654040170 2024-09-17T18:19:07.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.885+0000 7f6670ff9640 1 -- 172.21.15.121:0/2701437616 <== mgr.4105 v2:172.21.15.173:6812/2423786037 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34177 (secure 0 0 0) 0x7f66740707d0 con 0x7f6654040170 2024-09-17T18:19:07.888 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:19:07.888 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-09-17T18:19:07.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.887+0000 7f667a15a640 10 librados: watch_flush enter 2024-09-17T18:19:07.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.887+0000 7f667a15a640 10 librados: watch_flush exit 2024-09-17T18:19:07.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.887+0000 7f667a15a640 20 client.4328.objecter shutdown clearing up homeless session... 2024-09-17T18:19:07.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.887+0000 7f667a15a640 10 client.4328.objecter successfully canceled tick 2024-09-17T18:19:07.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.887+0000 7f667a15a640 1 -- 172.21.15.121:0/2701437616 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6654040170 msgr2=0x7f6654042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:07.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.887+0000 7f667a15a640 1 --2- 172.21.15.121:0/2701437616 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6654040170 0x7f6654042630 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6674135f10 tx=0x7f665c0079b0 comp rx=0 tx=0).stop 2024-09-17T18:19:07.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.887+0000 7f667a15a640 10 monclient: shutdown 2024-09-17T18:19:07.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.887+0000 7f667a15a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:07.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.887+0000 7f667a15a640 1 -- 172.21.15.121:0/2701437616 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f667406ec00 msgr2=0x7f667412f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:07.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.888+0000 7f667a15a640 1 --2- 172.21.15.121:0/2701437616 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f667406ec00 0x7f667412f040 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f66640090b0 tx=0x7f66640046c0 comp rx=0 tx=0).stop 2024-09-17T18:19:07.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.888+0000 7f667a15a640 1 -- 172.21.15.121:0/2701437616 shutdown_connections 2024-09-17T18:19:07.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.888+0000 7f667a15a640 1 --2- 172.21.15.121:0/2701437616 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6654040170 0x7f6654042630 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:07.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.888+0000 7f667a15a640 1 --2- 172.21.15.121:0/2701437616 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f66741386b0 0x7f667412f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:07.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.888+0000 7f667a15a640 1 --2- 172.21.15.121:0/2701437616 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66741391a0 0x7f66741cd730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:07.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.888+0000 7f667a15a640 1 --2- 172.21.15.121:0/2701437616 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f667406ec00 0x7f667412f040 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:07.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.888+0000 7f667a15a640 1 -- 172.21.15.121:0/2701437616 >> 172.21.15.121:0/2701437616 conn(0x7f66740a91e0 msgr2=0x7f6674132040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:07.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.888+0000 7f667a15a640 1 -- 172.21.15.121:0/2701437616 shutdown_connections 2024-09-17T18:19:07.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.888+0000 7f667a15a640 1 -- 172.21.15.121:0/2701437616 wait complete. 2024-09-17T18:19:07.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.888+0000 7f667a15a640 1 librados: shutdown 2024-09-17T18:19:07.900 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-09-17T18:19:07.496819+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163136,"kb_used_data":944,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067904,"statfs":{"total":579820584960,"available":579653533696,"internally_reserved":0,"allocated":966656,"data_stored":1313332,"data_compressed":59128,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":165927585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.970849"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T18:19:01.527478+0000","last_change":"2024-09-17T18:19:01.527658+0000","last_active":"2024-09-17T18:19:01.527478+0000","last_peered":"2024-09-17T18:19:01.527478+0000","last_clean":"2024-09-17T18:19:01.527478+0000","last_became_active":"2024-09-17T18:19:00.056752+0000","last_became_peered":"2024-09-17T18:19:00.056752+0000","last_unstale":"2024-09-17T18:19:01.527478+0000","last_undegraded":"2024-09-17T18:19:01.527478+0000","last_fullsized":"2024-09-17T18:19:01.527478+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:57:47.271964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049856399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T18:19:01.527858+0000","last_change":"2024-09-17T18:19:01.527858+0000","last_active":"2024-09-17T18:19:01.527858+0000","last_peered":"2024-09-17T18:19:01.527858+0000","last_clean":"2024-09-17T18:19:01.527858+0000","last_became_active":"2024-09-17T18:18:59.525295+0000","last_became_peered":"2024-09-17T18:18:59.525295+0000","last_unstale":"2024-09-17T18:19:01.527858+0000","last_undegraded":"2024-09-17T18:19:01.527858+0000","last_fullsized":"2024-09-17T18:19:01.527858+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:41:21.181263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062121300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T18:19:01.527332+0000","last_change":"2024-09-17T18:19:01.527563+0000","last_active":"2024-09-17T18:19:01.527332+0000","last_peered":"2024-09-17T18:19:01.527332+0000","last_clean":"2024-09-17T18:19:01.527332+0000","last_became_active":"2024-09-17T18:19:00.244851+0000","last_became_peered":"2024-09-17T18:19:00.244851+0000","last_unstale":"2024-09-17T18:19:01.527332+0000","last_undegraded":"2024-09-17T18:19:01.527332+0000","last_fullsized":"2024-09-17T18:19:01.527332+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:19:57.036107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041282100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T18:19:01.527784+0000","last_change":"2024-09-17T18:19:01.527784+0000","last_active":"2024-09-17T18:19:01.527784+0000","last_peered":"2024-09-17T18:19:01.527784+0000","last_clean":"2024-09-17T18:19:01.527784+0000","last_became_active":"2024-09-17T18:19:00.244613+0000","last_became_peered":"2024-09-17T18:19:00.244613+0000","last_unstale":"2024-09-17T18:19:01.527784+0000","last_undegraded":"2024-09-17T18:19:01.527784+0000","last_fullsized":"2024-09-17T18:19:01.527784+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:31:24.599660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063646500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":22,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T18:19:01.532623+0000","last_change":"2024-09-17T18:19:01.532623+0000","last_active":"2024-09-17T18:19:01.532623+0000","last_peered":"2024-09-17T18:19:01.532623+0000","last_clean":"2024-09-17T18:19:01.532623+0000","last_became_active":"2024-09-17T18:18:59.526791+0000","last_became_peered":"2024-09-17T18:18:59.526791+0000","last_unstale":"2024-09-17T18:19:01.532623+0000","last_undegraded":"2024-09-17T18:19:01.532623+0000","last_fullsized":"2024-09-17T18:19:01.532623+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T21:46:24.217486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010088289999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T18:19:01.527597+0000","last_change":"2024-09-17T18:19:01.527848+0000","last_active":"2024-09-17T18:19:01.527597+0000","last_peered":"2024-09-17T18:19:01.527597+0000","last_clean":"2024-09-17T18:19:01.527597+0000","last_became_active":"2024-09-17T18:18:59.529390+0000","last_became_peered":"2024-09-17T18:18:59.529390+0000","last_unstale":"2024-09-17T18:19:01.527597+0000","last_undegraded":"2024-09-17T18:19:01.527597+0000","last_fullsized":"2024-09-17T18:19:01.527597+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:40:45.376247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065986199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T18:19:01.527258+0000","last_change":"2024-09-17T18:19:01.527425+0000","last_active":"2024-09-17T18:19:01.527258+0000","last_peered":"2024-09-17T18:19:01.527258+0000","last_clean":"2024-09-17T18:19:01.527258+0000","last_became_active":"2024-09-17T18:18:59.526855+0000","last_became_peered":"2024-09-17T18:18:59.526855+0000","last_unstale":"2024-09-17T18:19:01.527258+0000","last_undegraded":"2024-09-17T18:19:01.527258+0000","last_fullsized":"2024-09-17T18:19:01.527258+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:05:38.754621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032532500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T18:19:01.531845+0000","last_change":"2024-09-17T18:19:01.532053+0000","last_active":"2024-09-17T18:19:01.531845+0000","last_peered":"2024-09-17T18:19:01.531845+0000","last_clean":"2024-09-17T18:19:01.531845+0000","last_became_active":"2024-09-17T18:18:59.526797+0000","last_became_peered":"2024-09-17T18:18:59.526797+0000","last_unstale":"2024-09-17T18:19:01.531845+0000","last_undegraded":"2024-09-17T18:19:01.531845+0000","last_fullsized":"2024-09-17T18:19:01.531845+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:19:28.196155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000425759,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"15'72","reported_seq":105,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T18:19:01.527033+0000","last_change":"2024-09-17T18:19:00.245854+0000","last_active":"2024-09-17T18:19:01.527033+0000","last_peered":"2024-09-17T18:19:01.527033+0000","last_clean":"2024-09-17T18:19:01.527033+0000","last_became_active":"2024-09-17T18:19:00.244903+0000","last_became_peered":"2024-09-17T18:19:00.244903+0000","last_unstale":"2024-09-17T18:19:01.527033+0000","last_undegraded":"2024-09-17T18:19:01.527033+0000","last_fullsized":"2024-09-17T18:19:01.527033+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:55:38.336150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59128,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607557,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":139264,"data_stored":42573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607556,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":143360,"data_stored":42592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":143360,"data_stored":42592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":200704,"data_stored":571501,"data_compressed":29564,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":139264,"data_stored":42573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":200704,"data_stored":571501,"data_compressed":29564,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29564,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29564,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T18:19:07.901 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T18:19:07.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.995+0000 7fbda1982640 10 monclient: get_monmap_and_config 2024-09-17T18:19:07.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.995+0000 7fbda1982640 10 monclient: build_initial_monmap 2024-09-17T18:19:07.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.995+0000 7fbda1982640 10 monclient: monmap: 2024-09-17T18:19:07.997 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:07.998 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:07.998 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:19:07.996450+0000 2024-09-17T18:19:07.998 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:19:07.996450+0000 2024-09-17T18:19:07.998 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:07.998 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:07.998 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:19:07.998 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:19:07.998 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:19:07.998 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:19:07.998 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:19:07.998 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:08.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.996+0000 7fbda1982640 1 Processor -- start 2024-09-17T18:19:08.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.996+0000 7fbda1982640 1 -- start start 2024-09-17T18:19:08.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.996+0000 7fbda1982640 10 monclient: init 2024-09-17T18:19:08.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.996+0000 7fbda1982640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:08.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.996+0000 7fbda1982640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-17T18:19:08.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.996+0000 7fbda1982640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbd9c1386b0 addr v1:172.21.15.173:6789/0 2024-09-17T18:19:08.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.996+0000 7fbda1982640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbd9c06ec00 0x7fbd9c138a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:08.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.996+0000 7fbda1982640 10 monclient(hunting): picked mon.noname-b con 0x7fbd9c06ec00 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:08.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.996+0000 7fbda1982640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fbd9c138fd0 0x7fbd9c13fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:08.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.996+0000 7fbda1982640 10 monclient(hunting): picked mon.noname-c con 0x7fbd9c138fd0 addr v2:172.21.15.173:3300/0 2024-09-17T18:19:08.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.997+0000 7fbda1982640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd9c0a5570 con 0x7fbd9c1386b0 2024-09-17T18:19:08.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.997+0000 7fbda1982640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:08.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.997+0000 7fbda1982640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fbd9c06f830 con 0x7fbd9c138fd0 2024-09-17T18:19:08.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.997+0000 7fbda1982640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:08.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.997+0000 7fbda1982640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbd9c06fb10 con 0x7fbd9c06ec00 2024-09-17T18:19:08.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.997+0000 7fbda1982640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:08.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.997+0000 7fbda1982640 10 monclient(hunting): authenticate will time out at 751.713030s 2024-09-17T18:19:08.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.997+0000 7fbd9a7fc640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbd9c06ec00 0x7fbd9c138a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:08.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.997+0000 7fbd9a7fc640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbd9c06ec00 0x7fbd9c138a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:59702/0 (socket says 172.21.15.121:59702) 2024-09-17T18:19:08.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.997+0000 7fbd9a7fc640 1 -- 172.21.15.121:0/2895830066 learned_addr learned my addr 172.21.15.121:0/2895830066 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:08.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.997+0000 7fbd9a7fc640 10 monclient(hunting): get_auth_request con 0x7fbd9c06ec00 auth_method 0 2024-09-17T18:19:08.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.997+0000 7fbd9a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:08.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.997+0000 7fbd9a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:08.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.997+0000 7fbd9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:08.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.997+0000 7fbd9a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:08.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.997+0000 7fbd9a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:08.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.997+0000 7fbd9a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:08.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.997+0000 7fbd9b7fe640 1 --2- 172.21.15.121:0/2895830066 >> v2:172.21.15.173:3300/0 conn(0x7fbd9c138fd0 0x7fbd9c13fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:08.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.997+0000 7fbd9b7fe640 10 monclient(hunting): get_auth_request con 0x7fbd9c138fd0 auth_method 0 2024-09-17T18:19:08.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.997+0000 7fbd9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:08.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.997+0000 7fbd9b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:08.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.997+0000 7fbd9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:08.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.997+0000 7fbd9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:08.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.997+0000 7fbd9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:08.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.997+0000 7fbd9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:08.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.997+0000 7fbd9a7fc640 10 monclient(hunting): handle_auth_done global_id 4331 payload 274 2024-09-17T18:19:08.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.997+0000 7fbd9a7fc640 1 -- 172.21.15.121:0/2895830066 >> v2:172.21.15.173:3300/0 conn(0x7fbd9c138fd0 msgr2=0x7fbd9c13fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:08.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.997+0000 7fbd9a7fc640 1 --2- 172.21.15.121:0/2895830066 >> v2:172.21.15.173:3300/0 conn(0x7fbd9c138fd0 0x7fbd9c13fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:08.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.997+0000 7fbd9a7fc640 1 -- 172.21.15.121:0/2895830066 >> v1:172.21.15.173:6789/0 conn(0x7fbd9c1386b0 legacy=0x7fbd9c0707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:08.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.997+0000 7fbd9a7fc640 10 monclient: _finish_hunting 0 2024-09-17T18:19:08.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.997+0000 7fbd9a7fc640 1 monclient: found mon.noname-b 2024-09-17T18:19:08.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.997+0000 7fbd9a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:08.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.997+0000 7fbd9a7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:19:08.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:07.997+0000 7fbd9a7fc640 1 -- 172.21.15.121:0/2895830066 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd9c139420 con 0x7fbd9c06ec00 2024-09-17T18:19:08.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.000+0000 7fbd9a7fc640 1 -- 172.21.15.121:0/2895830066 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbd9c06ec00 msgr2=0x7fbd9c138a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:08.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.000+0000 7fbd9a7fc640 1 --2- 172.21.15.121:0/2895830066 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbd9c06ec00 0x7fbd9c138a90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbd90009870 tx=0x7fbd9002a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7e7be31bf5460e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:08.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.000+0000 7fbd99ffb640 1 -- 172.21.15.121:0/2895830066 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd90037070 con 0x7fbd9c06ec00 2024-09-17T18:19:08.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.000+0000 7fbd99ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:08.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.000+0000 7fbd99ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:19:08.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.000+0000 7fbd99ffb640 10 monclient: dump: 2024-09-17T18:19:08.011 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:08.011 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:08.011 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.011 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.011 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:08.011 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:08.011 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:08.011 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:08.011 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:08.011 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:08.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.000+0000 7fbd99ffb640 10 monclient: _finish_auth 0 2024-09-17T18:19:08.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.000+0000 7fbd99ffb640 10 monclient: _check_auth_tickets 2024-09-17T18:19:08.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.000+0000 7fbd99ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:08.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.000+0000 7fbd99ffb640 1 -- 172.21.15.121:0/2895830066 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd90032640 con 0x7fbd9c06ec00 2024-09-17T18:19:08.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.000+0000 7fbda1982640 5 monclient: authenticate success, global_id 4331 2024-09-17T18:19:08.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.000+0000 7fbda1982640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:19:08.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.000+0000 7fbd99ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:08.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.000+0000 7fbda1982640 10 monclient: get_monmap_and_config success 2024-09-17T18:19:08.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.000+0000 7fbd99ffb640 1 -- 172.21.15.121:0/2895830066 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd90032a10 con 0x7fbd9c06ec00 2024-09-17T18:19:08.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.000+0000 7fbd99ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:08.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.000+0000 7fbda1982640 4 set_mon_vals no callback set 2024-09-17T18:19:08.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.000+0000 7fbd99ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:19:08.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.000+0000 7fbd99ffb640 10 monclient: dump: 2024-09-17T18:19:08.012 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:08.012 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:08.013 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.013 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.013 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:08.013 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:08.013 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:08.013 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:08.013 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:08.013 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:08.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.000+0000 7fbda1982640 10 monclient: shutdown 2024-09-17T18:19:08.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.000+0000 7fbda1982640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:08.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.000+0000 7fbda1982640 1 -- 172.21.15.121:0/2895830066 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbd9c06ec00 msgr2=0x7fbd9c138a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:08.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.000+0000 7fbda1982640 1 --2- 172.21.15.121:0/2895830066 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbd9c06ec00 0x7fbd9c138a90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbd90009870 tx=0x7fbd9002a130 comp rx=0 tx=0).stop 2024-09-17T18:19:08.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.000+0000 7fbda1982640 1 -- 172.21.15.121:0/2895830066 shutdown_connections 2024-09-17T18:19:08.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.000+0000 7fbda1982640 1 --2- 172.21.15.121:0/2895830066 >> v2:172.21.15.173:3300/0 conn(0x7fbd9c138fd0 0x7fbd9c13fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:08.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.000+0000 7fbda1982640 1 --2- 172.21.15.121:0/2895830066 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbd9c06ec00 0x7fbd9c138a90 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:08.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.000+0000 7fbda1982640 1 -- 172.21.15.121:0/2895830066 >> 172.21.15.121:0/2895830066 conn(0x7fbd9c0a91e0 msgr2=0x7fbd9c0a95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:08.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.000+0000 7fbda1982640 1 -- 172.21.15.121:0/2895830066 shutdown_connections 2024-09-17T18:19:08.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.000+0000 7fbda1982640 1 -- 172.21.15.121:0/2895830066 wait complete. 2024-09-17T18:19:08.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.001+0000 7fbda1982640 10 monclient: build_initial_monmap 2024-09-17T18:19:08.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.001+0000 7fbda1982640 10 monclient: monmap: 2024-09-17T18:19:08.014 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:08.014 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:19:08.014 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:19:08.014 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:19:08.014 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:08.014 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:08.014 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:19:08.014 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:19:08.015 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:19:08.015 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:08.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.001+0000 7fbda1982640 1 librados: starting msgr at 2024-09-17T18:19:08.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.001+0000 7fbda1982640 1 librados: starting objecter 2024-09-17T18:19:08.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.001+0000 7fbda1982640 1 Processor -- start 2024-09-17T18:19:08.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbda1982640 1 -- start start 2024-09-17T18:19:08.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbda1982640 1 librados: setting wanted keys 2024-09-17T18:19:08.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbda1982640 1 librados: calling monclient init 2024-09-17T18:19:08.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbda1982640 10 monclient: init 2024-09-17T18:19:08.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbda1982640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:08.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbda1982640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T18:19:08.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbda1982640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbd9c06ec00 0x7fbd9c0a2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:08.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbda1982640 10 monclient(hunting): picked mon.noname-b con 0x7fbd9c06ec00 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:08.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbda1982640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd9c1386b0 0x7fbd9c132170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:08.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbda1982640 10 monclient(hunting): picked mon.noname-a con 0x7fbd9c1386b0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:08.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbda1982640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbd9c138fd0 0x7fbd9c1326b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:08.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbda1982640 10 monclient(hunting): picked mon.noname-c con 0x7fbd9c138fd0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:08.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbda1982640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:08.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbda1982640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fbd9c1404c0 con 0x7fbd9c1386b0 2024-09-17T18:19:08.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbda1982640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:08.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbda1982640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbd9c140340 con 0x7fbd9c06ec00 2024-09-17T18:19:08.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbda1982640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:08.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbda1982640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbd9c140640 con 0x7fbd9c138fd0 2024-09-17T18:19:08.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbda1982640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:08.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbda1982640 10 monclient(hunting): authenticate will time out at 751.718450s 2024-09-17T18:19:08.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbd9b7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbd9c138fd0 0x7fbd9c1326b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:08.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbd9a7fc640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd9c1386b0 0x7fbd9c132170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:08.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbd9b7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbd9c138fd0 0x7fbd9c1326b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:59716/0 (socket says 172.21.15.121:59716) 2024-09-17T18:19:08.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbd9b7fe640 1 -- 172.21.15.121:0/3450778314 learned_addr learned my addr 172.21.15.121:0/3450778314 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:08.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbd9b7fe640 10 monclient(hunting): get_auth_request con 0x7fbd9c138fd0 auth_method 0 2024-09-17T18:19:08.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbd9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:08.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbd9b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:08.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbd9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:08.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbd9a7fc640 10 monclient(hunting): get_auth_request con 0x7fbd9c1386b0 auth_method 0 2024-09-17T18:19:08.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbd9a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:08.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbd9a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:08.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbd9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:08.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbd9affd640 1 --2- 172.21.15.121:0/3450778314 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbd9c06ec00 0x7fbd9c0a2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:08.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbd9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:08.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbd9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:08.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbd9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:08.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbd9a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:08.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbd9a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:08.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbd9a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:08.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbd9affd640 10 monclient(hunting): get_auth_request con 0x7fbd9c06ec00 auth_method 0 2024-09-17T18:19:08.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbd9affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:08.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbd9affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:08.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbd9affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:08.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbd9affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:08.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbd9affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:08.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbd9affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:08.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbd9b7fe640 10 monclient(hunting): handle_auth_done global_id 4334 payload 824 2024-09-17T18:19:08.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbd9b7fe640 1 -- 172.21.15.121:0/3450778314 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbd9c06ec00 msgr2=0x7fbd9c0a2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:08.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbd9b7fe640 1 --2- 172.21.15.121:0/3450778314 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbd9c06ec00 0x7fbd9c0a2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:08.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbd9b7fe640 1 -- 172.21.15.121:0/3450778314 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd9c1386b0 msgr2=0x7fbd9c132170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:08.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbd9b7fe640 1 --2- 172.21.15.121:0/3450778314 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd9c1386b0 0x7fbd9c132170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:08.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbd9b7fe640 10 monclient: _finish_hunting 0 2024-09-17T18:19:08.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbd9b7fe640 1 monclient: found mon.noname-c 2024-09-17T18:19:08.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbd9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:08.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbd9b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:19:08.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbd9b7fe640 1 -- 172.21.15.121:0/3450778314 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd9c132bf0 con 0x7fbd9c138fd0 2024-09-17T18:19:08.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.002+0000 7fbd9a7fc640 1 --2- 172.21.15.121:0/3450778314 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd9c1386b0 0x7fbd9c132170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:19:08.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.005+0000 7fbd9b7fe640 1 -- 172.21.15.121:0/3450778314 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbd9c138fd0 msgr2=0x7fbd9c1326b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:08.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.005+0000 7fbd9b7fe640 1 --2- 172.21.15.121:0/3450778314 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbd9c138fd0 0x7fbd9c1326b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbd9400efc0 tx=0x7fbd9400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:08.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.005+0000 7fbd83fff640 10 client.?.objecter ms_handle_connect 0x7fbd9c138fd0 2024-09-17T18:19:08.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.005+0000 7fbd83fff640 10 client.?.objecter resend_mon_ops 2024-09-17T18:19:08.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.005+0000 7fbd83fff640 1 -- 172.21.15.121:0/3450778314 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd94018020 con 0x7fbd9c138fd0 2024-09-17T18:19:08.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.005+0000 7fbd83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:08.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.005+0000 7fbd83fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:19:08.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.005+0000 7fbd83fff640 10 monclient: dump: 2024-09-17T18:19:08.020 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:08.020 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:08.020 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.020 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.020 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:08.020 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:08.020 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:08.020 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:08.020 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:08.020 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:08.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.005+0000 7fbd83fff640 10 monclient: _finish_auth 0 2024-09-17T18:19:08.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.005+0000 7fbd83fff640 10 monclient: _check_auth_tickets 2024-09-17T18:19:08.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.005+0000 7fbd83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:08.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.005+0000 7fbd83fff640 1 -- 172.21.15.121:0/3450778314 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd94014680 con 0x7fbd9c138fd0 2024-09-17T18:19:08.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.005+0000 7fbda1982640 5 monclient: authenticate success, global_id 4334 2024-09-17T18:19:08.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.005+0000 7fbda1982640 10 monclient: _renew_subs 2024-09-17T18:19:08.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.005+0000 7fbda1982640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:08.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.005+0000 7fbda1982640 1 -- 172.21.15.121:0/3450778314 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd9c0a8510 con 0x7fbd9c138fd0 2024-09-17T18:19:08.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.005+0000 7fbd83fff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:08.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.005+0000 7fbd83fff640 1 -- 172.21.15.121:0/3450778314 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd94014a70 con 0x7fbd9c138fd0 2024-09-17T18:19:08.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.005+0000 7fbd83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:08.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.005+0000 7fbd83fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:19:08.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.005+0000 7fbd83fff640 10 monclient: dump: 2024-09-17T18:19:08.021 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:08.021 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:08.021 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.022 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.022 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:08.022 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:08.022 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:08.022 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:08.022 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:08.022 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:08.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.005+0000 7fbda1982640 10 client.4334.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:08.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.005+0000 7fbda1982640 10 monclient: _renew_subs 2024-09-17T18:19:08.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.005+0000 7fbda1982640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:08.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.005+0000 7fbd997fa640 4 set_mon_vals no callback set 2024-09-17T18:19:08.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.005+0000 7fbda1982640 1 -- 172.21.15.121:0/3450778314 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd9c0a1ef0 con 0x7fbd9c138fd0 2024-09-17T18:19:08.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.005+0000 7fbda1982640 1 librados: init done 2024-09-17T18:19:08.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.006+0000 7fbd83fff640 1 -- 172.21.15.121:0/3450778314 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fbd94005d10 con 0x7fbd9c138fd0 2024-09-17T18:19:08.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.006+0000 7fbda1982640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:08.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.006+0000 7fbda1982640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:08.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.006+0000 7fbda1982640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:08.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.006+0000 7fbda1982640 1 -- 172.21.15.121:0/3450778314 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd9c138a90 con 0x7fbd9c138fd0 2024-09-17T18:19:08.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.006+0000 7fbd83fff640 1 --2- 172.21.15.121:0/3450778314 >> v2:172.21.15.173:6812/2423786037 conn(0x7fbd880401a0 0x7fbd88042660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:08.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.006+0000 7fbd83fff640 1 -- 172.21.15.121:0/3450778314 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(17..17 src has 1..17) ==== 3534+0+0 (secure 0 0 0) 0x7fbd94053f10 con 0x7fbd9c138fd0 2024-09-17T18:19:08.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.006+0000 7fbd83fff640 10 client.4334.objecter ms_dispatch 0x7fbd9c000b90 osd_map(17..17 src has 1..17) 2024-09-17T18:19:08.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.006+0000 7fbd83fff640 3 client.4334.objecter handle_osd_map got epochs [17,17] > 0 2024-09-17T18:19:08.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.006+0000 7fbd83fff640 3 client.4334.objecter handle_osd_map decoding full epoch 17 2024-09-17T18:19:08.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.006+0000 7fbd83fff640 20 client.4334.objecter dump_active .. 0 homeless 2024-09-17T18:19:08.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.006+0000 7fbd9affd640 1 --2- 172.21.15.121:0/3450778314 >> v2:172.21.15.173:6812/2423786037 conn(0x7fbd880401a0 0x7fbd88042660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:08.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.006+0000 7fbd9affd640 10 monclient: get_auth_request con 0x7fbd880401a0 auth_method 0 2024-09-17T18:19:08.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.009+0000 7fbd9affd640 1 --2- 172.21.15.121:0/3450778314 >> v2:172.21.15.173:6812/2423786037 conn(0x7fbd880401a0 0x7fbd88042660 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c13fe00 tx=0x7fbd8c007e30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:08.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.009+0000 7fbd83fff640 10 client.4334.objecter ms_handle_connect 0x7fbd880401a0 2024-09-17T18:19:08.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.010+0000 7fbd83fff640 1 -- 172.21.15.121:0/3450778314 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fbd94022030 con 0x7fbd9c138fd0 2024-09-17T18:19:08.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.010+0000 7fbd83fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:08.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.010+0000 7fbd83fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:19:08.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.183+0000 7fbda1982640 1 -- 172.21.15.121:0/3450778314 --> v2:172.21.15.173:6812/2423786037 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbd9c0a2230 con 0x7fbd880401a0 2024-09-17T18:19:08.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.185+0000 7fbd83fff640 1 -- 172.21.15.121:0/3450778314 <== mgr.4105 v2:172.21.15.173:6812/2423786037 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34177 (secure 0 0 0) 0x7fbd9c0a2230 con 0x7fbd880401a0 2024-09-17T18:19:08.187 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:19:08.187 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-09-17T18:19:08.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.186+0000 7fbda1982640 10 librados: watch_flush enter 2024-09-17T18:19:08.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.186+0000 7fbda1982640 10 librados: watch_flush exit 2024-09-17T18:19:08.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.186+0000 7fbda1982640 20 client.4334.objecter shutdown clearing up homeless session... 2024-09-17T18:19:08.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.186+0000 7fbda1982640 10 client.4334.objecter successfully canceled tick 2024-09-17T18:19:08.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.187+0000 7fbda1982640 1 -- 172.21.15.121:0/3450778314 >> v2:172.21.15.173:6812/2423786037 conn(0x7fbd880401a0 msgr2=0x7fbd88042660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:08.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.187+0000 7fbda1982640 1 --2- 172.21.15.121:0/3450778314 >> v2:172.21.15.173:6812/2423786037 conn(0x7fbd880401a0 0x7fbd88042660 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c13fe00 tx=0x7fbd8c007e30 comp rx=0 tx=0).stop 2024-09-17T18:19:08.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.187+0000 7fbda1982640 10 monclient: shutdown 2024-09-17T18:19:08.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.187+0000 7fbda1982640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:08.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.187+0000 7fbda1982640 1 -- 172.21.15.121:0/3450778314 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbd9c138fd0 msgr2=0x7fbd9c1326b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:08.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.187+0000 7fbda1982640 1 --2- 172.21.15.121:0/3450778314 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbd9c138fd0 0x7fbd9c1326b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbd9400efc0 tx=0x7fbd9400c490 comp rx=0 tx=0).stop 2024-09-17T18:19:08.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.187+0000 7fbda1982640 1 -- 172.21.15.121:0/3450778314 shutdown_connections 2024-09-17T18:19:08.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.187+0000 7fbda1982640 1 --2- 172.21.15.121:0/3450778314 >> v2:172.21.15.173:6812/2423786037 conn(0x7fbd880401a0 0x7fbd88042660 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:08.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.187+0000 7fbda1982640 1 --2- 172.21.15.121:0/3450778314 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbd9c138fd0 0x7fbd9c1326b0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:08.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.187+0000 7fbda1982640 1 --2- 172.21.15.121:0/3450778314 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd9c1386b0 0x7fbd9c132170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:08.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.187+0000 7fbda1982640 1 --2- 172.21.15.121:0/3450778314 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbd9c06ec00 0x7fbd9c0a2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:08.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.187+0000 7fbda1982640 1 -- 172.21.15.121:0/3450778314 >> 172.21.15.121:0/3450778314 conn(0x7fbd9c0a91e0 msgr2=0x7fbd9c074050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:08.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.187+0000 7fbda1982640 1 -- 172.21.15.121:0/3450778314 shutdown_connections 2024-09-17T18:19:08.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.187+0000 7fbda1982640 1 -- 172.21.15.121:0/3450778314 wait complete. 2024-09-17T18:19:08.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.187+0000 7fbda1982640 1 librados: shutdown 2024-09-17T18:19:08.199 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-09-17T18:19:07.496819+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163136,"kb_used_data":944,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067904,"statfs":{"total":579820584960,"available":579653533696,"internally_reserved":0,"allocated":966656,"data_stored":1313332,"data_compressed":59128,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":165927585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.970849"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T18:19:01.527478+0000","last_change":"2024-09-17T18:19:01.527658+0000","last_active":"2024-09-17T18:19:01.527478+0000","last_peered":"2024-09-17T18:19:01.527478+0000","last_clean":"2024-09-17T18:19:01.527478+0000","last_became_active":"2024-09-17T18:19:00.056752+0000","last_became_peered":"2024-09-17T18:19:00.056752+0000","last_unstale":"2024-09-17T18:19:01.527478+0000","last_undegraded":"2024-09-17T18:19:01.527478+0000","last_fullsized":"2024-09-17T18:19:01.527478+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:57:47.271964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049856399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T18:19:01.527858+0000","last_change":"2024-09-17T18:19:01.527858+0000","last_active":"2024-09-17T18:19:01.527858+0000","last_peered":"2024-09-17T18:19:01.527858+0000","last_clean":"2024-09-17T18:19:01.527858+0000","last_became_active":"2024-09-17T18:18:59.525295+0000","last_became_peered":"2024-09-17T18:18:59.525295+0000","last_unstale":"2024-09-17T18:19:01.527858+0000","last_undegraded":"2024-09-17T18:19:01.527858+0000","last_fullsized":"2024-09-17T18:19:01.527858+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:41:21.181263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062121300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T18:19:01.527332+0000","last_change":"2024-09-17T18:19:01.527563+0000","last_active":"2024-09-17T18:19:01.527332+0000","last_peered":"2024-09-17T18:19:01.527332+0000","last_clean":"2024-09-17T18:19:01.527332+0000","last_became_active":"2024-09-17T18:19:00.244851+0000","last_became_peered":"2024-09-17T18:19:00.244851+0000","last_unstale":"2024-09-17T18:19:01.527332+0000","last_undegraded":"2024-09-17T18:19:01.527332+0000","last_fullsized":"2024-09-17T18:19:01.527332+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:19:57.036107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041282100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T18:19:01.527784+0000","last_change":"2024-09-17T18:19:01.527784+0000","last_active":"2024-09-17T18:19:01.527784+0000","last_peered":"2024-09-17T18:19:01.527784+0000","last_clean":"2024-09-17T18:19:01.527784+0000","last_became_active":"2024-09-17T18:19:00.244613+0000","last_became_peered":"2024-09-17T18:19:00.244613+0000","last_unstale":"2024-09-17T18:19:01.527784+0000","last_undegraded":"2024-09-17T18:19:01.527784+0000","last_fullsized":"2024-09-17T18:19:01.527784+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:31:24.599660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063646500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":22,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T18:19:01.532623+0000","last_change":"2024-09-17T18:19:01.532623+0000","last_active":"2024-09-17T18:19:01.532623+0000","last_peered":"2024-09-17T18:19:01.532623+0000","last_clean":"2024-09-17T18:19:01.532623+0000","last_became_active":"2024-09-17T18:18:59.526791+0000","last_became_peered":"2024-09-17T18:18:59.526791+0000","last_unstale":"2024-09-17T18:19:01.532623+0000","last_undegraded":"2024-09-17T18:19:01.532623+0000","last_fullsized":"2024-09-17T18:19:01.532623+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T21:46:24.217486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010088289999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T18:19:01.527597+0000","last_change":"2024-09-17T18:19:01.527848+0000","last_active":"2024-09-17T18:19:01.527597+0000","last_peered":"2024-09-17T18:19:01.527597+0000","last_clean":"2024-09-17T18:19:01.527597+0000","last_became_active":"2024-09-17T18:18:59.529390+0000","last_became_peered":"2024-09-17T18:18:59.529390+0000","last_unstale":"2024-09-17T18:19:01.527597+0000","last_undegraded":"2024-09-17T18:19:01.527597+0000","last_fullsized":"2024-09-17T18:19:01.527597+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:40:45.376247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065986199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T18:19:01.527258+0000","last_change":"2024-09-17T18:19:01.527425+0000","last_active":"2024-09-17T18:19:01.527258+0000","last_peered":"2024-09-17T18:19:01.527258+0000","last_clean":"2024-09-17T18:19:01.527258+0000","last_became_active":"2024-09-17T18:18:59.526855+0000","last_became_peered":"2024-09-17T18:18:59.526855+0000","last_unstale":"2024-09-17T18:19:01.527258+0000","last_undegraded":"2024-09-17T18:19:01.527258+0000","last_fullsized":"2024-09-17T18:19:01.527258+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:05:38.754621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032532500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T18:19:01.531845+0000","last_change":"2024-09-17T18:19:01.532053+0000","last_active":"2024-09-17T18:19:01.531845+0000","last_peered":"2024-09-17T18:19:01.531845+0000","last_clean":"2024-09-17T18:19:01.531845+0000","last_became_active":"2024-09-17T18:18:59.526797+0000","last_became_peered":"2024-09-17T18:18:59.526797+0000","last_unstale":"2024-09-17T18:19:01.531845+0000","last_undegraded":"2024-09-17T18:19:01.531845+0000","last_fullsized":"2024-09-17T18:19:01.531845+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:19:28.196155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000425759,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"15'72","reported_seq":105,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T18:19:01.527033+0000","last_change":"2024-09-17T18:19:00.245854+0000","last_active":"2024-09-17T18:19:01.527033+0000","last_peered":"2024-09-17T18:19:01.527033+0000","last_clean":"2024-09-17T18:19:01.527033+0000","last_became_active":"2024-09-17T18:19:00.244903+0000","last_became_peered":"2024-09-17T18:19:00.244903+0000","last_unstale":"2024-09-17T18:19:01.527033+0000","last_undegraded":"2024-09-17T18:19:01.527033+0000","last_fullsized":"2024-09-17T18:19:01.527033+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:55:38.336150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59128,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607557,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":139264,"data_stored":42573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607556,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":143360,"data_stored":42592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":143360,"data_stored":42592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":200704,"data_stored":571501,"data_compressed":29564,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":139264,"data_stored":42573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":200704,"data_stored":571501,"data_compressed":29564,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29564,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29564,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T18:19:08.200 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-09-17T18:19:08.200 DEBUG:tasks.mon_thrash:Yielding 2024-09-17T18:19:08.200 INFO:teuthology.run_tasks:Running task workunit... 2024-09-17T18:19:08.215 INFO:tasks.workunit:Pulling workunits from ref 1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T18:19:08.215 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-17T18:19:08.215 DEBUG:teuthology.orchestra.run.smithi173:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-17T18:19:08.215 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-09-17T18:19:08.215 INFO:tasks.mon_thrash.mon_thrasher:seed: 1726597148, revive delay: 90.0, thrash delay: 1.0 thrash many: True, maintain quorum: True store thrash: True, probability: 50 freeze mon: prob 10.0 duration 15.0 2024-09-17T18:19:08.216 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T18:19:08.216 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:19:08.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T18:19:08.234 INFO:teuthology.orchestra.run.smithi173.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-17T18:19:08.235 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-17T18:19:08.292 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-17T18:19:08.292 DEBUG:teuthology.orchestra.run.smithi173:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-17T18:19:08.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.309+0000 7f6095b6b640 10 monclient: get_monmap_and_config 2024-09-17T18:19:08.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.309+0000 7f6095b6b640 10 monclient: build_initial_monmap 2024-09-17T18:19:08.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.309+0000 7f6095b6b640 10 monclient: monmap: 2024-09-17T18:19:08.311 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:08.311 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:08.312 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:19:08.310286+0000 2024-09-17T18:19:08.312 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:19:08.310286+0000 2024-09-17T18:19:08.312 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:08.312 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:08.312 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:19:08.312 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:19:08.312 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:19:08.312 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:19:08.312 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:19:08.312 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:08.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.310+0000 7f6095b6b640 1 Processor -- start 2024-09-17T18:19:08.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.310+0000 7f6095b6b640 1 -- start start 2024-09-17T18:19:08.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.310+0000 7f6095b6b640 10 monclient: init 2024-09-17T18:19:08.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.310+0000 7f6095b6b640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:08.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.310+0000 7f6095b6b640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-17T18:19:08.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.310+0000 7f6095b6b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f60900a0a70 addr v1:172.21.15.121:6789/0 2024-09-17T18:19:08.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.310+0000 7f6095b6b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f60901348f0 addr v1:172.21.15.173:6789/0 2024-09-17T18:19:08.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.310+0000 7f6095b6b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f609013b330 0x7f609013d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:08.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.310+0000 7f6095b6b640 10 monclient(hunting): picked mon.noname-b con 0x7f609013b330 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:08.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.310+0000 7f6095b6b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f609009f5b0 con 0x7f60900a0a70 2024-09-17T18:19:08.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.311+0000 7f6095b6b640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60900a1560 con 0x7f60901348f0 2024-09-17T18:19:08.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.311+0000 7f6095b6b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:08.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.311+0000 7f6095b6b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f609009ef80 con 0x7f609013b330 2024-09-17T18:19:08.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.311+0000 7f6095b6b640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:08.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.311+0000 7f6095b6b640 10 monclient(hunting): authenticate will time out at 752.027035s 2024-09-17T18:19:08.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.311+0000 7f608ffff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f609013b330 0x7f609013d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:08.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.311+0000 7f608f7fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f60900a0a70 0x7f60900a0e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:49672/0 (socket says 172.21.15.121:49672) 2024-09-17T18:19:08.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.311+0000 7f608f7fe640 1 -- 172.21.15.121:0/933614751 learned_addr learned my addr 172.21.15.121:0/933614751 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:19:08.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.311+0000 7f608ffff640 10 monclient(hunting): get_auth_request con 0x7f609013b330 auth_method 0 2024-09-17T18:19:08.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.311+0000 7f608ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:08.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.311+0000 7f608ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:08.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.311+0000 7f608ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:08.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.311+0000 7f608ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:08.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.311+0000 7f608ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:08.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.311+0000 7f608ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:08.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.311+0000 7f608ffff640 10 monclient(hunting): handle_auth_done global_id 4337 payload 274 2024-09-17T18:19:08.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.311+0000 7f608ffff640 1 -- 172.21.15.121:0/933614751 >> v1:172.21.15.173:6789/0 conn(0x7f60901348f0 legacy=0x7f6090139c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:08.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.311+0000 7f608ffff640 1 -- 172.21.15.121:0/933614751 >> v1:172.21.15.121:6789/0 conn(0x7f60900a0a70 legacy=0x7f60900a0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:08.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.311+0000 7f608ffff640 10 monclient: _finish_hunting 0 2024-09-17T18:19:08.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.311+0000 7f608ffff640 1 monclient: found mon.noname-b 2024-09-17T18:19:08.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.311+0000 7f608ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:08.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.311+0000 7f608ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:19:08.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.311+0000 7f608ffff640 1 -- 172.21.15.121:0/933614751 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f609009eca0 con 0x7f609013b330 2024-09-17T18:19:08.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.314+0000 7f608ffff640 1 -- 172.21.15.121:0/933614751 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f609013b330 msgr2=0x7f609013d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:08.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.314+0000 7f608ffff640 1 --2- 172.21.15.121:0/933614751 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f609013b330 0x7f609013d720 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6088009dc0 tx=0x7f608802a480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ab8aa27f8783fe server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:08.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.314+0000 7f608e7fc640 1 -- 172.21.15.121:0/933614751 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6088007a30 con 0x7f609013b330 2024-09-17T18:19:08.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.314+0000 7f608e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:08.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.314+0000 7f608e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:19:08.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.314+0000 7f608e7fc640 10 monclient: dump: 2024-09-17T18:19:08.323 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:08.323 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:08.323 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.323 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.323 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:08.323 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:08.323 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:08.324 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:08.324 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:08.324 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:08.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.314+0000 7f608e7fc640 10 monclient: _finish_auth 0 2024-09-17T18:19:08.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.314+0000 7f608e7fc640 10 monclient: _check_auth_tickets 2024-09-17T18:19:08.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.314+0000 7f608e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:08.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.314+0000 7f6095b6b640 5 monclient: authenticate success, global_id 4337 2024-09-17T18:19:08.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.314+0000 7f608e7fc640 1 -- 172.21.15.121:0/933614751 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6088007bd0 con 0x7f609013b330 2024-09-17T18:19:08.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.314+0000 7f6095b6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:19:08.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.314+0000 7f608e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:08.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.314+0000 7f6095b6b640 10 monclient: get_monmap_and_config success 2024-09-17T18:19:08.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.314+0000 7f608e7fc640 1 -- 172.21.15.121:0/933614751 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6088031400 con 0x7f609013b330 2024-09-17T18:19:08.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.314+0000 7f6095b6b640 4 set_mon_vals no callback set 2024-09-17T18:19:08.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.314+0000 7f608e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:08.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.314+0000 7f608e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:19:08.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.314+0000 7f608e7fc640 10 monclient: dump: 2024-09-17T18:19:08.325 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:08.325 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:08.325 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.325 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.325 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:08.325 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:08.325 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:08.325 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:08.325 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:08.325 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:08.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.314+0000 7f6095b6b640 10 monclient: shutdown 2024-09-17T18:19:08.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.314+0000 7f6095b6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:08.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.314+0000 7f6095b6b640 1 -- 172.21.15.121:0/933614751 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f609013b330 msgr2=0x7f609013d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:08.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.314+0000 7f6095b6b640 1 --2- 172.21.15.121:0/933614751 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f609013b330 0x7f609013d720 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6088009dc0 tx=0x7f608802a480 comp rx=0 tx=0).stop 2024-09-17T18:19:08.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.314+0000 7f6095b6b640 1 -- 172.21.15.121:0/933614751 shutdown_connections 2024-09-17T18:19:08.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.314+0000 7f6095b6b640 1 --2- 172.21.15.121:0/933614751 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f609013b330 0x7f609013d720 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:08.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.314+0000 7f6095b6b640 1 -- 172.21.15.121:0/933614751 >> 172.21.15.121:0/933614751 conn(0x7f60900a3b50 msgr2=0x7f60900a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:08.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.314+0000 7f6095b6b640 1 -- 172.21.15.121:0/933614751 shutdown_connections 2024-09-17T18:19:08.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.314+0000 7f6095b6b640 1 -- 172.21.15.121:0/933614751 wait complete. 2024-09-17T18:19:08.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.315+0000 7f6095b6b640 10 monclient: build_initial_monmap 2024-09-17T18:19:08.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.315+0000 7f6095b6b640 10 monclient: monmap: 2024-09-17T18:19:08.326 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:08.326 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:19:08.326 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:19:08.327 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:19:08.327 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:08.327 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:08.327 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:19:08.327 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:19:08.327 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:19:08.327 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:08.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.315+0000 7f6095b6b640 1 librados: starting msgr at 2024-09-17T18:19:08.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.315+0000 7f6095b6b640 1 librados: starting objecter 2024-09-17T18:19:08.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.315+0000 7f6095b6b640 1 Processor -- start 2024-09-17T18:19:08.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.315+0000 7f6095b6b640 1 -- start start 2024-09-17T18:19:08.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.315+0000 7f6095b6b640 1 librados: setting wanted keys 2024-09-17T18:19:08.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.315+0000 7f6095b6b640 1 librados: calling monclient init 2024-09-17T18:19:08.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.315+0000 7f6095b6b640 10 monclient: init 2024-09-17T18:19:08.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f6095b6b640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:08.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f6095b6b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:19:08.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f6095b6b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60900a0a70 0x7f6090133cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:08.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f6095b6b640 10 monclient(hunting): picked mon.noname-a con 0x7f60900a0a70 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:08.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f6095b6b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60901348f0 0x7f6090134200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:08.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f6095b6b640 10 monclient(hunting): picked mon.noname-b con 0x7f60901348f0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:08.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f6095b6b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f609013b330 0x7f6090072ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:08.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f6095b6b640 10 monclient(hunting): picked mon.noname-c con 0x7f609013b330 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:08.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f6095b6b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:08.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f6095b6b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f609009de60 con 0x7f60900a0a70 2024-09-17T18:19:08.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f6095b6b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:08.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f6095b6b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f609009dce0 con 0x7f60901348f0 2024-09-17T18:19:08.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f6095b6b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:08.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f6095b6b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f60900a1a50 con 0x7f609013b330 2024-09-17T18:19:08.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f6095b6b640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:08.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f6095b6b640 10 monclient(hunting): authenticate will time out at 752.032432s 2024-09-17T18:19:08.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f608ffff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f609013b330 0x7f6090072ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:08.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f608f7fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60900a0a70 0x7f6090133cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:08.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f608ffff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f609013b330 0x7f6090072ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:59732/0 (socket says 172.21.15.121:59732) 2024-09-17T18:19:08.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f608ffff640 1 -- 172.21.15.121:0/2007774432 learned_addr learned my addr 172.21.15.121:0/2007774432 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:08.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f608ffff640 10 monclient(hunting): get_auth_request con 0x7f609013b330 auth_method 0 2024-09-17T18:19:08.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f608ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:08.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f608ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:08.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f608ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:08.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f608f7fe640 10 monclient(hunting): get_auth_request con 0x7f60900a0a70 auth_method 0 2024-09-17T18:19:08.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f608f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:08.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f608f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:08.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f608f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:08.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f608ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:08.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f608ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:08.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f608ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:08.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f608f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:08.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f608f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:08.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f608effd640 1 --2- 172.21.15.121:0/2007774432 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60901348f0 0x7f6090134200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:08.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f608f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:08.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f608effd640 10 monclient(hunting): get_auth_request con 0x7f60901348f0 auth_method 0 2024-09-17T18:19:08.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f608effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:08.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f608effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:08.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f608effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:08.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f608effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:08.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f608effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:08.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f608effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:08.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f608ffff640 10 monclient(hunting): handle_auth_done global_id 4340 payload 824 2024-09-17T18:19:08.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f608ffff640 1 -- 172.21.15.121:0/2007774432 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60901348f0 msgr2=0x7f6090134200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:08.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f608ffff640 1 --2- 172.21.15.121:0/2007774432 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60901348f0 0x7f6090134200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:08.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f608ffff640 1 -- 172.21.15.121:0/2007774432 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60900a0a70 msgr2=0x7f6090133cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:08.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f608ffff640 1 --2- 172.21.15.121:0/2007774432 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60900a0a70 0x7f6090133cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:08.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f608ffff640 10 monclient: _finish_hunting 0 2024-09-17T18:19:08.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f608ffff640 1 monclient: found mon.noname-c 2024-09-17T18:19:08.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f608ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:08.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f608ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:19:08.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f608ffff640 1 -- 172.21.15.121:0/2007774432 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60901415e0 con 0x7f609013b330 2024-09-17T18:19:08.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.316+0000 7f608f7fe640 1 --2- 172.21.15.121:0/2007774432 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60900a0a70 0x7f6090133cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:19:08.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.319+0000 7f608ffff640 1 -- 172.21.15.121:0/2007774432 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f609013b330 msgr2=0x7f6090072ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:08.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.319+0000 7f608ffff640 1 --2- 172.21.15.121:0/2007774432 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f609013b330 0x7f6090072ca0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f608802aa70 tx=0x7f6088004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:08.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.319+0000 7f6077fff640 10 client.?.objecter ms_handle_connect 0x7f609013b330 2024-09-17T18:19:08.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.319+0000 7f6077fff640 10 client.?.objecter resend_mon_ops 2024-09-17T18:19:08.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.319+0000 7f6077fff640 1 -- 172.21.15.121:0/2007774432 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6088040070 con 0x7f609013b330 2024-09-17T18:19:08.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.319+0000 7f6077fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:08.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.319+0000 7f6077fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:19:08.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.319+0000 7f6077fff640 10 monclient: dump: 2024-09-17T18:19:08.332 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:08.332 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:08.333 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.333 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.333 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:08.333 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:08.333 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:08.333 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:08.333 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:08.333 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:08.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.319+0000 7f6077fff640 10 monclient: _finish_auth 0 2024-09-17T18:19:08.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.319+0000 7f6077fff640 10 monclient: _check_auth_tickets 2024-09-17T18:19:08.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.319+0000 7f6077fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:08.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.319+0000 7f6077fff640 1 -- 172.21.15.121:0/2007774432 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6088045460 con 0x7f609013b330 2024-09-17T18:19:08.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.319+0000 7f6095b6b640 5 monclient: authenticate success, global_id 4340 2024-09-17T18:19:08.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.319+0000 7f6095b6b640 10 monclient: _renew_subs 2024-09-17T18:19:08.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.319+0000 7f6095b6b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:08.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.319+0000 7f6095b6b640 1 -- 172.21.15.121:0/2007774432 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6090140760 con 0x7f609013b330 2024-09-17T18:19:08.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.319+0000 7f6077fff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:08.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.319+0000 7f6077fff640 1 -- 172.21.15.121:0/2007774432 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6088007a30 con 0x7f609013b330 2024-09-17T18:19:08.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.319+0000 7f6077fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:08.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.319+0000 7f6077fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:19:08.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.319+0000 7f6077fff640 10 monclient: dump: 2024-09-17T18:19:08.334 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:08.334 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:08.334 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.334 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.334 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:08.334 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:08.335 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:08.335 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:08.335 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:08.335 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:08.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.319+0000 7f6095b6b640 10 client.4340.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:08.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.319+0000 7f6095b6b640 10 monclient: _renew_subs 2024-09-17T18:19:08.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.319+0000 7f6095b6b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:08.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.319+0000 7f6095b6b640 1 -- 172.21.15.121:0/2007774432 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f609013a940 con 0x7f609013b330 2024-09-17T18:19:08.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.319+0000 7f6095b6b640 1 librados: init done 2024-09-17T18:19:08.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.319+0000 7f608dffb640 4 set_mon_vals no callback set 2024-09-17T18:19:08.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.320+0000 7f6077fff640 1 -- 172.21.15.121:0/2007774432 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f6088039040 con 0x7f609013b330 2024-09-17T18:19:08.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.320+0000 7f6095b6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:08.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.320+0000 7f6095b6b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:08.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.320+0000 7f6095b6b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:08.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.320+0000 7f6095b6b640 1 -- 172.21.15.121:0/2007774432 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f609009d360 con 0x7f609013b330 2024-09-17T18:19:08.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.320+0000 7f6077fff640 1 --2- 172.21.15.121:0/2007774432 >> v2:172.21.15.173:6812/2423786037 conn(0x7f607c040170 0x7f607c042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:08.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.320+0000 7f6077fff640 1 -- 172.21.15.121:0/2007774432 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(17..17 src has 1..17) ==== 3534+0+0 (secure 0 0 0) 0x7f6088074980 con 0x7f609013b330 2024-09-17T18:19:08.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.320+0000 7f6077fff640 10 client.4340.objecter ms_dispatch 0x7f6090000b90 osd_map(17..17 src has 1..17) 2024-09-17T18:19:08.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.320+0000 7f6077fff640 3 client.4340.objecter handle_osd_map got epochs [17,17] > 0 2024-09-17T18:19:08.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.320+0000 7f6077fff640 3 client.4340.objecter handle_osd_map decoding full epoch 17 2024-09-17T18:19:08.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.320+0000 7f6077fff640 20 client.4340.objecter dump_active .. 0 homeless 2024-09-17T18:19:08.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.320+0000 7f608f7fe640 1 --2- 172.21.15.121:0/2007774432 >> v2:172.21.15.173:6812/2423786037 conn(0x7f607c040170 0x7f607c042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:08.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.320+0000 7f608f7fe640 10 monclient: get_auth_request con 0x7f607c040170 auth_method 0 2024-09-17T18:19:08.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.323+0000 7f6077fff640 10 client.4340.objecter ms_handle_connect 0x7f607c040170 2024-09-17T18:19:08.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.323+0000 7f608f7fe640 1 --2- 172.21.15.121:0/2007774432 >> v2:172.21.15.173:6812/2423786037 conn(0x7f607c040170 0x7f607c042630 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6080007c90 tx=0x7f6080007be0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:08.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.324+0000 7f6077fff640 1 -- 172.21.15.121:0/2007774432 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f609009d360 con 0x7f609013b330 2024-09-17T18:19:08.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.324+0000 7f6077fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:08.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.324+0000 7f6077fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:19:08.352 INFO:tasks.workunit:timeout=3h 2024-09-17T18:19:08.352 INFO:tasks.workunit:cleanup=True 2024-09-17T18:19:08.353 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T18:19:08.410 INFO:tasks.workunit.client.0.smithi173.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-17T18:19:08.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.566+0000 7f6095b6b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:19:08.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.566+0000 7f6095b6b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:19:08.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.566+0000 7f6095b6b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:08.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.566+0000 7f6095b6b640 1 -- 172.21.15.121:0/2007774432 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f60900a0e50 con 0x7f609013b330 2024-09-17T18:19:08.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.567+0000 7f6077fff640 1 -- 172.21.15.121:0/2007774432 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f60901415e0 con 0x7f609013b330 2024-09-17T18:19:08.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.567+0000 7f6077fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:19:08.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.567+0000 7f6077fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:19:08.569 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:19:08.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.569+0000 7f6095b6b640 10 librados: watch_flush enter 2024-09-17T18:19:08.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.569+0000 7f6095b6b640 10 librados: watch_flush exit 2024-09-17T18:19:08.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.569+0000 7f6095b6b640 20 client.4340.objecter shutdown clearing up homeless session... 2024-09-17T18:19:08.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.569+0000 7f6095b6b640 10 client.4340.objecter successfully canceled tick 2024-09-17T18:19:08.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.569+0000 7f6095b6b640 1 -- 172.21.15.121:0/2007774432 >> v2:172.21.15.173:6812/2423786037 conn(0x7f607c040170 msgr2=0x7f607c042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:08.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.569+0000 7f6095b6b640 1 --2- 172.21.15.121:0/2007774432 >> v2:172.21.15.173:6812/2423786037 conn(0x7f607c040170 0x7f607c042630 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6080007c90 tx=0x7f6080007be0 comp rx=0 tx=0).stop 2024-09-17T18:19:08.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.569+0000 7f6095b6b640 10 monclient: shutdown 2024-09-17T18:19:08.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.569+0000 7f6095b6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:08.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.569+0000 7f6095b6b640 1 -- 172.21.15.121:0/2007774432 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f609013b330 msgr2=0x7f6090072ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:08.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.569+0000 7f6095b6b640 1 --2- 172.21.15.121:0/2007774432 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f609013b330 0x7f6090072ca0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f608802aa70 tx=0x7f6088004a40 comp rx=0 tx=0).stop 2024-09-17T18:19:08.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.569+0000 7f6095b6b640 1 -- 172.21.15.121:0/2007774432 shutdown_connections 2024-09-17T18:19:08.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.569+0000 7f6095b6b640 1 --2- 172.21.15.121:0/2007774432 >> v2:172.21.15.173:6812/2423786037 conn(0x7f607c040170 0x7f607c042630 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:08.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.569+0000 7f6095b6b640 1 --2- 172.21.15.121:0/2007774432 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60901348f0 0x7f6090134200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:08.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.569+0000 7f6095b6b640 1 --2- 172.21.15.121:0/2007774432 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f609013b330 0x7f6090072ca0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:08.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.569+0000 7f6095b6b640 1 --2- 172.21.15.121:0/2007774432 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60900a0a70 0x7f6090133cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:08.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.569+0000 7f6095b6b640 1 -- 172.21.15.121:0/2007774432 >> 172.21.15.121:0/2007774432 conn(0x7f60900a3b50 msgr2=0x7f60900a1e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:08.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.569+0000 7f6095b6b640 1 -- 172.21.15.121:0/2007774432 shutdown_connections 2024-09-17T18:19:08.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.569+0000 7f6095b6b640 1 -- 172.21.15.121:0/2007774432 wait complete. 2024-09-17T18:19:08.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.569+0000 7f6095b6b640 1 librados: shutdown 2024-09-17T18:19:08.582 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T18:19:08.582 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-17T18:19:08.582 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T18:19:08.582 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T18:19:08.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.679+0000 7f419c623640 10 monclient: get_monmap_and_config 2024-09-17T18:19:08.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.679+0000 7f419c623640 10 monclient: build_initial_monmap 2024-09-17T18:19:08.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.679+0000 7f419c623640 10 monclient: monmap: 2024-09-17T18:19:08.681 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:08.681 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:08.681 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:19:08.679802+0000 2024-09-17T18:19:08.682 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:19:08.679802+0000 2024-09-17T18:19:08.682 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:08.682 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:08.682 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:19:08.682 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:19:08.682 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:19:08.682 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:19:08.682 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:19:08.682 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:08.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.679+0000 7f419c623640 1 Processor -- start 2024-09-17T18:19:08.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.679+0000 7f419c623640 1 -- start start 2024-09-17T18:19:08.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.679+0000 7f419c623640 10 monclient: init 2024-09-17T18:19:08.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.680+0000 7f419c623640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:08.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.680+0000 7f419c623640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-17T18:19:08.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.680+0000 7f419c623640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41940700f0 0x7f4194134e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:08.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.680+0000 7f419c623640 10 monclient(hunting): picked mon.noname-b con 0x7f41940700f0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:08.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.680+0000 7f419c623640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f419409e810 addr v1:172.21.15.173:6789/0 2024-09-17T18:19:08.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.680+0000 7f419c623640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f419409ec30 addr v1:172.21.15.121:6789/0 2024-09-17T18:19:08.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.680+0000 7f419c623640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4194139180 con 0x7f419409ec30 2024-09-17T18:19:08.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.680+0000 7f419c623640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f419409ca60 con 0x7f419409e810 2024-09-17T18:19:08.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.680+0000 7f419c623640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:08.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.680+0000 7f419c623640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f41940704d0 con 0x7f41940700f0 2024-09-17T18:19:08.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.680+0000 7f419c623640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:08.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.680+0000 7f419c623640 10 monclient(hunting): authenticate will time out at 752.396634s 2024-09-17T18:19:08.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.680+0000 7f419a3bf640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41940700f0 0x7f4194134e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:08.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.680+0000 7f419abc0640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f419409ec30 0x7f4194138a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:49688/0 (socket says 172.21.15.121:49688) 2024-09-17T18:19:08.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.680+0000 7f419abc0640 1 -- 172.21.15.121:0/3036322637 learned_addr learned my addr 172.21.15.121:0/3036322637 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:19:08.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.680+0000 7f419a3bf640 10 monclient(hunting): get_auth_request con 0x7f41940700f0 auth_method 0 2024-09-17T18:19:08.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.680+0000 7f419a3bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:08.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.680+0000 7f419a3bf640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:08.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.680+0000 7f419a3bf640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:08.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.680+0000 7f419a3bf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:08.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.680+0000 7f419a3bf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:08.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.680+0000 7f419a3bf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:08.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.680+0000 7f419a3bf640 10 monclient(hunting): handle_auth_done global_id 4343 payload 274 2024-09-17T18:19:08.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.680+0000 7f419a3bf640 1 -- 172.21.15.121:0/3036322637 >> v1:172.21.15.173:6789/0 conn(0x7f419409e810 legacy=0x7f4194138360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:08.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.680+0000 7f419a3bf640 1 -- 172.21.15.121:0/3036322637 >> v1:172.21.15.121:6789/0 conn(0x7f419409ec30 legacy=0x7f4194138a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:08.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.680+0000 7f419a3bf640 10 monclient: _finish_hunting 0 2024-09-17T18:19:08.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.680+0000 7f419a3bf640 1 monclient: found mon.noname-b 2024-09-17T18:19:08.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.680+0000 7f419a3bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:08.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.680+0000 7f419a3bf640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:19:08.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.680+0000 7f419a3bf640 1 -- 172.21.15.121:0/3036322637 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f419409cd20 con 0x7f41940700f0 2024-09-17T18:19:08.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.683+0000 7f419a3bf640 1 -- 172.21.15.121:0/3036322637 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41940700f0 msgr2=0x7f4194134e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:08.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.683+0000 7f419a3bf640 1 --2- 172.21.15.121:0/3036322637 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41940700f0 0x7f4194134e50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f41900097d0 tx=0x7f419002a030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=615c78b3768028da server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:08.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.683+0000 7f41993bd640 1 -- 172.21.15.121:0/3036322637 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4190037070 con 0x7f41940700f0 2024-09-17T18:19:08.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.683+0000 7f41993bd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:08.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.683+0000 7f41993bd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:19:08.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.683+0000 7f41993bd640 10 monclient: dump: 2024-09-17T18:19:08.694 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:08.694 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:08.694 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.694 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.695 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:08.695 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:08.695 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:08.695 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:08.695 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:08.695 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:08.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.683+0000 7f41993bd640 10 monclient: _finish_auth 0 2024-09-17T18:19:08.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.683+0000 7f41993bd640 10 monclient: _check_auth_tickets 2024-09-17T18:19:08.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.683+0000 7f41993bd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:08.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.683+0000 7f419c623640 5 monclient: authenticate success, global_id 4343 2024-09-17T18:19:08.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.683+0000 7f419c623640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:19:08.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.683+0000 7f41993bd640 1 -- 172.21.15.121:0/3036322637 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4190032640 con 0x7f41940700f0 2024-09-17T18:19:08.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.683+0000 7f41993bd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:08.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.683+0000 7f419c623640 10 monclient: get_monmap_and_config success 2024-09-17T18:19:08.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.683+0000 7f41993bd640 1 -- 172.21.15.121:0/3036322637 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4190032a10 con 0x7f41940700f0 2024-09-17T18:19:08.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.683+0000 7f41993bd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:08.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.683+0000 7f419c623640 4 set_mon_vals no callback set 2024-09-17T18:19:08.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.683+0000 7f41993bd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:19:08.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.683+0000 7f41993bd640 10 monclient: dump: 2024-09-17T18:19:08.697 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:08.697 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:08.697 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.697 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.697 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:08.697 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:08.698 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:08.698 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:08.698 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:08.698 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:08.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.684+0000 7f419c623640 10 monclient: shutdown 2024-09-17T18:19:08.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.684+0000 7f419c623640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:08.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.684+0000 7f419c623640 1 -- 172.21.15.121:0/3036322637 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41940700f0 msgr2=0x7f4194134e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:08.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.684+0000 7f419c623640 1 --2- 172.21.15.121:0/3036322637 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41940700f0 0x7f4194134e50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f41900097d0 tx=0x7f419002a030 comp rx=0 tx=0).stop 2024-09-17T18:19:08.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.684+0000 7f419c623640 1 -- 172.21.15.121:0/3036322637 shutdown_connections 2024-09-17T18:19:08.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.684+0000 7f419c623640 1 --2- 172.21.15.121:0/3036322637 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41940700f0 0x7f4194134e50 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:08.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.684+0000 7f419c623640 1 -- 172.21.15.121:0/3036322637 >> 172.21.15.121:0/3036322637 conn(0x7f41940ab830 msgr2=0x7f41940a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:08.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.684+0000 7f419c623640 1 -- 172.21.15.121:0/3036322637 shutdown_connections 2024-09-17T18:19:08.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.684+0000 7f419c623640 1 -- 172.21.15.121:0/3036322637 wait complete. 2024-09-17T18:19:08.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.685+0000 7f419c623640 10 monclient: build_initial_monmap 2024-09-17T18:19:08.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.685+0000 7f419c623640 10 monclient: monmap: 2024-09-17T18:19:08.699 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:08.699 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:19:08.699 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:19:08.699 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:19:08.699 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:08.699 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:08.699 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:19:08.699 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:19:08.699 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:19:08.699 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:08.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.685+0000 7f419c623640 1 librados: starting msgr at 2024-09-17T18:19:08.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.685+0000 7f419c623640 1 librados: starting objecter 2024-09-17T18:19:08.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.685+0000 7f419c623640 1 Processor -- start 2024-09-17T18:19:08.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.685+0000 7f419c623640 1 -- start start 2024-09-17T18:19:08.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.685+0000 7f419c623640 1 librados: setting wanted keys 2024-09-17T18:19:08.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.685+0000 7f419c623640 1 librados: calling monclient init 2024-09-17T18:19:08.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.685+0000 7f419c623640 10 monclient: init 2024-09-17T18:19:08.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.685+0000 7f419c623640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:08.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.685+0000 7f419c623640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T18:19:08.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.685+0000 7f419c623640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41940700f0 0x7f4194131d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:08.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.685+0000 7f419c623640 10 monclient(hunting): picked mon.noname-b con 0x7f41940700f0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:08.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.685+0000 7f419c623640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f419409e810 0x7f4194132290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:08.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.685+0000 7f419c623640 10 monclient(hunting): picked mon.noname-a con 0x7f419409e810 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:08.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.685+0000 7f419c623640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f419409ec30 0x7f41940a0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:08.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.685+0000 7f419c623640 10 monclient(hunting): picked mon.noname-c con 0x7f419409ec30 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:08.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.685+0000 7f419c623640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:08.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.685+0000 7f419c623640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4194135390 con 0x7f419409e810 2024-09-17T18:19:08.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.685+0000 7f419c623640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:08.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.685+0000 7f419c623640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f419409d2e0 con 0x7f41940700f0 2024-09-17T18:19:08.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.685+0000 7f419c623640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:08.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.685+0000 7f419c623640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4194135510 con 0x7f419409ec30 2024-09-17T18:19:08.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.685+0000 7f419c623640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:08.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.685+0000 7f419c623640 10 monclient(hunting): authenticate will time out at 752.401998s 2024-09-17T18:19:08.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.686+0000 7f419abc0640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f419409ec30 0x7f41940a0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:08.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.686+0000 7f4199bbe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f419409e810 0x7f4194132290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:08.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.686+0000 7f419abc0640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f419409ec30 0x7f41940a0020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:59758/0 (socket says 172.21.15.121:59758) 2024-09-17T18:19:08.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.686+0000 7f419abc0640 1 -- 172.21.15.121:0/2328684205 learned_addr learned my addr 172.21.15.121:0/2328684205 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:08.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.686+0000 7f419abc0640 10 monclient(hunting): get_auth_request con 0x7f419409ec30 auth_method 0 2024-09-17T18:19:08.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.686+0000 7f4199bbe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f419409e810 0x7f4194132290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41250/0 (socket says 172.21.15.121:41250) 2024-09-17T18:19:08.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.686+0000 7f419abc0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:08.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.686+0000 7f419abc0640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:08.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.686+0000 7f419abc0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:08.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.686+0000 7f4199bbe640 10 monclient(hunting): get_auth_request con 0x7f419409e810 auth_method 0 2024-09-17T18:19:08.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.686+0000 7f4199bbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:08.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.686+0000 7f4199bbe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:08.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.686+0000 7f4199bbe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:08.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.686+0000 7f419abc0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:08.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.686+0000 7f419abc0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:08.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.686+0000 7f419abc0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:08.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.686+0000 7f4199bbe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:08.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.686+0000 7f4199bbe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:08.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.686+0000 7f4199bbe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:08.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.686+0000 7f419a3bf640 1 --2- 172.21.15.121:0/2328684205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41940700f0 0x7f4194131d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:08.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.686+0000 7f419abc0640 10 monclient(hunting): handle_auth_done global_id 4346 payload 824 2024-09-17T18:19:08.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.686+0000 7f419abc0640 1 -- 172.21.15.121:0/2328684205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41940700f0 msgr2=0x7f4194131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:08.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.686+0000 7f419abc0640 1 --2- 172.21.15.121:0/2328684205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41940700f0 0x7f4194131d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:08.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.686+0000 7f419abc0640 1 -- 172.21.15.121:0/2328684205 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f419409e810 msgr2=0x7f4194132290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:08.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.686+0000 7f419abc0640 1 --2- 172.21.15.121:0/2328684205 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f419409e810 0x7f4194132290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:08.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.686+0000 7f419abc0640 10 monclient: _finish_hunting 0 2024-09-17T18:19:08.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.686+0000 7f419abc0640 1 monclient: found mon.noname-c 2024-09-17T18:19:08.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.686+0000 7f419abc0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:08.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.686+0000 7f419abc0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:19:08.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.686+0000 7f419abc0640 1 -- 172.21.15.121:0/2328684205 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41940a1ae0 con 0x7f419409ec30 2024-09-17T18:19:08.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.686+0000 7f419a3bf640 10 monclient: get_auth_request con 0x7f41940700f0 auth_method 0 2024-09-17T18:19:08.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.686+0000 7f4199bbe640 1 --2- 172.21.15.121:0/2328684205 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f419409e810 0x7f4194132290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:19:08.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.686+0000 7f419a3bf640 1 --2- 172.21.15.121:0/2328684205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41940700f0 0x7f4194131d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T18:19:08.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.688+0000 7f419abc0640 1 -- 172.21.15.121:0/2328684205 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f419409ec30 msgr2=0x7f41940a0020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:08.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.688+0000 7f419abc0640 1 --2- 172.21.15.121:0/2328684205 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f419409ec30 0x7f41940a0020 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f418c00caf0 tx=0x7f418c00a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:08.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.688+0000 7f4186ffd640 10 client.?.objecter ms_handle_connect 0x7f419409ec30 2024-09-17T18:19:08.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.688+0000 7f4186ffd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:19:08.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.688+0000 7f4186ffd640 1 -- 172.21.15.121:0/2328684205 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f418c012400 con 0x7f419409ec30 2024-09-17T18:19:08.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.688+0000 7f4186ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:08.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.688+0000 7f4186ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:19:08.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.688+0000 7f4186ffd640 10 monclient: dump: 2024-09-17T18:19:08.704 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:08.704 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:08.704 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.704 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.704 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:08.705 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:08.705 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:08.705 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:08.705 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:08.705 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:08.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.688+0000 7f4186ffd640 10 monclient: _finish_auth 0 2024-09-17T18:19:08.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.688+0000 7f4186ffd640 10 monclient: _check_auth_tickets 2024-09-17T18:19:08.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.688+0000 7f4186ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:08.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.688+0000 7f4186ffd640 1 -- 172.21.15.121:0/2328684205 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f418c0125c0 con 0x7f419409ec30 2024-09-17T18:19:08.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.688+0000 7f419c623640 5 monclient: authenticate success, global_id 4346 2024-09-17T18:19:08.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.688+0000 7f419c623640 10 monclient: _renew_subs 2024-09-17T18:19:08.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.688+0000 7f419c623640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:08.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.688+0000 7f419c623640 1 -- 172.21.15.121:0/2328684205 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f419413b020 con 0x7f419409ec30 2024-09-17T18:19:08.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.688+0000 7f4186ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:08.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.688+0000 7f4186ffd640 1 -- 172.21.15.121:0/2328684205 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f418c0129b0 con 0x7f419409ec30 2024-09-17T18:19:08.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.688+0000 7f4186ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:08.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.688+0000 7f4186ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:19:08.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.688+0000 7f4186ffd640 10 monclient: dump: 2024-09-17T18:19:08.706 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:08.706 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:08.706 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.706 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.706 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:08.706 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:08.706 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:08.706 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:08.706 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:08.706 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:08.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.688+0000 7f4198bbc640 4 set_mon_vals no callback set 2024-09-17T18:19:08.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.688+0000 7f419c623640 10 client.4346.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:08.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.688+0000 7f419c623640 10 monclient: _renew_subs 2024-09-17T18:19:08.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.688+0000 7f419c623640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:08.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.688+0000 7f419c623640 1 -- 172.21.15.121:0/2328684205 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f41941327d0 con 0x7f419409ec30 2024-09-17T18:19:08.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.688+0000 7f419c623640 1 librados: init done 2024-09-17T18:19:08.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.689+0000 7f419c623640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:08.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.689+0000 7f419c623640 1 --2- 172.21.15.121:0/2328684205 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4194135f30 0x7f4194136310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:08.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.689+0000 7f419c623640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:08.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.689+0000 7f419c623640 1 -- 172.21.15.121:0/2328684205 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f41940a23f0 con 0x7f4194135f30 2024-09-17T18:19:08.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.689+0000 7f419a3bf640 1 --2- 172.21.15.121:0/2328684205 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4194135f30 0x7f4194136310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:08.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.689+0000 7f419a3bf640 10 monclient: get_auth_request con 0x7f4194135f30 auth_method 0 2024-09-17T18:19:08.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.689+0000 7f419a3bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:08.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.689+0000 7f419a3bf640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:08.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.689+0000 7f419a3bf640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:08.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.689+0000 7f4186ffd640 1 -- 172.21.15.121:0/2328684205 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f418c012b90 con 0x7f419409ec30 2024-09-17T18:19:08.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.689+0000 7f419a3bf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:08.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.689+0000 7f419a3bf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:08.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.689+0000 7f419a3bf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:08.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.689+0000 7f419a3bf640 10 monclient(hunting): handle_auth_done global_id 4353 payload 824 2024-09-17T18:19:08.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.689+0000 7f419a3bf640 1 -- 172.21.15.121:0/2328684205 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f41941348c0 con 0x7f4194135f30 2024-09-17T18:19:08.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.690+0000 7f4186ffd640 1 --2- 172.21.15.121:0/2328684205 >> v2:172.21.15.173:6812/2423786037 conn(0x7f41780401c0 0x7f4178042680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:08.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.690+0000 7f4186ffd640 1 -- 172.21.15.121:0/2328684205 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(17..17 src has 1..17) ==== 3534+0+0 (secure 0 0 0) 0x7f418c053600 con 0x7f419409ec30 2024-09-17T18:19:08.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.690+0000 7f4186ffd640 10 client.4346.objecter ms_dispatch 0x7f4194000b90 osd_map(17..17 src has 1..17) 2024-09-17T18:19:08.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.690+0000 7f4186ffd640 3 client.4346.objecter handle_osd_map got epochs [17,17] > 0 2024-09-17T18:19:08.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.690+0000 7f4186ffd640 3 client.4346.objecter handle_osd_map decoding full epoch 17 2024-09-17T18:19:08.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.690+0000 7f4186ffd640 20 client.4346.objecter dump_active .. 0 homeless 2024-09-17T18:19:08.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.690+0000 7f4199bbe640 1 --2- 172.21.15.121:0/2328684205 >> v2:172.21.15.173:6812/2423786037 conn(0x7f41780401c0 0x7f4178042680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:08.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.690+0000 7f4199bbe640 10 monclient: get_auth_request con 0x7f41780401c0 auth_method 0 2024-09-17T18:19:08.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.692+0000 7f4186ffd640 10 client.4346.objecter ms_handle_connect 0x7f4194135f30 2024-09-17T18:19:08.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.692+0000 7f4186ffd640 10 client.4346.objecter resend_mon_ops 2024-09-17T18:19:08.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.692+0000 7f419a3bf640 1 -- 172.21.15.121:0/2328684205 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4194135f30 msgr2=0x7f4194136310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:08.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.692+0000 7f419a3bf640 1 --2- 172.21.15.121:0/2328684205 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4194135f30 0x7f4194136310 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f41940a5870 tx=0x7f41900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:08.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.692+0000 7f4186ffd640 1 -- 172.21.15.121:0/2328684205 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4190042070 con 0x7f4194135f30 2024-09-17T18:19:08.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.692+0000 7f4186ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:08.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.692+0000 7f4186ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:19:08.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.692+0000 7f4186ffd640 10 monclient: dump: 2024-09-17T18:19:08.709 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:08.709 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:08.709 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.709 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.710 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:08.710 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:08.710 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:08.710 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:08.710 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:08.710 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:08.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.692+0000 7f4186ffd640 1 -- 172.21.15.121:0/2328684205 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f419003e660 con 0x7f4194135f30 2024-09-17T18:19:08.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.692+0000 7f4186ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:08.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.692+0000 7f4186ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:19:08.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.692+0000 7f4186ffd640 1 -- 172.21.15.121:0/2328684205 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4194135f30 msgr2=0x7f4194136310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:08.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.692+0000 7f4186ffd640 1 --2- 172.21.15.121:0/2328684205 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4194135f30 0x7f4194136310 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f41940a5870 tx=0x7f41900046c0 comp rx=0 tx=0).stop 2024-09-17T18:19:08.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.692+0000 7f4186ffd640 1 -- 172.21.15.121:0/2328684205 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4194135f30 msgr2=0x7f4194136310 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:19:08.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.692+0000 7f4186ffd640 1 --2- 172.21.15.121:0/2328684205 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4194135f30 0x7f4194136310 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f41940a5870 tx=0x7f41900046c0 comp rx=0 tx=0).stop 2024-09-17T18:19:08.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.692+0000 7f4199bbe640 1 --2- 172.21.15.121:0/2328684205 >> v2:172.21.15.173:6812/2423786037 conn(0x7f41780401c0 0x7f4178042680 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f419413a6b0 tx=0x7f4188007e30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:08.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.692+0000 7f4186ffd640 10 client.4346.objecter ms_handle_connect 0x7f41780401c0 2024-09-17T18:19:08.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.698+0000 7f419c623640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:19:08.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.698+0000 7f419c623640 1 --2- 172.21.15.121:0/2328684205 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4194136850 0x7f41940a0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:08.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.698+0000 7f419c623640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:08.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.698+0000 7f419c623640 1 -- 172.21.15.121:0/2328684205 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f419413ab60 con 0x7f4194136850 2024-09-17T18:19:08.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.698+0000 7f419a3bf640 1 --2- 172.21.15.121:0/2328684205 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4194136850 0x7f41940a0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:08.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.698+0000 7f419a3bf640 10 monclient: get_auth_request con 0x7f4194136850 auth_method 0 2024-09-17T18:19:08.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.698+0000 7f419a3bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:08.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.698+0000 7f419a3bf640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:08.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.698+0000 7f419a3bf640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:08.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.698+0000 7f419a3bf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:08.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.698+0000 7f419a3bf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:08.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.698+0000 7f419a3bf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:08.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.698+0000 7f419a3bf640 10 monclient(hunting): handle_auth_done global_id 4356 payload 824 2024-09-17T18:19:08.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.699+0000 7f419a3bf640 1 -- 172.21.15.121:0/2328684205 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f419406a660 con 0x7f4194136850 2024-09-17T18:19:08.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.701+0000 7f4186ffd640 10 client.4346.objecter ms_handle_connect 0x7f4194136850 2024-09-17T18:19:08.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.701+0000 7f419a3bf640 1 -- 172.21.15.121:0/2328684205 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4194136850 msgr2=0x7f41940a0560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:08.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.701+0000 7f4186ffd640 10 client.4346.objecter resend_mon_ops 2024-09-17T18:19:08.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.701+0000 7f419a3bf640 1 --2- 172.21.15.121:0/2328684205 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4194136850 0x7f41940a0560 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4194139a50 tx=0x7f4190008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:08.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.701+0000 7f4186ffd640 1 -- 172.21.15.121:0/2328684205 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4190037040 con 0x7f4194136850 2024-09-17T18:19:08.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.701+0000 7f4186ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:08.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.701+0000 7f4186ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:19:08.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.701+0000 7f4186ffd640 10 monclient: dump: 2024-09-17T18:19:08.715 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:08.715 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:08.715 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.715 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.715 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:08.715 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:08.715 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:08.715 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:08.715 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:08.715 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:08.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.701+0000 7f4186ffd640 1 -- 172.21.15.121:0/2328684205 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4721 (secure 0 0 0) 0x7f4190003610 con 0x7f4194136850 2024-09-17T18:19:08.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.701+0000 7f4186ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:19:08.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.701+0000 7f4186ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:19:08.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.701+0000 7f4186ffd640 1 -- 172.21.15.121:0/2328684205 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4194136850 msgr2=0x7f41940a0560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:08.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.701+0000 7f4186ffd640 1 --2- 172.21.15.121:0/2328684205 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4194136850 0x7f41940a0560 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4194139a50 tx=0x7f4190008030 comp rx=0 tx=0).stop 2024-09-17T18:19:08.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.701+0000 7f4186ffd640 1 -- 172.21.15.121:0/2328684205 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4194136850 msgr2=0x7f41940a0560 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:19:08.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.701+0000 7f4186ffd640 1 --2- 172.21.15.121:0/2328684205 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4194136850 0x7f41940a0560 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:08.716 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:19:08.716 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:19:08.716 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:19:08.716 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "leader", 2024-09-17T18:19:08.716 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 4, 2024-09-17T18:19:08.716 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 21686, 2024-09-17T18:19:08.716 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:19:08.716 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:19:08.717 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:19:08.717 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:19:08.717 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:19:08.717 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 21, 2024-09-17T18:19:08.717 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:19:08.717 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:19:08.717 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:19:08.717 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:19:08.717 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:19:08.717 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:19:08.717 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:19:08.717 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:19:08.717 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:19:08.717 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:19:08.718 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:19:08.718 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:19:08.718 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:19:08.718 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:19:08.718 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:19:08.718 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:19:08.718 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:19:08.718 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:19:08.718 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:19:08.718 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:19:08.718 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:19:08.718 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:19:08.718 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:19:08.719 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:19:08.719 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:19:08.719 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:19:08.719 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:19:08.719 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:19:08.719 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:19:08.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:08.719 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:19:08.719 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:19:08.719 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:19:08.719 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:19:08.719 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:19:08.720 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:19:08.720 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:19:08.720 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:19:08.720 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:19:08.720 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:19:08.720 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:19:08.720 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:19:08.720 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:19:08.720 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:19:08.720 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:19:08.720 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:19:08.720 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:19:08.720 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:19:08.721 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:19:08.721 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:19:08.721 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:19:08.721 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:19:08.721 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:19:08.721 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:19:08.721 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:19:08.721 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:19:08.721 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:19:08.721 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:19:08.721 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:19:08.721 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:19:08.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:08.722 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:19:08.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:08.722 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:19:08.722 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:19:08.722 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:19:08.722 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:19:08.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:08.722 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:19:08.722 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:19:08.722 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:19:08.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:08.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:08.722 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:19:08.722 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:19:08.723 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:19:08.723 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:19:08.723 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:19:08.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:08.723 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:19:08.723 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:19:08.723 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:19:08.723 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:19:08.723 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:19:08.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:08.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:08.723 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:19:08.724 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:19:08.724 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:19:08.724 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:19:08.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:08.724 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:19:08.724 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:19:08.724 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:19:08.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:08.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:08.724 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:19:08.724 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:19:08.724 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:19:08.724 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:19:08.725 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:19:08.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:08.725 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:19:08.725 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:19:08.725 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:19:08.725 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:19:08.725 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:19:08.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:08.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:08.725 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:19:08.725 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:19:08.725 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:19:08.725 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:19:08.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:08.726 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:19:08.726 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:19:08.726 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:19:08.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:08.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:08.726 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:19:08.726 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:19:08.726 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:19:08.726 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:19:08.726 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:19:08.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:08.726 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:19:08.726 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:19:08.726 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:19:08.727 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:19:08.727 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:19:08.727 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:19:08.727 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:19:08.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:08.727 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:19:08.727 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:19:08.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:08.727 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:19:08.727 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:19:08.727 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:19:08.727 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:19:08.727 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:19:08.727 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:19:08.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:08.728 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:19:08.728 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:19:08.728 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-17T18:19:08.728 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:19:08.728 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:19:08.728 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:19:08.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:08.728 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:19:08.728 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:19:08.728 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:19:08.728 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:19:08.729 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:19:08.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:08.729 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:19:08.729 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:19:08.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.701+0000 7f419c623640 10 librados: watch_flush enter 2024-09-17T18:19:08.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.701+0000 7f419c623640 10 librados: watch_flush exit 2024-09-17T18:19:08.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.701+0000 7f419c623640 20 client.4346.objecter shutdown clearing up homeless session... 2024-09-17T18:19:08.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.701+0000 7f419c623640 10 client.4346.objecter successfully canceled tick 2024-09-17T18:19:08.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.701+0000 7f419c623640 1 -- 172.21.15.121:0/2328684205 >> v2:172.21.15.173:6812/2423786037 conn(0x7f41780401c0 msgr2=0x7f4178042680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:08.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.701+0000 7f419c623640 1 --2- 172.21.15.121:0/2328684205 >> v2:172.21.15.173:6812/2423786037 conn(0x7f41780401c0 0x7f4178042680 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f419413a6b0 tx=0x7f4188007e30 comp rx=0 tx=0).stop 2024-09-17T18:19:08.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.701+0000 7f419c623640 10 monclient: shutdown 2024-09-17T18:19:08.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.701+0000 7f419c623640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:08.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.701+0000 7f419c623640 1 -- 172.21.15.121:0/2328684205 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f419409ec30 msgr2=0x7f41940a0020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:08.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.701+0000 7f419c623640 1 --2- 172.21.15.121:0/2328684205 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f419409ec30 0x7f41940a0020 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f418c00caf0 tx=0x7f418c00a060 comp rx=0 tx=0).stop 2024-09-17T18:19:08.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.701+0000 7f419abc0640 1 -- 172.21.15.121:0/2328684205 reap_dead start 2024-09-17T18:19:08.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.701+0000 7f419abc0640 1 -- 172.21.15.121:0/2328684205 reap_dead start 2024-09-17T18:19:08.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.701+0000 7f419c623640 1 -- 172.21.15.121:0/2328684205 shutdown_connections 2024-09-17T18:19:08.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.701+0000 7f419c623640 1 -- 172.21.15.121:0/2328684205 >> 172.21.15.121:0/2328684205 conn(0x7f41940ab830 msgr2=0x7f4194133ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:08.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.702+0000 7f419c623640 1 -- 172.21.15.121:0/2328684205 shutdown_connections 2024-09-17T18:19:08.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.702+0000 7f419c623640 1 -- 172.21.15.121:0/2328684205 wait complete. 2024-09-17T18:19:08.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.702+0000 7f419c623640 1 librados: shutdown 2024-09-17T18:19:08.732 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:19:08.732 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T18:19:08.732 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T18:19:08.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.827+0000 7f6f11f00640 10 monclient: get_monmap_and_config 2024-09-17T18:19:08.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.827+0000 7f6f11f00640 10 monclient: build_initial_monmap 2024-09-17T18:19:08.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.827+0000 7f6f11f00640 10 monclient: monmap: 2024-09-17T18:19:08.829 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:08.830 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:08.830 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:19:08.828361+0000 2024-09-17T18:19:08.830 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:19:08.828361+0000 2024-09-17T18:19:08.830 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:08.830 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:08.830 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:19:08.830 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:19:08.830 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:19:08.830 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:19:08.830 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:19:08.831 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:08.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.828+0000 7f6f11f00640 1 Processor -- start 2024-09-17T18:19:08.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.828+0000 7f6f11f00640 1 -- start start 2024-09-17T18:19:08.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.828+0000 7f6f11f00640 10 monclient: init 2024-09-17T18:19:08.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.828+0000 7f6f11f00640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:08.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.828+0000 7f6f11f00640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-17T18:19:08.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.829+0000 7f6f11f00640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6f0c0a3a00 addr v1:172.21.15.121:6789/0 2024-09-17T18:19:08.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.829+0000 7f6f11f00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f6f0c0a44f0 0x7f6f0c13ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:08.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.829+0000 7f6f11f00640 10 monclient(hunting): picked mon.noname-c con 0x7f6f0c0a44f0 addr v2:172.21.15.173:3300/0 2024-09-17T18:19:08.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.829+0000 7f6f11f00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f0c13bf40 0x7f6f0c13e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:08.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.829+0000 7f6f11f00640 10 monclient(hunting): picked mon.noname-b con 0x7f6f0c13bf40 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:08.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.829+0000 7f6f11f00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f0c09ee00 con 0x7f6f0c0a3a00 2024-09-17T18:19:08.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.829+0000 7f6f11f00640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:08.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.829+0000 7f6f11f00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f6f0c0a0a00 con 0x7f6f0c0a44f0 2024-09-17T18:19:08.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.829+0000 7f6f11f00640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:08.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.829+0000 7f6f11f00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6f0c076bc0 con 0x7f6f0c13bf40 2024-09-17T18:19:08.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.829+0000 7f6f11f00640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:08.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.829+0000 7f6f11f00640 10 monclient(hunting): authenticate will time out at 752.545067s 2024-09-17T18:19:08.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.829+0000 7f6f0b7fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f6f0c0a3a00 0x7f6f0c0a3de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:49700/0 (socket says 172.21.15.121:49700) 2024-09-17T18:19:08.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.829+0000 7f6f0bfff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f0c13bf40 0x7f6f0c13e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:08.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.829+0000 7f6f0b7fe640 1 -- 172.21.15.121:0/1481795170 learned_addr learned my addr 172.21.15.121:0/1481795170 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:19:08.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.829+0000 7f6f0bfff640 10 monclient(hunting): get_auth_request con 0x7f6f0c13bf40 auth_method 0 2024-09-17T18:19:08.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.829+0000 7f6f0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:08.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.829+0000 7f6f0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:08.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.829+0000 7f6f0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:08.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.829+0000 7f6f0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:08.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.829+0000 7f6f0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:08.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.829+0000 7f6f0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:08.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.829+0000 7f6f0affd640 1 --2- 172.21.15.121:0/1481795170 >> v2:172.21.15.173:3300/0 conn(0x7f6f0c0a44f0 0x7f6f0c13ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:08.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.829+0000 7f6f0bfff640 10 monclient(hunting): handle_auth_done global_id 4349 payload 274 2024-09-17T18:19:08.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.829+0000 7f6f0bfff640 1 -- 172.21.15.121:0/1481795170 >> v2:172.21.15.173:3300/0 conn(0x7f6f0c0a44f0 msgr2=0x7f6f0c13ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:08.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.829+0000 7f6f0bfff640 1 --2- 172.21.15.121:0/1481795170 >> v2:172.21.15.173:3300/0 conn(0x7f6f0c0a44f0 0x7f6f0c13ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:08.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.829+0000 7f6f0bfff640 1 -- 172.21.15.121:0/1481795170 >> v1:172.21.15.121:6789/0 conn(0x7f6f0c0a3a00 legacy=0x7f6f0c0a3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:08.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.829+0000 7f6f0bfff640 10 monclient: _finish_hunting 0 2024-09-17T18:19:08.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.829+0000 7f6f0bfff640 1 monclient: found mon.noname-b 2024-09-17T18:19:08.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.829+0000 7f6f0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:08.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.829+0000 7f6f0bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:19:08.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.829+0000 7f6f0bfff640 1 -- 172.21.15.121:0/1481795170 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f0c0a0040 con 0x7f6f0c13bf40 2024-09-17T18:19:08.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.829+0000 7f6f0affd640 10 monclient: get_auth_request con 0x7f6f0c0a44f0 auth_method 0 2024-09-17T18:19:08.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.829+0000 7f6f0affd640 1 --2- 172.21.15.121:0/1481795170 >> v2:172.21.15.173:3300/0 conn(0x7f6f0c0a44f0 0x7f6f0c13ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T18:19:08.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.832+0000 7f6f0bfff640 1 -- 172.21.15.121:0/1481795170 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f0c13bf40 msgr2=0x7f6f0c13e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:08.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.832+0000 7f6f0bfff640 1 --2- 172.21.15.121:0/1481795170 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f0c13bf40 0x7f6f0c13e330 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6f040097a0 tx=0x7f6f04029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a68ea7dd254ac5a server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:08.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.832+0000 7f6f0a7fc640 1 -- 172.21.15.121:0/1481795170 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f04037070 con 0x7f6f0c13bf40 2024-09-17T18:19:08.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.832+0000 7f6f0a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:08.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.832+0000 7f6f0a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:19:08.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.832+0000 7f6f0a7fc640 10 monclient: dump: 2024-09-17T18:19:08.843 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:08.843 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:08.843 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.843 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.843 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:08.844 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:08.844 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:08.844 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:08.844 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:08.844 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:08.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.832+0000 7f6f0a7fc640 10 monclient: _finish_auth 0 2024-09-17T18:19:08.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.832+0000 7f6f0a7fc640 10 monclient: _check_auth_tickets 2024-09-17T18:19:08.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.832+0000 7f6f0a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:08.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.832+0000 7f6f0a7fc640 1 -- 172.21.15.121:0/1481795170 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f04032640 con 0x7f6f0c13bf40 2024-09-17T18:19:08.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.832+0000 7f6f11f00640 5 monclient: authenticate success, global_id 4349 2024-09-17T18:19:08.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.832+0000 7f6f0a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:08.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.832+0000 7f6f11f00640 10 monclient: get_monmap_and_config success 2024-09-17T18:19:08.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.832+0000 7f6f0a7fc640 1 -- 172.21.15.121:0/1481795170 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f04032a10 con 0x7f6f0c13bf40 2024-09-17T18:19:08.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.832+0000 7f6f11f00640 4 set_mon_vals no callback set 2024-09-17T18:19:08.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.832+0000 7f6f0a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:08.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.832+0000 7f6f0a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:19:08.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.832+0000 7f6f0a7fc640 10 monclient: dump: 2024-09-17T18:19:08.845 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:08.845 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:08.845 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.845 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.845 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:08.845 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:08.845 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:08.846 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:08.846 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:08.846 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:08.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.832+0000 7f6f11f00640 10 monclient: shutdown 2024-09-17T18:19:08.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.832+0000 7f6f11f00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:08.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.832+0000 7f6f11f00640 1 -- 172.21.15.121:0/1481795170 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f0c13bf40 msgr2=0x7f6f0c13e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:08.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.832+0000 7f6f11f00640 1 --2- 172.21.15.121:0/1481795170 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f0c13bf40 0x7f6f0c13e330 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6f040097a0 tx=0x7f6f04029fd0 comp rx=0 tx=0).stop 2024-09-17T18:19:08.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.832+0000 7f6f11f00640 1 -- 172.21.15.121:0/1481795170 shutdown_connections 2024-09-17T18:19:08.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.832+0000 7f6f11f00640 1 --2- 172.21.15.121:0/1481795170 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f0c13bf40 0x7f6f0c13e330 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:08.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.832+0000 7f6f11f00640 1 --2- 172.21.15.121:0/1481795170 >> v2:172.21.15.173:3300/0 conn(0x7f6f0c0a44f0 0x7f6f0c13ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:08.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.832+0000 7f6f11f00640 1 -- 172.21.15.121:0/1481795170 >> 172.21.15.121:0/1481795170 conn(0x7f6f0c0a8680 msgr2=0x7f6f0c0a4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:08.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.832+0000 7f6f11f00640 1 -- 172.21.15.121:0/1481795170 shutdown_connections 2024-09-17T18:19:08.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.832+0000 7f6f11f00640 1 -- 172.21.15.121:0/1481795170 wait complete. 2024-09-17T18:19:08.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.833+0000 7f6f11f00640 10 monclient: build_initial_monmap 2024-09-17T18:19:08.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.833+0000 7f6f11f00640 10 monclient: monmap: 2024-09-17T18:19:08.847 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:08.847 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:19:08.847 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:19:08.847 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:19:08.847 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:08.847 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:08.847 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:19:08.847 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:19:08.847 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:19:08.847 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:08.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.833+0000 7f6f11f00640 1 librados: starting msgr at 2024-09-17T18:19:08.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.833+0000 7f6f11f00640 1 librados: starting objecter 2024-09-17T18:19:08.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.833+0000 7f6f11f00640 1 Processor -- start 2024-09-17T18:19:08.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.833+0000 7f6f11f00640 1 -- start start 2024-09-17T18:19:08.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f11f00640 1 librados: setting wanted keys 2024-09-17T18:19:08.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f11f00640 1 librados: calling monclient init 2024-09-17T18:19:08.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f11f00640 10 monclient: init 2024-09-17T18:19:08.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f11f00640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:08.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f11f00640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:19:08.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f11f00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f0c0a3a00 0x7f6f0c1cb540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:08.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f11f00640 10 monclient(hunting): picked mon.noname-a con 0x7f6f0c0a3a00 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:08.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f11f00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f0c0a44f0 0x7f6f0c136c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:08.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f11f00640 10 monclient(hunting): picked mon.noname-b con 0x7f6f0c0a44f0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:08.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f11f00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f0c13bf40 0x7f6f0c1371a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:08.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f11f00640 10 monclient(hunting): picked mon.noname-c con 0x7f6f0c13bf40 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:08.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f11f00640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:08.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f11f00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6f0c0a5470 con 0x7f6f0c0a3a00 2024-09-17T18:19:08.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f11f00640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:08.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f11f00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6f0c09f7a0 con 0x7f6f0c0a44f0 2024-09-17T18:19:08.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f11f00640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:08.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f11f00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6f0c0a55f0 con 0x7f6f0c13bf40 2024-09-17T18:19:08.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f11f00640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:08.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f11f00640 10 monclient(hunting): authenticate will time out at 752.550451s 2024-09-17T18:19:08.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f0b7fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f0c0a3a00 0x7f6f0c1cb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:08.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f0bfff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f0c13bf40 0x7f6f0c1371a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:08.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f0b7fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f0c0a3a00 0x7f6f0c1cb540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41260/0 (socket says 172.21.15.121:41260) 2024-09-17T18:19:08.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f0b7fe640 1 -- 172.21.15.121:0/4155364178 learned_addr learned my addr 172.21.15.121:0/4155364178 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:08.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f0b7fe640 10 monclient(hunting): get_auth_request con 0x7f6f0c0a3a00 auth_method 0 2024-09-17T18:19:08.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f0b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:08.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f0b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:08.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:08.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f0bfff640 10 monclient(hunting): get_auth_request con 0x7f6f0c13bf40 auth_method 0 2024-09-17T18:19:08.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:08.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:08.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:08.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f0b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:08.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f0b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:08.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f0b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:08.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f0affd640 1 --2- 172.21.15.121:0/4155364178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f0c0a44f0 0x7f6f0c136c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:08.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:08.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:08.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:08.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f0affd640 10 monclient(hunting): get_auth_request con 0x7f6f0c0a44f0 auth_method 0 2024-09-17T18:19:08.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f0affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:08.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f0affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:08.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f0affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:08.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f0affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:08.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f0affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:08.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f0affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:08.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f0b7fe640 10 monclient(hunting): handle_auth_done global_id 4359 payload 824 2024-09-17T18:19:08.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f0b7fe640 1 -- 172.21.15.121:0/4155364178 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f0c13bf40 msgr2=0x7f6f0c1371a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:08.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f0b7fe640 1 --2- 172.21.15.121:0/4155364178 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f0c13bf40 0x7f6f0c1371a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:08.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f0b7fe640 1 -- 172.21.15.121:0/4155364178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f0c0a44f0 msgr2=0x7f6f0c136c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:08.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f0b7fe640 1 --2- 172.21.15.121:0/4155364178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f0c0a44f0 0x7f6f0c136c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:08.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f0b7fe640 10 monclient: _finish_hunting 0 2024-09-17T18:19:08.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f0b7fe640 1 monclient: found mon.noname-a 2024-09-17T18:19:08.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:08.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f0b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:19:08.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f0b7fe640 1 -- 172.21.15.121:0/4155364178 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f0c125730 con 0x7f6f0c0a3a00 2024-09-17T18:19:08.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.834+0000 7f6f0bfff640 1 --2- 172.21.15.121:0/4155364178 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f0c13bf40 0x7f6f0c1371a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:19:08.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.837+0000 7f6f0b7fe640 1 -- 172.21.15.121:0/4155364178 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f0c0a3a00 msgr2=0x7f6f0c1cb540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:08.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.837+0000 7f6f0b7fe640 1 --2- 172.21.15.121:0/4155364178 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f0c0a3a00 0x7f6f0c1cb540 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6efc00cc10 tx=0x7f6efc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:08.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.837+0000 7f6ef3fff640 10 client.?.objecter ms_handle_connect 0x7f6f0c0a3a00 2024-09-17T18:19:08.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.837+0000 7f6ef3fff640 10 client.?.objecter resend_mon_ops 2024-09-17T18:19:08.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.837+0000 7f6ef3fff640 1 -- 172.21.15.121:0/4155364178 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6efc012400 con 0x7f6f0c0a3a00 2024-09-17T18:19:08.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.837+0000 7f6ef3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:08.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.837+0000 7f6ef3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:19:08.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.837+0000 7f6ef3fff640 10 monclient: dump: 2024-09-17T18:19:08.853 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:08.853 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:08.853 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.853 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.853 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:08.853 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:08.853 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:08.853 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:08.853 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:08.853 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:08.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.837+0000 7f6ef3fff640 10 monclient: _finish_auth 0 2024-09-17T18:19:08.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.837+0000 7f6ef3fff640 10 monclient: _check_auth_tickets 2024-09-17T18:19:08.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.837+0000 7f6ef3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:08.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.837+0000 7f6f11f00640 5 monclient: authenticate success, global_id 4359 2024-09-17T18:19:08.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.837+0000 7f6ef3fff640 1 -- 172.21.15.121:0/4155364178 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6efc0125e0 con 0x7f6f0c0a3a00 2024-09-17T18:19:08.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.837+0000 7f6f11f00640 10 monclient: _renew_subs 2024-09-17T18:19:08.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.837+0000 7f6f11f00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:08.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.837+0000 7f6f11f00640 1 -- 172.21.15.121:0/4155364178 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f0c1240b0 con 0x7f6f0c0a3a00 2024-09-17T18:19:08.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.837+0000 7f6ef3fff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:08.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.837+0000 7f6ef3fff640 1 -- 172.21.15.121:0/4155364178 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6efc0129d0 con 0x7f6f0c0a3a00 2024-09-17T18:19:08.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.837+0000 7f6ef3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:08.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.837+0000 7f6ef3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:19:08.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.837+0000 7f6ef3fff640 10 monclient: dump: 2024-09-17T18:19:08.854 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:08.854 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:08.854 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.854 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.854 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:08.854 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:08.854 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:08.855 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:08.855 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:08.855 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:08.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.837+0000 7f6f09ffb640 4 set_mon_vals no callback set 2024-09-17T18:19:08.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.837+0000 7f6f11f00640 10 client.4359.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:08.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.837+0000 7f6f11f00640 10 monclient: _renew_subs 2024-09-17T18:19:08.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.837+0000 7f6f11f00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:08.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.837+0000 7f6f11f00640 1 -- 172.21.15.121:0/4155364178 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f0c1cc5c0 con 0x7f6f0c0a3a00 2024-09-17T18:19:08.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.837+0000 7f6f11f00640 1 librados: init done 2024-09-17T18:19:08.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.838+0000 7f6f11f00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:08.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.838+0000 7f6f11f00640 1 --2- 172.21.15.121:0/4155364178 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f0c1ccfe0 0x7f6f0c02d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:08.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.838+0000 7f6f11f00640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:08.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.838+0000 7f6f11f00640 1 -- 172.21.15.121:0/4155364178 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6f0c09f1e0 con 0x7f6f0c1ccfe0 2024-09-17T18:19:08.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.838+0000 7f6f0affd640 1 --2- 172.21.15.121:0/4155364178 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f0c1ccfe0 0x7f6f0c02d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:08.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.838+0000 7f6ef3fff640 1 -- 172.21.15.121:0/4155364178 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f6efc012bf0 con 0x7f6f0c0a3a00 2024-09-17T18:19:08.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.838+0000 7f6f0affd640 10 monclient: get_auth_request con 0x7f6f0c1ccfe0 auth_method 0 2024-09-17T18:19:08.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.838+0000 7f6f0affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:08.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.838+0000 7f6f0affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:08.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.838+0000 7f6f0affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:08.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.838+0000 7f6f0affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:08.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.838+0000 7f6f0affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:08.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.838+0000 7f6f0affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:08.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.838+0000 7f6f0affd640 10 monclient(hunting): handle_auth_done global_id 4355 payload 824 2024-09-17T18:19:08.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.838+0000 7f6f0affd640 1 -- 172.21.15.121:0/4155364178 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6f0c13bc40 con 0x7f6f0c1ccfe0 2024-09-17T18:19:08.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.838+0000 7f6ef3fff640 1 --2- 172.21.15.121:0/4155364178 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6ef80401d0 0x7f6ef8042690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:08.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.838+0000 7f6ef3fff640 1 -- 172.21.15.121:0/4155364178 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 3534+0+0 (secure 0 0 0) 0x7f6efc0536c0 con 0x7f6f0c0a3a00 2024-09-17T18:19:08.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.838+0000 7f6ef3fff640 10 client.4359.objecter ms_dispatch 0x7f6f0c000b90 osd_map(17..17 src has 1..17) 2024-09-17T18:19:08.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.838+0000 7f6ef3fff640 3 client.4359.objecter handle_osd_map got epochs [17,17] > 0 2024-09-17T18:19:08.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.838+0000 7f6ef3fff640 3 client.4359.objecter handle_osd_map decoding full epoch 17 2024-09-17T18:19:08.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.838+0000 7f6ef3fff640 20 client.4359.objecter dump_active .. 0 homeless 2024-09-17T18:19:08.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.838+0000 7f6f0bfff640 1 --2- 172.21.15.121:0/4155364178 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6ef80401d0 0x7f6ef8042690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:08.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.838+0000 7f6f0bfff640 10 monclient: get_auth_request con 0x7f6ef80401d0 auth_method 0 2024-09-17T18:19:08.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.840+0000 7f6f0affd640 1 -- 172.21.15.121:0/4155364178 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f0c1ccfe0 msgr2=0x7f6f0c02d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:08.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.840+0000 7f6ef3fff640 10 client.4359.objecter ms_handle_connect 0x7f6f0c1ccfe0 2024-09-17T18:19:08.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.840+0000 7f6ef3fff640 10 client.4359.objecter resend_mon_ops 2024-09-17T18:19:08.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.840+0000 7f6f0affd640 1 --2- 172.21.15.121:0/4155364178 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f0c1ccfe0 0x7f6f0c02d440 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c09e080 tx=0x7f6f0000fce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:08.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.840+0000 7f6ef3fff640 1 -- 172.21.15.121:0/4155364178 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0000de30 con 0x7f6f0c1ccfe0 2024-09-17T18:19:08.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.840+0000 7f6ef3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:08.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.840+0000 7f6ef3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:19:08.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.840+0000 7f6ef3fff640 10 monclient: dump: 2024-09-17T18:19:08.857 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:08.858 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:08.858 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.858 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.858 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:08.858 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:08.858 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:08.858 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:08.858 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:08.858 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:08.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.841+0000 7f6ef3fff640 1 -- 172.21.15.121:0/4155364178 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6f0000de30 con 0x7f6f0c1ccfe0 2024-09-17T18:19:08.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.841+0000 7f6ef3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:08.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.841+0000 7f6ef3fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:19:08.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.841+0000 7f6ef3fff640 1 -- 172.21.15.121:0/4155364178 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f0c1ccfe0 msgr2=0x7f6f0c02d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:08.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.841+0000 7f6ef3fff640 1 --2- 172.21.15.121:0/4155364178 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f0c1ccfe0 0x7f6f0c02d440 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c09e080 tx=0x7f6f0000fce0 comp rx=0 tx=0).stop 2024-09-17T18:19:08.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.841+0000 7f6ef3fff640 1 -- 172.21.15.121:0/4155364178 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f0c1ccfe0 msgr2=0x7f6f0c02d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:19:08.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.841+0000 7f6ef3fff640 1 --2- 172.21.15.121:0/4155364178 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f0c1ccfe0 0x7f6f0c02d440 secure :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c09e080 tx=0x7f6f0000fce0 comp rx=0 tx=0).stop 2024-09-17T18:19:08.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.841+0000 7f6ef3fff640 10 client.4359.objecter ms_handle_connect 0x7f6ef80401d0 2024-09-17T18:19:08.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.841+0000 7f6f0bfff640 1 --2- 172.21.15.121:0/4155364178 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6ef80401d0 0x7f6ef8042690 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c0a7f50 tx=0x7f6f0402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:08.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.847+0000 7f6f11f00640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:19:08.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.847+0000 7f6f11f00640 1 --2- 172.21.15.121:0/4155364178 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f0c02d980 0x7f6f0c02dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:08.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.847+0000 7f6f11f00640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:08.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.847+0000 7f6f11f00640 1 -- 172.21.15.121:0/4155364178 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6f0c073700 con 0x7f6f0c02d980 2024-09-17T18:19:08.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.847+0000 7f6f0affd640 1 --2- 172.21.15.121:0/4155364178 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f0c02d980 0x7f6f0c02dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:08.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.847+0000 7f6f0affd640 10 monclient: get_auth_request con 0x7f6f0c02d980 auth_method 0 2024-09-17T18:19:08.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.847+0000 7f6f0affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:08.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.847+0000 7f6f0affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:08.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.847+0000 7f6f0affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:08.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.847+0000 7f6f0affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:08.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.847+0000 7f6f0affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:08.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.847+0000 7f6f0affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:08.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.847+0000 7f6f0affd640 10 monclient(hunting): handle_auth_done global_id 4358 payload 824 2024-09-17T18:19:08.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.847+0000 7f6f0affd640 1 -- 172.21.15.121:0/4155364178 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6f0c0a2450 con 0x7f6f0c02d980 2024-09-17T18:19:08.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.849+0000 7f6f0affd640 1 -- 172.21.15.121:0/4155364178 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f0c02d980 msgr2=0x7f6f0c02dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:08.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.849+0000 7f6ef3fff640 10 client.4359.objecter ms_handle_connect 0x7f6f0c02d980 2024-09-17T18:19:08.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.849+0000 7f6ef3fff640 10 client.4359.objecter resend_mon_ops 2024-09-17T18:19:08.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.849+0000 7f6f0affd640 1 --2- 172.21.15.121:0/4155364178 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f0c02d980 0x7f6f0c02dd90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c09e080 tx=0x7f6f000046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:08.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.849+0000 7f6ef3fff640 1 -- 172.21.15.121:0/4155364178 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f00003a30 con 0x7f6f0c02d980 2024-09-17T18:19:08.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.849+0000 7f6ef3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:08.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.849+0000 7f6ef3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:19:08.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.849+0000 7f6ef3fff640 10 monclient: dump: 2024-09-17T18:19:08.862 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:08.862 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:08.862 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.862 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.862 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:08.862 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:08.862 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:08.862 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:08.862 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:08.863 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:08.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.850+0000 7f6ef3fff640 1 -- 172.21.15.121:0/4155364178 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4719 (secure 0 0 0) 0x7f6f00003a30 con 0x7f6f0c02d980 2024-09-17T18:19:08.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.850+0000 7f6ef3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:19:08.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.850+0000 7f6ef3fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:19:08.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.850+0000 7f6ef3fff640 1 -- 172.21.15.121:0/4155364178 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f0c02d980 msgr2=0x7f6f0c02dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:08.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.850+0000 7f6ef3fff640 1 --2- 172.21.15.121:0/4155364178 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f0c02d980 0x7f6f0c02dd90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c09e080 tx=0x7f6f000046c0 comp rx=0 tx=0).stop 2024-09-17T18:19:08.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.850+0000 7f6ef3fff640 1 -- 172.21.15.121:0/4155364178 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f0c02d980 msgr2=0x7f6f0c02dd90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:19:08.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.850+0000 7f6ef3fff640 1 --2- 172.21.15.121:0/4155364178 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f0c02d980 0x7f6f0c02dd90 secure :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c09e080 tx=0x7f6f000046c0 comp rx=0 tx=0).stop 2024-09-17T18:19:08.863 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:19:08.863 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:19:08.863 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:19:08.863 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:19:08.864 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 4, 2024-09-17T18:19:08.864 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 21833, 2024-09-17T18:19:08.864 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:19:08.864 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:19:08.864 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:19:08.864 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:19:08.864 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:19:08.864 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 21, 2024-09-17T18:19:08.864 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:19:08.864 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:19:08.864 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:19:08.864 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:19:08.864 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:19:08.864 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:19:08.865 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:19:08.865 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:19:08.865 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:19:08.865 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:19:08.865 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:19:08.865 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:19:08.865 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:19:08.865 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:19:08.865 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:19:08.865 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:19:08.865 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:19:08.866 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:19:08.866 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:19:08.866 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:19:08.866 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:19:08.866 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:19:08.866 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:19:08.866 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:19:08.866 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:19:08.866 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:19:08.866 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:19:08.866 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:19:08.866 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:19:08.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:08.866 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:19:08.867 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:19:08.867 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:19:08.867 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:19:08.867 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:19:08.867 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:19:08.867 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:19:08.867 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:19:08.867 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:19:08.867 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:19:08.867 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:19:08.867 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:19:08.867 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:19:08.867 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:19:08.867 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:19:08.867 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:19:08.868 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:19:08.868 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:19:08.868 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:19:08.868 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:19:08.868 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:19:08.868 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:19:08.868 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:19:08.868 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:19:08.868 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:19:08.868 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:19:08.868 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:19:08.868 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:19:08.868 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:19:08.868 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:19:08.869 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:08.869 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:19:08.869 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:08.869 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:19:08.869 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:19:08.869 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:19:08.869 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:19:08.869 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:08.869 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:19:08.869 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:19:08.869 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:19:08.869 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:08.869 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:08.870 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:19:08.870 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:19:08.870 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:19:08.870 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:19:08.870 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:19:08.870 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:08.870 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:19:08.870 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:19:08.870 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:19:08.870 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:19:08.870 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:19:08.870 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:08.870 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:08.870 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:19:08.870 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:19:08.871 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:19:08.871 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:19:08.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:08.871 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:19:08.871 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:19:08.871 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:19:08.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:08.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:08.871 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:19:08.871 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:19:08.871 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:19:08.871 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:19:08.871 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:19:08.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:08.872 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:19:08.872 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:19:08.872 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:19:08.872 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:19:08.872 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:19:08.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:08.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:08.872 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:19:08.872 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:19:08.872 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:19:08.872 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:19:08.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:08.872 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:19:08.873 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:19:08.873 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:19:08.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:08.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:08.873 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:19:08.873 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:19:08.873 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:19:08.873 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:19:08.873 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:19:08.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:08.874 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:19:08.874 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:19:08.874 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:19:08.874 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:19:08.874 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:19:08.874 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:19:08.874 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:19:08.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:08.874 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:19:08.874 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:19:08.874 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:08.874 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:19:08.875 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:19:08.875 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:19:08.875 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:19:08.875 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:19:08.875 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:19:08.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:08.875 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:19:08.875 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:19:08.875 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:19:08.875 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:19:08.875 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:19:08.875 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:19:08.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:08.876 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:19:08.876 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:19:08.876 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:19:08.876 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:19:08.876 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:19:08.876 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:08.876 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:19:08.876 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:19:08.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.850+0000 7f6f11f00640 10 librados: watch_flush enter 2024-09-17T18:19:08.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.850+0000 7f6f11f00640 10 librados: watch_flush exit 2024-09-17T18:19:08.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.850+0000 7f6f11f00640 20 client.4359.objecter shutdown clearing up homeless session... 2024-09-17T18:19:08.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.850+0000 7f6f11f00640 10 client.4359.objecter successfully canceled tick 2024-09-17T18:19:08.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.850+0000 7f6f11f00640 1 -- 172.21.15.121:0/4155364178 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6ef80401d0 msgr2=0x7f6ef8042690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:08.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.850+0000 7f6f11f00640 1 --2- 172.21.15.121:0/4155364178 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6ef80401d0 0x7f6ef8042690 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c0a7f50 tx=0x7f6f0402e000 comp rx=0 tx=0).stop 2024-09-17T18:19:08.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.850+0000 7f6f11f00640 10 monclient: shutdown 2024-09-17T18:19:08.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.850+0000 7f6f11f00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:08.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.850+0000 7f6f11f00640 1 -- 172.21.15.121:0/4155364178 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f0c0a3a00 msgr2=0x7f6f0c1cb540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:08.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.850+0000 7f6f11f00640 1 --2- 172.21.15.121:0/4155364178 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f0c0a3a00 0x7f6f0c1cb540 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6efc00cc10 tx=0x7f6efc00a430 comp rx=0 tx=0).stop 2024-09-17T18:19:08.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.850+0000 7f6f0bfff640 1 -- 172.21.15.121:0/4155364178 reap_dead start 2024-09-17T18:19:08.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.850+0000 7f6f0bfff640 1 -- 172.21.15.121:0/4155364178 reap_dead start 2024-09-17T18:19:08.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.850+0000 7f6f11f00640 1 -- 172.21.15.121:0/4155364178 shutdown_connections 2024-09-17T18:19:08.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.850+0000 7f6f11f00640 1 -- 172.21.15.121:0/4155364178 >> 172.21.15.121:0/4155364178 conn(0x7f6f0c0a8680 msgr2=0x7f6f0c072ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:08.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.850+0000 7f6f11f00640 1 -- 172.21.15.121:0/4155364178 shutdown_connections 2024-09-17T18:19:08.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.850+0000 7f6f11f00640 1 -- 172.21.15.121:0/4155364178 wait complete. 2024-09-17T18:19:08.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.850+0000 7f6f11f00640 1 librados: shutdown 2024-09-17T18:19:08.879 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:19:08.880 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T18:19:08.880 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T18:19:08.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.974+0000 7fcb8bc23640 10 monclient: get_monmap_and_config 2024-09-17T18:19:08.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.974+0000 7fcb8bc23640 10 monclient: build_initial_monmap 2024-09-17T18:19:08.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.974+0000 7fcb8bc23640 10 monclient: monmap: 2024-09-17T18:19:08.977 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:08.977 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:08.977 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:19:08.975507+0000 2024-09-17T18:19:08.977 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:19:08.975507+0000 2024-09-17T18:19:08.977 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:08.977 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:08.977 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:19:08.977 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:19:08.977 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:19:08.977 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:19:08.978 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:19:08.978 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:08.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.975+0000 7fcb8bc23640 1 Processor -- start 2024-09-17T18:19:08.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.975+0000 7fcb8bc23640 1 -- start start 2024-09-17T18:19:08.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.975+0000 7fcb8bc23640 10 monclient: init 2024-09-17T18:19:08.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.976+0000 7fcb8bc23640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:08.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.976+0000 7fcb8bc23640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-17T18:19:08.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.976+0000 7fcb8bc23640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcb840a0a00 addr v1:172.21.15.121:6789/0 2024-09-17T18:19:08.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.976+0000 7fcb8bc23640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fcb84137c90 0x7fcb8413a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:08.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.976+0000 7fcb8bc23640 10 monclient(hunting): picked mon.noname-c con 0x7fcb84137c90 addr v2:172.21.15.173:3300/0 2024-09-17T18:19:08.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.976+0000 7fcb8bc23640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fcb8413a5c0 0x7fcb8413c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:08.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.976+0000 7fcb8bc23640 10 monclient(hunting): picked mon.noname-a con 0x7fcb8413a5c0 addr v2:172.21.15.121:3300/0 2024-09-17T18:19:08.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.976+0000 7fcb8bc23640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb840a3820 con 0x7fcb840a0a00 2024-09-17T18:19:08.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.976+0000 7fcb8bc23640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:08.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.976+0000 7fcb8bc23640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fcb840a2ea0 con 0x7fcb8413a5c0 2024-09-17T18:19:08.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.976+0000 7fcb8bc23640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:08.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.976+0000 7fcb8bc23640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fcb8409e800 con 0x7fcb84137c90 2024-09-17T18:19:08.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.976+0000 7fcb8bc23640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:08.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.976+0000 7fcb8bc23640 10 monclient(hunting): authenticate will time out at 752.692286s 2024-09-17T18:19:08.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.976+0000 7fcb899bf640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fcb840a0a00 0x7fcb840a0de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:49710/0 (socket says 172.21.15.121:49710) 2024-09-17T18:19:08.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.976+0000 7fcb899bf640 1 -- 172.21.15.121:0/603410791 learned_addr learned my addr 172.21.15.121:0/603410791 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:19:08.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.976+0000 7fcb8a1c0640 1 --2- 172.21.15.121:0/603410791 >> v2:172.21.15.121:3300/0 conn(0x7fcb8413a5c0 0x7fcb8413c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:08.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.976+0000 7fcb8a1c0640 10 monclient(hunting): get_auth_request con 0x7fcb8413a5c0 auth_method 0 2024-09-17T18:19:08.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.976+0000 7fcb8a1c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:08.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.976+0000 7fcb8a1c0640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:08.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.976+0000 7fcb8a1c0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:08.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.976+0000 7fcb8a1c0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:08.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.976+0000 7fcb8a1c0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:08.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.976+0000 7fcb8a1c0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:08.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.976+0000 7fcb891be640 1 --2- 172.21.15.121:0/603410791 >> v2:172.21.15.173:3300/0 conn(0x7fcb84137c90 0x7fcb8413a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:08.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.976+0000 7fcb891be640 10 monclient(hunting): get_auth_request con 0x7fcb84137c90 auth_method 0 2024-09-17T18:19:08.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.976+0000 7fcb891be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:08.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.976+0000 7fcb891be640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:08.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.976+0000 7fcb891be640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:08.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.976+0000 7fcb891be640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:08.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.976+0000 7fcb891be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:08.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.976+0000 7fcb891be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:08.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.976+0000 7fcb8a1c0640 10 monclient(hunting): handle_auth_done global_id 4362 payload 274 2024-09-17T18:19:08.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.976+0000 7fcb8a1c0640 1 -- 172.21.15.121:0/603410791 >> v2:172.21.15.173:3300/0 conn(0x7fcb84137c90 msgr2=0x7fcb8413a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:08.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.976+0000 7fcb8a1c0640 1 --2- 172.21.15.121:0/603410791 >> v2:172.21.15.173:3300/0 conn(0x7fcb84137c90 0x7fcb8413a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:08.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.976+0000 7fcb8a1c0640 1 -- 172.21.15.121:0/603410791 >> v1:172.21.15.121:6789/0 conn(0x7fcb840a0a00 legacy=0x7fcb840a0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:08.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.976+0000 7fcb8a1c0640 10 monclient: _finish_hunting 0 2024-09-17T18:19:08.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.976+0000 7fcb8a1c0640 1 monclient: found mon.noname-a 2024-09-17T18:19:08.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.976+0000 7fcb8a1c0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:08.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.976+0000 7fcb8a1c0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:19:08.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.976+0000 7fcb8a1c0640 1 -- 172.21.15.121:0/603410791 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb840a2c30 con 0x7fcb8413a5c0 2024-09-17T18:19:08.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.979+0000 7fcb8a1c0640 1 -- 172.21.15.121:0/603410791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb8413a5c0 msgr2=0x7fcb8413c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:08.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.979+0000 7fcb8a1c0640 1 --2- 172.21.15.121:0/603410791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb8413a5c0 0x7fcb8413c9b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcb840a3080 tx=0x7fcb7c02a4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7c889759e5195c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:08.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.979+0000 7fcb889bd640 1 -- 172.21.15.121:0/603410791 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb7c007950 con 0x7fcb8413a5c0 2024-09-17T18:19:08.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.979+0000 7fcb889bd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:08.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.979+0000 7fcb889bd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:19:08.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.979+0000 7fcb889bd640 10 monclient: dump: 2024-09-17T18:19:08.992 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:08.992 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:08.992 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.993 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.993 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:08.993 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:08.993 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:08.993 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:08.993 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:08.993 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:08.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.979+0000 7fcb889bd640 10 monclient: _finish_auth 0 2024-09-17T18:19:08.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.979+0000 7fcb889bd640 10 monclient: _check_auth_tickets 2024-09-17T18:19:08.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.979+0000 7fcb889bd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:08.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.979+0000 7fcb889bd640 1 -- 172.21.15.121:0/603410791 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb7c007af0 con 0x7fcb8413a5c0 2024-09-17T18:19:08.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.979+0000 7fcb8bc23640 5 monclient: authenticate success, global_id 4362 2024-09-17T18:19:08.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.979+0000 7fcb8bc23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:19:08.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.979+0000 7fcb889bd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:08.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.979+0000 7fcb8bc23640 10 monclient: get_monmap_and_config success 2024-09-17T18:19:08.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.979+0000 7fcb889bd640 1 -- 172.21.15.121:0/603410791 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb7c031400 con 0x7fcb8413a5c0 2024-09-17T18:19:08.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.979+0000 7fcb889bd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:08.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.979+0000 7fcb8bc23640 4 set_mon_vals no callback set 2024-09-17T18:19:08.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.979+0000 7fcb889bd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:19:08.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.979+0000 7fcb889bd640 10 monclient: dump: 2024-09-17T18:19:08.994 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:08.994 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:08.994 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.994 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:08.994 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:08.995 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:08.995 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:08.995 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:08.995 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:08.995 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:08.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.980+0000 7fcb8bc23640 10 monclient: shutdown 2024-09-17T18:19:08.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.980+0000 7fcb8bc23640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:08.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.980+0000 7fcb8bc23640 1 -- 172.21.15.121:0/603410791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb8413a5c0 msgr2=0x7fcb8413c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:08.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.980+0000 7fcb8bc23640 1 --2- 172.21.15.121:0/603410791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb8413a5c0 0x7fcb8413c9b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcb840a3080 tx=0x7fcb7c02a4d0 comp rx=0 tx=0).stop 2024-09-17T18:19:08.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.980+0000 7fcb8bc23640 1 -- 172.21.15.121:0/603410791 shutdown_connections 2024-09-17T18:19:08.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.980+0000 7fcb8bc23640 1 --2- 172.21.15.121:0/603410791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb8413a5c0 0x7fcb8413c9b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:08.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.980+0000 7fcb8bc23640 1 --2- 172.21.15.121:0/603410791 >> v2:172.21.15.173:3300/0 conn(0x7fcb84137c90 0x7fcb8413a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:08.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.980+0000 7fcb8bc23640 1 -- 172.21.15.121:0/603410791 >> 172.21.15.121:0/603410791 conn(0x7fcb840a7830 msgr2=0x7fcb840a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:08.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.980+0000 7fcb8bc23640 1 -- 172.21.15.121:0/603410791 shutdown_connections 2024-09-17T18:19:08.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.980+0000 7fcb8bc23640 1 -- 172.21.15.121:0/603410791 wait complete. 2024-09-17T18:19:08.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.980+0000 7fcb8bc23640 10 monclient: build_initial_monmap 2024-09-17T18:19:08.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.980+0000 7fcb8bc23640 10 monclient: monmap: 2024-09-17T18:19:08.996 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:08.996 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:19:08.996 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:19:08.996 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:19:08.996 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:08.996 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:08.996 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:19:08.996 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:19:08.997 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:19:08.997 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:08.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb8bc23640 1 librados: starting msgr at 2024-09-17T18:19:08.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb8bc23640 1 librados: starting objecter 2024-09-17T18:19:08.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb8bc23640 1 Processor -- start 2024-09-17T18:19:08.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb8bc23640 1 -- start start 2024-09-17T18:19:08.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb8bc23640 1 librados: setting wanted keys 2024-09-17T18:19:08.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb8bc23640 1 librados: calling monclient init 2024-09-17T18:19:08.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb8bc23640 10 monclient: init 2024-09-17T18:19:08.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb8bc23640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:08.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb8bc23640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T18:19:08.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb8bc23640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb840a0a00 0x7fcb84073ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:08.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb8bc23640 10 monclient(hunting): picked mon.noname-a con 0x7fcb840a0a00 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:08.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb8bc23640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb84137c90 0x7fcb841347b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:08.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb8bc23640 10 monclient(hunting): picked mon.noname-c con 0x7fcb84137c90 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:08.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb8bc23640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb8413a5c0 0x7fcb84134cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:08.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb8bc23640 10 monclient(hunting): picked mon.noname-b con 0x7fcb8413a5c0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:08.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb8bc23640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:08.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb8bc23640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcb8409de60 con 0x7fcb840a0a00 2024-09-17T18:19:08.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb8bc23640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:08.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb8bc23640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcb8409dce0 con 0x7fcb8413a5c0 2024-09-17T18:19:08.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb8bc23640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:08.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb8bc23640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcb840a3120 con 0x7fcb84137c90 2024-09-17T18:19:08.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb8bc23640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:08.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb8bc23640 10 monclient(hunting): authenticate will time out at 752.697651s 2024-09-17T18:19:08.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb899bf640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb840a0a00 0x7fcb84073ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:08.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb891be640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb84137c90 0x7fcb841347b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:08.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb899bf640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb840a0a00 0x7fcb84073ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41276/0 (socket says 172.21.15.121:41276) 2024-09-17T18:19:08.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb891be640 10 monclient(hunting): get_auth_request con 0x7fcb84137c90 auth_method 0 2024-09-17T18:19:08.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb891be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:08.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb891be640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:08.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb899bf640 1 -- 172.21.15.121:0/1274449169 learned_addr learned my addr 172.21.15.121:0/1274449169 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:08.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb891be640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:08.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb899bf640 10 monclient(hunting): get_auth_request con 0x7fcb840a0a00 auth_method 0 2024-09-17T18:19:08.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb899bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:08.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb899bf640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:08.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb899bf640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:08.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb899bf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:08.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb899bf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:08.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb899bf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:09.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb891be640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:09.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb891be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:09.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb891be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:09.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb8a1c0640 1 --2- 172.21.15.121:0/1274449169 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb8413a5c0 0x7fcb84134cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:09.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb8a1c0640 10 monclient(hunting): get_auth_request con 0x7fcb8413a5c0 auth_method 0 2024-09-17T18:19:09.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb8a1c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:09.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb8a1c0640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:09.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.981+0000 7fcb8a1c0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:09.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.982+0000 7fcb8a1c0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:09.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.982+0000 7fcb8a1c0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:09.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.982+0000 7fcb8a1c0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:09.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.982+0000 7fcb899bf640 10 monclient(hunting): handle_auth_done global_id 4365 payload 824 2024-09-17T18:19:09.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.982+0000 7fcb899bf640 1 -- 172.21.15.121:0/1274449169 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb84137c90 msgr2=0x7fcb841347b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:09.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.982+0000 7fcb899bf640 1 --2- 172.21.15.121:0/1274449169 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb84137c90 0x7fcb841347b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:09.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.982+0000 7fcb899bf640 1 -- 172.21.15.121:0/1274449169 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb8413a5c0 msgr2=0x7fcb84134cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:09.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.982+0000 7fcb899bf640 1 --2- 172.21.15.121:0/1274449169 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb8413a5c0 0x7fcb84134cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:09.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.982+0000 7fcb899bf640 10 monclient: _finish_hunting 0 2024-09-17T18:19:09.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.982+0000 7fcb899bf640 1 monclient: found mon.noname-a 2024-09-17T18:19:09.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.982+0000 7fcb899bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:09.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.982+0000 7fcb899bf640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:19:09.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.982+0000 7fcb899bf640 1 -- 172.21.15.121:0/1274449169 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb84137300 con 0x7fcb840a0a00 2024-09-17T18:19:09.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.982+0000 7fcb891be640 1 --2- 172.21.15.121:0/1274449169 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb84137c90 0x7fcb841347b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:19:09.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.984+0000 7fcb899bf640 1 -- 172.21.15.121:0/1274449169 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb840a0a00 msgr2=0x7fcb84073ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:09.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.984+0000 7fcb899bf640 1 --2- 172.21.15.121:0/1274449169 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb840a0a00 0x7fcb84073ca0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcb8000c460 tx=0x7fcb8000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:09.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.984+0000 7fcb7a7fc640 10 client.?.objecter ms_handle_connect 0x7fcb840a0a00 2024-09-17T18:19:09.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.984+0000 7fcb7a7fc640 10 client.?.objecter resend_mon_ops 2024-09-17T18:19:09.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.984+0000 7fcb7a7fc640 1 -- 172.21.15.121:0/1274449169 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb80007070 con 0x7fcb840a0a00 2024-09-17T18:19:09.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.984+0000 7fcb7a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:09.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.984+0000 7fcb7a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:19:09.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.984+0000 7fcb7a7fc640 10 monclient: dump: 2024-09-17T18:19:09.002 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:09.002 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:09.002 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:09.002 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:09.002 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:09.002 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:09.002 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:09.002 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:09.002 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:09.002 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:09.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.984+0000 7fcb7a7fc640 10 monclient: _finish_auth 0 2024-09-17T18:19:09.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.984+0000 7fcb7a7fc640 10 monclient: _check_auth_tickets 2024-09-17T18:19:09.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.984+0000 7fcb7a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:09.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.984+0000 7fcb8bc23640 5 monclient: authenticate success, global_id 4365 2024-09-17T18:19:09.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.984+0000 7fcb7a7fc640 1 -- 172.21.15.121:0/1274449169 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb80007230 con 0x7fcb840a0a00 2024-09-17T18:19:09.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.984+0000 7fcb8bc23640 10 monclient: _renew_subs 2024-09-17T18:19:09.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.984+0000 7fcb8bc23640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:09.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.984+0000 7fcb8bc23640 1 -- 172.21.15.121:0/1274449169 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb8409e240 con 0x7fcb840a0a00 2024-09-17T18:19:09.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.984+0000 7fcb7a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:09.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.984+0000 7fcb7a7fc640 1 -- 172.21.15.121:0/1274449169 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb80007620 con 0x7fcb840a0a00 2024-09-17T18:19:09.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.984+0000 7fcb7a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:09.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.984+0000 7fcb7a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:19:09.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.984+0000 7fcb7a7fc640 10 monclient: dump: 2024-09-17T18:19:09.003 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:09.003 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:09.003 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:09.003 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:09.003 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:09.003 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:09.004 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:09.004 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:09.004 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:09.004 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:09.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.984+0000 7fcb7bfff640 4 set_mon_vals no callback set 2024-09-17T18:19:09.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.984+0000 7fcb8bc23640 10 client.4365.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:09.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.984+0000 7fcb8bc23640 10 monclient: _renew_subs 2024-09-17T18:19:09.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.984+0000 7fcb8bc23640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:09.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.984+0000 7fcb8bc23640 1 -- 172.21.15.121:0/1274449169 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb84122e40 con 0x7fcb840a0a00 2024-09-17T18:19:09.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.984+0000 7fcb8bc23640 1 librados: init done 2024-09-17T18:19:09.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.985+0000 7fcb8bc23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:09.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.985+0000 7fcb8bc23640 1 --2- 172.21.15.121:0/1274449169 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb84122270 0x7fcb8402d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:09.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.985+0000 7fcb8bc23640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:09.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.985+0000 7fcb8bc23640 1 -- 172.21.15.121:0/1274449169 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcb84124d70 con 0x7fcb84122270 2024-09-17T18:19:09.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.985+0000 7fcb899bf640 1 --2- 172.21.15.121:0/1274449169 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb840a0a00 0x7fcb84073ca0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcb8000c460 tx=0x7fcb8000c920 comp rx=0 tx=0).queue_received will delay after 1.7266e+09 on 0x7fcb80052c00 osd_map(17..17 src has 1..17) 2024-09-17T18:19:09.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.985+0000 7fcb7a7fc640 1 -- 172.21.15.121:0/1274449169 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fcb8001b400 con 0x7fcb840a0a00 2024-09-17T18:19:09.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.985+0000 7fcb891be640 1 --2- 172.21.15.121:0/1274449169 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb84122270 0x7fcb8402d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:09.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.985+0000 7fcb891be640 10 monclient: get_auth_request con 0x7fcb84122270 auth_method 0 2024-09-17T18:19:09.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.985+0000 7fcb891be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:09.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.985+0000 7fcb891be640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:09.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.985+0000 7fcb891be640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:09.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.985+0000 7fcb891be640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:09.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.985+0000 7fcb891be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:09.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.985+0000 7fcb891be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:09.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.985+0000 7fcb891be640 10 monclient(hunting): handle_auth_done global_id 4354 payload 824 2024-09-17T18:19:09.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.985+0000 7fcb7a7fc640 1 --2- 172.21.15.121:0/1274449169 >> v2:172.21.15.173:6812/2423786037 conn(0x7fcb68040220 0x7fcb680426e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:09.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.985+0000 7fcb891be640 1 -- 172.21.15.121:0/1274449169 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcb840a1f10 con 0x7fcb84122270 2024-09-17T18:19:09.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.985+0000 7fcb8a1c0640 1 --2- 172.21.15.121:0/1274449169 >> v2:172.21.15.173:6812/2423786037 conn(0x7fcb68040220 0x7fcb680426e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:09.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.985+0000 7fcb8a1c0640 10 monclient: get_auth_request con 0x7fcb68040220 auth_method 0 2024-09-17T18:19:09.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.988+0000 7fcb891be640 1 -- 172.21.15.121:0/1274449169 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb84122270 msgr2=0x7fcb8402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:09.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.988+0000 7fcb7a7fc640 10 client.4365.objecter ms_handle_connect 0x7fcb84122270 2024-09-17T18:19:09.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.988+0000 7fcb891be640 1 --2- 172.21.15.121:0/1274449169 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb84122270 0x7fcb8402d440 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcb74011cf0 tx=0x7fcb7400f610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:09.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.988+0000 7fcb7a7fc640 10 client.4365.objecter resend_mon_ops 2024-09-17T18:19:09.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.988+0000 7fcb7a7fc640 1 -- 172.21.15.121:0/1274449169 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb74004030 con 0x7fcb84122270 2024-09-17T18:19:09.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.988+0000 7fcb7a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:09.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.988+0000 7fcb7a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:19:09.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.988+0000 7fcb7a7fc640 10 monclient: dump: 2024-09-17T18:19:09.006 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:09.006 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:09.006 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:09.006 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:09.006 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:09.007 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:09.007 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:09.007 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:09.007 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:09.007 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:09.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.988+0000 7fcb7a7fc640 10 client.4365.objecter ms_handle_connect 0x7fcb68040220 2024-09-17T18:19:09.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.988+0000 7fcb8a1c0640 1 --2- 172.21.15.121:0/1274449169 >> v2:172.21.15.173:6812/2423786037 conn(0x7fcb68040220 0x7fcb680426e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c034000 tx=0x7fcb7c02f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:09.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.988+0000 7fcb7a7fc640 1 -- 172.21.15.121:0/1274449169 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcb74004030 con 0x7fcb84122270 2024-09-17T18:19:09.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.988+0000 7fcb7a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:09.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.988+0000 7fcb7a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:19:09.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.988+0000 7fcb7a7fc640 1 -- 172.21.15.121:0/1274449169 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb84122270 msgr2=0x7fcb8402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:09.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.988+0000 7fcb7a7fc640 1 --2- 172.21.15.121:0/1274449169 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb84122270 0x7fcb8402d440 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcb74011cf0 tx=0x7fcb7400f610 comp rx=0 tx=0).stop 2024-09-17T18:19:09.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.988+0000 7fcb7a7fc640 1 -- 172.21.15.121:0/1274449169 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb84122270 msgr2=0x7fcb8402d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:19:09.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.988+0000 7fcb7a7fc640 1 --2- 172.21.15.121:0/1274449169 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb84122270 0x7fcb8402d440 secure :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcb74011cf0 tx=0x7fcb7400f610 comp rx=0 tx=0).stop 2024-09-17T18:19:09.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.994+0000 7fcb8bc23640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:19:09.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.994+0000 7fcb8bc23640 1 --2- 172.21.15.121:0/1274449169 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb84122650 0x7fcb8402d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:09.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.994+0000 7fcb8bc23640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:09.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.994+0000 7fcb8bc23640 1 -- 172.21.15.121:0/1274449169 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcb840709d0 con 0x7fcb84122650 2024-09-17T18:19:09.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.994+0000 7fcb891be640 1 --2- 172.21.15.121:0/1274449169 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb84122650 0x7fcb8402d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:09.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.995+0000 7fcb891be640 10 monclient: get_auth_request con 0x7fcb84122650 auth_method 0 2024-09-17T18:19:09.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.995+0000 7fcb891be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:09.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.995+0000 7fcb891be640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:09.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.995+0000 7fcb891be640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:09.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.995+0000 7fcb891be640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:09.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.995+0000 7fcb891be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:09.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.995+0000 7fcb891be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:09.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.995+0000 7fcb891be640 10 monclient(hunting): handle_auth_done global_id 4357 payload 824 2024-09-17T18:19:09.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.995+0000 7fcb891be640 1 -- 172.21.15.121:0/1274449169 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcb84135770 con 0x7fcb84122650 2024-09-17T18:19:09.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.997+0000 7fcb891be640 1 -- 172.21.15.121:0/1274449169 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb84122650 msgr2=0x7fcb8402d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:09.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.997+0000 7fcb891be640 1 --2- 172.21.15.121:0/1274449169 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb84122650 0x7fcb8402d980 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcb74011cf0 tx=0x7fcb740046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:09.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.997+0000 7fcb7a7fc640 10 client.4365.objecter ms_handle_connect 0x7fcb84122650 2024-09-17T18:19:09.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.997+0000 7fcb7a7fc640 10 client.4365.objecter resend_mon_ops 2024-09-17T18:19:09.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.997+0000 7fcb7a7fc640 1 -- 172.21.15.121:0/1274449169 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb74003c30 con 0x7fcb84122650 2024-09-17T18:19:09.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.997+0000 7fcb7a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:09.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.997+0000 7fcb7a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:19:09.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.997+0000 7fcb7a7fc640 10 monclient: dump: 2024-09-17T18:19:09.011 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:09.011 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:09.011 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:09.011 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:09.011 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:09.011 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:09.011 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:09.011 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:09.011 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:09.011 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:09.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.998+0000 7fcb7a7fc640 1 -- 172.21.15.121:0/1274449169 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4886 (secure 0 0 0) 0x7fcb74003c30 con 0x7fcb84122650 2024-09-17T18:19:09.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.998+0000 7fcb7a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:19:09.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.998+0000 7fcb7a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:19:09.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.998+0000 7fcb7a7fc640 1 -- 172.21.15.121:0/1274449169 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb84122650 msgr2=0x7fcb8402d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:09.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.998+0000 7fcb7a7fc640 1 --2- 172.21.15.121:0/1274449169 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb84122650 0x7fcb8402d980 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcb74011cf0 tx=0x7fcb740046c0 comp rx=0 tx=0).stop 2024-09-17T18:19:09.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.998+0000 7fcb7a7fc640 1 -- 172.21.15.121:0/1274449169 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb84122650 msgr2=0x7fcb8402d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:19:09.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.998+0000 7fcb7a7fc640 1 --2- 172.21.15.121:0/1274449169 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb84122650 0x7fcb8402d980 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcb74011cf0 tx=0x7fcb740046c0 comp rx=0 tx=0).stop 2024-09-17T18:19:09.012 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:19:09.012 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:19:09.012 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:19:09.012 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:19:09.012 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 4, 2024-09-17T18:19:09.012 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 21943, 2024-09-17T18:19:09.012 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:19:09.012 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:19:09.012 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:19:09.013 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:19:09.013 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:19:09.013 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 21, 2024-09-17T18:19:09.013 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:19:09.013 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:19:09.013 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:19:09.013 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:19:09.013 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:19:09.013 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:19:09.013 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:19:09.013 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:19:09.013 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:19:09.013 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:19:09.013 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:19:09.014 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:19:09.014 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:19:09.014 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:19:09.014 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:19:09.014 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:19:09.014 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:19:09.014 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:19:09.014 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:19:09.014 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:19:09.014 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:19:09.014 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:19:09.015 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:19:09.015 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:19:09.015 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:19:09.015 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:19:09.015 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:19:09.015 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:19:09.015 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:19:09.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:09.015 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:19:09.015 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:19:09.015 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:19:09.015 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:19:09.015 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:19:09.015 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:19:09.016 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:19:09.016 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:19:09.016 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:19:09.016 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:19:09.016 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:19:09.016 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:19:09.016 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:19:09.016 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:19:09.016 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:19:09.016 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:19:09.016 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:19:09.016 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:19:09.016 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:19:09.017 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:19:09.017 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:19:09.017 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:19:09.017 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:19:09.017 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:19:09.017 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:19:09.017 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:19:09.017 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:19:09.017 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:19:09.017 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:19:09.017 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:19:09.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:09.017 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:19:09.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:09.018 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:19:09.018 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:19:09.018 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:19:09.018 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:19:09.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:09.018 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:19:09.018 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:19:09.018 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:19:09.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:09.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:09.018 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:19:09.018 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:19:09.018 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:19:09.018 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:19:09.019 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:19:09.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:09.019 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:19:09.019 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:19:09.019 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:19:09.019 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:19:09.019 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:19:09.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:09.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:09.019 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:19:09.019 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:19:09.019 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:19:09.019 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:19:09.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:09.020 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:19:09.020 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:19:09.020 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:19:09.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:09.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:09.020 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:19:09.020 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:19:09.020 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:19:09.020 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:19:09.020 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:19:09.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:09.020 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:19:09.020 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:19:09.020 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:19:09.021 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:19:09.021 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:19:09.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:09.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:09.021 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:19:09.021 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:19:09.021 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:19:09.021 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:19:09.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:09.021 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:19:09.021 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:19:09.021 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:19:09.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:09.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:09.022 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:19:09.022 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:19:09.022 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:19:09.022 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:19:09.022 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:19:09.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:09.022 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:19:09.022 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:19:09.022 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:19:09.022 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:19:09.022 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:19:09.022 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:19:09.022 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:19:09.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:09.023 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:19:09.023 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:19:09.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:09.023 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:19:09.023 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:19:09.023 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:19:09.023 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:19:09.023 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:19:09.023 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:19:09.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:09.023 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:19:09.023 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:19:09.023 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 3 2024-09-17T18:19:09.023 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:19:09.024 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:19:09.024 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:19:09.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:09.024 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:19:09.024 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:19:09.024 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 5 2024-09-17T18:19:09.024 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:19:09.024 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:19:09.024 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": [ 2024-09-17T18:19:09.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:09.024 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:19:09.024 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:19:09.024 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:19:09.024 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:19:09.025 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:19:09.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:09.025 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:19:09.025 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:19:09.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.998+0000 7fcb8bc23640 10 librados: watch_flush enter 2024-09-17T18:19:09.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.998+0000 7fcb8bc23640 10 librados: watch_flush exit 2024-09-17T18:19:09.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.998+0000 7fcb8bc23640 20 client.4365.objecter shutdown clearing up homeless session... 2024-09-17T18:19:09.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.998+0000 7fcb8bc23640 10 client.4365.objecter successfully canceled tick 2024-09-17T18:19:09.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.998+0000 7fcb8bc23640 1 -- 172.21.15.121:0/1274449169 >> v2:172.21.15.173:6812/2423786037 conn(0x7fcb68040220 msgr2=0x7fcb680426e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:09.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.998+0000 7fcb8bc23640 1 --2- 172.21.15.121:0/1274449169 >> v2:172.21.15.173:6812/2423786037 conn(0x7fcb68040220 0x7fcb680426e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c034000 tx=0x7fcb7c02f000 comp rx=0 tx=0).stop 2024-09-17T18:19:09.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.998+0000 7fcb8bc23640 10 monclient: shutdown 2024-09-17T18:19:09.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.998+0000 7fcb8bc23640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:09.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.998+0000 7fcb8bc23640 1 -- 172.21.15.121:0/1274449169 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb840a0a00 msgr2=0x7fcb84073ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:09.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.998+0000 7fcb8bc23640 1 --2- 172.21.15.121:0/1274449169 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb840a0a00 0x7fcb84073ca0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcb8000c460 tx=0x7fcb8000c920 comp rx=0 tx=0).stop 2024-09-17T18:19:09.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.998+0000 7fcb8a1c0640 1 -- 172.21.15.121:0/1274449169 reap_dead start 2024-09-17T18:19:09.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.998+0000 7fcb7a7fc640 1 -- 172.21.15.121:0/1274449169 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 3534+0+0 (unknown 0 0 0) 0x7fcb80052c00 con 0x7fcb840a0a00 2024-09-17T18:19:09.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.998+0000 7fcb7a7fc640 10 client.4365.objecter ms_dispatch 0x7fcb84000b90 osd_map(17..17 src has 1..17) 2024-09-17T18:19:09.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.998+0000 7fcb7a7fc640 10 librados: disconnected, discarding osd_map(17..17 src has 1..17) 2024-09-17T18:19:09.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.998+0000 7fcb8a1c0640 1 -- 172.21.15.121:0/1274449169 reap_dead start 2024-09-17T18:19:09.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.998+0000 7fcb8bc23640 1 -- 172.21.15.121:0/1274449169 shutdown_connections 2024-09-17T18:19:09.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.998+0000 7fcb8bc23640 1 -- 172.21.15.121:0/1274449169 >> 172.21.15.121:0/1274449169 conn(0x7fcb840a7830 msgr2=0x7fcb841360a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:09.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.998+0000 7fcb8bc23640 1 -- 172.21.15.121:0/1274449169 shutdown_connections 2024-09-17T18:19:09.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.998+0000 7fcb8bc23640 1 -- 172.21.15.121:0/1274449169 wait complete. 2024-09-17T18:19:09.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:08.998+0000 7fcb8bc23640 1 librados: shutdown 2024-09-17T18:19:09.028 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:19:09.029 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-17T18:19:09.029 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-17T18:19:09.029 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-17T18:19:09.029 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c store 2024-09-17T18:19:09.029 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c sync_force --yes-i-really-mean-it 2024-09-17T18:19:09.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.125+0000 7fbfc5352640 10 monclient: get_monmap_and_config 2024-09-17T18:19:09.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.125+0000 7fbfc5352640 10 monclient: build_initial_monmap 2024-09-17T18:19:09.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.125+0000 7fbfc5352640 10 monclient: monmap: 2024-09-17T18:19:09.127 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:09.127 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:09.128 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:19:09.126285+0000 2024-09-17T18:19:09.128 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:19:09.126285+0000 2024-09-17T18:19:09.128 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:09.128 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:09.128 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:19:09.128 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:19:09.128 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:19:09.128 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:19:09.128 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:19:09.128 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:09.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.126+0000 7fbfc5352640 1 Processor -- start 2024-09-17T18:19:09.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.126+0000 7fbfc5352640 1 -- start start 2024-09-17T18:19:09.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.126+0000 7fbfc5352640 10 monclient: init 2024-09-17T18:19:09.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.126+0000 7fbfc5352640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:09.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.126+0000 7fbfc5352640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-17T18:19:09.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.126+0000 7fbfc5352640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbfc00a5c00 addr v1:172.21.15.173:6789/0 2024-09-17T18:19:09.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.126+0000 7fbfc5352640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbfc00a5fe0 addr v1:172.21.15.121:6789/0 2024-09-17T18:19:09.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.126+0000 7fbfc5352640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fbfc013ec80 0x7fbfc0141070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:09.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.126+0000 7fbfc5352640 10 monclient(hunting): picked mon.noname-c con 0x7fbfc013ec80 addr v2:172.21.15.173:3300/0 2024-09-17T18:19:09.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.126+0000 7fbfc5352640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfc009d510 con 0x7fbfc00a5fe0 2024-09-17T18:19:09.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.126+0000 7fbfc5352640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfc00a7480 con 0x7fbfc00a5c00 2024-09-17T18:19:09.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.126+0000 7fbfc5352640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:09.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.126+0000 7fbfc5352640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fbfc009cfe0 con 0x7fbfc013ec80 2024-09-17T18:19:09.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.126+0000 7fbfc5352640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:09.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.126+0000 7fbfc5352640 10 monclient(hunting): authenticate will time out at 752.842865s 2024-09-17T18:19:09.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.126+0000 7fbfb67fc640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fbfc00a5fe0 0x7fbfc00a63f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:49844/0 (socket says 172.21.15.121:49844) 2024-09-17T18:19:09.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.126+0000 7fbfb67fc640 1 -- 172.21.15.121:0/2708250252 learned_addr learned my addr 172.21.15.121:0/2708250252 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:19:09.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.127+0000 7fbfbf7fe640 1 --2- 172.21.15.121:0/2708250252 >> v2:172.21.15.173:3300/0 conn(0x7fbfc013ec80 0x7fbfc0141070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:09.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.127+0000 7fbfbf7fe640 10 monclient(hunting): get_auth_request con 0x7fbfc013ec80 auth_method 0 2024-09-17T18:19:09.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.127+0000 7fbfbf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:09.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.127+0000 7fbfbf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:09.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.127+0000 7fbfbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:09.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.127+0000 7fbfbf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:09.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.127+0000 7fbfbf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:09.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.127+0000 7fbfbf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:09.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.127+0000 7fbfbf7fe640 10 monclient(hunting): handle_auth_done global_id 4360 payload 274 2024-09-17T18:19:09.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.127+0000 7fbfbf7fe640 1 -- 172.21.15.121:0/2708250252 >> v1:172.21.15.173:6789/0 conn(0x7fbfc00a5c00 legacy=0x7fbfc009d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:09.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.127+0000 7fbfbf7fe640 1 -- 172.21.15.121:0/2708250252 >> v1:172.21.15.121:6789/0 conn(0x7fbfc00a5fe0 legacy=0x7fbfc00a63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:09.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.127+0000 7fbfbf7fe640 10 monclient: _finish_hunting 0 2024-09-17T18:19:09.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.127+0000 7fbfbf7fe640 1 monclient: found mon.noname-c 2024-09-17T18:19:09.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.127+0000 7fbfbf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:09.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.127+0000 7fbfbf7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.173:3300/0 2024-09-17T18:19:09.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.127+0000 7fbfbf7fe640 1 -- 172.21.15.121:0/2708250252 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfc00a8ae0 con 0x7fbfc013ec80 2024-09-17T18:19:09.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.130+0000 7fbfbf7fe640 1 -- 172.21.15.121:0/2708250252 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbfc013ec80 msgr2=0x7fbfc0141070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:09.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.130+0000 7fbfbf7fe640 1 --2- 172.21.15.121:0/2708250252 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbfc013ec80 0x7fbfc0141070 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbfb8009d90 tx=0x7fbfb802a530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9d10bb14f6c53aa server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:09.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.130+0000 7fbfbe7fc640 1 -- 172.21.15.121:0/2708250252 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb8007a30 con 0x7fbfc013ec80 2024-09-17T18:19:09.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.130+0000 7fbfbe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:09.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.130+0000 7fbfbe7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:19:09.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.130+0000 7fbfbe7fc640 10 monclient: dump: 2024-09-17T18:19:09.141 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:09.141 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:09.141 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:09.141 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:09.141 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:09.141 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:09.141 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:09.141 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:09.141 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:09.141 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:09.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.130+0000 7fbfbe7fc640 10 monclient: _finish_auth 0 2024-09-17T18:19:09.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.130+0000 7fbfbe7fc640 10 monclient: _check_auth_tickets 2024-09-17T18:19:09.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.130+0000 7fbfbe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:09.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.130+0000 7fbfbe7fc640 1 -- 172.21.15.121:0/2708250252 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfb8007bd0 con 0x7fbfc013ec80 2024-09-17T18:19:09.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.130+0000 7fbfc5352640 5 monclient: authenticate success, global_id 4360 2024-09-17T18:19:09.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.130+0000 7fbfc5352640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:19:09.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.130+0000 7fbfbe7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:09.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.130+0000 7fbfbe7fc640 1 -- 172.21.15.121:0/2708250252 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb8031400 con 0x7fbfc013ec80 2024-09-17T18:19:09.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.130+0000 7fbfc5352640 10 monclient: get_monmap_and_config success 2024-09-17T18:19:09.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.130+0000 7fbfc5352640 4 set_mon_vals no callback set 2024-09-17T18:19:09.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.130+0000 7fbfbe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:09.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.130+0000 7fbfbe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:19:09.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.130+0000 7fbfbe7fc640 10 monclient: dump: 2024-09-17T18:19:09.142 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:09.142 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:09.143 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:09.143 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:09.143 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:09.143 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:09.143 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:09.143 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:09.143 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:09.143 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:09.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.131+0000 7fbfc5352640 10 monclient: shutdown 2024-09-17T18:19:09.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.131+0000 7fbfc5352640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:09.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.131+0000 7fbfc5352640 1 -- 172.21.15.121:0/2708250252 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbfc013ec80 msgr2=0x7fbfc0141070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:09.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.131+0000 7fbfc5352640 1 --2- 172.21.15.121:0/2708250252 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbfc013ec80 0x7fbfc0141070 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbfb8009d90 tx=0x7fbfb802a530 comp rx=0 tx=0).stop 2024-09-17T18:19:09.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.131+0000 7fbfc5352640 1 -- 172.21.15.121:0/2708250252 shutdown_connections 2024-09-17T18:19:09.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.131+0000 7fbfc5352640 1 --2- 172.21.15.121:0/2708250252 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbfc013ec80 0x7fbfc0141070 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:09.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.131+0000 7fbfc5352640 1 -- 172.21.15.121:0/2708250252 >> 172.21.15.121:0/2708250252 conn(0x7fbfc00ab9e0 msgr2=0x7fbfc00a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:09.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.131+0000 7fbfc5352640 1 -- 172.21.15.121:0/2708250252 shutdown_connections 2024-09-17T18:19:09.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.131+0000 7fbfc5352640 1 -- 172.21.15.121:0/2708250252 wait complete. 2024-09-17T18:19:09.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.131+0000 7fbfc5352640 10 monclient: build_initial_monmap 2024-09-17T18:19:09.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.131+0000 7fbfc5352640 10 monclient: monmap: 2024-09-17T18:19:09.144 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:09.144 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:19:09.144 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:19:09.144 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:19:09.144 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:09.144 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:09.144 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:19:09.144 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:19:09.144 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:19:09.144 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:09.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.131+0000 7fbfc5352640 1 librados: starting msgr at 2024-09-17T18:19:09.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.131+0000 7fbfc5352640 1 librados: starting objecter 2024-09-17T18:19:09.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.132+0000 7fbfc5352640 1 Processor -- start 2024-09-17T18:19:09.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.132+0000 7fbfc5352640 1 -- start start 2024-09-17T18:19:09.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.132+0000 7fbfc5352640 1 librados: setting wanted keys 2024-09-17T18:19:09.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.132+0000 7fbfc5352640 1 librados: calling monclient init 2024-09-17T18:19:09.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.132+0000 7fbfc5352640 10 monclient: init 2024-09-17T18:19:09.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.132+0000 7fbfc5352640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:09.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.132+0000 7fbfc5352640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T18:19:09.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.132+0000 7fbfc5352640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbfc00a5c00 0x7fbfc01cd950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:09.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.132+0000 7fbfc5352640 10 monclient(hunting): picked mon.noname-b con 0x7fbfc00a5c00 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:09.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.132+0000 7fbfc5352640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbfc00a5fe0 0x7fbfc01ce120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:09.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.132+0000 7fbfc5352640 10 monclient(hunting): picked mon.noname-a con 0x7fbfc00a5fe0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:09.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.132+0000 7fbfc5352640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbfc013ec80 0x7fbfc013bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:09.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.132+0000 7fbfc5352640 10 monclient(hunting): picked mon.noname-c con 0x7fbfc013ec80 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:09.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.132+0000 7fbfc5352640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:09.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.132+0000 7fbfc5352640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc01415b0 con 0x7fbfc00a5fe0 2024-09-17T18:19:09.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.132+0000 7fbfc5352640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:09.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.132+0000 7fbfc5352640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc009cd00 con 0x7fbfc00a5c00 2024-09-17T18:19:09.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.132+0000 7fbfc5352640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:09.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.132+0000 7fbfc5352640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbfc0141730 con 0x7fbfc013ec80 2024-09-17T18:19:09.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.132+0000 7fbfc5352640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:09.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.132+0000 7fbfc5352640 10 monclient(hunting): authenticate will time out at 752.848739s 2024-09-17T18:19:09.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.132+0000 7fbfb67fc640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbfc00a5fe0 0x7fbfc01ce120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:09.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.132+0000 7fbfb67fc640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbfc00a5fe0 0x7fbfc01ce120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43634/0 (socket says 172.21.15.121:43634) 2024-09-17T18:19:09.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.132+0000 7fbfb67fc640 1 -- 172.21.15.121:0/1820598356 learned_addr learned my addr 172.21.15.121:0/1820598356 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:09.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.132+0000 7fbfb67fc640 10 monclient(hunting): get_auth_request con 0x7fbfc00a5fe0 auth_method 0 2024-09-17T18:19:09.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.132+0000 7fbfb67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:09.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.132+0000 7fbfb67fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:09.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.132+0000 7fbfb67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:09.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.132+0000 7fbfbf7fe640 1 --2- 172.21.15.121:0/1820598356 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbfc013ec80 0x7fbfc013bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:09.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.132+0000 7fbfbf7fe640 10 monclient(hunting): get_auth_request con 0x7fbfc013ec80 auth_method 0 2024-09-17T18:19:09.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.132+0000 7fbfbf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:09.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.132+0000 7fbfbf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:09.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.132+0000 7fbfbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:09.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.132+0000 7fbfb67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:09.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.132+0000 7fbfb67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:09.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.132+0000 7fbfb67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:09.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.132+0000 7fbfbf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:09.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.132+0000 7fbfbf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:09.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.132+0000 7fbfbf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:09.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.133+0000 7fbfbeffd640 1 --2- 172.21.15.121:0/1820598356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbfc00a5c00 0x7fbfc01cd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:09.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.133+0000 7fbfbeffd640 10 monclient(hunting): get_auth_request con 0x7fbfc00a5c00 auth_method 0 2024-09-17T18:19:09.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.133+0000 7fbfbeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:09.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.133+0000 7fbfbeffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:09.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.133+0000 7fbfbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:09.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.133+0000 7fbfbf7fe640 10 monclient(hunting): handle_auth_done global_id 4364 payload 824 2024-09-17T18:19:09.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.133+0000 7fbfbf7fe640 1 -- 172.21.15.121:0/1820598356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbfc00a5c00 msgr2=0x7fbfc01cd950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:09.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.133+0000 7fbfbf7fe640 1 --2- 172.21.15.121:0/1820598356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbfc00a5c00 0x7fbfc01cd950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:09.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.133+0000 7fbfbf7fe640 1 -- 172.21.15.121:0/1820598356 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbfc00a5fe0 msgr2=0x7fbfc01ce120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:09.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.133+0000 7fbfbf7fe640 1 --2- 172.21.15.121:0/1820598356 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbfc00a5fe0 0x7fbfc01ce120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:09.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.133+0000 7fbfbf7fe640 10 monclient: _finish_hunting 0 2024-09-17T18:19:09.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.133+0000 7fbfbf7fe640 1 monclient: found mon.noname-c 2024-09-17T18:19:09.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.133+0000 7fbfbf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:09.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.133+0000 7fbfbf7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:19:09.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.133+0000 7fbfbf7fe640 1 -- 172.21.15.121:0/1820598356 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfc0122540 con 0x7fbfc013ec80 2024-09-17T18:19:09.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.133+0000 7fbfbeffd640 1 --2- 172.21.15.121:0/1820598356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbfc00a5c00 0x7fbfc01cd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T18:19:09.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.133+0000 7fbfb67fc640 1 --2- 172.21.15.121:0/1820598356 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbfc00a5fe0 0x7fbfc01ce120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:19:09.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.135+0000 7fbfbf7fe640 1 -- 172.21.15.121:0/1820598356 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbfc013ec80 msgr2=0x7fbfc013bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:09.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.135+0000 7fbfbf7fe640 1 --2- 172.21.15.121:0/1820598356 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbfc013ec80 0x7fbfc013bb00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbfc00a6d30 tx=0x7fbfb8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:09.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.135+0000 7fbfb7fff640 10 client.?.objecter ms_handle_connect 0x7fbfc013ec80 2024-09-17T18:19:09.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.135+0000 7fbfb7fff640 10 client.?.objecter resend_mon_ops 2024-09-17T18:19:09.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.135+0000 7fbfb7fff640 1 -- 172.21.15.121:0/1820598356 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb8040070 con 0x7fbfc013ec80 2024-09-17T18:19:09.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.135+0000 7fbfb7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:09.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.135+0000 7fbfb7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:19:09.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.135+0000 7fbfb7fff640 10 monclient: dump: 2024-09-17T18:19:09.149 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:09.149 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:09.149 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:09.149 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:09.149 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:09.149 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:09.150 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:09.150 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:09.150 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:09.150 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:09.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.135+0000 7fbfb7fff640 10 monclient: _finish_auth 0 2024-09-17T18:19:09.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.135+0000 7fbfb7fff640 10 monclient: _check_auth_tickets 2024-09-17T18:19:09.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.135+0000 7fbfb7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:09.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.135+0000 7fbfb7fff640 1 -- 172.21.15.121:0/1820598356 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfb80456a0 con 0x7fbfc013ec80 2024-09-17T18:19:09.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.135+0000 7fbfb7fff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:09.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.135+0000 7fbfb7fff640 1 -- 172.21.15.121:0/1820598356 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb8045a90 con 0x7fbfc013ec80 2024-09-17T18:19:09.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.135+0000 7fbfb7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:09.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.135+0000 7fbfb7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:19:09.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.135+0000 7fbfb7fff640 10 monclient: dump: 2024-09-17T18:19:09.150 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:09.150 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:09.151 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:09.151 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:09.151 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:09.151 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:09.151 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:09.151 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:09.151 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:09.151 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:09.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.135+0000 7fbfc5352640 5 monclient: authenticate success, global_id 4364 2024-09-17T18:19:09.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.135+0000 7fbfc5352640 10 monclient: _renew_subs 2024-09-17T18:19:09.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.135+0000 7fbfc5352640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:09.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.135+0000 7fbfc5352640 1 -- 172.21.15.121:0/1820598356 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfc013e820 con 0x7fbfc013ec80 2024-09-17T18:19:09.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.135+0000 7fbfbdffb640 4 set_mon_vals no callback set 2024-09-17T18:19:09.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.135+0000 7fbfc5352640 10 client.4364.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:09.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.135+0000 7fbfc5352640 10 monclient: _renew_subs 2024-09-17T18:19:09.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.135+0000 7fbfc5352640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:19:09.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.135+0000 7fbfc5352640 1 -- 172.21.15.121:0/1820598356 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfc0121ca0 con 0x7fbfc013ec80 2024-09-17T18:19:09.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.135+0000 7fbfc5352640 1 librados: init done 2024-09-17T18:19:09.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.136+0000 7fbfc5352640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:09.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.136+0000 7fbfc5352640 1 --2- 172.21.15.121:0/1820598356 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbfc01ce660 0x7fbfc002d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:09.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.136+0000 7fbfc5352640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:09.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.136+0000 7fbfc5352640 1 -- 172.21.15.121:0/1820598356 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbfc00aaa70 con 0x7fbfc01ce660 2024-09-17T18:19:09.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.136+0000 7fbfb7fff640 1 -- 172.21.15.121:0/1820598356 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fbfb8039040 con 0x7fbfc013ec80 2024-09-17T18:19:09.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.136+0000 7fbfbeffd640 1 --2- 172.21.15.121:0/1820598356 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbfc01ce660 0x7fbfc002d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:09.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.136+0000 7fbfbeffd640 10 monclient: get_auth_request con 0x7fbfc01ce660 auth_method 0 2024-09-17T18:19:09.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.136+0000 7fbfbeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:09.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.136+0000 7fbfbeffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:09.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.136+0000 7fbfbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:09.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.136+0000 7fbfbeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:09.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.136+0000 7fbfbeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:09.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.136+0000 7fbfbeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:09.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.136+0000 7fbfbeffd640 10 monclient(hunting): handle_auth_done global_id 4367 payload 824 2024-09-17T18:19:09.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.136+0000 7fbfb7fff640 1 --2- 172.21.15.121:0/1820598356 >> v2:172.21.15.173:6812/2423786037 conn(0x7fbfa0040170 0x7fbfa0042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:09.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.136+0000 7fbfbeffd640 1 -- 172.21.15.121:0/1820598356 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbfc009d930 con 0x7fbfc01ce660 2024-09-17T18:19:09.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.136+0000 7fbfb7fff640 1 -- 172.21.15.121:0/1820598356 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(17..17 src has 1..17) ==== 3534+0+0 (secure 0 0 0) 0x7fbfb8074980 con 0x7fbfc013ec80 2024-09-17T18:19:09.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.136+0000 7fbfb7fff640 10 client.4364.objecter ms_dispatch 0x7fbfc0000b90 osd_map(17..17 src has 1..17) 2024-09-17T18:19:09.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.136+0000 7fbfb7fff640 3 client.4364.objecter handle_osd_map got epochs [17,17] > 0 2024-09-17T18:19:09.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.136+0000 7fbfb7fff640 3 client.4364.objecter handle_osd_map decoding full epoch 17 2024-09-17T18:19:09.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.137+0000 7fbfb7fff640 20 client.4364.objecter dump_active .. 0 homeless 2024-09-17T18:19:09.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.138+0000 7fbfb67fc640 1 --2- 172.21.15.121:0/1820598356 >> v2:172.21.15.173:6812/2423786037 conn(0x7fbfa0040170 0x7fbfa0042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:09.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.138+0000 7fbfb67fc640 10 monclient: get_auth_request con 0x7fbfa0040170 auth_method 0 2024-09-17T18:19:09.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.139+0000 7fbfbeffd640 1 -- 172.21.15.121:0/1820598356 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbfc01ce660 msgr2=0x7fbfc002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:09.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.139+0000 7fbfb7fff640 10 client.4364.objecter ms_handle_connect 0x7fbfc01ce660 2024-09-17T18:19:09.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.139+0000 7fbfb7fff640 10 client.4364.objecter resend_mon_ops 2024-09-17T18:19:09.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.139+0000 7fbfbeffd640 1 --2- 172.21.15.121:0/1820598356 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbfc01ce660 0x7fbfc002d440 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbfb000e500 tx=0x7fbfb000e9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:09.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.139+0000 7fbfb7fff640 1 -- 172.21.15.121:0/1820598356 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb0013800 con 0x7fbfc01ce660 2024-09-17T18:19:09.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.139+0000 7fbfb7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:09.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.139+0000 7fbfb7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:19:09.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.139+0000 7fbfb7fff640 10 monclient: dump: 2024-09-17T18:19:09.156 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:09.156 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:09.156 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:09.156 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:09.156 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:09.156 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:09.156 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:09.157 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:09.157 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:09.157 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:09.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.139+0000 7fbfb7fff640 1 -- 172.21.15.121:0/1820598356 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbfb0013800 con 0x7fbfc01ce660 2024-09-17T18:19:09.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.139+0000 7fbfb7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:09.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.139+0000 7fbfb7fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:19:09.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.139+0000 7fbfb7fff640 1 -- 172.21.15.121:0/1820598356 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbfc01ce660 msgr2=0x7fbfc002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:09.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.139+0000 7fbfb7fff640 1 --2- 172.21.15.121:0/1820598356 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbfc01ce660 0x7fbfc002d440 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbfb000e500 tx=0x7fbfb000e9c0 comp rx=0 tx=0).stop 2024-09-17T18:19:09.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.139+0000 7fbfb7fff640 1 -- 172.21.15.121:0/1820598356 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbfc01ce660 msgr2=0x7fbfc002d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:19:09.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.139+0000 7fbfb7fff640 1 --2- 172.21.15.121:0/1820598356 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbfc01ce660 0x7fbfc002d440 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:09.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.142+0000 7fbfb67fc640 1 --2- 172.21.15.121:0/1820598356 >> v2:172.21.15.173:6812/2423786037 conn(0x7fbfa0040170 0x7fbfa0042630 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbfa8007d50 tx=0x7fbfa8007ca0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:09.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.142+0000 7fbfb7fff640 10 client.4364.objecter ms_handle_connect 0x7fbfa0040170 2024-09-17T18:19:09.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.146+0000 7fbfc5352640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-17T18:19:09.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.146+0000 7fbfc5352640 1 --2- 172.21.15.121:0/1820598356 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbfc002d980 0x7fbfc002dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:09.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.146+0000 7fbfc5352640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:09.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.146+0000 7fbfc5352640 1 -- 172.21.15.121:0/1820598356 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbfc00008d0 con 0x7fbfc002d980 2024-09-17T18:19:09.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.146+0000 7fbfbeffd640 1 --2- 172.21.15.121:0/1820598356 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbfc002d980 0x7fbfc002dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:09.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.146+0000 7fbfbeffd640 10 monclient: get_auth_request con 0x7fbfc002d980 auth_method 0 2024-09-17T18:19:09.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.146+0000 7fbfbeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:09.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.146+0000 7fbfbeffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:09.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.146+0000 7fbfbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:09.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.146+0000 7fbfbeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:09.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.146+0000 7fbfbeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:09.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.146+0000 7fbfbeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:09.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.146+0000 7fbfbeffd640 10 monclient(hunting): handle_auth_done global_id 4370 payload 824 2024-09-17T18:19:09.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.147+0000 7fbfbeffd640 1 -- 172.21.15.121:0/1820598356 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7fbfc0139210 con 0x7fbfc002d980 2024-09-17T18:19:09.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.149+0000 7fbfbeffd640 1 -- 172.21.15.121:0/1820598356 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbfc002d980 msgr2=0x7fbfc002dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:09.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.149+0000 7fbfb7fff640 10 client.4364.objecter ms_handle_connect 0x7fbfc002d980 2024-09-17T18:19:09.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.149+0000 7fbfb7fff640 10 client.4364.objecter resend_mon_ops 2024-09-17T18:19:09.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.149+0000 7fbfbeffd640 1 --2- 172.21.15.121:0/1820598356 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbfc002d980 0x7fbfc002dd60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbfb000e500 tx=0x7fbfb0003760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:09.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.149+0000 7fbfb7fff640 1 -- 172.21.15.121:0/1820598356 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb0006df0 con 0x7fbfc002d980 2024-09-17T18:19:09.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.149+0000 7fbfb7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:09.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.149+0000 7fbfb7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:19:09.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.149+0000 7fbfb7fff640 10 monclient: dump: 2024-09-17T18:19:09.159 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:09.159 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:09.159 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:09.159 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:09.159 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:09.159 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:09.159 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:09.159 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:09.159 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:09.160 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:09.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.149+0000 7fbfb7fff640 1 -- 172.21.15.121:0/1820598356 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (secure 0 0 0) 0x7fbfb0006df0 con 0x7fbfc002d980 2024-09-17T18:19:09.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.149+0000 7fbfb7fff640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-17T18:19:09.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.149+0000 7fbfb7fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:19:09.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.149+0000 7fbfb7fff640 1 -- 172.21.15.121:0/1820598356 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbfc002d980 msgr2=0x7fbfc002dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:09.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.149+0000 7fbfb7fff640 1 --2- 172.21.15.121:0/1820598356 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbfc002d980 0x7fbfc002dd60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbfb000e500 tx=0x7fbfb0003760 comp rx=0 tx=0).stop 2024-09-17T18:19:09.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.149+0000 7fbfb7fff640 1 -- 172.21.15.121:0/1820598356 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbfc002d980 msgr2=0x7fbfc002dd60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:19:09.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.149+0000 7fbfb7fff640 1 --2- 172.21.15.121:0/1820598356 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbfc002d980 0x7fbfc002dd60 secure :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbfb000e500 tx=0x7fbfb0003760 comp rx=0 tx=0).stop 2024-09-17T18:19:09.160 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:19:09.160 INFO:teuthology.orchestra.run.smithi121.stdout: "ret": 0, 2024-09-17T18:19:09.161 INFO:teuthology.orchestra.run.smithi121.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-09-17T18:19:09.161 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:19:09.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.149+0000 7fbfc5352640 10 librados: watch_flush enter 2024-09-17T18:19:09.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.150+0000 7fbfc5352640 10 librados: watch_flush exit 2024-09-17T18:19:09.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.150+0000 7fbfc5352640 20 client.4364.objecter shutdown clearing up homeless session... 2024-09-17T18:19:09.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.150+0000 7fbfc5352640 10 client.4364.objecter successfully canceled tick 2024-09-17T18:19:09.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.150+0000 7fbfc5352640 1 -- 172.21.15.121:0/1820598356 >> v2:172.21.15.173:6812/2423786037 conn(0x7fbfa0040170 msgr2=0x7fbfa0042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:09.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.150+0000 7fbfc5352640 1 --2- 172.21.15.121:0/1820598356 >> v2:172.21.15.173:6812/2423786037 conn(0x7fbfa0040170 0x7fbfa0042630 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbfa8007d50 tx=0x7fbfa8007ca0 comp rx=0 tx=0).stop 2024-09-17T18:19:09.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.150+0000 7fbfc5352640 10 monclient: shutdown 2024-09-17T18:19:09.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.150+0000 7fbfc5352640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:09.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.150+0000 7fbfc5352640 1 -- 172.21.15.121:0/1820598356 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbfc013ec80 msgr2=0x7fbfc013bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:09.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.150+0000 7fbfc5352640 1 --2- 172.21.15.121:0/1820598356 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbfc013ec80 0x7fbfc013bb00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbfc00a6d30 tx=0x7fbfb8004a40 comp rx=0 tx=0).stop 2024-09-17T18:19:09.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.150+0000 7fbfbf7fe640 1 -- 172.21.15.121:0/1820598356 reap_dead start 2024-09-17T18:19:09.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.150+0000 7fbfbf7fe640 1 -- 172.21.15.121:0/1820598356 reap_dead start 2024-09-17T18:19:09.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.150+0000 7fbfc5352640 1 -- 172.21.15.121:0/1820598356 shutdown_connections 2024-09-17T18:19:09.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.150+0000 7fbfc5352640 1 -- 172.21.15.121:0/1820598356 >> 172.21.15.121:0/1820598356 conn(0x7fbfc00ab9e0 msgr2=0x7fbfc00a1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:09.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.150+0000 7fbfc5352640 1 -- 172.21.15.121:0/1820598356 shutdown_connections 2024-09-17T18:19:09.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.150+0000 7fbfc5352640 1 -- 172.21.15.121:0/1820598356 wait complete. 2024-09-17T18:19:09.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.150+0000 7fbfc5352640 1 librados: shutdown 2024-09-17T18:19:09.163 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-17T18:19:09.164 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-17T18:19:09.164 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T18:19:09.174 INFO:tasks.ceph.mon.c:Stopped 2024-09-17T18:19:09.174 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-17T18:19:09.174 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:19:09.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.309+0000 7f7d3aa16640 10 monclient: get_monmap_and_config 2024-09-17T18:19:09.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.309+0000 7f7d3aa16640 10 monclient: build_initial_monmap 2024-09-17T18:19:09.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.309+0000 7f7d3aa16640 10 monclient: monmap: 2024-09-17T18:19:09.311 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:09.311 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:09.311 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:19:09.309717+0000 2024-09-17T18:19:09.311 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:19:09.309717+0000 2024-09-17T18:19:09.311 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:09.311 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:09.311 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:19:09.311 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:19:09.311 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:19:09.311 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:19:09.311 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:19:09.311 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:09.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.309+0000 7f7d3aa16640 1 Processor -- start 2024-09-17T18:19:09.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.310+0000 7f7d3aa16640 1 -- start start 2024-09-17T18:19:09.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.310+0000 7f7d3aa16640 10 monclient: init 2024-09-17T18:19:09.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.310+0000 7f7d3aa16640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:09.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.310+0000 7f7d3aa16640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-17T18:19:09.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.310+0000 7f7d3aa16640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f7d340a5c00 0x7f7d3409d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:09.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.310+0000 7f7d3aa16640 10 monclient(hunting): picked mon.noname-c con 0x7f7d340a5c00 addr v2:172.21.15.173:3300/0 2024-09-17T18:19:09.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.310+0000 7f7d3aa16640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7d340a5fe0 addr v1:172.21.15.173:6789/0 2024-09-17T18:19:09.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.310+0000 7f7d3aa16640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d3413dc00 0x7f7d3413fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:09.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.310+0000 7f7d3aa16640 10 monclient(hunting): picked mon.noname-b con 0x7f7d3413dc00 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:09.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.310+0000 7f7d3aa16640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d340a8a80 con 0x7f7d340a5fe0 2024-09-17T18:19:09.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.310+0000 7f7d3aa16640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:09.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.310+0000 7f7d3aa16640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f7d340a7110 con 0x7f7d340a5c00 2024-09-17T18:19:09.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.310+0000 7f7d3aa16640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:09.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.310+0000 7f7d3aa16640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7d3409de00 con 0x7f7d3413dc00 2024-09-17T18:19:09.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.310+0000 7f7d3aa16640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:09.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.310+0000 7f7d3aa16640 10 monclient(hunting): authenticate will time out at 753.026575s 2024-09-17T18:19:09.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.310+0000 7f7d38fb3640 1 -- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d3413dc00 msgr2=0x7f7d3413fff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:19:09.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.310+0000 7f7d38fb3640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d3413dc00 0x7f7d3413fff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:19:09.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.310+0000 7f7d33fff640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f7d340a5c00 0x7f7d3409d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:09.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.310+0000 7f7d2bfff640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f7d340a5fe0 0x7f7d340a63f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.121:34640/0 (socket says 172.21.15.121:34640) 2024-09-17T18:19:09.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.310+0000 7f7d2bfff640 1 -- 172.21.15.121:0/938295557 learned_addr learned my addr 172.21.15.121:0/938295557 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:19:09.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.310+0000 7f7d33fff640 10 monclient(hunting): get_auth_request con 0x7f7d340a5c00 auth_method 0 2024-09-17T18:19:09.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.310+0000 7f7d33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:09.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.310+0000 7f7d33fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:09.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.310+0000 7f7d33fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:09.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.310+0000 7f7d33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:09.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.310+0000 7f7d33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:09.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.311+0000 7f7d33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:09.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.311+0000 7f7d33fff640 10 monclient(hunting): handle_auth_done global_id 4366 payload 274 2024-09-17T18:19:09.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.311+0000 7f7d33fff640 1 -- 172.21.15.121:0/938295557 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d3413dc00 msgr2=0x7f7d3413fff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:19:09.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.311+0000 7f7d33fff640 1 --2- 172.21.15.121:0/938295557 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d3413dc00 0x7f7d3413fff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:09.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.311+0000 7f7d33fff640 1 -- 172.21.15.121:0/938295557 >> v1:172.21.15.173:6789/0 conn(0x7f7d340a5fe0 legacy=0x7f7d340a63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:09.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.311+0000 7f7d33fff640 10 monclient: _finish_hunting 0 2024-09-17T18:19:09.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.311+0000 7f7d33fff640 1 monclient: found mon.noname-c 2024-09-17T18:19:09.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.311+0000 7f7d33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:09.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.311+0000 7f7d33fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.173:3300/0 2024-09-17T18:19:09.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.311+0000 7f7d33fff640 1 -- 172.21.15.121:0/938295557 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d340a74d0 con 0x7f7d340a5c00 2024-09-17T18:19:09.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.314+0000 7f7d33fff640 1 -- 172.21.15.121:0/938295557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d340a5c00 msgr2=0x7f7d3409d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:09.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.314+0000 7f7d33fff640 1 --2- 172.21.15.121:0/938295557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d340a5c00 0x7f7d3409d8c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7d240097a0 tx=0x7f7d24029fe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=52b1b360d629d876 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:09.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.314+0000 7f7d337fe640 1 -- 172.21.15.121:0/938295557 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d24037070 con 0x7f7d340a5c00 2024-09-17T18:19:09.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.314+0000 7f7d337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:09.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.314+0000 7f7d337fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:19:09.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.314+0000 7f7d337fe640 10 monclient: dump: 2024-09-17T18:19:09.324 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:09.324 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:09.324 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:09.324 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:09.324 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:09.324 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:09.324 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:09.324 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:09.324 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:09.324 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:09.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.314+0000 7f7d337fe640 10 monclient: _finish_auth 0 2024-09-17T18:19:09.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.314+0000 7f7d337fe640 10 monclient: _check_auth_tickets 2024-09-17T18:19:09.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.314+0000 7f7d337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:09.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.314+0000 7f7d337fe640 1 -- 172.21.15.121:0/938295557 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d24032640 con 0x7f7d340a5c00 2024-09-17T18:19:09.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.314+0000 7f7d3aa16640 5 monclient: authenticate success, global_id 4366 2024-09-17T18:19:09.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.314+0000 7f7d3aa16640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:19:09.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.314+0000 7f7d337fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:09.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.314+0000 7f7d337fe640 1 -- 172.21.15.121:0/938295557 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d24032a10 con 0x7f7d340a5c00 2024-09-17T18:19:09.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.314+0000 7f7d3aa16640 10 monclient: get_monmap_and_config success 2024-09-17T18:19:09.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.314+0000 7f7d3aa16640 4 set_mon_vals no callback set 2024-09-17T18:19:09.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.314+0000 7f7d337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:09.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.314+0000 7f7d337fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:19:09.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.314+0000 7f7d337fe640 10 monclient: dump: 2024-09-17T18:19:09.325 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:09.325 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:09.325 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:09.325 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:09.326 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:09.326 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:09.326 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:09.326 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:09.326 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:09.326 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:09.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.314+0000 7f7d3aa16640 10 monclient: shutdown 2024-09-17T18:19:09.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.314+0000 7f7d3aa16640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:09.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.314+0000 7f7d3aa16640 1 -- 172.21.15.121:0/938295557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d340a5c00 msgr2=0x7f7d3409d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:09.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.314+0000 7f7d3aa16640 1 --2- 172.21.15.121:0/938295557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d340a5c00 0x7f7d3409d8c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7d240097a0 tx=0x7f7d24029fe0 comp rx=0 tx=0).stop 2024-09-17T18:19:09.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.314+0000 7f7d3aa16640 1 -- 172.21.15.121:0/938295557 shutdown_connections 2024-09-17T18:19:09.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.314+0000 7f7d3aa16640 1 --2- 172.21.15.121:0/938295557 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d3413dc00 0x7f7d3413fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:09.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.314+0000 7f7d3aa16640 1 --2- 172.21.15.121:0/938295557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d340a5c00 0x7f7d3409d8c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:09.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.314+0000 7f7d3aa16640 1 -- 172.21.15.121:0/938295557 >> 172.21.15.121:0/938295557 conn(0x7f7d340ab9b0 msgr2=0x7f7d340a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:09.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.315+0000 7f7d3aa16640 1 -- 172.21.15.121:0/938295557 shutdown_connections 2024-09-17T18:19:09.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.315+0000 7f7d3aa16640 1 -- 172.21.15.121:0/938295557 wait complete. 2024-09-17T18:19:09.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.315+0000 7f7d3aa16640 10 monclient: build_initial_monmap 2024-09-17T18:19:09.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.315+0000 7f7d3aa16640 10 monclient: monmap: 2024-09-17T18:19:09.327 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:09.327 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:19:09.327 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:19:09.327 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:19:09.327 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:09.327 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:09.327 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:19:09.327 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:19:09.327 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:19:09.328 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:09.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.315+0000 7f7d3aa16640 1 librados: starting msgr at 2024-09-17T18:19:09.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.315+0000 7f7d3aa16640 1 librados: starting objecter 2024-09-17T18:19:09.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.315+0000 7f7d3aa16640 1 Processor -- start 2024-09-17T18:19:09.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.315+0000 7f7d3aa16640 1 -- start start 2024-09-17T18:19:09.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.315+0000 7f7d3aa16640 1 librados: setting wanted keys 2024-09-17T18:19:09.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.315+0000 7f7d3aa16640 1 librados: calling monclient init 2024-09-17T18:19:09.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.315+0000 7f7d3aa16640 10 monclient: init 2024-09-17T18:19:09.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d3aa16640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:09.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d3aa16640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:19:09.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d3aa16640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d340a5c00 0x7f7d341cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:09.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d3aa16640 10 monclient(hunting): picked mon.noname-a con 0x7f7d340a5c00 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:09.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d3aa16640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d340a5fe0 0x7f7d34138e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:09.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d3aa16640 10 monclient(hunting): picked mon.noname-b con 0x7f7d340a5fe0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:09.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d3aa16640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d3413dc00 0x7f7d3413be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:09.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d3aa16640 10 monclient(hunting): picked mon.noname-c con 0x7f7d3413dc00 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:09.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d3aa16640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:09.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d3aa16640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7d340703d0 con 0x7f7d340a5c00 2024-09-17T18:19:09.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d3aa16640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:09.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d3aa16640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7d34070550 con 0x7f7d340a5fe0 2024-09-17T18:19:09.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d3aa16640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:09.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d3aa16640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7d34140530 con 0x7f7d3413dc00 2024-09-17T18:19:09.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d3aa16640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:09.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d3aa16640 10 monclient(hunting): authenticate will time out at 753.032402s 2024-09-17T18:19:09.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d38fb3640 1 -- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d3413dc00 msgr2=0x7f7d3413be40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:19:09.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d38fb3640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d3413dc00 0x7f7d3413be40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:19:09.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d33fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d340a5c00 0x7f7d341cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:09.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d33fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d340a5c00 0x7f7d341cdb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43660/0 (socket says 172.21.15.121:43660) 2024-09-17T18:19:09.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d33fff640 1 -- 172.21.15.121:0/2469482771 learned_addr learned my addr 172.21.15.121:0/2469482771 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:09.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d33fff640 10 monclient(hunting): get_auth_request con 0x7f7d340a5c00 auth_method 0 2024-09-17T18:19:09.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:09.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d33fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:09.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d33fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:09.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:09.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:09.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:09.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d2bfff640 1 --2- 172.21.15.121:0/2469482771 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d340a5fe0 0x7f7d34138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:09.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d2bfff640 10 monclient(hunting): get_auth_request con 0x7f7d340a5fe0 auth_method 0 2024-09-17T18:19:09.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d2bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:09.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d2bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:09.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:09.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d2bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:09.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d2bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:09.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d2bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:09.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d33fff640 10 monclient(hunting): handle_auth_done global_id 4371 payload 824 2024-09-17T18:19:09.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d33fff640 1 -- 172.21.15.121:0/2469482771 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d3413dc00 msgr2=0x7f7d3413be40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:19:09.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d33fff640 1 --2- 172.21.15.121:0/2469482771 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d3413dc00 0x7f7d3413be40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:09.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d33fff640 1 -- 172.21.15.121:0/2469482771 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d340a5fe0 msgr2=0x7f7d34138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:09.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d33fff640 1 --2- 172.21.15.121:0/2469482771 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d340a5fe0 0x7f7d34138e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:09.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d33fff640 10 monclient: _finish_hunting 0 2024-09-17T18:19:09.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d33fff640 1 monclient: found mon.noname-a 2024-09-17T18:19:09.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:09.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d33fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:19:09.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.316+0000 7f7d33fff640 1 -- 172.21.15.121:0/2469482771 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d34122f20 con 0x7f7d340a5c00 2024-09-17T18:19:09.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.319+0000 7f7d33fff640 1 -- 172.21.15.121:0/2469482771 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d340a5c00 msgr2=0x7f7d341cdb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:09.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.319+0000 7f7d33fff640 1 --2- 172.21.15.121:0/2469482771 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d340a5c00 0x7f7d341cdb50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7d240097a0 tx=0x7f7d240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:09.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.319+0000 7f7d317fa640 10 client.?.objecter ms_handle_connect 0x7f7d340a5c00 2024-09-17T18:19:09.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.319+0000 7f7d317fa640 10 client.?.objecter resend_mon_ops 2024-09-17T18:19:09.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.319+0000 7f7d317fa640 1 -- 172.21.15.121:0/2469482771 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d24041070 con 0x7f7d340a5c00 2024-09-17T18:19:09.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.319+0000 7f7d317fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:09.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.319+0000 7f7d317fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:19:09.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.319+0000 7f7d317fa640 10 monclient: dump: 2024-09-17T18:19:09.332 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:09.332 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:09.332 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:09.332 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:09.332 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:09.332 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:09.333 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:09.333 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:09.333 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:09.333 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:09.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.319+0000 7f7d317fa640 10 monclient: _finish_auth 0 2024-09-17T18:19:09.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.319+0000 7f7d317fa640 10 monclient: _check_auth_tickets 2024-09-17T18:19:09.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.319+0000 7f7d317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:09.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.319+0000 7f7d3aa16640 5 monclient: authenticate success, global_id 4371 2024-09-17T18:19:09.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.319+0000 7f7d3aa16640 10 monclient: _renew_subs 2024-09-17T18:19:09.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.319+0000 7f7d317fa640 1 -- 172.21.15.121:0/2469482771 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d2403ca80 con 0x7f7d340a5c00 2024-09-17T18:19:09.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.319+0000 7f7d3aa16640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:09.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.319+0000 7f7d3aa16640 1 -- 172.21.15.121:0/2469482771 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d340aabe0 con 0x7f7d340a5c00 2024-09-17T18:19:09.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.319+0000 7f7d317fa640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:09.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.319+0000 7f7d317fa640 1 -- 172.21.15.121:0/2469482771 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d24037040 con 0x7f7d340a5c00 2024-09-17T18:19:09.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.319+0000 7f7d317fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:09.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.319+0000 7f7d317fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:19:09.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.319+0000 7f7d317fa640 10 monclient: dump: 2024-09-17T18:19:09.334 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:09.334 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:09.334 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:09.334 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:09.334 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:09.334 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:09.334 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:09.334 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:09.335 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:09.335 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:09.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.319+0000 7f7d3aa16640 10 client.4371.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:09.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.319+0000 7f7d3aa16640 10 monclient: _renew_subs 2024-09-17T18:19:09.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.319+0000 7f7d32ffd640 4 set_mon_vals no callback set 2024-09-17T18:19:09.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.319+0000 7f7d3aa16640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:09.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.319+0000 7f7d3aa16640 1 -- 172.21.15.121:0/2469482771 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d341ce9e0 con 0x7f7d340a5c00 2024-09-17T18:19:09.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.319+0000 7f7d3aa16640 1 librados: init done 2024-09-17T18:19:09.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.320+0000 7f7d317fa640 1 -- 172.21.15.121:0/2469482771 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f7d24044070 con 0x7f7d340a5c00 2024-09-17T18:19:09.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.320+0000 7f7d3aa16640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:09.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.320+0000 7f7d3aa16640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:09.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.320+0000 7f7d3aa16640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:09.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.320+0000 7f7d3aa16640 1 -- 172.21.15.121:0/2469482771 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d340a4fb0 con 0x7f7d340a5c00 2024-09-17T18:19:09.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.320+0000 7f7d317fa640 1 --2- 172.21.15.121:0/2469482771 >> v2:172.21.15.173:6812/2423786037 conn(0x7f7d14040120 0x7f7d140425e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:09.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.320+0000 7f7d317fa640 1 -- 172.21.15.121:0/2469482771 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 3534+0+0 (secure 0 0 0) 0x7f7d240734b0 con 0x7f7d340a5c00 2024-09-17T18:19:09.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.320+0000 7f7d317fa640 10 client.4371.objecter ms_dispatch 0x7f7d34000b90 osd_map(17..17 src has 1..17) 2024-09-17T18:19:09.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.320+0000 7f7d317fa640 3 client.4371.objecter handle_osd_map got epochs [17,17] > 0 2024-09-17T18:19:09.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.320+0000 7f7d317fa640 3 client.4371.objecter handle_osd_map decoding full epoch 17 2024-09-17T18:19:09.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.320+0000 7f7d317fa640 20 client.4371.objecter dump_active .. 0 homeless 2024-09-17T18:19:09.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.320+0000 7f7d2bfff640 1 --2- 172.21.15.121:0/2469482771 >> v2:172.21.15.173:6812/2423786037 conn(0x7f7d14040120 0x7f7d140425e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:09.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.320+0000 7f7d2bfff640 10 monclient: get_auth_request con 0x7f7d14040120 auth_method 0 2024-09-17T18:19:09.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.323+0000 7f7d317fa640 10 client.4371.objecter ms_handle_connect 0x7f7d14040120 2024-09-17T18:19:09.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.323+0000 7f7d2bfff640 1 --2- 172.21.15.121:0/2469482771 >> v2:172.21.15.173:6812/2423786037 conn(0x7f7d14040120 0x7f7d140425e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c007c90 tx=0x7f7d1c007be0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:09.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.324+0000 7f7d317fa640 1 -- 172.21.15.121:0/2469482771 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7d340a4fb0 con 0x7f7d340a5c00 2024-09-17T18:19:09.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.324+0000 7f7d317fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:09.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.324+0000 7f7d317fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:19:09.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.565+0000 7f7d3aa16640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:19:09.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.565+0000 7f7d3aa16640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:19:09.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.565+0000 7f7d3aa16640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:09.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.565+0000 7f7d3aa16640 1 -- 172.21.15.121:0/2469482771 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7d3409d8c0 con 0x7f7d340a5c00 2024-09-17T18:19:09.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.565+0000 7f7d317fa640 1 -- 172.21.15.121:0/2469482771 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f7d2404b070 con 0x7f7d340a5c00 2024-09-17T18:19:09.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.565+0000 7f7d317fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:19:09.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.565+0000 7f7d317fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:19:09.568 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:19:09.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.567+0000 7f7d3aa16640 10 librados: watch_flush enter 2024-09-17T18:19:09.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.567+0000 7f7d3aa16640 10 librados: watch_flush exit 2024-09-17T18:19:09.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.568+0000 7f7d3aa16640 20 client.4371.objecter shutdown clearing up homeless session... 2024-09-17T18:19:09.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.568+0000 7f7d3aa16640 10 client.4371.objecter successfully canceled tick 2024-09-17T18:19:09.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.568+0000 7f7d3aa16640 1 -- 172.21.15.121:0/2469482771 >> v2:172.21.15.173:6812/2423786037 conn(0x7f7d14040120 msgr2=0x7f7d140425e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:09.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.568+0000 7f7d3aa16640 1 --2- 172.21.15.121:0/2469482771 >> v2:172.21.15.173:6812/2423786037 conn(0x7f7d14040120 0x7f7d140425e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c007c90 tx=0x7f7d1c007be0 comp rx=0 tx=0).stop 2024-09-17T18:19:09.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.568+0000 7f7d3aa16640 10 monclient: shutdown 2024-09-17T18:19:09.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.568+0000 7f7d3aa16640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:09.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.568+0000 7f7d3aa16640 1 -- 172.21.15.121:0/2469482771 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d340a5c00 msgr2=0x7f7d341cdb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:09.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.568+0000 7f7d3aa16640 1 --2- 172.21.15.121:0/2469482771 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d340a5c00 0x7f7d341cdb50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7d240097a0 tx=0x7f7d240046c0 comp rx=0 tx=0).stop 2024-09-17T18:19:09.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.568+0000 7f7d3aa16640 1 -- 172.21.15.121:0/2469482771 shutdown_connections 2024-09-17T18:19:09.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.568+0000 7f7d3aa16640 1 --2- 172.21.15.121:0/2469482771 >> v2:172.21.15.173:6812/2423786037 conn(0x7f7d14040120 0x7f7d140425e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:09.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.568+0000 7f7d3aa16640 1 --2- 172.21.15.121:0/2469482771 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d340a5fe0 0x7f7d34138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:09.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.568+0000 7f7d3aa16640 1 --2- 172.21.15.121:0/2469482771 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d3413dc00 0x7f7d3413be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:09.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.568+0000 7f7d3aa16640 1 --2- 172.21.15.121:0/2469482771 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d340a5c00 0x7f7d341cdb50 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:09.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.568+0000 7f7d3aa16640 1 -- 172.21.15.121:0/2469482771 >> 172.21.15.121:0/2469482771 conn(0x7f7d340ab9b0 msgr2=0x7f7d340a48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:09.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.568+0000 7f7d3aa16640 1 -- 172.21.15.121:0/2469482771 shutdown_connections 2024-09-17T18:19:09.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.568+0000 7f7d3aa16640 1 -- 172.21.15.121:0/2469482771 wait complete. 2024-09-17T18:19:09.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:09.568+0000 7f7d3aa16640 1 librados: shutdown 2024-09-17T18:19:12.581 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:19:12.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.686+0000 7fef56416640 10 monclient: get_monmap_and_config 2024-09-17T18:19:12.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.686+0000 7fef56416640 10 monclient: build_initial_monmap 2024-09-17T18:19:12.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.686+0000 7fef56416640 10 monclient: monmap: 2024-09-17T18:19:12.688 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:12.689 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:12.689 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:19:12.687206+0000 2024-09-17T18:19:12.689 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:19:12.687206+0000 2024-09-17T18:19:12.689 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:12.689 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:12.689 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:19:12.689 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:19:12.690 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:19:12.690 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:19:12.690 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:19:12.690 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:12.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.687+0000 7fef56416640 1 Processor -- start 2024-09-17T18:19:12.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.687+0000 7fef56416640 1 -- start start 2024-09-17T18:19:12.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.687+0000 7fef56416640 10 monclient: init 2024-09-17T18:19:12.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.687+0000 7fef56416640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:12.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.687+0000 7fef56416640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-17T18:19:12.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.687+0000 7fef56416640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fef500a0a70 addr v1:172.21.15.121:6789/0 2024-09-17T18:19:12.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.687+0000 7fef56416640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fef501348f0 0x7fef50139c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:12.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.687+0000 7fef56416640 10 monclient(hunting): picked mon.noname-b con 0x7fef501348f0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:12.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.687+0000 7fef56416640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fef5013a150 0x7fef5013c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:12.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.687+0000 7fef56416640 10 monclient(hunting): picked mon.noname-a con 0x7fef5013a150 addr v2:172.21.15.121:3300/0 2024-09-17T18:19:12.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.687+0000 7fef56416640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef5009f5b0 con 0x7fef500a0a70 2024-09-17T18:19:12.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.687+0000 7fef56416640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:12.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.687+0000 7fef56416640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fef5009ef80 con 0x7fef5013a150 2024-09-17T18:19:12.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.687+0000 7fef56416640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:12.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.687+0000 7fef56416640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fef500a1560 con 0x7fef501348f0 2024-09-17T18:19:12.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.687+0000 7fef56416640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:12.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.687+0000 7fef56416640 10 monclient(hunting): authenticate will time out at 756.403978s 2024-09-17T18:19:12.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.687+0000 7fef4f7fe640 1 -- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fef501348f0 msgr2=0x7fef50139c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:19:12.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.688+0000 7fef4f7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fef501348f0 0x7fef50139c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:19:12.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.688+0000 7fef549b3640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fef5013a150 0x7fef5013c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:12.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.688+0000 7fef4ffff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fef500a0a70 0x7fef500a0e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:49856/0 (socket says 172.21.15.121:49856) 2024-09-17T18:19:12.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.688+0000 7fef549b3640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fef5013a150 0x7fef5013c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43664/0 (socket says 172.21.15.121:43664) 2024-09-17T18:19:12.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.688+0000 7fef4ffff640 1 -- 172.21.15.121:0/1299240727 learned_addr learned my addr 172.21.15.121:0/1299240727 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:19:12.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.688+0000 7fef549b3640 10 monclient(hunting): get_auth_request con 0x7fef5013a150 auth_method 0 2024-09-17T18:19:12.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.688+0000 7fef549b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:12.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.688+0000 7fef549b3640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:12.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.688+0000 7fef549b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:12.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.688+0000 7fef549b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:12.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.688+0000 7fef549b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:12.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.688+0000 7fef549b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:12.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.688+0000 7fef549b3640 10 monclient(hunting): handle_auth_done global_id 4374 payload 274 2024-09-17T18:19:12.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.688+0000 7fef549b3640 1 -- 172.21.15.121:0/1299240727 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fef501348f0 msgr2=0x7fef50139c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:19:12.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.688+0000 7fef549b3640 1 --2- 172.21.15.121:0/1299240727 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fef501348f0 0x7fef50139c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:12.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.688+0000 7fef549b3640 1 -- 172.21.15.121:0/1299240727 >> v1:172.21.15.121:6789/0 conn(0x7fef500a0a70 legacy=0x7fef500a0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:12.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.688+0000 7fef549b3640 10 monclient: _finish_hunting 0 2024-09-17T18:19:12.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.688+0000 7fef549b3640 1 monclient: found mon.noname-a 2024-09-17T18:19:12.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.688+0000 7fef549b3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:12.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.688+0000 7fef549b3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:19:12.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.688+0000 7fef549b3640 1 -- 172.21.15.121:0/1299240727 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef5009eca0 con 0x7fef5013a150 2024-09-17T18:19:12.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.691+0000 7fef549b3640 1 -- 172.21.15.121:0/1299240727 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef5013a150 msgr2=0x7fef5013c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:12.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.691+0000 7fef549b3640 1 --2- 172.21.15.121:0/1299240727 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef5013a150 0x7fef5013c540 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fef4800aed0 tx=0x7fef4802b630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c3035a4d1634f43 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:12.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.691+0000 7fef4effd640 1 -- 172.21.15.121:0/1299240727 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef48037070 con 0x7fef5013a150 2024-09-17T18:19:12.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.691+0000 7fef4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:12.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.691+0000 7fef4effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:19:12.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.691+0000 7fef4effd640 10 monclient: dump: 2024-09-17T18:19:12.702 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:12.702 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:12.702 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:12.702 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:12.702 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:12.702 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:12.702 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:12.702 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:12.702 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:12.702 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:12.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.691+0000 7fef4effd640 10 monclient: _finish_auth 0 2024-09-17T18:19:12.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.691+0000 7fef4effd640 10 monclient: _check_auth_tickets 2024-09-17T18:19:12.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.691+0000 7fef4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:12.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.691+0000 7fef4effd640 1 -- 172.21.15.121:0/1299240727 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef48008850 con 0x7fef5013a150 2024-09-17T18:19:12.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.691+0000 7fef56416640 5 monclient: authenticate success, global_id 4374 2024-09-17T18:19:12.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.691+0000 7fef56416640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:19:12.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.691+0000 7fef4effd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:12.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.691+0000 7fef56416640 10 monclient: get_monmap_and_config success 2024-09-17T18:19:12.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.691+0000 7fef4effd640 1 -- 172.21.15.121:0/1299240727 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef48008c20 con 0x7fef5013a150 2024-09-17T18:19:12.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.691+0000 7fef56416640 4 set_mon_vals no callback set 2024-09-17T18:19:12.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.691+0000 7fef4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:12.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.691+0000 7fef4effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:19:12.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.691+0000 7fef4effd640 10 monclient: dump: 2024-09-17T18:19:12.703 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:12.704 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:12.704 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:12.704 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:12.704 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:12.704 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:12.704 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:12.704 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:12.704 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:12.704 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:12.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.692+0000 7fef56416640 10 monclient: shutdown 2024-09-17T18:19:12.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.692+0000 7fef56416640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:12.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.692+0000 7fef56416640 1 -- 172.21.15.121:0/1299240727 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef5013a150 msgr2=0x7fef5013c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:12.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.692+0000 7fef56416640 1 --2- 172.21.15.121:0/1299240727 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef5013a150 0x7fef5013c540 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fef4800aed0 tx=0x7fef4802b630 comp rx=0 tx=0).stop 2024-09-17T18:19:12.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.692+0000 7fef56416640 1 -- 172.21.15.121:0/1299240727 shutdown_connections 2024-09-17T18:19:12.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.692+0000 7fef56416640 1 --2- 172.21.15.121:0/1299240727 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef5013a150 0x7fef5013c540 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:12.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.692+0000 7fef56416640 1 --2- 172.21.15.121:0/1299240727 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fef501348f0 0x7fef50139c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:12.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.692+0000 7fef56416640 1 -- 172.21.15.121:0/1299240727 >> 172.21.15.121:0/1299240727 conn(0x7fef500a3b50 msgr2=0x7fef500a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:12.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.692+0000 7fef56416640 1 -- 172.21.15.121:0/1299240727 shutdown_connections 2024-09-17T18:19:12.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.692+0000 7fef56416640 1 -- 172.21.15.121:0/1299240727 wait complete. 2024-09-17T18:19:12.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.692+0000 7fef56416640 10 monclient: build_initial_monmap 2024-09-17T18:19:12.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.692+0000 7fef56416640 10 monclient: monmap: 2024-09-17T18:19:12.705 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:12.705 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:19:12.705 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:19:12.705 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:19:12.705 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:12.705 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:12.705 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:19:12.706 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:19:12.706 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:19:12.706 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:12.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.692+0000 7fef56416640 1 librados: starting msgr at 2024-09-17T18:19:12.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.692+0000 7fef56416640 1 librados: starting objecter 2024-09-17T18:19:12.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.693+0000 7fef56416640 1 Processor -- start 2024-09-17T18:19:12.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.693+0000 7fef56416640 1 -- start start 2024-09-17T18:19:12.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.693+0000 7fef56416640 1 librados: setting wanted keys 2024-09-17T18:19:12.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.693+0000 7fef56416640 1 librados: calling monclient init 2024-09-17T18:19:12.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.693+0000 7fef56416640 10 monclient: init 2024-09-17T18:19:12.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.693+0000 7fef56416640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:12.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.693+0000 7fef56416640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T18:19:12.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.693+0000 7fef56416640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef500a0a70 0x7fef50071120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:12.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.693+0000 7fef56416640 10 monclient(hunting): picked mon.noname-b con 0x7fef500a0a70 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:12.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.693+0000 7fef56416640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fef501348f0 0x7fef50071660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:12.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.693+0000 7fef56416640 10 monclient(hunting): picked mon.noname-c con 0x7fef501348f0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:12.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.693+0000 7fef56416640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef5013a150 0x7fef50071ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:12.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.693+0000 7fef56416640 10 monclient(hunting): picked mon.noname-a con 0x7fef5013a150 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:12.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.693+0000 7fef56416640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:12.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.693+0000 7fef56416640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fef5009dce0 con 0x7fef5013a150 2024-09-17T18:19:12.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.693+0000 7fef56416640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:12.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.693+0000 7fef56416640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fef500a38e0 con 0x7fef500a0a70 2024-09-17T18:19:12.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.693+0000 7fef56416640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:12.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.693+0000 7fef56416640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fef5009de60 con 0x7fef501348f0 2024-09-17T18:19:12.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.693+0000 7fef56416640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:12.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.693+0000 7fef56416640 10 monclient(hunting): authenticate will time out at 756.409783s 2024-09-17T18:19:12.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.693+0000 7fef4f7fe640 1 -- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fef501348f0 msgr2=0x7fef50071660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:19:12.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.693+0000 7fef4f7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fef501348f0 0x7fef50071660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:19:12.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.693+0000 7fef549b3640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef5013a150 0x7fef50071ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:12.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.693+0000 7fef549b3640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef5013a150 0x7fef50071ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43670/0 (socket says 172.21.15.121:43670) 2024-09-17T18:19:12.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.693+0000 7fef549b3640 1 -- 172.21.15.121:0/3484364301 learned_addr learned my addr 172.21.15.121:0/3484364301 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:12.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.693+0000 7fef549b3640 10 monclient(hunting): get_auth_request con 0x7fef5013a150 auth_method 0 2024-09-17T18:19:12.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.693+0000 7fef549b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:12.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.693+0000 7fef549b3640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:12.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.693+0000 7fef549b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:12.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.693+0000 7fef549b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:12.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.693+0000 7fef549b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:12.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.693+0000 7fef549b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:12.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.694+0000 7fef4ffff640 1 --2- 172.21.15.121:0/3484364301 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef500a0a70 0x7fef50071120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:12.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.694+0000 7fef4ffff640 10 monclient(hunting): get_auth_request con 0x7fef500a0a70 auth_method 0 2024-09-17T18:19:12.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.694+0000 7fef4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:12.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.694+0000 7fef4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:12.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.694+0000 7fef4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:12.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.694+0000 7fef4ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:12.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.694+0000 7fef4ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:12.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.694+0000 7fef4ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:12.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.694+0000 7fef549b3640 10 monclient(hunting): handle_auth_done global_id 4377 payload 824 2024-09-17T18:19:12.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.694+0000 7fef549b3640 1 -- 172.21.15.121:0/3484364301 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fef501348f0 msgr2=0x7fef50071660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:19:12.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.694+0000 7fef549b3640 1 --2- 172.21.15.121:0/3484364301 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fef501348f0 0x7fef50071660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:12.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.694+0000 7fef549b3640 1 -- 172.21.15.121:0/3484364301 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef500a0a70 msgr2=0x7fef50071120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:12.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.694+0000 7fef549b3640 1 --2- 172.21.15.121:0/3484364301 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef500a0a70 0x7fef50071120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:12.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.694+0000 7fef549b3640 10 monclient: _finish_hunting 0 2024-09-17T18:19:12.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.694+0000 7fef549b3640 1 monclient: found mon.noname-a 2024-09-17T18:19:12.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.694+0000 7fef549b3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:12.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.694+0000 7fef549b3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:19:12.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.694+0000 7fef549b3640 1 -- 172.21.15.121:0/3484364301 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef5011db20 con 0x7fef5013a150 2024-09-17T18:19:12.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.696+0000 7fef549b3640 1 -- 172.21.15.121:0/3484364301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef5013a150 msgr2=0x7fef50071ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:12.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.696+0000 7fef549b3640 1 --2- 172.21.15.121:0/3484364301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef5013a150 0x7fef50071ba0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fef4802b210 tx=0x7fef48004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:12.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.696+0000 7fef4cff9640 10 client.?.objecter ms_handle_connect 0x7fef5013a150 2024-09-17T18:19:12.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.696+0000 7fef4cff9640 10 client.?.objecter resend_mon_ops 2024-09-17T18:19:12.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.696+0000 7fef4cff9640 1 -- 172.21.15.121:0/3484364301 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef48037050 con 0x7fef5013a150 2024-09-17T18:19:12.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.696+0000 7fef4cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:12.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.696+0000 7fef4cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:19:12.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.696+0000 7fef4cff9640 10 monclient: dump: 2024-09-17T18:19:12.711 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:12.711 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:12.711 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:12.712 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:12.712 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:12.712 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:12.712 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:12.712 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:12.712 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:12.712 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:12.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.696+0000 7fef4cff9640 10 monclient: _finish_auth 0 2024-09-17T18:19:12.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.696+0000 7fef4cff9640 10 monclient: _check_auth_tickets 2024-09-17T18:19:12.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.696+0000 7fef4cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:12.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.696+0000 7fef4cff9640 1 -- 172.21.15.121:0/3484364301 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef48043070 con 0x7fef5013a150 2024-09-17T18:19:12.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.696+0000 7fef56416640 5 monclient: authenticate success, global_id 4377 2024-09-17T18:19:12.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.696+0000 7fef56416640 10 monclient: _renew_subs 2024-09-17T18:19:12.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.696+0000 7fef56416640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:12.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.696+0000 7fef56416640 1 -- 172.21.15.121:0/3484364301 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef5009d700 con 0x7fef5013a150 2024-09-17T18:19:12.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.696+0000 7fef4cff9640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:12.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.696+0000 7fef4cff9640 1 -- 172.21.15.121:0/3484364301 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef48032d50 con 0x7fef5013a150 2024-09-17T18:19:12.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.696+0000 7fef4cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:12.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.696+0000 7fef4cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:19:12.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.696+0000 7fef4cff9640 10 monclient: dump: 2024-09-17T18:19:12.713 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:12.713 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:12.713 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:12.713 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:12.713 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:12.714 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:12.714 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:12.714 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:12.714 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:12.714 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:12.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.696+0000 7fef56416640 10 client.4377.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:12.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.696+0000 7fef56416640 10 monclient: _renew_subs 2024-09-17T18:19:12.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.696+0000 7fef4e7fc640 4 set_mon_vals no callback set 2024-09-17T18:19:12.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.696+0000 7fef56416640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:12.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.696+0000 7fef56416640 1 -- 172.21.15.121:0/3484364301 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef50073000 con 0x7fef5013a150 2024-09-17T18:19:12.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.696+0000 7fef56416640 1 librados: init done 2024-09-17T18:19:12.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.697+0000 7fef4cff9640 1 -- 172.21.15.121:0/3484364301 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fef48048460 con 0x7fef5013a150 2024-09-17T18:19:12.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.697+0000 7fef56416640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:12.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.697+0000 7fef56416640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:12.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.697+0000 7fef56416640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:12.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.697+0000 7fef56416640 1 -- 172.21.15.121:0/3484364301 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef500742c0 con 0x7fef5013a150 2024-09-17T18:19:12.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.697+0000 7fef4cff9640 1 --2- 172.21.15.121:0/3484364301 >> v2:172.21.15.173:6812/2423786037 conn(0x7fef3c040170 0x7fef3c042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:12.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.697+0000 7fef4cff9640 1 -- 172.21.15.121:0/3484364301 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 3534+0+0 (secure 0 0 0) 0x7fef48074380 con 0x7fef5013a150 2024-09-17T18:19:12.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.697+0000 7fef4cff9640 10 client.4377.objecter ms_dispatch 0x7fef50000b90 osd_map(17..17 src has 1..17) 2024-09-17T18:19:12.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.697+0000 7fef4cff9640 3 client.4377.objecter handle_osd_map got epochs [17,17] > 0 2024-09-17T18:19:12.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.697+0000 7fef4cff9640 3 client.4377.objecter handle_osd_map decoding full epoch 17 2024-09-17T18:19:12.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.698+0000 7fef4cff9640 20 client.4377.objecter dump_active .. 0 homeless 2024-09-17T18:19:12.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.698+0000 7fef4ffff640 1 --2- 172.21.15.121:0/3484364301 >> v2:172.21.15.173:6812/2423786037 conn(0x7fef3c040170 0x7fef3c042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:12.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.698+0000 7fef4ffff640 10 monclient: get_auth_request con 0x7fef3c040170 auth_method 0 2024-09-17T18:19:12.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.700+0000 7fef4cff9640 10 client.4377.objecter ms_handle_connect 0x7fef3c040170 2024-09-17T18:19:12.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.700+0000 7fef4ffff640 1 --2- 172.21.15.121:0/3484364301 >> v2:172.21.15.173:6812/2423786037 conn(0x7fef3c040170 0x7fef3c042630 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fef40007c90 tx=0x7fef40007be0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:12.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.702+0000 7fef4cff9640 1 -- 172.21.15.121:0/3484364301 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fef48073260 con 0x7fef5013a150 2024-09-17T18:19:12.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.702+0000 7fef4cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:12.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.702+0000 7fef4cff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:19:12.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.941+0000 7fef56416640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:19:12.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.941+0000 7fef56416640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:19:12.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.941+0000 7fef56416640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:12.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.941+0000 7fef56416640 1 -- 172.21.15.121:0/3484364301 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fef500a0e50 con 0x7fef5013a150 2024-09-17T18:19:12.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.942+0000 7fef4cff9640 1 -- 172.21.15.121:0/3484364301 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fef48048c70 con 0x7fef5013a150 2024-09-17T18:19:12.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.942+0000 7fef4cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:19:12.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.942+0000 7fef4cff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:19:12.945 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":25,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:19:12.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.943+0000 7fef56416640 10 librados: watch_flush enter 2024-09-17T18:19:12.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.943+0000 7fef56416640 10 librados: watch_flush exit 2024-09-17T18:19:12.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.944+0000 7fef56416640 20 client.4377.objecter shutdown clearing up homeless session... 2024-09-17T18:19:12.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.944+0000 7fef56416640 10 client.4377.objecter successfully canceled tick 2024-09-17T18:19:12.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.944+0000 7fef56416640 1 -- 172.21.15.121:0/3484364301 >> v2:172.21.15.173:6812/2423786037 conn(0x7fef3c040170 msgr2=0x7fef3c042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:12.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.944+0000 7fef56416640 1 --2- 172.21.15.121:0/3484364301 >> v2:172.21.15.173:6812/2423786037 conn(0x7fef3c040170 0x7fef3c042630 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fef40007c90 tx=0x7fef40007be0 comp rx=0 tx=0).stop 2024-09-17T18:19:12.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.944+0000 7fef56416640 10 monclient: shutdown 2024-09-17T18:19:12.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.944+0000 7fef56416640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:12.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.944+0000 7fef56416640 1 -- 172.21.15.121:0/3484364301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef5013a150 msgr2=0x7fef50071ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:12.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.944+0000 7fef56416640 1 --2- 172.21.15.121:0/3484364301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef5013a150 0x7fef50071ba0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fef4802b210 tx=0x7fef48004a40 comp rx=0 tx=0).stop 2024-09-17T18:19:12.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.944+0000 7fef56416640 1 -- 172.21.15.121:0/3484364301 shutdown_connections 2024-09-17T18:19:12.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.944+0000 7fef56416640 1 --2- 172.21.15.121:0/3484364301 >> v2:172.21.15.173:6812/2423786037 conn(0x7fef3c040170 0x7fef3c042630 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:12.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.944+0000 7fef56416640 1 --2- 172.21.15.121:0/3484364301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef5013a150 0x7fef50071ba0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:12.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.944+0000 7fef56416640 1 --2- 172.21.15.121:0/3484364301 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fef501348f0 0x7fef50071660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:12.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.944+0000 7fef56416640 1 --2- 172.21.15.121:0/3484364301 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef500a0a70 0x7fef50071120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:12.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.944+0000 7fef56416640 1 -- 172.21.15.121:0/3484364301 >> 172.21.15.121:0/3484364301 conn(0x7fef500a3b50 msgr2=0x7fef500a66c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:12.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.944+0000 7fef56416640 1 -- 172.21.15.121:0/3484364301 shutdown_connections 2024-09-17T18:19:12.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.944+0000 7fef56416640 1 -- 172.21.15.121:0/3484364301 wait complete. 2024-09-17T18:19:12.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:12.944+0000 7fef56416640 1 librados: shutdown 2024-09-17T18:19:15.957 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:19:16.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.063+0000 7fb4c8375640 10 monclient: get_monmap_and_config 2024-09-17T18:19:16.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.064+0000 7fb4c8375640 10 monclient: build_initial_monmap 2024-09-17T18:19:16.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.064+0000 7fb4c8375640 10 monclient: monmap: 2024-09-17T18:19:16.066 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:16.066 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:16.067 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:19:16.064616+0000 2024-09-17T18:19:16.067 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:19:16.064616+0000 2024-09-17T18:19:16.067 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:16.067 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:16.067 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:19:16.067 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:19:16.067 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:19:16.067 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:19:16.067 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:19:16.067 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:16.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.064+0000 7fb4c8375640 1 Processor -- start 2024-09-17T18:19:16.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.064+0000 7fb4c8375640 1 -- start start 2024-09-17T18:19:16.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.064+0000 7fb4c8375640 10 monclient: init 2024-09-17T18:19:16.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.065+0000 7fb4c8375640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:16.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.065+0000 7fb4c8375640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-17T18:19:16.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.065+0000 7fb4c8375640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb4c01358b0 addr v1:172.21.15.173:6789/0 2024-09-17T18:19:16.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.065+0000 7fb4c8375640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4c0135c90 0x7fb4c01350d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:16.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.065+0000 7fb4c8375640 10 monclient(hunting): picked mon.noname-b con 0x7fb4c0135c90 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:16.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.065+0000 7fb4c8375640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb4c0138ce0 0x7fb4c0140910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:16.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.065+0000 7fb4c8375640 10 monclient(hunting): picked mon.noname-a con 0x7fb4c0138ce0 addr v2:172.21.15.121:3300/0 2024-09-17T18:19:16.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.065+0000 7fb4c8375640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4c009d510 con 0x7fb4c01358b0 2024-09-17T18:19:16.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.065+0000 7fb4c8375640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:16.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.065+0000 7fb4c8375640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fb4c00a72b0 con 0x7fb4c0138ce0 2024-09-17T18:19:16.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.065+0000 7fb4c8375640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:16.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.065+0000 7fb4c8375640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb4c006a600 con 0x7fb4c0135c90 2024-09-17T18:19:16.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.065+0000 7fb4c8375640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:16.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.065+0000 7fb4c8375640 10 monclient(hunting): authenticate will time out at 759.781484s 2024-09-17T18:19:16.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.065+0000 7fb4c5910640 1 -- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4c0135c90 msgr2=0x7fb4c01350d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:19:16.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.065+0000 7fb4c5910640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4c0135c90 0x7fb4c01350d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:19:16.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.065+0000 7fb4c6912640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb4c0138ce0 0x7fb4c0140910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:16.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.065+0000 7fb4c6912640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb4c0138ce0 0x7fb4c0140910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43686/0 (socket says 172.21.15.121:43686) 2024-09-17T18:19:16.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.065+0000 7fb4c6912640 1 -- 172.21.15.121:0/3319021273 learned_addr learned my addr 172.21.15.121:0/3319021273 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:16.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.065+0000 7fb4c6912640 10 monclient(hunting): get_auth_request con 0x7fb4c0138ce0 auth_method 0 2024-09-17T18:19:16.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.065+0000 7fb4c6912640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:16.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.065+0000 7fb4c6912640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:16.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.065+0000 7fb4c6912640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:16.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.065+0000 7fb4c6912640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:16.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.065+0000 7fb4c6912640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:16.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.065+0000 7fb4c6912640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:16.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.065+0000 7fb4c6912640 10 monclient(hunting): handle_auth_done global_id 4380 payload 274 2024-09-17T18:19:16.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.065+0000 7fb4c6912640 1 -- 172.21.15.121:0/3319021273 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4c0135c90 msgr2=0x7fb4c01350d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:19:16.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.065+0000 7fb4c6912640 1 --2- 172.21.15.121:0/3319021273 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4c0135c90 0x7fb4c01350d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:16.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.065+0000 7fb4c6912640 1 -- 172.21.15.121:0/3319021273 >> v1:172.21.15.173:6789/0 conn(0x7fb4c01358b0 legacy=0x7fb4c01385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:16.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.065+0000 7fb4c6912640 10 monclient: _finish_hunting 0 2024-09-17T18:19:16.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.065+0000 7fb4c6912640 1 monclient: found mon.noname-a 2024-09-17T18:19:16.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.065+0000 7fb4c6912640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:16.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.065+0000 7fb4c6912640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:19:16.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.065+0000 7fb4c6912640 1 -- 172.21.15.121:0/3319021273 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4c009dca0 con 0x7fb4c0138ce0 2024-09-17T18:19:16.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.068+0000 7fb4c6912640 1 -- 172.21.15.121:0/3319021273 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4c0138ce0 msgr2=0x7fb4c0140910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:16.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.068+0000 7fb4c6912640 1 --2- 172.21.15.121:0/3319021273 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4c0138ce0 0x7fb4c0140910 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb4b8009870 tx=0x7fb4b802a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71af5b611874c9e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:16.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.068+0000 7fb4c510f640 1 -- 172.21.15.121:0/3319021273 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4b8037070 con 0x7fb4c0138ce0 2024-09-17T18:19:16.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.068+0000 7fb4c510f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:16.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.068+0000 7fb4c510f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:19:16.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.068+0000 7fb4c510f640 10 monclient: dump: 2024-09-17T18:19:16.083 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:16.083 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:16.083 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:16.083 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:16.083 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:16.083 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:16.083 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:16.083 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:16.083 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:16.084 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:16.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.068+0000 7fb4c510f640 10 monclient: _finish_auth 0 2024-09-17T18:19:16.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.068+0000 7fb4c510f640 10 monclient: _check_auth_tickets 2024-09-17T18:19:16.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.068+0000 7fb4c510f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:16.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.068+0000 7fb4c510f640 1 -- 172.21.15.121:0/3319021273 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4b8032640 con 0x7fb4c0138ce0 2024-09-17T18:19:16.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.068+0000 7fb4c8375640 5 monclient: authenticate success, global_id 4380 2024-09-17T18:19:16.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.068+0000 7fb4c8375640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:19:16.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.068+0000 7fb4c510f640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:16.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.068+0000 7fb4c8375640 10 monclient: get_monmap_and_config success 2024-09-17T18:19:16.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.068+0000 7fb4c510f640 1 -- 172.21.15.121:0/3319021273 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4b8032a10 con 0x7fb4c0138ce0 2024-09-17T18:19:16.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.068+0000 7fb4c510f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:16.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.068+0000 7fb4c8375640 4 set_mon_vals no callback set 2024-09-17T18:19:16.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.068+0000 7fb4c510f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:19:16.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.068+0000 7fb4c510f640 10 monclient: dump: 2024-09-17T18:19:16.085 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:16.085 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:16.085 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:16.085 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:16.085 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:16.085 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:16.085 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:16.085 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:16.086 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:16.086 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:16.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.069+0000 7fb4c8375640 10 monclient: shutdown 2024-09-17T18:19:16.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.069+0000 7fb4c8375640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:16.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.069+0000 7fb4c8375640 1 -- 172.21.15.121:0/3319021273 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4c0138ce0 msgr2=0x7fb4c0140910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:16.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.069+0000 7fb4c8375640 1 --2- 172.21.15.121:0/3319021273 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4c0138ce0 0x7fb4c0140910 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb4b8009870 tx=0x7fb4b802a0d0 comp rx=0 tx=0).stop 2024-09-17T18:19:16.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.069+0000 7fb4c8375640 1 -- 172.21.15.121:0/3319021273 shutdown_connections 2024-09-17T18:19:16.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.069+0000 7fb4c8375640 1 --2- 172.21.15.121:0/3319021273 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4c0138ce0 0x7fb4c0140910 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:16.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.069+0000 7fb4c8375640 1 --2- 172.21.15.121:0/3319021273 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4c0135c90 0x7fb4c01350d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:16.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.069+0000 7fb4c8375640 1 -- 172.21.15.121:0/3319021273 >> 172.21.15.121:0/3319021273 conn(0x7fb4c00a7f80 msgr2=0x7fb4c00a8390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:16.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.069+0000 7fb4c8375640 1 -- 172.21.15.121:0/3319021273 shutdown_connections 2024-09-17T18:19:16.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.069+0000 7fb4c8375640 1 -- 172.21.15.121:0/3319021273 wait complete. 2024-09-17T18:19:16.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.070+0000 7fb4c8375640 10 monclient: build_initial_monmap 2024-09-17T18:19:16.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.070+0000 7fb4c8375640 10 monclient: monmap: 2024-09-17T18:19:16.087 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:16.087 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:19:16.087 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:19:16.087 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:19:16.087 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:16.087 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:16.087 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:19:16.087 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:19:16.087 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:19:16.088 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:16.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.070+0000 7fb4c8375640 1 librados: starting msgr at 2024-09-17T18:19:16.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.070+0000 7fb4c8375640 1 librados: starting objecter 2024-09-17T18:19:16.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.070+0000 7fb4c8375640 1 Processor -- start 2024-09-17T18:19:16.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.070+0000 7fb4c8375640 1 -- start start 2024-09-17T18:19:16.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.070+0000 7fb4c8375640 1 librados: setting wanted keys 2024-09-17T18:19:16.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.070+0000 7fb4c8375640 1 librados: calling monclient init 2024-09-17T18:19:16.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.070+0000 7fb4c8375640 10 monclient: init 2024-09-17T18:19:16.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c8375640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:16.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c8375640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T18:19:16.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c8375640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4c01358b0 0x7fb4c01ca570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:16.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c8375640 10 monclient(hunting): picked mon.noname-b con 0x7fb4c01358b0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:16.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c8375640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4c0135c90 0x7fb4c0132240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:16.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c8375640 10 monclient(hunting): picked mon.noname-a con 0x7fb4c0135c90 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:16.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c8375640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4c0138ce0 0x7fb4c01cdab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:16.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c8375640 10 monclient(hunting): picked mon.noname-c con 0x7fb4c0138ce0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:16.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c8375640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:16.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c8375640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c00707a0 con 0x7fb4c0135c90 2024-09-17T18:19:16.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c8375640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:16.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c8375640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c0070620 con 0x7fb4c01358b0 2024-09-17T18:19:16.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c8375640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:16.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c8375640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb4c0139190 con 0x7fb4c0138ce0 2024-09-17T18:19:16.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c8375640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:16.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c8375640 10 monclient(hunting): authenticate will time out at 759.787145s 2024-09-17T18:19:16.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c6912640 1 -- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4c0138ce0 msgr2=0x7fb4c01cdab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:19:16.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c6912640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4c0138ce0 0x7fb4c01cdab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:19:16.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c5910640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4c0135c90 0x7fb4c0132240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:16.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c5910640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4c0135c90 0x7fb4c0132240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43700/0 (socket says 172.21.15.121:43700) 2024-09-17T18:19:16.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c6111640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4c01358b0 0x7fb4c01ca570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:16.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c5910640 1 -- 172.21.15.121:0/1768887447 learned_addr learned my addr 172.21.15.121:0/1768887447 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:16.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c5910640 10 monclient(hunting): get_auth_request con 0x7fb4c0135c90 auth_method 0 2024-09-17T18:19:16.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c5910640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:16.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c5910640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:16.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c5910640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:16.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c6111640 10 monclient(hunting): get_auth_request con 0x7fb4c01358b0 auth_method 0 2024-09-17T18:19:16.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c6111640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:16.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c6111640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:16.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c6111640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:16.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c5910640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:16.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c5910640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:16.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c5910640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:16.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c6111640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:16.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c6111640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:16.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c6111640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:16.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c5910640 10 monclient(hunting): handle_auth_done global_id 4383 payload 824 2024-09-17T18:19:16.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c5910640 1 -- 172.21.15.121:0/1768887447 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4c0138ce0 msgr2=0x7fb4c01cdab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:19:16.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c5910640 1 --2- 172.21.15.121:0/1768887447 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4c0138ce0 0x7fb4c01cdab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:16.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c5910640 1 -- 172.21.15.121:0/1768887447 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4c01358b0 msgr2=0x7fb4c01ca570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:16.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c5910640 1 --2- 172.21.15.121:0/1768887447 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4c01358b0 0x7fb4c01ca570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:16.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c5910640 10 monclient: _finish_hunting 0 2024-09-17T18:19:16.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c5910640 1 monclient: found mon.noname-a 2024-09-17T18:19:16.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c5910640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:16.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c5910640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:19:16.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.071+0000 7fb4c5910640 1 -- 172.21.15.121:0/1768887447 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4c01224e0 con 0x7fb4c0135c90 2024-09-17T18:19:16.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.073+0000 7fb4c5910640 1 -- 172.21.15.121:0/1768887447 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4c0135c90 msgr2=0x7fb4c0132240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:16.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.073+0000 7fb4c5910640 1 --2- 172.21.15.121:0/1768887447 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4c0135c90 0x7fb4c0132240 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb4b000b520 tx=0x7fb4b000b9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:16.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.073+0000 7fb4b6ffd640 10 client.?.objecter ms_handle_connect 0x7fb4c0135c90 2024-09-17T18:19:16.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.073+0000 7fb4b6ffd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:19:16.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.073+0000 7fb4b6ffd640 1 -- 172.21.15.121:0/1768887447 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4b0003da0 con 0x7fb4c0135c90 2024-09-17T18:19:16.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.073+0000 7fb4b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:16.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.073+0000 7fb4b6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:19:16.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.073+0000 7fb4b6ffd640 10 monclient: dump: 2024-09-17T18:19:16.094 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:16.094 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:16.094 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:16.094 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:16.094 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:16.094 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:16.094 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:16.094 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:16.094 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:16.094 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:16.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.073+0000 7fb4b6ffd640 10 monclient: _finish_auth 0 2024-09-17T18:19:16.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.073+0000 7fb4b6ffd640 10 monclient: _check_auth_tickets 2024-09-17T18:19:16.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.073+0000 7fb4b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:16.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.073+0000 7fb4c8375640 5 monclient: authenticate success, global_id 4383 2024-09-17T18:19:16.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.073+0000 7fb4b6ffd640 1 -- 172.21.15.121:0/1768887447 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4b0015070 con 0x7fb4c0135c90 2024-09-17T18:19:16.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.073+0000 7fb4b6ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:16.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.073+0000 7fb4c8375640 10 monclient: _renew_subs 2024-09-17T18:19:16.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.073+0000 7fb4b6ffd640 1 -- 172.21.15.121:0/1768887447 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4b0005290 con 0x7fb4c0135c90 2024-09-17T18:19:16.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.073+0000 7fb4c8375640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:16.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.073+0000 7fb4c8375640 1 -- 172.21.15.121:0/1768887447 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4c0134a30 con 0x7fb4c0135c90 2024-09-17T18:19:16.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.073+0000 7fb4b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:16.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.073+0000 7fb4b6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:19:16.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.073+0000 7fb4b6ffd640 10 monclient: dump: 2024-09-17T18:19:16.096 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:16.096 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:16.096 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:16.096 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:16.096 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:16.096 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:16.096 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:16.096 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:16.096 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:16.096 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:16.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.073+0000 7fb4c490e640 4 set_mon_vals no callback set 2024-09-17T18:19:16.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.074+0000 7fb4c8375640 10 client.4383.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:16.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.074+0000 7fb4c8375640 10 monclient: _renew_subs 2024-09-17T18:19:16.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.074+0000 7fb4c8375640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:16.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.074+0000 7fb4c8375640 1 -- 172.21.15.121:0/1768887447 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4c0122780 con 0x7fb4c0135c90 2024-09-17T18:19:16.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.074+0000 7fb4c8375640 1 librados: init done 2024-09-17T18:19:16.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.074+0000 7fb4b6ffd640 1 -- 172.21.15.121:0/1768887447 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fb4b00054b0 con 0x7fb4c0135c90 2024-09-17T18:19:16.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.075+0000 7fb4c8375640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:16.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.075+0000 7fb4c8375640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:16.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.075+0000 7fb4c8375640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:16.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.075+0000 7fb4c8375640 1 -- 172.21.15.121:0/1768887447 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4c01350d0 con 0x7fb4c0135c90 2024-09-17T18:19:16.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.075+0000 7fb4b6ffd640 1 --2- 172.21.15.121:0/1768887447 >> v2:172.21.15.173:6812/2423786037 conn(0x7fb4a4040120 0x7fb4a40425e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:16.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.075+0000 7fb4b6ffd640 1 -- 172.21.15.121:0/1768887447 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 3534+0+0 (secure 0 0 0) 0x7fb4b0052960 con 0x7fb4c0135c90 2024-09-17T18:19:16.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.075+0000 7fb4b6ffd640 10 client.4383.objecter ms_dispatch 0x7fb4c0000b90 osd_map(17..17 src has 1..17) 2024-09-17T18:19:16.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.075+0000 7fb4b6ffd640 3 client.4383.objecter handle_osd_map got epochs [17,17] > 0 2024-09-17T18:19:16.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.075+0000 7fb4b6ffd640 3 client.4383.objecter handle_osd_map decoding full epoch 17 2024-09-17T18:19:16.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.075+0000 7fb4b6ffd640 20 client.4383.objecter dump_active .. 0 homeless 2024-09-17T18:19:16.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.075+0000 7fb4c6111640 1 --2- 172.21.15.121:0/1768887447 >> v2:172.21.15.173:6812/2423786037 conn(0x7fb4a4040120 0x7fb4a40425e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:16.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.075+0000 7fb4c6111640 10 monclient: get_auth_request con 0x7fb4a4040120 auth_method 0 2024-09-17T18:19:16.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.077+0000 7fb4b6ffd640 10 client.4383.objecter ms_handle_connect 0x7fb4a4040120 2024-09-17T18:19:16.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.077+0000 7fb4c6111640 1 --2- 172.21.15.121:0/1768887447 >> v2:172.21.15.173:6812/2423786037 conn(0x7fb4a4040120 0x7fb4a40425e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc005660 tx=0x7fb4bc003d90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:16.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.079+0000 7fb4b6ffd640 1 -- 172.21.15.121:0/1768887447 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb4b0051840 con 0x7fb4c0135c90 2024-09-17T18:19:16.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.079+0000 7fb4b6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:16.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.079+0000 7fb4b6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:19:16.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.322+0000 7fb4c8375640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:19:16.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.322+0000 7fb4c8375640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:19:16.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.322+0000 7fb4c8375640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:16.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.322+0000 7fb4c8375640 1 -- 172.21.15.121:0/1768887447 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb4c00a7090 con 0x7fb4c0135c90 2024-09-17T18:19:16.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.323+0000 7fb4b6ffd640 1 -- 172.21.15.121:0/1768887447 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fb4b001a460 con 0x7fb4c0135c90 2024-09-17T18:19:16.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.323+0000 7fb4b6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:19:16.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.323+0000 7fb4b6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:19:16.326 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":29,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:19:16.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.325+0000 7fb4c8375640 10 librados: watch_flush enter 2024-09-17T18:19:16.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.325+0000 7fb4c8375640 10 librados: watch_flush exit 2024-09-17T18:19:16.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.325+0000 7fb4c8375640 20 client.4383.objecter shutdown clearing up homeless session... 2024-09-17T18:19:16.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.325+0000 7fb4c8375640 10 client.4383.objecter successfully canceled tick 2024-09-17T18:19:16.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.325+0000 7fb4c8375640 1 -- 172.21.15.121:0/1768887447 >> v2:172.21.15.173:6812/2423786037 conn(0x7fb4a4040120 msgr2=0x7fb4a40425e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:16.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.325+0000 7fb4c8375640 1 --2- 172.21.15.121:0/1768887447 >> v2:172.21.15.173:6812/2423786037 conn(0x7fb4a4040120 0x7fb4a40425e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc005660 tx=0x7fb4bc003d90 comp rx=0 tx=0).stop 2024-09-17T18:19:16.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.325+0000 7fb4c8375640 10 monclient: shutdown 2024-09-17T18:19:16.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.325+0000 7fb4c8375640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:16.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.325+0000 7fb4c8375640 1 -- 172.21.15.121:0/1768887447 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4c0135c90 msgr2=0x7fb4c0132240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:16.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.325+0000 7fb4c8375640 1 --2- 172.21.15.121:0/1768887447 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4c0135c90 0x7fb4c0132240 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb4b000b520 tx=0x7fb4b000b9e0 comp rx=0 tx=0).stop 2024-09-17T18:19:16.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.325+0000 7fb4c8375640 1 -- 172.21.15.121:0/1768887447 shutdown_connections 2024-09-17T18:19:16.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.325+0000 7fb4c8375640 1 --2- 172.21.15.121:0/1768887447 >> v2:172.21.15.173:6812/2423786037 conn(0x7fb4a4040120 0x7fb4a40425e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:16.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.325+0000 7fb4c8375640 1 --2- 172.21.15.121:0/1768887447 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4c0138ce0 0x7fb4c01cdab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:16.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.325+0000 7fb4c8375640 1 --2- 172.21.15.121:0/1768887447 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4c0135c90 0x7fb4c0132240 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:16.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.325+0000 7fb4c8375640 1 --2- 172.21.15.121:0/1768887447 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4c01358b0 0x7fb4c01ca570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:16.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.325+0000 7fb4c8375640 1 -- 172.21.15.121:0/1768887447 >> 172.21.15.121:0/1768887447 conn(0x7fb4c00a7f80 msgr2=0x7fb4c00a9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:16.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.325+0000 7fb4c8375640 1 -- 172.21.15.121:0/1768887447 shutdown_connections 2024-09-17T18:19:16.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.325+0000 7fb4c8375640 1 -- 172.21.15.121:0/1768887447 wait complete. 2024-09-17T18:19:16.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:16.325+0000 7fb4c8375640 1 librados: shutdown 2024-09-17T18:19:19.339 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:19:19.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.446+0000 7f3dc9117640 10 monclient: get_monmap_and_config 2024-09-17T18:19:19.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.446+0000 7f3dc9117640 10 monclient: build_initial_monmap 2024-09-17T18:19:19.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.446+0000 7f3dc9117640 10 monclient: monmap: 2024-09-17T18:19:19.449 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:19.449 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:19.449 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:19:19.447501+0000 2024-09-17T18:19:19.449 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:19:19.447501+0000 2024-09-17T18:19:19.449 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:19.449 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:19.449 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:19:19.449 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:19:19.449 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:19:19.449 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:19:19.449 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:19:19.449 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:19.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.447+0000 7f3dc9117640 1 Processor -- start 2024-09-17T18:19:19.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.447+0000 7f3dc9117640 1 -- start start 2024-09-17T18:19:19.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.447+0000 7f3dc9117640 10 monclient: init 2024-09-17T18:19:19.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.448+0000 7f3dc9117640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:19.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.448+0000 7f3dc9117640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-17T18:19:19.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.448+0000 7f3dc9117640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3dc4136800 0x7f3dc4136170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:19.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.448+0000 7f3dc9117640 10 monclient(hunting): picked mon.noname-b con 0x7f3dc4136800 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:19.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.448+0000 7f3dc9117640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f3dc4136be0 0x7f3dc4132c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:19.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.448+0000 7f3dc9117640 10 monclient(hunting): picked mon.noname-a con 0x7f3dc4136be0 addr v2:172.21.15.121:3300/0 2024-09-17T18:19:19.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.448+0000 7f3dc9117640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3dc4133190 addr v1:172.21.15.121:6789/0 2024-09-17T18:19:19.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.448+0000 7f3dc9117640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dc40a0c20 con 0x7f3dc4133190 2024-09-17T18:19:19.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.448+0000 7f3dc9117640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:19.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.448+0000 7f3dc9117640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f3dc40a2cb0 con 0x7f3dc4136be0 2024-09-17T18:19:19.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.448+0000 7f3dc9117640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:19.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.448+0000 7f3dc9117640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3dc40a2f90 con 0x7f3dc4136800 2024-09-17T18:19:19.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.448+0000 7f3dc9117640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:19.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.448+0000 7f3dc9117640 10 monclient(hunting): authenticate will time out at 763.164117s 2024-09-17T18:19:19.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.448+0000 7f3dc2d9d640 1 -- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3dc4136800 msgr2=0x7f3dc4136170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:19:19.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.448+0000 7f3dc2d9d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3dc4136800 0x7f3dc4136170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:19:19.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.448+0000 7f3dc259c640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f3dc4136be0 0x7f3dc4132c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:19.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.448+0000 7f3dc259c640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f3dc4136be0 0x7f3dc4132c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38494/0 (socket says 172.21.15.121:38494) 2024-09-17T18:19:19.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.448+0000 7f3dc259c640 1 -- 172.21.15.121:0/720812916 learned_addr learned my addr 172.21.15.121:0/720812916 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:19.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.448+0000 7f3dc259c640 10 monclient(hunting): get_auth_request con 0x7f3dc4136be0 auth_method 0 2024-09-17T18:19:19.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.448+0000 7f3dc359e640 1 --1- 172.21.15.121:0/720812916 >> v1:172.21.15.121:6789/0 conn(0x7f3dc4133190 0x7f3dc413d500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:58288/0 (socket says 172.21.15.121:58288) 2024-09-17T18:19:19.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.448+0000 7f3dc259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:19.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.448+0000 7f3dc259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:19.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.448+0000 7f3dc259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:19.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.448+0000 7f3dc259c640 1 -- 172.21.15.121:0/720812916 >> v2:172.21.15.121:3300/0 conn(0x7f3dc4136be0 msgr2=0x7f3dc4132c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T18:19:19.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.448+0000 7f3dc259c640 1 -- 172.21.15.121:0/720812916 >> v2:172.21.15.121:3300/0 conn(0x7f3dc4136be0 msgr2=0x7f3dc4132c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:19:19.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.448+0000 7f3dc259c640 1 --2- 172.21.15.121:0/720812916 >> v2:172.21.15.121:3300/0 conn(0x7f3dc4136be0 0x7f3dc4132c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:19:19.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.448+0000 7f3dc259c640 1 --2- 172.21.15.121:0/720812916 >> v2:172.21.15.121:3300/0 conn(0x7f3dc4136be0 0x7f3dc4132c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:19:19.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.450+0000 7f3dc359e640 1 -- 172.21.15.121:0/720812916 >> v1:172.21.15.121:6789/0 conn(0x7f3dc4133190 legacy=0x7f3dc413d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:19.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.649+0000 7f3dc2d9d640 1 -- 172.21.15.121:0/720812916 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3dc4136800 msgr2=0x7f3dc4136170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:19:19.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.649+0000 7f3dc2d9d640 1 --2- 172.21.15.121:0/720812916 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3dc4136800 0x7f3dc4136170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T18:19:19.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.649+0000 7f3dc259c640 1 --2- 172.21.15.121:0/720812916 >> v2:172.21.15.121:3300/0 conn(0x7f3dc4136be0 0x7f3dc4132c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:19.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.649+0000 7f3dc259c640 10 monclient(hunting): get_auth_request con 0x7f3dc4136be0 auth_method 0 2024-09-17T18:19:19.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.649+0000 7f3dc259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:19.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.649+0000 7f3dc259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:19.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.649+0000 7f3dc259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:19:19.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.649+0000 7f3dc259c640 1 -- 172.21.15.121:0/720812916 >> v2:172.21.15.121:3300/0 conn(0x7f3dc4136be0 msgr2=0x7f3dc4132c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:19:19.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.649+0000 7f3dc259c640 1 -- 172.21.15.121:0/720812916 >> v2:172.21.15.121:3300/0 conn(0x7f3dc4136be0 msgr2=0x7f3dc4132c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:19:19.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.649+0000 7f3dc259c640 1 --2- 172.21.15.121:0/720812916 >> v2:172.21.15.121:3300/0 conn(0x7f3dc4136be0 0x7f3dc4132c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:19:19.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:19.649+0000 7f3dc259c640 1 --2- 172.21.15.121:0/720812916 >> v2:172.21.15.121:3300/0 conn(0x7f3dc4136be0 0x7f3dc4132c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T18:19:20.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:20.050+0000 7f3dc2d9d640 1 -- 172.21.15.121:0/720812916 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3dc4136800 msgr2=0x7f3dc4136170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:19:20.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:20.050+0000 7f3dc2d9d640 1 --2- 172.21.15.121:0/720812916 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3dc4136800 0x7f3dc4136170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T18:19:20.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:20.050+0000 7f3dc259c640 1 --2- 172.21.15.121:0/720812916 >> v2:172.21.15.121:3300/0 conn(0x7f3dc4136be0 0x7f3dc4132c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:20.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:20.050+0000 7f3dc259c640 10 monclient(hunting): get_auth_request con 0x7f3dc4136be0 auth_method 0 2024-09-17T18:19:20.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:20.050+0000 7f3dc259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:20.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:20.050+0000 7f3dc259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:20.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:20.050+0000 7f3dc259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:19:20.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:20.050+0000 7f3dc259c640 1 -- 172.21.15.121:0/720812916 >> v2:172.21.15.121:3300/0 conn(0x7f3dc4136be0 msgr2=0x7f3dc4132c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T18:19:20.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:20.050+0000 7f3dc259c640 1 -- 172.21.15.121:0/720812916 >> v2:172.21.15.121:3300/0 conn(0x7f3dc4136be0 msgr2=0x7f3dc4132c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:19:20.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:20.050+0000 7f3dc259c640 1 --2- 172.21.15.121:0/720812916 >> v2:172.21.15.121:3300/0 conn(0x7f3dc4136be0 0x7f3dc4132c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:19:20.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:20.050+0000 7f3dc259c640 1 --2- 172.21.15.121:0/720812916 >> v2:172.21.15.121:3300/0 conn(0x7f3dc4136be0 0x7f3dc4132c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T18:19:20.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:20.851+0000 7f3dc2d9d640 1 -- 172.21.15.121:0/720812916 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3dc4136800 msgr2=0x7f3dc4136170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:19:20.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:20.851+0000 7f3dc2d9d640 1 --2- 172.21.15.121:0/720812916 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3dc4136800 0x7f3dc4136170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T18:19:20.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:20.851+0000 7f3dc259c640 1 --2- 172.21.15.121:0/720812916 >> v2:172.21.15.121:3300/0 conn(0x7f3dc4136be0 0x7f3dc4132c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:20.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:20.851+0000 7f3dc259c640 10 monclient(hunting): get_auth_request con 0x7f3dc4136be0 auth_method 0 2024-09-17T18:19:20.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:20.851+0000 7f3dc259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:20.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:20.851+0000 7f3dc259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:20.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:20.851+0000 7f3dc259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:19:20.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:20.851+0000 7f3dc259c640 1 -- 172.21.15.121:0/720812916 >> v2:172.21.15.121:3300/0 conn(0x7f3dc4136be0 msgr2=0x7f3dc4132c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:19:20.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:20.851+0000 7f3dc259c640 1 -- 172.21.15.121:0/720812916 >> v2:172.21.15.121:3300/0 conn(0x7f3dc4136be0 msgr2=0x7f3dc4132c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:19:20.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:20.851+0000 7f3dc259c640 1 --2- 172.21.15.121:0/720812916 >> v2:172.21.15.121:3300/0 conn(0x7f3dc4136be0 0x7f3dc4132c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:19:20.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:20.851+0000 7f3dc259c640 1 --2- 172.21.15.121:0/720812916 >> v2:172.21.15.121:3300/0 conn(0x7f3dc4136be0 0x7f3dc4132c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T18:19:22.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:22.448+0000 7f3dc0d99640 10 monclient(hunting): tick 2024-09-17T18:19:22.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:22.448+0000 7f3dc0d99640 10 monclient(hunting): _check_auth_tickets 2024-09-17T18:19:22.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:22.448+0000 7f3dc0d99640 1 monclient(hunting): continuing hunt 2024-09-17T18:19:22.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:22.448+0000 7f3dc0d99640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T18:19:22.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:22.448+0000 7f3dc0d99640 1 -- 172.21.15.121:0/720812916 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3dc4136800 msgr2=0x7f3dc4136170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:19:22.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:22.448+0000 7f3dc0d99640 1 --2- 172.21.15.121:0/720812916 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3dc4136800 0x7f3dc4136170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:22.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:22.448+0000 7f3dc0d99640 1 -- 172.21.15.121:0/720812916 >> v2:172.21.15.121:3300/0 conn(0x7f3dc4136be0 msgr2=0x7f3dc4132c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:19:22.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:22.448+0000 7f3dc0d99640 1 --2- 172.21.15.121:0/720812916 >> v2:172.21.15.121:3300/0 conn(0x7f3dc4136be0 0x7f3dc4132c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:22.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:22.448+0000 7f3dc0d99640 1 -- 172.21.15.121:0/720812916 >> v1:172.21.15.121:6789/0 conn(0x7f3dc4133190 legacy=0x7f3dc413d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:22.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:22.448+0000 7f3dc0d99640 10 monclient: _add_conns ranks=[1,4] 2024-09-17T18:19:22.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:22.448+0000 7f3dc0d99640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3da4003760 addr v1:172.21.15.173:6789/0 2024-09-17T18:19:22.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:22.448+0000 7f3dc0d99640 1 --2- 172.21.15.121:0/720812916 >> v2:172.21.15.173:3300/0 conn(0x7f3da4007490 0x7f3da4009950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:22.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:22.448+0000 7f3dc0d99640 10 monclient(hunting): picked mon.noname-c con 0x7f3da4007490 addr v2:172.21.15.173:3300/0 2024-09-17T18:19:22.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:22.448+0000 7f3dc0d99640 1 -- 172.21.15.121:0/720812916 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dc40a3270 con 0x7f3da4003760 2024-09-17T18:19:22.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:22.448+0000 7f3dc0d99640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:22.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:22.448+0000 7f3dc0d99640 1 -- 172.21.15.121:0/720812916 --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f3dc40a2cb0 con 0x7f3da4007490 2024-09-17T18:19:22.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:22.448+0000 7f3dc0d99640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:22.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:22.448+0000 7f3dc259c640 1 --2- 172.21.15.121:0/720812916 >> v2:172.21.15.173:3300/0 conn(0x7f3da4007490 0x7f3da4009950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:22.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:22.448+0000 7f3dc259c640 10 monclient(hunting): get_auth_request con 0x7f3da4007490 auth_method 0 2024-09-17T18:19:22.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:22.448+0000 7f3dc259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:22.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:22.448+0000 7f3dc259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:22.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:22.448+0000 7f3dc259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:22.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:22.448+0000 7f3dc259c640 1 -- 172.21.15.121:0/720812916 >> v2:172.21.15.173:3300/0 conn(0x7f3da4007490 msgr2=0x7f3da4009950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T18:19:22.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:22.448+0000 7f3dc259c640 1 -- 172.21.15.121:0/720812916 >> v2:172.21.15.173:3300/0 conn(0x7f3da4007490 msgr2=0x7f3da4009950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:19:22.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:22.448+0000 7f3dc259c640 1 --2- 172.21.15.121:0/720812916 >> v2:172.21.15.173:3300/0 conn(0x7f3da4007490 0x7f3da4009950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:19:22.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:22.448+0000 7f3dc259c640 1 --2- 172.21.15.121:0/720812916 >> v2:172.21.15.173:3300/0 conn(0x7f3da4007490 0x7f3da4009950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:19:22.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:22.450+0000 7f3dc2d9d640 1 -- 172.21.15.121:0/720812916 >> v1:172.21.15.173:6789/0 conn(0x7f3da4003760 legacy=0x7f3da4005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:22.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:22.649+0000 7f3dc259c640 1 --2- 172.21.15.121:0/720812916 >> v2:172.21.15.173:3300/0 conn(0x7f3da4007490 0x7f3da4009950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:22.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:22.649+0000 7f3dc259c640 10 monclient(hunting): get_auth_request con 0x7f3da4007490 auth_method 0 2024-09-17T18:19:22.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:22.649+0000 7f3dc259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:22.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:22.649+0000 7f3dc259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:22.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:22.649+0000 7f3dc259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:19:22.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:22.649+0000 7f3dc259c640 1 -- 172.21.15.121:0/720812916 >> v2:172.21.15.173:3300/0 conn(0x7f3da4007490 msgr2=0x7f3da4009950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T18:19:22.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:22.649+0000 7f3dc259c640 1 -- 172.21.15.121:0/720812916 >> v2:172.21.15.173:3300/0 conn(0x7f3da4007490 msgr2=0x7f3da4009950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:19:22.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:22.649+0000 7f3dc259c640 1 --2- 172.21.15.121:0/720812916 >> v2:172.21.15.173:3300/0 conn(0x7f3da4007490 0x7f3da4009950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:19:22.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:22.649+0000 7f3dc259c640 1 --2- 172.21.15.121:0/720812916 >> v2:172.21.15.173:3300/0 conn(0x7f3da4007490 0x7f3da4009950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T18:19:23.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:23.050+0000 7f3dc259c640 1 --2- 172.21.15.121:0/720812916 >> v2:172.21.15.173:3300/0 conn(0x7f3da4007490 0x7f3da4009950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:23.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:23.050+0000 7f3dc259c640 10 monclient(hunting): get_auth_request con 0x7f3da4007490 auth_method 0 2024-09-17T18:19:23.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:23.050+0000 7f3dc259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:23.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:23.050+0000 7f3dc259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:23.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:23.050+0000 7f3dc259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:19:23.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:23.050+0000 7f3dc259c640 1 -- 172.21.15.121:0/720812916 >> v2:172.21.15.173:3300/0 conn(0x7f3da4007490 msgr2=0x7f3da4009950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T18:19:23.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:23.050+0000 7f3dc259c640 1 -- 172.21.15.121:0/720812916 >> v2:172.21.15.173:3300/0 conn(0x7f3da4007490 msgr2=0x7f3da4009950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:19:23.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:23.050+0000 7f3dc259c640 1 --2- 172.21.15.121:0/720812916 >> v2:172.21.15.173:3300/0 conn(0x7f3da4007490 0x7f3da4009950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:19:23.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:23.050+0000 7f3dc259c640 1 --2- 172.21.15.121:0/720812916 >> v2:172.21.15.173:3300/0 conn(0x7f3da4007490 0x7f3da4009950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T18:19:23.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:23.851+0000 7f3dc259c640 1 --2- 172.21.15.121:0/720812916 >> v2:172.21.15.173:3300/0 conn(0x7f3da4007490 0x7f3da4009950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:23.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:23.851+0000 7f3dc259c640 10 monclient(hunting): get_auth_request con 0x7f3da4007490 auth_method 0 2024-09-17T18:19:23.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:23.851+0000 7f3dc259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:23.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:23.851+0000 7f3dc259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:23.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:23.851+0000 7f3dc259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:19:23.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:23.851+0000 7f3dc259c640 1 -- 172.21.15.121:0/720812916 >> v2:172.21.15.173:3300/0 conn(0x7f3da4007490 msgr2=0x7f3da4009950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T18:19:23.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:23.851+0000 7f3dc259c640 1 -- 172.21.15.121:0/720812916 >> v2:172.21.15.173:3300/0 conn(0x7f3da4007490 msgr2=0x7f3da4009950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:19:23.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:23.851+0000 7f3dc259c640 1 --2- 172.21.15.121:0/720812916 >> v2:172.21.15.173:3300/0 conn(0x7f3da4007490 0x7f3da4009950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:19:23.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:23.851+0000 7f3dc259c640 1 --2- 172.21.15.121:0/720812916 >> v2:172.21.15.173:3300/0 conn(0x7f3da4007490 0x7f3da4009950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T18:19:24.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc1d9b640 1 -- 172.21.15.121:0/720812916 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 203448081 0 0) 0x7f3dc40a3270 con 0x7f3da4003760 2024-09-17T18:19:24.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc1d9b640 10 monclient(hunting): _negotiate 2024-09-17T18:19:24.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:24.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:24.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc1d9b640 10 monclient(hunting): my global_id is 4378 2024-09-17T18:19:24.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc1d9b640 1 -- 172.21.15.121:0/720812916 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3db0003520 con 0x7f3da4003760 2024-09-17T18:19:24.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc1d9b640 1 -- 172.21.15.121:0/720812916 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 340475319 0 0) 0x7f3db0003520 con 0x7f3da4003760 2024-09-17T18:19:24.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc1d9b640 1 -- 172.21.15.121:0/720812916 >> v2:172.21.15.173:3300/0 conn(0x7f3da4007490 msgr2=0x7f3da4009950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:19:24.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc1d9b640 1 --2- 172.21.15.121:0/720812916 >> v2:172.21.15.173:3300/0 conn(0x7f3da4007490 0x7f3da4009950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:24.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc1d9b640 10 monclient: _finish_hunting 0 2024-09-17T18:19:24.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc1d9b640 1 monclient: found mon.noname-c-legacy 2024-09-17T18:19:24.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:24.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc1d9b640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.173:6789/0 2024-09-17T18:19:24.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc1d9b640 1 -- 172.21.15.121:0/720812916 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3da400b0e0 con 0x7f3da4003760 2024-09-17T18:19:24.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc1d9b640 10 monclient: _finish_auth 0 2024-09-17T18:19:24.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc1d9b640 10 monclient: _check_auth_tickets 2024-09-17T18:19:24.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:24.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc1d9b640 1 -- 172.21.15.121:0/720812916 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4002019445 0 0) 0x7f3db4004980 con 0x7f3da4003760 2024-09-17T18:19:24.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc9117640 5 monclient: authenticate success, global_id 4378 2024-09-17T18:19:24.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc9117640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:19:24.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:24.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc1d9b640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-17T18:19:24.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc1d9b640 10 monclient: dump: 2024-09-17T18:19:24.054 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:24.054 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:24.054 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:24.054 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:24.054 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:24.054 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:24.054 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:24.054 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:24.054 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:24.054 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:24.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc1d9b640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] but i'm connected to v1:172.21.15.173:6789/0, reconnecting 2024-09-17T18:19:24.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc1d9b640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:24.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc1d9b640 1 -- 172.21.15.121:0/720812916 >> v1:172.21.15.173:6789/0 conn(0x7f3da4003760 legacy=0x7f3da4005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:24.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc1d9b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T18:19:24.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc359e640 1 -- 172.21.15.121:0/720812916 reap_dead start 2024-09-17T18:19:24.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc1d9b640 1 --2- 172.21.15.121:0/720812916 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3db00039b0 0x7f3db0003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:24.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc1d9b640 10 monclient(hunting): picked mon.a con 0x7f3db00039b0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:24.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc1d9b640 1 --2- 172.21.15.121:0/720812916 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3db0004800 0x7f3db0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:24.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc1d9b640 10 monclient(hunting): picked mon.c con 0x7f3db0004800 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:24.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc1d9b640 1 --2- 172.21.15.121:0/720812916 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3db0005290 0x7f3db000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:24.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc1d9b640 10 monclient(hunting): picked mon.b con 0x7f3db0005290 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:24.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc1d9b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:24.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc1d9b640 1 -- 172.21.15.121:0/720812916 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc40a2cb0 con 0x7f3db00039b0 2024-09-17T18:19:24.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc1d9b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:24.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc1d9b640 1 -- 172.21.15.121:0/720812916 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3db000c1e0 con 0x7f3db0005290 2024-09-17T18:19:24.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc1d9b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:24.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc1d9b640 1 -- 172.21.15.121:0/720812916 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3db000c3e0 con 0x7f3db0004800 2024-09-17T18:19:24.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc1d9b640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:24.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc1d9b640 10 monclient(hunting): _finish_auth 0 2024-09-17T18:19:24.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc259c640 1 -- 172.21.15.121:0/720812916 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3db0004800 msgr2=0x7f3db0004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:19:24.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc259c640 1 --2- 172.21.15.121:0/720812916 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3db0004800 0x7f3db0004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:19:24.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc9117640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-17T18:19:24.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc2d9d640 1 --2- 172.21.15.121:0/720812916 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3db00039b0 0x7f3db0003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:24.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc2d9d640 10 monclient(hunting): get_auth_request con 0x7f3db00039b0 auth_method 0 2024-09-17T18:19:24.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:24.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:24.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc2d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:19:24.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc359e640 1 --2- 172.21.15.121:0/720812916 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3db0005290 0x7f3db000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:24.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc359e640 10 monclient(hunting): get_auth_request con 0x7f3db0005290 auth_method 0 2024-09-17T18:19:24.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:24.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc359e640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:24.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc359e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:19:24.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:24.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:24.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.043+0000 7f3dc2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:19:24.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.044+0000 7f3dc359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:24.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.044+0000 7f3dc359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:24.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.044+0000 7f3dc359e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:19:24.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.044+0000 7f3dc2d9d640 10 monclient(hunting): handle_auth_done global_id 4378 payload 293 2024-09-17T18:19:24.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.044+0000 7f3dc2d9d640 1 -- 172.21.15.121:0/720812916 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3db0004800 msgr2=0x7f3db0004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:19:24.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.044+0000 7f3dc2d9d640 1 --2- 172.21.15.121:0/720812916 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3db0004800 0x7f3db0004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:24.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.044+0000 7f3dc2d9d640 1 -- 172.21.15.121:0/720812916 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3db0005290 msgr2=0x7f3db000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:24.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.044+0000 7f3dc2d9d640 1 --2- 172.21.15.121:0/720812916 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3db0005290 0x7f3db000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:24.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.044+0000 7f3dc2d9d640 10 monclient: _finish_hunting 0 2024-09-17T18:19:24.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.044+0000 7f3dc2d9d640 1 monclient: found mon.a 2024-09-17T18:19:24.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.044+0000 7f3dc2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:24.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.044+0000 7f3dc2d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:24.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.044+0000 7f3dc2d9d640 1 -- 172.21.15.121:0/720812916 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3db0003520 con 0x7f3db00039b0 2024-09-17T18:19:24.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.044+0000 7f3dc2d9d640 10 monclient: _finish_auth 0 2024-09-17T18:19:24.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.044+0000 7f3dc2d9d640 10 monclient: _check_auth_tickets 2024-09-17T18:19:24.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.044+0000 7f3dc2d9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:24.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.047+0000 7f3dc2d9d640 1 -- 172.21.15.121:0/720812916 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3db00039b0 msgr2=0x7f3db0003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:24.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.047+0000 7f3dc2d9d640 1 --2- 172.21.15.121:0/720812916 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3db00039b0 0x7f3db0003db0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3db4003d00 tx=0x7f3db0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eb72ba29c486ddf server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:24.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.047+0000 7f3dc1d9b640 1 -- 172.21.15.121:0/720812916 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db400a8f0 con 0x7f3db00039b0 2024-09-17T18:19:24.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.047+0000 7f3dc1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:24.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.047+0000 7f3dc1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:19:24.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.047+0000 7f3dc1d9b640 10 monclient: dump: 2024-09-17T18:19:24.058 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:24.059 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:24.059 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:24.059 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:24.059 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:24.059 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:24.059 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:24.059 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:24.059 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:24.059 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:24.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.047+0000 7f3dc1d9b640 1 -- 172.21.15.121:0/720812916 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3db40062d0 con 0x7f3db00039b0 2024-09-17T18:19:24.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.047+0000 7f3dc9117640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:19:24.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.047+0000 7f3dc1d9b640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:24.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.047+0000 7f3dc1d9b640 1 -- 172.21.15.121:0/720812916 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db40066c0 con 0x7f3db00039b0 2024-09-17T18:19:24.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.047+0000 7f3dc9117640 10 monclient: get_monmap_and_config success 2024-09-17T18:19:24.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.047+0000 7f3dc1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:24.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.047+0000 7f3dc9117640 4 set_mon_vals no callback set 2024-09-17T18:19:24.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.047+0000 7f3dc1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:19:24.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.047+0000 7f3dc1d9b640 10 monclient: dump: 2024-09-17T18:19:24.060 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:24.060 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:24.060 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:24.060 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:24.060 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:24.060 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:24.060 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:24.060 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:24.060 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:24.060 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:24.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.048+0000 7f3dc9117640 10 monclient: shutdown 2024-09-17T18:19:24.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.048+0000 7f3dc9117640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:24.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.048+0000 7f3dc9117640 1 -- 172.21.15.121:0/720812916 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3db00039b0 msgr2=0x7f3db0003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:24.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.048+0000 7f3dc9117640 1 --2- 172.21.15.121:0/720812916 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3db00039b0 0x7f3db0003db0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3db4003d00 tx=0x7f3db0005830 comp rx=0 tx=0).stop 2024-09-17T18:19:24.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.048+0000 7f3dc9117640 1 -- 172.21.15.121:0/720812916 shutdown_connections 2024-09-17T18:19:24.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.048+0000 7f3dc9117640 1 --2- 172.21.15.121:0/720812916 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3db0005290 0x7f3db000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:24.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.048+0000 7f3dc9117640 1 --2- 172.21.15.121:0/720812916 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3db0004800 0x7f3db0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:24.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.048+0000 7f3dc9117640 1 --2- 172.21.15.121:0/720812916 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3db00039b0 0x7f3db0003db0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:24.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.048+0000 7f3dc9117640 1 -- 172.21.15.121:0/720812916 >> 172.21.15.121:0/720812916 conn(0x7f3dc40a3b50 msgr2=0x7f3dc40a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:24.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.048+0000 7f3dc9117640 1 -- 172.21.15.121:0/720812916 shutdown_connections 2024-09-17T18:19:24.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.048+0000 7f3dc9117640 1 -- 172.21.15.121:0/720812916 wait complete. 2024-09-17T18:19:24.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.049+0000 7f3dc9117640 10 monclient: build_initial_monmap 2024-09-17T18:19:24.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.049+0000 7f3dc9117640 10 monclient: monmap: 2024-09-17T18:19:24.061 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:24.061 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:19:24.061 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:19:24.062 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:19:24.062 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:24.062 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:24.062 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:19:24.062 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:19:24.062 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:19:24.062 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:24.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.049+0000 7f3dc9117640 1 librados: starting msgr at 2024-09-17T18:19:24.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.049+0000 7f3dc9117640 1 librados: starting objecter 2024-09-17T18:19:24.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.050+0000 7f3dc9117640 1 Processor -- start 2024-09-17T18:19:24.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.050+0000 7f3dc9117640 1 -- start start 2024-09-17T18:19:24.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.050+0000 7f3dc9117640 1 librados: setting wanted keys 2024-09-17T18:19:24.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.050+0000 7f3dc9117640 1 librados: calling monclient init 2024-09-17T18:19:24.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.050+0000 7f3dc9117640 10 monclient: init 2024-09-17T18:19:24.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.051+0000 7f3dc9117640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:24.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.051+0000 7f3dc9117640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:19:24.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.051+0000 7f3dc9117640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3db00039b0 0x7f3dc41cb210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:24.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.051+0000 7f3dc9117640 10 monclient(hunting): picked mon.noname-a con 0x7f3db00039b0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:24.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.051+0000 7f3dc9117640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3db0004800 0x7f3dc412f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:24.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.051+0000 7f3dc9117640 10 monclient(hunting): picked mon.noname-b con 0x7f3db0004800 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:24.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.051+0000 7f3dc9117640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3db0005290 0x7f3dc412ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:24.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.051+0000 7f3dc9117640 10 monclient(hunting): picked mon.noname-c con 0x7f3db0005290 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:24.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.051+0000 7f3dc9117640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:24.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.051+0000 7f3dc9117640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc409dce0 con 0x7f3db00039b0 2024-09-17T18:19:24.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.051+0000 7f3dc9117640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:24.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.051+0000 7f3dc9117640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc4137020 con 0x7f3db0004800 2024-09-17T18:19:24.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.051+0000 7f3dc9117640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:24.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.051+0000 7f3dc9117640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3dc409de60 con 0x7f3db0005290 2024-09-17T18:19:24.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.051+0000 7f3dc9117640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:24.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.051+0000 7f3dc9117640 10 monclient(hunting): authenticate will time out at 767.767436s 2024-09-17T18:19:24.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.051+0000 7f3dc359e640 1 -- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3db0005290 msgr2=0x7f3dc412ff10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:19:24.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.051+0000 7f3dc359e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3db0005290 0x7f3dc412ff10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:19:24.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.051+0000 7f3dc259c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3db0004800 0x7f3dc412f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:24.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.051+0000 7f3dc259c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3db0004800 0x7f3dc412f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.121:52650/0 (socket says 172.21.15.121:52650) 2024-09-17T18:19:24.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.051+0000 7f3dc259c640 1 -- 172.21.15.121:0/4039236579 learned_addr learned my addr 172.21.15.121:0/4039236579 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:24.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.051+0000 7f3dc259c640 10 monclient(hunting): get_auth_request con 0x7f3db0004800 auth_method 0 2024-09-17T18:19:24.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.051+0000 7f3dc259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:24.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.051+0000 7f3dc259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:24.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.051+0000 7f3dc259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:24.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.051+0000 7f3dc259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:24.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.051+0000 7f3dc259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:24.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.051+0000 7f3dc259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:24.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.051+0000 7f3dc259c640 10 monclient(hunting): handle_auth_done global_id 4381 payload 824 2024-09-17T18:19:24.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.051+0000 7f3dc259c640 1 -- 172.21.15.121:0/4039236579 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3db0005290 msgr2=0x7f3dc412ff10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:19:24.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.051+0000 7f3dc259c640 1 --2- 172.21.15.121:0/4039236579 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3db0005290 0x7f3dc412ff10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:24.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.051+0000 7f3dc259c640 1 -- 172.21.15.121:0/4039236579 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3db00039b0 msgr2=0x7f3dc41cb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:24.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.051+0000 7f3dc259c640 1 --2- 172.21.15.121:0/4039236579 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3db00039b0 0x7f3dc41cb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:24.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.051+0000 7f3dc259c640 10 monclient: _finish_hunting 0 2024-09-17T18:19:24.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.051+0000 7f3dc259c640 1 monclient: found mon.noname-b 2024-09-17T18:19:24.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.051+0000 7f3dc259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:24.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.051+0000 7f3dc259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.173:3300/0 2024-09-17T18:19:24.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.051+0000 7f3dc259c640 1 -- 172.21.15.121:0/4039236579 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dc41cb750 con 0x7f3db0004800 2024-09-17T18:19:24.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.054+0000 7f3dc259c640 1 -- 172.21.15.121:0/4039236579 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3db0004800 msgr2=0x7f3dc412f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:24.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.054+0000 7f3dc259c640 1 --2- 172.21.15.121:0/4039236579 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3db0004800 0x7f3dc412f9d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3dac0169c0 tx=0x7f3dac016e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:24.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.054+0000 7f3dbb7fe640 10 client.?.objecter ms_handle_connect 0x7f3db0004800 2024-09-17T18:19:24.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.054+0000 7f3dbb7fe640 10 client.?.objecter resend_mon_ops 2024-09-17T18:19:24.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.055+0000 7f3dbb7fe640 1 -- 172.21.15.121:0/4039236579 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dac027070 con 0x7f3db0004800 2024-09-17T18:19:24.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.055+0000 7f3dbb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:24.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.055+0000 7f3dbb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:19:24.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.055+0000 7f3dbb7fe640 10 monclient: dump: 2024-09-17T18:19:24.069 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:24.069 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:24.069 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:24.069 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:24.069 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:24.070 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:24.070 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:24.070 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:24.070 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:24.070 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:24.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.055+0000 7f3dbb7fe640 10 monclient: _finish_auth 0 2024-09-17T18:19:24.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.055+0000 7f3dbb7fe640 10 monclient: _check_auth_tickets 2024-09-17T18:19:24.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.055+0000 7f3dbb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:24.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.056+0000 7f3dc9117640 5 monclient: authenticate success, global_id 4381 2024-09-17T18:19:24.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.056+0000 7f3dbb7fe640 1 -- 172.21.15.121:0/4039236579 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dac015d60 con 0x7f3db0004800 2024-09-17T18:19:24.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.056+0000 7f3dbb7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:24.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.056+0000 7f3dbb7fe640 1 -- 172.21.15.121:0/4039236579 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dac00f040 con 0x7f3db0004800 2024-09-17T18:19:24.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.056+0000 7f3dc9117640 10 monclient: _renew_subs 2024-09-17T18:19:24.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.056+0000 7f3dc9117640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:19:24.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.056+0000 7f3dc9117640 1 -- 172.21.15.121:0/4039236579 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dc4130450 con 0x7f3db0004800 2024-09-17T18:19:24.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.056+0000 7f3dbb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:24.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.056+0000 7f3dbb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:19:24.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.056+0000 7f3dbb7fe640 10 monclient: dump: 2024-09-17T18:19:24.071 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:24.071 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:24.071 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:24.071 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:24.071 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:24.071 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:24.071 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:24.071 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:24.071 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:24.071 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:24.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.056+0000 7f3dc159a640 4 set_mon_vals no callback set 2024-09-17T18:19:24.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.056+0000 7f3dc9117640 10 client.4381.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:24.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.056+0000 7f3dc9117640 10 monclient: _renew_subs 2024-09-17T18:19:24.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.056+0000 7f3dc9117640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:19:24.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.056+0000 7f3dc9117640 1 -- 172.21.15.121:0/4039236579 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dc4120e10 con 0x7f3db0004800 2024-09-17T18:19:24.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.056+0000 7f3dc9117640 1 librados: init done 2024-09-17T18:19:24.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.056+0000 7f3dbb7fe640 1 -- 172.21.15.121:0/4039236579 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f3dac02a070 con 0x7f3db0004800 2024-09-17T18:19:24.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.057+0000 7f3dbb7fe640 1 --2- 172.21.15.121:0/4039236579 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3da4040cd0 0x7f3da4043190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:24.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.057+0000 7f3dbb7fe640 1 -- 172.21.15.121:0/4039236579 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 3534+0+0 (secure 0 0 0) 0x7f3dac054100 con 0x7f3db0004800 2024-09-17T18:19:24.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.057+0000 7f3dbb7fe640 10 client.4381.objecter ms_dispatch 0x7f3dc4000b90 osd_map(17..17 src has 1..17) 2024-09-17T18:19:24.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.057+0000 7f3dbb7fe640 3 client.4381.objecter handle_osd_map got epochs [17,17] > 0 2024-09-17T18:19:24.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.057+0000 7f3dbb7fe640 3 client.4381.objecter handle_osd_map decoding full epoch 17 2024-09-17T18:19:24.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.057+0000 7f3dc9117640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:24.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.057+0000 7f3dc9117640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:24.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.057+0000 7f3dc9117640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:19:24.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.057+0000 7f3dbb7fe640 20 client.4381.objecter dump_active .. 0 homeless 2024-09-17T18:19:24.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.057+0000 7f3dc2d9d640 1 --2- 172.21.15.121:0/4039236579 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3da4040cd0 0x7f3da4043190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:24.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.057+0000 7f3dc9117640 1 -- 172.21.15.121:0/4039236579 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dc4132c50 con 0x7f3db0004800 2024-09-17T18:19:24.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.057+0000 7f3dc2d9d640 10 monclient: get_auth_request con 0x7f3da4040cd0 auth_method 0 2024-09-17T18:19:24.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.060+0000 7f3dbb7fe640 10 client.4381.objecter ms_handle_connect 0x7f3da4040cd0 2024-09-17T18:19:24.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.060+0000 7f3dc2d9d640 1 --2- 172.21.15.121:0/4039236579 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3da4040cd0 0x7f3da4043190 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3db4033000 tx=0x7f3db4006000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:24.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.062+0000 7f3dbb7fe640 1 -- 172.21.15.121:0/4039236579 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3dac014d60 con 0x7f3db0004800 2024-09-17T18:19:24.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.062+0000 7f3dbb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:24.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.062+0000 7f3dbb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:19:24.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.309+0000 7f3dc9117640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:19:24.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.309+0000 7f3dc9117640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:19:24.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.309+0000 7f3dc9117640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:19:24.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.309+0000 7f3dc9117640 1 -- 172.21.15.121:0/4039236579 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3dc41365e0 con 0x7f3db0004800 2024-09-17T18:19:24.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.309+0000 7f3dbb7fe640 1 -- 172.21.15.121:0/4039236579 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f3dac017ce0 con 0x7f3db0004800 2024-09-17T18:19:24.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.309+0000 7f3dbb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:19:24.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.309+0000 7f3dbb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:19:24.313 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":8,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:19:24.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.311+0000 7f3dc9117640 10 librados: watch_flush enter 2024-09-17T18:19:24.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.312+0000 7f3dc9117640 10 librados: watch_flush exit 2024-09-17T18:19:24.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.312+0000 7f3dc9117640 20 client.4381.objecter shutdown clearing up homeless session... 2024-09-17T18:19:24.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.312+0000 7f3dc9117640 10 client.4381.objecter successfully canceled tick 2024-09-17T18:19:24.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.312+0000 7f3dc9117640 1 -- 172.21.15.121:0/4039236579 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3da4040cd0 msgr2=0x7f3da4043190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:24.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.312+0000 7f3dc9117640 1 --2- 172.21.15.121:0/4039236579 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3da4040cd0 0x7f3da4043190 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3db4033000 tx=0x7f3db4006000 comp rx=0 tx=0).stop 2024-09-17T18:19:24.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.312+0000 7f3dc9117640 10 monclient: shutdown 2024-09-17T18:19:24.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.312+0000 7f3dc9117640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:24.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.312+0000 7f3dc9117640 1 -- 172.21.15.121:0/4039236579 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3db0004800 msgr2=0x7f3dc412f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:24.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.312+0000 7f3dc9117640 1 --2- 172.21.15.121:0/4039236579 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3db0004800 0x7f3dc412f9d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3dac0169c0 tx=0x7f3dac016e80 comp rx=0 tx=0).stop 2024-09-17T18:19:24.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.312+0000 7f3dc9117640 1 -- 172.21.15.121:0/4039236579 shutdown_connections 2024-09-17T18:19:24.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.312+0000 7f3dc9117640 1 --2- 172.21.15.121:0/4039236579 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3da4040cd0 0x7f3da4043190 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:24.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.312+0000 7f3dc9117640 1 --2- 172.21.15.121:0/4039236579 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3db0004800 0x7f3dc412f9d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:24.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.312+0000 7f3dc9117640 1 --2- 172.21.15.121:0/4039236579 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3db0005290 0x7f3dc412ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:24.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.312+0000 7f3dc9117640 1 --2- 172.21.15.121:0/4039236579 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3db00039b0 0x7f3dc41cb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:24.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.312+0000 7f3dc9117640 1 -- 172.21.15.121:0/4039236579 >> 172.21.15.121:0/4039236579 conn(0x7f3dc40a3b50 msgr2=0x7f3dc40a5210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:24.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.312+0000 7f3dc9117640 1 -- 172.21.15.121:0/4039236579 shutdown_connections 2024-09-17T18:19:24.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.312+0000 7f3dc9117640 1 -- 172.21.15.121:0/4039236579 wait complete. 2024-09-17T18:19:24.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.312+0000 7f3dc9117640 1 librados: shutdown 2024-09-17T18:19:24.325 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-17T18:19:24.326 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-17T18:19:24.326 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T18:19:24.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.422+0000 7f54d30c6640 10 monclient: get_monmap_and_config 2024-09-17T18:19:24.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.422+0000 7f54d30c6640 10 monclient: build_initial_monmap 2024-09-17T18:19:24.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.422+0000 7f54d30c6640 10 monclient: monmap: 2024-09-17T18:19:24.424 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:24.424 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:24.424 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:19:24.423142+0000 2024-09-17T18:19:24.424 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:19:24.423142+0000 2024-09-17T18:19:24.425 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:24.425 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:24.425 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:19:24.425 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:19:24.425 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:19:24.425 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:19:24.425 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:19:24.425 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:24.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.423+0000 7f54d30c6640 1 Processor -- start 2024-09-17T18:19:24.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.423+0000 7f54d30c6640 1 -- start start 2024-09-17T18:19:24.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.423+0000 7f54d30c6640 10 monclient: init 2024-09-17T18:19:24.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.423+0000 7f54d30c6640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:24.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.423+0000 7f54d30c6640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-17T18:19:24.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.423+0000 7f54d30c6640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f54cc09e550 0x7f54cc09e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:24.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.423+0000 7f54d30c6640 10 monclient(hunting): picked mon.noname-b con 0x7f54cc09e550 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:24.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.423+0000 7f54d30c6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54cc09ee70 addr v1:172.21.15.121:6789/0 2024-09-17T18:19:24.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.423+0000 7f54d30c6640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f54cc09f290 0x7f54cc13cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:24.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.423+0000 7f54d30c6640 10 monclient(hunting): picked mon.noname-a con 0x7f54cc09f290 addr v2:172.21.15.121:3300/0 2024-09-17T18:19:24.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.423+0000 7f54d30c6640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54cc0a4460 con 0x7f54cc09ee70 2024-09-17T18:19:24.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.423+0000 7f54d30c6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:24.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.423+0000 7f54d30c6640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f54cc0a4950 con 0x7f54cc09f290 2024-09-17T18:19:24.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.423+0000 7f54d30c6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:24.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.423+0000 7f54d30c6640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f54cc0a2b60 con 0x7f54cc09e550 2024-09-17T18:19:24.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.423+0000 7f54d30c6640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:24.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.423+0000 7f54d30c6640 10 monclient(hunting): authenticate will time out at 768.139853s 2024-09-17T18:19:24.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.423+0000 7f54d0e62640 1 -- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f54cc09e550 msgr2=0x7f54cc09e930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:19:24.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.423+0000 7f54d0e62640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f54cc09e550 0x7f54cc09e930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:19:24.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.423+0000 7f54cbfff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f54cc09ee70 0x7f54cc139840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:58292/0 (socket says 172.21.15.121:58292) 2024-09-17T18:19:24.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.423+0000 7f54cbfff640 1 -- 172.21.15.121:0/2443241364 learned_addr learned my addr 172.21.15.121:0/2443241364 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:19:24.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.423+0000 7f54d1663640 1 --2- 172.21.15.121:0/2443241364 >> v2:172.21.15.121:3300/0 conn(0x7f54cc09f290 0x7f54cc13cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:24.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.424+0000 7f54d1663640 10 monclient(hunting): get_auth_request con 0x7f54cc09f290 auth_method 0 2024-09-17T18:19:24.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.424+0000 7f54d1663640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:24.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.424+0000 7f54d1663640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:24.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.424+0000 7f54d1663640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:24.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.424+0000 7f54d1663640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:24.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.424+0000 7f54d1663640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:24.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.424+0000 7f54d1663640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:24.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.424+0000 7f54d1663640 10 monclient(hunting): handle_auth_done global_id 4386 payload 274 2024-09-17T18:19:24.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.424+0000 7f54d1663640 1 -- 172.21.15.121:0/2443241364 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f54cc09e550 msgr2=0x7f54cc09e930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:19:24.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.424+0000 7f54d1663640 1 --2- 172.21.15.121:0/2443241364 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f54cc09e550 0x7f54cc09e930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:24.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.424+0000 7f54d1663640 1 -- 172.21.15.121:0/2443241364 >> v1:172.21.15.121:6789/0 conn(0x7f54cc09ee70 legacy=0x7f54cc139840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:19:24.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.424+0000 7f54d1663640 10 monclient: _finish_hunting 0 2024-09-17T18:19:24.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.424+0000 7f54d1663640 1 monclient: found mon.noname-a 2024-09-17T18:19:24.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.424+0000 7f54d1663640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:24.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.424+0000 7f54d1663640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:19:24.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.424+0000 7f54d1663640 1 -- 172.21.15.121:0/2443241364 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54cc09f840 con 0x7f54cc09f290 2024-09-17T18:19:24.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.427+0000 7f54d1663640 1 -- 172.21.15.121:0/2443241364 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54cc09f290 msgr2=0x7f54cc13cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:24.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.427+0000 7f54d1663640 1 --2- 172.21.15.121:0/2443241364 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54cc09f290 0x7f54cc13cf70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f54cc09fbb0 tx=0x7f54bc029fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=929443d5bd947fc2 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:24.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.427+0000 7f54cb7fe640 1 -- 172.21.15.121:0/2443241364 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54bc037070 con 0x7f54cc09f290 2024-09-17T18:19:24.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.427+0000 7f54cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:24.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.427+0000 7f54cb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:19:24.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.427+0000 7f54cb7fe640 10 monclient: dump: 2024-09-17T18:19:24.439 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:24.439 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:24.439 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:24.439 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:24.439 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:24.439 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:24.439 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:24.439 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:24.439 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:24.439 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:24.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.427+0000 7f54cb7fe640 10 monclient: _finish_auth 0 2024-09-17T18:19:24.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.427+0000 7f54cb7fe640 10 monclient: _check_auth_tickets 2024-09-17T18:19:24.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.427+0000 7f54cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:24.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.427+0000 7f54d30c6640 5 monclient: authenticate success, global_id 4386 2024-09-17T18:19:24.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.427+0000 7f54cb7fe640 1 -- 172.21.15.121:0/2443241364 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54bc032440 con 0x7f54cc09f290 2024-09-17T18:19:24.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.427+0000 7f54cb7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:24.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.427+0000 7f54d30c6640 10 monclient: get_monmap_and_config success 2024-09-17T18:19:24.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.427+0000 7f54cb7fe640 1 -- 172.21.15.121:0/2443241364 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54bc032810 con 0x7f54cc09f290 2024-09-17T18:19:24.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.427+0000 7f54d30c6640 4 set_mon_vals no callback set 2024-09-17T18:19:24.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.427+0000 7f54cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:24.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.427+0000 7f54cb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:19:24.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.427+0000 7f54cb7fe640 10 monclient: dump: 2024-09-17T18:19:24.441 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:24.441 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:24.441 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:24.441 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:24.441 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:24.441 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:24.441 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:24.441 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:24.441 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:24.441 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:24.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.427+0000 7f54d30c6640 10 monclient: shutdown 2024-09-17T18:19:24.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.427+0000 7f54d30c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:24.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.427+0000 7f54d30c6640 1 -- 172.21.15.121:0/2443241364 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54cc09f290 msgr2=0x7f54cc13cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:24.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.427+0000 7f54d30c6640 1 --2- 172.21.15.121:0/2443241364 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54cc09f290 0x7f54cc13cf70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f54cc09fbb0 tx=0x7f54bc029fb0 comp rx=0 tx=0).stop 2024-09-17T18:19:24.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.427+0000 7f54d30c6640 1 -- 172.21.15.121:0/2443241364 shutdown_connections 2024-09-17T18:19:24.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.427+0000 7f54d30c6640 1 --2- 172.21.15.121:0/2443241364 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54cc09f290 0x7f54cc13cf70 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:24.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.427+0000 7f54d30c6640 1 --2- 172.21.15.121:0/2443241364 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f54cc09e550 0x7f54cc09e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:24.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.427+0000 7f54d30c6640 1 -- 172.21.15.121:0/2443241364 >> 172.21.15.121:0/2443241364 conn(0x7f54cc0a7830 msgr2=0x7f54cc0a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:24.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.428+0000 7f54d30c6640 1 -- 172.21.15.121:0/2443241364 shutdown_connections 2024-09-17T18:19:24.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.428+0000 7f54d30c6640 1 -- 172.21.15.121:0/2443241364 wait complete. 2024-09-17T18:19:24.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.428+0000 7f54d30c6640 10 monclient: build_initial_monmap 2024-09-17T18:19:24.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.428+0000 7f54d30c6640 10 monclient: monmap: 2024-09-17T18:19:24.442 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:24.442 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:19:24.442 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:19:24.443 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:19:24.443 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:24.443 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:24.443 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:19:24.443 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:19:24.443 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:19:24.443 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:24.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.428+0000 7f54d30c6640 1 librados: starting msgr at 2024-09-17T18:19:24.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.428+0000 7f54d30c6640 1 librados: starting objecter 2024-09-17T18:19:24.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.428+0000 7f54d30c6640 1 Processor -- start 2024-09-17T18:19:24.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.428+0000 7f54d30c6640 1 -- start start 2024-09-17T18:19:24.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.428+0000 7f54d30c6640 1 librados: setting wanted keys 2024-09-17T18:19:24.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.428+0000 7f54d30c6640 1 librados: calling monclient init 2024-09-17T18:19:24.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.428+0000 7f54d30c6640 10 monclient: init 2024-09-17T18:19:24.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54d30c6640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:24.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54d30c6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:19:24.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54d30c6640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54cc09e550 0x7f54cc133f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:24.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54d30c6640 10 monclient(hunting): picked mon.noname-a con 0x7f54cc09e550 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:24.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54d30c6640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f54cc09ee70 0x7f54cc134450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:24.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54d30c6640 10 monclient(hunting): picked mon.noname-b con 0x7f54cc09ee70 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:24.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54d30c6640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f54cc09f290 0x7f54cc13ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:24.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54d30c6640 10 monclient(hunting): picked mon.noname-c con 0x7f54cc09f290 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:24.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54d30c6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:24.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54d30c6640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f54cc09de60 con 0x7f54cc09e550 2024-09-17T18:19:24.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54d30c6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:24.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54d30c6640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f54cc09dce0 con 0x7f54cc09ee70 2024-09-17T18:19:24.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54d30c6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:24.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54d30c6640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f54cc0747e0 con 0x7f54cc09f290 2024-09-17T18:19:24.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54d30c6640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:24.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54d30c6640 10 monclient(hunting): authenticate will time out at 768.145507s 2024-09-17T18:19:24.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54d1663640 1 -- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f54cc09f290 msgr2=0x7f54cc13ab80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:19:24.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54d1663640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f54cc09f290 0x7f54cc13ab80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:19:24.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54d0e62640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54cc09e550 0x7f54cc133f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:24.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54d0e62640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54cc09e550 0x7f54cc133f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38556/0 (socket says 172.21.15.121:38556) 2024-09-17T18:19:24.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54d0e62640 1 -- 172.21.15.121:0/3819592791 learned_addr learned my addr 172.21.15.121:0/3819592791 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:24.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54d0e62640 10 monclient(hunting): get_auth_request con 0x7f54cc09e550 auth_method 0 2024-09-17T18:19:24.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54d0e62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:24.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54d0e62640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:24.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54d0e62640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:24.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54d0e62640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:24.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54d0e62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:24.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54cbfff640 1 --2- 172.21.15.121:0/3819592791 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f54cc09ee70 0x7f54cc134450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:24.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54d0e62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:24.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54cbfff640 10 monclient(hunting): get_auth_request con 0x7f54cc09ee70 auth_method 0 2024-09-17T18:19:24.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:24.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:24.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:24.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:24.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:24.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:24.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54d0e62640 10 monclient(hunting): handle_auth_done global_id 4389 payload 824 2024-09-17T18:19:24.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54d0e62640 1 -- 172.21.15.121:0/3819592791 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f54cc09f290 msgr2=0x7f54cc13ab80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:19:24.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54d0e62640 1 --2- 172.21.15.121:0/3819592791 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f54cc09f290 0x7f54cc13ab80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:24.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54d0e62640 1 -- 172.21.15.121:0/3819592791 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f54cc09ee70 msgr2=0x7f54cc134450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:24.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54d0e62640 1 --2- 172.21.15.121:0/3819592791 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f54cc09ee70 0x7f54cc134450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:24.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54d0e62640 10 monclient: _finish_hunting 0 2024-09-17T18:19:24.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54d0e62640 1 monclient: found mon.noname-a 2024-09-17T18:19:24.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54d0e62640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:24.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54d0e62640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:19:24.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.429+0000 7f54d0e62640 1 -- 172.21.15.121:0/3819592791 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54cc11ecc0 con 0x7f54cc09e550 2024-09-17T18:19:24.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.432+0000 7f54d0e62640 1 -- 172.21.15.121:0/3819592791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54cc09e550 msgr2=0x7f54cc133f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:24.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.432+0000 7f54d0e62640 1 --2- 172.21.15.121:0/3819592791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54cc09e550 0x7f54cc133f10 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f54c400b580 tx=0x7f54c400ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:24.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.432+0000 7f54c97fa640 10 client.?.objecter ms_handle_connect 0x7f54cc09e550 2024-09-17T18:19:24.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.432+0000 7f54c97fa640 10 client.?.objecter resend_mon_ops 2024-09-17T18:19:24.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.432+0000 7f54c97fa640 1 -- 172.21.15.121:0/3819592791 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c40050f0 con 0x7f54cc09e550 2024-09-17T18:19:24.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.432+0000 7f54c97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:24.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.432+0000 7f54c97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:19:24.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.432+0000 7f54c97fa640 10 monclient: dump: 2024-09-17T18:19:24.448 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:24.448 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:24.448 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:24.448 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:24.448 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:24.449 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:24.449 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:24.449 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:24.449 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:24.449 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:24.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.432+0000 7f54c97fa640 10 monclient: _finish_auth 0 2024-09-17T18:19:24.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.432+0000 7f54c97fa640 10 monclient: _check_auth_tickets 2024-09-17T18:19:24.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.432+0000 7f54c97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:24.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.432+0000 7f54c97fa640 1 -- 172.21.15.121:0/3819592791 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54c4015070 con 0x7f54cc09e550 2024-09-17T18:19:24.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.432+0000 7f54d30c6640 5 monclient: authenticate success, global_id 4389 2024-09-17T18:19:24.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.432+0000 7f54c97fa640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:24.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.432+0000 7f54d30c6640 10 monclient: _renew_subs 2024-09-17T18:19:24.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.432+0000 7f54c97fa640 1 -- 172.21.15.121:0/3819592791 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c40054d0 con 0x7f54cc09e550 2024-09-17T18:19:24.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.432+0000 7f54d30c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:24.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.432+0000 7f54d30c6640 1 -- 172.21.15.121:0/3819592791 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54cc11efa0 con 0x7f54cc09e550 2024-09-17T18:19:24.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.432+0000 7f54c97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:24.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.432+0000 7f54c97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:19:24.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.432+0000 7f54caffd640 4 set_mon_vals no callback set 2024-09-17T18:19:24.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.432+0000 7f54c97fa640 10 monclient: dump: 2024-09-17T18:19:24.450 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:24.450 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:24.450 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:24.450 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:24.450 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:24.450 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:24.450 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:24.451 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:24.451 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:24.451 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:24.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.432+0000 7f54d30c6640 10 client.4389.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:24.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.432+0000 7f54d30c6640 10 monclient: _renew_subs 2024-09-17T18:19:24.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.432+0000 7f54d30c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:24.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.432+0000 7f54d30c6640 1 -- 172.21.15.121:0/3819592791 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54cc134990 con 0x7f54cc09e550 2024-09-17T18:19:24.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.432+0000 7f54d30c6640 1 librados: init done 2024-09-17T18:19:24.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.433+0000 7f54d30c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:24.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.433+0000 7f54d30c6640 1 --2- 172.21.15.121:0/3819592791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54cc13b880 0x7f54cc0a1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:24.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.433+0000 7f54d30c6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:24.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.433+0000 7f54d30c6640 1 -- 172.21.15.121:0/3819592791 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f54cc09e930 con 0x7f54cc13b880 2024-09-17T18:19:24.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.433+0000 7f54cbfff640 1 --2- 172.21.15.121:0/3819592791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54cc13b880 0x7f54cc0a1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:24.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.433+0000 7f54c97fa640 1 -- 172.21.15.121:0/3819592791 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f54c4005670 con 0x7f54cc09e550 2024-09-17T18:19:24.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.433+0000 7f54cbfff640 10 monclient: get_auth_request con 0x7f54cc13b880 auth_method 0 2024-09-17T18:19:24.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.433+0000 7f54cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:24.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.433+0000 7f54cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:24.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.433+0000 7f54cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:24.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.433+0000 7f54cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:24.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.433+0000 7f54cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:24.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.433+0000 7f54cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:24.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.433+0000 7f54c97fa640 1 --2- 172.21.15.121:0/3819592791 >> v2:172.21.15.173:6812/2423786037 conn(0x7f54ac040150 0x7f54ac042610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:24.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.433+0000 7f54cbfff640 10 monclient(hunting): handle_auth_done global_id 4392 payload 824 2024-09-17T18:19:24.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.433+0000 7f54cbfff640 1 -- 172.21.15.121:0/3819592791 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f54cc11e700 con 0x7f54cc13b880 2024-09-17T18:19:24.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.433+0000 7f54c97fa640 1 -- 172.21.15.121:0/3819592791 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 3534+0+0 (secure 0 0 0) 0x7f54c4052db0 con 0x7f54cc09e550 2024-09-17T18:19:24.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.433+0000 7f54c97fa640 10 client.4389.objecter ms_dispatch 0x7f54cc000b90 osd_map(17..17 src has 1..17) 2024-09-17T18:19:24.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.433+0000 7f54c97fa640 3 client.4389.objecter handle_osd_map got epochs [17,17] > 0 2024-09-17T18:19:24.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.433+0000 7f54c97fa640 3 client.4389.objecter handle_osd_map decoding full epoch 17 2024-09-17T18:19:24.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.433+0000 7f54c97fa640 20 client.4389.objecter dump_active .. 0 homeless 2024-09-17T18:19:24.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.433+0000 7f54d1663640 1 --2- 172.21.15.121:0/3819592791 >> v2:172.21.15.173:6812/2423786037 conn(0x7f54ac040150 0x7f54ac042610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:24.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.433+0000 7f54d1663640 10 monclient: get_auth_request con 0x7f54ac040150 auth_method 0 2024-09-17T18:19:24.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.435+0000 7f54c97fa640 10 client.4389.objecter ms_handle_connect 0x7f54cc13b880 2024-09-17T18:19:24.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.435+0000 7f54c97fa640 10 client.4389.objecter resend_mon_ops 2024-09-17T18:19:24.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.435+0000 7f54cbfff640 1 -- 172.21.15.121:0/3819592791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54cc13b880 msgr2=0x7f54cc0a1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:24.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.435+0000 7f54cbfff640 1 --2- 172.21.15.121:0/3819592791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54cc13b880 0x7f54cc0a1a50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f54b4007980 tx=0x7f54b400f5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:24.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.435+0000 7f54c97fa640 1 -- 172.21.15.121:0/3819592791 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b4017030 con 0x7f54cc13b880 2024-09-17T18:19:24.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.435+0000 7f54c97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:24.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.435+0000 7f54c97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:19:24.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.435+0000 7f54c97fa640 10 monclient: dump: 2024-09-17T18:19:24.453 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:24.453 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:24.453 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:24.454 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:24.454 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:24.454 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:24.454 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:24.454 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:24.454 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:24.454 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:24.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.435+0000 7f54c97fa640 1 -- 172.21.15.121:0/3819592791 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f54b4006020 con 0x7f54cc13b880 2024-09-17T18:19:24.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.435+0000 7f54c97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:24.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.435+0000 7f54c97fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:19:24.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.436+0000 7f54c97fa640 1 -- 172.21.15.121:0/3819592791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54cc13b880 msgr2=0x7f54cc0a1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:24.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.436+0000 7f54c97fa640 1 --2- 172.21.15.121:0/3819592791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54cc13b880 0x7f54cc0a1a50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f54b4007980 tx=0x7f54b400f5a0 comp rx=0 tx=0).stop 2024-09-17T18:19:24.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.436+0000 7f54c97fa640 1 -- 172.21.15.121:0/3819592791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54cc13b880 msgr2=0x7f54cc0a1a50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:19:24.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.436+0000 7f54c97fa640 1 --2- 172.21.15.121:0/3819592791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54cc13b880 0x7f54cc0a1a50 secure :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f54b4007980 tx=0x7f54b400f5a0 comp rx=0 tx=0).stop 2024-09-17T18:19:24.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.436+0000 7f54d1663640 1 --2- 172.21.15.121:0/3819592791 >> v2:172.21.15.173:6812/2423786037 conn(0x7f54ac040150 0x7f54ac042610 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f54cc09e1a0 tx=0x7f54bc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:24.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.436+0000 7f54c97fa640 10 client.4389.objecter ms_handle_connect 0x7f54ac040150 2024-09-17T18:19:24.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.441+0000 7f54d30c6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:19:24.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.441+0000 7f54d30c6640 1 --2- 172.21.15.121:0/3819592791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54cc13bc60 0x7f54cc0a1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:24.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.441+0000 7f54d30c6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:24.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.441+0000 7f54d30c6640 1 -- 172.21.15.121:0/3819592791 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f54cc11e940 con 0x7f54cc13bc60 2024-09-17T18:19:24.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.442+0000 7f54cbfff640 1 --2- 172.21.15.121:0/3819592791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54cc13bc60 0x7f54cc0a1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:24.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.442+0000 7f54cbfff640 10 monclient: get_auth_request con 0x7f54cc13bc60 auth_method 0 2024-09-17T18:19:24.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.442+0000 7f54cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:24.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.442+0000 7f54cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:24.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.442+0000 7f54cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:24.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.442+0000 7f54cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:24.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.442+0000 7f54cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:24.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.442+0000 7f54cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:24.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.442+0000 7f54cbfff640 10 monclient(hunting): handle_auth_done global_id 4395 payload 824 2024-09-17T18:19:24.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.442+0000 7f54cbfff640 1 -- 172.21.15.121:0/3819592791 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f54cc06cd40 con 0x7f54cc13bc60 2024-09-17T18:19:24.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.444+0000 7f54cbfff640 1 -- 172.21.15.121:0/3819592791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54cc13bc60 msgr2=0x7f54cc0a1f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:24.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.444+0000 7f54c97fa640 10 client.4389.objecter ms_handle_connect 0x7f54cc13bc60 2024-09-17T18:19:24.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.444+0000 7f54c97fa640 10 client.4389.objecter resend_mon_ops 2024-09-17T18:19:24.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.444+0000 7f54cbfff640 1 --2- 172.21.15.121:0/3819592791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54cc13bc60 0x7f54cc0a1f90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f54b4007980 tx=0x7f54b4004cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:24.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.444+0000 7f54c97fa640 1 -- 172.21.15.121:0/3819592791 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b401d240 con 0x7f54cc13bc60 2024-09-17T18:19:24.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.444+0000 7f54c97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:24.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.444+0000 7f54c97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:19:24.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.444+0000 7f54c97fa640 10 monclient: dump: 2024-09-17T18:19:24.460 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:24.460 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:24.460 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:24.461 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:24.461 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:24.461 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:24.461 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:24.461 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:24.461 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:24.461 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:24.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.444+0000 7f54c97fa640 1 -- 172.21.15.121:0/3819592791 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4709 (secure 0 0 0) 0x7f54b401d240 con 0x7f54cc13bc60 2024-09-17T18:19:24.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.444+0000 7f54c97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:19:24.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.444+0000 7f54c97fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:19:24.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.444+0000 7f54c97fa640 1 -- 172.21.15.121:0/3819592791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54cc13bc60 msgr2=0x7f54cc0a1f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:24.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.444+0000 7f54c97fa640 1 --2- 172.21.15.121:0/3819592791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54cc13bc60 0x7f54cc0a1f90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f54b4007980 tx=0x7f54b4004cc0 comp rx=0 tx=0).stop 2024-09-17T18:19:24.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.444+0000 7f54c97fa640 1 -- 172.21.15.121:0/3819592791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54cc13bc60 msgr2=0x7f54cc0a1f90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:19:24.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.444+0000 7f54c97fa640 1 --2- 172.21.15.121:0/3819592791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54cc13bc60 0x7f54cc0a1f90 secure :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f54b4007980 tx=0x7f54b4004cc0 comp rx=0 tx=0).stop 2024-09-17T18:19:24.462 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:19:24.462 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:19:24.462 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:19:24.462 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "leader", 2024-09-17T18:19:24.462 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 8, 2024-09-17T18:19:24.462 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 37429, 2024-09-17T18:19:24.462 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:19:24.462 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:19:24.462 INFO:teuthology.orchestra.run.smithi121.stdout: 1 2024-09-17T18:19:24.463 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:19:24.463 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 0, 2024-09-17T18:19:24.463 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:19:24.463 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:19:24.463 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:19:24.463 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:19:24.463 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:19:24.463 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:19:24.463 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:19:24.463 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:19:24.463 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:19:24.463 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:19:24.463 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:19:24.463 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:19:24.463 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:19:24.463 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:19:24.464 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:19:24.464 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:19:24.464 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:19:24.464 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:19:24.464 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:19:24.464 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:19:24.464 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:19:24.464 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:19:24.464 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:19:24.464 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:19:24.464 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:19:24.465 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:19:24.465 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:19:24.465 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:19:24.465 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:19:24.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:24.465 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:19:24.465 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:19:24.465 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:19:24.465 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:19:24.465 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:19:24.465 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:19:24.466 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:19:24.466 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:19:24.466 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:19:24.466 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:19:24.466 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:19:24.466 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:19:24.466 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:19:24.466 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:19:24.466 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:19:24.466 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:19:24.466 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:19:24.467 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:19:24.467 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:19:24.467 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:19:24.467 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:19:24.467 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:19:24.467 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:19:24.467 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:19:24.467 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:19:24.467 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:19:24.467 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:19:24.467 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:19:24.468 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:19:24.468 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:19:24.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:24.468 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:19:24.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:24.468 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:19:24.468 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:19:24.468 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:19:24.468 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:19:24.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:24.468 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:19:24.468 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:19:24.469 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:19:24.469 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:24.469 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:24.469 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:19:24.469 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:19:24.469 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:19:24.469 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:19:24.469 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:19:24.469 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:24.469 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:19:24.469 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:19:24.469 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:19:24.469 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:19:24.469 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:19:24.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:24.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:24.470 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:19:24.470 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:19:24.470 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:19:24.470 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:19:24.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:24.470 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:19:24.470 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:19:24.470 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:19:24.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:24.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:24.470 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:19:24.470 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:19:24.470 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:19:24.471 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:19:24.471 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:19:24.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:24.471 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:19:24.471 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:19:24.471 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:19:24.471 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:19:24.471 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:19:24.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:24.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:24.471 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:19:24.471 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:19:24.471 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:19:24.471 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:19:24.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:24.472 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:19:24.472 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:19:24.472 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:19:24.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:24.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:24.472 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:19:24.472 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:19:24.472 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:19:24.472 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:19:24.472 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:19:24.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:24.472 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:19:24.472 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:19:24.472 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:19:24.472 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:19:24.473 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:19:24.473 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:19:24.473 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:19:24.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:24.473 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:19:24.473 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:19:24.473 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:24.473 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:19:24.473 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:19:24.473 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:19:24.473 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:19:24.473 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:19:24.473 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:19:24.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:24.474 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:19:24.474 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:19:24.474 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-17T18:19:24.474 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:19:24.474 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:19:24.474 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:19:24.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:24.474 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:19:24.474 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:19:24.474 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-17T18:19:24.474 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:19:24.474 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:19:24.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:24.474 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:19:24.474 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:19:24.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.445+0000 7f54d30c6640 10 librados: watch_flush enter 2024-09-17T18:19:24.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.445+0000 7f54d30c6640 10 librados: watch_flush exit 2024-09-17T18:19:24.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.445+0000 7f54d30c6640 20 client.4389.objecter shutdown clearing up homeless session... 2024-09-17T18:19:24.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.445+0000 7f54d30c6640 10 client.4389.objecter successfully canceled tick 2024-09-17T18:19:24.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.445+0000 7f54d30c6640 1 -- 172.21.15.121:0/3819592791 >> v2:172.21.15.173:6812/2423786037 conn(0x7f54ac040150 msgr2=0x7f54ac042610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:24.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.445+0000 7f54d30c6640 1 --2- 172.21.15.121:0/3819592791 >> v2:172.21.15.173:6812/2423786037 conn(0x7f54ac040150 0x7f54ac042610 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f54cc09e1a0 tx=0x7f54bc02e000 comp rx=0 tx=0).stop 2024-09-17T18:19:24.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.445+0000 7f54d30c6640 10 monclient: shutdown 2024-09-17T18:19:24.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.445+0000 7f54d30c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:24.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.445+0000 7f54d30c6640 1 -- 172.21.15.121:0/3819592791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54cc09e550 msgr2=0x7f54cc133f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:24.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.445+0000 7f54d30c6640 1 --2- 172.21.15.121:0/3819592791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54cc09e550 0x7f54cc133f10 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f54c400b580 tx=0x7f54c400ba40 comp rx=0 tx=0).stop 2024-09-17T18:19:24.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.445+0000 7f54d1663640 1 -- 172.21.15.121:0/3819592791 reap_dead start 2024-09-17T18:19:24.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.445+0000 7f54d1663640 1 -- 172.21.15.121:0/3819592791 reap_dead start 2024-09-17T18:19:24.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.445+0000 7f54d30c6640 1 -- 172.21.15.121:0/3819592791 shutdown_connections 2024-09-17T18:19:24.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.445+0000 7f54d30c6640 1 -- 172.21.15.121:0/3819592791 >> 172.21.15.121:0/3819592791 conn(0x7f54cc0a7830 msgr2=0x7f54cc137a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:24.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.445+0000 7f54d30c6640 1 -- 172.21.15.121:0/3819592791 shutdown_connections 2024-09-17T18:19:24.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.445+0000 7f54d30c6640 1 -- 172.21.15.121:0/3819592791 wait complete. 2024-09-17T18:19:24.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.445+0000 7f54d30c6640 1 librados: shutdown 2024-09-17T18:19:24.477 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T18:19:24.478 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-17T18:19:24.478 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T18:19:24.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.573+0000 7ff679660640 10 monclient: get_monmap_and_config 2024-09-17T18:19:24.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.573+0000 7ff679660640 10 monclient: build_initial_monmap 2024-09-17T18:19:24.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.573+0000 7ff679660640 10 monclient: monmap: 2024-09-17T18:19:24.576 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:24.576 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:24.576 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:19:24.574481+0000 2024-09-17T18:19:24.576 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:19:24.574481+0000 2024-09-17T18:19:24.576 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:24.577 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:24.577 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:19:24.577 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:19:24.577 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:19:24.577 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:19:24.577 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:19:24.577 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:24.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.574+0000 7ff679660640 1 Processor -- start 2024-09-17T18:19:24.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.574+0000 7ff679660640 1 -- start start 2024-09-17T18:19:24.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.574+0000 7ff679660640 10 monclient: init 2024-09-17T18:19:24.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.574+0000 7ff679660640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:24.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.574+0000 7ff679660640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-17T18:19:24.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.574+0000 7ff679660640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7ff6740a4a10 0x7ff6740a4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:24.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.575+0000 7ff679660640 10 monclient(hunting): picked mon.noname-c con 0x7ff6740a4a10 addr v2:172.21.15.173:3300/0 2024-09-17T18:19:24.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.575+0000 7ff679660640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff67409fc90 addr v1:172.21.15.173:6789/0 2024-09-17T18:19:24.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.575+0000 7ff679660640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff6740a5330 0x7ff674136ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:24.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.575+0000 7ff679660640 10 monclient(hunting): picked mon.noname-b con 0x7ff6740a5330 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:24.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.575+0000 7ff679660640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6740a1d80 con 0x7ff67409fc90 2024-09-17T18:19:24.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.575+0000 7ff679660640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:24.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.575+0000 7ff679660640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7ff6740a2060 con 0x7ff6740a4a10 2024-09-17T18:19:24.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.575+0000 7ff679660640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:24.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.575+0000 7ff679660640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff6740a2340 con 0x7ff6740a5330 2024-09-17T18:19:24.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.575+0000 7ff679660640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:24.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.575+0000 7ff679660640 10 monclient(hunting): authenticate will time out at 768.291057s 2024-09-17T18:19:24.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.575+0000 7ff6737fe640 1 -- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff6740a5330 msgr2=0x7ff674136ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:19:24.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.575+0000 7ff6737fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff6740a5330 0x7ff674136ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:19:24.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.575+0000 7ff672ffd640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7ff6740a4a10 0x7ff6740a4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:24.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.575+0000 7ff672ffd640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7ff6740a4a10 0x7ff6740a4df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.121:52658/0 (socket says 172.21.15.121:52658) 2024-09-17T18:19:24.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.575+0000 7ff672ffd640 1 -- 172.21.15.121:0/2318952847 learned_addr learned my addr 172.21.15.121:0/2318952847 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:24.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.575+0000 7ff672ffd640 10 monclient(hunting): get_auth_request con 0x7ff6740a4a10 auth_method 0 2024-09-17T18:19:24.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.575+0000 7ff672ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:24.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.575+0000 7ff672ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:24.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.575+0000 7ff672ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:24.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.575+0000 7ff672ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:24.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.575+0000 7ff672ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:24.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.575+0000 7ff672ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:24.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.577+0000 7ff6727fc640 1 -- 172.21.15.121:0/2318952847 >> v1:172.21.15.173:6789/0 conn(0x7ff67409fc90 legacy=0x7ff6741367c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:24.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.577+0000 7ff672ffd640 10 monclient(hunting): handle_auth_done global_id 4387 payload 274 2024-09-17T18:19:24.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.577+0000 7ff672ffd640 1 -- 172.21.15.121:0/2318952847 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff6740a5330 msgr2=0x7ff674136ed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:19:24.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.577+0000 7ff672ffd640 1 --2- 172.21.15.121:0/2318952847 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff6740a5330 0x7ff674136ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:24.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.577+0000 7ff672ffd640 1 -- 172.21.15.121:0/2318952847 >> v1:172.21.15.173:6789/0 conn(0x7ff67409fc90 legacy=0x7ff6741367c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:24.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.577+0000 7ff672ffd640 10 monclient: _finish_hunting 0 2024-09-17T18:19:24.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.577+0000 7ff672ffd640 1 monclient: found mon.noname-c 2024-09-17T18:19:24.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.577+0000 7ff672ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:24.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.577+0000 7ff672ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.173:3300/0 2024-09-17T18:19:24.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.577+0000 7ff672ffd640 1 -- 172.21.15.121:0/2318952847 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6740a57e0 con 0x7ff6740a4a10 2024-09-17T18:19:24.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.580+0000 7ff672ffd640 1 -- 172.21.15.121:0/2318952847 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff6740a4a10 msgr2=0x7ff6740a4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:24.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.580+0000 7ff672ffd640 1 --2- 172.21.15.121:0/2318952847 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff6740a4a10 0x7ff6740a4df0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff66c009ed0 tx=0x7ff66c02a6d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a88ba3102ce3837 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:24.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.581+0000 7ff671ffb640 1 -- 172.21.15.121:0/2318952847 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff66c036070 con 0x7ff6740a4a10 2024-09-17T18:19:24.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.581+0000 7ff671ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:24.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.581+0000 7ff671ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:19:24.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.581+0000 7ff671ffb640 10 monclient: dump: 2024-09-17T18:19:24.587 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:24.587 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:24.587 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:24.587 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:24.587 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:24.587 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:24.587 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:24.587 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:24.587 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:24.588 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:24.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.581+0000 7ff671ffb640 10 monclient: _finish_auth 0 2024-09-17T18:19:24.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.581+0000 7ff671ffb640 10 monclient: _check_auth_tickets 2024-09-17T18:19:24.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.581+0000 7ff671ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:24.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.581+0000 7ff671ffb640 1 -- 172.21.15.121:0/2318952847 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff66c007800 con 0x7ff6740a4a10 2024-09-17T18:19:24.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.581+0000 7ff671ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:24.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.581+0000 7ff671ffb640 1 -- 172.21.15.121:0/2318952847 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff66c007bd0 con 0x7ff6740a4a10 2024-09-17T18:19:24.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.581+0000 7ff679660640 5 monclient: authenticate success, global_id 4387 2024-09-17T18:19:24.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.581+0000 7ff679660640 10 monclient: get_monmap_and_config success 2024-09-17T18:19:24.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.581+0000 7ff679660640 4 set_mon_vals no callback set 2024-09-17T18:19:24.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.581+0000 7ff671ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:24.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.581+0000 7ff671ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:19:24.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.581+0000 7ff671ffb640 10 monclient: dump: 2024-09-17T18:19:24.589 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:24.589 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:24.589 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:24.589 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:24.589 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:24.589 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:24.590 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:24.590 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:24.590 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:24.590 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:24.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.581+0000 7ff679660640 10 monclient: shutdown 2024-09-17T18:19:24.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.581+0000 7ff679660640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:24.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.581+0000 7ff679660640 1 -- 172.21.15.121:0/2318952847 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff6740a4a10 msgr2=0x7ff6740a4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:24.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.581+0000 7ff679660640 1 --2- 172.21.15.121:0/2318952847 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff6740a4a10 0x7ff6740a4df0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff66c009ed0 tx=0x7ff66c02a6d0 comp rx=0 tx=0).stop 2024-09-17T18:19:24.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.581+0000 7ff679660640 1 -- 172.21.15.121:0/2318952847 shutdown_connections 2024-09-17T18:19:24.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.581+0000 7ff679660640 1 --2- 172.21.15.121:0/2318952847 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff6740a5330 0x7ff674136ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:24.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.581+0000 7ff679660640 1 --2- 172.21.15.121:0/2318952847 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff6740a4a10 0x7ff6740a4df0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:24.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.581+0000 7ff679660640 1 -- 172.21.15.121:0/2318952847 >> 172.21.15.121:0/2318952847 conn(0x7ff67406db20 msgr2=0x7ff6740a5b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:24.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.581+0000 7ff679660640 1 -- 172.21.15.121:0/2318952847 shutdown_connections 2024-09-17T18:19:24.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.581+0000 7ff679660640 1 -- 172.21.15.121:0/2318952847 wait complete. 2024-09-17T18:19:24.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.582+0000 7ff679660640 10 monclient: build_initial_monmap 2024-09-17T18:19:24.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.582+0000 7ff679660640 10 monclient: monmap: 2024-09-17T18:19:24.591 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:19:24.591 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:19:24.591 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:19:24.591 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:19:24.591 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:19:24.591 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:24.591 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:19:24.592 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:19:24.592 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:19:24.592 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:24.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.582+0000 7ff679660640 1 librados: starting msgr at 2024-09-17T18:19:24.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.582+0000 7ff679660640 1 librados: starting objecter 2024-09-17T18:19:24.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.582+0000 7ff679660640 1 Processor -- start 2024-09-17T18:19:24.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.582+0000 7ff679660640 1 -- start start 2024-09-17T18:19:24.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.582+0000 7ff679660640 1 librados: setting wanted keys 2024-09-17T18:19:24.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.582+0000 7ff679660640 1 librados: calling monclient init 2024-09-17T18:19:24.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.582+0000 7ff679660640 10 monclient: init 2024-09-17T18:19:24.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff679660640 10 monclient: _reopen_session rank -1 2024-09-17T18:19:24.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff679660640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T18:19:24.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff679660640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff67409fc90 0x7ff6741cda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:24.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff679660640 10 monclient(hunting): picked mon.noname-b con 0x7ff67409fc90 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:19:24.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff679660640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff6740a4a10 0x7ff6741cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:24.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff679660640 10 monclient(hunting): picked mon.noname-c con 0x7ff6740a4a10 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:19:24.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff679660640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6740a5330 0x7ff67406ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:24.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff679660640 10 monclient(hunting): picked mon.noname-a con 0x7ff6740a5330 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:19:24.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff679660640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:24.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff679660640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff6740a6600 con 0x7ff6740a5330 2024-09-17T18:19:24.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff679660640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:24.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff679660640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff6740a6480 con 0x7ff67409fc90 2024-09-17T18:19:24.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff679660640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:24.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff679660640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff6740a17a0 con 0x7ff6740a4a10 2024-09-17T18:19:24.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff679660640 10 monclient(hunting): _renew_subs 2024-09-17T18:19:24.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff679660640 10 monclient(hunting): authenticate will time out at 768.299170s 2024-09-17T18:19:24.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff6727fc640 1 -- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff6740a4a10 msgr2=0x7ff6741cdfb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:19:24.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff6727fc640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff6740a4a10 0x7ff6741cdfb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:19:24.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff6737fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6740a5330 0x7ff67406ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:24.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff6737fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6740a5330 0x7ff67406ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38582/0 (socket says 172.21.15.121:38582) 2024-09-17T18:19:24.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff6737fe640 1 -- 172.21.15.121:0/2134096591 learned_addr learned my addr 172.21.15.121:0/2134096591 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:19:24.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff6737fe640 10 monclient(hunting): get_auth_request con 0x7ff6740a5330 auth_method 0 2024-09-17T18:19:24.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff6737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:24.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff6737fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:24.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff6737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:24.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff672ffd640 1 --2- 172.21.15.121:0/2134096591 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff67409fc90 0x7ff6741cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:24.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff672ffd640 10 monclient(hunting): get_auth_request con 0x7ff67409fc90 auth_method 0 2024-09-17T18:19:24.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff672ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:24.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff672ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:24.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff672ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:24.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff6737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:24.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff6737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:24.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff6737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:24.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff672ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:24.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff672ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:24.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff672ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:24.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff6737fe640 10 monclient(hunting): handle_auth_done global_id 4398 payload 824 2024-09-17T18:19:24.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff6737fe640 1 -- 172.21.15.121:0/2134096591 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff6740a4a10 msgr2=0x7ff6741cdfb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:19:24.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff6737fe640 1 --2- 172.21.15.121:0/2134096591 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff6740a4a10 0x7ff6741cdfb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:24.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff6737fe640 1 -- 172.21.15.121:0/2134096591 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff67409fc90 msgr2=0x7ff6741cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:24.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff6737fe640 1 --2- 172.21.15.121:0/2134096591 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff67409fc90 0x7ff6741cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:24.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff6737fe640 10 monclient: _finish_hunting 0 2024-09-17T18:19:24.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff6737fe640 1 monclient: found mon.noname-a 2024-09-17T18:19:24.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff6737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:19:24.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff6737fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:19:24.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.583+0000 7ff6737fe640 1 -- 172.21.15.121:0/2134096591 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff674129d30 con 0x7ff6740a5330 2024-09-17T18:19:24.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.585+0000 7ff6737fe640 1 -- 172.21.15.121:0/2134096591 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6740a5330 msgr2=0x7ff67406ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:24.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.585+0000 7ff6737fe640 1 --2- 172.21.15.121:0/2134096591 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6740a5330 0x7ff67406ec30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff66400b410 tx=0x7ff66400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:24.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.585+0000 7ff66b7fe640 10 client.?.objecter ms_handle_connect 0x7ff6740a5330 2024-09-17T18:19:24.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.585+0000 7ff66b7fe640 10 client.?.objecter resend_mon_ops 2024-09-17T18:19:24.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.585+0000 7ff66b7fe640 1 -- 172.21.15.121:0/2134096591 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff664017070 con 0x7ff6740a5330 2024-09-17T18:19:24.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.585+0000 7ff66b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:24.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.586+0000 7ff66b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:19:24.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.586+0000 7ff66b7fe640 10 monclient: dump: 2024-09-17T18:19:24.603 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:24.603 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:24.603 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:24.603 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:24.603 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:24.603 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:24.603 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:24.603 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:24.603 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:24.603 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:24.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.586+0000 7ff66b7fe640 10 monclient: _finish_auth 0 2024-09-17T18:19:24.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.586+0000 7ff66b7fe640 10 monclient: _check_auth_tickets 2024-09-17T18:19:24.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.586+0000 7ff66b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:19:24.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.586+0000 7ff66b7fe640 1 -- 172.21.15.121:0/2134096591 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6640060d0 con 0x7ff6740a5330 2024-09-17T18:19:24.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.586+0000 7ff679660640 5 monclient: authenticate success, global_id 4398 2024-09-17T18:19:24.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.586+0000 7ff679660640 10 monclient: _renew_subs 2024-09-17T18:19:24.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.586+0000 7ff679660640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:24.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.586+0000 7ff679660640 1 -- 172.21.15.121:0/2134096591 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff67412a010 con 0x7ff6740a5330 2024-09-17T18:19:24.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.586+0000 7ff66b7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:19:24.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.586+0000 7ff66b7fe640 1 -- 172.21.15.121:0/2134096591 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6640064c0 con 0x7ff6740a5330 2024-09-17T18:19:24.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.586+0000 7ff66b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:24.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.586+0000 7ff66b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:19:24.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.586+0000 7ff66b7fe640 10 monclient: dump: 2024-09-17T18:19:24.604 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:24.604 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:24.604 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:24.604 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:24.605 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:24.605 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:24.605 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:24.605 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:24.605 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:24.605 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:24.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.586+0000 7ff6717fa640 4 set_mon_vals no callback set 2024-09-17T18:19:24.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.586+0000 7ff679660640 10 client.4398.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:19:24.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.586+0000 7ff679660640 10 monclient: _renew_subs 2024-09-17T18:19:24.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.586+0000 7ff679660640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:19:24.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.586+0000 7ff679660640 1 -- 172.21.15.121:0/2134096591 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6740a01f0 con 0x7ff6740a5330 2024-09-17T18:19:24.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.586+0000 7ff679660640 1 librados: init done 2024-09-17T18:19:24.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.586+0000 7ff679660640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:24.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.586+0000 7ff679660640 1 --2- 172.21.15.121:0/2134096591 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff67406f930 0x7ff67402d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:24.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.586+0000 7ff679660640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:24.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.586+0000 7ff679660640 1 -- 172.21.15.121:0/2134096591 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff6740a4df0 con 0x7ff67406f930 2024-09-17T18:19:24.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.586+0000 7ff66b7fe640 1 -- 172.21.15.121:0/2134096591 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7ff66401a070 con 0x7ff6740a5330 2024-09-17T18:19:24.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.587+0000 7ff672ffd640 1 --2- 172.21.15.121:0/2134096591 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff67406f930 0x7ff67402d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:24.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.587+0000 7ff672ffd640 10 monclient: get_auth_request con 0x7ff67406f930 auth_method 0 2024-09-17T18:19:24.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.587+0000 7ff672ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:24.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.587+0000 7ff672ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:24.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.587+0000 7ff672ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:24.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.587+0000 7ff672ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:24.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.587+0000 7ff672ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:24.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.587+0000 7ff672ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:24.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.587+0000 7ff66b7fe640 1 --2- 172.21.15.121:0/2134096591 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff654040150 0x7ff654042610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:24.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.587+0000 7ff672ffd640 10 monclient(hunting): handle_auth_done global_id 4396 payload 824 2024-09-17T18:19:24.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.587+0000 7ff672ffd640 1 -- 172.21.15.121:0/2134096591 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff674129b10 con 0x7ff67406f930 2024-09-17T18:19:24.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.587+0000 7ff66b7fe640 1 -- 172.21.15.121:0/2134096591 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 3534+0+0 (secure 0 0 0) 0x7ff664052260 con 0x7ff6740a5330 2024-09-17T18:19:24.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.587+0000 7ff66b7fe640 10 client.4398.objecter ms_dispatch 0x7ff674000b90 osd_map(17..17 src has 1..17) 2024-09-17T18:19:24.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.587+0000 7ff66b7fe640 3 client.4398.objecter handle_osd_map got epochs [17,17] > 0 2024-09-17T18:19:24.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.587+0000 7ff66b7fe640 3 client.4398.objecter handle_osd_map decoding full epoch 17 2024-09-17T18:19:24.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.587+0000 7ff66b7fe640 20 client.4398.objecter dump_active .. 0 homeless 2024-09-17T18:19:24.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.587+0000 7ff6727fc640 1 --2- 172.21.15.121:0/2134096591 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff654040150 0x7ff654042610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:24.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.587+0000 7ff6727fc640 10 monclient: get_auth_request con 0x7ff654040150 auth_method 0 2024-09-17T18:19:24.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.589+0000 7ff66b7fe640 10 client.4398.objecter ms_handle_connect 0x7ff67406f930 2024-09-17T18:19:24.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.589+0000 7ff672ffd640 1 -- 172.21.15.121:0/2134096591 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff67406f930 msgr2=0x7ff67402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:24.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.589+0000 7ff66b7fe640 10 client.4398.objecter resend_mon_ops 2024-09-17T18:19:24.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.589+0000 7ff672ffd640 1 --2- 172.21.15.121:0/2134096591 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff67406f930 0x7ff67402d440 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff674075ef0 tx=0x7ff66c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:24.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.589+0000 7ff66b7fe640 1 -- 172.21.15.121:0/2134096591 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff66c043070 con 0x7ff67406f930 2024-09-17T18:19:24.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.589+0000 7ff66b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:24.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.589+0000 7ff66b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:19:24.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.589+0000 7ff66b7fe640 10 monclient: dump: 2024-09-17T18:19:24.608 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:24.608 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:24.608 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:24.608 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:24.608 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:24.608 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:24.609 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:24.609 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:24.609 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:24.609 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:24.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.589+0000 7ff66b7fe640 10 client.4398.objecter ms_handle_connect 0x7ff654040150 2024-09-17T18:19:24.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.589+0000 7ff6727fc640 1 --2- 172.21.15.121:0/2134096591 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff654040150 0x7ff654042610 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff65c0014b0 tx=0x7ff65c0079b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:24.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.589+0000 7ff66b7fe640 1 -- 172.21.15.121:0/2134096591 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff66c03da80 con 0x7ff67406f930 2024-09-17T18:19:24.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.589+0000 7ff66b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:19:24.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.589+0000 7ff66b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:19:24.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.589+0000 7ff66b7fe640 1 -- 172.21.15.121:0/2134096591 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff67406f930 msgr2=0x7ff67402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:24.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.589+0000 7ff66b7fe640 1 --2- 172.21.15.121:0/2134096591 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff67406f930 0x7ff67402d440 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff674075ef0 tx=0x7ff66c004a40 comp rx=0 tx=0).stop 2024-09-17T18:19:24.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.589+0000 7ff66b7fe640 1 -- 172.21.15.121:0/2134096591 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff67406f930 msgr2=0x7ff67402d440 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:19:24.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.589+0000 7ff66b7fe640 1 --2- 172.21.15.121:0/2134096591 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff67406f930 0x7ff67402d440 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:24.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.595+0000 7ff679660640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:19:24.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.595+0000 7ff679660640 1 --2- 172.21.15.121:0/2134096591 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff67406fd10 0x7ff67402d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:19:24.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.595+0000 7ff679660640 10 monclient(hunting): start opening mon connection 2024-09-17T18:19:24.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.595+0000 7ff679660640 1 -- 172.21.15.121:0/2134096591 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff674120d60 con 0x7ff67406fd10 2024-09-17T18:19:24.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.596+0000 7ff672ffd640 1 --2- 172.21.15.121:0/2134096591 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff67406fd10 0x7ff67402d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:19:24.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.596+0000 7ff672ffd640 10 monclient: get_auth_request con 0x7ff67406fd10 auth_method 0 2024-09-17T18:19:24.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.596+0000 7ff672ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:19:24.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.596+0000 7ff672ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:19:24.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.596+0000 7ff672ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:19:24.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.596+0000 7ff672ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:19:24.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.596+0000 7ff672ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:19:24.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.596+0000 7ff672ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:19:24.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.596+0000 7ff672ffd640 10 monclient(hunting): handle_auth_done global_id 4399 payload 824 2024-09-17T18:19:24.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.596+0000 7ff672ffd640 1 -- 172.21.15.121:0/2134096591 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff674137060 con 0x7ff67406fd10 2024-09-17T18:19:24.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.598+0000 7ff672ffd640 1 -- 172.21.15.121:0/2134096591 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff67406fd10 msgr2=0x7ff67402d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:19:24.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.598+0000 7ff66b7fe640 10 client.4398.objecter ms_handle_connect 0x7ff67406fd10 2024-09-17T18:19:24.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.598+0000 7ff672ffd640 1 --2- 172.21.15.121:0/2134096591 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff67406fd10 0x7ff67402d980 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff66c0093e0 tx=0x7ff66c02f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:19:24.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.598+0000 7ff66b7fe640 10 client.4398.objecter resend_mon_ops 2024-09-17T18:19:24.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.598+0000 7ff66b7fe640 1 -- 172.21.15.121:0/2134096591 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff66c036040 con 0x7ff67406fd10 2024-09-17T18:19:24.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.598+0000 7ff66b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:19:24.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.598+0000 7ff66b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:19:24.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.598+0000 7ff66b7fe640 10 monclient: dump: 2024-09-17T18:19:24.611 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:19:24.611 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:19:24.611 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:24.612 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:19:24.612 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:19:24.612 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:19:24.612 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:19:24.612 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:19:24.612 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:19:24.612 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:19:24.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.599+0000 7ff66b7fe640 1 -- 172.21.15.121:0/2134096591 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4874 (secure 0 0 0) 0x7ff66c004250 con 0x7ff67406fd10 2024-09-17T18:19:24.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.599+0000 7ff66b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:19:24.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.599+0000 7ff66b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:19:24.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.599+0000 7ff66b7fe640 1 -- 172.21.15.121:0/2134096591 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff67406fd10 msgr2=0x7ff67402d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:24.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.599+0000 7ff66b7fe640 1 --2- 172.21.15.121:0/2134096591 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff67406fd10 0x7ff67402d980 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff66c0093e0 tx=0x7ff66c02f000 comp rx=0 tx=0).stop 2024-09-17T18:19:24.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.599+0000 7ff66b7fe640 1 -- 172.21.15.121:0/2134096591 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff67406fd10 msgr2=0x7ff67402d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:19:24.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.599+0000 7ff66b7fe640 1 --2- 172.21.15.121:0/2134096591 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff67406fd10 0x7ff67402d980 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:24.613 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:19:24.613 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:19:24.613 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:19:24.613 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:19:24.613 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 8, 2024-09-17T18:19:24.613 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 37544, 2024-09-17T18:19:24.613 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:19:24.613 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:19:24.613 INFO:teuthology.orchestra.run.smithi121.stdout: 1 2024-09-17T18:19:24.614 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:19:24.614 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 0, 2024-09-17T18:19:24.614 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:19:24.614 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:19:24.614 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:19:24.614 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:19:24.614 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:19:24.614 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:19:24.614 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:19:24.614 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:19:24.614 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:19:24.614 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:19:24.614 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:19:24.614 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:19:24.614 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:19:24.615 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:19:24.615 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:19:24.615 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:19:24.615 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:19:24.615 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:19:24.615 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:19:24.615 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:19:24.615 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:19:24.615 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:19:24.615 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:19:24.615 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:19:24.615 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:19:24.615 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:19:24.615 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:19:24.615 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:19:24.615 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:19:24.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:24.616 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:19:24.616 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:19:24.616 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:19:24.616 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:19:24.616 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:19:24.616 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:19:24.616 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:19:24.616 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:19:24.616 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:19:24.616 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:19:24.616 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:19:24.616 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:19:24.616 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:19:24.617 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:19:24.617 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:19:24.617 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:19:24.617 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:19:24.617 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:19:24.617 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:19:24.617 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:19:24.617 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:19:24.617 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:19:24.617 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:19:24.617 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:19:24.618 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:19:24.618 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:19:24.618 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:19:24.618 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:19:24.618 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:19:24.618 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:19:24.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:24.618 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:19:24.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:24.618 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:19:24.618 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:19:24.618 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:19:24.618 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:19:24.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:24.618 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:19:24.619 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:19:24.619 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:19:24.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:24.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:24.619 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:19:24.619 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:19:24.619 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:19:24.619 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:19:24.619 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:19:24.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:24.619 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:19:24.619 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:19:24.619 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:19:24.619 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:19:24.619 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:19:24.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:24.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:24.620 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:19:24.620 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:19:24.620 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:19:24.620 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:19:24.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:24.620 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:19:24.620 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:19:24.620 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:19:24.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:24.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:24.620 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:19:24.621 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:19:24.621 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:19:24.621 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:19:24.621 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:19:24.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:24.621 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:19:24.621 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:19:24.621 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:19:24.621 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:19:24.621 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:19:24.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:24.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:24.621 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:19:24.621 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:19:24.621 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:19:24.622 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:19:24.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:24.622 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:19:24.622 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:19:24.622 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:19:24.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:24.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:24.622 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:19:24.622 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:19:24.622 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:19:24.622 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:19:24.622 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:19:24.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:24.622 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:19:24.623 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:19:24.623 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:19:24.623 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:19:24.623 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:19:24.623 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:19:24.623 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:19:24.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:24.623 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:19:24.623 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:19:24.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:24.623 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:19:24.623 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:19:24.623 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:19:24.624 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:19:24.624 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:19:24.624 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:19:24.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:24.624 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:19:24.624 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:19:24.624 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 4 2024-09-17T18:19:24.624 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:19:24.624 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:19:24.624 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:19:24.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:24.625 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:19:24.625 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:19:24.625 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 5 2024-09-17T18:19:24.625 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:19:24.625 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:19:24.625 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": [ 2024-09-17T18:19:24.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:19:24.625 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:19:24.625 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:19:24.625 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:19:24.625 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:19:24.625 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:19:24.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:19:24.625 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:19:24.625 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:19:24.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.599+0000 7ff679660640 10 librados: watch_flush enter 2024-09-17T18:19:24.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.599+0000 7ff679660640 10 librados: watch_flush exit 2024-09-17T18:19:24.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.599+0000 7ff679660640 20 client.4398.objecter shutdown clearing up homeless session... 2024-09-17T18:19:24.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.599+0000 7ff679660640 10 client.4398.objecter successfully canceled tick 2024-09-17T18:19:24.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.599+0000 7ff679660640 1 -- 172.21.15.121:0/2134096591 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff654040150 msgr2=0x7ff654042610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:24.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.599+0000 7ff679660640 1 --2- 172.21.15.121:0/2134096591 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff654040150 0x7ff654042610 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff65c0014b0 tx=0x7ff65c0079b0 comp rx=0 tx=0).stop 2024-09-17T18:19:24.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.599+0000 7ff679660640 10 monclient: shutdown 2024-09-17T18:19:24.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.599+0000 7ff679660640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:19:24.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.599+0000 7ff679660640 1 -- 172.21.15.121:0/2134096591 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6740a5330 msgr2=0x7ff67406ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:19:24.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.599+0000 7ff6737fe640 1 -- 172.21.15.121:0/2134096591 reap_dead start 2024-09-17T18:19:24.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.599+0000 7ff679660640 1 --2- 172.21.15.121:0/2134096591 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6740a5330 0x7ff67406ec30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff66400b410 tx=0x7ff66400b8d0 comp rx=0 tx=0).stop 2024-09-17T18:19:24.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.599+0000 7ff679660640 1 -- 172.21.15.121:0/2134096591 shutdown_connections 2024-09-17T18:19:24.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.599+0000 7ff679660640 1 --2- 172.21.15.121:0/2134096591 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6740a5330 0x7ff67406ec30 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:19:24.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.599+0000 7ff679660640 1 -- 172.21.15.121:0/2134096591 >> 172.21.15.121:0/2134096591 conn(0x7ff67406db20 msgr2=0x7ff6740a6f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:19:24.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.599+0000 7ff679660640 1 -- 172.21.15.121:0/2134096591 shutdown_connections 2024-09-17T18:19:24.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.599+0000 7ff679660640 1 -- 172.21.15.121:0/2134096591 wait complete. 2024-09-17T18:19:24.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:19:24.599+0000 7ff679660640 1 librados: shutdown 2024-09-17T18:19:24.628 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T18:19:24.628 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-17T18:19:59.665 INFO:tasks.workunit.client.0.smithi173.stderr:Note: switching to '1889a165b5128787ec693450a3d1c5df5d6dc203'. 2024-09-17T18:19:59.665 INFO:tasks.workunit.client.0.smithi173.stderr: 2024-09-17T18:19:59.665 INFO:tasks.workunit.client.0.smithi173.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-17T18:19:59.666 INFO:tasks.workunit.client.0.smithi173.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-17T18:19:59.666 INFO:tasks.workunit.client.0.smithi173.stderr:state without impacting any branches by switching back to a branch. 2024-09-17T18:19:59.666 INFO:tasks.workunit.client.0.smithi173.stderr: 2024-09-17T18:19:59.666 INFO:tasks.workunit.client.0.smithi173.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-17T18:19:59.666 INFO:tasks.workunit.client.0.smithi173.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-17T18:19:59.666 INFO:tasks.workunit.client.0.smithi173.stderr: 2024-09-17T18:19:59.666 INFO:tasks.workunit.client.0.smithi173.stderr: git switch -c 2024-09-17T18:19:59.666 INFO:tasks.workunit.client.0.smithi173.stderr: 2024-09-17T18:19:59.666 INFO:tasks.workunit.client.0.smithi173.stderr:Or undo this operation with: 2024-09-17T18:19:59.666 INFO:tasks.workunit.client.0.smithi173.stderr: 2024-09-17T18:19:59.666 INFO:tasks.workunit.client.0.smithi173.stderr: git switch - 2024-09-17T18:19:59.666 INFO:tasks.workunit.client.0.smithi173.stderr: 2024-09-17T18:19:59.667 INFO:tasks.workunit.client.0.smithi173.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-17T18:19:59.667 INFO:tasks.workunit.client.0.smithi173.stderr: 2024-09-17T18:19:59.667 INFO:tasks.workunit.client.0.smithi173.stderr:HEAD is now at 1889a165b51 Merge branch 'wip-matanb-cleanup-snapmap' of https://github.com/Matan-B/ceph into wip-bharath10-testing-2024-09-17-1348 2024-09-17T18:19:59.669 DEBUG:teuthology.orchestra.run.smithi173:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-17T18:19:59.737 INFO:tasks.workunit.client.0.smithi173.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-17T18:19:59.738 INFO:tasks.workunit.client.0.smithi173.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-17T18:19:59.738 INFO:tasks.workunit.client.0.smithi173.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-17T18:20:00.324 INFO:tasks.workunit.client.0.smithi173.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-17T18:20:00.390 INFO:tasks.workunit.client.0.smithi173.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-17T18:20:00.429 INFO:tasks.workunit.client.0.smithi173.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-17T18:20:00.431 INFO:tasks.workunit.client.0.smithi173.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-17T18:20:00.431 INFO:tasks.workunit.client.0.smithi173.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-17T18:20:00.469 INFO:tasks.workunit.client.0.smithi173.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-17T18:20:00.472 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T18:20:00.472 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-17T18:20:00.528 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-09-17T18:20:00.529 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-09-17T18:20:00.530 DEBUG:teuthology.orchestra.run.smithi173:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1889a165b5128787ec693450a3d1c5df5d6dc203 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-09-17T18:20:00.588 INFO:tasks.workunit.client.0.smithi173.stderr:+ parallel=1 2024-09-17T18:20:00.588 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' '' = --serial ']' 2024-09-17T18:20:00.588 INFO:tasks.workunit.client.0.smithi173.stderr:+ crimson=0 2024-09-17T18:20:00.588 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' '' = --crimson ']' 2024-09-17T18:20:00.589 INFO:tasks.workunit.client.0.smithi173.stderr:+ color= 2024-09-17T18:20:00.589 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -t 1 ']' 2024-09-17T18:20:00.589 INFO:tasks.workunit.client.0.smithi173.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-09-17T18:20:00.589 INFO:tasks.workunit.client.0.smithi173.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-09-17T18:20:00.589 INFO:tasks.workunit.client.0.smithi173.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-09-17T18:20:00.590 INFO:tasks.workunit.client.0.smithi173.stderr:+ declare -A pids 2024-09-17T18:20:00.590 INFO:tasks.workunit.client.0.smithi173.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-17T18:20:00.590 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1 -eq 1 ']' 2024-09-17T18:20:00.590 INFO:tasks.workunit.client.0.smithi173.stderr:++ printf %25s api_aio 2024-09-17T18:20:00.590 INFO:tasks.workunit.client.0.smithi173.stderr:+ r=' api_aio' 2024-09-17T18:20:00.590 INFO:tasks.workunit.client.0.smithi173.stderr:++ echo api_aio 2024-09-17T18:20:00.590 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '{print $1}' 2024-09-17T18:20:00.591 INFO:tasks.workunit.client.0.smithi173.stderr:+ ff=api_aio 2024-09-17T18:20:00.591 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=34924 2024-09-17T18:20:00.591 INFO:tasks.workunit.client.0.smithi173.stderr:+ echo 'test api_aio on pid 34924' 2024-09-17T18:20:00.592 INFO:tasks.workunit.client.0.smithi173.stdout:test api_aio on pid 34924 2024-09-17T18:20:00.595 INFO:tasks.workunit.client.0.smithi173.stderr:+ pids[$f]=34924 2024-09-17T18:20:00.595 INFO:tasks.workunit.client.0.smithi173.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-17T18:20:00.595 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1 -eq 1 ']' 2024-09-17T18:20:00.595 INFO:tasks.workunit.client.0.smithi173.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-09-17T18:20:00.595 INFO:tasks.workunit.client.0.smithi173.stderr:++ printf %25s api_aio_pp 2024-09-17T18:20:00.595 INFO:tasks.workunit.client.0.smithi173.stderr:+ r=' api_aio_pp' 2024-09-17T18:20:00.595 INFO:tasks.workunit.client.0.smithi173.stderr:++ echo api_aio_pp 2024-09-17T18:20:00.595 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '{print $1}' 2024-09-17T18:20:00.595 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -f /etc/bashrc ']' 2024-09-17T18:20:00.596 INFO:tasks.workunit.client.0.smithi173.stderr:+ . /etc/bashrc 2024-09-17T18:20:00.596 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -z '' ']' 2024-09-17T18:20:00.596 INFO:tasks.workunit.client.0.smithi173.stderr:++ BASHRCSOURCED=Y 2024-09-17T18:20:00.596 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.596 INFO:tasks.workunit.client.0.smithi173.stderr:++ shopt -q login_shell 2024-09-17T18:20:00.596 INFO:tasks.workunit.client.0.smithi173.stderr:+++ umask 2024-09-17T18:20:00.596 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 0022 -eq 0 ']' 2024-09-17T18:20:00.596 INFO:tasks.workunit.client.0.smithi173.stderr:++ SHELL=/bin/bash 2024-09-17T18:20:00.596 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.596 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-17T18:20:00.596 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.596 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-17T18:20:00.596 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-17T18:20:00.596 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.597 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-17T18:20:00.597 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.597 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-17T18:20:00.597 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:00.597 INFO:tasks.workunit.client.0.smithi173.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:00.597 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.597 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-17T18:20:00.597 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.597 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-17T18:20:00.597 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.597 INFO:tasks.workunit.client.0.smithi173.stderr:+ ff=api_aio_pp 2024-09-17T18:20:00.597 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=34931 2024-09-17T18:20:00.598 INFO:tasks.workunit.client.0.smithi173.stdout:test api_aio_pp on pid 34931 2024-09-17T18:20:00.601 INFO:tasks.workunit.client.0.smithi173.stderr:+ echo 'test api_aio_pp on pid 34931' 2024-09-17T18:20:00.601 INFO:tasks.workunit.client.0.smithi173.stderr:+ pids[$f]=34931 2024-09-17T18:20:00.601 INFO:tasks.workunit.client.0.smithi173.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-17T18:20:00.601 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1 -eq 1 ']' 2024-09-17T18:20:00.601 INFO:tasks.workunit.client.0.smithi173.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-09-17T18:20:00.601 INFO:tasks.workunit.client.0.smithi173.stderr:++ printf %25s api_io 2024-09-17T18:20:00.602 INFO:tasks.workunit.client.0.smithi173.stderr:+ r=' api_io' 2024-09-17T18:20:00.602 INFO:tasks.workunit.client.0.smithi173.stderr:++ echo api_io 2024-09-17T18:20:00.602 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '{print $1}' 2024-09-17T18:20:00.602 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -f /etc/bashrc ']' 2024-09-17T18:20:00.602 INFO:tasks.workunit.client.0.smithi173.stderr:+ . /etc/bashrc 2024-09-17T18:20:00.602 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -z '' ']' 2024-09-17T18:20:00.602 INFO:tasks.workunit.client.0.smithi173.stderr:++ BASHRCSOURCED=Y 2024-09-17T18:20:00.602 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.602 INFO:tasks.workunit.client.0.smithi173.stderr:++ shopt -q login_shell 2024-09-17T18:20:00.602 INFO:tasks.workunit.client.0.smithi173.stderr:+++ umask 2024-09-17T18:20:00.602 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 0022 -eq 0 ']' 2024-09-17T18:20:00.602 INFO:tasks.workunit.client.0.smithi173.stderr:++ SHELL=/bin/bash 2024-09-17T18:20:00.602 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'grep=grep --color=auto' 2024-09-17T18:20:00.602 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.603 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-17T18:20:00.603 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-17T18:20:00.603 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-17T18:20:00.603 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.603 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-17T18:20:00.603 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.603 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-17T18:20:00.603 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.603 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorls.sh 2024-09-17T18:20:00.603 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-17T18:20:00.603 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' '!' -t 0 ']' 2024-09-17T18:20:00.603 INFO:tasks.workunit.client.0.smithi173.stderr:+++ return 2024-09-17T18:20:00.603 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.603 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-17T18:20:00.604 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.604 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-17T18:20:00.604 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.604 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-17T18:20:00.604 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.604 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-17T18:20:00.604 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export S_COLORS=auto 2024-09-17T18:20:00.604 INFO:tasks.workunit.client.0.smithi173.stderr:+++ S_COLORS=auto 2024-09-17T18:20:00.604 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:00.604 INFO:tasks.workunit.client.0.smithi173.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:00.604 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.604 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.604 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-17T18:20:00.605 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-17T18:20:00.605 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.605 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.605 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-17T18:20:00.605 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-17T18:20:00.605 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.605 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.605 INFO:tasks.workunit.client.0.smithi173.stderr:+ ff=api_io 2024-09-17T18:20:00.605 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=34940 2024-09-17T18:20:00.605 INFO:tasks.workunit.client.0.smithi173.stdout:test api_io on pid 34940 2024-09-17T18:20:00.609 INFO:tasks.workunit.client.0.smithi173.stderr:+ echo 'test api_io on pid 34940' 2024-09-17T18:20:00.610 INFO:tasks.workunit.client.0.smithi173.stderr:+ pids[$f]=34940 2024-09-17T18:20:00.610 INFO:tasks.workunit.client.0.smithi173.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-17T18:20:00.610 INFO:tasks.workunit.client.0.smithi173.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-09-17T18:20:00.610 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1 -eq 1 ']' 2024-09-17T18:20:00.610 INFO:tasks.workunit.client.0.smithi173.stderr:++ printf %25s api_io_pp 2024-09-17T18:20:00.610 INFO:tasks.workunit.client.0.smithi173.stderr:+ r=' api_io_pp' 2024-09-17T18:20:00.610 INFO:tasks.workunit.client.0.smithi173.stderr:++ echo api_io_pp 2024-09-17T18:20:00.610 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '{print $1}' 2024-09-17T18:20:00.610 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -f /etc/bashrc ']' 2024-09-17T18:20:00.610 INFO:tasks.workunit.client.0.smithi173.stderr:+ . /etc/bashrc 2024-09-17T18:20:00.610 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'grep=grep --color=auto' 2024-09-17T18:20:00.610 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -z '' ']' 2024-09-17T18:20:00.610 INFO:tasks.workunit.client.0.smithi173.stderr:++ BASHRCSOURCED=Y 2024-09-17T18:20:00.610 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.611 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-17T18:20:00.611 INFO:tasks.workunit.client.0.smithi173.stderr:++ shopt -q login_shell 2024-09-17T18:20:00.611 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-17T18:20:00.611 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-17T18:20:00.611 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.611 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-17T18:20:00.611 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.611 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-17T18:20:00.611 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorls.sh 2024-09-17T18:20:00.611 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-17T18:20:00.611 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' '!' -t 0 ']' 2024-09-17T18:20:00.611 INFO:tasks.workunit.client.0.smithi173.stderr:+++ return 2024-09-17T18:20:00.611 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.611 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.611 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-17T18:20:00.612 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-17T18:20:00.612 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.612 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.612 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-17T18:20:00.612 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-17T18:20:00.612 INFO:tasks.workunit.client.0.smithi173.stderr:+++ umask 2024-09-17T18:20:00.612 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export S_COLORS=auto 2024-09-17T18:20:00.612 INFO:tasks.workunit.client.0.smithi173.stderr:+++ S_COLORS=auto 2024-09-17T18:20:00.612 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-17T18:20:00.612 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.612 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-17T18:20:00.612 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.612 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.612 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-17T18:20:00.613 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.613 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 0022 -eq 0 ']' 2024-09-17T18:20:00.613 INFO:tasks.workunit.client.0.smithi173.stderr:++ SHELL=/bin/bash 2024-09-17T18:20:00.613 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.613 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-17T18:20:00.613 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.613 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-17T18:20:00.613 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-17T18:20:00.613 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.613 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-17T18:20:00.613 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.613 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-17T18:20:00.613 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:00.613 INFO:tasks.workunit.client.0.smithi173.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:00.614 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.614 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-17T18:20:00.614 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.614 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-17T18:20:00.614 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.614 INFO:tasks.workunit.client.0.smithi173.stderr:+ ff=api_io_pp 2024-09-17T18:20:00.614 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=34953 2024-09-17T18:20:00.614 INFO:tasks.workunit.client.0.smithi173.stdout:test api_io_pp on pid 34953 2024-09-17T18:20:00.619 INFO:tasks.workunit.client.0.smithi173.stderr:+ echo 'test api_io_pp on pid 34953' 2024-09-17T18:20:00.619 INFO:tasks.workunit.client.0.smithi173.stderr:+ pids[$f]=34953 2024-09-17T18:20:00.619 INFO:tasks.workunit.client.0.smithi173.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-17T18:20:00.620 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1 -eq 1 ']' 2024-09-17T18:20:00.620 INFO:tasks.workunit.client.0.smithi173.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-09-17T18:20:00.620 INFO:tasks.workunit.client.0.smithi173.stderr:++ printf %25s api_asio 2024-09-17T18:20:00.620 INFO:tasks.workunit.client.0.smithi173.stderr:+ r=' api_asio' 2024-09-17T18:20:00.620 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-17T18:20:00.620 INFO:tasks.workunit.client.0.smithi173.stderr:++ echo api_asio 2024-09-17T18:20:00.620 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-17T18:20:00.620 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-17T18:20:00.620 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.620 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-17T18:20:00.620 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-17T18:20:00.620 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.620 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-17T18:20:00.621 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-17T18:20:00.621 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '{print $1}' 2024-09-17T18:20:00.621 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-17T18:20:00.621 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -f /etc/bashrc ']' 2024-09-17T18:20:00.621 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-17T18:20:00.621 INFO:tasks.workunit.client.0.smithi173.stderr:+ . /etc/bashrc 2024-09-17T18:20:00.621 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.621 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.621 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-17T18:20:00.621 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-17T18:20:00.621 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.621 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-17T18:20:00.621 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.621 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/gawk.sh 2024-09-17T18:20:00.622 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-17T18:20:00.622 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -z '' ']' 2024-09-17T18:20:00.622 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.622 INFO:tasks.workunit.client.0.smithi173.stderr:++ BASHRCSOURCED=Y 2024-09-17T18:20:00.622 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.622 INFO:tasks.workunit.client.0.smithi173.stderr:++ shopt -q login_shell 2024-09-17T18:20:00.622 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.622 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-17T18:20:00.622 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.622 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/lang.sh 2024-09-17T18:20:00.622 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup 2024-09-17T18:20:00.622 INFO:tasks.workunit.client.0.smithi173.stderr:+++ umask 2024-09-17T18:20:00.622 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 0022 -eq 0 ']' 2024-09-17T18:20:00.622 INFO:tasks.workunit.client.0.smithi173.stderr:++ SHELL=/bin/bash 2024-09-17T18:20:00.623 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'grep=grep --color=auto' 2024-09-17T18:20:00.623 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/locale 2024-09-17T18:20:00.623 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.623 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-17T18:20:00.623 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.623 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-17T18:20:00.623 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-17T18:20:00.623 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-17T18:20:00.623 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.623 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-17T18:20:00.623 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.623 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorls.sh 2024-09-17T18:20:00.623 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-17T18:20:00.623 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.623 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-17T18:20:00.624 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.624 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-17T18:20:00.624 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' '!' -t 0 ']' 2024-09-17T18:20:00.624 INFO:tasks.workunit.client.0.smithi173.stderr:+++ return 2024-09-17T18:20:00.624 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:00.624 INFO:tasks.workunit.client.0.smithi173.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:00.624 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.624 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.624 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-17T18:20:00.624 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.624 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-17T18:20:00.624 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-17T18:20:00.624 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.624 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.624 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-17T18:20:00.625 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export S_COLORS=auto 2024-09-17T18:20:00.625 INFO:tasks.workunit.client.0.smithi173.stderr:+++ S_COLORS=auto 2024-09-17T18:20:00.625 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.625 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-17T18:20:00.625 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.625 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-17T18:20:00.625 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.625 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:00.625 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-17T18:20:00.625 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:00.625 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-17T18:20:00.625 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-17T18:20:00.625 INFO:tasks.workunit.client.0.smithi173.stderr:+ ff=api_asio 2024-09-17T18:20:00.626 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-17T18:20:00.626 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=34968 2024-09-17T18:20:00.626 INFO:tasks.workunit.client.0.smithi173.stdout:test api_asio on pid 34968 2024-09-17T18:20:00.632 INFO:tasks.workunit.client.0.smithi173.stderr:+ echo 'test api_asio on pid 34968' 2024-09-17T18:20:00.632 INFO:tasks.workunit.client.0.smithi173.stderr:+ pids[$f]=34968 2024-09-17T18:20:00.632 INFO:tasks.workunit.client.0.smithi173.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-17T18:20:00.632 INFO:tasks.workunit.client.0.smithi173.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-09-17T18:20:00.633 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1 -eq 1 ']' 2024-09-17T18:20:00.633 INFO:tasks.workunit.client.0.smithi173.stderr:++ printf %25s api_list 2024-09-17T18:20:00.633 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-17T18:20:00.633 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-17T18:20:00.633 INFO:tasks.workunit.client.0.smithi173.stderr:+ r=' api_list' 2024-09-17T18:20:00.633 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-17T18:20:00.633 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.633 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-17T18:20:00.634 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.634 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-17T18:20:00.634 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-17T18:20:00.634 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.634 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-17T18:20:00.634 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.634 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/gawk.sh 2024-09-17T18:20:00.634 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.634 INFO:tasks.workunit.client.0.smithi173.stderr:++ echo api_list 2024-09-17T18:20:00.635 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'grep=grep --color=auto' 2024-09-17T18:20:00.635 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-17T18:20:00.635 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-17T18:20:00.635 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.635 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/lang.sh 2024-09-17T18:20:00.635 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '{print $1}' 2024-09-17T18:20:00.635 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup 2024-09-17T18:20:00.635 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-17T18:20:00.635 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.635 INFO:tasks.workunit.client.0.smithi173.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-17T18:20:00.635 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-17T18:20:00.635 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.635 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorls.sh 2024-09-17T18:20:00.635 INFO:tasks.workunit.client.0.smithi173.stderr:++++ export LANG=en_US.UTF-8 2024-09-17T18:20:00.636 INFO:tasks.workunit.client.0.smithi173.stderr:++++ LANG=en_US.UTF-8 2024-09-17T18:20:00.636 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:00.636 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' '!' -t 0 ']' 2024-09-17T18:20:00.636 INFO:tasks.workunit.client.0.smithi173.stderr:+++ return 2024-09-17T18:20:00.636 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-17T18:20:00.636 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.636 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-17T18:20:00.636 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.636 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:00.636 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-17T18:20:00.636 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG=en_US.UTF-8 2024-09-17T18:20:00.636 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -f /etc/bashrc ']' 2024-09-17T18:20:00.636 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup config 2024-09-17T18:20:00.636 INFO:tasks.workunit.client.0.smithi173.stderr:+ . /etc/bashrc 2024-09-17T18:20:00.637 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export S_COLORS=auto 2024-09-17T18:20:00.637 INFO:tasks.workunit.client.0.smithi173.stderr:+++ S_COLORS=auto 2024-09-17T18:20:00.637 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.637 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-17T18:20:00.637 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.637 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-17T18:20:00.637 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:00.637 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.637 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:00.637 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -z '' ']' 2024-09-17T18:20:00.637 INFO:tasks.workunit.client.0.smithi173.stderr:++ BASHRCSOURCED=Y 2024-09-17T18:20:00.637 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' dumb = linux ']' 2024-09-17T18:20:00.637 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.637 INFO:tasks.workunit.client.0.smithi173.stderr:++ shopt -q login_shell 2024-09-17T18:20:00.638 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.638 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-17T18:20:00.638 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/locale 2024-09-17T18:20:00.638 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.638 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/less.sh 2024-09-17T18:20:00.638 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-17T18:20:00.638 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.638 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-17T18:20:00.638 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.638 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/which2.sh 2024-09-17T18:20:00.638 INFO:tasks.workunit.client.0.smithi173.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-17T18:20:00.638 INFO:tasks.workunit.client.0.smithi173.stderr:+++ umask 2024-09-17T18:20:00.638 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-17T18:20:00.638 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-17T18:20:00.639 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-17T18:20:00.639 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.639 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-17T18:20:00.639 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.639 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-17T18:20:00.639 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 0022 -eq 0 ']' 2024-09-17T18:20:00.639 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-17T18:20:00.639 INFO:tasks.workunit.client.0.smithi173.stderr:++ SHELL=/bin/bash 2024-09-17T18:20:00.639 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.639 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.639 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-17T18:20:00.639 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.640 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-17T18:20:00.640 INFO:tasks.workunit.client.0.smithi173.stderr:+++++ readlink /proc/34924/exe 2024-09-17T18:20:00.640 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-17T18:20:00.640 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.640 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-17T18:20:00.640 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.640 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-17T18:20:00.640 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:00.640 INFO:tasks.workunit.client.0.smithi173.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:00.640 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.640 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-17T18:20:00.640 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.640 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-17T18:20:00.641 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.641 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:00.641 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-17T18:20:00.641 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:00.641 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-17T18:20:00.641 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-17T18:20:00.641 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-17T18:20:00.641 INFO:tasks.workunit.client.0.smithi173.stderr:++++ basename /usr/bin/bash 2024-09-17T18:20:00.641 INFO:tasks.workunit.client.0.smithi173.stderr:+ ff=api_list 2024-09-17T18:20:00.641 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_declare='declare -f' 2024-09-17T18:20:00.641 INFO:tasks.workunit.client.0.smithi173.stdout:test api_list on pid 34985 2024-09-17T18:20:00.650 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=34985 2024-09-17T18:20:00.650 INFO:tasks.workunit.client.0.smithi173.stderr:+ echo 'test api_list on pid 34985' 2024-09-17T18:20:00.650 INFO:tasks.workunit.client.0.smithi173.stderr:+ pids[$f]=34985 2024-09-17T18:20:00.650 INFO:tasks.workunit.client.0.smithi173.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-17T18:20:00.650 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1 -eq 1 ']' 2024-09-17T18:20:00.650 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_opt=-f 2024-09-17T18:20:00.651 INFO:tasks.workunit.client.0.smithi173.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-09-17T18:20:00.651 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export which_declare 2024-09-17T18:20:00.651 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export -f which 2024-09-17T18:20:00.651 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset i 2024-09-17T18:20:00.651 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset -f pathmunge 2024-09-17T18:20:00.651 INFO:tasks.workunit.client.0.smithi173.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-17T18:20:00.651 INFO:tasks.workunit.client.0.smithi173.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-17T18:20:00.651 INFO:tasks.workunit.client.0.smithi173.stderr:++++ export LANG=en_US.UTF-8 2024-09-17T18:20:00.651 INFO:tasks.workunit.client.0.smithi173.stderr:+ export PATH 2024-09-17T18:20:00.651 INFO:tasks.workunit.client.0.smithi173.stderr:++++ LANG=en_US.UTF-8 2024-09-17T18:20:00.651 INFO:tasks.workunit.client.0.smithi173.stderr:++ printf %25s api_lock 2024-09-17T18:20:00.651 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-17T18:20:00.651 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:00.651 INFO:tasks.workunit.client.0.smithi173.stderr:+ unset rc 2024-09-17T18:20:00.651 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-17T18:20:00.651 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-17T18:20:00.652 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:00.652 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG=en_US.UTF-8 2024-09-17T18:20:00.652 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-17T18:20:00.652 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup config 2024-09-17T18:20:00.652 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-17T18:20:00.652 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.652 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:00.652 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-17T18:20:00.652 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.652 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-17T18:20:00.652 INFO:tasks.workunit.client.0.smithi173.stderr:+ r=' api_lock' 2024-09-17T18:20:00.652 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-17T18:20:00.652 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.652 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:00.653 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' dumb = linux ']' 2024-09-17T18:20:00.653 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.653 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-17T18:20:00.653 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.653 INFO:tasks.workunit.client.0.smithi173.stderr:+ tee ceph_test_rados_api_aio.log 2024-09-17T18:20:00.653 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/less.sh 2024-09-17T18:20:00.653 INFO:tasks.workunit.client.0.smithi173.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2024-09-17T18:20:00.653 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-17T18:20:00.653 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.653 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-17T18:20:00.653 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-17T18:20:00.653 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-17T18:20:00.653 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.653 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/which2.sh 2024-09-17T18:20:00.654 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '{print $1}' 2024-09-17T18:20:00.654 INFO:tasks.workunit.client.0.smithi173.stderr:+ sed 's/^/ api_aio: /' 2024-09-17T18:20:00.654 INFO:tasks.workunit.client.0.smithi173.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-17T18:20:00.654 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-17T18:20:00.654 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.654 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-17T18:20:00.654 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.654 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-17T18:20:00.654 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -f /etc/bashrc ']' 2024-09-17T18:20:00.654 INFO:tasks.workunit.client.0.smithi173.stderr:+ . /etc/bashrc 2024-09-17T18:20:00.654 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-17T18:20:00.654 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.654 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-17T18:20:00.654 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.654 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/gawk.sh 2024-09-17T18:20:00.655 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -z '' ']' 2024-09-17T18:20:00.655 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'grep=grep --color=auto' 2024-09-17T18:20:00.655 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.655 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-17T18:20:00.655 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.655 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/lang.sh 2024-09-17T18:20:00.655 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup 2024-09-17T18:20:00.655 INFO:tasks.workunit.client.0.smithi173.stderr:+++++ readlink /proc/34931/exe 2024-09-17T18:20:00.655 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-17T18:20:00.655 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-17T18:20:00.655 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.655 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-17T18:20:00.655 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.655 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorls.sh 2024-09-17T18:20:00.655 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' '!' -t 0 ']' 2024-09-17T18:20:00.655 INFO:tasks.workunit.client.0.smithi173.stderr:+++ return 2024-09-17T18:20:00.656 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.656 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-17T18:20:00.656 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.656 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-17T18:20:00.656 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export S_COLORS=auto 2024-09-17T18:20:00.656 INFO:tasks.workunit.client.0.smithi173.stderr:+++ S_COLORS=auto 2024-09-17T18:20:00.656 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.656 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-17T18:20:00.656 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.656 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-17T18:20:00.656 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/locale 2024-09-17T18:20:00.656 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.656 INFO:tasks.workunit.client.0.smithi173.stderr:++ BASHRCSOURCED=Y 2024-09-17T18:20:00.657 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.657 INFO:tasks.workunit.client.0.smithi173.stderr:++ shopt -q login_shell 2024-09-17T18:20:00.657 INFO:tasks.workunit.client.0.smithi173.stderr:++ echo api_lock 2024-09-17T18:20:00.657 INFO:tasks.workunit.client.0.smithi173.stderr:+++ umask 2024-09-17T18:20:00.657 INFO:tasks.workunit.client.0.smithi173.stderr:++++ basename /usr/bin/bash 2024-09-17T18:20:00.657 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 0022 -eq 0 ']' 2024-09-17T18:20:00.657 INFO:tasks.workunit.client.0.smithi173.stderr:++ SHELL=/bin/bash 2024-09-17T18:20:00.657 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.657 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-17T18:20:00.657 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.657 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-17T18:20:00.657 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-17T18:20:00.657 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.657 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-17T18:20:00.657 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.658 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-17T18:20:00.658 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:00.658 INFO:tasks.workunit.client.0.smithi173.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:00.658 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.658 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-17T18:20:00.658 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.658 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-17T18:20:00.658 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.658 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:00.658 INFO:tasks.workunit.client.0.smithi173.stderr:+ ff=api_lock 2024-09-17T18:20:00.658 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-17T18:20:00.658 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:00.658 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-17T18:20:00.658 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-17T18:20:00.659 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35003 2024-09-17T18:20:00.659 INFO:tasks.workunit.client.0.smithi173.stdout:test api_lock on pid 35003 2024-09-17T18:20:00.666 INFO:tasks.workunit.client.0.smithi173.stderr:+ echo 'test api_lock on pid 35003' 2024-09-17T18:20:00.666 INFO:tasks.workunit.client.0.smithi173.stderr:+ pids[$f]=35003 2024-09-17T18:20:00.666 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_declare='declare -f' 2024-09-17T18:20:00.666 INFO:tasks.workunit.client.0.smithi173.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-17T18:20:00.666 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1 -eq 1 ']' 2024-09-17T18:20:00.666 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_opt=-f 2024-09-17T18:20:00.666 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-17T18:20:00.667 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export which_declare 2024-09-17T18:20:00.667 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export -f which 2024-09-17T18:20:00.667 INFO:tasks.workunit.client.0.smithi173.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-09-17T18:20:00.667 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset i 2024-09-17T18:20:00.667 INFO:tasks.workunit.client.0.smithi173.stderr:++ printf %25s api_lock_pp 2024-09-17T18:20:00.667 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset -f pathmunge 2024-09-17T18:20:00.667 INFO:tasks.workunit.client.0.smithi173.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-17T18:20:00.667 INFO:tasks.workunit.client.0.smithi173.stderr:+ r=' api_lock_pp' 2024-09-17T18:20:00.667 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-17T18:20:00.667 INFO:tasks.workunit.client.0.smithi173.stderr:+ export PATH 2024-09-17T18:20:00.667 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-17T18:20:00.667 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-17T18:20:00.667 INFO:tasks.workunit.client.0.smithi173.stderr:+ unset rc 2024-09-17T18:20:00.667 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-17T18:20:00.667 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.668 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-17T18:20:00.668 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.668 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-17T18:20:00.668 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-17T18:20:00.668 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.668 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-17T18:20:00.668 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.668 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/gawk.sh 2024-09-17T18:20:00.668 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.668 INFO:tasks.workunit.client.0.smithi173.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2024-09-17T18:20:00.668 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-17T18:20:00.668 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.668 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '{print $1}' 2024-09-17T18:20:00.669 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/lang.sh 2024-09-17T18:20:00.669 INFO:tasks.workunit.client.0.smithi173.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-09-17T18:20:00.669 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup 2024-09-17T18:20:00.669 INFO:tasks.workunit.client.0.smithi173.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-17T18:20:00.669 INFO:tasks.workunit.client.0.smithi173.stderr:++++ export LANG=en_US.UTF-8 2024-09-17T18:20:00.669 INFO:tasks.workunit.client.0.smithi173.stderr:++++ LANG=en_US.UTF-8 2024-09-17T18:20:00.669 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/locale 2024-09-17T18:20:00.669 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -f /etc/bashrc ']' 2024-09-17T18:20:00.669 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:00.669 INFO:tasks.workunit.client.0.smithi173.stderr:+ . /etc/bashrc 2024-09-17T18:20:00.669 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-17T18:20:00.669 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:00.669 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG=en_US.UTF-8 2024-09-17T18:20:00.670 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup config 2024-09-17T18:20:00.670 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:00.670 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -z '' ']' 2024-09-17T18:20:00.670 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:00.670 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' dumb = linux ']' 2024-09-17T18:20:00.670 INFO:tasks.workunit.client.0.smithi173.stderr:+ sed 's/^/ api_aio_pp: /' 2024-09-17T18:20:00.670 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.670 INFO:tasks.workunit.client.0.smithi173.stderr:++ BASHRCSOURCED=Y 2024-09-17T18:20:00.670 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-17T18:20:00.670 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.670 INFO:tasks.workunit.client.0.smithi173.stderr:++ shopt -q login_shell 2024-09-17T18:20:00.670 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.670 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/less.sh 2024-09-17T18:20:00.670 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-17T18:20:00.670 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.671 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-17T18:20:00.671 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.671 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/which2.sh 2024-09-17T18:20:00.671 INFO:tasks.workunit.client.0.smithi173.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-17T18:20:00.671 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-17T18:20:00.671 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-17T18:20:00.671 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-17T18:20:00.671 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.671 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-17T18:20:00.671 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.671 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-17T18:20:00.671 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-17T18:20:00.671 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.671 INFO:tasks.workunit.client.0.smithi173.stderr:++ echo api_lock_pp 2024-09-17T18:20:00.671 INFO:tasks.workunit.client.0.smithi173.stderr:+++++ readlink /proc/34940/exe 2024-09-17T18:20:00.671 INFO:tasks.workunit.client.0.smithi173.stderr:+++ umask 2024-09-17T18:20:00.672 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 0022 -eq 0 ']' 2024-09-17T18:20:00.672 INFO:tasks.workunit.client.0.smithi173.stderr:++ SHELL=/bin/bash 2024-09-17T18:20:00.672 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.672 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-17T18:20:00.672 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.672 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-17T18:20:00.672 INFO:tasks.workunit.client.0.smithi173.stderr:+ ff=api_lock_pp 2024-09-17T18:20:00.672 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-17T18:20:00.672 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.672 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-17T18:20:00.672 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.672 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-17T18:20:00.672 INFO:tasks.workunit.client.0.smithi173.stderr:++++ basename /usr/bin/bash 2024-09-17T18:20:00.672 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:00.673 INFO:tasks.workunit.client.0.smithi173.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:00.673 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.673 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-17T18:20:00.673 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.673 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'grep=grep --color=auto' 2024-09-17T18:20:00.673 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-17T18:20:00.673 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.673 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-17T18:20:00.673 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-17T18:20:00.673 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.673 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35020 2024-09-17T18:20:00.673 INFO:tasks.workunit.client.0.smithi173.stderr:+ echo 'test api_lock_pp on pid 35020' 2024-09-17T18:20:00.674 INFO:tasks.workunit.client.0.smithi173.stdout:test api_lock_pp on pid 35020 2024-09-17T18:20:00.682 INFO:tasks.workunit.client.0.smithi173.stderr:+ pids[$f]=35020 2024-09-17T18:20:00.682 INFO:tasks.workunit.client.0.smithi173.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-17T18:20:00.682 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1 -eq 1 ']' 2024-09-17T18:20:00.682 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-17T18:20:00.682 INFO:tasks.workunit.client.0.smithi173.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-09-17T18:20:00.682 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.682 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorls.sh 2024-09-17T18:20:00.682 INFO:tasks.workunit.client.0.smithi173.stderr:++ printf %25s api_misc 2024-09-17T18:20:00.683 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' '!' -t 0 ']' 2024-09-17T18:20:00.683 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:00.683 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-17T18:20:00.683 INFO:tasks.workunit.client.0.smithi173.stderr:+++ return 2024-09-17T18:20:00.683 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.683 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-17T18:20:00.683 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.683 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-17T18:20:00.683 INFO:tasks.workunit.client.0.smithi173.stderr:+ r=' api_misc' 2024-09-17T18:20:00.683 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export S_COLORS=auto 2024-09-17T18:20:00.683 INFO:tasks.workunit.client.0.smithi173.stderr:+++ S_COLORS=auto 2024-09-17T18:20:00.683 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:00.683 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-17T18:20:00.683 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-17T18:20:00.684 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.684 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-17T18:20:00.684 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.684 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_declare='declare -f' 2024-09-17T18:20:00.684 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-17T18:20:00.684 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.684 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_opt=-f 2024-09-17T18:20:00.684 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export which_declare 2024-09-17T18:20:00.684 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export -f which 2024-09-17T18:20:00.684 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-17T18:20:00.684 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset i 2024-09-17T18:20:00.684 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset -f pathmunge 2024-09-17T18:20:00.684 INFO:tasks.workunit.client.0.smithi173.stderr:++ echo api_misc 2024-09-17T18:20:00.684 INFO:tasks.workunit.client.0.smithi173.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-17T18:20:00.684 INFO:tasks.workunit.client.0.smithi173.stderr:+ export PATH 2024-09-17T18:20:00.685 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '{print $1}' 2024-09-17T18:20:00.685 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -f /etc/bashrc ']' 2024-09-17T18:20:00.685 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-17T18:20:00.685 INFO:tasks.workunit.client.0.smithi173.stderr:+ unset rc 2024-09-17T18:20:00.685 INFO:tasks.workunit.client.0.smithi173.stderr:+ . /etc/bashrc 2024-09-17T18:20:00.685 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -z '' ']' 2024-09-17T18:20:00.685 INFO:tasks.workunit.client.0.smithi173.stderr:++ BASHRCSOURCED=Y 2024-09-17T18:20:00.685 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.685 INFO:tasks.workunit.client.0.smithi173.stderr:++ shopt -q login_shell 2024-09-17T18:20:00.685 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-17T18:20:00.685 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-17T18:20:00.685 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-17T18:20:00.685 INFO:tasks.workunit.client.0.smithi173.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2024-09-17T18:20:00.686 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.686 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-17T18:20:00.686 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.686 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-17T18:20:00.686 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-17T18:20:00.686 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.686 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-17T18:20:00.686 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.686 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/gawk.sh 2024-09-17T18:20:00.686 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.686 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-17T18:20:00.686 INFO:tasks.workunit.client.0.smithi173.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-17T18:20:00.686 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.686 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/lang.sh 2024-09-17T18:20:00.686 INFO:tasks.workunit.client.0.smithi173.stderr:++++ export LANG=en_US.UTF-8 2024-09-17T18:20:00.687 INFO:tasks.workunit.client.0.smithi173.stderr:++++ LANG=en_US.UTF-8 2024-09-17T18:20:00.687 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup 2024-09-17T18:20:00.687 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:00.687 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-17T18:20:00.687 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:00.687 INFO:tasks.workunit.client.0.smithi173.stderr:+++ umask 2024-09-17T18:20:00.687 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG=en_US.UTF-8 2024-09-17T18:20:00.687 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup config 2024-09-17T18:20:00.687 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:00.687 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:00.687 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 0022 -eq 0 ']' 2024-09-17T18:20:00.687 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' dumb = linux ']' 2024-09-17T18:20:00.687 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.687 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-17T18:20:00.688 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.688 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/less.sh 2024-09-17T18:20:00.688 INFO:tasks.workunit.client.0.smithi173.stderr:++ SHELL=/bin/bash 2024-09-17T18:20:00.688 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-17T18:20:00.688 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.688 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-17T18:20:00.688 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.688 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/which2.sh 2024-09-17T18:20:00.688 INFO:tasks.workunit.client.0.smithi173.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-17T18:20:00.688 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.688 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-17T18:20:00.688 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.688 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-17T18:20:00.688 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-17T18:20:00.688 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.689 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-17T18:20:00.689 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.689 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-17T18:20:00.689 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:00.689 INFO:tasks.workunit.client.0.smithi173.stderr:+ sed 's/^/ api_io: /' 2024-09-17T18:20:00.689 INFO:tasks.workunit.client.0.smithi173.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:00.689 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.689 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'grep=grep --color=auto' 2024-09-17T18:20:00.689 INFO:tasks.workunit.client.0.smithi173.stderr:+++++ readlink /proc/34953/exe 2024-09-17T18:20:00.689 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-17T18:20:00.689 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.689 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-17T18:20:00.689 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.689 INFO:tasks.workunit.client.0.smithi173.stderr:+ ff=api_misc 2024-09-17T18:20:00.689 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-17T18:20:00.690 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-17T18:20:00.690 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.690 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-17T18:20:00.690 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35038 2024-09-17T18:20:00.690 INFO:tasks.workunit.client.0.smithi173.stdout:test api_misc on pid 35038 2024-09-17T18:20:00.699 INFO:tasks.workunit.client.0.smithi173.stderr:+ echo 'test api_misc on pid 35038' 2024-09-17T18:20:00.699 INFO:tasks.workunit.client.0.smithi173.stderr:+ pids[$f]=35038 2024-09-17T18:20:00.699 INFO:tasks.workunit.client.0.smithi173.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-17T18:20:00.699 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1 -eq 1 ']' 2024-09-17T18:20:00.699 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.699 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorls.sh 2024-09-17T18:20:00.699 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' '!' -t 0 ']' 2024-09-17T18:20:00.699 INFO:tasks.workunit.client.0.smithi173.stderr:+++ return 2024-09-17T18:20:00.699 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.699 INFO:tasks.workunit.client.0.smithi173.stderr:++ printf %25s api_misc_pp 2024-09-17T18:20:00.699 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-17T18:20:00.700 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.700 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-17T18:20:00.700 INFO:tasks.workunit.client.0.smithi173.stderr:+ r=' api_misc_pp' 2024-09-17T18:20:00.700 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export S_COLORS=auto 2024-09-17T18:20:00.700 INFO:tasks.workunit.client.0.smithi173.stderr:+++ S_COLORS=auto 2024-09-17T18:20:00.700 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.700 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-17T18:20:00.700 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.700 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-17T18:20:00.700 INFO:tasks.workunit.client.0.smithi173.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-09-17T18:20:00.700 INFO:tasks.workunit.client.0.smithi173.stderr:++++ basename /usr/bin/bash 2024-09-17T18:20:00.700 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.700 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-17T18:20:00.700 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-17T18:20:00.701 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-17T18:20:00.701 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.701 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-17T18:20:00.701 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.701 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-17T18:20:00.701 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-17T18:20:00.701 INFO:tasks.workunit.client.0.smithi173.stderr:+ tee ceph_test_rados_api_io.log 2024-09-17T18:20:00.701 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.701 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_declare='declare -f' 2024-09-17T18:20:00.701 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_opt=-f 2024-09-17T18:20:00.701 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export which_declare 2024-09-17T18:20:00.701 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export -f which 2024-09-17T18:20:00.701 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset i 2024-09-17T18:20:00.702 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/locale 2024-09-17T18:20:00.702 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset -f pathmunge 2024-09-17T18:20:00.702 INFO:tasks.workunit.client.0.smithi173.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-17T18:20:00.702 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -f /etc/bashrc ']' 2024-09-17T18:20:00.702 INFO:tasks.workunit.client.0.smithi173.stderr:+ export PATH 2024-09-17T18:20:00.702 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-17T18:20:00.702 INFO:tasks.workunit.client.0.smithi173.stderr:+ . /etc/bashrc 2024-09-17T18:20:00.702 INFO:tasks.workunit.client.0.smithi173.stderr:+ unset rc 2024-09-17T18:20:00.702 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -z '' ']' 2024-09-17T18:20:00.702 INFO:tasks.workunit.client.0.smithi173.stderr:++ BASHRCSOURCED=Y 2024-09-17T18:20:00.702 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.702 INFO:tasks.workunit.client.0.smithi173.stderr:++ shopt -q login_shell 2024-09-17T18:20:00.702 INFO:tasks.workunit.client.0.smithi173.stderr:+ sed 's/^/ api_io_pp: /' 2024-09-17T18:20:00.702 INFO:tasks.workunit.client.0.smithi173.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-09-17T18:20:00.703 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:00.703 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'grep=grep --color=auto' 2024-09-17T18:20:00.703 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-17T18:20:00.703 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-17T18:20:00.703 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-17T18:20:00.703 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.703 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-17T18:20:00.703 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.703 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorls.sh 2024-09-17T18:20:00.703 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' '!' -t 0 ']' 2024-09-17T18:20:00.703 INFO:tasks.workunit.client.0.smithi173.stderr:+++ return 2024-09-17T18:20:00.703 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.703 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-17T18:20:00.703 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.703 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-17T18:20:00.704 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export S_COLORS=auto 2024-09-17T18:20:00.704 INFO:tasks.workunit.client.0.smithi173.stderr:+++ S_COLORS=auto 2024-09-17T18:20:00.704 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.704 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-17T18:20:00.704 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.704 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-17T18:20:00.704 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.704 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:00.704 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-17T18:20:00.704 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-17T18:20:00.704 INFO:tasks.workunit.client.0.smithi173.stderr:++ echo api_misc_pp 2024-09-17T18:20:00.704 INFO:tasks.workunit.client.0.smithi173.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2024-09-17T18:20:00.704 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '{print $1}' 2024-09-17T18:20:00.705 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-17T18:20:00.705 INFO:tasks.workunit.client.0.smithi173.stderr:+++ umask 2024-09-17T18:20:00.705 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 0022 -eq 0 ']' 2024-09-17T18:20:00.705 INFO:tasks.workunit.client.0.smithi173.stderr:++ SHELL=/bin/bash 2024-09-17T18:20:00.705 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.705 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-17T18:20:00.705 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.705 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-17T18:20:00.705 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-17T18:20:00.705 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.705 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-17T18:20:00.705 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.705 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-17T18:20:00.705 INFO:tasks.workunit.client.0.smithi173.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-17T18:20:00.706 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:00.706 INFO:tasks.workunit.client.0.smithi173.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:00.706 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.706 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-17T18:20:00.706 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.706 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-17T18:20:00.706 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.706 INFO:tasks.workunit.client.0.smithi173.stderr:++++ export LANG=en_US.UTF-8 2024-09-17T18:20:00.706 INFO:tasks.workunit.client.0.smithi173.stderr:++++ LANG=en_US.UTF-8 2024-09-17T18:20:00.706 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:00.706 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-17T18:20:00.706 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:00.706 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG=en_US.UTF-8 2024-09-17T18:20:00.706 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup config 2024-09-17T18:20:00.707 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:00.707 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:00.707 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' dumb = linux ']' 2024-09-17T18:20:00.707 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.707 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-17T18:20:00.707 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.707 INFO:tasks.workunit.client.0.smithi173.stderr:+ ff=api_misc_pp 2024-09-17T18:20:00.707 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/less.sh 2024-09-17T18:20:00.707 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-17T18:20:00.707 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35057 2024-09-17T18:20:00.707 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.707 INFO:tasks.workunit.client.0.smithi173.stderr:+ echo 'test api_misc_pp on pid 35057' 2024-09-17T18:20:00.708 INFO:tasks.workunit.client.0.smithi173.stdout:test api_misc_pp on pid 35057 2024-09-17T18:20:00.721 INFO:tasks.workunit.client.0.smithi173.stderr:+ pids[$f]=35057 2024-09-17T18:20:00.722 INFO:tasks.workunit.client.0.smithi173.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-17T18:20:00.722 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1 -eq 1 ']' 2024-09-17T18:20:00.722 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-17T18:20:00.722 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.722 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/which2.sh 2024-09-17T18:20:00.722 INFO:tasks.workunit.client.0.smithi173.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-17T18:20:00.722 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-17T18:20:00.722 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-17T18:20:00.722 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-17T18:20:00.722 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.722 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-17T18:20:00.722 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.722 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-17T18:20:00.723 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-17T18:20:00.723 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.723 INFO:tasks.workunit.client.0.smithi173.stderr:+++++ readlink /proc/34968/exe 2024-09-17T18:20:00.723 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-17T18:20:00.723 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-17T18:20:00.723 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-17T18:20:00.723 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.723 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-17T18:20:00.723 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.723 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-17T18:20:00.723 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-17T18:20:00.723 INFO:tasks.workunit.client.0.smithi173.stderr:++++ basename /usr/bin/bash 2024-09-17T18:20:00.723 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.723 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-17T18:20:00.724 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.724 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/gawk.sh 2024-09-17T18:20:00.724 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.724 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-17T18:20:00.724 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.724 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/lang.sh 2024-09-17T18:20:00.724 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup 2024-09-17T18:20:00.724 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'grep=grep --color=auto' 2024-09-17T18:20:00.724 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-17T18:20:00.724 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-17T18:20:00.724 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.724 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-17T18:20:00.724 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.724 INFO:tasks.workunit.client.0.smithi173.stderr:++ printf %25s api_tier_pp 2024-09-17T18:20:00.725 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorls.sh 2024-09-17T18:20:00.725 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' '!' -t 0 ']' 2024-09-17T18:20:00.725 INFO:tasks.workunit.client.0.smithi173.stderr:+++ return 2024-09-17T18:20:00.725 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.725 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_declare='declare -f' 2024-09-17T18:20:00.725 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_opt=-f 2024-09-17T18:20:00.725 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-17T18:20:00.725 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export which_declare 2024-09-17T18:20:00.725 INFO:tasks.workunit.client.0.smithi173.stderr:+ r=' api_tier_pp' 2024-09-17T18:20:00.725 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export -f which 2024-09-17T18:20:00.725 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset i 2024-09-17T18:20:00.725 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.725 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-17T18:20:00.725 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export S_COLORS=auto 2024-09-17T18:20:00.725 INFO:tasks.workunit.client.0.smithi173.stderr:+++ S_COLORS=auto 2024-09-17T18:20:00.726 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.726 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-17T18:20:00.726 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.726 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-17T18:20:00.726 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset -f pathmunge 2024-09-17T18:20:00.726 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.726 INFO:tasks.workunit.client.0.smithi173.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-17T18:20:00.726 INFO:tasks.workunit.client.0.smithi173.stderr:+ export PATH 2024-09-17T18:20:00.726 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-17T18:20:00.726 INFO:tasks.workunit.client.0.smithi173.stderr:+ unset rc 2024-09-17T18:20:00.726 INFO:tasks.workunit.client.0.smithi173.stderr:++ echo api_tier_pp 2024-09-17T18:20:00.726 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-17T18:20:00.726 INFO:tasks.workunit.client.0.smithi173.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2024-09-17T18:20:00.726 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-17T18:20:00.726 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-17T18:20:00.726 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.727 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-17T18:20:00.727 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.727 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-17T18:20:00.727 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-17T18:20:00.727 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.727 INFO:tasks.workunit.client.0.smithi173.stderr:+ tee ceph_test_rados_api_asio.log 2024-09-17T18:20:00.727 INFO:tasks.workunit.client.0.smithi173.stderr:+ sed 's/^/ api_asio: /' 2024-09-17T18:20:00.727 INFO:tasks.workunit.client.0.smithi173.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-09-17T18:20:00.727 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-17T18:20:00.727 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-17T18:20:00.727 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -f /etc/bashrc ']' 2024-09-17T18:20:00.727 INFO:tasks.workunit.client.0.smithi173.stderr:+ . /etc/bashrc 2024-09-17T18:20:00.727 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-17T18:20:00.727 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.727 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-17T18:20:00.728 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -z '' ']' 2024-09-17T18:20:00.728 INFO:tasks.workunit.client.0.smithi173.stderr:++ BASHRCSOURCED=Y 2024-09-17T18:20:00.728 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.728 INFO:tasks.workunit.client.0.smithi173.stderr:++ shopt -q login_shell 2024-09-17T18:20:00.728 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.728 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-17T18:20:00.728 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-17T18:20:00.728 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.728 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-17T18:20:00.728 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.728 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/gawk.sh 2024-09-17T18:20:00.728 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-17T18:20:00.728 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-17T18:20:00.728 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-17T18:20:00.728 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.729 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.729 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-17T18:20:00.729 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.729 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-17T18:20:00.729 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-17T18:20:00.729 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-17T18:20:00.729 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.729 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/lang.sh 2024-09-17T18:20:00.729 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup 2024-09-17T18:20:00.729 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.729 INFO:tasks.workunit.client.0.smithi173.stderr:+++ umask 2024-09-17T18:20:00.729 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-17T18:20:00.729 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-17T18:20:00.729 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-17T18:20:00.729 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/locale 2024-09-17T18:20:00.730 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.730 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-17T18:20:00.730 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.730 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-17T18:20:00.730 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '{print $1}' 2024-09-17T18:20:00.730 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-17T18:20:00.730 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.730 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-17T18:20:00.730 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.730 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/gawk.sh 2024-09-17T18:20:00.730 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.730 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-17T18:20:00.730 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.730 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/lang.sh 2024-09-17T18:20:00.731 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup 2024-09-17T18:20:00.731 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 0022 -eq 0 ']' 2024-09-17T18:20:00.731 INFO:tasks.workunit.client.0.smithi173.stderr:++ SHELL=/bin/bash 2024-09-17T18:20:00.731 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/locale 2024-09-17T18:20:00.731 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.731 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-17T18:20:00.731 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.731 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-17T18:20:00.731 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-17T18:20:00.731 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.731 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-17T18:20:00.731 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.731 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-17T18:20:00.731 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:00.731 INFO:tasks.workunit.client.0.smithi173.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:00.732 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.732 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-17T18:20:00.732 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.732 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-17T18:20:00.732 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.732 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:00.732 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-17T18:20:00.732 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:00.732 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-17T18:20:00.732 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-17T18:20:00.732 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-17T18:20:00.732 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-17T18:20:00.732 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-17T18:20:00.733 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.733 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-17T18:20:00.733 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.733 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-17T18:20:00.733 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-17T18:20:00.733 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-17T18:20:00.733 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.733 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-17T18:20:00.733 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.733 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/gawk.sh 2024-09-17T18:20:00.733 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.734 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-17T18:20:00.734 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.734 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/lang.sh 2024-09-17T18:20:00.734 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup 2024-09-17T18:20:00.734 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/locale 2024-09-17T18:20:00.734 INFO:tasks.workunit.client.0.smithi173.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-17T18:20:00.734 INFO:tasks.workunit.client.0.smithi173.stderr:++++ export LANG=en_US.UTF-8 2024-09-17T18:20:00.734 INFO:tasks.workunit.client.0.smithi173.stderr:++++ LANG=en_US.UTF-8 2024-09-17T18:20:00.734 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:00.734 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/locale 2024-09-17T18:20:00.734 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-17T18:20:00.734 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:00.734 INFO:tasks.workunit.client.0.smithi173.stderr:+ ff=api_tier_pp 2024-09-17T18:20:00.734 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG=en_US.UTF-8 2024-09-17T18:20:00.734 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup config 2024-09-17T18:20:00.735 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:00.735 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35089 2024-09-17T18:20:00.735 INFO:tasks.workunit.client.0.smithi173.stderr:+ echo 'test api_tier_pp on pid 35089' 2024-09-17T18:20:00.735 INFO:tasks.workunit.client.0.smithi173.stdout:test api_tier_pp on pid 35089 2024-09-17T18:20:00.747 INFO:tasks.workunit.client.0.smithi173.stderr:+ pids[$f]=35089 2024-09-17T18:20:00.747 INFO:tasks.workunit.client.0.smithi173.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-17T18:20:00.747 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1 -eq 1 ']' 2024-09-17T18:20:00.747 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:00.747 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' dumb = linux ']' 2024-09-17T18:20:00.747 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.747 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-17T18:20:00.747 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.747 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/less.sh 2024-09-17T18:20:00.747 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-17T18:20:00.748 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.748 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-17T18:20:00.748 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.748 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/which2.sh 2024-09-17T18:20:00.748 INFO:tasks.workunit.client.0.smithi173.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-17T18:20:00.748 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:00.748 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-17T18:20:00.748 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:00.748 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-17T18:20:00.748 INFO:tasks.workunit.client.0.smithi173.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-09-17T18:20:00.748 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-17T18:20:00.748 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:00.748 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-17T18:20:00.749 INFO:tasks.workunit.client.0.smithi173.stderr:++ printf %25s api_pool 2024-09-17T18:20:00.749 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:00.749 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-17T18:20:00.749 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-17T18:20:00.749 INFO:tasks.workunit.client.0.smithi173.stderr:+ r=' api_pool' 2024-09-17T18:20:00.749 INFO:tasks.workunit.client.0.smithi173.stderr:+++++ readlink /proc/35003/exe 2024-09-17T18:20:00.749 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-17T18:20:00.749 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -f /etc/bashrc ']' 2024-09-17T18:20:00.749 INFO:tasks.workunit.client.0.smithi173.stderr:++++ basename /usr/bin/bash 2024-09-17T18:20:00.749 INFO:tasks.workunit.client.0.smithi173.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-17T18:20:00.749 INFO:tasks.workunit.client.0.smithi173.stderr:++++ export LANG=en_US.UTF-8 2024-09-17T18:20:00.749 INFO:tasks.workunit.client.0.smithi173.stderr:+ . /etc/bashrc 2024-09-17T18:20:00.749 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'grep=grep --color=auto' 2024-09-17T18:20:00.749 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_declare='declare -f' 2024-09-17T18:20:00.750 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_opt=-f 2024-09-17T18:20:00.750 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export which_declare 2024-09-17T18:20:00.750 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-17T18:20:00.750 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -z '' ']' 2024-09-17T18:20:00.750 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-17T18:20:00.750 INFO:tasks.workunit.client.0.smithi173.stderr:++ BASHRCSOURCED=Y 2024-09-17T18:20:00.750 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.750 INFO:tasks.workunit.client.0.smithi173.stderr:++ shopt -q login_shell 2024-09-17T18:20:00.750 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.750 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-17T18:20:00.751 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.751 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorls.sh 2024-09-17T18:20:00.751 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' '!' -t 0 ']' 2024-09-17T18:20:00.751 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export -f which 2024-09-17T18:20:00.751 INFO:tasks.workunit.client.0.smithi173.stderr:+++ return 2024-09-17T18:20:00.751 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.751 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-17T18:20:00.751 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.751 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-17T18:20:00.751 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export S_COLORS=auto 2024-09-17T18:20:00.751 INFO:tasks.workunit.client.0.smithi173.stderr:+++ S_COLORS=auto 2024-09-17T18:20:00.751 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.751 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-17T18:20:00.751 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.752 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-17T18:20:00.752 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.752 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset i 2024-09-17T18:20:00.752 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset -f pathmunge 2024-09-17T18:20:00.752 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-17T18:20:00.752 INFO:tasks.workunit.client.0.smithi173.stderr:++++ LANG=en_US.UTF-8 2024-09-17T18:20:00.752 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:00.752 INFO:tasks.workunit.client.0.smithi173.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-17T18:20:00.752 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-17T18:20:00.752 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:00.752 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG=en_US.UTF-8 2024-09-17T18:20:00.752 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup config 2024-09-17T18:20:00.752 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:00.752 INFO:tasks.workunit.client.0.smithi173.stderr:+ export PATH 2024-09-17T18:20:00.753 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:00.753 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' dumb = linux ']' 2024-09-17T18:20:00.753 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.753 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-17T18:20:00.753 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.753 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/less.sh 2024-09-17T18:20:00.753 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-17T18:20:00.753 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-17T18:20:00.753 INFO:tasks.workunit.client.0.smithi173.stderr:+ unset rc 2024-09-17T18:20:00.753 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.753 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-17T18:20:00.753 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.753 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/which2.sh 2024-09-17T18:20:00.753 INFO:tasks.workunit.client.0.smithi173.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-17T18:20:00.754 INFO:tasks.workunit.client.0.smithi173.stderr:+ tee ceph_test_rados_api_lock.log 2024-09-17T18:20:00.754 INFO:tasks.workunit.client.0.smithi173.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2024-09-17T18:20:00.754 INFO:tasks.workunit.client.0.smithi173.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-17T18:20:00.754 INFO:tasks.workunit.client.0.smithi173.stderr:++++ export LANG=en_US.UTF-8 2024-09-17T18:20:00.754 INFO:tasks.workunit.client.0.smithi173.stderr:++++ LANG=en_US.UTF-8 2024-09-17T18:20:00.754 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:00.754 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-17T18:20:00.754 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:00.754 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG=en_US.UTF-8 2024-09-17T18:20:00.754 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup config 2024-09-17T18:20:00.754 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:00.754 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:00.754 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' dumb = linux ']' 2024-09-17T18:20:00.754 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.755 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-17T18:20:00.755 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.755 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/less.sh 2024-09-17T18:20:00.755 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-17T18:20:00.755 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.755 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-17T18:20:00.755 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.755 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/which2.sh 2024-09-17T18:20:00.755 INFO:tasks.workunit.client.0.smithi173.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-17T18:20:00.755 INFO:tasks.workunit.client.0.smithi173.stderr:+++ umask 2024-09-17T18:20:00.755 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 0022 -eq 0 ']' 2024-09-17T18:20:00.755 INFO:tasks.workunit.client.0.smithi173.stderr:++ SHELL=/bin/bash 2024-09-17T18:20:00.755 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.755 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-17T18:20:00.755 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.756 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-17T18:20:00.756 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-17T18:20:00.756 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.756 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-17T18:20:00.756 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.756 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-17T18:20:00.756 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:00.756 INFO:tasks.workunit.client.0.smithi173.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:00.756 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.756 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-17T18:20:00.756 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.756 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-17T18:20:00.756 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.756 INFO:tasks.workunit.client.0.smithi173.stderr:+ sed 's/^/ api_lock: /' 2024-09-17T18:20:00.757 INFO:tasks.workunit.client.0.smithi173.stderr:++ echo api_pool 2024-09-17T18:20:00.757 INFO:tasks.workunit.client.0.smithi173.stderr:+++++ readlink /proc/35020/exe 2024-09-17T18:20:00.757 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '{print $1}' 2024-09-17T18:20:00.757 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-17T18:20:00.757 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-17T18:20:00.757 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-17T18:20:00.757 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.757 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-17T18:20:00.757 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.757 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-17T18:20:00.757 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-17T18:20:00.757 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.757 INFO:tasks.workunit.client.0.smithi173.stderr:++++ basename /usr/bin/bash 2024-09-17T18:20:00.757 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_declare='declare -f' 2024-09-17T18:20:00.757 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_opt=-f 2024-09-17T18:20:00.758 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export which_declare 2024-09-17T18:20:00.758 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export -f which 2024-09-17T18:20:00.758 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset i 2024-09-17T18:20:00.758 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset -f pathmunge 2024-09-17T18:20:00.758 INFO:tasks.workunit.client.0.smithi173.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-17T18:20:00.758 INFO:tasks.workunit.client.0.smithi173.stderr:+ export PATH 2024-09-17T18:20:00.758 INFO:tasks.workunit.client.0.smithi173.stderr:+ ff=api_pool 2024-09-17T18:20:00.758 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-17T18:20:00.758 INFO:tasks.workunit.client.0.smithi173.stderr:+ unset rc 2024-09-17T18:20:00.758 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35109 2024-09-17T18:20:00.758 INFO:tasks.workunit.client.0.smithi173.stderr:+ echo 'test api_pool on pid 35109' 2024-09-17T18:20:00.759 INFO:tasks.workunit.client.0.smithi173.stdout:test api_pool on pid 35109 2024-09-17T18:20:00.769 INFO:tasks.workunit.client.0.smithi173.stderr:+ pids[$f]=35109 2024-09-17T18:20:00.769 INFO:tasks.workunit.client.0.smithi173.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-17T18:20:00.769 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1 -eq 1 ']' 2024-09-17T18:20:00.769 INFO:tasks.workunit.client.0.smithi173.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-09-17T18:20:00.769 INFO:tasks.workunit.client.0.smithi173.stderr:++ printf %25s api_snapshots 2024-09-17T18:20:00.769 INFO:tasks.workunit.client.0.smithi173.stderr:+ r=' api_snapshots' 2024-09-17T18:20:00.769 INFO:tasks.workunit.client.0.smithi173.stderr:++ echo api_snapshots 2024-09-17T18:20:00.769 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -f /etc/bashrc ']' 2024-09-17T18:20:00.769 INFO:tasks.workunit.client.0.smithi173.stderr:+ . /etc/bashrc 2024-09-17T18:20:00.769 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -z '' ']' 2024-09-17T18:20:00.770 INFO:tasks.workunit.client.0.smithi173.stderr:++ BASHRCSOURCED=Y 2024-09-17T18:20:00.770 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.770 INFO:tasks.workunit.client.0.smithi173.stderr:++ shopt -q login_shell 2024-09-17T18:20:00.770 INFO:tasks.workunit.client.0.smithi173.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2024-09-17T18:20:00.770 INFO:tasks.workunit.client.0.smithi173.stderr:+++ umask 2024-09-17T18:20:00.770 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:00.770 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-17T18:20:00.770 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:00.770 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-17T18:20:00.770 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-17T18:20:00.770 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 0022 -eq 0 ']' 2024-09-17T18:20:00.770 INFO:tasks.workunit.client.0.smithi173.stderr:++ SHELL=/bin/bash 2024-09-17T18:20:00.770 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.770 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-17T18:20:00.771 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.771 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-17T18:20:00.771 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-17T18:20:00.771 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.771 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-17T18:20:00.771 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.771 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-17T18:20:00.771 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:00.771 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-17T18:20:00.771 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-17T18:20:00.771 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-17T18:20:00.771 INFO:tasks.workunit.client.0.smithi173.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:00.771 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.771 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-17T18:20:00.771 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.772 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-17T18:20:00.772 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-17T18:20:00.772 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.772 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-17T18:20:00.772 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.772 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-17T18:20:00.772 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.772 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/gawk.sh 2024-09-17T18:20:00.772 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-17T18:20:00.772 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.772 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.772 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-17T18:20:00.772 INFO:tasks.workunit.client.0.smithi173.stderr:+++++ readlink /proc/35038/exe 2024-09-17T18:20:00.772 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.773 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/lang.sh 2024-09-17T18:20:00.773 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup 2024-09-17T18:20:00.773 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-17T18:20:00.773 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.773 INFO:tasks.workunit.client.0.smithi173.stderr:+ sed 's/^/ api_lock_pp: /' 2024-09-17T18:20:00.773 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/locale 2024-09-17T18:20:00.773 INFO:tasks.workunit.client.0.smithi173.stderr:++++ basename /usr/bin/bash 2024-09-17T18:20:00.773 INFO:tasks.workunit.client.0.smithi173.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-17T18:20:00.773 INFO:tasks.workunit.client.0.smithi173.stderr:++++ export LANG=en_US.UTF-8 2024-09-17T18:20:00.773 INFO:tasks.workunit.client.0.smithi173.stderr:++++ LANG=en_US.UTF-8 2024-09-17T18:20:00.773 INFO:tasks.workunit.client.0.smithi173.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-09-17T18:20:00.773 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:00.773 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_declare='declare -f' 2024-09-17T18:20:00.773 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_opt=-f 2024-09-17T18:20:00.774 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-17T18:20:00.774 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export which_declare 2024-09-17T18:20:00.774 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export -f which 2024-09-17T18:20:00.774 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset i 2024-09-17T18:20:00.774 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset -f pathmunge 2024-09-17T18:20:00.774 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:00.774 INFO:tasks.workunit.client.0.smithi173.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-17T18:20:00.774 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG=en_US.UTF-8 2024-09-17T18:20:00.774 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup config 2024-09-17T18:20:00.774 INFO:tasks.workunit.client.0.smithi173.stderr:+ export PATH 2024-09-17T18:20:00.774 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-17T18:20:00.774 INFO:tasks.workunit.client.0.smithi173.stderr:+ unset rc 2024-09-17T18:20:00.774 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:00.774 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:00.775 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' dumb = linux ']' 2024-09-17T18:20:00.775 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.775 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-17T18:20:00.775 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.775 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/less.sh 2024-09-17T18:20:00.775 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-17T18:20:00.775 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.775 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-17T18:20:00.775 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.775 INFO:tasks.workunit.client.0.smithi173.stderr:+ tee ceph_test_rados_api_misc.log 2024-09-17T18:20:00.775 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/which2.sh 2024-09-17T18:20:00.775 INFO:tasks.workunit.client.0.smithi173.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-17T18:20:00.775 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'grep=grep --color=auto' 2024-09-17T18:20:00.775 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-17T18:20:00.775 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-17T18:20:00.776 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.776 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-17T18:20:00.776 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.776 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorls.sh 2024-09-17T18:20:00.776 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' '!' -t 0 ']' 2024-09-17T18:20:00.776 INFO:tasks.workunit.client.0.smithi173.stderr:+++ return 2024-09-17T18:20:00.776 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.776 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-17T18:20:00.776 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.776 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-17T18:20:00.776 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export S_COLORS=auto 2024-09-17T18:20:00.776 INFO:tasks.workunit.client.0.smithi173.stderr:+++ S_COLORS=auto 2024-09-17T18:20:00.776 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.776 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-17T18:20:00.777 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.777 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-17T18:20:00.777 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.777 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '{print $1}' 2024-09-17T18:20:00.777 INFO:tasks.workunit.client.0.smithi173.stderr:+++++ readlink /proc/34985/exe 2024-09-17T18:20:00.777 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:00.777 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-17T18:20:00.777 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:00.777 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-17T18:20:00.777 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-17T18:20:00.777 INFO:tasks.workunit.client.0.smithi173.stderr:++++ basename /usr/bin/bash 2024-09-17T18:20:00.777 INFO:tasks.workunit.client.0.smithi173.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2024-09-17T18:20:00.777 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_declare='declare -f' 2024-09-17T18:20:00.777 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_opt=-f 2024-09-17T18:20:00.778 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export which_declare 2024-09-17T18:20:00.778 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export -f which 2024-09-17T18:20:00.778 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset i 2024-09-17T18:20:00.778 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset -f pathmunge 2024-09-17T18:20:00.778 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-17T18:20:00.778 INFO:tasks.workunit.client.0.smithi173.stderr:+ ff=api_snapshots 2024-09-17T18:20:00.778 INFO:tasks.workunit.client.0.smithi173.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-17T18:20:00.778 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35151 2024-09-17T18:20:00.778 INFO:tasks.workunit.client.0.smithi173.stdout:test api_snapshots on pid 35151 2024-09-17T18:20:00.783 INFO:tasks.workunit.client.0.smithi173.stderr:+ echo 'test api_snapshots on pid 35151' 2024-09-17T18:20:00.784 INFO:tasks.workunit.client.0.smithi173.stderr:+ pids[$f]=35151 2024-09-17T18:20:00.784 INFO:tasks.workunit.client.0.smithi173.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-17T18:20:00.784 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1 -eq 1 ']' 2024-09-17T18:20:00.784 INFO:tasks.workunit.client.0.smithi173.stderr:+ export PATH 2024-09-17T18:20:00.784 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-17T18:20:00.784 INFO:tasks.workunit.client.0.smithi173.stderr:+ unset rc 2024-09-17T18:20:00.784 INFO:tasks.workunit.client.0.smithi173.stderr:+ sed 's/^/ api_list: /' 2024-09-17T18:20:00.784 INFO:tasks.workunit.client.0.smithi173.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-17T18:20:00.784 INFO:tasks.workunit.client.0.smithi173.stderr:++++ export LANG=en_US.UTF-8 2024-09-17T18:20:00.784 INFO:tasks.workunit.client.0.smithi173.stderr:++++ LANG=en_US.UTF-8 2024-09-17T18:20:00.784 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:00.784 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-17T18:20:00.784 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:00.784 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG=en_US.UTF-8 2024-09-17T18:20:00.785 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup config 2024-09-17T18:20:00.785 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:00.785 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:00.785 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' dumb = linux ']' 2024-09-17T18:20:00.785 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.785 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-17T18:20:00.785 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.785 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/less.sh 2024-09-17T18:20:00.785 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-17T18:20:00.785 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.785 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-17T18:20:00.785 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.785 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/which2.sh 2024-09-17T18:20:00.786 INFO:tasks.workunit.client.0.smithi173.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-17T18:20:00.786 INFO:tasks.workunit.client.0.smithi173.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-09-17T18:20:00.786 INFO:tasks.workunit.client.0.smithi173.stderr:+ tee ceph_test_rados_api_list.log 2024-09-17T18:20:00.786 INFO:tasks.workunit.client.0.smithi173.stderr:++ printf %25s api_snapshots_pp 2024-09-17T18:20:00.786 INFO:tasks.workunit.client.0.smithi173.stderr:+ r=' api_snapshots_pp' 2024-09-17T18:20:00.786 INFO:tasks.workunit.client.0.smithi173.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2024-09-17T18:20:00.786 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '{print $1}' 2024-09-17T18:20:00.786 INFO:tasks.workunit.client.0.smithi173.stderr:++ echo api_snapshots_pp 2024-09-17T18:20:00.786 INFO:tasks.workunit.client.0.smithi173.stderr:+ sed 's/^/ api_misc: /' 2024-09-17T18:20:00.786 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -f /etc/bashrc ']' 2024-09-17T18:20:00.786 INFO:tasks.workunit.client.0.smithi173.stderr:+ . /etc/bashrc 2024-09-17T18:20:00.786 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'grep=grep --color=auto' 2024-09-17T18:20:00.786 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-17T18:20:00.786 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-17T18:20:00.787 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -z '' ']' 2024-09-17T18:20:00.787 INFO:tasks.workunit.client.0.smithi173.stderr:++ BASHRCSOURCED=Y 2024-09-17T18:20:00.787 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.787 INFO:tasks.workunit.client.0.smithi173.stderr:++ shopt -q login_shell 2024-09-17T18:20:00.787 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.787 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-17T18:20:00.787 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.787 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorls.sh 2024-09-17T18:20:00.787 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' '!' -t 0 ']' 2024-09-17T18:20:00.787 INFO:tasks.workunit.client.0.smithi173.stderr:+++ return 2024-09-17T18:20:00.787 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.787 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-17T18:20:00.787 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.787 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-17T18:20:00.787 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export S_COLORS=auto 2024-09-17T18:20:00.788 INFO:tasks.workunit.client.0.smithi173.stderr:+++ S_COLORS=auto 2024-09-17T18:20:00.788 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.788 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-17T18:20:00.788 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.788 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-17T18:20:00.788 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.788 INFO:tasks.workunit.client.0.smithi173.stderr:+ ff=api_snapshots_pp 2024-09-17T18:20:00.788 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35190 2024-09-17T18:20:00.788 INFO:tasks.workunit.client.0.smithi173.stdout:test api_snapshots_pp on pid 35190 2024-09-17T18:20:00.793 INFO:tasks.workunit.client.0.smithi173.stderr:+ echo 'test api_snapshots_pp on pid 35190' 2024-09-17T18:20:00.793 INFO:tasks.workunit.client.0.smithi173.stderr:+ pids[$f]=35190 2024-09-17T18:20:00.793 INFO:tasks.workunit.client.0.smithi173.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-17T18:20:00.793 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1 -eq 1 ']' 2024-09-17T18:20:00.793 INFO:tasks.workunit.client.0.smithi173.stderr:+++ umask 2024-09-17T18:20:00.793 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 0022 -eq 0 ']' 2024-09-17T18:20:00.793 INFO:tasks.workunit.client.0.smithi173.stderr:++ SHELL=/bin/bash 2024-09-17T18:20:00.793 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.793 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-17T18:20:00.794 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.794 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-17T18:20:00.794 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-17T18:20:00.794 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.794 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-17T18:20:00.794 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.794 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-17T18:20:00.794 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:00.794 INFO:tasks.workunit.client.0.smithi173.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:00.794 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.794 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-17T18:20:00.794 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.794 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-17T18:20:00.794 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.795 INFO:tasks.workunit.client.0.smithi173.stderr:++ printf %25s api_stat 2024-09-17T18:20:00.795 INFO:tasks.workunit.client.0.smithi173.stderr:+ r=' api_stat' 2024-09-17T18:20:00.795 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-17T18:20:00.795 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-17T18:20:00.795 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-17T18:20:00.795 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.795 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-17T18:20:00.795 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.795 INFO:tasks.workunit.client.0.smithi173.stderr:+++++ readlink /proc/35057/exe 2024-09-17T18:20:00.795 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-17T18:20:00.795 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-17T18:20:00.795 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.795 INFO:tasks.workunit.client.0.smithi173.stderr:++ echo api_stat 2024-09-17T18:20:00.795 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '{print $1}' 2024-09-17T18:20:00.795 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-17T18:20:00.796 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-17T18:20:00.796 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-17T18:20:00.796 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.796 INFO:tasks.workunit.client.0.smithi173.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-09-17T18:20:00.796 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-17T18:20:00.796 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.796 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-17T18:20:00.796 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-17T18:20:00.796 INFO:tasks.workunit.client.0.smithi173.stderr:++++ basename /usr/bin/bash 2024-09-17T18:20:00.796 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.796 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-17T18:20:00.796 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.796 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/gawk.sh 2024-09-17T18:20:00.797 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.797 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-17T18:20:00.797 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.797 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/lang.sh 2024-09-17T18:20:00.797 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup 2024-09-17T18:20:00.797 INFO:tasks.workunit.client.0.smithi173.stderr:+ ff=api_stat 2024-09-17T18:20:00.797 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35216 2024-09-17T18:20:00.797 INFO:tasks.workunit.client.0.smithi173.stderr:+ echo 'test api_stat on pid 35216' 2024-09-17T18:20:00.797 INFO:tasks.workunit.client.0.smithi173.stdout:test api_stat on pid 35216 2024-09-17T18:20:00.802 INFO:tasks.workunit.client.0.smithi173.stderr:+ pids[$f]=35216 2024-09-17T18:20:00.802 INFO:tasks.workunit.client.0.smithi173.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-17T18:20:00.803 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1 -eq 1 ']' 2024-09-17T18:20:00.803 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -f /etc/bashrc ']' 2024-09-17T18:20:00.803 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_declare='declare -f' 2024-09-17T18:20:00.803 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_opt=-f 2024-09-17T18:20:00.803 INFO:tasks.workunit.client.0.smithi173.stderr:+ . /etc/bashrc 2024-09-17T18:20:00.803 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export which_declare 2024-09-17T18:20:00.803 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export -f which 2024-09-17T18:20:00.803 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset i 2024-09-17T18:20:00.803 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset -f pathmunge 2024-09-17T18:20:00.803 INFO:tasks.workunit.client.0.smithi173.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-17T18:20:00.803 INFO:tasks.workunit.client.0.smithi173.stderr:+ export PATH 2024-09-17T18:20:00.803 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-17T18:20:00.803 INFO:tasks.workunit.client.0.smithi173.stderr:+ unset rc 2024-09-17T18:20:00.803 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -z '' ']' 2024-09-17T18:20:00.803 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'grep=grep --color=auto' 2024-09-17T18:20:00.804 INFO:tasks.workunit.client.0.smithi173.stderr:++ BASHRCSOURCED=Y 2024-09-17T18:20:00.804 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-17T18:20:00.804 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.804 INFO:tasks.workunit.client.0.smithi173.stderr:++ shopt -q login_shell 2024-09-17T18:20:00.804 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-17T18:20:00.804 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.804 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-17T18:20:00.804 INFO:tasks.workunit.client.0.smithi173.stderr:+++ umask 2024-09-17T18:20:00.804 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/locale 2024-09-17T18:20:00.804 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.804 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorls.sh 2024-09-17T18:20:00.804 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' '!' -t 0 ']' 2024-09-17T18:20:00.804 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 0022 -eq 0 ']' 2024-09-17T18:20:00.805 INFO:tasks.workunit.client.0.smithi173.stderr:++ SHELL=/bin/bash 2024-09-17T18:20:00.805 INFO:tasks.workunit.client.0.smithi173.stderr:+++ return 2024-09-17T18:20:00.805 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.805 INFO:tasks.workunit.client.0.smithi173.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-09-17T18:20:00.805 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.805 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-17T18:20:00.805 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.805 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-17T18:20:00.805 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export S_COLORS=auto 2024-09-17T18:20:00.805 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-17T18:20:00.805 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.805 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-17T18:20:00.805 INFO:tasks.workunit.client.0.smithi173.stderr:+++ S_COLORS=auto 2024-09-17T18:20:00.805 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.806 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-17T18:20:00.806 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.806 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-17T18:20:00.806 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.806 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-17T18:20:00.806 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:00.806 INFO:tasks.workunit.client.0.smithi173.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:00.806 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.806 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-17T18:20:00.806 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.806 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-17T18:20:00.806 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-17T18:20:00.806 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.806 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.807 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-17T18:20:00.807 INFO:tasks.workunit.client.0.smithi173.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2024-09-17T18:20:00.807 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.807 INFO:tasks.workunit.client.0.smithi173.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-09-17T18:20:00.807 INFO:tasks.workunit.client.0.smithi173.stderr:++ printf %25s api_stat_pp 2024-09-17T18:20:00.807 INFO:tasks.workunit.client.0.smithi173.stderr:+ r=' api_stat_pp' 2024-09-17T18:20:00.807 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -f /etc/bashrc ']' 2024-09-17T18:20:00.807 INFO:tasks.workunit.client.0.smithi173.stderr:+ . /etc/bashrc 2024-09-17T18:20:00.807 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -z '' ']' 2024-09-17T18:20:00.807 INFO:tasks.workunit.client.0.smithi173.stderr:++ BASHRCSOURCED=Y 2024-09-17T18:20:00.807 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.807 INFO:tasks.workunit.client.0.smithi173.stderr:++ shopt -q login_shell 2024-09-17T18:20:00.807 INFO:tasks.workunit.client.0.smithi173.stderr:++ echo api_stat_pp 2024-09-17T18:20:00.807 INFO:tasks.workunit.client.0.smithi173.stderr:+ sed 's/^/ api_misc_pp: /' 2024-09-17T18:20:00.807 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '{print $1}' 2024-09-17T18:20:00.808 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-09-17T18:20:00.808 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: [----------] Global test environment set-up. 2024-09-17T18:20:00.808 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: [----------] 12 tests from AsioRados 2024-09-17T18:20:00.808 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.680+0000 7f2fc5d77780 10 monclient: get_monmap_and_config 2024-09-17T18:20:00.808 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.680+0000 7f2fc5d77780 10 monclient: build_initial_monmap 2024-09-17T18:20:00.808 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.680+0000 7f2fc5d77780 10 monclient: monmap: 2024-09-17T18:20:00.808 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: epoch 0 2024-09-17T18:20:00.808 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:20:00.808 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: last_changed 0.000000 2024-09-17T18:20:00.808 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: created 0.000000 2024-09-17T18:20:00.809 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: min_mon_release 0 (unknown) 2024-09-17T18:20:00.809 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: election_strategy: 1 2024-09-17T18:20:00.809 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:20:00.809 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:20:00.809 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:20:00.809 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.809 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.680+0000 7f2fc5d77780 1 Processor -- start 2024-09-17T18:20:00.809 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.680+0000 7f2fc5d77780 1 -- start start 2024-09-17T18:20:00.809 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.680+0000 7f2fc5d77780 10 monclient: init 2024-09-17T18:20:00.809 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.681+0000 7f2fc5d77780 10 monclient: _reopen_session rank -1 2024-09-17T18:20:00.809 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.681+0000 7f2fc5d77780 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:20:00.809 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.681+0000 7f2fc5d77780 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5588a7b1fa60 0x5588a7b04fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:20:00.809 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.681+0000 7f2fc5d77780 10 monclient(hunting): picked mon.noname-a con 0x5588a7b1fa60 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:20:00.810 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.681+0000 7f2fc5d77780 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588a7b34140 0x5588a7b34520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:20:00.810 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.681+0000 7f2fc5d77780 10 monclient(hunting): picked mon.noname-b con 0x5588a7b34140 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:20:00.810 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.681+0000 7f2fc5d77780 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5588a7b34a60 0x5588a7c6e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:20:00.810 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.681+0000 7f2fc5d77780 10 monclient(hunting): picked mon.noname-c con 0x5588a7b34a60 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:20:00.810 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.681+0000 7f2fc5d77780 10 monclient(hunting): start opening mon connection 2024-09-17T18:20:00.810 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.681+0000 7f2fc5d77780 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5588a7a5dd50 con 0x5588a7b1fa60 2024-09-17T18:20:00.810 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.681+0000 7f2fc5d77780 10 monclient(hunting): start opening mon connection 2024-09-17T18:20:00.810 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.681+0000 7f2fc5d77780 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5588a7abc420 con 0x5588a7b34140 2024-09-17T18:20:00.810 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.681+0000 7f2fc5d77780 10 monclient(hunting): start opening mon connection 2024-09-17T18:20:00.810 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.681+0000 7f2fc5d77780 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x5588a7b30c50 con 0x5588a7b34a60 2024-09-17T18:20:00.810 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.681+0000 7f2fc5d77780 10 monclient(hunting): _renew_subs 2024-09-17T18:20:00.810 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.681+0000 7f2fc5d77780 10 monclient(hunting): authenticate will time out at 798.473368s 2024-09-17T18:20:00.811 INFO:tasks.workunit.client.0.smithi173.stderr:+++ umask 2024-09-17T18:20:00.811 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:00.811 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-17T18:20:00.811 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:00.811 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-17T18:20:00.812 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.681+0000 7f2fc4066640 1 -- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5588a7b34a60 msgr2=0x5588a7c6e7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:20:00.812 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.681+0000 7f2fc4066640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5588a7b34a60 0x5588a7c6e7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:20:00.812 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.681+0000 7f2fc3064640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588a7b34140 0x5588a7b34520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:20:00.812 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.681+0000 7f2fc3865640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5588a7b1fa60 0x5588a7b04fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:20:00.812 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.681+0000 7f2fc3865640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5588a7b1fa60 0x5588a7b04fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.173:34702/0 (socket says 172.21.15.173:34702) 2024-09-17T18:20:00.812 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.681+0000 7f2fc3865640 1 -- 172.21.15.173:0/2093952871 learned_addr learned my addr 172.21.15.173:0/2093952871 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-09-17T18:20:00.812 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.681+0000 7f2fc3865640 10 monclient(hunting): get_auth_request con 0x5588a7b1fa60 auth_method 0 2024-09-17T18:20:00.812 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.681+0000 7f2fc3865640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:20:00.812 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.681+0000 7f2fc3865640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:20:00.812 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.681+0000 7f2fc3865640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:20:00.812 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.681+0000 7f2fc3064640 10 monclient(hunting): get_auth_request con 0x5588a7b34140 auth_method 0 2024-09-17T18:20:00.813 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.681+0000 7f2fc3064640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:20:00.813 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.681+0000 7f2fc3064640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:20:00.813 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.681+0000 7f2fc3064640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:20:00.813 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.681+0000 7f2fc3064640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:20:00.813 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.681+0000 7f2fc3064640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:20:00.813 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.681+0000 7f2fc3064640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:20:00.813 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.681+0000 7f2fc3865640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:20:00.813 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.681+0000 7f2fc3865640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:20:00.813 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.682+0000 7f2fc3865640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:20:00.813 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.682+0000 7f2fc3064640 10 monclient(hunting): handle_auth_done global_id 4426 payload 274 2024-09-17T18:20:00.813 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.682+0000 7f2fc3064640 1 -- 172.21.15.173:0/2093952871 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5588a7b34a60 msgr2=0x5588a7c6e7a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:20:00.817 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-17T18:20:00.817 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 0022 -eq 0 ']' 2024-09-17T18:20:00.817 INFO:tasks.workunit.client.0.smithi173.stderr:++ SHELL=/bin/bash 2024-09-17T18:20:00.817 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.817 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-17T18:20:00.817 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.817 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-17T18:20:00.817 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-17T18:20:00.817 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.817 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-17T18:20:00.817 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.817 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-17T18:20:00.818 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:00.818 INFO:tasks.workunit.client.0.smithi173.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:00.818 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.818 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-17T18:20:00.818 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.818 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-17T18:20:00.818 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-17T18:20:00.818 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.818 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-17T18:20:00.818 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-17T18:20:00.818 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-17T18:20:00.818 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.818 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-17T18:20:00.818 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.818 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-17T18:20:00.819 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-17T18:20:00.819 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.819 INFO:tasks.workunit.client.0.smithi173.stderr:+ ff=api_stat_pp 2024-09-17T18:20:00.819 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'grep=grep --color=auto' 2024-09-17T18:20:00.820 INFO:tasks.workunit.client.0.smithi173.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-17T18:20:00.820 INFO:tasks.workunit.client.0.smithi173.stderr:++++ export LANG=en_US.UTF-8 2024-09-17T18:20:00.820 INFO:tasks.workunit.client.0.smithi173.stderr:++++ LANG=en_US.UTF-8 2024-09-17T18:20:00.820 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:00.820 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-17T18:20:00.820 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35243 2024-09-17T18:20:00.820 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.682+0000 7f2fc3064640 1 --2- 172.21.15.173:0/2093952871 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5588a7b34a60 0x5588a7c6e7a0 unknown :-1 s=START_CONNECtest api_stat_pp on pid 35243 2024-09-17T18:20:00.824 INFO:tasks.workunit.client.0.smithi173.stderr:+ echo 'test api_stat_pp on pid 35243' 2024-09-17T18:20:00.824 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:00.824 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG=en_US.UTF-8 2024-09-17T18:20:00.824 INFO:tasks.workunit.client.0.smithi173.stderr:+ pids[$f]=35243 2024-09-17T18:20:00.825 INFO:tasks.workunit.client.0.smithi173.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-17T18:20:00.825 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup config 2024-09-17T18:20:00.825 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1 -eq 1 ']' 2024-09-17T18:20:00.825 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:00.825 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-17T18:20:00.825 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:00.825 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' dumb = linux ']' 2024-09-17T18:20:00.825 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.825 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-17T18:20:00.825 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.825 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/less.sh 2024-09-17T18:20:00.825 INFO:tasks.workunit.client.0.smithi173.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-09-17T18:20:00.826 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-17T18:20:00.826 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-17T18:20:00.826 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.826 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-17T18:20:00.826 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.826 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.826 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/which2.sh 2024-09-17T18:20:00.826 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-17T18:20:00.826 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.826 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorls.sh 2024-09-17T18:20:00.826 INFO:tasks.workunit.client.0.smithi173.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-17T18:20:00.826 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' '!' -t 0 ']' 2024-09-17T18:20:00.826 INFO:tasks.workunit.client.0.smithi173.stderr:+++ return 2024-09-17T18:20:00.827 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.827 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-17T18:20:00.827 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.827 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-17T18:20:00.827 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export S_COLORS=auto 2024-09-17T18:20:00.827 INFO:tasks.workunit.client.0.smithi173.stderr:+++ S_COLORS=auto 2024-09-17T18:20:00.827 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.827 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-17T18:20:00.827 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.827 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-17T18:20:00.827 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.827 INFO:tasks.workunit.client.0.smithi173.stderr:++ printf %25s api_watch_notify 2024-09-17T18:20:00.827 INFO:tasks.workunit.client.0.smithi173.stderr:+ r=' api_watch_notify' 2024-09-17T18:20:00.828 INFO:tasks.workunit.client.0.smithi173.stderr:+++++ readlink /proc/35109/exe 2024-09-17T18:20:00.828 INFO:tasks.workunit.client.0.smithi173.stdout:T pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:20:00.828 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.682+0000 7f2fc3064640 1 -- 172.21.15.173:0/2093952871 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5588a7b1fa60 msgr2=0x5588a7b04fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:20:00.828 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.682+0000 7f2fc3064640 1 --2- 172.21.15.173:0/2093952871 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5588a7b1fa60 0x5588a7b04fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:20:00.828 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.682+0000 7f2fc3064640 10 monclient: _finish_hunting 0 2024-09-17T18:20:00.828 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.682+0000 7f2fc3064640 1 monclient: found mon.noname-b 2024-09-17T18:20:00.828 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.682+0000 7f2fc3064640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:20:00.829 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.682+0000 7f2fc3064640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.173:3300/0 2024-09-17T18:20:00.829 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.682+0000 7f2fc3064640 1 -- 172.21.15.173:0/2093952871 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588a7bc2c40 con 0x5588a7b34140 2024-09-17T18:20:00.829 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.684+0000 7f2fc3064640 1 -- 172.21.15.173:0/2093952871 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588a7b34140 msgr2=0x5588a7b34520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:20:00.829 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.684+0000 7f2fc3064640 1 --2- 172.21.15.173:0/2093952871 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588a7b34140 0x5588a7b34520 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8002d20 tx=0x7f2fb800b5b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a22a465b023b9ad6 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:20:00.829 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.685+0000 7f2fc2062640 1 -- 172.21.15.173:0/2093952871 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb800f600 con 0x5588a7b34140 2024-09-17T18:20:00.829 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.686+0000 7f2fc2062640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:20:00.829 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.686+0000 7f2fc2062640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:20:00.829 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.686+0000 7f2fc2062640 10 monclient: dump: 2024-09-17T18:20:00.829 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: epoch 1 2024-09-17T18:20:00.829 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:20:00.829 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:20:00.829 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:20:00.830 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: min_mon_release 19 (squid) 2024-09-17T18:20:00.830 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: election_strategy: 1 2024-09-17T18:20:00.830 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:20:00.830 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:20:00.830 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:20:00.830 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.830 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.686+0000 7f2fc2062640 10 monclient: _finish_auth 0 2024-09-17T18:20:00.830 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.686+0000 7f2fc2062640 10 monclient: _check_auth_tickets 2024-09-17T18:20:00.830 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.686+0000 7f2fc2062640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:20:00.830 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.686+0000 7f2fc5d77780 5 monclient: authenticate success, global_id 4426 2024-09-17T18:20:00.830 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.686+0000 7f2fc5d77780 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:20:00.830 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.686+0000 7f2fc2062640 1 -- 172.21.15.173:0/2093952871 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fb8014070 con 0x5588a7b34140 2024-09-17T18:20:00.830 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.686+0000 7f2fc2062640 10 monclient: handle_config config(0 keys) 2024-09-17T18:20:00.832 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -f /etc/bashrc ']' 2024-09-17T18:20:00.832 INFO:tasks.workunit.client.0.smithi173.stderr:+ . /etc/bashrc 2024-09-17T18:20:00.832 INFO:tasks.workunit.client.0.smithi173.stderr:++++ basename /usr/bin/bash 2024-09-17T18:20:00.832 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -z '' ']' 2024-09-17T18:20:00.832 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '{print $1}' 2024-09-17T18:20:00.832 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_declare='declare -f' 2024-09-17T18:20:00.832 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_opt=-f 2024-09-17T18:20:00.833 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export which_declare 2024-09-17T18:20:00.833 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export -f which 2024-09-17T18:20:00.833 INFO:tasks.workunit.client.0.smithi173.stderr:++ BASHRCSOURCED=Y 2024-09-17T18:20:00.833 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset i 2024-09-17T18:20:00.833 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset -f pathmunge 2024-09-17T18:20:00.833 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.833 INFO:tasks.workunit.client.0.smithi173.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-17T18:20:00.833 INFO:tasks.workunit.client.0.smithi173.stderr:++ shopt -q login_shell 2024-09-17T18:20:00.833 INFO:tasks.workunit.client.0.smithi173.stderr:+ export PATH 2024-09-17T18:20:00.833 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-17T18:20:00.834 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.686+0000 7f2fc5d77780 10 monclient: get_monmap_and_config success 2024-09-17T18:20:00.834 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.686+0000 7f2fc5d77780 4 set_mon_vals no callback set 2024-09-17T18:20:00.834 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.687+0000 7f2fc5d77780 10 monclient: shutdown 2024-09-17T18:20:00.834 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.687+0000 7f2fc5d77780 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:20:00.834 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.687+0000 7f2fc5d77780 1 -- 172.21.15.173:0/2093952871 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588a7b34140 msgr2=0x5588a7b34520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:20:00.834 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.687+0000 7f2fc5d77780 1 --2- 172.21.15.173:0/2093952871 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588a7b34140 0x5588a7b34520 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8002d20 tx=0x7f2fb800b5b0 comp rx=0 tx=0).stop 2024-09-17T18:20:00.834 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.687+0000 7f2fc5d77780 1 -- 172.21.15.173:0/2093952871 shutdown_connections 2024-09-17T18:20:00.834 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.687+0000 7f2fc5d77780 1 --2- 172.21.15.173:0/2093952871 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588a7b34140 0x5588a7b34520 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:20:00.834 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.687+0000 7f2fc5d77780 1 --2- 172.21.15.173:0/2093952871 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5588a7b34a60 0x5588a7c6e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:20:00.834 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.687+0000 7f2fc5d77780 1 --2- 172.21.15.173:0/2093952871 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5588a7b1fa60 0x5588a7b04fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:20:00.834 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.687+0000 7f2fc5d77780 1 -- 172.21.15.173:0/2093952871 >> 172.21.15.173:0/2093952871 conn(0x5588a7b03bf0 msgr2=0x5588a7b06f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:20:00.834 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.687+0000 7f2fc5d77780 1 -- 172.21.15.173:0/2093952871 shutdown_connections 2024-09-17T18:20:00.835 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.687+0000 7f2fc5d77780 1 -- 172.21.15.173:0/2093952871 wait complete. 2024-09-17T18:20:00.835 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.687+0000 7f2fc5d77780 10 monclient: build_initial_monmap 2024-09-17T18:20:00.835 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.687+0000 7f2fc5d77780 10 monclient: monmap: 2024-09-17T18:20:00.835 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: epoch 0 2024-09-17T18:20:00.835 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:20:00.835 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: last_changed 0.000000 2024-09-17T18:20:00.835 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: created 0.000000 2024-09-17T18:20:00.835 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: min_mon_release 0 (unknown) 2024-09-17T18:20:00.835 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: election_strategy: 1 2024-09-17T18:20:00.835 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:20:00.836 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:20:00.836 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:20:00.836 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.836 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.688+0000 7f2fc5d77780 1 librados: starting msgr at 2024-09-17T18:20:00.836 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.688+0000 7f2fc5d77780 1 librados: starting objecter 2024-09-17T18:20:00.836 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.688+0000 7f2fc5d77780 1 Processor -- start 2024-09-17T18:20:00.836 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.688+0000 7f2fc5d77780 1 -- start start 2024-09-17T18:20:00.836 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.688+0000 7f2fc5d77780 1 librados: setting wanted keys 2024-09-17T18:20:00.836 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.688+0000 7f2fc5d77780 1 librados: calling monclient init 2024-09-17T18:20:00.836 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.688+0000 7f2fc5d77780 10 monclient: init 2024-09-17T18:20:00.836 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.688+0000 7f2fc5d77780 10 monclient: _reopen_session rank -1 2024-09-17T18:20:00.836 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.688+0000 7f2fc5d77780 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:20:00.837 INFO:tasks.workunit.client.0.smithi173.stderr:+ unset rc 2024-09-17T18:20:00.837 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.688+0000 7f2fc5d77780 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5588a7b1fa60 0x5588a7b1ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:20:00.837 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.688+0000 7f2fc5d77780 10 monclient(hunting): picked mon.noname-a con 0x5588a7b1fa60 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:20:00.837 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.688+0000 7f2fc5d77780 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588a7b34a60 0x5588a7c6f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:20:00.837 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.688+0000 7f2fc5d77780 10 monclient(hunting): picked mon.noname-b con 0x5588a7b34a60 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:20:00.837 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.688+0000 7f2fc5d77780 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5588a7c6fcf0 0x5588a7c70180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:20:00.837 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.688+0000 7f2fc5d77780 10 monclient(hunting): picked mon.noname-c con 0x5588a7c6fcf0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:20:00.838 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.688+0000 7f2fc5d77780 10 monclient(hunting): start opening mon connection 2024-09-17T18:20:00.838 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.688+0000 7f2fc5d77780 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5588a7b361e0 con 0x5588a7b1fa60 2024-09-17T18:20:00.838 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.688+0000 7f2fc5d77780 10 monclient(hunting): start opening mon connection 2024-09-17T18:20:00.838 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.688+0000 7f2fc5d77780 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5588a7b35dc0 con 0x5588a7b34a60 2024-09-17T18:20:00.838 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.688+0000 7f2fc5d77780 10 monclient(hunting): start opening mon connection 2024-09-17T18:20:00.838 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.688+0000 7f2fc5d77780 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x5588a7b34e90 con 0x5588a7c6fcf0 2024-09-17T18:20:00.838 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.688+0000 7f2fc5d77780 10 monclient(hunting): _renew_subs 2024-09-17T18:20:00.838 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.688+0000 7f2fc5d77780 10 monclient(hunting): authenticate will time out at 798.480989s 2024-09-17T18:20:00.838 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.689+0000 7f2fc4066640 1 -- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5588a7c6fcf0 msgr2=0x5588a7c70180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:20:00.838 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.689+0000 7f2fc4066640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5588a7c6fcf0 0x5588a7c70180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:20:00.838 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.689+0000 7f2fc3865640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5588a7b1fa60 0x5588a7b1ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:20:00.838 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.689+0000 7f2fc3064640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588a7b34a60 0x5588a7c6f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:20:00.839 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.689+0000 7f2fc3865640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5588a7b1fa60 0x5588a7b1ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.173:34706/0 (socket says 172.21.15.173:34706) 2024-09-17T18:20:00.841 INFO:tasks.workunit.client.0.smithi173.stderr:+++ umask 2024-09-17T18:20:00.841 INFO:tasks.workunit.client.0.smithi173.stderr:++ echo api_watch_notify 2024-09-17T18:20:00.841 INFO:tasks.workunit.client.0.smithi173.stderr:+ ff=api_watch_notify 2024-09-17T18:20:00.841 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-17T18:20:00.841 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-17T18:20:00.841 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-17T18:20:00.841 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.841 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-17T18:20:00.841 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.841 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-17T18:20:00.841 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-17T18:20:00.841 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.842 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'grep=grep --color=auto' 2024-09-17T18:20:00.842 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 0022 -eq 0 ']' 2024-09-17T18:20:00.842 INFO:tasks.workunit.client.0.smithi173.stderr:++ SHELL=/bin/bash 2024-09-17T18:20:00.842 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-17T18:20:00.842 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.842 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-17T18:20:00.842 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.842 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-17T18:20:00.842 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35264 2024-09-17T18:20:00.843 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.689+0000 7f2fc3865640 1 -- 172.21.15.173:0/56431533 learned_addr learned my addr 172.21.15.173:0/56431533 (peer_addr_fortest api_watch_notify on pid 35264 2024-09-17T18:20:00.844 INFO:tasks.workunit.client.0.smithi173.stderr:+ echo 'test api_watch_notify on pid 35264' 2024-09-17T18:20:00.844 INFO:tasks.workunit.client.0.smithi173.stderr:+ pids[$f]=35264 2024-09-17T18:20:00.844 INFO:tasks.workunit.client.0.smithi173.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-17T18:20:00.844 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1 -eq 1 ']' 2024-09-17T18:20:00.844 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-17T18:20:00.844 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-17T18:20:00.844 INFO:tasks.workunit.client.0.smithi173.stdout:_me v2:172.21.15.173:0/0) 2024-09-17T18:20:00.845 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.689+0000 7f2fc3865640 10 monclient(hunting): get_auth_request con 0x5588a7b1fa60 auth_method 0 2024-09-17T18:20:00.845 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.689+0000 7f2fc3865640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:20:00.845 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.689+0000 7f2fc3865640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:20:00.845 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.689+0000 7f2fc3865640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:20:00.845 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.689+0000 7f2fc3064640 10 monclient(hunting): get_auth_request con 0x5588a7b34a60 auth_method 0 2024-09-17T18:20:00.845 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.689+0000 7f2fc3064640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:20:00.845 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.689+0000 7f2fc3064640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:20:00.845 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.689+0000 7f2fc3064640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:20:00.845 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.689+0000 7f2fc3064640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:20:00.845 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.689+0000 7f2fc3064640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:20:00.846 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.689+0000 7f2fc3064640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:20:00.846 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.689+0000 7f2fc3865640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:20:00.846 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.689+0000 7f2fc3865640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:20:00.846 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.689+0000 7f2fc3865640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:20:00.846 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.689+0000 7f2fc3865640 10 monclient(hunting): handle_auth_done global_id 4413 payload 824 2024-09-17T18:20:00.846 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.689+0000 7f2fc3865640 1 -- 172.21.15.173:0/56431533 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5588a7c6fcf0 msgr2=0x5588a7c70180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:20:00.846 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.689+0000 7f2fc3865640 1 --2- 172.21.15.173:0/56431533 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5588a7c6fcf0 0x5588a7c70180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:20:00.846 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.689+0000 7f2fc3865640 1 -- 172.21.15.173:0/56431533 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588a7b34a60 msgr2=0x5588a7c6f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:20:00.846 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.689+0000 7f2fc3865640 1 --2- 172.21.15.173:0/56431533 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588a7b34a60 0x5588a7c6f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:20:00.846 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.689+0000 7f2fc3865640 10 monclient: _finish_hunting 0 2024-09-17T18:20:00.846 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.689+0000 7f2fc3865640 1 monclient: found mon.noname-a 2024-09-17T18:20:00.847 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.689+0000 7f2fc3865640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:20:00.847 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.689+0000 7f2fc3865640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:20:00.847 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.689+0000 7f2fc3865640 1 -- 172.21.15.173:0/56431533 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588a7b33e70 con 0x5588a7b1fa60 2024-09-17T18:20:00.847 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.689+0000 7f2fc3064640 1 --2- 172.21.15.173:0/56431533 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588a7b34a60 0x5588a7c6f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:20:00.849 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.850 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-17T18:20:00.850 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.850 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorls.sh 2024-09-17T18:20:00.850 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' '!' -t 0 ']' 2024-09-17T18:20:00.850 INFO:tasks.workunit.client.0.smithi173.stderr:+++ return 2024-09-17T18:20:00.850 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.850 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-17T18:20:00.850 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.850 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-17T18:20:00.850 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.850 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export S_COLORS=auto 2024-09-17T18:20:00.850 INFO:tasks.workunit.client.0.smithi173.stderr:+++ S_COLORS=auto 2024-09-17T18:20:00.851 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.851 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-17T18:20:00.851 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.851 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-17T18:20:00.851 INFO:tasks.workunit.client.0.smithi173.stderr:++ printf %25s api_watch_notify_pp 2024-09-17T18:20:00.851 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.851 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-17T18:20:00.851 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.851 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-17T18:20:00.851 INFO:tasks.workunit.client.0.smithi173.stderr:+ r=' api_watch_notify_pp' 2024-09-17T18:20:00.851 INFO:tasks.workunit.client.0.smithi173.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-09-17T18:20:00.852 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:00.852 INFO:tasks.workunit.client.0.smithi173.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:00.852 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.852 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-17T18:20:00.852 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.852 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-17T18:20:00.852 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.852 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '{print $1}' 2024-09-17T18:20:00.852 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -f /etc/bashrc ']' 2024-09-17T18:20:00.853 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.692+0000 7f2fc3865640 1 -- 172.21.15.173:0/56431533 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5588a7b1fa60 msgr2=0x5588a7b1ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:20:00.853 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.692+0000 7f2fc3865640 1 --2- 172.21.15.173:0/56431533 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5588a7b1fa60 0x5588a7b1ead0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2fb4029d60 tx=0x7f2fb4002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:20:00.853 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.692+0000 7f2fc1060640 10 client.?.objecter ms_handle_connect 0x5588a7b1fa60 2024-09-17T18:20:00.853 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.692+0000 7f2fc1060640 10 client.?.objecter resend_mon_ops 2024-09-17T18:20:00.853 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.694+0000 7f2fc1060640 1 -- 172.21.15.173:0/56431533 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb4051c00 con 0x5588a7b1fa60 2024-09-17T18:20:00.853 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.694+0000 7f2fc1060640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:20:00.853 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.694+0000 7f2fc1060640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:20:00.853 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.694+0000 7f2fc1060640 10 monclient: dump: 2024-09-17T18:20:00.853 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: epoch 1 2024-09-17T18:20:00.853 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:20:00.854 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:20:00.854 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:20:00.854 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: min_mon_release 19 (squid) 2024-09-17T18:20:00.854 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: election_strategy: 1 2024-09-17T18:20:00.854 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:20:00.854 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:20:00.854 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:20:00.854 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.854 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.694+0000 7f2fc1060640 10 monclient: _finish_auth 0 2024-09-17T18:20:00.854 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.694+0000 7f2fc1060640 10 monclient: _check_auth_tickets 2024-09-17T18:20:00.854 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.694+0000 7f2fc1060640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:20:00.854 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.694+0000 7f2fc1060640 1 -- 172.21.15.173:0/56431533 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fb4051da0 con 0x5588a7b1fa60 2024-09-17T18:20:00.854 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.694+0000 7f2fc1060640 10 monclient: handle_config config(0 keys) 2024-09-17T18:20:00.854 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.694+0000 7f2fc1060640 1 -- 172.21.15.173:0/56431533 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb4008790 con 0x5588a7b1fa60 2024-09-17T18:20:00.855 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.694+0000 7f2fc1060640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:20:00.855 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.694+0000 7f2fc1060640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:20:00.855 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.694+0000 7f2fc1060640 10 monclient: dump: 2024-09-17T18:20:00.855 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: epoch 1 2024-09-17T18:20:00.855 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:20:00.855 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:20:00.855 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:20:00.855 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: min_mon_release 19 (squid) 2024-09-17T18:20:00.855 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: election_strategy: 1 2024-09-17T18:20:00.855 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:20:00.855 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:20:00.855 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:20:00.855 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.856 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.694+0000 7f2fc5d77780 5 monclient: authenticate success, global_id 4413 2024-09-17T18:20:00.856 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.694+0000 7f2fc5d77780 10 monclient: _renew_subs 2024-09-17T18:20:00.859 INFO:tasks.workunit.client.0.smithi173.stderr:+ . /etc/bashrc 2024-09-17T18:20:00.860 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -z '' ']' 2024-09-17T18:20:00.860 INFO:tasks.workunit.client.0.smithi173.stderr:++ BASHRCSOURCED=Y 2024-09-17T18:20:00.860 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.860 INFO:tasks.workunit.client.0.smithi173.stderr:++ shopt -q login_shell 2024-09-17T18:20:00.860 INFO:tasks.workunit.client.0.smithi173.stderr:++ echo api_watch_notify_pp 2024-09-17T18:20:00.860 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-17T18:20:00.860 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-17T18:20:00.860 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-17T18:20:00.860 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.860 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-17T18:20:00.860 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.861 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-17T18:20:00.861 INFO:tasks.workunit.client.0.smithi173.stderr:+++ umask 2024-09-17T18:20:00.861 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-17T18:20:00.861 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.861 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-17T18:20:00.861 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.861 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/gawk.sh 2024-09-17T18:20:00.861 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.861 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-17T18:20:00.861 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.861 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/lang.sh 2024-09-17T18:20:00.861 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup 2024-09-17T18:20:00.862 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 0022 -eq 0 ']' 2024-09-17T18:20:00.862 INFO:tasks.workunit.client.0.smithi173.stderr:++ SHELL=/bin/bash 2024-09-17T18:20:00.862 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.862 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-17T18:20:00.862 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.862 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-17T18:20:00.862 INFO:tasks.workunit.client.0.smithi173.stderr:+ ff=api_watch_notify_pp 2024-09-17T18:20:00.862 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-17T18:20:00.862 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.862 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-17T18:20:00.862 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.862 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-17T18:20:00.863 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:00.863 INFO:tasks.workunit.client.0.smithi173.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:00.863 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.863 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-17T18:20:00.863 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35277 2024-09-17T18:20:00.863 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:2test api_watch_notify_pp on pid 35277 2024-09-17T18:20:00.865 INFO:tasks.workunit.client.0.smithi173.stderr:+ echo 'test api_watch_notify_pp on pid 35277' 2024-09-17T18:20:00.866 INFO:tasks.workunit.client.0.smithi173.stderr:+ pids[$f]=35277 2024-09-17T18:20:00.866 INFO:tasks.workunit.client.0.smithi173.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-17T18:20:00.866 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1 -eq 1 ']' 2024-09-17T18:20:00.866 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.866 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-17T18:20:00.866 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.866 INFO:tasks.workunit.client.0.smithi173.stderr:++ printf %25s api_cmd 2024-09-17T18:20:00.866 INFO:tasks.workunit.client.0.smithi173.stderr:+ r=' api_cmd' 2024-09-17T18:20:00.866 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-17T18:20:00.866 INFO:tasks.workunit.client.0.smithi173.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-09-17T18:20:00.867 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-17T18:20:00.867 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-17T18:20:00.867 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.867 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-17T18:20:00.867 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.867 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-17T18:20:00.867 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-17T18:20:00.867 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.867 INFO:tasks.workunit.client.0.smithi173.stdout:0:00.694+0000 7f2fc5d77780 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:20:00.868 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.694+0000 7f2fc5d77780 1 -- 172.21.15.173:0/56431533 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5588a7b06430 con 0x5588a7b1fa60 2024-09-17T18:20:00.868 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.694+0000 7f2fc5d77780 10 client.4413.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:20:00.868 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.694+0000 7f2fc5d77780 10 monclient: _renew_subs 2024-09-17T18:20:00.868 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.694+0000 7f2fc5d77780 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:20:00.868 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.694+0000 7f2fc5d77780 1 -- 172.21.15.173:0/56431533 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588a7c72190 con 0x5588a7b1fa60 2024-09-17T18:20:00.868 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.694+0000 7f2fc5d77780 1 librados: init done 2024-09-17T18:20:00.868 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.694+0000 7f2fc5d77780 10 librados: wait_for_osdmap waiting 2024-09-17T18:20:00.868 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.695+0000 7f2fc1060640 1 -- 172.21.15.173:0/56431533 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f2fb4008930 con 0x5588a7b1fa60 2024-09-17T18:20:00.868 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.696+0000 7f2fc1060640 1 --2- 172.21.15.173:0/56431533 >> v2:172.21.15.173:6812/2423786037 conn(0x7f2fb003e390 0x7f2fb0040850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:20:00.868 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.696+0000 7f2fc1060640 1 -- 172.21.15.173:0/56431533 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 3534+0+0 (secure 0 0 0) 0x7f2fb40955a0 con 0x5588a7b1fa60 2024-09-17T18:20:00.868 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.696+0000 7f2fc1060640 10 client.4413.objecter ms_dispatch 0x5588a7ab1240 osd_map(17..17 src has 1..17) 2024-09-17T18:20:00.868 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.696+0000 7f2fc1060640 3 client.4413.objecter handle_osd_map got epochs [17,17] > 0 2024-09-17T18:20:00.869 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.696+0000 7f2fc1060640 3 client.4413.objecter handle_osd_map decoding full epoch 17 2024-09-17T18:20:00.869 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.696+0000 7f2fc1060640 20 client.4413.objecter dump_active .. 0 homeless 2024-09-17T18:20:00.869 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.696+0000 7f2fc5d77780 10 librados: wait_for_osdmap done waiting 2024-09-17T18:20:00.869 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.696+0000 7f2fc5d77780 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:20:00.869 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.696+0000 7f2fc5d77780 1 -- 172.21.15.173:0/56431533 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2fb40515c0 con 0x5588a7b1fa60 2024-09-17T18:20:00.869 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.696+0000 7f2fc3064640 1 --2- 172.21.15.173:0/56431533 >> v2:172.21.15.173:6812/2423786037 conn(0x7f2fb003e390 0x7f2fb0040850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:20:00.869 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.697+0000 7f2fc3064640 10 monclient: get_auth_request con 0x7f2fb003e390 auth_method 0 2024-09-17T18:20:00.869 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.697+0000 7f2fc1060640 1 -- 172.21.15.173:0/56431533 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f2fb406a560 con 0x5588a7b1fa60 2024-09-17T18:20:00.869 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.697+0000 7f2fc1060640 10 monclient: handle_get_version_reply finishing 1 version 17 2024-09-17T18:20:00.869 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.697+0000 7f2fc1861640 4 set_mon_vals no callback set 2024-09-17T18:20:00.869 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.698+0000 7f2fc1861640 10 client.4413.objecter _get_latest_version latest 17, have it 2024-09-17T18:20:00.869 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.699+0000 7f2fc5d77780 10 client.4413.objecter create_pool name=ceph_test_rados_api_asio 2024-09-17T18:20:00.878 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -f /etc/bashrc ']' 2024-09-17T18:20:00.878 INFO:tasks.workunit.client.0.smithi173.stderr:+ . /etc/bashrc 2024-09-17T18:20:00.878 INFO:tasks.workunit.client.0.smithi173.stderr:++ echo api_cmd 2024-09-17T18:20:00.878 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -z '' ']' 2024-09-17T18:20:00.878 INFO:tasks.workunit.client.0.smithi173.stderr:++ BASHRCSOURCED=Y 2024-09-17T18:20:00.878 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.878 INFO:tasks.workunit.client.0.smithi173.stderr:++ shopt -q login_shell 2024-09-17T18:20:00.879 INFO:tasks.workunit.client.0.smithi173.stderr:+++ umask 2024-09-17T18:20:00.879 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-17T18:20:00.879 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-17T18:20:00.879 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-17T18:20:00.879 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.879 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-17T18:20:00.879 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.879 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-17T18:20:00.879 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-17T18:20:00.879 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.879 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 0022 -eq 0 ']' 2024-09-17T18:20:00.879 INFO:tasks.workunit.client.0.smithi173.stderr:++ SHELL=/bin/bash 2024-09-17T18:20:00.879 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.879 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-17T18:20:00.880 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.880 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-17T18:20:00.880 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-17T18:20:00.880 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.880 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-17T18:20:00.880 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.880 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-17T18:20:00.880 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-17T18:20:00.880 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-17T18:20:00.880 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-17T18:20:00.880 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.880 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-17T18:20:00.880 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.881 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-17T18:20:00.881 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:00.881 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-17T18:20:00.881 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.881 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-17T18:20:00.881 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.881 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/gawk.sh 2024-09-17T18:20:00.881 INFO:tasks.workunit.client.0.smithi173.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:00.881 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.881 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.881 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-17T18:20:00.881 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.881 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/lang.sh 2024-09-17T18:20:00.881 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-17T18:20:00.882 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup 2024-09-17T18:20:00.882 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.882 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-17T18:20:00.882 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.882 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/locale 2024-09-17T18:20:00.882 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-17T18:20:00.882 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '{print $1}' 2024-09-17T18:20:00.882 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-17T18:20:00.882 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-17T18:20:00.882 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.882 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-17T18:20:00.882 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.882 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-17T18:20:00.882 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-17T18:20:00.883 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.883 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-17T18:20:00.883 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.883 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/gawk.sh 2024-09-17T18:20:00.883 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.883 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:00.883 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-17T18:20:00.883 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-17T18:20:00.883 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:00.883 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-17T18:20:00.883 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-17T18:20:00.883 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.883 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/lang.sh 2024-09-17T18:20:00.884 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup 2024-09-17T18:20:00.884 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/locale 2024-09-17T18:20:00.884 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'grep=grep --color=auto' 2024-09-17T18:20:00.884 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-17T18:20:00.884 INFO:tasks.workunit.client.0.smithi173.stderr:+ sed 's/^/ api_pool: /' 2024-09-17T18:20:00.884 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-17T18:20:00.884 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.884 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-17T18:20:00.884 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.884 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorls.sh 2024-09-17T18:20:00.884 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'grep=grep --color=auto' 2024-09-17T18:20:00.884 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' '!' -t 0 ']' 2024-09-17T18:20:00.884 INFO:tasks.workunit.client.0.smithi173.stderr:+++ return 2024-09-17T18:20:00.884 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.885 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-17T18:20:00.885 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.885 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-17T18:20:00.885 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export S_COLORS=auto 2024-09-17T18:20:00.885 INFO:tasks.workunit.client.0.smithi173.stderr:+++ S_COLORS=auto 2024-09-17T18:20:00.885 INFO:tasks.workunit.client.0.smithi173.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2024-09-17T18:20:00.885 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.885 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-17T18:20:00.885 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.885 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-17T18:20:00.885 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.885 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-17T18:20:00.885 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-17T18:20:00.885 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.886 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-17T18:20:00.886 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.886 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorls.sh 2024-09-17T18:20:00.886 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' '!' -t 0 ']' 2024-09-17T18:20:00.886 INFO:tasks.workunit.client.0.smithi173.stderr:+++ return 2024-09-17T18:20:00.886 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.886 INFO:tasks.workunit.client.0.smithi173.stderr:+ ff=api_cmd 2024-09-17T18:20:00.886 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-17T18:20:00.886 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.886 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-17T18:20:00.886 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35311 2024-09-17T18:20:00.886 INFO:tasks.workunit.client.0.smithi173.stderr:+ echo 'test api_cmd on pid 35311' 2024-09-17T18:20:00.886 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.699+0000 7f2fc5d77780 10 clitest api_cmd on pid 35311 2024-09-17T18:20:00.893 INFO:tasks.workunit.client.0.smithi173.stderr:+ pids[$f]=35311 2024-09-17T18:20:00.893 INFO:tasks.workunit.client.0.smithi173.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-17T18:20:00.893 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1 -eq 1 ']' 2024-09-17T18:20:00.893 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export S_COLORS=auto 2024-09-17T18:20:00.893 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-17T18:20:00.893 INFO:tasks.workunit.client.0.smithi173.stderr:+++ S_COLORS=auto 2024-09-17T18:20:00.893 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.893 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-17T18:20:00.893 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.894 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-17T18:20:00.894 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/locale 2024-09-17T18:20:00.894 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.894 INFO:tasks.workunit.client.0.smithi173.stderr:++ printf %25s api_cmd_pp 2024-09-17T18:20:00.894 INFO:tasks.workunit.client.0.smithi173.stderr:+ r=' api_cmd_pp' 2024-09-17T18:20:00.894 INFO:tasks.workunit.client.0.smithi173.stderr:+ tee ceph_test_rados_api_pool.log 2024-09-17T18:20:00.894 INFO:tasks.workunit.client.0.smithi173.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-17T18:20:00.894 INFO:tasks.workunit.client.0.smithi173.stderr:++++ export LANG=en_US.UTF-8 2024-09-17T18:20:00.894 INFO:tasks.workunit.client.0.smithi173.stderr:++++ LANG=en_US.UTF-8 2024-09-17T18:20:00.894 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:00.895 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-17T18:20:00.895 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:00.895 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG=en_US.UTF-8 2024-09-17T18:20:00.895 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup config 2024-09-17T18:20:00.895 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:00.895 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:00.895 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' dumb = linux ']' 2024-09-17T18:20:00.895 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.895 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-17T18:20:00.895 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.895 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/less.sh 2024-09-17T18:20:00.895 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-17T18:20:00.895 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.895 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-17T18:20:00.896 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.896 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/which2.sh 2024-09-17T18:20:00.896 INFO:tasks.workunit.client.0.smithi173.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-17T18:20:00.896 INFO:tasks.workunit.client.0.smithi173.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-09-17T18:20:00.896 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-17T18:20:00.896 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-17T18:20:00.896 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-17T18:20:00.896 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.896 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-17T18:20:00.896 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.896 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-17T18:20:00.896 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-17T18:20:00.897 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.897 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-17T18:20:00.897 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.897 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/gawk.sh 2024-09-17T18:20:00.897 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.897 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-17T18:20:00.897 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.897 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/lang.sh 2024-09-17T18:20:00.897 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup 2024-09-17T18:20:00.897 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:00.897 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-17T18:20:00.897 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:00.897 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-17T18:20:00.898 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-17T18:20:00.898 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '{print $1}' 2024-09-17T18:20:00.898 INFO:tasks.workunit.client.0.smithi173.stderr:+++++ readlink /proc/35216/exe 2024-09-17T18:20:00.898 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-17T18:20:00.898 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-17T18:20:00.898 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-17T18:20:00.898 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.898 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-17T18:20:00.898 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.898 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-17T18:20:00.899 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-17T18:20:00.899 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.899 INFO:tasks.workunit.client.0.smithi173.stderr:++ echo api_cmd_pp 2024-09-17T18:20:00.899 INFO:tasks.workunit.client.0.smithi173.stderr:+ ff=api_cmd_pp 2024-09-17T18:20:00.899 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:00.899 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35340 2024-09-17T18:20:00.899 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-17T18:20:00.899 INFO:tasks.workunit.client.0.smithi173.stdout:test api_cmd_pp on pid 35340 2024-09-17T18:20:00.918 INFO:tasks.workunit.client.0.smithi173.stderr:+ echo 'test api_cmd_pp on pid 35340' 2024-09-17T18:20:00.918 INFO:tasks.workunit.client.0.smithi173.stderr:+ pids[$f]=35340 2024-09-17T18:20:00.918 INFO:tasks.workunit.client.0.smithi173.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-17T18:20:00.918 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1 -eq 1 ']' 2024-09-17T18:20:00.918 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:00.918 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-17T18:20:00.918 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-17T18:20:00.918 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -f /etc/bashrc ']' 2024-09-17T18:20:00.918 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'grep=grep --color=auto' 2024-09-17T18:20:00.918 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-17T18:20:00.919 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-17T18:20:00.919 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.919 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-17T18:20:00.919 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/locale 2024-09-17T18:20:00.919 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.919 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorls.sh 2024-09-17T18:20:00.919 INFO:tasks.workunit.client.0.smithi173.stderr:+ . /etc/bashrc 2024-09-17T18:20:00.919 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' '!' -t 0 ']' 2024-09-17T18:20:00.919 INFO:tasks.workunit.client.0.smithi173.stderr:+++ return 2024-09-17T18:20:00.919 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.919 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-17T18:20:00.919 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.919 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-17T18:20:00.920 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export S_COLORS=auto 2024-09-17T18:20:00.920 INFO:tasks.workunit.client.0.smithi173.stderr:+++ S_COLORS=auto 2024-09-17T18:20:00.920 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.920 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-17T18:20:00.920 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.920 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-17T18:20:00.920 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.920 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -z '' ']' 2024-09-17T18:20:00.920 INFO:tasks.workunit.client.0.smithi173.stderr:++ printf %25s api_service 2024-09-17T18:20:00.920 INFO:tasks.workunit.client.0.smithi173.stderr:++++ basename /usr/bin/bash 2024-09-17T18:20:00.920 INFO:tasks.workunit.client.0.smithi173.stderr:++ BASHRCSOURCED=Y 2024-09-17T18:20:00.920 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.920 INFO:tasks.workunit.client.0.smithi173.stderr:++ shopt -q login_shell 2024-09-17T18:20:00.920 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-17T18:20:00.921 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:00.921 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-17T18:20:00.921 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_declare='declare -f' 2024-09-17T18:20:00.921 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_opt=-f 2024-09-17T18:20:00.921 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:00.921 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export which_declare 2024-09-17T18:20:00.921 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export -f which 2024-09-17T18:20:00.921 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset i 2024-09-17T18:20:00.921 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset -f pathmunge 2024-09-17T18:20:00.921 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-17T18:20:00.921 INFO:tasks.workunit.client.0.smithi173.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-17T18:20:00.921 INFO:tasks.workunit.client.0.smithi173.stderr:+ export PATH 2024-09-17T18:20:00.921 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-17T18:20:00.922 INFO:tasks.workunit.client.0.smithi173.stderr:+ unset rc 2024-09-17T18:20:00.922 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-17T18:20:00.922 INFO:tasks.workunit.client.0.smithi173.stderr:+ r=' api_service' 2024-09-17T18:20:00.922 INFO:tasks.workunit.client.0.smithi173.stderr:+++ umask 2024-09-17T18:20:00.922 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 0022 -eq 0 ']' 2024-09-17T18:20:00.922 INFO:tasks.workunit.client.0.smithi173.stderr:++ SHELL=/bin/bash 2024-09-17T18:20:00.922 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.922 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-17T18:20:00.922 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.922 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-17T18:20:00.922 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-17T18:20:00.922 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.923 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-17T18:20:00.923 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.923 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-17T18:20:00.923 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:00.923 INFO:tasks.workunit.client.0.smithi173.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:00.923 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.923 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-17T18:20:00.923 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.923 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-17T18:20:00.923 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.923 INFO:tasks.workunit.client.0.smithi173.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2024-09-17T18:20:00.923 INFO:tasks.workunit.client.0.smithi173.stderr:+ tee ceph_test_rados_api_stat.log 2024-09-17T18:20:00.923 INFO:tasks.workunit.client.0.smithi173.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-17T18:20:00.923 INFO:tasks.workunit.client.0.smithi173.stderr:++++ export LANG=en_US.UTF-8 2024-09-17T18:20:00.924 INFO:tasks.workunit.client.0.smithi173.stderr:++++ LANG=en_US.UTF-8 2024-09-17T18:20:00.924 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:00.924 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-17T18:20:00.924 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:00.924 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG=en_US.UTF-8 2024-09-17T18:20:00.924 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup config 2024-09-17T18:20:00.924 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:00.924 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:00.924 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' dumb = linux ']' 2024-09-17T18:20:00.924 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.924 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-17T18:20:00.924 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.924 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/less.sh 2024-09-17T18:20:00.925 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-17T18:20:00.925 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.925 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-17T18:20:00.925 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.925 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/which2.sh 2024-09-17T18:20:00.925 INFO:tasks.workunit.client.0.smithi173.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-17T18:20:00.925 INFO:tasks.workunit.client.0.smithi173.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-09-17T18:20:00.925 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-17T18:20:00.925 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -f /etc/bashrc ']' 2024-09-17T18:20:00.925 INFO:tasks.workunit.client.0.smithi173.stderr:+ . /etc/bashrc 2024-09-17T18:20:00.925 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -z '' ']' 2024-09-17T18:20:00.925 INFO:tasks.workunit.client.0.smithi173.stderr:++ BASHRCSOURCED=Y 2024-09-17T18:20:00.925 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.926 INFO:tasks.workunit.client.0.smithi173.stderr:++ shopt -q login_shell 2024-09-17T18:20:00.926 INFO:tasks.workunit.client.0.smithi173.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-17T18:20:00.926 INFO:tasks.workunit.client.0.smithi173.stderr:++++ export LANG=en_US.UTF-8 2024-09-17T18:20:00.926 INFO:tasks.workunit.client.0.smithi173.stderr:++++ LANG=en_US.UTF-8 2024-09-17T18:20:00.926 INFO:tasks.workunit.client.0.smithi173.stderr:+ sed 's/^/ api_stat: /' 2024-09-17T18:20:00.926 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:00.926 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-17T18:20:00.926 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:00.926 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG=en_US.UTF-8 2024-09-17T18:20:00.926 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup config 2024-09-17T18:20:00.926 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:00.926 INFO:tasks.workunit.client.0.smithi173.stderr:+++++ readlink /proc/35151/exe 2024-09-17T18:20:00.926 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:00.926 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' dumb = linux ']' 2024-09-17T18:20:00.927 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.927 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-17T18:20:00.927 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.927 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/less.sh 2024-09-17T18:20:00.927 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-17T18:20:00.927 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.927 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-17T18:20:00.927 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.927 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/which2.sh 2024-09-17T18:20:00.927 INFO:tasks.workunit.client.0.smithi173.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-17T18:20:00.927 INFO:tasks.workunit.client.0.smithi173.stderr:+++ umask 2024-09-17T18:20:00.927 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 0022 -eq 0 ']' 2024-09-17T18:20:00.927 INFO:tasks.workunit.client.0.smithi173.stderr:++ SHELL=/bin/bash 2024-09-17T18:20:00.927 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '{print $1}' 2024-09-17T18:20:00.928 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.928 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-17T18:20:00.928 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.928 INFO:tasks.workunit.client.0.smithi173.stderr:++ echo api_service 2024-09-17T18:20:00.928 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-17T18:20:00.928 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-17T18:20:00.928 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.928 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-17T18:20:00.928 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.928 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-17T18:20:00.928 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-17T18:20:00.928 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:00.929 INFO:tasks.workunit.client.0.smithi173.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:00.929 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.929 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-17T18:20:00.929 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.929 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-17T18:20:00.929 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.929 INFO:tasks.workunit.client.0.smithi173.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-17T18:20:00.929 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-17T18:20:00.929 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-17T18:20:00.929 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-17T18:20:00.929 INFO:tasks.workunit.client.0.smithi173.stderr:++++ export LANG=en_US.UTF-8 2024-09-17T18:20:00.929 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.929 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-17T18:20:00.929 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.930 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-17T18:20:00.930 INFO:tasks.workunit.client.0.smithi173.stderr:++++ LANG=en_US.UTF-8 2024-09-17T18:20:00.930 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-17T18:20:00.930 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.930 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:00.930 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-17T18:20:00.930 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:00.930 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG=en_US.UTF-8 2024-09-17T18:20:00.930 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup config 2024-09-17T18:20:00.930 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:00.930 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:00.930 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-17T18:20:00.930 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' dumb = linux ']' 2024-09-17T18:20:00.930 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-17T18:20:00.931 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-17T18:20:00.931 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.931 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-17T18:20:00.931 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.931 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-17T18:20:00.931 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.931 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-17T18:20:00.931 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.931 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-17T18:20:00.931 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.931 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/less.sh 2024-09-17T18:20:00.931 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-17T18:20:00.931 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.932 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-17T18:20:00.932 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.932 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/which2.sh 2024-09-17T18:20:00.932 INFO:tasks.workunit.client.0.smithi173.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-17T18:20:00.932 INFO:tasks.workunit.client.0.smithi173.stderr:+++++ readlink /proc/35190/exe 2024-09-17T18:20:00.932 INFO:tasks.workunit.client.0.smithi173.stderr:++++ basename /usr/bin/bash 2024-09-17T18:20:00.932 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'grep=grep --color=auto' 2024-09-17T18:20:00.932 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-17T18:20:00.932 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-17T18:20:00.932 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.932 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-17T18:20:00.932 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.932 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorls.sh 2024-09-17T18:20:00.933 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' '!' -t 0 ']' 2024-09-17T18:20:00.933 INFO:tasks.workunit.client.0.smithi173.stderr:+++ return 2024-09-17T18:20:00.933 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.933 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-17T18:20:00.933 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.933 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-17T18:20:00.933 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export S_COLORS=auto 2024-09-17T18:20:00.933 INFO:tasks.workunit.client.0.smithi173.stderr:+++ S_COLORS=auto 2024-09-17T18:20:00.933 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-17T18:20:00.933 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.933 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-17T18:20:00.933 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-17T18:20:00.933 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.934 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-17T18:20:00.934 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-17T18:20:00.934 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.934 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-17T18:20:00.934 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.934 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-17T18:20:00.934 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.934 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-17T18:20:00.934 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.934 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/gawk.sh 2024-09-17T18:20:00.934 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-17T18:20:00.934 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.934 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.934 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-17T18:20:00.935 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.935 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/lang.sh 2024-09-17T18:20:00.935 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup 2024-09-17T18:20:00.935 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_declare='declare -f' 2024-09-17T18:20:00.935 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_opt=-f 2024-09-17T18:20:00.935 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export which_declare 2024-09-17T18:20:00.935 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export -f which 2024-09-17T18:20:00.935 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset i 2024-09-17T18:20:00.935 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset -f pathmunge 2024-09-17T18:20:00.935 INFO:tasks.workunit.client.0.smithi173.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-17T18:20:00.935 INFO:tasks.workunit.client.0.smithi173.stderr:+ export PATH 2024-09-17T18:20:00.935 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-17T18:20:00.935 INFO:tasks.workunit.client.0.smithi173.stderr:+ unset rc 2024-09-17T18:20:00.936 INFO:tasks.workunit.client.0.smithi173.stderr:+ ff=api_service 2024-09-17T18:20:00.936 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35387 2024-09-17T18:20:00.936 INFO:tasks.workunit.client.0.smithi173.stderr:+ echo 'test api_service on pid 35387' 2024-09-17T18:20:00.936 INFO:tasks.workunit.client.0.smithi173.stdout:test api_service on pid 35387 2024-09-17T18:20:00.958 INFO:tasks.workunit.client.0.smithi173.stderr:+ pids[$f]=35387 2024-09-17T18:20:00.958 INFO:tasks.workunit.client.0.smithi173.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-17T18:20:00.958 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1 -eq 1 ']' 2024-09-17T18:20:00.958 INFO:tasks.workunit.client.0.smithi173.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-09-17T18:20:00.958 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/locale 2024-09-17T18:20:00.958 INFO:tasks.workunit.client.0.smithi173.stderr:++++ basename /usr/bin/bash 2024-09-17T18:20:00.958 INFO:tasks.workunit.client.0.smithi173.stderr:+++++ readlink /proc/35089/exe 2024-09-17T18:20:00.958 INFO:tasks.workunit.client.0.smithi173.stderr:++ printf %25s api_service_pp 2024-09-17T18:20:00.958 INFO:tasks.workunit.client.0.smithi173.stderr:+ r=' api_service_pp' 2024-09-17T18:20:00.959 INFO:tasks.workunit.client.0.smithi173.stderr:+ sed 's/^/ api_snapshots: /' 2024-09-17T18:20:00.959 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_declare='declare -f' 2024-09-17T18:20:00.959 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_opt=-f 2024-09-17T18:20:00.959 INFO:tasks.workunit.client.0.smithi173.stderr:++++ basename /usr/bin/bash 2024-09-17T18:20:00.959 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export which_declare 2024-09-17T18:20:00.959 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export -f which 2024-09-17T18:20:00.959 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset i 2024-09-17T18:20:00.959 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset -f pathmunge 2024-09-17T18:20:00.959 INFO:tasks.workunit.client.0.smithi173.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-17T18:20:00.959 INFO:tasks.workunit.client.0.smithi173.stderr:+ export PATH 2024-09-17T18:20:00.959 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-17T18:20:00.959 INFO:tasks.workunit.client.0.smithi173.stderr:+ unset rc 2024-09-17T18:20:00.959 INFO:tasks.workunit.client.0.smithi173.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-09-17T18:20:00.960 INFO:tasks.workunit.client.0.smithi173.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2024-09-17T18:20:00.960 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_declare='declare -f' 2024-09-17T18:20:00.960 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_opt=-f 2024-09-17T18:20:00.960 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export which_declare 2024-09-17T18:20:00.960 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export -f which 2024-09-17T18:20:00.960 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset i 2024-09-17T18:20:00.960 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset -f pathmunge 2024-09-17T18:20:00.960 INFO:tasks.workunit.client.0.smithi173.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-17T18:20:00.960 INFO:tasks.workunit.client.0.smithi173.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2024-09-17T18:20:00.960 INFO:tasks.workunit.client.0.smithi173.stderr:+ export PATH 2024-09-17T18:20:00.960 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-17T18:20:00.960 INFO:tasks.workunit.client.0.smithi173.stderr:+ unset rc 2024-09-17T18:20:00.960 INFO:tasks.workunit.client.0.smithi173.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-09-17T18:20:00.960 INFO:tasks.workunit.client.0.smithi173.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-09-17T18:20:00.961 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'grep=grep --color=auto' 2024-09-17T18:20:00.961 INFO:tasks.workunit.client.0.smithi173.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-09-17T18:20:00.961 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-17T18:20:00.961 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-17T18:20:00.961 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.961 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-17T18:20:00.961 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.961 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorls.sh 2024-09-17T18:20:00.961 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' '!' -t 0 ']' 2024-09-17T18:20:00.961 INFO:tasks.workunit.client.0.smithi173.stderr:+++ return 2024-09-17T18:20:00.961 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.961 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-17T18:20:00.961 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.962 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-17T18:20:00.962 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export S_COLORS=auto 2024-09-17T18:20:00.962 INFO:tasks.workunit.client.0.smithi173.stderr:+++ S_COLORS=auto 2024-09-17T18:20:00.962 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.962 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-17T18:20:00.962 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-17T18:20:00.962 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.962 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-17T18:20:00.962 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-17T18:20:00.962 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-17T18:20:00.962 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.962 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-17T18:20:00.962 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.962 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-17T18:20:00.962 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-17T18:20:00.963 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.963 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.963 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -f /etc/bashrc ']' 2024-09-17T18:20:00.963 INFO:tasks.workunit.client.0.smithi173.stderr:+ . /etc/bashrc 2024-09-17T18:20:00.963 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -z '' ']' 2024-09-17T18:20:00.963 INFO:tasks.workunit.client.0.smithi173.stderr:++ BASHRCSOURCED=Y 2024-09-17T18:20:00.963 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.963 INFO:tasks.workunit.client.0.smithi173.stderr:++ shopt -q login_shell 2024-09-17T18:20:00.963 INFO:tasks.workunit.client.0.smithi173.stderr:+++ umask 2024-09-17T18:20:00.963 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 0022 -eq 0 ']' 2024-09-17T18:20:00.963 INFO:tasks.workunit.client.0.smithi173.stderr:++ SHELL=/bin/bash 2024-09-17T18:20:00.963 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.963 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-17T18:20:00.963 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.964 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-17T18:20:00.964 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-17T18:20:00.964 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.964 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-17T18:20:00.964 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.964 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-17T18:20:00.964 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:00.964 INFO:tasks.workunit.client.0.smithi173.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:00.964 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.964 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-17T18:20:00.964 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.964 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-17T18:20:00.964 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.964 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-17T18:20:00.964 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-17T18:20:00.965 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-17T18:20:00.965 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-17T18:20:00.965 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-17T18:20:00.965 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.965 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-17T18:20:00.965 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.965 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-17T18:20:00.965 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-17T18:20:00.965 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.965 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-17T18:20:00.965 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.965 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-17T18:20:00.965 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.965 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-17T18:20:00.965 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-17T18:20:00.966 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.966 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-17T18:20:00.966 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.966 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/gawk.sh 2024-09-17T18:20:00.966 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.966 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-17T18:20:00.966 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.966 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/lang.sh 2024-09-17T18:20:00.966 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup 2024-09-17T18:20:00.966 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-17T18:20:00.966 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-17T18:20:00.966 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-17T18:20:00.966 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.966 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-17T18:20:00.967 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.967 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-17T18:20:00.967 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-17T18:20:00.967 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.967 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-17T18:20:00.967 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.967 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/gawk.sh 2024-09-17T18:20:00.967 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.967 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-17T18:20:00.967 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.967 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/lang.sh 2024-09-17T18:20:00.967 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup 2024-09-17T18:20:00.967 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/locale 2024-09-17T18:20:00.967 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'grep=grep --color=auto' 2024-09-17T18:20:00.968 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-17T18:20:00.968 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-17T18:20:00.968 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/locale 2024-09-17T18:20:00.968 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.968 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-17T18:20:00.968 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.968 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorls.sh 2024-09-17T18:20:00.968 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' '!' -t 0 ']' 2024-09-17T18:20:00.968 INFO:tasks.workunit.client.0.smithi173.stderr:+++ return 2024-09-17T18:20:00.968 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.968 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-17T18:20:00.968 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.969 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-17T18:20:00.969 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export S_COLORS=auto 2024-09-17T18:20:00.969 INFO:tasks.workunit.client.0.smithi173.stderr:+++ S_COLORS=auto 2024-09-17T18:20:00.969 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.969 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-17T18:20:00.969 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.969 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-17T18:20:00.969 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-17T18:20:00.969 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.969 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-17T18:20:00.969 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-17T18:20:00.969 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.969 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-17T18:20:00.969 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.970 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-17T18:20:00.970 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-17T18:20:00.970 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.970 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-17T18:20:00.970 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.970 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/gawk.sh 2024-09-17T18:20:00.970 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.970 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-17T18:20:00.970 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.970 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/lang.sh 2024-09-17T18:20:00.970 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup 2024-09-17T18:20:00.970 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:00.970 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-17T18:20:00.970 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:00.971 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:00.971 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-17T18:20:00.971 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-17T18:20:00.971 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:00.971 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-17T18:20:00.971 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-17T18:20:00.971 INFO:tasks.workunit.client.0.smithi173.stderr:++ echo api_service_pp 2024-09-17T18:20:00.971 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-17T18:20:00.971 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-17T18:20:00.971 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/locale 2024-09-17T18:20:00.971 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-17T18:20:00.971 INFO:tasks.workunit.client.0.smithi173.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2024-09-17T18:20:00.971 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:00.971 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-17T18:20:00.972 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:00.972 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-17T18:20:00.972 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-17T18:20:00.972 INFO:tasks.workunit.client.0.smithi173.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-17T18:20:00.972 INFO:tasks.workunit.client.0.smithi173.stderr:++++ export LANG=en_US.UTF-8 2024-09-17T18:20:00.972 INFO:tasks.workunit.client.0.smithi173.stderr:++++ LANG=en_US.UTF-8 2024-09-17T18:20:00.972 INFO:tasks.workunit.client.0.smithi173.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-17T18:20:00.972 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:00.972 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-17T18:20:00.972 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:00.972 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG=en_US.UTF-8 2024-09-17T18:20:00.972 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup config 2024-09-17T18:20:00.972 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:00.972 INFO:tasks.workunit.client.0.smithi173.stderr:++++ export LANG=en_US.UTF-8 2024-09-17T18:20:00.972 INFO:tasks.workunit.client.0.smithi173.stderr:++++ LANG=en_US.UTF-8 2024-09-17T18:20:00.973 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:00.973 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' dumb = linux ']' 2024-09-17T18:20:00.973 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.973 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-17T18:20:00.973 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.973 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/less.sh 2024-09-17T18:20:00.973 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:00.973 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-17T18:20:00.973 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.973 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-17T18:20:00.973 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.973 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/which2.sh 2024-09-17T18:20:00.973 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-17T18:20:00.973 INFO:tasks.workunit.client.0.smithi173.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-17T18:20:00.974 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:00.974 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG=en_US.UTF-8 2024-09-17T18:20:00.974 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup config 2024-09-17T18:20:00.974 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:00.974 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '{print $1}' 2024-09-17T18:20:00.974 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:00.974 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' dumb = linux ']' 2024-09-17T18:20:00.974 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.974 INFO:tasks.workunit.client.0.smithi173.stderr:+ sed 's/^/ api_tier_pp: /' 2024-09-17T18:20:00.974 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-17T18:20:00.974 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.974 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/less.sh 2024-09-17T18:20:00.974 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-17T18:20:00.974 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.974 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-17T18:20:00.975 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.975 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/which2.sh 2024-09-17T18:20:00.975 INFO:tasks.workunit.client.0.smithi173.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-17T18:20:00.975 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-17T18:20:00.975 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-17T18:20:00.975 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-17T18:20:00.975 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.975 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-17T18:20:00.975 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.975 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-17T18:20:00.975 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-17T18:20:00.975 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:00.975 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:00.975 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-17T18:20:00.976 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:00.976 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-17T18:20:00.976 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-17T18:20:00.976 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-17T18:20:00.976 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-17T18:20:00.976 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-17T18:20:00.976 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.976 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-17T18:20:00.976 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.976 INFO:tasks.workunit.client.0.smithi173.stderr:+ ff=api_service_pp 2024-09-17T18:20:00.976 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-17T18:20:00.976 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-17T18:20:00.976 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.976 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-17T18:20:00.977 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.977 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/gawk.sh 2024-09-17T18:20:00.977 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35439 2024-09-17T18:20:00.977 INFO:tasks.workunit.client.0.smithi173.stderr:+ echo 'test api_service_pp on pid 35439' 2024-09-17T18:20:00.977 INFO:tasks.workunit.client.0.smithi173.stdout:test api_service_pp on pid 35439 2024-09-17T18:20:00.984 INFO:tasks.workunit.client.0.smithi173.stderr:+ pids[$f]=35439 2024-09-17T18:20:00.984 INFO:tasks.workunit.client.0.smithi173.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-17T18:20:00.984 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1 -eq 1 ']' 2024-09-17T18:20:00.984 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.984 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-17T18:20:00.984 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.984 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/lang.sh 2024-09-17T18:20:00.984 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup 2024-09-17T18:20:00.984 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-17T18:20:00.984 INFO:tasks.workunit.client.0.smithi173.stderr:++ printf %25s api_c_write_operations 2024-09-17T18:20:00.984 INFO:tasks.workunit.client.0.smithi173.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-17T18:20:00.984 INFO:tasks.workunit.client.0.smithi173.stderr:+ r=' api_c_write_operations' 2024-09-17T18:20:00.984 INFO:tasks.workunit.client.0.smithi173.stderr:++++ export LANG=en_US.UTF-8 2024-09-17T18:20:00.985 INFO:tasks.workunit.client.0.smithi173.stderr:++++ LANG=en_US.UTF-8 2024-09-17T18:20:00.985 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:00.985 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-17T18:20:00.985 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:00.985 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG=en_US.UTF-8 2024-09-17T18:20:00.985 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup config 2024-09-17T18:20:00.985 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:00.985 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:00.985 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-17T18:20:00.985 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' dumb = linux ']' 2024-09-17T18:20:00.985 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.985 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-17T18:20:00.985 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.986 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/less.sh 2024-09-17T18:20:00.986 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-17T18:20:00.986 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.986 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-17T18:20:00.986 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.986 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/which2.sh 2024-09-17T18:20:00.986 INFO:tasks.workunit.client.0.smithi173.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-17T18:20:00.986 INFO:tasks.workunit.client.0.smithi173.stderr:+++++ readlink /proc/35277/exe 2024-09-17T18:20:00.986 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '{print $1}' 2024-09-17T18:20:00.986 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/locale 2024-09-17T18:20:00.986 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:00.986 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-17T18:20:00.986 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:00.986 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-17T18:20:00.987 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-17T18:20:00.987 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-17T18:20:00.987 INFO:tasks.workunit.client.0.smithi173.stderr:++++ basename /usr/bin/bash 2024-09-17T18:20:00.987 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_declare='declare -f' 2024-09-17T18:20:00.987 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_opt=-f 2024-09-17T18:20:00.987 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export which_declare 2024-09-17T18:20:00.987 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export -f which 2024-09-17T18:20:00.987 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset i 2024-09-17T18:20:00.987 INFO:tasks.workunit.client.0.smithi173.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-17T18:20:00.987 INFO:tasks.workunit.client.0.smithi173.stderr:++++ export LANG=en_US.UTF-8 2024-09-17T18:20:00.987 INFO:tasks.workunit.client.0.smithi173.stderr:++++ LANG=en_US.UTF-8 2024-09-17T18:20:00.987 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:00.987 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-17T18:20:00.987 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:00.987 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset -f pathmunge 2024-09-17T18:20:00.988 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG=en_US.UTF-8 2024-09-17T18:20:00.988 INFO:tasks.workunit.client.0.smithi173.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-17T18:20:00.988 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup config 2024-09-17T18:20:00.988 INFO:tasks.workunit.client.0.smithi173.stderr:+ export PATH 2024-09-17T18:20:00.988 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:00.988 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-17T18:20:00.988 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:00.988 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' dumb = linux ']' 2024-09-17T18:20:00.988 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.988 INFO:tasks.workunit.client.0.smithi173.stderr:+ unset rc 2024-09-17T18:20:00.988 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-17T18:20:00.988 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.988 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/less.sh 2024-09-17T18:20:00.989 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-17T18:20:00.989 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:00.989 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-17T18:20:00.989 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:00.989 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/which2.sh 2024-09-17T18:20:00.989 INFO:tasks.workunit.client.0.smithi173.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-17T18:20:00.989 INFO:tasks.workunit.client.0.smithi173.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-09-17T18:20:00.989 INFO:tasks.workunit.client.0.smithi173.stderr:++ echo api_c_write_operations 2024-09-17T18:20:00.989 INFO:tasks.workunit.client.0.smithi173.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-09-17T18:20:00.989 INFO:tasks.workunit.client.0.smithi173.stderr:+ ff=api_c_write_operations 2024-09-17T18:20:00.989 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35464 2024-09-17T18:20:00.990 INFO:tasks.workunit.client.0.smithi173.stdout:test api_c_write_operations on pid 35464 2024-09-17T18:20:01.001 INFO:tasks.workunit.client.0.smithi173.stderr:+ echo 'test api_c_write_operations on pid 35464' 2024-09-17T18:20:01.001 INFO:tasks.workunit.client.0.smithi173.stderr:+ pids[$f]=35464 2024-09-17T18:20:01.001 INFO:tasks.workunit.client.0.smithi173.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-17T18:20:01.001 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1 -eq 1 ']' 2024-09-17T18:20:01.002 INFO:tasks.workunit.client.0.smithi173.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-17T18:20:01.002 INFO:tasks.workunit.client.0.smithi173.stderr:++++ export LANG=en_US.UTF-8 2024-09-17T18:20:01.002 INFO:tasks.workunit.client.0.smithi173.stderr:++++ LANG=en_US.UTF-8 2024-09-17T18:20:01.002 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:01.002 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-17T18:20:01.002 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:01.002 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG=en_US.UTF-8 2024-09-17T18:20:01.002 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup config 2024-09-17T18:20:01.002 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:01.002 INFO:tasks.workunit.client.0.smithi173.stderr:++ printf %25s api_c_read_operations 2024-09-17T18:20:01.002 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:01.002 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' dumb = linux ']' 2024-09-17T18:20:01.002 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.002 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-17T18:20:01.003 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.003 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/less.sh 2024-09-17T18:20:01.003 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-17T18:20:01.003 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.003 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-17T18:20:01.003 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.003 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/which2.sh 2024-09-17T18:20:01.003 INFO:tasks.workunit.client.0.smithi173.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-17T18:20:01.003 INFO:tasks.workunit.client.0.smithi173.stderr:+ r=' api_c_read_operations' 2024-09-17T18:20:01.003 INFO:tasks.workunit.client.0.smithi173.stderr:++ echo api_c_read_operations 2024-09-17T18:20:01.003 INFO:tasks.workunit.client.0.smithi173.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-09-17T18:20:01.003 INFO:tasks.workunit.client.0.smithi173.stderr:+++++ readlink /proc/35340/exe 2024-09-17T18:20:01.003 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-17T18:20:01.003 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-17T18:20:01.004 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-17T18:20:01.004 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.004 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-17T18:20:01.004 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.004 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-17T18:20:01.004 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -f /etc/bashrc ']' 2024-09-17T18:20:01.004 INFO:tasks.workunit.client.0.smithi173.stderr:+ . /etc/bashrc 2024-09-17T18:20:01.004 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-17T18:20:01.004 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.004 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-17T18:20:01.004 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.004 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/gawk.sh 2024-09-17T18:20:01.004 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -z '' ']' 2024-09-17T18:20:01.005 INFO:tasks.workunit.client.0.smithi173.stderr:++ BASHRCSOURCED=Y 2024-09-17T18:20:01.005 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.005 INFO:tasks.workunit.client.0.smithi173.stderr:++ shopt -q login_shell 2024-09-17T18:20:01.005 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.005 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-17T18:20:01.005 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.005 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/lang.sh 2024-09-17T18:20:01.005 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup 2024-09-17T18:20:01.005 INFO:tasks.workunit.client.0.smithi173.stderr:++++ basename /usr/bin/bash 2024-09-17T18:20:01.005 INFO:tasks.workunit.client.0.smithi173.stderr:+++++ readlink /proc/35264/exe 2024-09-17T18:20:01.005 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_declare='declare -f' 2024-09-17T18:20:01.005 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_opt=-f 2024-09-17T18:20:01.005 INFO:tasks.workunit.client.0.smithi173.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2024-09-17T18:20:01.005 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export which_declare 2024-09-17T18:20:01.006 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export -f which 2024-09-17T18:20:01.006 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset i 2024-09-17T18:20:01.006 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset -f pathmunge 2024-09-17T18:20:01.006 INFO:tasks.workunit.client.0.smithi173.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-17T18:20:01.006 INFO:tasks.workunit.client.0.smithi173.stderr:+ export PATH 2024-09-17T18:20:01.006 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-17T18:20:01.006 INFO:tasks.workunit.client.0.smithi173.stderr:+ unset rc 2024-09-17T18:20:01.006 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/locale 2024-09-17T18:20:01.006 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:01.006 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-17T18:20:01.006 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:01.006 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-17T18:20:01.006 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-17T18:20:01.006 INFO:tasks.workunit.client.0.smithi173.stderr:+++ umask 2024-09-17T18:20:01.007 INFO:tasks.workunit.client.0.smithi173.stderr:++++ basename /usr/bin/bash 2024-09-17T18:20:01.007 INFO:tasks.workunit.client.0.smithi173.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-09-17T18:20:01.007 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 0022 -eq 0 ']' 2024-09-17T18:20:01.007 INFO:tasks.workunit.client.0.smithi173.stderr:++ SHELL=/bin/bash 2024-09-17T18:20:01.007 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.007 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-17T18:20:01.007 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.007 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-17T18:20:01.007 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-17T18:20:01.007 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.007 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-17T18:20:01.007 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.007 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-17T18:20:01.007 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:01.007 INFO:tasks.workunit.client.0.smithi173.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:01.008 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_declare='declare -f' 2024-09-17T18:20:01.008 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_opt=-f 2024-09-17T18:20:01.008 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.008 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export which_declare 2024-09-17T18:20:01.008 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export -f which 2024-09-17T18:20:01.008 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset i 2024-09-17T18:20:01.008 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset -f pathmunge 2024-09-17T18:20:01.008 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-17T18:20:01.008 INFO:tasks.workunit.client.0.smithi173.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-17T18:20:01.008 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.008 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-17T18:20:01.008 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.008 INFO:tasks.workunit.client.0.smithi173.stderr:+ export PATH 2024-09-17T18:20:01.008 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-17T18:20:01.009 INFO:tasks.workunit.client.0.smithi173.stderr:+ unset rc 2024-09-17T18:20:01.009 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-17T18:20:01.009 INFO:tasks.workunit.client.0.smithi173.stderr:+ sed 's/^/ api_watch_notify: /' 2024-09-17T18:20:01.009 INFO:tasks.workunit.client.0.smithi173.stderr:+++++ readlink /proc/35243/exe 2024-09-17T18:20:01.009 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '{print $1}' 2024-09-17T18:20:01.009 INFO:tasks.workunit.client.0.smithi173.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-09-17T18:20:01.009 INFO:tasks.workunit.client.0.smithi173.stderr:++++ basename /usr/bin/bash 2024-09-17T18:20:01.009 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_declare='declare -f' 2024-09-17T18:20:01.009 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_opt=-f 2024-09-17T18:20:01.009 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export which_declare 2024-09-17T18:20:01.009 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export -f which 2024-09-17T18:20:01.009 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset i 2024-09-17T18:20:01.009 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset -f pathmunge 2024-09-17T18:20:01.009 INFO:tasks.workunit.client.0.smithi173.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-17T18:20:01.010 INFO:tasks.workunit.client.0.smithi173.stderr:+ export PATH 2024-09-17T18:20:01.010 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-17T18:20:01.010 INFO:tasks.workunit.client.0.smithi173.stderr:+ unset rc 2024-09-17T18:20:01.010 INFO:tasks.workunit.client.0.smithi173.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-17T18:20:01.010 INFO:tasks.workunit.client.0.smithi173.stderr:++++ export LANG=en_US.UTF-8 2024-09-17T18:20:01.010 INFO:tasks.workunit.client.0.smithi173.stderr:++++ LANG=en_US.UTF-8 2024-09-17T18:20:01.010 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:01.010 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-17T18:20:01.010 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:01.010 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG=en_US.UTF-8 2024-09-17T18:20:01.010 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup config 2024-09-17T18:20:01.010 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:01.010 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:01.010 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' dumb = linux ']' 2024-09-17T18:20:01.011 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.011 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-17T18:20:01.011 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.011 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/less.sh 2024-09-17T18:20:01.011 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-17T18:20:01.011 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.011 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-17T18:20:01.011 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.011 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/which2.sh 2024-09-17T18:20:01.011 INFO:tasks.workunit.client.0.smithi173.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-17T18:20:01.011 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -f /etc/bashrc ']' 2024-09-17T18:20:01.011 INFO:tasks.workunit.client.0.smithi173.stderr:+ . /etc/bashrc 2024-09-17T18:20:01.011 INFO:tasks.workunit.client.0.smithi173.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2024-09-17T18:20:01.011 INFO:tasks.workunit.client.0.smithi173.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2024-09-17T18:20:01.011 INFO:tasks.workunit.client.0.smithi173.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-09-17T18:20:01.012 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -z '' ']' 2024-09-17T18:20:01.012 INFO:tasks.workunit.client.0.smithi173.stderr:++ BASHRCSOURCED=Y 2024-09-17T18:20:01.012 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.012 INFO:tasks.workunit.client.0.smithi173.stderr:++ shopt -q login_shell 2024-09-17T18:20:01.012 INFO:tasks.workunit.client.0.smithi173.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-09-17T18:20:01.012 INFO:tasks.workunit.client.0.smithi173.stderr:+ ff=api_c_read_operations 2024-09-17T18:20:01.012 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35518 2024-09-17T18:20:01.012 INFO:tasks.workunit.client.0.smithi173.stderr:+ echo 'test api_c_read_operations on pid 35518' 2024-09-17T18:20:01.012 INFO:tasks.workunit.client.0.smithi173.stdout:test api_c_read_operations on pid 35518 2024-09-17T18:20:01.019 INFO:tasks.workunit.client.0.smithi173.stderr:+ pids[$f]=35518 2024-09-17T18:20:01.019 INFO:tasks.workunit.client.0.smithi173.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-17T18:20:01.019 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1 -eq 1 ']' 2024-09-17T18:20:01.020 INFO:tasks.workunit.client.0.smithi173.stderr:++ printf %25s list_parallel 2024-09-17T18:20:01.020 INFO:tasks.workunit.client.0.smithi173.stderr:+ r=' list_parallel' 2024-09-17T18:20:01.020 INFO:tasks.workunit.client.0.smithi173.stderr:+++ umask 2024-09-17T18:20:01.020 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 0022 -eq 0 ']' 2024-09-17T18:20:01.020 INFO:tasks.workunit.client.0.smithi173.stderr:++ SHELL=/bin/bash 2024-09-17T18:20:01.020 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.020 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-17T18:20:01.020 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.020 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-17T18:20:01.020 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-17T18:20:01.020 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.020 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-17T18:20:01.020 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.020 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-17T18:20:01.021 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:01.021 INFO:tasks.workunit.client.0.smithi173.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:01.021 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.021 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-17T18:20:01.021 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.021 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-17T18:20:01.021 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.021 INFO:tasks.workunit.client.0.smithi173.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-09-17T18:20:01.021 INFO:tasks.workunit.client.0.smithi173.stderr:+++++ readlink /proc/35311/exe 2024-09-17T18:20:01.021 INFO:tasks.workunit.client.0.smithi173.stderr:++++ basename /usr/bin/bash 2024-09-17T18:20:01.021 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -f /etc/bashrc ']' 2024-09-17T18:20:01.024 INFO:tasks.workunit.client.0.smithi173.stderr:+ . /etc/bashrc 2024-09-17T18:20:01.024 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -z '' ']' 2024-09-17T18:20:01.024 INFO:tasks.workunit.client.0.smithi173.stderr:++ BASHRCSOURCED=Y 2024-09-17T18:20:01.024 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.024 INFO:tasks.workunit.client.0.smithi173.stderr:++ shopt -q login_shell 2024-09-17T18:20:01.024 INFO:tasks.workunit.client.0.smithi173.stderr:++ echo list_parallel 2024-09-17T18:20:01.024 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_declare='declare -f' 2024-09-17T18:20:01.024 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_opt=-f 2024-09-17T18:20:01.024 INFO:tasks.workunit.client.0.smithi173.stderr:+++ umask 2024-09-17T18:20:01.024 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export which_declare 2024-09-17T18:20:01.025 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export -f which 2024-09-17T18:20:01.025 INFO:tasks.workunit.client.0.smithi173.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2024-09-17T18:20:01.025 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset i 2024-09-17T18:20:01.025 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset -f pathmunge 2024-09-17T18:20:01.025 INFO:tasks.workunit.client.0.smithi173.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-17T18:20:01.025 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 0022 -eq 0 ']' 2024-09-17T18:20:01.025 INFO:tasks.workunit.client.0.smithi173.stderr:++ SHELL=/bin/bash 2024-09-17T18:20:01.025 INFO:tasks.workunit.client.0.smithi173.stderr:+ export PATH 2024-09-17T18:20:01.025 INFO:tasks.workunit.client.0.smithi173.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-09-17T18:20:01.025 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-17T18:20:01.025 INFO:tasks.workunit.client.0.smithi173.stderr:+ unset rc 2024-09-17T18:20:01.025 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '{print $1}' 2024-09-17T18:20:01.025 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.025 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-17T18:20:01.025 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.026 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-17T18:20:01.026 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-17T18:20:01.026 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.026 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-17T18:20:01.026 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.026 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-17T18:20:01.026 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:01.026 INFO:tasks.workunit.client.0.smithi173.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:01.026 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.026 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-17T18:20:01.026 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.026 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-17T18:20:01.026 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.026 INFO:tasks.workunit.client.0.smithi173.stderr:+++++ readlink /proc/35387/exe 2024-09-17T18:20:01.027 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'grep=grep --color=auto' 2024-09-17T18:20:01.027 INFO:tasks.workunit.client.0.smithi173.stderr:++++ basename /usr/bin/bash 2024-09-17T18:20:01.027 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-17T18:20:01.027 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-17T18:20:01.027 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.027 INFO:tasks.workunit.client.0.smithi173.stderr:+ sed 's/^/ api_stat_pp: /' 2024-09-17T18:20:01.027 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-17T18:20:01.027 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.027 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorls.sh 2024-09-17T18:20:01.027 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' '!' -t 0 ']' 2024-09-17T18:20:01.027 INFO:tasks.workunit.client.0.smithi173.stderr:+++ return 2024-09-17T18:20:01.027 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.027 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-17T18:20:01.027 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.028 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-17T18:20:01.028 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export S_COLORS=auto 2024-09-17T18:20:01.028 INFO:tasks.workunit.client.0.smithi173.stderr:+++ S_COLORS=auto 2024-09-17T18:20:01.028 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.028 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-17T18:20:01.028 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.028 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-17T18:20:01.028 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.028 INFO:tasks.workunit.client.0.smithi173.stderr:+ ff=list_parallel 2024-09-17T18:20:01.028 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35552 2024-09-17T18:20:01.028 INFO:tasks.workunit.client.0.smithi173.stdout:test list_parallel on pid 35552 2024-09-17T18:20:01.034 INFO:tasks.workunit.client.0.smithi173.stderr:+ echo 'test list_parallel on pid 35552' 2024-09-17T18:20:01.034 INFO:tasks.workunit.client.0.smithi173.stderr:+ pids[$f]=35552 2024-09-17T18:20:01.034 INFO:tasks.workunit.client.0.smithi173.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-17T18:20:01.035 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1 -eq 1 ']' 2024-09-17T18:20:01.035 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'grep=grep --color=auto' 2024-09-17T18:20:01.035 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-17T18:20:01.035 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-17T18:20:01.035 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.035 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-17T18:20:01.035 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.035 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorls.sh 2024-09-17T18:20:01.035 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' '!' -t 0 ']' 2024-09-17T18:20:01.035 INFO:tasks.workunit.client.0.smithi173.stderr:+++ return 2024-09-17T18:20:01.035 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.035 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-17T18:20:01.035 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.035 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-17T18:20:01.036 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export S_COLORS=auto 2024-09-17T18:20:01.036 INFO:tasks.workunit.client.0.smithi173.stderr:+++ S_COLORS=auto 2024-09-17T18:20:01.036 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.036 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-17T18:20:01.036 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.036 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-17T18:20:01.036 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.036 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_declare='declare -f' 2024-09-17T18:20:01.036 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_opt=-f 2024-09-17T18:20:01.036 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export which_declare 2024-09-17T18:20:01.036 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export -f which 2024-09-17T18:20:01.036 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset i 2024-09-17T18:20:01.036 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset -f pathmunge 2024-09-17T18:20:01.037 INFO:tasks.workunit.client.0.smithi173.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-17T18:20:01.037 INFO:tasks.workunit.client.0.smithi173.stderr:+ export PATH 2024-09-17T18:20:01.037 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-17T18:20:01.037 INFO:tasks.workunit.client.0.smithi173.stderr:+ unset rc 2024-09-17T18:20:01.037 INFO:tasks.workunit.client.0.smithi173.stderr:++ printf %25s open_pools_parallel 2024-09-17T18:20:01.037 INFO:tasks.workunit.client.0.smithi173.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2024-09-17T18:20:01.037 INFO:tasks.workunit.client.0.smithi173.stderr:+ r=' open_pools_parallel' 2024-09-17T18:20:01.037 INFO:tasks.workunit.client.0.smithi173.stderr:+ sed 's/^/ api_cmd: /' 2024-09-17T18:20:01.037 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'grep=grep --color=auto' 2024-09-17T18:20:01.037 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-17T18:20:01.037 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-17T18:20:01.037 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.037 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-17T18:20:01.037 INFO:tasks.workunit.client.0.smithi173.stderr:++ echo open_pools_parallel 2024-09-17T18:20:01.037 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.038 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorls.sh 2024-09-17T18:20:01.038 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' '!' -t 0 ']' 2024-09-17T18:20:01.038 INFO:tasks.workunit.client.0.smithi173.stderr:+++ return 2024-09-17T18:20:01.038 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.038 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-17T18:20:01.038 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.038 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-17T18:20:01.038 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export S_COLORS=auto 2024-09-17T18:20:01.038 INFO:tasks.workunit.client.0.smithi173.stderr:+++ S_COLORS=auto 2024-09-17T18:20:01.038 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.038 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-17T18:20:01.038 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.038 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-17T18:20:01.038 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.039 INFO:tasks.workunit.client.0.smithi173.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-09-17T18:20:01.039 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '{print $1}' 2024-09-17T18:20:01.039 INFO:tasks.workunit.client.0.smithi173.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2024-09-17T18:20:01.039 INFO:tasks.workunit.client.0.smithi173.stderr:+ tee ceph_test_rados_api_service.log 2024-09-17T18:20:01.039 INFO:tasks.workunit.client.0.smithi173.stderr:+ sed 's/^/ api_service: /' 2024-09-17T18:20:01.039 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -f /etc/bashrc ']' 2024-09-17T18:20:01.039 INFO:tasks.workunit.client.0.smithi173.stderr:+ . /etc/bashrc 2024-09-17T18:20:01.039 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -z '' ']' 2024-09-17T18:20:01.039 INFO:tasks.workunit.client.0.smithi173.stderr:++ BASHRCSOURCED=Y 2024-09-17T18:20:01.039 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.039 INFO:tasks.workunit.client.0.smithi173.stderr:++ shopt -q login_shell 2024-09-17T18:20:01.039 INFO:tasks.workunit.client.0.smithi173.stderr:+++ umask 2024-09-17T18:20:01.039 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 0022 -eq 0 ']' 2024-09-17T18:20:01.039 INFO:tasks.workunit.client.0.smithi173.stderr:++ SHELL=/bin/bash 2024-09-17T18:20:01.039 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.040 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-17T18:20:01.040 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.040 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-17T18:20:01.040 INFO:tasks.workunit.client.0.smithi173.stderr:+ ff=open_pools_parallel 2024-09-17T18:20:01.040 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-17T18:20:01.040 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35599 2024-09-17T18:20:01.040 INFO:tasks.workunit.client.0.smithi173.stdout:test open_pools_parallel on pid 35599 2024-09-17T18:20:01.054 INFO:tasks.workunit.client.0.smithi173.stderr:+ echo 'test open_pools_parallel on pid 35599' 2024-09-17T18:20:01.054 INFO:tasks.workunit.client.0.smithi173.stderr:+ pids[$f]=35599 2024-09-17T18:20:01.054 INFO:tasks.workunit.client.0.smithi173.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-17T18:20:01.054 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1 -eq 1 ']' 2024-09-17T18:20:01.055 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.055 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-17T18:20:01.055 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.055 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-17T18:20:01.055 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:01.055 INFO:tasks.workunit.client.0.smithi173.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:01.055 INFO:tasks.workunit.client.0.smithi173.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-09-17T18:20:01.055 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.055 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-17T18:20:01.055 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.055 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-17T18:20:01.055 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.055 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-17T18:20:01.056 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-17T18:20:01.056 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-17T18:20:01.056 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.056 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-17T18:20:01.056 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.056 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-17T18:20:01.056 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-17T18:20:01.056 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.056 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -f /etc/bashrc ']' 2024-09-17T18:20:01.056 INFO:tasks.workunit.client.0.smithi173.stderr:+ . /etc/bashrc 2024-09-17T18:20:01.056 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -z '' ']' 2024-09-17T18:20:01.056 INFO:tasks.workunit.client.0.smithi173.stderr:++ BASHRCSOURCED=Y 2024-09-17T18:20:01.056 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.056 INFO:tasks.workunit.client.0.smithi173.stderr:++ shopt -q login_shell 2024-09-17T18:20:01.057 INFO:tasks.workunit.client.0.smithi173.stderr:+++ umask 2024-09-17T18:20:01.057 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 0022 -eq 0 ']' 2024-09-17T18:20:01.057 INFO:tasks.workunit.client.0.smithi173.stderr:++ SHELL=/bin/bash 2024-09-17T18:20:01.057 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.057 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-17T18:20:01.057 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.057 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-17T18:20:01.057 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-17T18:20:01.057 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.057 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-17T18:20:01.057 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.057 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-17T18:20:01.057 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:01.057 INFO:tasks.workunit.client.0.smithi173.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:01.058 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.058 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-17T18:20:01.058 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.058 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-17T18:20:01.058 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.058 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-17T18:20:01.058 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-17T18:20:01.058 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-17T18:20:01.058 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.058 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'grep=grep --color=auto' 2024-09-17T18:20:01.058 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-17T18:20:01.058 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-17T18:20:01.058 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-17T18:20:01.058 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.059 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-17T18:20:01.059 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.059 INFO:tasks.workunit.client.0.smithi173.stderr:+ tee ceph_test_rados_api_cmd.log 2024-09-17T18:20:01.059 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorls.sh 2024-09-17T18:20:01.059 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' '!' -t 0 ']' 2024-09-17T18:20:01.059 INFO:tasks.workunit.client.0.smithi173.stderr:+++ return 2024-09-17T18:20:01.059 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.059 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-17T18:20:01.059 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.059 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-17T18:20:01.059 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export S_COLORS=auto 2024-09-17T18:20:01.059 INFO:tasks.workunit.client.0.smithi173.stderr:+++ S_COLORS=auto 2024-09-17T18:20:01.059 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.059 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.060 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-17T18:20:01.060 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-17T18:20:01.060 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-17T18:20:01.060 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.060 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-17T18:20:01.060 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.060 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.060 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-17T18:20:01.060 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.060 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/gawk.sh 2024-09-17T18:20:01.060 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.060 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-17T18:20:01.060 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.060 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/lang.sh 2024-09-17T18:20:01.061 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup 2024-09-17T18:20:01.061 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/locale 2024-09-17T18:20:01.061 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'grep=grep --color=auto' 2024-09-17T18:20:01.061 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-17T18:20:01.061 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-17T18:20:01.061 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.061 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-17T18:20:01.061 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.061 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorls.sh 2024-09-17T18:20:01.061 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' '!' -t 0 ']' 2024-09-17T18:20:01.061 INFO:tasks.workunit.client.0.smithi173.stderr:+++ return 2024-09-17T18:20:01.061 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:01.061 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.061 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-17T18:20:01.062 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.062 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-17T18:20:01.062 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export S_COLORS=auto 2024-09-17T18:20:01.062 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-17T18:20:01.062 INFO:tasks.workunit.client.0.smithi173.stderr:+++ S_COLORS=auto 2024-09-17T18:20:01.062 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.062 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:01.062 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-17T18:20:01.062 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-17T18:20:01.062 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.062 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-17T18:20:01.062 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-17T18:20:01.062 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-17T18:20:01.062 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-17T18:20:01.063 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.063 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-17T18:20:01.063 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.063 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-17T18:20:01.063 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.063 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-17T18:20:01.063 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-17T18:20:01.063 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.063 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-17T18:20:01.063 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-17T18:20:01.063 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-17T18:20:01.063 INFO:tasks.workunit.client.0.smithi173.stderr:++ printf %25s delete_pools_parallel 2024-09-17T18:20:01.063 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-17T18:20:01.063 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.064 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-17T18:20:01.064 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.064 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-17T18:20:01.064 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-17T18:20:01.064 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.064 INFO:tasks.workunit.client.0.smithi173.stderr:+ r=' delete_pools_parallel' 2024-09-17T18:20:01.064 INFO:tasks.workunit.client.0.smithi173.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-17T18:20:01.064 INFO:tasks.workunit.client.0.smithi173.stderr:++++ export LANG=en_US.UTF-8 2024-09-17T18:20:01.064 INFO:tasks.workunit.client.0.smithi173.stderr:++++ LANG=en_US.UTF-8 2024-09-17T18:20:01.064 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:01.064 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-17T18:20:01.064 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:01.064 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG=en_US.UTF-8 2024-09-17T18:20:01.064 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup config 2024-09-17T18:20:01.065 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:01.065 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '{print $1}' 2024-09-17T18:20:01.065 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:01.065 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' dumb = linux ']' 2024-09-17T18:20:01.065 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.065 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-17T18:20:01.065 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.065 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/less.sh 2024-09-17T18:20:01.065 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-17T18:20:01.065 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.065 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-17T18:20:01.065 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.065 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/which2.sh 2024-09-17T18:20:01.065 INFO:tasks.workunit.client.0.smithi173.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-17T18:20:01.065 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-17T18:20:01.066 INFO:tasks.workunit.client.0.smithi173.stderr:++ echo delete_pools_parallel 2024-09-17T18:20:01.066 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-17T18:20:01.066 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-17T18:20:01.066 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.066 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-17T18:20:01.066 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.066 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-17T18:20:01.066 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-17T18:20:01.066 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.066 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-17T18:20:01.066 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.066 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/gawk.sh 2024-09-17T18:20:01.066 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.066 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-17T18:20:01.067 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.067 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/lang.sh 2024-09-17T18:20:01.067 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup 2024-09-17T18:20:01.067 INFO:tasks.workunit.client.0.smithi173.stderr:+ ff=delete_pools_parallel 2024-09-17T18:20:01.067 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35628 2024-09-17T18:20:01.067 INFO:tasks.workunit.client.0.smithi173.stderr:+ echo 'test delete_pools_parallel on pid 35628' 2024-09-17T18:20:01.067 INFO:tasks.workunit.client.0.smithi173.stdout:test delete_pools_parallel on pid 35628 2024-09-17T18:20:01.081 INFO:tasks.workunit.client.0.smithi173.stderr:+ pids[$f]=35628 2024-09-17T18:20:01.081 INFO:tasks.workunit.client.0.smithi173.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-17T18:20:01.082 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1 -eq 1 ']' 2024-09-17T18:20:01.082 INFO:tasks.workunit.client.0.smithi173.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-09-17T18:20:01.082 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -f /etc/bashrc ']' 2024-09-17T18:20:01.082 INFO:tasks.workunit.client.0.smithi173.stderr:+ . /etc/bashrc 2024-09-17T18:20:01.082 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -z '' ']' 2024-09-17T18:20:01.082 INFO:tasks.workunit.client.0.smithi173.stderr:++ BASHRCSOURCED=Y 2024-09-17T18:20:01.082 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.082 INFO:tasks.workunit.client.0.smithi173.stderr:++ shopt -q login_shell 2024-09-17T18:20:01.082 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-17T18:20:01.082 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-17T18:20:01.082 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-17T18:20:01.082 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.082 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-17T18:20:01.082 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.083 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-17T18:20:01.083 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-17T18:20:01.083 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.083 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-17T18:20:01.083 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.083 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/gawk.sh 2024-09-17T18:20:01.083 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.083 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-17T18:20:01.083 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.083 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/lang.sh 2024-09-17T18:20:01.083 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup 2024-09-17T18:20:01.083 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/locale 2024-09-17T18:20:01.083 INFO:tasks.workunit.client.0.smithi173.stderr:+++ umask 2024-09-17T18:20:01.083 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 0022 -eq 0 ']' 2024-09-17T18:20:01.083 INFO:tasks.workunit.client.0.smithi173.stderr:++ SHELL=/bin/bash 2024-09-17T18:20:01.084 INFO:tasks.workunit.client.0.smithi173.stderr:++ printf %25s cls 2024-09-17T18:20:01.084 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.084 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-17T18:20:01.084 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.084 INFO:tasks.workunit.client.0.smithi173.stderr:+ r=' cls' 2024-09-17T18:20:01.084 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-17T18:20:01.084 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-17T18:20:01.084 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.084 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-17T18:20:01.084 INFO:tasks.workunit.client.0.smithi173.stderr:+++++ readlink /proc/35464/exe 2024-09-17T18:20:01.084 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.084 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-17T18:20:01.084 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-17T18:20:01.084 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-17T18:20:01.085 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-17T18:20:01.085 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.085 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-17T18:20:01.085 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.085 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-17T18:20:01.085 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-17T18:20:01.085 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.085 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:01.085 INFO:tasks.workunit.client.0.smithi173.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:01.085 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.085 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-17T18:20:01.085 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.085 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-17T18:20:01.085 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/locale 2024-09-17T18:20:01.086 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.086 INFO:tasks.workunit.client.0.smithi173.stderr:++++ basename /usr/bin/bash 2024-09-17T18:20:01.086 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_declare='declare -f' 2024-09-17T18:20:01.086 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_opt=-f 2024-09-17T18:20:01.086 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export which_declare 2024-09-17T18:20:01.086 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export -f which 2024-09-17T18:20:01.086 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset i 2024-09-17T18:20:01.086 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset -f pathmunge 2024-09-17T18:20:01.086 INFO:tasks.workunit.client.0.smithi173.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-17T18:20:01.086 INFO:tasks.workunit.client.0.smithi173.stderr:+ export PATH 2024-09-17T18:20:01.086 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-17T18:20:01.086 INFO:tasks.workunit.client.0.smithi173.stderr:+ unset rc 2024-09-17T18:20:01.086 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:01.086 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-17T18:20:01.087 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:01.087 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-17T18:20:01.087 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-17T18:20:01.087 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:01.087 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-17T18:20:01.087 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:01.087 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-17T18:20:01.087 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-17T18:20:01.087 INFO:tasks.workunit.client.0.smithi173.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-09-17T18:20:01.087 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-17T18:20:01.087 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-17T18:20:01.087 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-17T18:20:01.087 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.087 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-17T18:20:01.088 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.088 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-17T18:20:01.088 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-17T18:20:01.088 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.088 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-17T18:20:01.088 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.088 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/gawk.sh 2024-09-17T18:20:01.088 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.088 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-17T18:20:01.088 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.088 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/lang.sh 2024-09-17T18:20:01.089 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup 2024-09-17T18:20:01.089 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/locale 2024-09-17T18:20:01.089 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-17T18:20:01.089 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-17T18:20:01.089 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-17T18:20:01.089 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.089 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-17T18:20:01.089 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.089 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-17T18:20:01.089 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-17T18:20:01.089 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.089 INFO:tasks.workunit.client.0.smithi173.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2024-09-17T18:20:01.089 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:01.090 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-17T18:20:01.090 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:01.090 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-17T18:20:01.090 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-17T18:20:01.090 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-17T18:20:01.090 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-17T18:20:01.090 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-17T18:20:01.090 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.090 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-17T18:20:01.090 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.090 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-17T18:20:01.090 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-17T18:20:01.090 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-17T18:20:01.090 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.090 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-17T18:20:01.091 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.091 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/gawk.sh 2024-09-17T18:20:01.091 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.091 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-17T18:20:01.091 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.091 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/lang.sh 2024-09-17T18:20:01.091 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup 2024-09-17T18:20:01.091 INFO:tasks.workunit.client.0.smithi173.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-17T18:20:01.091 INFO:tasks.workunit.client.0.smithi173.stderr:++++ export LANG=en_US.UTF-8 2024-09-17T18:20:01.091 INFO:tasks.workunit.client.0.smithi173.stderr:++++ LANG=en_US.UTF-8 2024-09-17T18:20:01.091 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:01.091 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-17T18:20:01.091 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:01.091 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG=en_US.UTF-8 2024-09-17T18:20:01.092 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup config 2024-09-17T18:20:01.092 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:01.092 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:01.092 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' dumb = linux ']' 2024-09-17T18:20:01.092 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.092 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-17T18:20:01.092 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.092 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/less.sh 2024-09-17T18:20:01.092 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-17T18:20:01.092 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.092 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-17T18:20:01.092 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.092 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/which2.sh 2024-09-17T18:20:01.092 INFO:tasks.workunit.client.0.smithi173.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-17T18:20:01.093 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/locale 2024-09-17T18:20:01.093 INFO:tasks.workunit.client.0.smithi173.stderr:++ echo cls 2024-09-17T18:20:01.093 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '{print $1}' 2024-09-17T18:20:01.093 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'grep=grep --color=auto' 2024-09-17T18:20:01.093 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-17T18:20:01.093 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-17T18:20:01.093 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.093 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-17T18:20:01.093 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.093 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorls.sh 2024-09-17T18:20:01.093 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-17T18:20:01.093 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' '!' -t 0 ']' 2024-09-17T18:20:01.093 INFO:tasks.workunit.client.0.smithi173.stderr:+++ return 2024-09-17T18:20:01.093 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.094 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-17T18:20:01.094 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.094 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-17T18:20:01.094 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export S_COLORS=auto 2024-09-17T18:20:01.094 INFO:tasks.workunit.client.0.smithi173.stderr:+++ S_COLORS=auto 2024-09-17T18:20:01.094 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.094 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-17T18:20:01.094 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.094 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-17T18:20:01.094 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.094 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:01.094 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-17T18:20:01.094 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:01.094 INFO:tasks.workunit.client.0.smithi173.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-09-17T18:20:01.095 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-17T18:20:01.095 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-17T18:20:01.095 INFO:tasks.workunit.client.0.smithi173.stderr:+ ff=cls 2024-09-17T18:20:01.095 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35688 2024-09-17T18:20:01.095 INFO:tasks.workunit.client.0.smithi173.stderr:+ echo 'test cls on pid 35688' 2024-09-17T18:20:01.095 INFO:tasks.workunit.client.0.smithi173.stdout:test cls on pid 35688 2024-09-17T18:20:01.104 INFO:tasks.workunit.client.0.smithi173.stderr:+ pids[$f]=35688 2024-09-17T18:20:01.104 INFO:tasks.workunit.client.0.smithi173.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-17T18:20:01.104 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1 -eq 1 ']' 2024-09-17T18:20:01.104 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-17T18:20:01.104 INFO:tasks.workunit.client.0.smithi173.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-17T18:20:01.104 INFO:tasks.workunit.client.0.smithi173.stderr:++++ export LANG=en_US.UTF-8 2024-09-17T18:20:01.104 INFO:tasks.workunit.client.0.smithi173.stderr:++++ LANG=en_US.UTF-8 2024-09-17T18:20:01.104 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:01.104 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-17T18:20:01.104 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:01.104 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG=en_US.UTF-8 2024-09-17T18:20:01.105 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup config 2024-09-17T18:20:01.105 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:01.105 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-17T18:20:01.105 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:01.105 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' dumb = linux ']' 2024-09-17T18:20:01.105 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.105 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-17T18:20:01.105 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.105 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/less.sh 2024-09-17T18:20:01.105 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-17T18:20:01.105 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.105 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-17T18:20:01.105 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.105 INFO:tasks.workunit.client.0.smithi173.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-17T18:20:01.105 INFO:tasks.workunit.client.0.smithi173.stderr:++++ export LANG=en_US.UTF-8 2024-09-17T18:20:01.106 INFO:tasks.workunit.client.0.smithi173.stderr:++++ LANG=en_US.UTF-8 2024-09-17T18:20:01.106 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/which2.sh 2024-09-17T18:20:01.106 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:01.106 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-17T18:20:01.106 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:01.106 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG=en_US.UTF-8 2024-09-17T18:20:01.106 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup config 2024-09-17T18:20:01.106 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:01.106 INFO:tasks.workunit.client.0.smithi173.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-17T18:20:01.106 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:01.106 INFO:tasks.workunit.client.0.smithi173.stderr:+++++ readlink /proc/35599/exe 2024-09-17T18:20:01.106 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' dumb = linux ']' 2024-09-17T18:20:01.106 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.106 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-17T18:20:01.106 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.106 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/less.sh 2024-09-17T18:20:01.107 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-17T18:20:01.107 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.107 INFO:tasks.workunit.client.0.smithi173.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-09-17T18:20:01.107 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-17T18:20:01.107 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.107 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/which2.sh 2024-09-17T18:20:01.107 INFO:tasks.workunit.client.0.smithi173.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-17T18:20:01.107 INFO:tasks.workunit.client.0.smithi173.stderr:++ printf %25s cmd 2024-09-17T18:20:01.107 INFO:tasks.workunit.client.0.smithi173.stderr:+ r=' cmd' 2024-09-17T18:20:01.107 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '{print $1}' 2024-09-17T18:20:01.107 INFO:tasks.workunit.client.0.smithi173.stderr:+++++ readlink /proc/35518/exe 2024-09-17T18:20:01.107 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-17T18:20:01.107 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -f /etc/bashrc ']' 2024-09-17T18:20:01.108 INFO:tasks.workunit.client.0.smithi173.stderr:+ . /etc/bashrc 2024-09-17T18:20:01.108 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-17T18:20:01.108 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-17T18:20:01.108 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -z '' ']' 2024-09-17T18:20:01.108 INFO:tasks.workunit.client.0.smithi173.stderr:++ BASHRCSOURCED=Y 2024-09-17T18:20:01.108 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.108 INFO:tasks.workunit.client.0.smithi173.stderr:++ shopt -q login_shell 2024-09-17T18:20:01.108 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.108 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-17T18:20:01.108 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.108 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-17T18:20:01.108 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-17T18:20:01.108 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.108 INFO:tasks.workunit.client.0.smithi173.stderr:++++ basename /usr/bin/bash 2024-09-17T18:20:01.109 INFO:tasks.workunit.client.0.smithi173.stderr:++++ basename /usr/bin/bash 2024-09-17T18:20:01.109 INFO:tasks.workunit.client.0.smithi173.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-17T18:20:01.109 INFO:tasks.workunit.client.0.smithi173.stderr:++++ export LANG=en_US.UTF-8 2024-09-17T18:20:01.109 INFO:tasks.workunit.client.0.smithi173.stderr:++++ LANG=en_US.UTF-8 2024-09-17T18:20:01.109 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:01.109 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-17T18:20:01.109 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:01.109 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG=en_US.UTF-8 2024-09-17T18:20:01.109 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup config 2024-09-17T18:20:01.109 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:01.109 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:01.109 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' dumb = linux ']' 2024-09-17T18:20:01.109 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.109 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-17T18:20:01.110 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.110 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/less.sh 2024-09-17T18:20:01.110 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-17T18:20:01.110 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.110 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-17T18:20:01.110 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.110 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/which2.sh 2024-09-17T18:20:01.110 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_declare='declare -f' 2024-09-17T18:20:01.110 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_opt=-f 2024-09-17T18:20:01.110 INFO:tasks.workunit.client.0.smithi173.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-17T18:20:01.110 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export which_declare 2024-09-17T18:20:01.110 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export -f which 2024-09-17T18:20:01.110 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset i 2024-09-17T18:20:01.110 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset -f pathmunge 2024-09-17T18:20:01.111 INFO:tasks.workunit.client.0.smithi173.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-17T18:20:01.111 INFO:tasks.workunit.client.0.smithi173.stderr:++ echo cmd 2024-09-17T18:20:01.111 INFO:tasks.workunit.client.0.smithi173.stderr:+ export PATH 2024-09-17T18:20:01.111 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-17T18:20:01.111 INFO:tasks.workunit.client.0.smithi173.stderr:+ unset rc 2024-09-17T18:20:01.111 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_declare='declare -f' 2024-09-17T18:20:01.111 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_opt=-f 2024-09-17T18:20:01.111 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export which_declare 2024-09-17T18:20:01.111 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export -f which 2024-09-17T18:20:01.111 INFO:tasks.workunit.client.0.smithi173.stderr:+ ff=cmd 2024-09-17T18:20:01.111 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset i 2024-09-17T18:20:01.111 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset -f pathmunge 2024-09-17T18:20:01.111 INFO:tasks.workunit.client.0.smithi173.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-17T18:20:01.111 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35718 2024-09-17T18:20:01.112 INFO:tasks.workunit.client.0.smithi173.stdout:test cmd on pid 35718 2024-09-17T18:20:01.121 INFO:tasks.workunit.client.0.smithi173.stderr:+ echo 'test cmd on pid 35718' 2024-09-17T18:20:01.121 INFO:tasks.workunit.client.0.smithi173.stderr:+ pids[$f]=35718 2024-09-17T18:20:01.121 INFO:tasks.workunit.client.0.smithi173.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-17T18:20:01.121 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1 -eq 1 ']' 2024-09-17T18:20:01.121 INFO:tasks.workunit.client.0.smithi173.stderr:+ export PATH 2024-09-17T18:20:01.121 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-17T18:20:01.121 INFO:tasks.workunit.client.0.smithi173.stderr:+ unset rc 2024-09-17T18:20:01.121 INFO:tasks.workunit.client.0.smithi173.stderr:++ printf %25s handler_error 2024-09-17T18:20:01.121 INFO:tasks.workunit.client.0.smithi173.stderr:+ r=' handler_error' 2024-09-17T18:20:01.121 INFO:tasks.workunit.client.0.smithi173.stderr:+++++ readlink /proc/35439/exe 2024-09-17T18:20:01.121 INFO:tasks.workunit.client.0.smithi173.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-09-17T18:20:01.121 INFO:tasks.workunit.client.0.smithi173.stderr:++++ basename /usr/bin/bash 2024-09-17T18:20:01.121 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -f /etc/bashrc ']' 2024-09-17T18:20:01.122 INFO:tasks.workunit.client.0.smithi173.stderr:+ . /etc/bashrc 2024-09-17T18:20:01.122 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -z '' ']' 2024-09-17T18:20:01.122 INFO:tasks.workunit.client.0.smithi173.stderr:++ BASHRCSOURCED=Y 2024-09-17T18:20:01.122 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.122 INFO:tasks.workunit.client.0.smithi173.stderr:++ shopt -q login_shell 2024-09-17T18:20:01.122 INFO:tasks.workunit.client.0.smithi173.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2024-09-17T18:20:01.122 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_declare='declare -f' 2024-09-17T18:20:01.122 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_opt=-f 2024-09-17T18:20:01.122 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export which_declare 2024-09-17T18:20:01.122 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export -f which 2024-09-17T18:20:01.122 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset i 2024-09-17T18:20:01.122 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset -f pathmunge 2024-09-17T18:20:01.122 INFO:tasks.workunit.client.0.smithi173.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-17T18:20:01.122 INFO:tasks.workunit.client.0.smithi173.stderr:+ export PATH 2024-09-17T18:20:01.123 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-17T18:20:01.123 INFO:tasks.workunit.client.0.smithi173.stderr:+ unset rc 2024-09-17T18:20:01.123 INFO:tasks.workunit.client.0.smithi173.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-09-17T18:20:01.123 INFO:tasks.workunit.client.0.smithi173.stderr:+++ umask 2024-09-17T18:20:01.123 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 0022 -eq 0 ']' 2024-09-17T18:20:01.123 INFO:tasks.workunit.client.0.smithi173.stderr:++ SHELL=/bin/bash 2024-09-17T18:20:01.123 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.123 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-17T18:20:01.123 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.123 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-17T18:20:01.123 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-17T18:20:01.123 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.123 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-17T18:20:01.123 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-17T18:20:01.124 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-17T18:20:01.124 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-17T18:20:01.124 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.124 INFO:tasks.workunit.client.0.smithi173.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2024-09-17T18:20:01.124 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-17T18:20:01.124 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.124 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-17T18:20:01.124 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.124 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:01.124 INFO:tasks.workunit.client.0.smithi173.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:01.124 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-17T18:20:01.124 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.124 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-17T18:20:01.124 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.125 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-17T18:20:01.125 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-17T18:20:01.125 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.125 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.125 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-17T18:20:01.125 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.125 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/gawk.sh 2024-09-17T18:20:01.125 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.125 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-17T18:20:01.125 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.125 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/lang.sh 2024-09-17T18:20:01.125 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup 2024-09-17T18:20:01.125 INFO:tasks.workunit.client.0.smithi173.stderr:+ sed 's/^/ api_service_pp: /' 2024-09-17T18:20:01.125 INFO:tasks.workunit.client.0.smithi173.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-09-17T18:20:01.126 INFO:tasks.workunit.client.0.smithi173.stderr:+++ umask 2024-09-17T18:20:01.126 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 0022 -eq 0 ']' 2024-09-17T18:20:01.126 INFO:tasks.workunit.client.0.smithi173.stderr:++ SHELL=/bin/bash 2024-09-17T18:20:01.126 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.126 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-17T18:20:01.126 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.126 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-17T18:20:01.126 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-17T18:20:01.126 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.126 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-17T18:20:01.126 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.126 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-17T18:20:01.126 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:01.126 INFO:tasks.workunit.client.0.smithi173.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:01.126 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.127 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-17T18:20:01.127 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.127 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-17T18:20:01.127 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.127 INFO:tasks.workunit.client.0.smithi173.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-09-17T18:20:01.127 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'grep=grep --color=auto' 2024-09-17T18:20:01.127 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-17T18:20:01.127 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-17T18:20:01.127 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.127 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-17T18:20:01.127 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.127 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorls.sh 2024-09-17T18:20:01.127 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' '!' -t 0 ']' 2024-09-17T18:20:01.127 INFO:tasks.workunit.client.0.smithi173.stderr:+++ return 2024-09-17T18:20:01.128 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.128 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-17T18:20:01.128 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.128 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-17T18:20:01.128 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export S_COLORS=auto 2024-09-17T18:20:01.128 INFO:tasks.workunit.client.0.smithi173.stderr:+++ S_COLORS=auto 2024-09-17T18:20:01.128 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.128 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-17T18:20:01.128 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.128 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-17T18:20:01.128 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.128 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '{print $1}' 2024-09-17T18:20:01.128 INFO:tasks.workunit.client.0.smithi173.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2024-09-17T18:20:01.128 INFO:tasks.workunit.client.0.smithi173.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-09-17T18:20:01.129 INFO:tasks.workunit.client.0.smithi173.stderr:+++++ readlink /proc/35552/exe 2024-09-17T18:20:01.129 INFO:tasks.workunit.client.0.smithi173.stderr:++++ basename /usr/bin/bash 2024-09-17T18:20:01.129 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_declare='declare -f' 2024-09-17T18:20:01.129 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_opt=-f 2024-09-17T18:20:01.129 INFO:tasks.workunit.client.0.smithi173.stderr:++ echo handler_error 2024-09-17T18:20:01.129 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export which_declare 2024-09-17T18:20:01.129 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export -f which 2024-09-17T18:20:01.129 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset i 2024-09-17T18:20:01.129 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset -f pathmunge 2024-09-17T18:20:01.129 INFO:tasks.workunit.client.0.smithi173.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-17T18:20:01.129 INFO:tasks.workunit.client.0.smithi173.stderr:+ export PATH 2024-09-17T18:20:01.129 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-17T18:20:01.129 INFO:tasks.workunit.client.0.smithi173.stderr:+ ff=handler_error 2024-09-17T18:20:01.129 INFO:tasks.workunit.client.0.smithi173.stderr:+ unset rc 2024-09-17T18:20:01.130 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35766 2024-09-17T18:20:01.130 INFO:tasks.workunit.client.0.smithi173.stderr:+ echo 'test handler_error on pid 35766' 2024-09-17T18:20:01.130 INFO:tasks.workunit.client.0.smithi173.stdout:test handler_error on pid 35766 2024-09-17T18:20:01.135 INFO:tasks.workunit.client.0.smithi173.stderr:+ pids[$f]=35766 2024-09-17T18:20:01.135 INFO:tasks.workunit.client.0.smithi173.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-17T18:20:01.135 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1 -eq 1 ']' 2024-09-17T18:20:01.135 INFO:tasks.workunit.client.0.smithi173.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-09-17T18:20:01.135 INFO:tasks.workunit.client.0.smithi173.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2024-09-17T18:20:01.135 INFO:tasks.workunit.client.0.smithi173.stderr:+ sed 's/^/ list_parallel: /' 2024-09-17T18:20:01.136 INFO:tasks.workunit.client.0.smithi173.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-09-17T18:20:01.136 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/locale 2024-09-17T18:20:01.136 INFO:tasks.workunit.client.0.smithi173.stderr:+ tee ceph_test_rados_list_parallel.log 2024-09-17T18:20:01.136 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'grep=grep --color=auto' 2024-09-17T18:20:01.136 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-17T18:20:01.136 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-17T18:20:01.136 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.136 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-17T18:20:01.136 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.136 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorls.sh 2024-09-17T18:20:01.136 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' '!' -t 0 ']' 2024-09-17T18:20:01.136 INFO:tasks.workunit.client.0.smithi173.stderr:+++ return 2024-09-17T18:20:01.136 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.136 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-17T18:20:01.137 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.137 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-17T18:20:01.137 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export S_COLORS=auto 2024-09-17T18:20:01.137 INFO:tasks.workunit.client.0.smithi173.stderr:+++ S_COLORS=auto 2024-09-17T18:20:01.137 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.137 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-17T18:20:01.137 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.137 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-17T18:20:01.137 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.137 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -f /etc/bashrc ']' 2024-09-17T18:20:01.137 INFO:tasks.workunit.client.0.smithi173.stderr:+ . /etc/bashrc 2024-09-17T18:20:01.137 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -z '' ']' 2024-09-17T18:20:01.137 INFO:tasks.workunit.client.0.smithi173.stderr:++ BASHRCSOURCED=Y 2024-09-17T18:20:01.137 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.138 INFO:tasks.workunit.client.0.smithi173.stderr:++ shopt -q login_shell 2024-09-17T18:20:01.138 INFO:tasks.workunit.client.0.smithi173.stderr:+++ umask 2024-09-17T18:20:01.138 INFO:tasks.workunit.client.0.smithi173.stderr:++ printf %25s io 2024-09-17T18:20:01.138 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 0022 -eq 0 ']' 2024-09-17T18:20:01.138 INFO:tasks.workunit.client.0.smithi173.stderr:++ SHELL=/bin/bash 2024-09-17T18:20:01.138 INFO:tasks.workunit.client.0.smithi173.stderr:+ r=' io' 2024-09-17T18:20:01.138 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.138 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-17T18:20:01.138 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.138 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-17T18:20:01.138 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-17T18:20:01.138 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.138 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-17T18:20:01.138 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.139 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-17T18:20:01.139 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '{print $1}' 2024-09-17T18:20:01.139 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:01.139 INFO:tasks.workunit.client.0.smithi173.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:01.139 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.139 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-17T18:20:01.139 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.139 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-17T18:20:01.139 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.139 INFO:tasks.workunit.client.0.smithi173.stderr:++ echo io 2024-09-17T18:20:01.139 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-17T18:20:01.139 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-17T18:20:01.139 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-17T18:20:01.139 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.139 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-17T18:20:01.139 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.140 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-17T18:20:01.140 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-17T18:20:01.140 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.140 INFO:tasks.workunit.client.0.smithi173.stderr:+ ff=io 2024-09-17T18:20:01.140 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35800 2024-09-17T18:20:01.140 INFO:tasks.workunit.client.0.smithi173.stderr:+ echo 'test io on pid 35800' 2024-09-17T18:20:01.140 INFO:tasks.workunit.client.0.smithi173.stdout:test io on pid 35800 2024-09-17T18:20:01.141 INFO:tasks.workunit.client.0.smithi173.stderr:+ pids[$f]=35800 2024-09-17T18:20:01.142 INFO:tasks.workunit.client.0.smithi173.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-17T18:20:01.142 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1 -eq 1 ']' 2024-09-17T18:20:01.142 INFO:tasks.workunit.client.0.smithi173.stderr:++ printf %25s ec_io 2024-09-17T18:20:01.142 INFO:tasks.workunit.client.0.smithi173.stderr:+ r=' ec_io' 2024-09-17T18:20:01.142 INFO:tasks.workunit.client.0.smithi173.stderr:++ echo ec_io 2024-09-17T18:20:01.142 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '{print $1}' 2024-09-17T18:20:01.142 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:01.142 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-17T18:20:01.142 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:01.142 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-17T18:20:01.142 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-17T18:20:01.142 INFO:tasks.workunit.client.0.smithi173.stderr:+ ff=ec_io 2024-09-17T18:20:01.142 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35817 2024-09-17T18:20:01.142 INFO:tasks.workunit.client.0.smithi173.stderr:+ echo 'test ec_io on pid 35817' 2024-09-17T18:20:01.143 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-17T18:20:01.143 INFO:tasks.workunit.client.0.smithi173.stdout:test ec_io on pid 35817 2024-09-17T18:20:01.147 INFO:tasks.workunit.client.0.smithi173.stderr:+ pids[$f]=35817 2024-09-17T18:20:01.148 INFO:tasks.workunit.client.0.smithi173.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-17T18:20:01.148 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1 -eq 1 ']' 2024-09-17T18:20:01.148 INFO:tasks.workunit.client.0.smithi173.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-09-17T18:20:01.148 INFO:tasks.workunit.client.0.smithi173.stderr:++ printf %25s list 2024-09-17T18:20:01.148 INFO:tasks.workunit.client.0.smithi173.stderr:+ r=' list' 2024-09-17T18:20:01.148 INFO:tasks.workunit.client.0.smithi173.stderr:++ echo list 2024-09-17T18:20:01.148 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -f /etc/bashrc ']' 2024-09-17T18:20:01.148 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '{print $1}' 2024-09-17T18:20:01.148 INFO:tasks.workunit.client.0.smithi173.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-17T18:20:01.148 INFO:tasks.workunit.client.0.smithi173.stderr:+ . /etc/bashrc 2024-09-17T18:20:01.148 INFO:tasks.workunit.client.0.smithi173.stderr:++++ export LANG=en_US.UTF-8 2024-09-17T18:20:01.148 INFO:tasks.workunit.client.0.smithi173.stderr:++++ LANG=en_US.UTF-8 2024-09-17T18:20:01.149 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:01.149 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -z '' ']' 2024-09-17T18:20:01.149 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-17T18:20:01.149 INFO:tasks.workunit.client.0.smithi173.stderr:++ BASHRCSOURCED=Y 2024-09-17T18:20:01.149 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.149 INFO:tasks.workunit.client.0.smithi173.stderr:++ shopt -q login_shell 2024-09-17T18:20:01.149 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:01.149 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG=en_US.UTF-8 2024-09-17T18:20:01.149 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup config 2024-09-17T18:20:01.149 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:01.149 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:01.149 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' dumb = linux ']' 2024-09-17T18:20:01.150 INFO:tasks.workunit.client.0.smithi173.stderr:+++ umask 2024-09-17T18:20:01.150 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.150 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-17T18:20:01.150 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.150 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 0022 -eq 0 ']' 2024-09-17T18:20:01.150 INFO:tasks.workunit.client.0.smithi173.stderr:++ SHELL=/bin/bash 2024-09-17T18:20:01.150 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/less.sh 2024-09-17T18:20:01.150 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.150 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-17T18:20:01.150 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.150 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-17T18:20:01.150 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.150 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/which2.sh 2024-09-17T18:20:01.150 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-17T18:20:01.150 INFO:tasks.workunit.client.0.smithi173.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-17T18:20:01.151 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.151 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-17T18:20:01.151 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-17T18:20:01.151 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.151 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-17T18:20:01.151 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.151 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-17T18:20:01.151 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:01.151 INFO:tasks.workunit.client.0.smithi173.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:01.151 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.151 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-17T18:20:01.151 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.151 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-17T18:20:01.151 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.151 INFO:tasks.workunit.client.0.smithi173.stderr:+++++ readlink /proc/35628/exe 2024-09-17T18:20:01.151 INFO:tasks.workunit.client.0.smithi173.stderr:++++ basename /usr/bin/bash 2024-09-17T18:20:01.152 INFO:tasks.workunit.client.0.smithi173.stderr:+ ff=list 2024-09-17T18:20:01.152 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35829 2024-09-17T18:20:01.152 INFO:tasks.workunit.client.0.smithi173.stderr:+ echo 'test list on pid 35829' 2024-09-17T18:20:01.152 INFO:tasks.workunit.client.0.smithi173.stdout:test list on pid 35829 2024-09-17T18:20:01.181 INFO:tasks.workunit.client.0.smithi173.stderr:+ pids[$f]=35829 2024-09-17T18:20:01.181 INFO:tasks.workunit.client.0.smithi173.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-17T18:20:01.182 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1 -eq 1 ']' 2024-09-17T18:20:01.182 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_declare='declare -f' 2024-09-17T18:20:01.182 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_opt=-f 2024-09-17T18:20:01.182 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export which_declare 2024-09-17T18:20:01.182 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export -f which 2024-09-17T18:20:01.182 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset i 2024-09-17T18:20:01.182 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset -f pathmunge 2024-09-17T18:20:01.182 INFO:tasks.workunit.client.0.smithi173.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-17T18:20:01.182 INFO:tasks.workunit.client.0.smithi173.stderr:+ export PATH 2024-09-17T18:20:01.182 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-17T18:20:01.182 INFO:tasks.workunit.client.0.smithi173.stderr:+ unset rc 2024-09-17T18:20:01.182 INFO:tasks.workunit.client.0.smithi173.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2024-09-17T18:20:01.182 INFO:tasks.workunit.client.0.smithi173.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-09-17T18:20:01.182 INFO:tasks.workunit.client.0.smithi173.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-09-17T18:20:01.183 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -f /etc/bashrc ']' 2024-09-17T18:20:01.183 INFO:tasks.workunit.client.0.smithi173.stderr:+ . /etc/bashrc 2024-09-17T18:20:01.183 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'grep=grep --color=auto' 2024-09-17T18:20:01.183 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-17T18:20:01.183 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-17T18:20:01.183 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -z '' ']' 2024-09-17T18:20:01.183 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.183 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-17T18:20:01.183 INFO:tasks.workunit.client.0.smithi173.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-09-17T18:20:01.183 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.183 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorls.sh 2024-09-17T18:20:01.183 INFO:tasks.workunit.client.0.smithi173.stderr:++ BASHRCSOURCED=Y 2024-09-17T18:20:01.183 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' '!' -t 0 ']' 2024-09-17T18:20:01.183 INFO:tasks.workunit.client.0.smithi173.stderr:+++ return 2024-09-17T18:20:01.184 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.184 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-17T18:20:01.184 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.184 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-17T18:20:01.184 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.184 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export S_COLORS=auto 2024-09-17T18:20:01.184 INFO:tasks.workunit.client.0.smithi173.stderr:+++ S_COLORS=auto 2024-09-17T18:20:01.184 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.184 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-17T18:20:01.184 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.184 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-17T18:20:01.184 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.184 INFO:tasks.workunit.client.0.smithi173.stderr:++ shopt -q login_shell 2024-09-17T18:20:01.184 INFO:tasks.workunit.client.0.smithi173.stderr:+++ umask 2024-09-17T18:20:01.185 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 0022 -eq 0 ']' 2024-09-17T18:20:01.185 INFO:tasks.workunit.client.0.smithi173.stderr:++ SHELL=/bin/bash 2024-09-17T18:20:01.185 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.185 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-17T18:20:01.185 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.185 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-17T18:20:01.185 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-17T18:20:01.185 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.185 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-17T18:20:01.185 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.185 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-17T18:20:01.185 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:01.185 INFO:tasks.workunit.client.0.smithi173.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:01.185 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.186 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-17T18:20:01.186 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-17T18:20:01.186 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.186 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-17T18:20:01.186 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-17T18:20:01.186 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-17T18:20:01.186 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.186 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-17T18:20:01.186 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.186 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-17T18:20:01.186 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-17T18:20:01.186 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.186 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.186 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-17T18:20:01.187 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-17T18:20:01.187 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-17T18:20:01.187 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.187 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-17T18:20:01.187 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.187 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-17T18:20:01.187 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-17T18:20:01.187 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.187 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'grep=grep --color=auto' 2024-09-17T18:20:01.187 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-17T18:20:01.187 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-17T18:20:01.187 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.187 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-17T18:20:01.187 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.187 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorls.sh 2024-09-17T18:20:01.188 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' '!' -t 0 ']' 2024-09-17T18:20:01.188 INFO:tasks.workunit.client.0.smithi173.stderr:+++ return 2024-09-17T18:20:01.188 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.188 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-17T18:20:01.188 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.188 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-17T18:20:01.188 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export S_COLORS=auto 2024-09-17T18:20:01.188 INFO:tasks.workunit.client.0.smithi173.stderr:+++ S_COLORS=auto 2024-09-17T18:20:01.188 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.188 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-17T18:20:01.188 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-17T18:20:01.188 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-17T18:20:01.188 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-17T18:20:01.188 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.189 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.189 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-17T18:20:01.189 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.189 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-17T18:20:01.189 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-17T18:20:01.189 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-17T18:20:01.189 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.189 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-17T18:20:01.189 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.189 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/gawk.sh 2024-09-17T18:20:01.189 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.189 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.189 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-17T18:20:01.189 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.190 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/lang.sh 2024-09-17T18:20:01.190 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup 2024-09-17T18:20:01.190 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/locale 2024-09-17T18:20:01.190 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'grep=grep --color=auto' 2024-09-17T18:20:01.190 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-17T18:20:01.190 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-17T18:20:01.190 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.190 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-17T18:20:01.190 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.190 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorls.sh 2024-09-17T18:20:01.190 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' '!' -t 0 ']' 2024-09-17T18:20:01.190 INFO:tasks.workunit.client.0.smithi173.stderr:+++ return 2024-09-17T18:20:01.190 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.190 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-17T18:20:01.191 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.191 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-17T18:20:01.191 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export S_COLORS=auto 2024-09-17T18:20:01.191 INFO:tasks.workunit.client.0.smithi173.stderr:+++ S_COLORS=auto 2024-09-17T18:20:01.191 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.191 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-17T18:20:01.191 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.191 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-17T18:20:01.191 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.191 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:01.191 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-17T18:20:01.191 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:01.191 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-17T18:20:01.191 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-17T18:20:01.192 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-17T18:20:01.192 INFO:tasks.workunit.client.0.smithi173.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-09-17T18:20:01.192 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-17T18:20:01.192 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-17T18:20:01.192 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-17T18:20:01.192 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.192 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-17T18:20:01.192 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.192 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-17T18:20:01.192 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-17T18:20:01.192 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.192 INFO:tasks.workunit.client.0.smithi173.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-17T18:20:01.192 INFO:tasks.workunit.client.0.smithi173.stderr:++++ export LANG=en_US.UTF-8 2024-09-17T18:20:01.193 INFO:tasks.workunit.client.0.smithi173.stderr:++++ LANG=en_US.UTF-8 2024-09-17T18:20:01.193 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:01.193 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-17T18:20:01.193 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:01.193 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG=en_US.UTF-8 2024-09-17T18:20:01.193 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup config 2024-09-17T18:20:01.193 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:01.193 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:01.193 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' dumb = linux ']' 2024-09-17T18:20:01.193 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.193 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-17T18:20:01.193 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-17T18:20:01.193 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-17T18:20:01.193 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-17T18:20:01.194 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.194 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-17T18:20:01.194 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.194 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-17T18:20:01.194 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -f /etc/bashrc ']' 2024-09-17T18:20:01.194 INFO:tasks.workunit.client.0.smithi173.stderr:+ . /etc/bashrc 2024-09-17T18:20:01.194 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-17T18:20:01.194 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.194 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.194 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/less.sh 2024-09-17T18:20:01.194 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -z '' ']' 2024-09-17T18:20:01.194 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-17T18:20:01.194 INFO:tasks.workunit.client.0.smithi173.stderr:++ BASHRCSOURCED=Y 2024-09-17T18:20:01.195 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.195 INFO:tasks.workunit.client.0.smithi173.stderr:++ shopt -q login_shell 2024-09-17T18:20:01.195 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.195 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-17T18:20:01.195 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.195 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/which2.sh 2024-09-17T18:20:01.195 INFO:tasks.workunit.client.0.smithi173.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-17T18:20:01.195 INFO:tasks.workunit.client.0.smithi173.stderr:+++ umask 2024-09-17T18:20:01.195 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 0022 -eq 0 ']' 2024-09-17T18:20:01.196 INFO:tasks.workunit.client.0.smithi173.stderr:+++++ readlink /proc/35688/exe 2024-09-17T18:20:01.196 INFO:tasks.workunit.client.0.smithi173.stderr:++ SHELL=/bin/bash 2024-09-17T18:20:01.196 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.196 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-17T18:20:01.196 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.196 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-17T18:20:01.196 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-17T18:20:01.196 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.196 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-17T18:20:01.196 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.196 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-17T18:20:01.196 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:01.196 INFO:tasks.workunit.client.0.smithi173.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:01.197 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.197 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-17T18:20:01.197 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.197 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-17T18:20:01.197 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.197 INFO:tasks.workunit.client.0.smithi173.stderr:++++ basename /usr/bin/bash 2024-09-17T18:20:01.197 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-17T18:20:01.197 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-17T18:20:01.197 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_declare='declare -f' 2024-09-17T18:20:01.197 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_opt=-f 2024-09-17T18:20:01.198 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export which_declare 2024-09-17T18:20:01.198 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-17T18:20:01.198 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.198 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-17T18:20:01.198 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.198 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-17T18:20:01.198 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export -f which 2024-09-17T18:20:01.198 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-17T18:20:01.198 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset i 2024-09-17T18:20:01.198 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset -f pathmunge 2024-09-17T18:20:01.198 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-17T18:20:01.198 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-17T18:20:01.198 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.199 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-17T18:20:01.199 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-17T18:20:01.199 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.199 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/gawk.sh 2024-09-17T18:20:01.199 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.199 INFO:tasks.workunit.client.0.smithi173.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-17T18:20:01.199 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-17T18:20:01.199 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.199 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-17T18:20:01.199 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-17T18:20:01.199 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.200 INFO:tasks.workunit.client.0.smithi173.stderr:+ export PATH 2024-09-17T18:20:01.200 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-17T18:20:01.200 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.200 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.200 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-17T18:20:01.200 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/gawk.sh 2024-09-17T18:20:01.200 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.200 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-17T18:20:01.200 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/lang.sh 2024-09-17T18:20:01.200 INFO:tasks.workunit.client.0.smithi173.stderr:+ unset rc 2024-09-17T18:20:01.200 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup 2024-09-17T18:20:01.200 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.201 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-17T18:20:01.201 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.201 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/lang.sh 2024-09-17T18:20:01.201 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup 2024-09-17T18:20:01.201 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/locale 2024-09-17T18:20:01.201 INFO:tasks.workunit.client.0.smithi173.stderr:+ tee ceph_test_neorados_cls.log 2024-09-17T18:20:01.201 INFO:tasks.workunit.client.0.smithi173.stderr:+ ceph_test_neorados_cls 2024-09-17T18:20:01.201 INFO:tasks.workunit.client.0.smithi173.stderr:++ printf %25s ec_list 2024-09-17T18:20:01.201 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:01.202 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-17T18:20:01.202 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:01.202 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-17T18:20:01.202 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-17T18:20:01.202 INFO:tasks.workunit.client.0.smithi173.stderr:+ r=' ec_list' 2024-09-17T18:20:01.202 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-17T18:20:01.202 INFO:tasks.workunit.client.0.smithi173.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-17T18:20:01.202 INFO:tasks.workunit.client.0.smithi173.stderr:++++ export LANG=en_US.UTF-8 2024-09-17T18:20:01.202 INFO:tasks.workunit.client.0.smithi173.stderr:++++ LANG=en_US.UTF-8 2024-09-17T18:20:01.202 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:01.202 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-17T18:20:01.202 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-17T18:20:01.202 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-17T18:20:01.203 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-17T18:20:01.203 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.203 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-17T18:20:01.203 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.203 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-17T18:20:01.203 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-17T18:20:01.203 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.203 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-17T18:20:01.203 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.203 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/gawk.sh 2024-09-17T18:20:01.204 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:01.204 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG=en_US.UTF-8 2024-09-17T18:20:01.204 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.204 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-17T18:20:01.204 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.204 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/lang.sh 2024-09-17T18:20:01.204 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup config 2024-09-17T18:20:01.204 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup 2024-09-17T18:20:01.204 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:01.204 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:01.204 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' dumb = linux ']' 2024-09-17T18:20:01.205 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.205 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-17T18:20:01.205 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.205 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/less.sh 2024-09-17T18:20:01.205 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-17T18:20:01.205 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.205 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-17T18:20:01.205 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.205 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/which2.sh 2024-09-17T18:20:01.206 INFO:tasks.workunit.client.0.smithi173.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-17T18:20:01.206 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/locale 2024-09-17T18:20:01.206 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-17T18:20:01.206 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-17T18:20:01.206 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-17T18:20:01.206 INFO:tasks.workunit.client.0.smithi173.stderr:+++++ readlink /proc/35800/exe 2024-09-17T18:20:01.206 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.206 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-17T18:20:01.206 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.207 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-17T18:20:01.207 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-17T18:20:01.207 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.207 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-17T18:20:01.207 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.207 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/gawk.sh 2024-09-17T18:20:01.207 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.207 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-17T18:20:01.207 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.208 INFO:tasks.workunit.client.0.smithi173.stderr:++++ basename /usr/bin/bash 2024-09-17T18:20:01.208 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/lang.sh 2024-09-17T18:20:01.208 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup 2024-09-17T18:20:01.208 INFO:tasks.workunit.client.0.smithi173.stderr:++ echo ec_list 2024-09-17T18:20:01.208 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/locale 2024-09-17T18:20:01.208 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_declare='declare -f' 2024-09-17T18:20:01.208 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_opt=-f 2024-09-17T18:20:01.208 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export which_declare 2024-09-17T18:20:01.208 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export -f which 2024-09-17T18:20:01.208 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset i 2024-09-17T18:20:01.208 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset -f pathmunge 2024-09-17T18:20:01.208 INFO:tasks.workunit.client.0.smithi173.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-17T18:20:01.209 INFO:tasks.workunit.client.0.smithi173.stderr:+ export PATH 2024-09-17T18:20:01.209 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-17T18:20:01.209 INFO:tasks.workunit.client.0.smithi173.stderr:+ unset rc 2024-09-17T18:20:01.209 INFO:tasks.workunit.client.0.smithi173.stderr:+ ceph_test_neorados_io 2024-09-17T18:20:01.209 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/locale 2024-09-17T18:20:01.209 INFO:tasks.workunit.client.0.smithi173.stderr:+ tee ceph_test_neorados_io.log 2024-09-17T18:20:01.209 INFO:tasks.workunit.client.0.smithi173.stderr:+ sed 's/^/ io: /' 2024-09-17T18:20:01.209 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:01.209 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-17T18:20:01.209 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:01.209 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-17T18:20:01.209 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-17T18:20:01.209 INFO:tasks.workunit.client.0.smithi173.stderr:+ sed 's/^/ cls: /' 2024-09-17T18:20:01.210 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-17T18:20:01.210 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '{print $1}' 2024-09-17T18:20:01.210 INFO:tasks.workunit.client.0.smithi173.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-17T18:20:01.210 INFO:tasks.workunit.client.0.smithi173.stderr:++++ export LANG=en_US.UTF-8 2024-09-17T18:20:01.210 INFO:tasks.workunit.client.0.smithi173.stderr:++++ LANG=en_US.UTF-8 2024-09-17T18:20:01.210 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:01.210 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-17T18:20:01.210 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:01.210 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG=en_US.UTF-8 2024-09-17T18:20:01.210 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup config 2024-09-17T18:20:01.210 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:01.210 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:01.210 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' dumb = linux ']' 2024-09-17T18:20:01.210 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.211 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-17T18:20:01.211 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.211 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/less.sh 2024-09-17T18:20:01.211 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-17T18:20:01.211 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.211 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-17T18:20:01.211 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.211 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/which2.sh 2024-09-17T18:20:01.211 INFO:tasks.workunit.client.0.smithi173.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-17T18:20:01.211 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:01.211 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-17T18:20:01.211 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:01.211 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-17T18:20:01.212 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-17T18:20:01.212 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'grep=grep --color=auto' 2024-09-17T18:20:01.212 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-17T18:20:01.212 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-17T18:20:01.212 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.212 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-17T18:20:01.212 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.212 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorls.sh 2024-09-17T18:20:01.212 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' '!' -t 0 ']' 2024-09-17T18:20:01.212 INFO:tasks.workunit.client.0.smithi173.stderr:+++ return 2024-09-17T18:20:01.212 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.212 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-17T18:20:01.212 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.212 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-17T18:20:01.213 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export S_COLORS=auto 2024-09-17T18:20:01.213 INFO:tasks.workunit.client.0.smithi173.stderr:+++ S_COLORS=auto 2024-09-17T18:20:01.213 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.213 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-17T18:20:01.213 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.213 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-17T18:20:01.213 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.213 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:01.213 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-17T18:20:01.213 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:01.213 INFO:tasks.workunit.client.0.smithi173.stderr:+ ff=ec_list 2024-09-17T18:20:01.213 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35931 2024-09-17T18:20:01.213 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-17T18:20:01.213 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-17T18:20:01.213 INFO:tasks.workunit.client.0.smithi173.stderr:+ echo 'test ec_list on pid 35931' 2024-09-17T18:20:01.214 INFO:tasks.workunit.client.0.smithi173.stdout:test ec_list on pid 35931 2024-09-17T18:20:01.223 INFO:tasks.workunit.client.0.smithi173.stderr:+ pids[$f]=35931 2024-09-17T18:20:01.223 INFO:tasks.workunit.client.0.smithi173.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-17T18:20:01.223 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1 -eq 1 ']' 2024-09-17T18:20:01.223 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-17T18:20:01.223 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-17T18:20:01.223 INFO:tasks.workunit.client.0.smithi173.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-17T18:20:01.223 INFO:tasks.workunit.client.0.smithi173.stderr:++++ export LANG=en_US.UTF-8 2024-09-17T18:20:01.223 INFO:tasks.workunit.client.0.smithi173.stderr:++++ LANG=en_US.UTF-8 2024-09-17T18:20:01.223 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:01.224 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-17T18:20:01.224 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:01.224 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG=en_US.UTF-8 2024-09-17T18:20:01.224 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup config 2024-09-17T18:20:01.224 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:01.224 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:01.224 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' dumb = linux ']' 2024-09-17T18:20:01.224 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.224 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-17T18:20:01.224 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.224 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/less.sh 2024-09-17T18:20:01.224 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-17T18:20:01.224 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.225 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-17T18:20:01.225 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.225 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/which2.sh 2024-09-17T18:20:01.225 INFO:tasks.workunit.client.0.smithi173.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-17T18:20:01.225 INFO:tasks.workunit.client.0.smithi173.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-17T18:20:01.225 INFO:tasks.workunit.client.0.smithi173.stderr:++++ export LANG=en_US.UTF-8 2024-09-17T18:20:01.225 INFO:tasks.workunit.client.0.smithi173.stderr:++++ LANG=en_US.UTF-8 2024-09-17T18:20:01.225 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:01.225 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-17T18:20:01.225 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:01.225 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG=en_US.UTF-8 2024-09-17T18:20:01.225 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup config 2024-09-17T18:20:01.225 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:01.225 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:01.225 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' dumb = linux ']' 2024-09-17T18:20:01.226 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.226 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-17T18:20:01.226 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.226 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/less.sh 2024-09-17T18:20:01.226 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-17T18:20:01.226 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.226 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-17T18:20:01.226 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.228 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/which2.sh 2024-09-17T18:20:01.228 INFO:tasks.workunit.client.0.smithi173.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-17T18:20:01.228 INFO:tasks.workunit.client.0.smithi173.stderr:++ printf %25s misc 2024-09-17T18:20:01.228 INFO:tasks.workunit.client.0.smithi173.stderr:+ r=' misc' 2024-09-17T18:20:01.228 INFO:tasks.workunit.client.0.smithi173.stderr:+++++ readlink /proc/35766/exe 2024-09-17T18:20:01.228 INFO:tasks.workunit.client.0.smithi173.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-09-17T18:20:01.228 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -f /etc/bashrc ']' 2024-09-17T18:20:01.228 INFO:tasks.workunit.client.0.smithi173.stderr:+ . /etc/bashrc 2024-09-17T18:20:01.228 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -z '' ']' 2024-09-17T18:20:01.228 INFO:tasks.workunit.client.0.smithi173.stderr:++ BASHRCSOURCED=Y 2024-09-17T18:20:01.229 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.229 INFO:tasks.workunit.client.0.smithi173.stderr:++ shopt -q login_shell 2024-09-17T18:20:01.229 INFO:tasks.workunit.client.0.smithi173.stderr:++++ basename /usr/bin/bash 2024-09-17T18:20:01.229 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_declare='declare -f' 2024-09-17T18:20:01.229 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_opt=-f 2024-09-17T18:20:01.229 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export which_declare 2024-09-17T18:20:01.229 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export -f which 2024-09-17T18:20:01.229 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset i 2024-09-17T18:20:01.229 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset -f pathmunge 2024-09-17T18:20:01.229 INFO:tasks.workunit.client.0.smithi173.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-17T18:20:01.229 INFO:tasks.workunit.client.0.smithi173.stderr:+ export PATH 2024-09-17T18:20:01.229 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-17T18:20:01.229 INFO:tasks.workunit.client.0.smithi173.stderr:+ unset rc 2024-09-17T18:20:01.229 INFO:tasks.workunit.client.0.smithi173.stderr:+++++ readlink /proc/35817/exe 2024-09-17T18:20:01.230 INFO:tasks.workunit.client.0.smithi173.stderr:+++ umask 2024-09-17T18:20:01.230 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 0022 -eq 0 ']' 2024-09-17T18:20:01.230 INFO:tasks.workunit.client.0.smithi173.stderr:++ SHELL=/bin/bash 2024-09-17T18:20:01.230 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.230 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-17T18:20:01.230 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.230 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-17T18:20:01.230 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-17T18:20:01.230 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.230 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-17T18:20:01.230 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.230 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-17T18:20:01.230 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:01.230 INFO:tasks.workunit.client.0.smithi173.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:01.230 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.231 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-17T18:20:01.231 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.231 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-17T18:20:01.231 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.231 INFO:tasks.workunit.client.0.smithi173.stderr:++++ basename /usr/bin/bash 2024-09-17T18:20:01.231 INFO:tasks.workunit.client.0.smithi173.stderr:+++++ readlink /proc/35718/exe 2024-09-17T18:20:01.231 INFO:tasks.workunit.client.0.smithi173.stderr:++ echo misc 2024-09-17T18:20:01.231 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '{print $1}' 2024-09-17T18:20:01.231 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_declare='declare -f' 2024-09-17T18:20:01.231 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_opt=-f 2024-09-17T18:20:01.231 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export which_declare 2024-09-17T18:20:01.231 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export -f which 2024-09-17T18:20:01.231 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset i 2024-09-17T18:20:01.231 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset -f pathmunge 2024-09-17T18:20:01.231 INFO:tasks.workunit.client.0.smithi173.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-17T18:20:01.232 INFO:tasks.workunit.client.0.smithi173.stderr:+ export PATH 2024-09-17T18:20:01.232 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-17T18:20:01.232 INFO:tasks.workunit.client.0.smithi173.stderr:+ unset rc 2024-09-17T18:20:01.232 INFO:tasks.workunit.client.0.smithi173.stderr:+ tee ceph_test_neorados_handler_error.log 2024-09-17T18:20:01.232 INFO:tasks.workunit.client.0.smithi173.stderr:+ sed 's/^/ handler_error: /' 2024-09-17T18:20:01.232 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-17T18:20:01.232 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-17T18:20:01.232 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-17T18:20:01.232 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.233 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-17T18:20:01.233 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.233 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-17T18:20:01.233 INFO:tasks.workunit.client.0.smithi173.stderr:++++ basename /usr/bin/bash 2024-09-17T18:20:01.233 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-17T18:20:01.233 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.233 INFO:tasks.workunit.client.0.smithi173.stderr:+ ff=misc 2024-09-17T18:20:01.233 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=36020 2024-09-17T18:20:01.233 INFO:tasks.workunit.client.0.smithi173.stderr:+ echo 'test misc on pid 36020' 2024-09-17T18:20:01.234 INFO:tasks.workunit.client.0.smithi173.stdout:test misc on pid 36020 2024-09-17T18:20:01.243 INFO:tasks.workunit.client.0.smithi173.stderr:+ pids[$f]=36020 2024-09-17T18:20:01.243 INFO:tasks.workunit.client.0.smithi173.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-17T18:20:01.243 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1 -eq 1 ']' 2024-09-17T18:20:01.243 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_declare='declare -f' 2024-09-17T18:20:01.243 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_opt=-f 2024-09-17T18:20:01.243 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export which_declare 2024-09-17T18:20:01.243 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export -f which 2024-09-17T18:20:01.243 INFO:tasks.workunit.client.0.smithi173.stderr:+ sed 's/^/ ec_io: /' 2024-09-17T18:20:01.244 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset i 2024-09-17T18:20:01.244 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset -f pathmunge 2024-09-17T18:20:01.244 INFO:tasks.workunit.client.0.smithi173.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-17T18:20:01.244 INFO:tasks.workunit.client.0.smithi173.stderr:+ export PATH 2024-09-17T18:20:01.244 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-17T18:20:01.244 INFO:tasks.workunit.client.0.smithi173.stderr:+ unset rc 2024-09-17T18:20:01.244 INFO:tasks.workunit.client.0.smithi173.stderr:++ printf %25s pool 2024-09-17T18:20:01.244 INFO:tasks.workunit.client.0.smithi173.stderr:+ ceph_test_neorados_ec_io 2024-09-17T18:20:01.244 INFO:tasks.workunit.client.0.smithi173.stderr:+ r=' pool' 2024-09-17T18:20:01.244 INFO:tasks.workunit.client.0.smithi173.stderr:+ ceph_test_neorados_handler_error 2024-09-17T18:20:01.244 INFO:tasks.workunit.client.0.smithi173.stderr:+ ceph_test_neorados_cmd 2024-09-17T18:20:01.244 INFO:tasks.workunit.client.0.smithi173.stderr:+ tee ceph_test_neorados_ec_io.log 2024-09-17T18:20:01.244 INFO:tasks.workunit.client.0.smithi173.stderr:+ tee ceph_test_neorados_cmd.log 2024-09-17T18:20:01.245 INFO:tasks.workunit.client.0.smithi173.stderr:+ sed 's/^/ cmd: /' 2024-09-17T18:20:01.245 INFO:tasks.workunit.client.0.smithi173.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-09-17T18:20:01.245 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -f /etc/bashrc ']' 2024-09-17T18:20:01.245 INFO:tasks.workunit.client.0.smithi173.stderr:+ . /etc/bashrc 2024-09-17T18:20:01.245 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'grep=grep --color=auto' 2024-09-17T18:20:01.245 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-17T18:20:01.245 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-17T18:20:01.245 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.245 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-17T18:20:01.245 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.245 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorls.sh 2024-09-17T18:20:01.245 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' '!' -t 0 ']' 2024-09-17T18:20:01.245 INFO:tasks.workunit.client.0.smithi173.stderr:+++ return 2024-09-17T18:20:01.245 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.246 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-17T18:20:01.246 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.246 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-17T18:20:01.246 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export S_COLORS=auto 2024-09-17T18:20:01.246 INFO:tasks.workunit.client.0.smithi173.stderr:+++ S_COLORS=auto 2024-09-17T18:20:01.246 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.246 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-17T18:20:01.246 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.246 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-17T18:20:01.246 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.246 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -z '' ']' 2024-09-17T18:20:01.246 INFO:tasks.workunit.client.0.smithi173.stderr:++ BASHRCSOURCED=Y 2024-09-17T18:20:01.246 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.246 INFO:tasks.workunit.client.0.smithi173.stderr:++ shopt -q login_shell 2024-09-17T18:20:01.246 INFO:tasks.workunit.client.0.smithi173.stderr:+++ umask 2024-09-17T18:20:01.247 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 0022 -eq 0 ']' 2024-09-17T18:20:01.247 INFO:tasks.workunit.client.0.smithi173.stderr:++ SHELL=/bin/bash 2024-09-17T18:20:01.247 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.247 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-17T18:20:01.247 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.247 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-17T18:20:01.247 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-17T18:20:01.247 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.247 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-17T18:20:01.247 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.247 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-17T18:20:01.247 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:01.247 INFO:tasks.workunit.client.0.smithi173.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:01.247 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.248 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-17T18:20:01.248 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.248 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-17T18:20:01.248 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.248 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-17T18:20:01.248 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-17T18:20:01.248 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-17T18:20:01.248 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.248 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-17T18:20:01.248 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.248 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-17T18:20:01.248 INFO:tasks.workunit.client.0.smithi173.stderr:++ echo pool 2024-09-17T18:20:01.248 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-17T18:20:01.249 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.249 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-17T18:20:01.249 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.249 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/gawk.sh 2024-09-17T18:20:01.249 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.249 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-17T18:20:01.249 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.249 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/lang.sh 2024-09-17T18:20:01.249 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup 2024-09-17T18:20:01.249 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-17T18:20:01.249 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-17T18:20:01.249 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-17T18:20:01.249 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.249 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-17T18:20:01.250 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.250 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-17T18:20:01.250 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '{print $1}' 2024-09-17T18:20:01.250 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-17T18:20:01.250 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.250 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/locale 2024-09-17T18:20:01.250 INFO:tasks.workunit.client.0.smithi173.stderr:+ ff=pool 2024-09-17T18:20:01.250 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-17T18:20:01.250 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-17T18:20:01.250 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-17T18:20:01.250 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.250 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-17T18:20:01.250 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.250 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-17T18:20:01.251 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=36115 2024-09-17T18:20:01.251 INFO:tasks.workunit.client.0.smithi173.stderr:+ echo 'test pool on pid 36115' 2024-09-17T18:20:01.251 INFO:tasks.workunit.client.0.smithi173.stdout:test pool on pid 36115 2024-09-17T18:20:01.260 INFO:tasks.workunit.client.0.smithi173.stderr:+ pids[$f]=36115 2024-09-17T18:20:01.260 INFO:tasks.workunit.client.0.smithi173.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-17T18:20:01.260 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1 -eq 1 ']' 2024-09-17T18:20:01.260 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-17T18:20:01.260 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.260 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-17T18:20:01.261 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.261 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/gawk.sh 2024-09-17T18:20:01.261 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.261 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-17T18:20:01.261 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.261 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/lang.sh 2024-09-17T18:20:01.261 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup 2024-09-17T18:20:01.261 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/locale 2024-09-17T18:20:01.261 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:01.261 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-17T18:20:01.261 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:01.261 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-17T18:20:01.261 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-17T18:20:01.261 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-17T18:20:01.262 INFO:tasks.workunit.client.0.smithi173.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-09-17T18:20:01.262 INFO:tasks.workunit.client.0.smithi173.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-17T18:20:01.262 INFO:tasks.workunit.client.0.smithi173.stderr:++++ export LANG=en_US.UTF-8 2024-09-17T18:20:01.262 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -f /etc/bashrc ']' 2024-09-17T18:20:01.262 INFO:tasks.workunit.client.0.smithi173.stderr:+ . /etc/bashrc 2024-09-17T18:20:01.262 INFO:tasks.workunit.client.0.smithi173.stderr:++++ LANG=en_US.UTF-8 2024-09-17T18:20:01.262 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:01.262 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-17T18:20:01.262 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:01.262 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG=en_US.UTF-8 2024-09-17T18:20:01.262 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup config 2024-09-17T18:20:01.262 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:01.262 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:01.263 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' dumb = linux ']' 2024-09-17T18:20:01.263 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.263 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-17T18:20:01.263 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.263 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/less.sh 2024-09-17T18:20:01.263 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-17T18:20:01.263 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.263 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-17T18:20:01.263 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.263 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/which2.sh 2024-09-17T18:20:01.263 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -z '' ']' 2024-09-17T18:20:01.263 INFO:tasks.workunit.client.0.smithi173.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-17T18:20:01.263 INFO:tasks.workunit.client.0.smithi173.stderr:++ BASHRCSOURCED=Y 2024-09-17T18:20:01.263 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.264 INFO:tasks.workunit.client.0.smithi173.stderr:++ shopt -q login_shell 2024-09-17T18:20:01.264 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'grep=grep --color=auto' 2024-09-17T18:20:01.264 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-17T18:20:01.264 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-17T18:20:01.264 INFO:tasks.workunit.client.0.smithi173.stderr:+++ umask 2024-09-17T18:20:01.264 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.264 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-17T18:20:01.264 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.264 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorls.sh 2024-09-17T18:20:01.264 INFO:tasks.workunit.client.0.smithi173.stderr:++ printf %25s read_operations 2024-09-17T18:20:01.264 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' '!' -t 0 ']' 2024-09-17T18:20:01.264 INFO:tasks.workunit.client.0.smithi173.stderr:+++ return 2024-09-17T18:20:01.264 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.265 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-17T18:20:01.265 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.265 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-17T18:20:01.265 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export S_COLORS=auto 2024-09-17T18:20:01.265 INFO:tasks.workunit.client.0.smithi173.stderr:+++ S_COLORS=auto 2024-09-17T18:20:01.265 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.265 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-17T18:20:01.265 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.265 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-17T18:20:01.265 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.265 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 0022 -eq 0 ']' 2024-09-17T18:20:01.265 INFO:tasks.workunit.client.0.smithi173.stderr:++ SHELL=/bin/bash 2024-09-17T18:20:01.265 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.265 INFO:tasks.workunit.client.0.smithi173.stderr:+ r=' read_operations' 2024-09-17T18:20:01.265 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-17T18:20:01.266 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.266 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-17T18:20:01.266 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-17T18:20:01.266 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.266 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-17T18:20:01.266 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.266 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-17T18:20:01.266 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:01.266 INFO:tasks.workunit.client.0.smithi173.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:01.266 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.266 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-17T18:20:01.266 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.266 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-17T18:20:01.266 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.267 INFO:tasks.workunit.client.0.smithi173.stderr:+++++ readlink /proc/35931/exe 2024-09-17T18:20:01.267 INFO:tasks.workunit.client.0.smithi173.stderr:++++ basename /usr/bin/bash 2024-09-17T18:20:01.267 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_declare='declare -f' 2024-09-17T18:20:01.267 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_opt=-f 2024-09-17T18:20:01.267 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export which_declare 2024-09-17T18:20:01.267 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export -f which 2024-09-17T18:20:01.267 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset i 2024-09-17T18:20:01.267 INFO:tasks.workunit.client.0.smithi173.stderr:++ echo read_operations 2024-09-17T18:20:01.267 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset -f pathmunge 2024-09-17T18:20:01.267 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-17T18:20:01.267 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-17T18:20:01.267 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-17T18:20:01.267 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '{print $1}' 2024-09-17T18:20:01.267 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.268 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-17T18:20:01.268 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.268 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-17T18:20:01.268 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-17T18:20:01.268 INFO:tasks.workunit.client.0.smithi173.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-17T18:20:01.268 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.268 INFO:tasks.workunit.client.0.smithi173.stderr:+ export PATH 2024-09-17T18:20:01.268 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-17T18:20:01.268 INFO:tasks.workunit.client.0.smithi173.stderr:+ unset rc 2024-09-17T18:20:01.268 INFO:tasks.workunit.client.0.smithi173.stderr:+ ceph_test_neorados_ec_list 2024-09-17T18:20:01.268 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:01.268 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-17T18:20:01.268 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:01.268 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-17T18:20:01.269 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-17T18:20:01.269 INFO:tasks.workunit.client.0.smithi173.stderr:+ ff=read_operations 2024-09-17T18:20:01.269 INFO:tasks.workunit.client.0.smithi173.stderr:+ tee ceph_test_neorados_ec_list.log 2024-09-17T18:20:01.269 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=36195 2024-09-17T18:20:01.269 INFO:tasks.workunit.client.0.smithi173.stderr:+ echo 'test read_operations on pid 36195' 2024-09-17T18:20:01.269 INFO:tasks.workunit.client.0.smithi173.stdout:test read_operations on pid 36195 2024-09-17T18:20:01.280 INFO:tasks.workunit.client.0.smithi173.stderr:+ pids[$f]=36195 2024-09-17T18:20:01.281 INFO:tasks.workunit.client.0.smithi173.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-17T18:20:01.281 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1 -eq 1 ']' 2024-09-17T18:20:01.281 INFO:tasks.workunit.client.0.smithi173.stderr:+ sed 's/^/ ec_list: /' 2024-09-17T18:20:01.281 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'grep=grep --color=auto' 2024-09-17T18:20:01.281 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-17T18:20:01.281 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-17T18:20:01.281 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.281 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-17T18:20:01.281 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.281 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorls.sh 2024-09-17T18:20:01.281 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' '!' -t 0 ']' 2024-09-17T18:20:01.281 INFO:tasks.workunit.client.0.smithi173.stderr:+++ return 2024-09-17T18:20:01.281 INFO:tasks.workunit.client.0.smithi173.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-09-17T18:20:01.281 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.282 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-17T18:20:01.282 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.282 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-17T18:20:01.282 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export S_COLORS=auto 2024-09-17T18:20:01.282 INFO:tasks.workunit.client.0.smithi173.stderr:+++ S_COLORS=auto 2024-09-17T18:20:01.282 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.282 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-17T18:20:01.282 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.282 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-17T18:20:01.282 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.282 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-17T18:20:01.282 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-17T18:20:01.282 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -f /etc/bashrc ']' 2024-09-17T18:20:01.282 INFO:tasks.workunit.client.0.smithi173.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-17T18:20:01.283 INFO:tasks.workunit.client.0.smithi173.stderr:+ . /etc/bashrc 2024-09-17T18:20:01.283 INFO:tasks.workunit.client.0.smithi173.stderr:++++ export LANG=en_US.UTF-8 2024-09-17T18:20:01.283 INFO:tasks.workunit.client.0.smithi173.stderr:++++ LANG=en_US.UTF-8 2024-09-17T18:20:01.283 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:01.283 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-17T18:20:01.283 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:01.283 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG=en_US.UTF-8 2024-09-17T18:20:01.283 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup config 2024-09-17T18:20:01.283 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -z '' ']' 2024-09-17T18:20:01.283 INFO:tasks.workunit.client.0.smithi173.stderr:++ BASHRCSOURCED=Y 2024-09-17T18:20:01.283 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.283 INFO:tasks.workunit.client.0.smithi173.stderr:++ shopt -q login_shell 2024-09-17T18:20:01.283 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:01.283 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-17T18:20:01.284 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:01.284 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' dumb = linux ']' 2024-09-17T18:20:01.284 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.284 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-17T18:20:01.284 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.284 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/less.sh 2024-09-17T18:20:01.284 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-17T18:20:01.284 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-17T18:20:01.284 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.284 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-17T18:20:01.284 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.284 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/which2.sh 2024-09-17T18:20:01.284 INFO:tasks.workunit.client.0.smithi173.stderr:++ printf %25s snapshots 2024-09-17T18:20:01.284 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.285 INFO:tasks.workunit.client.0.smithi173.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-17T18:20:01.285 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-17T18:20:01.285 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.285 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-17T18:20:01.285 INFO:tasks.workunit.client.0.smithi173.stderr:+ r=' snapshots' 2024-09-17T18:20:01.285 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-17T18:20:01.285 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.285 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-17T18:20:01.285 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.285 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/gawk.sh 2024-09-17T18:20:01.285 INFO:tasks.workunit.client.0.smithi173.stderr:+++ umask 2024-09-17T18:20:01.285 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.285 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-17T18:20:01.285 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.286 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/lang.sh 2024-09-17T18:20:01.286 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup 2024-09-17T18:20:01.286 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 0022 -eq 0 ']' 2024-09-17T18:20:01.286 INFO:tasks.workunit.client.0.smithi173.stderr:++ SHELL=/bin/bash 2024-09-17T18:20:01.286 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.286 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-17T18:20:01.286 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.286 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-17T18:20:01.286 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/locale 2024-09-17T18:20:01.286 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '{print $1}' 2024-09-17T18:20:01.286 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-17T18:20:01.286 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.286 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-17T18:20:01.286 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.287 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-17T18:20:01.287 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:01.287 INFO:tasks.workunit.client.0.smithi173.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:01.287 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.287 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-17T18:20:01.287 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.287 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-17T18:20:01.287 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.287 INFO:tasks.workunit.client.0.smithi173.stderr:+++++ readlink /proc/35829/exe 2024-09-17T18:20:01.287 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:01.287 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-17T18:20:01.287 INFO:tasks.workunit.client.0.smithi173.stderr:++ echo snapshots 2024-09-17T18:20:01.287 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:01.287 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-17T18:20:01.287 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-17T18:20:01.288 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-17T18:20:01.288 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-17T18:20:01.288 INFO:tasks.workunit.client.0.smithi173.stderr:++++ basename /usr/bin/bash 2024-09-17T18:20:01.288 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-17T18:20:01.288 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-17T18:20:01.288 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.288 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-17T18:20:01.288 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.288 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-17T18:20:01.288 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-17T18:20:01.288 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.288 INFO:tasks.workunit.client.0.smithi173.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-17T18:20:01.288 INFO:tasks.workunit.client.0.smithi173.stderr:++++ export LANG=en_US.UTF-8 2024-09-17T18:20:01.288 INFO:tasks.workunit.client.0.smithi173.stderr:++++ LANG=en_US.UTF-8 2024-09-17T18:20:01.289 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:01.289 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-17T18:20:01.289 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:01.289 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG=en_US.UTF-8 2024-09-17T18:20:01.289 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup config 2024-09-17T18:20:01.289 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:01.289 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_declare='declare -f' 2024-09-17T18:20:01.289 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_opt=-f 2024-09-17T18:20:01.289 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export which_declare 2024-09-17T18:20:01.289 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export -f which 2024-09-17T18:20:01.289 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset i 2024-09-17T18:20:01.289 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset -f pathmunge 2024-09-17T18:20:01.289 INFO:tasks.workunit.client.0.smithi173.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-17T18:20:01.289 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:01.290 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'grep=grep --color=auto' 2024-09-17T18:20:01.290 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' dumb = linux ']' 2024-09-17T18:20:01.290 INFO:tasks.workunit.client.0.smithi173.stderr:+ export PATH 2024-09-17T18:20:01.290 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.290 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-17T18:20:01.290 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-17T18:20:01.290 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.290 INFO:tasks.workunit.client.0.smithi173.stderr:+ unset rc 2024-09-17T18:20:01.290 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/less.sh 2024-09-17T18:20:01.290 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-17T18:20:01.290 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-17T18:20:01.290 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-17T18:20:01.290 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.290 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.290 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-17T18:20:01.291 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.291 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/which2.sh 2024-09-17T18:20:01.291 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-17T18:20:01.291 INFO:tasks.workunit.client.0.smithi173.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-17T18:20:01.291 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.291 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorls.sh 2024-09-17T18:20:01.291 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' '!' -t 0 ']' 2024-09-17T18:20:01.291 INFO:tasks.workunit.client.0.smithi173.stderr:+ ceph_test_neorados_list 2024-09-17T18:20:01.291 INFO:tasks.workunit.client.0.smithi173.stderr:+++ return 2024-09-17T18:20:01.291 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.291 INFO:tasks.workunit.client.0.smithi173.stderr:+ tee ceph_test_neorados_list.log 2024-09-17T18:20:01.291 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-17T18:20:01.291 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.292 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-17T18:20:01.292 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export S_COLORS=auto 2024-09-17T18:20:01.292 INFO:tasks.workunit.client.0.smithi173.stderr:+++ S_COLORS=auto 2024-09-17T18:20:01.292 INFO:tasks.workunit.client.0.smithi173.stderr:+ sed 's/^/ list: /' 2024-09-17T18:20:01.292 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.292 INFO:tasks.workunit.client.0.smithi173.stderr:+ ff=snapshots 2024-09-17T18:20:01.292 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-17T18:20:01.292 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.292 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-17T18:20:01.292 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=36260 2024-09-17T18:20:01.292 INFO:tasks.workunit.client.0.smithi173.stderr:+++++ readlink /proc/36020/exe 2024-09-17T18:20:01.292 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.292 INFO:tasks.workunit.client.0.smithi173.stderr:+ echo 'test snapshots on pid 36260' 2024-09-17T18:20:01.293 INFO:tasks.workunit.client.0.smithi173.stdout:test snapshots on pid 36260 2024-09-17T18:20:01.297 INFO:tasks.workunit.client.0.smithi173.stderr:+ pids[$f]=36260 2024-09-17T18:20:01.297 INFO:tasks.workunit.client.0.smithi173.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-17T18:20:01.297 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1 -eq 1 ']' 2024-09-17T18:20:01.297 INFO:tasks.workunit.client.0.smithi173.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-09-17T18:20:01.298 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-17T18:20:01.298 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-17T18:20:01.298 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-17T18:20:01.298 INFO:tasks.workunit.client.0.smithi173.stderr:++++ basename /usr/bin/bash 2024-09-17T18:20:01.298 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.298 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-17T18:20:01.298 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.298 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-17T18:20:01.298 INFO:tasks.workunit.client.0.smithi173.stderr:++ printf %25s watch_notify 2024-09-17T18:20:01.298 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-17T18:20:01.298 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.298 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-17T18:20:01.298 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.298 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/gawk.sh 2024-09-17T18:20:01.299 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.299 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-17T18:20:01.299 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.299 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/lang.sh 2024-09-17T18:20:01.299 INFO:tasks.workunit.client.0.smithi173.stderr:+ r=' watch_notify' 2024-09-17T18:20:01.299 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup 2024-09-17T18:20:01.299 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -f /etc/bashrc ']' 2024-09-17T18:20:01.299 INFO:tasks.workunit.client.0.smithi173.stderr:+ . /etc/bashrc 2024-09-17T18:20:01.299 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -z '' ']' 2024-09-17T18:20:01.299 INFO:tasks.workunit.client.0.smithi173.stderr:++ BASHRCSOURCED=Y 2024-09-17T18:20:01.299 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/locale 2024-09-17T18:20:01.299 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.299 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_declare='declare -f' 2024-09-17T18:20:01.299 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_opt=-f 2024-09-17T18:20:01.300 INFO:tasks.workunit.client.0.smithi173.stderr:++ echo watch_notify 2024-09-17T18:20:01.300 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '{print $1}' 2024-09-17T18:20:01.300 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export which_declare 2024-09-17T18:20:01.300 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export -f which 2024-09-17T18:20:01.300 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset i 2024-09-17T18:20:01.300 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset -f pathmunge 2024-09-17T18:20:01.300 INFO:tasks.workunit.client.0.smithi173.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-17T18:20:01.300 INFO:tasks.workunit.client.0.smithi173.stderr:++ shopt -q login_shell 2024-09-17T18:20:01.300 INFO:tasks.workunit.client.0.smithi173.stderr:+ export PATH 2024-09-17T18:20:01.300 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-17T18:20:01.300 INFO:tasks.workunit.client.0.smithi173.stderr:+ unset rc 2024-09-17T18:20:01.300 INFO:tasks.workunit.client.0.smithi173.stderr:+ tee ceph_test_neorados_misc.log 2024-09-17T18:20:01.300 INFO:tasks.workunit.client.0.smithi173.stderr:+ ceph_test_neorados_misc 2024-09-17T18:20:01.300 INFO:tasks.workunit.client.0.smithi173.stderr:+ ff=watch_notify 2024-09-17T18:20:01.301 INFO:tasks.workunit.client.0.smithi173.stderr:+ sed 's/^/ misc: /' 2024-09-17T18:20:01.301 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-17T18:20:01.301 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-17T18:20:01.301 INFO:tasks.workunit.client.0.smithi173.stderr:+++ umask 2024-09-17T18:20:01.301 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-17T18:20:01.301 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.301 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-17T18:20:01.301 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.301 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-17T18:20:01.301 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-17T18:20:01.301 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:01.301 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-17T18:20:01.301 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=36274 2024-09-17T18:20:01.301 INFO:tasks.workunit.client.0.smithi173.stderr:+ echo 'test watch_notify on pid 36274' 2024-09-17T18:20:01.302 INFO:tasks.workunit.client.0.smithi173.stdout:test watch_notify on pid 36274 2024-09-17T18:20:01.308 INFO:tasks.workunit.client.0.smithi173.stderr:+ pids[$f]=36274 2024-09-17T18:20:01.308 INFO:tasks.workunit.client.0.smithi173.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-17T18:20:01.308 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1 -eq 1 ']' 2024-09-17T18:20:01.308 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 0022 -eq 0 ']' 2024-09-17T18:20:01.308 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:01.308 INFO:tasks.workunit.client.0.smithi173.stderr:++ SHELL=/bin/bash 2024-09-17T18:20:01.308 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-17T18:20:01.308 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.309 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-17T18:20:01.309 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.309 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-17T18:20:01.309 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.309 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-17T18:20:01.309 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-17T18:20:01.309 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.309 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-17T18:20:01.309 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.309 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-17T18:20:01.309 INFO:tasks.workunit.client.0.smithi173.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-09-17T18:20:01.309 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -f /etc/bashrc ']' 2024-09-17T18:20:01.309 INFO:tasks.workunit.client.0.smithi173.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-17T18:20:01.309 INFO:tasks.workunit.client.0.smithi173.stderr:++++ export LANG=en_US.UTF-8 2024-09-17T18:20:01.310 INFO:tasks.workunit.client.0.smithi173.stderr:++++ LANG=en_US.UTF-8 2024-09-17T18:20:01.310 INFO:tasks.workunit.client.0.smithi173.stderr:+ . /etc/bashrc 2024-09-17T18:20:01.310 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:01.310 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-17T18:20:01.310 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:01.310 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG=en_US.UTF-8 2024-09-17T18:20:01.310 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup config 2024-09-17T18:20:01.310 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:01.310 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:01.310 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' dumb = linux ']' 2024-09-17T18:20:01.310 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.310 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-17T18:20:01.310 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.310 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/less.sh 2024-09-17T18:20:01.310 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-17T18:20:01.311 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.311 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-17T18:20:01.311 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.311 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/which2.sh 2024-09-17T18:20:01.311 INFO:tasks.workunit.client.0.smithi173.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-17T18:20:01.311 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -z '' ']' 2024-09-17T18:20:01.311 INFO:tasks.workunit.client.0.smithi173.stderr:++ BASHRCSOURCED=Y 2024-09-17T18:20:01.311 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.311 INFO:tasks.workunit.client.0.smithi173.stderr:++ shopt -q login_shell 2024-09-17T18:20:01.311 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-17T18:20:01.311 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:01.311 INFO:tasks.workunit.client.0.smithi173.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:01.311 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.311 INFO:tasks.workunit.client.0.smithi173.stderr:++ printf %25s write_operations 2024-09-17T18:20:01.312 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-17T18:20:01.312 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-17T18:20:01.312 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-17T18:20:01.312 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.312 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-17T18:20:01.312 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-17T18:20:01.312 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.312 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.312 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-17T18:20:01.312 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.312 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-17T18:20:01.312 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-17T18:20:01.312 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.313 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-17T18:20:01.313 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.313 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/gawk.sh 2024-09-17T18:20:01.313 INFO:tasks.workunit.client.0.smithi173.stderr:+++ umask 2024-09-17T18:20:01.313 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.313 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-17T18:20:01.313 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.313 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/lang.sh 2024-09-17T18:20:01.313 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup 2024-09-17T18:20:01.313 INFO:tasks.workunit.client.0.smithi173.stderr:+ r=' write_operations' 2024-09-17T18:20:01.313 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 0022 -eq 0 ']' 2024-09-17T18:20:01.313 INFO:tasks.workunit.client.0.smithi173.stderr:++ SHELL=/bin/bash 2024-09-17T18:20:01.313 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.313 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-17T18:20:01.313 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/locale 2024-09-17T18:20:01.314 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.314 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-17T18:20:01.314 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-17T18:20:01.314 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.314 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-17T18:20:01.314 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.314 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-17T18:20:01.314 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:01.314 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '{print $1}' 2024-09-17T18:20:01.314 INFO:tasks.workunit.client.0.smithi173.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:01.314 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.314 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-17T18:20:01.314 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.314 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-17T18:20:01.315 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.315 INFO:tasks.workunit.client.0.smithi173.stderr:+++++ readlink /proc/36115/exe 2024-09-17T18:20:01.315 INFO:tasks.workunit.client.0.smithi173.stderr:++ echo write_operations 2024-09-17T18:20:01.315 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:01.315 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-17T18:20:01.315 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:01.315 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-17T18:20:01.315 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-17T18:20:01.315 INFO:tasks.workunit.client.0.smithi173.stderr:++++ basename /usr/bin/bash 2024-09-17T18:20:01.315 INFO:tasks.workunit.client.0.smithi173.stderr:+ ff=write_operations 2024-09-17T18:20:01.315 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=36313 2024-09-17T18:20:01.315 INFO:tasks.workunit.client.0.smithi173.stdout:test write_operations on pid 36313 2024-09-17T18:20:01.340 INFO:tasks.workunit.client.0.smithi173.stderr:+ echo 'test write_operations on pid 36313' 2024-09-17T18:20:01.340 INFO:tasks.workunit.client.0.smithi173.stderr:+ pids[$f]=36313 2024-09-17T18:20:01.340 INFO:tasks.workunit.client.0.smithi173.stderr:+ ret=0 2024-09-17T18:20:01.340 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1 -eq 1 ']' 2024-09-17T18:20:01.340 INFO:tasks.workunit.client.0.smithi173.stderr:+ for t in "${!pids[@]}" 2024-09-17T18:20:01.340 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35089 2024-09-17T18:20:01.340 INFO:tasks.workunit.client.0.smithi173.stderr:+ wait 35089 2024-09-17T18:20:01.340 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_declare='declare -f' 2024-09-17T18:20:01.340 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_opt=-f 2024-09-17T18:20:01.341 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export which_declare 2024-09-17T18:20:01.341 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export -f which 2024-09-17T18:20:01.341 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset i 2024-09-17T18:20:01.341 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset -f pathmunge 2024-09-17T18:20:01.341 INFO:tasks.workunit.client.0.smithi173.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-17T18:20:01.341 INFO:tasks.workunit.client.0.smithi173.stderr:+ export PATH 2024-09-17T18:20:01.341 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-17T18:20:01.341 INFO:tasks.workunit.client.0.smithi173.stderr:+ unset rc 2024-09-17T18:20:01.341 INFO:tasks.workunit.client.0.smithi173.stderr:+ ceph_test_neorados_pool 2024-09-17T18:20:01.341 INFO:tasks.workunit.client.0.smithi173.stderr:+ sed 's/^/ pool: /' 2024-09-17T18:20:01.341 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'grep=grep --color=auto' 2024-09-17T18:20:01.341 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-17T18:20:01.341 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-17T18:20:01.341 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.342 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-17T18:20:01.342 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.342 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorls.sh 2024-09-17T18:20:01.342 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' '!' -t 0 ']' 2024-09-17T18:20:01.342 INFO:tasks.workunit.client.0.smithi173.stderr:+++ return 2024-09-17T18:20:01.342 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.342 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-17T18:20:01.342 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.342 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-17T18:20:01.342 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export S_COLORS=auto 2024-09-17T18:20:01.342 INFO:tasks.workunit.client.0.smithi173.stderr:+++ S_COLORS=auto 2024-09-17T18:20:01.342 INFO:tasks.workunit.client.0.smithi173.stderr:+ tee ceph_test_neorados_pool.log 2024-09-17T18:20:01.342 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.343 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-17T18:20:01.343 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'grep=grep --color=auto' 2024-09-17T18:20:01.343 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-17T18:20:01.343 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.343 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-17T18:20:01.343 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-17T18:20:01.343 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.343 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-17T18:20:01.343 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.343 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorls.sh 2024-09-17T18:20:01.343 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-17T18:20:01.343 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' '!' -t 0 ']' 2024-09-17T18:20:01.343 INFO:tasks.workunit.client.0.smithi173.stderr:+++ return 2024-09-17T18:20:01.343 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.344 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-17T18:20:01.344 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.344 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-17T18:20:01.344 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.344 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export S_COLORS=auto 2024-09-17T18:20:01.344 INFO:tasks.workunit.client.0.smithi173.stderr:+++ S_COLORS=auto 2024-09-17T18:20:01.344 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.344 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-17T18:20:01.344 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.344 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-17T18:20:01.344 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.344 INFO:tasks.workunit.client.0.smithi173.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-09-17T18:20:01.344 INFO:tasks.workunit.client.0.smithi173.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-17T18:20:01.344 INFO:tasks.workunit.client.0.smithi173.stderr:++++ export LANG=en_US.UTF-8 2024-09-17T18:20:01.345 INFO:tasks.workunit.client.0.smithi173.stderr:++++ LANG=en_US.UTF-8 2024-09-17T18:20:01.345 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:01.345 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-17T18:20:01.345 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:01.345 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG=en_US.UTF-8 2024-09-17T18:20:01.345 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup config 2024-09-17T18:20:01.345 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:01.345 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -f /etc/bashrc ']' 2024-09-17T18:20:01.345 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:01.345 INFO:tasks.workunit.client.0.smithi173.stderr:+ . /etc/bashrc 2024-09-17T18:20:01.345 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' dumb = linux ']' 2024-09-17T18:20:01.345 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.345 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -z '' ']' 2024-09-17T18:20:01.345 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-17T18:20:01.346 INFO:tasks.workunit.client.0.smithi173.stderr:++ BASHRCSOURCED=Y 2024-09-17T18:20:01.346 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.346 INFO:tasks.workunit.client.0.smithi173.stderr:++ shopt -q login_shell 2024-09-17T18:20:01.346 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.346 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/less.sh 2024-09-17T18:20:01.346 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-17T18:20:01.346 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.346 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-17T18:20:01.346 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.346 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/which2.sh 2024-09-17T18:20:01.346 INFO:tasks.workunit.client.0.smithi173.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-17T18:20:01.346 INFO:tasks.workunit.client.0.smithi173.stderr:+++ umask 2024-09-17T18:20:01.346 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 0022 -eq 0 ']' 2024-09-17T18:20:01.346 INFO:tasks.workunit.client.0.smithi173.stderr:++ SHELL=/bin/bash 2024-09-17T18:20:01.346 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-17T18:20:01.347 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.347 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-17T18:20:01.347 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-17T18:20:01.347 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.347 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-17T18:20:01.347 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-17T18:20:01.347 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-17T18:20:01.347 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.347 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-17T18:20:01.347 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.347 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-17T18:20:01.347 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-17T18:20:01.347 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.348 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-17T18:20:01.348 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.348 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-17T18:20:01.348 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-17T18:20:01.348 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-17T18:20:01.348 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:01.348 INFO:tasks.workunit.client.0.smithi173.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-17T18:20:01.348 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.348 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.348 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-17T18:20:01.348 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.348 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-17T18:20:01.348 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.349 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-17T18:20:01.349 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.349 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-17T18:20:01.349 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.349 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-17T18:20:01.349 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-17T18:20:01.349 INFO:tasks.workunit.client.0.smithi173.stderr:+++++ readlink /proc/36195/exe 2024-09-17T18:20:01.349 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.349 INFO:tasks.workunit.client.0.smithi173.stderr:++++ basename /usr/bin/bash 2024-09-17T18:20:01.349 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_declare='declare -f' 2024-09-17T18:20:01.349 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_opt=-f 2024-09-17T18:20:01.349 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export which_declare 2024-09-17T18:20:01.349 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export -f which 2024-09-17T18:20:01.349 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset i 2024-09-17T18:20:01.350 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset -f pathmunge 2024-09-17T18:20:01.350 INFO:tasks.workunit.client.0.smithi173.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-17T18:20:01.350 INFO:tasks.workunit.client.0.smithi173.stderr:+ export PATH 2024-09-17T18:20:01.350 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-17T18:20:01.350 INFO:tasks.workunit.client.0.smithi173.stderr:+ unset rc 2024-09-17T18:20:01.350 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'grep=grep --color=auto' 2024-09-17T18:20:01.350 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-17T18:20:01.350 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-17T18:20:01.350 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-17T18:20:01.350 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-17T18:20:01.350 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.350 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-17T18:20:01.350 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-17T18:20:01.350 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.351 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-17T18:20:01.351 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorls.sh 2024-09-17T18:20:01.351 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.351 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' '!' -t 0 ']' 2024-09-17T18:20:01.351 INFO:tasks.workunit.client.0.smithi173.stderr:+++ return 2024-09-17T18:20:01.351 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.351 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-17T18:20:01.351 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-17T18:20:01.351 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.351 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-17T18:20:01.351 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-17T18:20:01.351 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.351 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export S_COLORS=auto 2024-09-17T18:20:01.351 INFO:tasks.workunit.client.0.smithi173.stderr:+++ S_COLORS=auto 2024-09-17T18:20:01.352 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-17T18:20:01.352 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.352 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.352 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-17T18:20:01.352 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-17T18:20:01.352 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.352 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-17T18:20:01.352 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.352 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-17T18:20:01.352 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.352 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-17T18:20:01.352 INFO:tasks.workunit.client.0.smithi173.stderr:+ ceph_test_neorados_read_operations 2024-09-17T18:20:01.352 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-17T18:20:01.353 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.353 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-17T18:20:01.353 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.353 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/gawk.sh 2024-09-17T18:20:01.353 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-17T18:20:01.353 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.353 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-17T18:20:01.353 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.353 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/gawk.sh 2024-09-17T18:20:01.353 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.353 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.353 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-17T18:20:01.353 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.353 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/lang.sh 2024-09-17T18:20:01.354 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup 2024-09-17T18:20:01.354 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-17T18:20:01.354 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.354 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/lang.sh 2024-09-17T18:20:01.354 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup 2024-09-17T18:20:01.354 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/locale 2024-09-17T18:20:01.354 INFO:tasks.workunit.client.0.smithi173.stderr:+ sed 's/^/ read_operations: /' 2024-09-17T18:20:01.354 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/locale 2024-09-17T18:20:01.354 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:01.354 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-17T18:20:01.354 INFO:tasks.workunit.client.0.smithi173.stderr:+ tee ceph_test_neorados_read_operations.log 2024-09-17T18:20:01.354 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:01.354 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-17T18:20:01.354 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-17T18:20:01.355 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-17T18:20:01.355 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-17T18:20:01.355 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-17T18:20:01.355 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.355 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-17T18:20:01.355 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:01.355 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-17T18:20:01.355 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.355 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:01.355 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-17T18:20:01.355 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-17T18:20:01.355 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-17T18:20:01.355 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-17T18:20:01.355 INFO:tasks.workunit.client.0.smithi173.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-17T18:20:01.356 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-17T18:20:01.356 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-17T18:20:01.356 INFO:tasks.workunit.client.0.smithi173.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-17T18:20:01.356 INFO:tasks.workunit.client.0.smithi173.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-17T18:20:01.356 INFO:tasks.workunit.client.0.smithi173.stderr:++++ export LANG=en_US.UTF-8 2024-09-17T18:20:01.356 INFO:tasks.workunit.client.0.smithi173.stderr:++++ LANG=en_US.UTF-8 2024-09-17T18:20:01.356 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:01.356 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-17T18:20:01.356 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:01.356 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG=en_US.UTF-8 2024-09-17T18:20:01.356 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup config 2024-09-17T18:20:01.356 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:01.356 INFO:tasks.workunit.client.0.smithi173.stderr:++++ export LANG=en_US.UTF-8 2024-09-17T18:20:01.356 INFO:tasks.workunit.client.0.smithi173.stderr:++++ LANG=en_US.UTF-8 2024-09-17T18:20:01.357 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:01.357 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-17T18:20:01.357 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:01.357 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG=en_US.UTF-8 2024-09-17T18:20:01.357 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup config 2024-09-17T18:20:01.357 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:01.357 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' dumb = linux ']' 2024-09-17T18:20:01.357 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:01.357 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.357 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-17T18:20:01.357 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.357 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/less.sh 2024-09-17T18:20:01.357 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:01.357 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-17T18:20:01.358 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.358 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-17T18:20:01.358 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.358 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/which2.sh 2024-09-17T18:20:01.358 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' dumb = linux ']' 2024-09-17T18:20:01.358 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.358 INFO:tasks.workunit.client.0.smithi173.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-17T18:20:01.358 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-17T18:20:01.358 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.358 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/less.sh 2024-09-17T18:20:01.358 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-17T18:20:01.358 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.358 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-17T18:20:01.359 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.359 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/which2.sh 2024-09-17T18:20:01.359 INFO:tasks.workunit.client.0.smithi173.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-17T18:20:01.359 INFO:tasks.workunit.client.0.smithi173.stderr:+++++ readlink /proc/36260/exe 2024-09-17T18:20:01.359 INFO:tasks.workunit.client.0.smithi173.stderr:+++++ readlink /proc/36274/exe 2024-09-17T18:20:01.359 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-17T18:20:01.359 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-17T18:20:01.359 INFO:tasks.workunit.client.0.smithi173.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-17T18:20:01.359 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.359 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-17T18:20:01.359 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.359 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-17T18:20:01.359 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-17T18:20:01.359 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.360 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-17T18:20:01.360 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.360 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/gawk.sh 2024-09-17T18:20:01.360 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.360 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-17T18:20:01.360 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.360 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/lang.sh 2024-09-17T18:20:01.360 INFO:tasks.workunit.client.0.smithi173.stderr:++++ basename /usr/bin/bash 2024-09-17T18:20:01.360 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup 2024-09-17T18:20:01.360 INFO:tasks.workunit.client.0.smithi173.stderr:++++ basename /usr/bin/bash 2024-09-17T18:20:01.360 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/locale 2024-09-17T18:20:01.360 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_declare='declare -f' 2024-09-17T18:20:01.360 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_opt=-f 2024-09-17T18:20:01.360 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export which_declare 2024-09-17T18:20:01.360 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export -f which 2024-09-17T18:20:01.361 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset i 2024-09-17T18:20:01.361 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset -f pathmunge 2024-09-17T18:20:01.361 INFO:tasks.workunit.client.0.smithi173.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-17T18:20:01.361 INFO:tasks.workunit.client.0.smithi173.stderr:+ export PATH 2024-09-17T18:20:01.361 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-17T18:20:01.361 INFO:tasks.workunit.client.0.smithi173.stderr:+ unset rc 2024-09-17T18:20:01.361 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_declare='declare -f' 2024-09-17T18:20:01.361 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_opt=-f 2024-09-17T18:20:01.361 INFO:tasks.workunit.client.0.smithi173.stderr:+ ceph_test_neorados_snapshots 2024-09-17T18:20:01.361 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:01.361 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export which_declare 2024-09-17T18:20:01.361 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-17T18:20:01.361 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export -f which 2024-09-17T18:20:01.361 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:01.362 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-17T18:20:01.362 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-17T18:20:01.362 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset i 2024-09-17T18:20:01.362 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset -f pathmunge 2024-09-17T18:20:01.362 INFO:tasks.workunit.client.0.smithi173.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-17T18:20:01.362 INFO:tasks.workunit.client.0.smithi173.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-17T18:20:01.362 INFO:tasks.workunit.client.0.smithi173.stderr:+ export PATH 2024-09-17T18:20:01.362 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-17T18:20:01.362 INFO:tasks.workunit.client.0.smithi173.stderr:+ unset rc 2024-09-17T18:20:01.362 INFO:tasks.workunit.client.0.smithi173.stderr:+ sed 's/^/ snapshots: /' 2024-09-17T18:20:01.362 INFO:tasks.workunit.client.0.smithi173.stderr:+ ceph_test_neorados_watch_notify 2024-09-17T18:20:01.362 INFO:tasks.workunit.client.0.smithi173.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-17T18:20:01.362 INFO:tasks.workunit.client.0.smithi173.stderr:++++ export LANG=en_US.UTF-8 2024-09-17T18:20:01.362 INFO:tasks.workunit.client.0.smithi173.stderr:++++ LANG=en_US.UTF-8 2024-09-17T18:20:01.363 INFO:tasks.workunit.client.0.smithi173.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-17T18:20:01.363 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-17T18:20:01.363 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:01.363 INFO:tasks.workunit.client.0.smithi173.stderr:+++ LANG=en_US.UTF-8 2024-09-17T18:20:01.363 INFO:tasks.workunit.client.0.smithi173.stderr:+++ unset LANG_backup config 2024-09-17T18:20:01.363 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n '' ']' 2024-09-17T18:20:01.363 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-17T18:20:01.363 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' dumb = linux ']' 2024-09-17T18:20:01.363 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.363 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-17T18:20:01.363 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.363 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/less.sh 2024-09-17T18:20:01.363 INFO:tasks.workunit.client.0.smithi173.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-17T18:20:01.363 INFO:tasks.workunit.client.0.smithi173.stderr:++ for i in /etc/profile.d/*.sh 2024-09-17T18:20:01.363 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-17T18:20:01.364 INFO:tasks.workunit.client.0.smithi173.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-09-17T18:20:01.364 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' '' ']' 2024-09-17T18:20:01.364 INFO:tasks.workunit.client.0.smithi173.stderr:++ . /etc/profile.d/which2.sh 2024-09-17T18:20:01.364 INFO:tasks.workunit.client.0.smithi173.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-17T18:20:01.364 INFO:tasks.workunit.client.0.smithi173.stderr:+ tee ceph_test_neorados_snapshots.log 2024-09-17T18:20:01.364 INFO:tasks.workunit.client.0.smithi173.stderr:+ sed 's/^/ watch_notify: /' 2024-09-17T18:20:01.364 INFO:tasks.workunit.client.0.smithi173.stderr:+++++ readlink /proc/36313/exe 2024-09-17T18:20:01.364 INFO:tasks.workunit.client.0.smithi173.stderr:++++ basename /usr/bin/bash 2024-09-17T18:20:01.364 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_declare='declare -f' 2024-09-17T18:20:01.364 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which_opt=-f 2024-09-17T18:20:01.364 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export which_declare 2024-09-17T18:20:01.364 INFO:tasks.workunit.client.0.smithi173.stderr:+++ export -f which 2024-09-17T18:20:01.364 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset i 2024-09-17T18:20:01.364 INFO:tasks.workunit.client.0.smithi173.stderr:++ unset -f pathmunge 2024-09-17T18:20:01.365 INFO:tasks.workunit.client.0.smithi173.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-17T18:20:01.365 INFO:tasks.workunit.client.0.smithi173.stderr:+ export PATH 2024-09-17T18:20:01.365 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-17T18:20:01.365 INFO:tasks.workunit.client.0.smithi173.stderr:+ unset rc 2024-09-17T18:20:01.365 INFO:tasks.workunit.client.0.smithi173.stderr:+ ceph_test_neorados_write_operations 2024-09-17T18:20:01.365 INFO:tasks.workunit.client.0.smithi173.stderr:+ tee ceph_test_neorados_write_operations.log 2024-09-17T18:20:01.365 INFO:tasks.workunit.client.0.smithi173.stderr:+ sed 's/^/ write_operations: /' 2024-09-17T18:20:01.365 INFO:tasks.workunit.client.0.smithi173.stdout:ent.4413.objecter pool_op_submit 1 2024-09-17T18:20:01.365 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.699+0000 7f2fc5d77780 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:20:01.366 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.699+0000 7f2fc5d77780 1 -- 172.21.15.173:0/56431533 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) -- 0x5588a7abed70 con 0x5588a7b1fa60 2024-09-17T18:20:01.366 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.699+0000 7f2fc3064640 1 --2- 172.21.15.173:0/56431533 >> v2:172.21.15.173:6812/2423786037 conn(0x7f2fb003e390 0x7f2fb0040850 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2fb800b5f0 tx=0x7f2fb8009000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:20:01.366 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:00.699+0000 7f2fc1060640 10 client.4413.objecter ms_handle_connect 0x7f2fb003e390 2024-09-17T18:20:01.366 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:01.109+0000 7f2fc1060640 1 -- 172.21.15.173:0/56431533 <== mon.0 v2:172.21.15.121:3300/0 7 ==== pool_op_reply(tid 1 (0) Success v18) ==== 43+0+0 (secure 0 0 0) 0x7f2fb401cd70 con 0x5588a7b1fa60 2024-09-17T18:20:01.366 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:01.109+0000 7f2fc1060640 10 client.4413.objecter ms_dispatch 0x5588a7ab1240 pool_op_reply(tid 1 (0) Success v18) 2024-09-17T18:20:01.366 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:01.109+0000 7f2fc1060640 10 client.4413.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v18) 2024-09-17T18:20:01.366 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:01.109+0000 7f2fc1060640 10 client.4413.objecter have request 1 at 0x5588a7c71cd0 Op: create 2024-09-17T18:20:01.366 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:01.109+0000 7f2fc1060640 20 client.4413.objecter waiting for client to reach epoch 18 before calling back 2024-09-17T18:20:01.366 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:01.109+0000 7f2fc1060640 10 client.4413.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:20:01.366 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:01.109+0000 7f2fc1060640 10 monclient: _renew_subs 2024-09-17T18:20:01.366 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:01.109+0000 7f2fc1060640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:20:01.366 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:01.109+0000 7f2fc1060640 1 -- 172.21.15.173:0/56431533 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=18}) -- 0x7f2fb00437b0 con 0x5588a7b1fa60 2024-09-17T18:20:01.366 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:01.109+0000 7f2fc1060640 10 client.4413.objecter done 2024-09-17T18:20:01.367 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:01.122+0000 7f2fc1060640 1 -- 172.21.15.173:0/56431533 <== mon.0 v2:172.21.15.121:3300/0 8 ==== osd_map(18..18 src has 1..18) ==== 11650+0+0 (secure 0 0 0) 0x7f2fb4060da0 con 0x5588a7b1fa60 2024-09-17T18:20:01.367 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:01.122+0000 7f2fc1060640 10 client.4413.objecter ms_dispatch 0x5588a7ab1240 osd_map(18..18 src has 1..18) 2024-09-17T18:20:01.367 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:01.122+0000 7f2fc1060640 3 client.4413.objecter handle_osd_map got epochs [18,18] > 17 2024-09-17T18:20:01.367 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:01.122+0000 7f2fc1060640 3 client.4413.objecter handle_osd_map decoding incremental epoch 18 2024-09-17T18:20:01.367 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:01.123+0000 7f2fc1060640 20 client.4413.objecter dump_active .. 0 homeless 2024-09-17T18:20:01.367 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:01.131+0000 7f2fc5d77780 10 librados: set snap read head -> 1 2024-09-17T18:20:01.367 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:01.131+0000 7f2fc5d77780 10 librados: writefull oid=exist nspace= 2024-09-17T18:20:01.367 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:01.131+0000 7f2fc5d77780 10 client.4413.objecter _op_submit op 0x5588a7c706c0 2024-09-17T18:20:01.367 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:01.131+0000 7f2fc5d77780 20 client.4413.objecter _calc_target epoch 18 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2024-09-17T18:20:01.367 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:01.131+0000 7f2fc5d77780 20 client.4413.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-17T18:20:01.367 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:01.131+0000 7f2fc5d77780 10 client.4413.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [3,0] primary 3 2024-09-17T18:20:02.320 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:01.131+0000 7f2fc5d77780 1 --2- 172.21.15.173:0/56431533 >> v2:172.21.15.173:6800/3493964597 conn(0x5588a7c86f10 0x5588a7c89300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:20:02.321 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:01.131+0000 7f2fc5d77780 20 client.4413.objecter _get_session s=0x5588a7b33720 osd=3 3 2024-09-17T18:20:02.321 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:01.131+0000 7f2fc5d77780 10 client.4413.objecter _op_submit oid exist '@5' '@5' [writefull 0~5 in=5b] tid 2 osd.3 2024-09-17T18:20:02.321 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:01.131+0000 7f2fc5d77780 20 client.4413.objecter get_session s=0x5588a7b33720 osd=3 3 2024-09-17T18:20:02.321 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:01.131+0000 7f2fc5d77780 15 client.4413.objecter _session_op_assign 3 2 2024-09-17T18:20:02.321 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:01.131+0000 7f2fc5d77780 15 client.4413.objecter _send_op 2 to 5.1 on osd.3 2024-09-17T18:20:02.321 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:01.131+0000 7f2fc5d77780 1 -- 172.21.15.173:0/56431533 --> v2:172.21.15.173:6800/3493964597 -- osd_op(unknown.0.0:2 5.1 5:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18) -- 0x5588a7b1f010 con 0x5588a7c86f10 2024-09-17T18:20:02.321 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:01.131+0000 7f2fc5d77780 20 client.4413.objecter put_session s=0x5588a7b33720 osd=3 4 2024-09-17T18:20:02.321 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:01.131+0000 7f2fc5d77780 5 client.4413.objecter 1 in flight 2024-09-17T18:20:02.321 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:01.132+0000 7f2fc4066640 1 --2- 172.21.15.173:0/56431533 >> v2:172.21.15.173:6800/3493964597 conn(0x5588a7c86f10 0x5588a7c89300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:20:02.321 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:01.132+0000 7f2fc4066640 10 monclient: get_auth_request con 0x5588a7c86f10 auth_method 0 2024-09-17T18:20:02.321 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:01.138+0000 7f2fc4066640 1 --2- 172.21.15.173:0/56431533 >> v2:172.21.15.173:6800/3493964597 conn(0x5588a7c86f10 0x5588a7c89300 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:20:02.321 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:01.138+0000 7f2fc1060640 10 client.4413.objecter ms_handle_connect 0x5588a7c86f10 2024-09-17T18:20:02.321 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.259+0000 7f2fc1060640 1 -- 172.21.15.173:0/56431533 <== osd.3 v2:172.21.15.173:6800/3493964597 1 ==== osd_map(19..19 src has 1..19) ==== 9540+0+0 (crc 0 0 0) 0x7f2fbc002690 con 0x5588a7c86f10 2024-09-17T18:20:02.322 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.259+0000 7f2fc1060640 10 client.4413.objecter ms_dispatch 0x5588a7ab1240 osd_map(19..19 src has 1..19) 2024-09-17T18:20:02.322 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.259+0000 7f2fc1060640 3 client.4413.objecter handle_osd_map got epochs [19,19] > 18 2024-09-17T18:20:02.322 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.259+0000 7f2fc1060640 3 client.4413.objecter handle_osd_map decoding incremental epoch 19 2024-09-17T18:20:02.322 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.259+0000 7f2fc1060640 10 client.4413.objecter checking op 2 2024-09-17T18:20:02.322 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.259+0000 7f2fc1060640 20 client.4413.objecter _calc_target epoch 19 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2024-09-17T18:20:02.322 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.259+0000 7f2fc1060640 20 client.4413.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-17T18:20:02.322 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.259+0000 7f2fc1060640 20 client.4413.objecter dump_active .. 0 homeless 2024-09-17T18:20:02.322 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.259+0000 7f2fc1060640 20 client.4413.objecter 2 5.d3a59201 osd.3 exist [writefull 0~5 in=5b] 2024-09-17T18:20:02.322 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.318+0000 7f2fc4066640 1 -- 172.21.15.173:0/56431533 <== osd.3 v2:172.21.15.173:6800/3493964597 2 ==== osd_op_reply(2 exist [writefull 0~5] v19'1 uv1 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f2fbc002890 con 0x5588a7c86f10 2024-09-17T18:20:02.322 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.318+0000 7f2fc4066640 10 client.4413.objecter ms_dispatch 0x5588a7ab1240 osd_op_reply(2 exist [writefull 0~5] v19'1 uv1 ondisk = 0) 2024-09-17T18:20:02.322 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.318+0000 7f2fc4066640 10 client.4413.objecter in handle_osd_op_reply 2024-09-17T18:20:02.322 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.318+0000 7f2fc4066640 7 client.4413.objecter handle_osd_op_reply 2 ondisk uv 1 in 5.1 attempt 0 2024-09-17T18:20:02.322 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.318+0000 7f2fc4066640 10 client.4413.objecter op 0 rval 0 len 0 2024-09-17T18:20:02.322 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.318+0000 7f2fc4066640 15 client.4413.objecter handle_osd_op_reply completed tid 2 2024-09-17T18:20:02.323 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.318+0000 7f2fc4066640 15 client.4413.objecter _finish_op 2 2024-09-17T18:20:02.323 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.318+0000 7f2fc4066640 20 client.4413.objecter put_session s=0x5588a7b33720 osd=3 4 2024-09-17T18:20:02.323 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.318+0000 7f2fc4066640 15 client.4413.objecter _session_op_remove 3 2 2024-09-17T18:20:02.323 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.318+0000 7f2fc4066640 5 client.4413.objecter 0 in flight 2024-09-17T18:20:02.323 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.318+0000 7f2fc5d77780 10 librados: Objecter returned from writefull r=0 2024-09-17T18:20:02.323 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-09-17T18:20:02.323 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.318+0000 7f2fc5d77780 10 client.4413.objecter _op_submit op 0x5588a7c706c0 2024-09-17T18:20:02.323 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.318+0000 7f2fc5d77780 20 client.4413.objecter _calc_target epoch 19 base exist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-17T18:20:02.323 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.318+0000 7f2fc5d77780 20 client.4413.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-17T18:20:02.323 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.318+0000 7f2fc5d77780 10 client.4413.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [3,0] primary 3 2024-09-17T18:20:02.323 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.318+0000 7f2fc5d77780 20 client.4413.objecter _get_session s=0x5588a7b33720 osd=3 3 2024-09-17T18:20:02.323 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.318+0000 7f2fc5d77780 10 client.4413.objecter _op_submit oid exist '@5' '@5' [read 0~256] tid 3 osd.3 2024-09-17T18:20:02.323 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.318+0000 7f2fc5d77780 20 client.4413.objecter get_session s=0x5588a7b33720 osd=3 3 2024-09-17T18:20:02.323 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.318+0000 7f2fc5d77780 15 client.4413.objecter _session_op_assign 3 3 2024-09-17T18:20:02.324 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.318+0000 7f2fc5d77780 15 client.4413.objecter _send_op 3 to 5.1 on osd.3 2024-09-17T18:20:02.324 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.318+0000 7f2fc5d77780 1 -- 172.21.15.173:0/56431533 --> v2:172.21.15.173:6800/3493964597 -- osd_op(unknown.0.0:3 5.1 5:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19) -- 0x5588a7bc11c0 con 0x5588a7c86f10 2024-09-17T18:20:02.324 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.318+0000 7f2fc5d77780 20 client.4413.objecter put_session s=0x5588a7b33720 osd=3 4 2024-09-17T18:20:02.324 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.318+0000 7f2fc5d77780 5 client.4413.objecter 1 in flight 2024-09-17T18:20:02.324 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.318+0000 7f2fc5d77780 10 client.4413.objecter _op_submit op 0x5588a7bc15d0 2024-09-17T18:20:02.324 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.318+0000 7f2fc5d77780 20 client.4413.objecter _calc_target epoch 19 base noexist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-17T18:20:02.324 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.318+0000 7f2fc5d77780 20 client.4413.objecter _calc_target target noexist @5 -> pgid 5.a7bbaa1 2024-09-17T18:20:02.324 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.318+0000 7f2fc5d77780 10 client.4413.objecter _calc_target raw pgid 5.a7bbaa1 -> actual 5.1 acting [3,0] primary 3 2024-09-17T18:20:02.324 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.318+0000 7f2fc5d77780 20 client.4413.objecter _get_session s=0x5588a7b33720 osd=3 4 2024-09-17T18:20:02.324 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.318+0000 7f2fc5d77780 10 client.4413.objecter _op_submit oid noexist '@5' '@5' [read 0~256] tid 4 osd.3 2024-09-17T18:20:02.324 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.318+0000 7f2fc5d77780 20 client.4413.objecter get_session s=0x5588a7b33720 osd=3 4 2024-09-17T18:20:02.324 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.318+0000 7f2fc5d77780 15 client.4413.objecter _session_op_assign 3 4 2024-09-17T18:20:02.325 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.318+0000 7f2fc5d77780 15 client.4413.objecter _send_op 4 to 5.1 on osd.3 2024-09-17T18:20:02.325 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.318+0000 7f2fc5d77780 1 -- 172.21.15.173:0/56431533 --> v2:172.21.15.173:6800/3493964597 -- osd_op(unknown.0.0:4 5.1 5:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19) -- 0x5588a7bc1a80 con 0x5588a7c86f10 2024-09-17T18:20:02.325 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.318+0000 7f2fc5d77780 20 client.4413.objecter put_session s=0x5588a7b33720 osd=3 5 2024-09-17T18:20:02.325 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.318+0000 7f2fc5d77780 5 client.4413.objecter 2 in flight 2024-09-17T18:20:02.326 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.320+0000 7f2fc4066640 1 -- 172.21.15.173:0/56431533 <== osd.3 v2:172.21.15.173:6800/3493964597 3 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f2fbc002890 con 0x5588a7c86f10 2024-09-17T18:20:02.326 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.320+0000 7f2fc4066640 10 client.4413.objecter ms_dispatch 0x5588a7ab1240 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-09-17T18:20:02.326 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.320+0000 7f2fc4066640 10 client.4413.objecter in handle_osd_op_reply 2024-09-17T18:20:02.326 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.320+0000 7f2fc4066640 7 client.4413.objecter handle_osd_op_reply 3 ondisk uv 1 in 5.1 attempt 0 2024-09-17T18:20:02.326 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.320+0000 7f2fc4066640 10 client.4413.objecter op 0 rval 0 len 5 2024-09-17T18:20:02.326 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.320+0000 7f2fc4066640 15 client.4413.objecter handle_osd_op_reply completed tid 3 2024-09-17T18:20:02.326 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.320+0000 7f2fc4066640 15 client.4413.objecter _finish_op 3 2024-09-17T18:20:02.326 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.320+0000 7f2fc4066640 20 client.4413.objecter put_session s=0x5588a7b33720 osd=3 5 2024-09-17T18:20:02.326 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.320+0000 7f2fc4066640 15 client.4413.objecter _session_op_remove 3 3 2024-09-17T18:20:02.326 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.320+0000 7f2fc4066640 5 client.4413.objecter 1 in flight 2024-09-17T18:20:02.326 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.320+0000 7f2fc4066640 1 -- 172.21.15.173:0/56431533 <== osd.3 v2:172.21.15.173:6800/3493964597 4 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f2fbc002890 con 0x5588a7c86f10 2024-09-17T18:20:02.326 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.320+0000 7f2fc4066640 10 client.4413.objecter ms_dispatch 0x5588a7ab1240 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-17T18:20:02.326 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.320+0000 7f2fc4066640 10 client.4413.objecter in handle_osd_op_reply 2024-09-17T18:20:02.326 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (2 ms) 2024-09-17T18:20:02.327 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.320+0000 7f2fc4066640 7 client.4413.objecter handle_osd_op_reply 4 ondisk uv 0 in 5.1 attempt 0 2024-09-17T18:20:02.327 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-09-17T18:20:02.327 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.320+0000 7f2fc4066640 10 client.4413.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-17T18:20:02.327 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.320+0000 7f2fc4066640 10 client.4413.objecter op 0 rval 0 len 0 2024-09-17T18:20:02.327 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.320+0000 7f2fc4066640 15 client.4413.objecter handle_osd_op_reply completed tid 4 2024-09-17T18:20:02.327 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.320+0000 7f2fc4066640 15 client.4413.objecter _finish_op 4 2024-09-17T18:20:02.327 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.320+0000 7f2fc4066640 20 client.4413.objecter put_session s=0x5588a7b33720 osd=3 4 2024-09-17T18:20:02.327 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.320+0000 7f2fc4066640 15 client.4413.objecter _session_op_remove 3 4 2024-09-17T18:20:02.327 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.320+0000 7f2fc4066640 5 client.4413.objecter 0 in flight 2024-09-17T18:20:02.327 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.320+0000 7f2fc5d77780 10 client.4413.objecter _op_submit op 0x5588a7bc15d0 2024-09-17T18:20:02.327 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.320+0000 7f2fc5d77780 20 client.4413.objecter _calc_target epoch 19 base exist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-17T18:20:02.327 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.320+0000 7f2fc5d77780 20 client.4413.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-17T18:20:02.327 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.320+0000 7f2fc5d77780 10 client.4413.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [3,0] primary 3 2024-09-17T18:20:02.327 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.320+0000 7f2fc5d77780 20 client.4413.objecter _get_session s=0x5588a7b33720 osd=3 3 2024-09-17T18:20:02.328 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.320+0000 7f2fc5d77780 10 client.4413.objecter _op_submit oid exist '@5' '@5' [read 0~256] tid 5 osd.3 2024-09-17T18:20:02.328 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.320+0000 7f2fc5d77780 20 client.4413.objecter get_session s=0x5588a7b33720 osd=3 3 2024-09-17T18:20:02.328 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.320+0000 7f2fc5d77780 15 client.4413.objecter _session_op_assign 3 5 2024-09-17T18:20:02.328 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.320+0000 7f2fc5d77780 15 client.4413.objecter _send_op 5 to 5.1 on osd.3 2024-09-17T18:20:02.328 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.320+0000 7f2fc5d77780 1 -- 172.21.15.173:0/56431533 --> v2:172.21.15.173:6800/3493964597 -- osd_op(unknown.0.0:5 5.1 5:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19) -- 0x5588a7bc1e20 con 0x5588a7c86f10 2024-09-17T18:20:02.328 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.320+0000 7f2fc5d77780 20 client.4413.objecter put_session s=0x5588a7b33720 osd=3 4 2024-09-17T18:20:02.328 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.320+0000 7f2fc5d77780 5 client.4413.objecter 1 in flight 2024-09-17T18:20:02.328 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.320+0000 7f2fc5d77780 10 client.4413.objecter _op_submit op 0x5588a7c706c0 2024-09-17T18:20:02.328 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.320+0000 7f2fc5d77780 20 client.4413.objecter _calc_target epoch 19 base noexist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-17T18:20:02.328 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc5d77780 20 client.4413.objecter _calc_target target noexist @5 -> pgid 5.a7bbaa1 2024-09-17T18:20:02.328 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc5d77780 10 client.4413.objecter _calc_target raw pgid 5.a7bbaa1 -> actual 5.1 acting [3,0] primary 3 2024-09-17T18:20:02.328 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc5d77780 20 client.4413.objecter _get_session s=0x5588a7b33720 osd=3 4 2024-09-17T18:20:02.328 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc5d77780 10 client.4413.objecter _op_submit oid noexist '@5' '@5' [read 0~256] tid 6 osd.3 2024-09-17T18:20:02.329 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc5d77780 20 client.4413.objecter get_session s=0x5588a7b33720 osd=3 4 2024-09-17T18:20:02.329 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc5d77780 15 client.4413.objecter _session_op_assign 3 6 2024-09-17T18:20:02.329 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc5d77780 15 client.4413.objecter _send_op 6 to 5.1 on osd.3 2024-09-17T18:20:02.329 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc5d77780 1 -- 172.21.15.173:0/56431533 --> v2:172.21.15.173:6800/3493964597 -- osd_op(unknown.0.0:6 5.1 5:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19) -- 0x5588a7c89880 con 0x5588a7c86f10 2024-09-17T18:20:02.329 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc5d77780 20 client.4413.objecter put_session s=0x5588a7b33720 osd=3 5 2024-09-17T18:20:02.329 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc5d77780 5 client.4413.objecter 2 in flight 2024-09-17T18:20:02.329 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (1 ms) 2024-09-17T18:20:02.329 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-09-17T18:20:02.329 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc4066640 1 -- 172.21.15.173:0/56431533 <== osd.3 v2:172.21.15.173:6800/3493964597 5 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f2fbc002890 con 0x5588a7c86f10 2024-09-17T18:20:02.329 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc4066640 10 client.4413.objecter ms_dispatch 0x5588a7ab1240 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-09-17T18:20:02.329 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc4066640 10 client.4413.objecter in handle_osd_op_reply 2024-09-17T18:20:02.329 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc4066640 7 client.4413.objecter handle_osd_op_reply 5 ondisk uv 1 in 5.1 attempt 0 2024-09-17T18:20:02.329 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc4066640 10 client.4413.objecter op 0 rval 0 len 5 2024-09-17T18:20:02.329 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc4066640 15 client.4413.objecter handle_osd_op_reply completed tid 5 2024-09-17T18:20:02.329 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc4066640 15 client.4413.objecter _finish_op 5 2024-09-17T18:20:02.330 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc4066640 20 client.4413.objecter put_session s=0x5588a7b33720 osd=3 5 2024-09-17T18:20:02.330 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc4066640 15 client.4413.objecter _session_op_remove 3 5 2024-09-17T18:20:02.330 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc4066640 5 client.4413.objecter 1 in flight 2024-09-17T18:20:02.330 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc4066640 1 -- 172.21.15.173:0/56431533 <== osd.3 v2:172.21.15.173:6800/3493964597 6 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f2fbc002890 con 0x5588a7c86f10 2024-09-17T18:20:02.330 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc4066640 10 client.4413.objecter ms_dispatch 0x5588a7ab1240 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-17T18:20:02.330 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc4066640 10 client.4413.objecter in handle_osd_op_reply 2024-09-17T18:20:02.330 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc4066640 7 client.4413.objecter handle_osd_op_reply 6 ondisk uv 0 in 5.1 attempt 0 2024-09-17T18:20:02.330 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc4066640 10 client.4413.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-17T18:20:02.330 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc4066640 10 client.4413.objecter op 0 rval 0 len 0 2024-09-17T18:20:02.330 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc4066640 15 client.4413.objecter handle_osd_op_reply completed tid 6 2024-09-17T18:20:02.330 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc4066640 15 client.4413.objecter _finish_op 6 2024-09-17T18:20:02.330 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc4066640 20 client.4413.objecter put_session s=0x5588a7b33720 osd=3 4 2024-09-17T18:20:02.330 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc4066640 15 client.4413.objecter _session_op_remove 3 6 2024-09-17T18:20:02.331 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc4066640 5 client.4413.objecter 0 in flight 2024-09-17T18:20:02.331 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc5d77780 10 client.4413.objecter _op_submit op 0x5588a7bc15b0 2024-09-17T18:20:02.331 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc5d77780 20 client.4413.objecter _calc_target epoch 19 base exist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-17T18:20:02.331 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc5d77780 20 client.4413.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-17T18:20:02.331 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc5d77780 10 client.4413.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [3,0] primary 3 2024-09-17T18:20:02.331 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc5d77780 20 client.4413.objecter _get_session s=0x5588a7b33720 osd=3 3 2024-09-17T18:20:02.331 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc5d77780 10 client.4413.objecter _op_submit oid exist '@5' '@5' [read 0~256] tid 7 osd.3 2024-09-17T18:20:02.331 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc5d77780 20 client.4413.objecter get_session s=0x5588a7b33720 osd=3 3 2024-09-17T18:20:02.331 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc5d77780 15 client.4413.objecter _session_op_assign 3 7 2024-09-17T18:20:02.331 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc5d77780 15 client.4413.objecter _send_op 7 to 5.1 on osd.3 2024-09-17T18:20:02.331 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc5d77780 1 -- 172.21.15.173:0/56431533 --> v2:172.21.15.173:6800/3493964597 -- osd_op(unknown.0.0:7 5.1 5:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19) -- 0x5588a7c706c0 con 0x5588a7c86f10 2024-09-17T18:20:02.331 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc5d77780 20 client.4413.objecter put_session s=0x5588a7b33720 osd=3 4 2024-09-17T18:20:02.331 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc5d77780 5 client.4413.objecter 1 in flight 2024-09-17T18:20:02.331 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc5d77780 10 client.4413.objecter _op_submit op 0x5588a7cc9c90 2024-09-17T18:20:02.332 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc5d77780 20 client.4413.objecter _calc_target epoch 19 base noexist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-17T18:20:02.332 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc5d77780 20 client.4413.objecter _calc_target target noexist @5 -> pgid 5.a7bbaa1 2024-09-17T18:20:02.332 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc5d77780 10 client.4413.objecter _calc_target raw pgid 5.a7bbaa1 -> actual 5.1 acting [3,0] primary 3 2024-09-17T18:20:02.332 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc5d77780 20 client.4413.objecter _get_session s=0x5588a7b33720 osd=3 4 2024-09-17T18:20:02.332 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc5d77780 10 client.4413.objecter _op_submit oid noexist '@5' '@5' [read 0~256] tid 8 osd.3 2024-09-17T18:20:02.332 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc5d77780 20 client.4413.objecter get_session s=0x5588a7b33720 osd=3 4 2024-09-17T18:20:02.332 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc5d77780 15 client.4413.objecter _session_op_assign 3 8 2024-09-17T18:20:02.332 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc5d77780 15 client.4413.objecter _send_op 8 to 5.1 on osd.3 2024-09-17T18:20:02.332 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc5d77780 1 -- 172.21.15.173:0/56431533 --> v2:172.21.15.173:6800/3493964597 -- osd_op(unknown.0.0:8 5.1 5:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19) -- 0x5588a7cca140 con 0x5588a7c86f10 2024-09-17T18:20:02.332 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc5d77780 20 client.4413.objecter put_session s=0x5588a7b33720 osd=3 5 2024-09-17T18:20:02.332 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.321+0000 7f2fc5d77780 5 client.4413.objecter 2 in flight 2024-09-17T18:20:02.332 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (1 ms) 2024-09-17T18:20:02.332 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-09-17T18:20:02.332 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.322+0000 7f2fc4066640 1 -- 172.21.15.173:0/56431533 <== osd.3 v2:172.21.15.173:6800/3493964597 7 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f2fbc002890 con 0x5588a7c86f10 2024-09-17T18:20:02.333 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.322+0000 7f2fc4066640 10 client.4413.objecter ms_dispatch 0x5588a7ab1240 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-09-17T18:20:02.333 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.322+0000 7f2fc4066640 10 client.4413.objecter in handle_osd_op_reply 2024-09-17T18:20:02.333 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.322+0000 7f2fc4066640 7 client.4413.objecter handle_osd_op_reply 7 ondisk uv 1 in 5.1 attempt 0 2024-09-17T18:20:02.333 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.322+0000 7f2fc4066640 10 client.4413.objecter op 0 rval 0 len 5 2024-09-17T18:20:02.333 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.322+0000 7f2fc4066640 15 client.4413.objecter handle_osd_op_reply completed tid 7 2024-09-17T18:20:02.333 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.322+0000 7f2fc4066640 15 client.4413.objecter _finish_op 7 2024-09-17T18:20:02.333 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.322+0000 7f2fc4066640 20 client.4413.objecter put_session s=0x5588a7b33720 osd=3 5 2024-09-17T18:20:02.333 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.322+0000 7f2fc4066640 15 client.4413.objecter _session_op_remove 3 7 2024-09-17T18:20:02.333 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.322+0000 7f2fc4066640 5 client.4413.objecter 1 in flight 2024-09-17T18:20:02.333 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.322+0000 7f2fc4066640 1 -- 172.21.15.173:0/56431533 <== osd.3 v2:172.21.15.173:6800/3493964597 8 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f2fbc002890 con 0x5588a7c86f10 2024-09-17T18:20:02.333 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.322+0000 7f2fc4066640 10 client.4413.objecter ms_dispatch 0x5588a7ab1240 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-17T18:20:02.333 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.322+0000 7f2fc4066640 10 client.4413.objecter in handle_osd_op_reply 2024-09-17T18:20:02.333 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.322+0000 7f2fc4066640 7 client.4413.objecter handle_osd_op_reply 8 ondisk uv 0 in 5.1 attempt 0 2024-09-17T18:20:02.334 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.322+0000 7f2fc4066640 10 client.4413.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-17T18:20:02.334 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.322+0000 7f2fc4066640 10 client.4413.objecter op 0 rval 0 len 0 2024-09-17T18:20:02.334 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.322+0000 7f2fc4066640 15 client.4413.objecter handle_osd_op_reply completed tid 8 2024-09-17T18:20:02.334 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.322+0000 7f2fc4066640 15 client.4413.objecter _finish_op 8 2024-09-17T18:20:02.334 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.322+0000 7f2fc4066640 20 client.4413.objecter put_session s=0x5588a7b33720 osd=3 4 2024-09-17T18:20:02.334 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.322+0000 7f2fc4066640 15 client.4413.objecter _session_op_remove 3 8 2024-09-17T18:20:02.334 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.322+0000 7f2fc4066640 5 client.4413.objecter 0 in flight 2024-09-17T18:20:02.334 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.322+0000 7f2fc5d77780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-09-17T18:20:02.334 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.322+0000 7f2fc5d77780 20 librados: queue_aio_write 0x5588a7a75760 completion 0x5588a7bc15b0 write_seq 1 2024-09-17T18:20:02.335 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.322+0000 7f2fc5d77780 10 client.4413.objecter _op_submit op 0x5588a7c8ac80 2024-09-17T18:20:02.335 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.322+0000 7f2fc5d77780 20 client.4413.objecter _calc_target epoch 19 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2024-09-17T18:20:02.335 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.322+0000 7f2fc5d77780 20 client.4413.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-17T18:20:02.335 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.322+0000 7f2fc5d77780 10 client.4413.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [3,0] primary 3 2024-09-17T18:20:02.335 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.322+0000 7f2fc5d77780 20 client.4413.objecter _get_session s=0x5588a7b33720 osd=3 3 2024-09-17T18:20:02.335 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.322+0000 7f2fc5d77780 10 client.4413.objecter _op_submit oid exist '@5' '@5' [write 0~5 in=5b] tid 9 osd.3 2024-09-17T18:20:02.335 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.322+0000 7f2fc5d77780 20 client.4413.objecter get_session s=0x5588a7b33720 osd=3 3 2024-09-17T18:20:02.335 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.322+0000 7f2fc5d77780 15 client.4413.objecter _session_op_assign 3 9 2024-09-17T18:20:02.335 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.322+0000 7f2fc5d77780 15 client.4413.objecter _send_op 9 to 5.1 on osd.3 2024-09-17T18:20:02.335 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.322+0000 7f2fc5d77780 1 -- 172.21.15.173:0/56431533 --> v2:172.21.15.173:6800/3493964597 -- osd_op(unknown.0.0:9 5.1 5:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19) -- 0x5588a7bc16c0 con 0x5588a7c86f10 2024-09-17T18:20:02.335 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.322+0000 7f2fc5d77780 20 client.4413.objecter put_session s=0x5588a7b33720 osd=3 4 2024-09-17T18:20:02.335 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.322+0000 7f2fc5d77780 5 client.4413.objecter 1 in flight 2024-09-17T18:20:02.335 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.322+0000 7f2fc5d77780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-09-17T18:20:02.336 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.331+0000 7f2fc4066640 1 -- 172.21.15.173:0/56431533 <== osd.3 v2:172.21.15.173:6800/3493964597 9 ==== osd_op_reply(9 exist [write 0~5] v19'2 uv2 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f2fbc002890 con 0x5588a7c86f10 2024-09-17T18:20:02.336 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.331+0000 7f2fc4066640 10 client.4413.objecter ms_dispatch 0x5588a7ab1240 osd_op_reply(9 exist [write 0~5] v19'2 uv2 ondisk = 0) 2024-09-17T18:20:02.336 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.331+0000 7f2fc4066640 10 client.4413.objecter in handle_osd_op_reply 2024-09-17T18:20:02.336 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.331+0000 7f2fc4066640 7 client.4413.objecter handle_osd_op_reply 9 ondisk uv 2 in 5.1 attempt 0 2024-09-17T18:20:02.336 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.331+0000 7f2fc4066640 10 client.4413.objecter op 0 rval 0 len 0 2024-09-17T18:20:02.336 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.331+0000 7f2fc4066640 15 client.4413.objecter handle_osd_op_reply completed tid 9 2024-09-17T18:20:02.336 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.331+0000 7f2fc4066640 15 client.4413.objecter _finish_op 9 2024-09-17T18:20:02.338 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.331+0000 7f2fc4066640 20 client.4413.objecter put_session s=0x5588a7b33720 osd=3 4 2024-09-17T18:20:02.338 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.331+0000 7f2fc4066640 15 client.4413.objecter _session_op_remove 3 9 2024-09-17T18:20:02.338 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.331+0000 7f2fc4066640 5 client.4413.objecter 0 in flight 2024-09-17T18:20:02.338 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.331+0000 7f2fc4066640 20 librados: complete_aio_write 0x5588a7bc15b0 2024-09-17T18:20:02.338 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (9 ms) 2024-09-17T18:20:02.338 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-09-17T18:20:02.338 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.331+0000 7f2fc5d77780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-09-17T18:20:02.338 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.331+0000 7f2fc5d77780 20 librados: queue_aio_write 0x5588a7a75760 completion 0x5588a7c8b1b0 write_seq 2 2024-09-17T18:20:02.339 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.331+0000 7f2fc5d77780 10 client.4413.objecter _op_submit op 0x5588a7c8ac80 2024-09-17T18:20:02.339 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.331+0000 7f2fc5d77780 20 client.4413.objecter _calc_target epoch 19 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2024-09-17T18:20:02.339 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.331+0000 7f2fc5d77780 20 client.4413.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-17T18:20:02.339 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.331+0000 7f2fc5d77780 10 client.4413.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [3,0] primary 3 2024-09-17T18:20:02.339 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.331+0000 7f2fc5d77780 20 client.4413.objecter _get_session s=0x5588a7b33720 osd=3 3 2024-09-17T18:20:02.339 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.331+0000 7f2fc5d77780 10 client.4413.objecter _op_submit oid exist '@5' '@5' [write 0~5 in=5b] tid 10 osd.3 2024-09-17T18:20:02.339 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.331+0000 7f2fc5d77780 20 client.4413.objecter get_session s=0x5588a7b33720 osd=3 3 2024-09-17T18:20:02.339 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.331+0000 7f2fc5d77780 15 client.4413.objecter _session_op_assign 3 10 2024-09-17T18:20:02.339 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.331+0000 7f2fc5d77780 15 client.4413.objecter _send_op 10 to 5.1 on osd.3 2024-09-17T18:20:02.339 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.331+0000 7f2fc5d77780 1 -- 172.21.15.173:0/56431533 --> v2:172.21.15.173:6800/3493964597 -- osd_op(unknown.0.0:10 5.1 5:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19) -- 0x5588a7bc16c0 con 0x5588a7c86f10 2024-09-17T18:20:02.339 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.331+0000 7f2fc5d77780 20 client.4413.objecter put_session s=0x5588a7b33720 osd=3 4 2024-09-17T18:20:02.339 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.331+0000 7f2fc5d77780 5 client.4413.objecter 1 in flight 2024-09-17T18:20:02.339 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.331+0000 7f2fc5d77780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-09-17T18:20:02.339 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.336+0000 7f2fc4066640 1 -- 172.21.15.173:0/56431533 <== osd.3 v2:172.21.15.173:6800/3493964597 10 ==== osd_op_reply(10 exist [write 0~5] v19'3 uv3 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f2fbc002890 con 0x5588a7c86f10 2024-09-17T18:20:02.339 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.336+0000 7f2fc4066640 10 client.4413.objecter ms_dispatch 0x5588a7ab1240 osd_op_reply(10 exist [write 0~5] v19'3 uv3 ondisk = 0) 2024-09-17T18:20:02.340 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.336+0000 7f2fc4066640 10 client.4413.objecter in handle_osd_op_reply 2024-09-17T18:20:02.340 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.336+0000 7f2fc4066640 7 client.4413.objecter handle_osd_op_reply 10 ondisk uv 3 in 5.1 attempt 0 2024-09-17T18:20:02.340 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.336+0000 7f2fc4066640 10 client.4413.objecter op 0 rval 0 len 0 2024-09-17T18:20:02.340 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.336+0000 7f2fc4066640 15 client.4413.objecter handle_osd_op_reply completed tid 10 2024-09-17T18:20:02.340 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.336+0000 7f2fc4066640 15 client.4413.objecter _finish_op 10 2024-09-17T18:20:02.340 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.336+0000 7f2fc4066640 20 client.4413.objecter put_session s=0x5588a7b33720 osd=3 4 2024-09-17T18:20:02.340 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.336+0000 7f2fc4066640 15 client.4413.objecter _session_op_remove 3 10 2024-09-17T18:20:02.347 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.336+0000 7f2fc4066640 5 client.4413.objecter 0 in flight 2024-09-17T18:20:02.347 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.336+0000 7f2fc4066640 20 librados: complete_aio_write 0x5588a7c8b1b0 2024-09-17T18:20:02.347 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (5 ms) 2024-09-17T18:20:02.347 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-09-17T18:20:02.347 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.336+0000 7f2fc5d77780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-09-17T18:20:02.347 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.336+0000 7f2fc5d77780 20 librados: queue_aio_write 0x5588a7a75760 completion 0x5588a7c8ac80 write_seq 3 2024-09-17T18:20:02.348 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.336+0000 7f2fc5d77780 10 client.4413.objecter _op_submit op 0x5588a7cad4c0 2024-09-17T18:20:02.348 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.336+0000 7f2fc5d77780 20 client.4413.objecter _calc_target epoch 19 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2024-09-17T18:20:02.348 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.336+0000 7f2fc5d77780 20 client.4413.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-17T18:20:02.348 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.336+0000 7f2fc5d77780 10 client.4413.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [3,0] primary 3 2024-09-17T18:20:02.348 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.336+0000 7f2fc5d77780 20 client.4413.objecter _get_session s=0x5588a7b33720 osd=3 3 2024-09-17T18:20:02.348 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.336+0000 7f2fc5d77780 10 client.4413.objecter _op_submit oid exist '@5' '@5' [write 0~5 in=5b] tid 11 osd.3 2024-09-17T18:20:02.348 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.336+0000 7f2fc5d77780 20 client.4413.objecter get_session s=0x5588a7b33720 osd=3 3 2024-09-17T18:20:02.348 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.336+0000 7f2fc5d77780 15 client.4413.objecter _session_op_assign 3 11 2024-09-17T18:20:02.348 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.336+0000 7f2fc5d77780 15 client.4413.objecter _send_op 11 to 5.1 on osd.3 2024-09-17T18:20:02.348 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.336+0000 7f2fc5d77780 1 -- 172.21.15.173:0/56431533 --> v2:172.21.15.173:6800/3493964597 -- osd_op(unknown.0.0:11 5.1 5:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19) -- 0x5588a7c8ad90 con 0x5588a7c86f10 2024-09-17T18:20:02.348 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.336+0000 7f2fc5d77780 20 client.4413.objecter put_session s=0x5588a7b33720 osd=3 4 2024-09-17T18:20:02.348 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.336+0000 7f2fc5d77780 5 client.4413.objecter 1 in flight 2024-09-17T18:20:02.349 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.336+0000 7f2fc5d77780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-09-17T18:20:02.349 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.345+0000 7f2fc4066640 1 -- 172.21.15.173:0/56431533 <== osd.3 v2:172.21.15.173:6800/3493964597 11 ==== osd_op_reply(11 exist [write 0~5] v19'4 uv4 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f2fbc002890 con 0x5588a7c86f10 2024-09-17T18:20:02.349 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.345+0000 7f2fc4066640 10 client.4413.objecter ms_dispatch 0x5588a7ab1240 osd_op_reply(11 exist [write 0~5] v19'4 uv4 ondisk = 0) 2024-09-17T18:20:02.349 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.345+0000 7f2fc4066640 10 client.4413.objecter in handle_osd_op_reply 2024-09-17T18:20:02.349 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.345+0000 7f2fc4066640 7 client.4413.objecter handle_osd_op_reply 11 ondisk uv 4 in 5.1 attempt 0 2024-09-17T18:20:02.349 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.345+0000 7f2fc4066640 10 client.4413.objecter op 0 rval 0 len 0 2024-09-17T18:20:02.349 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.345+0000 7f2fc4066640 15 client.4413.objecter handle_osd_op_reply completed tid 11 2024-09-17T18:20:02.349 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.345+0000 7f2fc4066640 15 client.4413.objecter _finish_op 11 2024-09-17T18:20:02.349 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.345+0000 7f2fc4066640 20 client.4413.objecter put_session s=0x5588a7b33720 osd=3 4 2024-09-17T18:20:02.349 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.345+0000 7f2fc4066640 15 client.4413.objecter _session_op_remove 3 11 2024-09-17T18:20:02.349 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.345+0000 7f2fc4066640 5 client.4413.objecter 0 in flight 2024-09-17T18:20:02.349 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.345+0000 7f2fc4066640 20 librados: complete_aio_write 0x5588a7c8ac80 2024-09-17T18:20:02.350 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (9 ms) 2024-09-17T18:20:02.350 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-09-17T18:20:02.350 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.345+0000 7f2fc5d77780 10 client.4413.objecter _op_submit op 0x5588a7c89c70 2024-09-17T18:20:02.350 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.345+0000 7f2fc5d77780 20 client.4413.objecter _calc_target epoch 19 base exist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-17T18:20:02.350 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.345+0000 7f2fc5d77780 20 client.4413.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-17T18:20:02.350 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.345+0000 7f2fc5d77780 10 client.4413.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [3,0] primary 3 2024-09-17T18:20:02.351 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.345+0000 7f2fc5d77780 20 client.4413.objecter _get_session s=0x5588a7b33720 osd=3 3 2024-09-17T18:20:02.351 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.345+0000 7f2fc5d77780 10 client.4413.objecter _op_submit oid exist '@5' '@5' [read 0~0] tid 12 osd.3 2024-09-17T18:20:02.351 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.345+0000 7f2fc5d77780 20 client.4413.objecter get_session s=0x5588a7b33720 osd=3 3 2024-09-17T18:20:02.351 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.345+0000 7f2fc5d77780 15 client.4413.objecter _session_op_assign 3 12 2024-09-17T18:20:02.351 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.345+0000 7f2fc5d77780 15 client.4413.objecter _send_op 12 to 5.1 on osd.3 2024-09-17T18:20:02.351 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.345+0000 7f2fc5d77780 1 -- 172.21.15.173:0/56431533 --> v2:172.21.15.173:6800/3493964597 -- osd_op(unknown.0.0:12 5.1 5:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19) -- 0x5588a7c8ad90 con 0x5588a7c86f10 2024-09-17T18:20:02.351 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.345+0000 7f2fc5d77780 20 client.4413.objecter put_session s=0x5588a7b33720 osd=3 4 2024-09-17T18:20:02.351 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.345+0000 7f2fc5d77780 5 client.4413.objecter 1 in flight 2024-09-17T18:20:02.351 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.345+0000 7f2fc5d77780 10 client.4413.objecter _op_submit op 0x5588a7c8a270 2024-09-17T18:20:02.351 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.345+0000 7f2fc5d77780 20 client.4413.objecter _calc_target epoch 19 base noexist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-17T18:20:02.351 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.345+0000 7f2fc5d77780 20 client.4413.objecter _calc_target target noexist @5 -> pgid 5.a7bbaa1 2024-09-17T18:20:02.351 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.345+0000 7f2fc5d77780 10 client.4413.objecter _calc_target raw pgid 5.a7bbaa1 -> actual 5.1 acting [3,0] primary 3 2024-09-17T18:20:02.351 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.345+0000 7f2fc5d77780 20 client.4413.objecter _get_session s=0x5588a7b33720 osd=3 4 2024-09-17T18:20:02.351 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.345+0000 7f2fc5d77780 10 client.4413.objecter _op_submit oid noexist '@5' '@5' [read 0~0] tid 13 osd.3 2024-09-17T18:20:02.352 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.345+0000 7f2fc5d77780 20 client.4413.objecter get_session s=0x5588a7b33720 osd=3 4 2024-09-17T18:20:02.352 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.345+0000 7f2fc5d77780 15 client.4413.objecter _session_op_assign 3 13 2024-09-17T18:20:02.352 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.345+0000 7f2fc5d77780 15 client.4413.objecter _send_op 13 to 5.1 on osd.3 2024-09-17T18:20:02.352 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.345+0000 7f2fc5d77780 1 -- 172.21.15.173:0/56431533 --> v2:172.21.15.173:6800/3493964597 -- osd_op(unknown.0.0:13 5.1 5:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19) -- 0x5588a7c8a720 con 0x5588a7c86f10 2024-09-17T18:20:02.352 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.345+0000 7f2fc5d77780 20 client.4413.objecter put_session s=0x5588a7b33720 osd=3 5 2024-09-17T18:20:02.352 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.345+0000 7f2fc5d77780 5 client.4413.objecter 2 in flight 2024-09-17T18:20:02.352 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc4066640 1 -- 172.21.15.173:0/56431533 <== osd.3 v2:172.21.15.173:6800/3493964597 12 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f2fbc002890 con 0x5588a7c86f10 2024-09-17T18:20:02.352 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc4066640 10 client.4413.objecter ms_dispatch 0x5588a7ab1240 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-09-17T18:20:02.352 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc4066640 10 client.4413.objecter in handle_osd_op_reply 2024-09-17T18:20:02.352 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc4066640 7 client.4413.objecter handle_osd_op_reply 12 ondisk uv 4 in 5.1 attempt 0 2024-09-17T18:20:02.352 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc4066640 10 client.4413.objecter op 0 rval 0 len 5 2024-09-17T18:20:02.352 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc4066640 15 client.4413.objecter handle_osd_op_reply completed tid 12 2024-09-17T18:20:02.352 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc4066640 15 client.4413.objecter _finish_op 12 2024-09-17T18:20:02.353 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc4066640 20 client.4413.objecter put_session s=0x5588a7b33720 osd=3 5 2024-09-17T18:20:02.353 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc4066640 15 client.4413.objecter _session_op_remove 3 12 2024-09-17T18:20:02.353 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc4066640 5 client.4413.objecter 1 in flight 2024-09-17T18:20:02.353 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (1 ms) 2024-09-17T18:20:02.353 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-09-17T18:20:02.353 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc4066640 1 -- 172.21.15.173:0/56431533 <== osd.3 v2:172.21.15.173:6800/3493964597 13 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f2fbc002890 con 0x5588a7c86f10 2024-09-17T18:20:02.353 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc4066640 10 client.4413.objecter ms_dispatch 0x5588a7ab1240 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-17T18:20:02.353 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc4066640 10 client.4413.objecter in handle_osd_op_reply 2024-09-17T18:20:02.353 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc4066640 7 client.4413.objecter handle_osd_op_reply 13 ondisk uv 0 in 5.1 attempt 0 2024-09-17T18:20:02.353 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc4066640 10 client.4413.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-17T18:20:02.353 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc4066640 10 client.4413.objecter op 0 rval 0 len 0 2024-09-17T18:20:02.353 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc4066640 15 client.4413.objecter handle_osd_op_reply completed tid 13 2024-09-17T18:20:02.353 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc4066640 15 client.4413.objecter _finish_op 13 2024-09-17T18:20:02.354 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc4066640 20 client.4413.objecter put_session s=0x5588a7b33720 osd=3 4 2024-09-17T18:20:02.354 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc4066640 15 client.4413.objecter _session_op_remove 3 13 2024-09-17T18:20:02.354 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc4066640 5 client.4413.objecter 0 in flight 2024-09-17T18:20:02.354 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc5d77780 10 client.4413.objecter _op_submit op 0x5588a7c89c70 2024-09-17T18:20:02.354 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc5d77780 20 client.4413.objecter _calc_target epoch 19 base exist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-17T18:20:02.354 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc5d77780 20 client.4413.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-17T18:20:02.354 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc5d77780 10 client.4413.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [3,0] primary 3 2024-09-17T18:20:02.354 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc5d77780 20 client.4413.objecter _get_session s=0x5588a7b33720 osd=3 3 2024-09-17T18:20:02.354 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc5d77780 10 client.4413.objecter _op_submit oid exist '@5' '@5' [read 0~0] tid 14 osd.3 2024-09-17T18:20:02.354 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc5d77780 20 client.4413.objecter get_session s=0x5588a7b33720 osd=3 3 2024-09-17T18:20:02.354 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc5d77780 15 client.4413.objecter _session_op_assign 3 14 2024-09-17T18:20:02.354 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc5d77780 15 client.4413.objecter _send_op 14 to 5.1 on osd.3 2024-09-17T18:20:02.354 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc5d77780 1 -- 172.21.15.173:0/56431533 --> v2:172.21.15.173:6800/3493964597 -- osd_op(unknown.0.0:14 5.1 5:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19) -- 0x5588a7c8ad90 con 0x5588a7c86f10 2024-09-17T18:20:02.354 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc5d77780 20 client.4413.objecter put_session s=0x5588a7b33720 osd=3 4 2024-09-17T18:20:02.355 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc5d77780 5 client.4413.objecter 1 in flight 2024-09-17T18:20:02.355 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc5d77780 10 client.4413.objecter _op_submit op 0x5588a7c8a3c0 2024-09-17T18:20:02.355 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc5d77780 20 client.4413.objecter _calc_target epoch 19 base noexist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-17T18:20:02.355 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc5d77780 20 client.4413.objecter _calc_target target noexist @5 -> pgid 5.a7bbaa1 2024-09-17T18:20:02.355 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc5d77780 10 client.4413.objecter _calc_target raw pgid 5.a7bbaa1 -> actual 5.1 acting [3,0] primary 3 2024-09-17T18:20:02.355 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc5d77780 20 client.4413.objecter _get_session s=0x5588a7b33720 osd=3 4 2024-09-17T18:20:02.355 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc5d77780 10 client.4413.objecter _op_submit oid noexist '@5' '@5' [read 0~0] tid 15 osd.3 2024-09-17T18:20:02.355 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc5d77780 20 client.4413.objecter get_session s=0x5588a7b33720 osd=3 4 2024-09-17T18:20:02.355 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc5d77780 15 client.4413.objecter _session_op_assign 3 15 2024-09-17T18:20:02.355 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc5d77780 15 client.4413.objecter _send_op 15 to 5.1 on osd.3 2024-09-17T18:20:02.355 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc5d77780 1 -- 172.21.15.173:0/56431533 --> v2:172.21.15.173:6800/3493964597 -- osd_op(unknown.0.0:15 5.1 5:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19) -- 0x5588a7cadb10 con 0x5588a7c86f10 2024-09-17T18:20:02.355 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc5d77780 20 client.4413.objecter put_session s=0x5588a7b33720 osd=3 5 2024-09-17T18:20:02.355 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.346+0000 7f2fc5d77780 5 client.4413.objecter 2 in flight 2024-09-17T18:20:02.355 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.347+0000 7f2fc4066640 1 -- 172.21.15.173:0/56431533 <== osd.3 v2:172.21.15.173:6800/3493964597 14 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f2fbc002890 con 0x5588a7c86f10 2024-09-17T18:20:02.356 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.347+0000 7f2fc4066640 10 client.4413.objecter ms_dispatch 0x5588a7ab1240 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-09-17T18:20:02.356 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.347+0000 7f2fc4066640 10 client.4413.objecter in handle_osd_op_reply 2024-09-17T18:20:02.356 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.347+0000 7f2fc4066640 7 client.4413.objecter handle_osd_op_reply 14 ondisk uv 4 in 5.1 attempt 0 2024-09-17T18:20:02.356 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.347+0000 7f2fc4066640 10 client.4413.objecter op 0 rval 0 len 5 2024-09-17T18:20:02.356 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.347+0000 7f2fc4066640 15 client.4413.objecter handle_osd_op_reply completed tid 14 2024-09-17T18:20:02.356 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.347+0000 7f2fc4066640 15 client.4413.objecter _finish_op 14 2024-09-17T18:20:02.356 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.347+0000 7f2fc4066640 20 client.4413.objecter put_session s=0x5588a7b33720 osd=3 5 2024-09-17T18:20:02.356 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.347+0000 7f2fc4066640 15 client.4413.objecter _session_op_remove 3 14 2024-09-17T18:20:02.356 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.347+0000 7f2fc4066640 5 client.4413.objecter 1 in flight 2024-09-17T18:20:02.356 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc4066640 1 -- 172.21.15.173:0/56431533 <== osd.3 v2:172.21.15.173:6800/3493964597 15 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f2fbc002890 con 0x5588a7c86f10 2024-09-17T18:20:02.356 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc4066640 10 client.4413.objecter ms_dispatch 0x5588a7ab1240 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-17T18:20:02.356 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc4066640 10 client.4413.objecter in handle_osd_op_reply 2024-09-17T18:20:02.356 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc4066640 7 client.4413.objecter handle_osd_op_reply 15 ondisk uv 0 in 5.1 attempt 0 2024-09-17T18:20:02.356 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc4066640 10 client.4413.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-17T18:20:02.357 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc4066640 10 client.4413.objecter op 0 rval 0 len 0 2024-09-17T18:20:02.357 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc4066640 15 client.4413.objecter handle_osd_op_reply completed tid 15 2024-09-17T18:20:02.357 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc4066640 15 client.4413.objecter _finish_op 15 2024-09-17T18:20:02.357 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc4066640 20 client.4413.objecter put_session s=0x5588a7b33720 osd=3 4 2024-09-17T18:20:02.357 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc4066640 15 client.4413.objecter _session_op_remove 3 15 2024-09-17T18:20:02.357 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc4066640 5 client.4413.objecter 0 in flight 2024-09-17T18:20:02.357 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (2 ms) 2024-09-17T18:20:02.357 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-09-17T18:20:02.357 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc5d77780 10 client.4413.objecter _op_submit op 0x5588a7cad4c0 2024-09-17T18:20:02.357 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc5d77780 20 client.4413.objecter _calc_target epoch 19 base exist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-17T18:20:02.357 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc5d77780 20 client.4413.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-17T18:20:02.357 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc5d77780 10 client.4413.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [3,0] primary 3 2024-09-17T18:20:02.357 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc5d77780 20 client.4413.objecter _get_session s=0x5588a7b33720 osd=3 3 2024-09-17T18:20:02.358 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc5d77780 10 client.4413.objecter _op_submit oid exist '@5' '@5' [read 0~0] tid 16 osd.3 2024-09-17T18:20:02.358 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc5d77780 20 client.4413.objecter get_session s=0x5588a7b33720 osd=3 3 2024-09-17T18:20:02.358 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc5d77780 15 client.4413.objecter _session_op_assign 3 16 2024-09-17T18:20:02.358 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc5d77780 15 client.4413.objecter _send_op 16 to 5.1 on osd.3 2024-09-17T18:20:02.358 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc5d77780 1 -- 172.21.15.173:0/56431533 --> v2:172.21.15.173:6800/3493964597 -- osd_op(unknown.0.0:16 5.1 5:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19) -- 0x5588a7c8ad90 con 0x5588a7c86f10 2024-09-17T18:20:02.358 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc5d77780 20 client.4413.objecter put_session s=0x5588a7b33720 osd=3 4 2024-09-17T18:20:02.358 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc5d77780 5 client.4413.objecter 1 in flight 2024-09-17T18:20:02.358 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc5d77780 10 client.4413.objecter _op_submit op 0x5588a7c8a3c0 2024-09-17T18:20:02.358 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc5d77780 20 client.4413.objecter _calc_target epoch 19 base noexist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-17T18:20:02.358 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc5d77780 20 client.4413.objecter _calc_target target noexist @5 -> pgid 5.a7bbaa1 2024-09-17T18:20:02.358 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc5d77780 10 client.4413.objecter _calc_target raw pgid 5.a7bbaa1 -> actual 5.1 acting [3,0] primary 3 2024-09-17T18:20:02.358 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc5d77780 20 client.4413.objecter _get_session s=0x5588a7b33720 osd=3 4 2024-09-17T18:20:02.358 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc5d77780 10 client.4413.objecter _op_submit oid noexist '@5' '@5' [read 0~0] tid 17 osd.3 2024-09-17T18:20:02.359 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc5d77780 20 client.4413.objecter get_session s=0x5588a7b33720 osd=3 4 2024-09-17T18:20:02.359 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc5d77780 15 client.4413.objecter _session_op_assign 3 17 2024-09-17T18:20:02.359 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc5d77780 15 client.4413.objecter _send_op 17 to 5.1 on osd.3 2024-09-17T18:20:02.359 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc5d77780 1 -- 172.21.15.173:0/56431533 --> v2:172.21.15.173:6800/3493964597 -- osd_op(unknown.0.0:17 5.1 5:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19) -- 0x5588a7c89c70 con 0x5588a7c86f10 2024-09-17T18:20:02.359 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc5d77780 20 client.4413.objecter put_session s=0x5588a7b33720 osd=3 5 2024-09-17T18:20:02.359 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc5d77780 5 client.4413.objecter 2 in flight 2024-09-17T18:20:02.359 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc4066640 1 -- 172.21.15.173:0/56431533 <== osd.3 v2:172.21.15.173:6800/3493964597 16 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f2fbc002890 con 0x5588a7c86f10 2024-09-17T18:20:02.360 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc4066640 10 client.4413.objecter ms_dispatch 0x5588a7ab1240 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-09-17T18:20:02.360 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc4066640 10 client.4413.objecter in handle_osd_op_reply 2024-09-17T18:20:02.360 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc4066640 7 client.4413.objecter handle_osd_op_reply 16 ondisk uv 4 in 5.1 attempt 0 2024-09-17T18:20:02.360 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc4066640 10 client.4413.objecter op 0 rval 0 len 5 2024-09-17T18:20:02.360 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc4066640 15 client.4413.objecter handle_osd_op_reply completed tid 16 2024-09-17T18:20:02.360 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc4066640 15 client.4413.objecter _finish_op 16 2024-09-17T18:20:02.360 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc4066640 20 client.4413.objecter put_session s=0x5588a7b33720 osd=3 5 2024-09-17T18:20:02.360 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc4066640 15 client.4413.objecter _session_op_remove 3 16 2024-09-17T18:20:02.360 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.348+0000 7f2fc4066640 5 client.4413.objecter 1 in flight 2024-09-17T18:20:02.360 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.349+0000 7f2fc4066640 1 -- 172.21.15.173:0/56431533 <== osd.3 v2:172.21.15.173:6800/3493964597 17 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f2fbc002890 con 0x5588a7c86f10 2024-09-17T18:20:02.360 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.349+0000 7f2fc4066640 10 client.4413.objecter ms_dispatch 0x5588a7ab1240 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-17T18:20:02.360 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.349+0000 7f2fc4066640 10 client.4413.objecter in handle_osd_op_reply 2024-09-17T18:20:02.360 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.349+0000 7f2fc4066640 7 client.4413.objecter handle_osd_op_reply 17 ondisk uv 0 in 5.1 attempt 0 2024-09-17T18:20:02.360 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.349+0000 7f2fc4066640 10 client.4413.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-17T18:20:02.360 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.349+0000 7f2fc4066640 10 client.4413.objecter op 0 rval 0 len 0 2024-09-17T18:20:02.361 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.349+0000 7f2fc4066640 15 client.4413.objecter handle_osd_op_reply completed tid 17 2024-09-17T18:20:02.361 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.349+0000 7f2fc4066640 15 client.4413.objecter _finish_op 17 2024-09-17T18:20:02.361 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.349+0000 7f2fc4066640 20 client.4413.objecter put_session s=0x5588a7b33720 osd=3 4 2024-09-17T18:20:02.361 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.349+0000 7f2fc4066640 15 client.4413.objecter _session_op_remove 3 17 2024-09-17T18:20:02.361 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.349+0000 7f2fc4066640 5 client.4413.objecter 0 in flight 2024-09-17T18:20:02.361 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (1 ms) 2024-09-17T18:20:02.361 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-09-17T18:20:02.361 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.349+0000 7f2fc5d77780 20 librados: queue_aio_write 0x5588a7a75760 completion 0x5588a7c8ad90 write_seq 4 2024-09-17T18:20:02.361 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.349+0000 7f2fc5d77780 10 client.4413.objecter _op_submit op 0x5588a7c8a3c0 2024-09-17T18:20:02.361 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.349+0000 7f2fc5d77780 20 client.4413.objecter _calc_target epoch 19 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2024-09-17T18:20:02.361 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.349+0000 7f2fc5d77780 20 client.4413.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-17T18:20:02.361 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.349+0000 7f2fc5d77780 10 client.4413.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [3,0] primary 3 2024-09-17T18:20:02.361 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.349+0000 7f2fc5d77780 20 client.4413.objecter _get_session s=0x5588a7b33720 osd=3 3 2024-09-17T18:20:02.361 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.349+0000 7f2fc5d77780 10 client.4413.objecter _op_submit oid exist '@5' '@5' [writefull 0~5 in=5b] tid 18 osd.3 2024-09-17T18:20:02.362 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.349+0000 7f2fc5d77780 20 client.4413.objecter get_session s=0x5588a7b33720 osd=3 3 2024-09-17T18:20:02.362 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.349+0000 7f2fc5d77780 15 client.4413.objecter _session_op_assign 3 18 2024-09-17T18:20:02.362 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.349+0000 7f2fc5d77780 15 client.4413.objecter _send_op 18 to 5.1 on osd.3 2024-09-17T18:20:02.362 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.349+0000 7f2fc5d77780 1 -- 172.21.15.173:0/56431533 --> v2:172.21.15.173:6800/3493964597 -- osd_op(unknown.0.0:18 5.1 5:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19) -- 0x5588a7cadb10 con 0x5588a7c86f10 2024-09-17T18:20:02.362 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.349+0000 7f2fc5d77780 20 client.4413.objecter put_session s=0x5588a7b33720 osd=3 4 2024-09-17T18:20:02.362 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.349+0000 7f2fc5d77780 5 client.4413.objecter 1 in flight 2024-09-17T18:20:02.362 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.357+0000 7f2fc4066640 1 -- 172.21.15.173:0/56431533 <== osd.3 v2:172.21.15.173:6800/3493964597 18 ==== osd_op_reply(18 exist [writefull 0~5] v19'5 uv5 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f2fbc002890 con 0x5588a7c86f10 2024-09-17T18:20:02.362 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.357+0000 7f2fc4066640 10 client.4413.objecter ms_dispatch 0x5588a7ab1240 osd_op_reply(18 exist [writefull 0~5] v19'5 uv5 ondisk = 0) 2024-09-17T18:20:02.362 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.357+0000 7f2fc4066640 10 client.4413.objecter in handle_osd_op_reply 2024-09-17T18:20:02.362 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.357+0000 7f2fc4066640 7 client.4413.objecter handle_osd_op_reply 18 ondisk uv 5 in 5.1 attempt 0 2024-09-17T18:20:02.362 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.357+0000 7f2fc4066640 10 client.4413.objecter op 0 rval 0 len 0 2024-09-17T18:20:02.362 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.357+0000 7f2fc4066640 15 client.4413.objecter handle_osd_op_reply completed tid 18 2024-09-17T18:20:02.362 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.357+0000 7f2fc4066640 15 client.4413.objecter _finish_op 18 2024-09-17T18:20:02.362 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.357+0000 7f2fc4066640 20 client.4413.objecter put_session s=0x5588a7b33720 osd=3 4 2024-09-17T18:20:02.363 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.357+0000 7f2fc4066640 15 client.4413.objecter _session_op_remove 3 18 2024-09-17T18:20:02.363 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.357+0000 7f2fc4066640 5 client.4413.objecter 0 in flight 2024-09-17T18:20:02.363 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.357+0000 7f2fc4066640 20 librados: complete_aio_write 0x5588a7c8ad90 2024-09-17T18:20:02.363 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (8 ms) 2024-09-17T18:20:02.363 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-09-17T18:20:02.363 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.357+0000 7f2fc5d77780 20 librados: queue_aio_write 0x5588a7a75760 completion 0x5588a7c8aea0 write_seq 5 2024-09-17T18:20:02.363 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.357+0000 7f2fc5d77780 10 client.4413.objecter _op_submit op 0x5588a7c8a3c0 2024-09-17T18:20:02.363 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.357+0000 7f2fc5d77780 20 client.4413.objecter _calc_target epoch 19 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2024-09-17T18:20:02.371 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.357+0000 7f2fc5d77780 20 client.4413.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-17T18:20:02.371 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.357+0000 7f2fc5d77780 10 client.4413.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [3,0] primary 3 2024-09-17T18:20:02.371 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.357+0000 7f2fc5d77780 20 client.4413.objecter _get_session s=0x5588a7b33720 osd=3 3 2024-09-17T18:20:02.371 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.357+0000 7f2fc5d77780 10 client.4413.objecter _op_submit oid exist '@5' '@5' [writefull 0~5 in=5b] tid 19 osd.3 2024-09-17T18:20:02.371 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.357+0000 7f2fc5d77780 20 client.4413.objecter get_session s=0x5588a7b33720 osd=3 3 2024-09-17T18:20:02.371 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.357+0000 7f2fc5d77780 15 client.4413.objecter _session_op_assign 3 19 2024-09-17T18:20:02.371 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.357+0000 7f2fc5d77780 15 client.4413.objecter _send_op 19 to 5.1 on osd.3 2024-09-17T18:20:02.371 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.357+0000 7f2fc5d77780 1 -- 172.21.15.173:0/56431533 --> v2:172.21.15.173:6800/3493964597 -- osd_op(unknown.0.0:19 5.1 5:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19) -- 0x5588a7cadb10 con 0x5588a7c86f10 2024-09-17T18:20:02.371 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.357+0000 7f2fc5d77780 20 client.4413.objecter put_session s=0x5588a7b33720 osd=3 4 2024-09-17T18:20:02.372 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.357+0000 7f2fc5d77780 5 client.4413.objecter 1 in flight 2024-09-17T18:20:02.372 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.369+0000 7f2fc4066640 1 -- 172.21.15.173:0/56431533 <== osd.3 v2:172.21.15.173:6800/3493964597 19 ==== osd_op_reply(19 exist [writefull 0~5] v19'6 uv6 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f2fbc002890 con 0x5588a7c86f10 2024-09-17T18:20:02.372 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.369+0000 7f2fc4066640 10 client.4413.objecter ms_dispatch 0x5588a7ab1240 osd_op_reply(19 exist [writefull 0~5] v19'6 uv6 ondisk = 0) 2024-09-17T18:20:02.372 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.369+0000 7f2fc4066640 10 client.4413.objecter in handle_osd_op_reply 2024-09-17T18:20:02.372 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.369+0000 7f2fc4066640 7 client.4413.objecter handle_osd_op_reply 19 ondisk uv 6 in 5.1 attempt 0 2024-09-17T18:20:02.372 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.369+0000 7f2fc4066640 10 client.4413.objecter op 0 rval 0 len 0 2024-09-17T18:20:02.372 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.369+0000 7f2fc4066640 15 client.4413.objecter handle_osd_op_reply completed tid 19 2024-09-17T18:20:02.372 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.369+0000 7f2fc4066640 15 client.4413.objecter _finish_op 19 2024-09-17T18:20:02.372 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.369+0000 7f2fc4066640 20 client.4413.objecter put_session s=0x5588a7b33720 osd=3 4 2024-09-17T18:20:02.372 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.369+0000 7f2fc4066640 15 client.4413.objecter _session_op_remove 3 19 2024-09-17T18:20:02.372 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.369+0000 7f2fc4066640 5 client.4413.objecter 0 in flight 2024-09-17T18:20:02.372 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.369+0000 7f2fc4066640 20 librados: complete_aio_write 0x5588a7c8aea0 2024-09-17T18:20:02.372 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (12 ms) 2024-09-17T18:20:02.373 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-09-17T18:20:02.373 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.369+0000 7f2fc5d77780 20 librados: queue_aio_write 0x5588a7a75760 completion 0x5588a7c8ad90 write_seq 6 2024-09-17T18:20:02.373 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.369+0000 7f2fc5d77780 10 client.4413.objecter _op_submit op 0x5588a7cad4c0 2024-09-17T18:20:02.373 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.369+0000 7f2fc5d77780 20 client.4413.objecter _calc_target epoch 19 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2024-09-17T18:20:02.373 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.369+0000 7f2fc5d77780 20 client.4413.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-17T18:20:02.373 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.369+0000 7f2fc5d77780 10 client.4413.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [3,0] primary 3 2024-09-17T18:20:02.373 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.369+0000 7f2fc5d77780 20 client.4413.objecter _get_session s=0x5588a7b33720 osd=3 3 2024-09-17T18:20:02.376 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.369+0000 7f2fc5d77780 10 client.4413.objecter _op_submit oid exist '@5' '@5' [writefull 0~5 in=5b] tid 20 osd.3 2024-09-17T18:20:02.376 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.369+0000 7f2fc5d77780 20 client.4413.objecter get_session s=0x5588a7b33720 osd=3 3 2024-09-17T18:20:02.376 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.369+0000 7f2fc5d77780 15 client.4413.objecter _session_op_assign 3 20 2024-09-17T18:20:02.376 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.369+0000 7f2fc5d77780 15 client.4413.objecter _send_op 20 to 5.1 on osd.3 2024-09-17T18:20:02.376 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.369+0000 7f2fc5d77780 1 -- 172.21.15.173:0/56431533 --> v2:172.21.15.173:6800/3493964597 -- osd_op(unknown.0.0:20 5.1 5:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19) -- 0x5588a7cadb10 con 0x5588a7c86f10 2024-09-17T18:20:02.376 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.369+0000 7f2fc5d77780 20 client.4413.objecter put_session s=0x5588a7b33720 osd=3 4 2024-09-17T18:20:02.377 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.369+0000 7f2fc5d77780 5 client.4413.objecter 1 in flight 2024-09-17T18:20:02.377 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.374+0000 7f2fc4066640 1 -- 172.21.15.173:0/56431533 <== osd.3 v2:172.21.15.173:6800/3493964597 20 ==== osd_op_reply(20 exist [writefull 0~5] v19'7 uv7 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f2fbc002890 con 0x5588a7c86f10 2024-09-17T18:20:02.377 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.374+0000 7f2fc4066640 10 client.4413.objecter ms_dispatch 0x5588a7ab1240 osd_op_reply(20 exist [writefull 0~5] v19'7 uv7 ondisk = 0) 2024-09-17T18:20:02.377 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.374+0000 7f2fc4066640 10 client.4413.objecter in handle_osd_op_reply 2024-09-17T18:20:02.377 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.374+0000 7f2fc4066640 7 client.4413.objecter handle_osd_op_reply 20 ondisk uv 7 in 5.1 attempt 0 2024-09-17T18:20:02.377 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.374+0000 7f2fc4066640 10 client.4413.objecter op 0 rval 0 len 0 2024-09-17T18:20:02.377 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.374+0000 7f2fc4066640 15 client.4413.objecter handle_osd_op_reply completed tid 20 2024-09-17T18:20:02.377 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.374+0000 7f2fc4066640 15 client.4413.objecter _finish_op 20 2024-09-17T18:20:02.377 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.374+0000 7f2fc4066640 20 client.4413.objecter put_session s=0x5588a7b33720 osd=3 4 2024-09-17T18:20:02.377 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.374+0000 7f2fc4066640 15 client.4413.objecter _session_op_remove 3 20 2024-09-17T18:20:02.377 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.374+0000 7f2fc4066640 5 client.4413.objecter 0 in flight 2024-09-17T18:20:02.377 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.374+0000 7f2fc4066640 20 librados: complete_aio_write 0x5588a7c8ad90 2024-09-17T18:20:02.377 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (5 ms) 2024-09-17T18:20:02.377 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.374+0000 7f2fc5d77780 10 librados: watch_flush enter 2024-09-17T18:20:02.378 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.374+0000 7f2fc5d77780 10 librados: watch_flush exit 2024-09-17T18:20:02.378 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.374+0000 7f2fc5d77780 10 client.4413.objecter close_session for osd.3 2024-09-17T18:20:02.378 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.374+0000 7f2fc5d77780 1 -- 172.21.15.173:0/56431533 >> v2:172.21.15.173:6800/3493964597 conn(0x5588a7c86f10 msgr2=0x5588a7c89300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:20:02.378 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.374+0000 7f2fc5d77780 1 --2- 172.21.15.173:0/56431533 >> v2:172.21.15.173:6800/3493964597 conn(0x5588a7c86f10 0x5588a7c89300 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:20:02.378 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.374+0000 7f2fc5d77780 20 client.4413.objecter put_session s=0x5588a7b33720 osd=3 1 2024-09-17T18:20:02.378 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.374+0000 7f2fc5d77780 20 client.4413.objecter shutdown clearing up homeless session... 2024-09-17T18:20:02.378 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.374+0000 7f2fc5d77780 10 client.4413.objecter successfully canceled tick 2024-09-17T18:20:02.378 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.374+0000 7f2fc5d77780 1 -- 172.21.15.173:0/56431533 >> v2:172.21.15.173:6812/2423786037 conn(0x7f2fb003e390 msgr2=0x7f2fb0040850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:20:02.380 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.374+0000 7f2fc5d77780 1 --2- 172.21.15.173:0/56431533 >> v2:172.21.15.173:6812/2423786037 conn(0x7f2fb003e390 0x7f2fb0040850 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2fb800b5f0 tx=0x7f2fb8009000 comp rx=0 tx=0).stop 2024-09-17T18:20:02.380 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.374+0000 7f2fc5d77780 10 monclient: shutdown 2024-09-17T18:20:02.380 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.374+0000 7f2fc5d77780 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:20:02.380 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.374+0000 7f2fc5d77780 1 -- 172.21.15.173:0/56431533 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5588a7b1fa60 msgr2=0x5588a7b1ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:20:02.380 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.374+0000 7f2fc5d77780 1 --2- 172.21.15.173:0/56431533 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5588a7b1fa60 0x5588a7b1ead0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2fb4029d60 tx=0x7f2fb4002fb0 comp rx=0 tx=0).stop 2024-09-17T18:20:02.380 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.374+0000 7f2fc4066640 1 -- 172.21.15.173:0/56431533 reap_dead start 2024-09-17T18:20:02.380 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.374+0000 7f2fc5d77780 1 -- 172.21.15.173:0/56431533 shutdown_connections 2024-09-17T18:20:02.380 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.374+0000 7f2fc5d77780 1 -- 172.21.15.173:0/56431533 >> 172.21.15.173:0/56431533 conn(0x5588a7b03bf0 msgr2=0x5588a7bc5a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:20:02.380 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.374+0000 7f2fc5d77780 1 -- 172.21.15.173:0/56431533 shutdown_connections 2024-09-17T18:20:02.380 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.374+0000 7f2fc5d77780 1 -- 172.21.15.173:0/56431533 wait complete. 2024-09-17T18:20:02.381 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.375+0000 7f2fc5d77780 1 librados: shutdown 2024-09-17T18:20:02.381 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: [----------] 12 tests from AsioRados (56 ms total) 2024-09-17T18:20:02.381 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: 2024-09-17T18:20:02.381 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: [----------] Global test environment tear-down 2024-09-17T18:20:02.381 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (1695 ms total) 2024-09-17T18:20:02.381 INFO:tasks.workunit.client.0.smithi173.stdout: api_asio: [ PASSED ] 12 tests. 2024-09-17T18:20:02.891 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-09-17T18:20:02.891 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: [----------] Global test environment set-up. 2024-09-17T18:20:02.891 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: [----------] 7 tests from LibRadosList 2024-09-17T18:20:02.891 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-09-17T18:20:02.891 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: [ OK ] LibRadosList.ListObjects (300 ms) 2024-09-17T18:20:02.891 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-09-17T18:20:02.891 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (40 ms) 2024-09-17T18:20:02.891 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-09-17T18:20:02.891 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: myset foo1,foo2,foo3 2024-09-17T18:20:02.892 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: foo1 2024-09-17T18:20:02.892 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: foo2 2024-09-17T18:20:02.892 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: foo3 2024-09-17T18:20:02.892 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: myset foo1,foo4,foo5 2024-09-17T18:20:02.892 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: foo4 2024-09-17T18:20:02.892 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: foo5 2024-09-17T18:20:02.892 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: foo1 2024-09-17T18:20:02.892 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: myset foo6,foo7 2024-09-17T18:20:02.892 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: foo7 2024-09-17T18:20:02.892 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: foo6 2024-09-17T18:20:02.892 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-09-17T18:20:02.892 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: ns1:foo4 2024-09-17T18:20:02.892 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: ns1:foo5 2024-09-17T18:20:02.892 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: ns2:foo7 2024-09-17T18:20:02.893 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: ns2:foo6 2024-09-17T18:20:02.893 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: ns1:foo1 2024-09-17T18:20:02.893 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: :foo1 2024-09-17T18:20:02.893 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: :foo2 2024-09-17T18:20:02.893 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: :foo3 2024-09-17T18:20:02.893 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (118 ms) 2024-09-17T18:20:02.893 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-09-17T18:20:02.893 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: 1 0 2024-09-17T18:20:02.893 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: 10 0 2024-09-17T18:20:02.893 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: 13 0 2024-09-17T18:20:02.893 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: 7 0 2024-09-17T18:20:02.893 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: 14 0 2024-09-17T18:20:02.893 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: 0 0 2024-09-17T18:20:02.894 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: 15 0 2024-09-17T18:20:02.894 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: 11 0 2024-09-17T18:20:02.894 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: 5 0 2024-09-17T18:20:02.894 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: 8 0 2024-09-17T18:20:02.894 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: 6 0 2024-09-17T18:20:02.894 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: 3 0 2024-09-17T18:20:02.894 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: 4 0 2024-09-17T18:20:02.894 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: 12 0 2024-09-17T18:20:02.894 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: 9 0 2024-09-17T18:20:02.894 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: 2 0 2024-09-17T18:20:02.894 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-09-17T18:20:02.894 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (154 ms) 2024-09-17T18:20:02.894 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-09-17T18:20:02.894 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: x cursor=MIN 2024-09-17T18:20:02.894 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > oid=1 cursor=8:02547ec2:::1:head 2024-09-17T18:20:02.895 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > oid=10 cursor=8:52ea6a34:::10:head 2024-09-17T18:20:02.895 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > oid=13 cursor=8:566253c9:::13:head 2024-09-17T18:20:02.895 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > oid=7 cursor=8:5c6b0b28:::7:head 2024-09-17T18:20:02.895 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > oid=14 cursor=8:62a1935d:::14:head 2024-09-17T18:20:02.895 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > oid=0 cursor=8:6cac518f:::0:head 2024-09-17T18:20:02.895 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > oid=15 cursor=8:863748b0:::15:head 2024-09-17T18:20:02.895 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > oid=11 cursor=8:89d3ae78:::11:head 2024-09-17T18:20:02.895 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > oid=5 cursor=8:b29083e3:::5:head 2024-09-17T18:20:02.895 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > oid=8 cursor=8:bd63b0f1:::8:head 2024-09-17T18:20:02.895 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > oid=6 cursor=8:c4fdafeb:::6:head 2024-09-17T18:20:02.895 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > oid=3 cursor=8:cfc208b3:::3:head 2024-09-17T18:20:02.895 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > oid=4 cursor=8:d83876eb:::4:head 2024-09-17T18:20:02.895 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > oid=12 cursor=8:de5d7c5f:::12:head 2024-09-17T18:20:02.895 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > oid=9 cursor=8:e960b815:::9:head 2024-09-17T18:20:02.896 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > oid=2 cursor=8:f905c69b:::2:head 2024-09-17T18:20:02.896 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: FIRST> seek to MIN oid=1 2024-09-17T18:20:02.896 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : oid=1 cursor=8:02547ec2:::1:head 2024-09-17T18:20:02.896 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : seek to 8:02547ec2:::1:head 2024-09-17T18:20:02.896 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > 8:02547ec2:::1:head -> 1 2024-09-17T18:20:02.896 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : oid=10 cursor=8:52ea6a34:::10:head 2024-09-17T18:20:02.896 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : seek to 8:52ea6a34:::10:head 2024-09-17T18:20:02.896 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > 8:52ea6a34:::10:head -> 10 2024-09-17T18:20:02.896 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : oid=13 cursor=8:566253c9:::13:head 2024-09-17T18:20:02.896 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : seek to 8:566253c9:::13:head 2024-09-17T18:20:02.896 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > 8:566253c9:::13:head -> 13 2024-09-17T18:20:02.896 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : oid=7 cursor=8:5c6b0b28:::7:head 2024-09-17T18:20:02.896 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : seek to 8:5c6b0b28:::7:head 2024-09-17T18:20:02.896 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > 8:5c6b0b28:::7:head -> 7 2024-09-17T18:20:02.897 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : oid=14 cursor=8:62a1935d:::14:head 2024-09-17T18:20:02.968 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : seek to 8:62a1935d:::14:head 2024-09-17T18:20:02.969 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > 8:62a1935d:::14:head -> 14 2024-09-17T18:20:02.969 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : oid=0 cursor=8:6cac518f:::0:head 2024-09-17T18:20:02.969 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : seek to 8:6cac518f:::0:head 2024-09-17T18:20:02.969 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > 8:6cac518f:::0:head -> 0 2024-09-17T18:20:02.969 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : oid=15 cursor=8:863748b0:::15:head 2024-09-17T18:20:02.969 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : seek to 8:863748b0:::15:head 2024-09-17T18:20:02.969 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > 8:863748b0:::15:head -> 15 2024-09-17T18:20:02.969 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : oid=11 cursor=8:89d3ae78:::11:head 2024-09-17T18:20:02.969 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : seek to 8:89d3ae78:::11:head 2024-09-17T18:20:02.969 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > 8:89d3ae78:::11:head -> 11 2024-09-17T18:20:02.969 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : oid=5 cursor=8:b29083e3:::5:head 2024-09-17T18:20:02.969 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : seek to 8:b29083e3:::5:head 2024-09-17T18:20:02.970 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > 8:b29083e3:::5:head -> 5 2024-09-17T18:20:02.970 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : oid=8 cursor=8:bd63b0f1:::8:head 2024-09-17T18:20:02.970 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : seek to 8:bd63b0f1:::8:head 2024-09-17T18:20:02.970 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > 8:bd63b0f1:::8:head -> 8 2024-09-17T18:20:02.970 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : oid=6 cursor=8:c4fdafeb:::6:head 2024-09-17T18:20:02.970 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : seek to 8:c4fdafeb:::6:head 2024-09-17T18:20:02.970 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > 8:c4fdafeb:::6:head -> 6 2024-09-17T18:20:02.970 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : oid=3 cursor=8:cfc208b3:::3:head 2024-09-17T18:20:02.970 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : seek to 8:cfc208b3:::3:head 2024-09-17T18:20:02.970 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > 8:cfc208b3:::3:head -> 3 2024-09-17T18:20:02.970 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : oid=4 cursor=8:d83876eb:::4:head 2024-09-17T18:20:02.970 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : seek to 8:d83876eb:::4:head 2024-09-17T18:20:02.970 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > 8:d83876eb:::4:head -> 4 2024-09-17T18:20:02.970 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : oid=12 cursor=8:de5d7c5f:::12:head 2024-09-17T18:20:02.971 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : seek to 8:de5d7c5f:::12:head 2024-09-17T18:20:02.971 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > 8:de5d7c5f:::12:head -> 12 2024-09-17T18:20:02.971 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : oid=9 cursor=8:e960b815:::9:head 2024-09-17T18:20:02.971 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : seek to 8:e960b815:::9:head 2024-09-17T18:20:02.971 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > 8:e960b815:::9:head -> 9 2024-09-17T18:20:02.971 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : oid=2 cursor=8:f905c69b:::2:head 2024-09-17T18:20:02.971 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : seek to 8:f905c69b:::2:head 2024-09-17T18:20:02.971 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > 8:f905c69b:::2:head -> 2 2024-09-17T18:20:02.971 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : seek to 8:52ea6a34:::10:head 2024-09-17T18:20:02.971 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : cursor()=8:52ea6a34:::10:head expected=8:52ea6a34:::10:head 2024-09-17T18:20:02.971 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > 8:52ea6a34:::10:head -> 10 2024-09-17T18:20:02.971 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : entry=10 expected=10 2024-09-17T18:20:02.971 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : seek to 8:02547ec2:::1:head 2024-09-17T18:20:02.971 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : cursor()=8:02547ec2:::1:head expected=8:02547ec2:::1:head 2024-09-17T18:20:02.972 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > 8:02547ec2:::1:head -> 1 2024-09-17T18:20:02.972 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : entry=1 expected=1 2024-09-17T18:20:02.972 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : seek to 8:de5d7c5f:::12:head 2024-09-17T18:20:02.972 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : cursor()=8:de5d7c5f:::12:head expected=8:de5d7c5f:::12:head 2024-09-17T18:20:02.972 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > 8:de5d7c5f:::12:head -> 12 2024-09-17T18:20:02.972 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : entry=12 expected=12 2024-09-17T18:20:02.972 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : seek to 8:d83876eb:::4:head 2024-09-17T18:20:02.972 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : cursor()=8:d83876eb:::4:head expected=8:d83876eb:::4:head 2024-09-17T18:20:02.972 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > 8:d83876eb:::4:head -> 4 2024-09-17T18:20:02.972 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : entry=4 expected=4 2024-09-17T18:20:02.972 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : seek to 8:cfc208b3:::3:head 2024-09-17T18:20:02.972 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : cursor()=8:cfc208b3:::3:head expected=8:cfc208b3:::3:head 2024-09-17T18:20:02.973 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > 8:cfc208b3:::3:head -> 3 2024-09-17T18:20:02.973 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : entry=3 expected=3 2024-09-17T18:20:02.973 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : seek to 8:c4fdafeb:::6:head 2024-09-17T18:20:02.973 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : cursor()=8:c4fdafeb:::6:head expected=8:c4fdafeb:::6:head 2024-09-17T18:20:02.973 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > 8:c4fdafeb:::6:head -> 6 2024-09-17T18:20:02.973 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : entry=6 expected=6 2024-09-17T18:20:02.973 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : seek to 8:b29083e3:::5:head 2024-09-17T18:20:02.973 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : cursor()=8:b29083e3:::5:head expected=8:b29083e3:::5:head 2024-09-17T18:20:02.973 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > 8:b29083e3:::5:head -> 5 2024-09-17T18:20:02.973 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : entry=5 expected=5 2024-09-17T18:20:02.973 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : seek to 8:5c6b0b28:::7:head 2024-09-17T18:20:02.973 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : cursor()=8:5c6b0b28:::7:head expected=8:5c6b0b28:::7:head 2024-09-17T18:20:02.974 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > 8:5c6b0b28:::7:head -> 7 2024-09-17T18:20:02.974 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : entry=7 expected=7 2024-09-17T18:20:02.974 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : seek to 8:f905c69b:::2:head 2024-09-17T18:20:03.166 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: cls: Running main() from gmock_main.cc 2024-09-17T18:20:03.166 INFO:tasks.workunit.client.0.smithi173.stdout: cls: [==========] Running 1 test from 1 test suite. 2024-09-17T18:20:03.166 INFO:tasks.workunit.client.0.smithi173.stdout: cls: [----------] Global test environment set-up. 2024-09-17T18:20:03.167 INFO:tasks.workunit.client.0.smithi173.stdout: cls: [----------] 1 test from NeoRadosCls 2024-09-17T18:20:03.167 INFO:tasks.workunit.client.0.smithi173.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-09-17T18:20:03.167 INFO:tasks.workunit.client.0.smithi173.stdout: cls: [ OK ] NeoRadosCls.DNE (2236 ms) 2024-09-17T18:20:03.167 INFO:tasks.workunit.client.0.smithi173.stdout: cls: [----------] 1 test from NeoRadosCls (2236 ms total) 2024-09-17T18:20:03.167 INFO:tasks.workunit.client.0.smithi173.stdout: cls: 2024-09-17T18:20:03.167 INFO:tasks.workunit.client.0.smithi173.stdout: cls: [----------] Global test environment tear-down 2024-09-17T18:20:03.167 INFO:tasks.workunit.client.0.smithi173.stdout: cls: [==========] 1 test from 1 test suite ran. (2236 ms total) 2024-09-17T18:20:03.167 INFO:tasks.workunit.client.0.smithi173.stdout: cls: [ PASSED ] 1 test. 2024-09-17T18:20:03.182 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-09-17T18:20:03.182 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-09-17T18:20:03.182 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-09-17T18:20:03.183 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-09-17T18:20:03.183 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-09-17T18:20:03.183 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-09-17T18:20:03.183 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-09-17T18:20:03.183 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (217 ms) 2024-09-17T18:20:03.183 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-09-17T18:20:03.183 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (25 ms) 2024-09-17T18:20:03.183 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-09-17T18:20:03.183 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (11 ms) 2024-09-17T18:20:03.183 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-09-17T18:20:03.183 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (43 ms) 2024-09-17T18:20:03.183 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-09-17T18:20:03.184 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (11 ms) 2024-09-17T18:20:03.184 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-09-17T18:20:03.184 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (15 ms) 2024-09-17T18:20:03.184 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-09-17T18:20:03.184 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (13 ms) 2024-09-17T18:20:03.184 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-09-17T18:20:03.184 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (9 ms) 2024-09-17T18:20:03.184 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-09-17T18:20:03.184 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (10 ms) 2024-09-17T18:20:03.184 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-09-17T18:20:03.184 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (5 ms) 2024-09-17T18:20:03.184 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-09-17T18:20:03.185 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (11 ms) 2024-09-17T18:20:03.185 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-09-17T18:20:03.185 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (5 ms) 2024-09-17T18:20:03.185 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-09-17T18:20:03.185 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (27 ms) 2024-09-17T18:20:03.185 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-09-17T18:20:03.185 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (32 ms) 2024-09-17T18:20:03.185 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-09-17T18:20:03.185 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (26 ms) 2024-09-17T18:20:03.185 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-09-17T18:20:03.185 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (5 ms) 2024-09-17T18:20:03.186 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (465 ms total) 2024-09-17T18:20:03.186 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: 2024-09-17T18:20:03.186 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-09-17T18:20:03.186 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (2303 ms total) 2024-09-17T18:20:03.186 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-09-17T18:20:03.186 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-09-17T18:20:03.186 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-09-17T18:20:03.186 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-09-17T18:20:03.186 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-09-17T18:20:03.187 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-09-17T18:20:03.187 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (68 ms) 2024-09-17T18:20:03.187 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-09-17T18:20:03.187 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (83 ms) 2024-09-17T18:20:03.187 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-09-17T18:20:03.187 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (2192 ms) 2024-09-17T18:20:03.187 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (2343 ms total) 2024-09-17T18:20:03.187 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd_pp: 2024-09-17T18:20:03.187 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-09-17T18:20:03.187 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (2343 ms total) 2024-09-17T18:20:03.187 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-09-17T18:20:03.190 INFO:tasks.workunit.client.0.smithi173.stdout: handler_error: Running main() from gmock_main.cc 2024-09-17T18:20:03.191 INFO:tasks.workunit.client.0.smithi173.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-09-17T18:20:03.191 INFO:tasks.workunit.client.0.smithi173.stdout: handler_error: [----------] Global test environment set-up. 2024-09-17T18:20:03.191 INFO:tasks.workunit.client.0.smithi173.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-09-17T18:20:03.191 INFO:tasks.workunit.client.0.smithi173.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-09-17T18:20:03.191 INFO:tasks.workunit.client.0.smithi173.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (2179 ms) 2024-09-17T18:20:03.191 INFO:tasks.workunit.client.0.smithi173.stdout: handler_error: [----------] 1 test from neocls_handler_error (2179 ms total) 2024-09-17T18:20:03.191 INFO:tasks.workunit.client.0.smithi173.stdout: handler_error: 2024-09-17T18:20:03.191 INFO:tasks.workunit.client.0.smithi173.stdout: handler_error: [----------] Global test environment tear-down 2024-09-17T18:20:03.191 INFO:tasks.workunit.client.0.smithi173.stdout: handler_error: [==========] 1 test from 1 test suite ran. (2179 ms total) 2024-09-17T18:20:03.191 INFO:tasks.workunit.client.0.smithi173.stdout: handler_error: [ PASSED ] 1 test. 2024-09-17T18:20:04.262 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: Running main() from gmock_main.cc 2024-09-17T18:20:04.262 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-09-17T18:20:04.262 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: [----------] Global test environment set-up. 2024-09-17T18:20:04.262 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-09-17T18:20:04.262 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-09-17T18:20:04.262 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (79 ms) 2024-09-17T18:20:04.262 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-09-17T18:20:04.262 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (61 ms) 2024-09-17T18:20:04.262 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-09-17T18:20:04.262 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (3210 ms) 2024-09-17T18:20:04.262 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-09-17T18:20:04.263 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:03.129856+0000 mon.a [INF] from='client.? 172.21.15.173:0/2793384439' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi173-36188-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi173-36188-1"}]': finished 2024-09-17T18:20:04.263 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:03.129918+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi173-35451-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi173-35451-1"}]': finished 2024-09-17T18:20:04.263 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:03.129979+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi173-36016-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi173-36016-1"}]': finished 2024-09-17T18:20:04.263 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:03.130046+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi173-35536-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-17T18:20:04.263 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:03.130101+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi173-34988-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-17T18:20:04.263 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:03.143168+0000 mon.a [INF] from='client.? 172.21.15.173:0/1292664106' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi173-35399-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-17T18:20:04.263 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:03.336667+0000 mon.b [INF] from='client.? 172.21.15.173:0/4178523083' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi173-35511-7"} : dispatch 2024-09-17T18:20:04.263 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:03.337403+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi173-35511-7"} : dispatch 2024-09-17T18:20:04.263 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:03.345352+0000 mon.b [INF] from='client.? 172.21.15.173:0/4178523083' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi173-35511-7"} : dispatch 2024-09-17T18:20:04.263 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:03.345898+0000 mon.b [INF] from='client.? 172.21.15.173:0/3116255395' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi173-35349-7"} : dispatch 2024-09-17T18:20:04.263 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:03.349901+0000 mon.a [INF] from='client.? 172.21.15.173:0/2416687734' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi173-35030-16"} : dispatch 2024-09-17T18:20:04.263 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:03.350074+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi173-35511-7"} : dispatch 2024-09-17T18:20:04.944 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:03.350409+0000 mon.a [INF] from='client.? 172.21.15.173:0/2416687734' entity='client. list_parallel: process_1_[35844]: starting. 2024-09-17T18:20:04.944 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_1_[35844]: creating pool ceph_test_rados_list_parallel.smithi173-35767 2024-09-17T18:20:04.944 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_1_[35844]: created object 0... 2024-09-17T18:20:04.944 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_1_[35844]: created object 25... 2024-09-17T18:20:04.944 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_1_[35844]: created object 49... 2024-09-17T18:20:04.944 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_1_[35844]: finishing. 2024-09-17T18:20:04.944 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_1_[35844]: shutting down. 2024-09-17T18:20:04.944 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_2_[35846]: starting. 2024-09-17T18:20:04.945 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_2_[35846]: listing objects. 2024-09-17T18:20:04.945 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_2_[35846]: listed object 0... 2024-09-17T18:20:04.945 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_2_[35846]: listed object 25... 2024-09-17T18:20:04.945 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_2_[35846]: saw 50 objects 2024-09-17T18:20:04.945 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_2_[35846]: shutting down. 2024-09-17T18:20:04.945 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: ******************************* 2024-09-17T18:20:04.945 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_3_[36548]: starting. 2024-09-17T18:20:04.945 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_3_[36548]: creating pool ceph_test_rados_list_parallel.smithi173-35767 2024-09-17T18:20:04.945 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_3_[36548]: created object 0... 2024-09-17T18:20:04.945 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_3_[36548]: created object 25... 2024-09-17T18:20:04.945 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_3_[36548]: created object 49... 2024-09-17T18:20:04.945 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_3_[36548]: finishing. 2024-09-17T18:20:04.945 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_3_[36548]: shutting down. 2024-09-17T18:20:04.946 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: ******************************* 2024-09-17T18:20:04.946 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_4_[36549]: starting. 2024-09-17T18:20:04.946 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_4_[36549]: listing objects. 2024-09-17T18:20:04.946 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_4_[36549]: listed object 0... 2024-09-17T18:20:04.946 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_4_[36549]: listed object 25... 2024-09-17T18:20:04.946 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_4_[36549]: saw 44 objects 2024-09-17T18:20:04.946 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_4_[36549]: shutting down. 2024-09-17T18:20:04.946 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: ******************************* 2024-09-17T18:20:04.946 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_5_[36550]: starting. 2024-09-17T18:20:04.946 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_5_[36550]: removed 25 objects... 2024-09-17T18:20:04.946 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_5_[36550]: removed half of the objects 2024-09-17T18:20:04.946 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_5_[36550]: removed 50 objects... 2024-09-17T18:20:04.946 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_5_[36550]: removed 50 objects 2024-09-17T18:20:04.946 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_5_[36550]: shutting down. 2024-09-17T18:20:04.947 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: ******************************* 2024-09-17T18:20:04.947 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: ******************************* 2024-09-17T18:20:04.947 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_6_[36599]: starting. 2024-09-17T18:20:04.947 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_6_[36599]: creating pool ceph_test_rados_list_parallel.smithi173-35767 2024-09-17T18:20:04.947 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_6_[36599]: created object 0... 2024-09-17T18:20:04.947 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_6_[36599]: created object 25... 2024-09-17T18:20:04.947 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_6_[36599]: created object 49... 2024-09-17T18:20:04.947 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_6_[36599]: finishing. 2024-09-17T18:20:04.947 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_6_[36599]: shutting down. 2024-09-17T18:20:04.947 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: ******************************* 2024-09-17T18:20:04.947 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: ******************************* 2024-09-17T18:20:04.947 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_7_[36600]: starting. 2024-09-17T18:20:04.947 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_7_[36600]: listing objects. 2024-09-17T18:20:04.948 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_7_[36600]: listed object 0... 2024-09-17T18:20:04.948 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_7_[36600]: listed object 25... 2024-09-17T18:20:04.948 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_7_[36600]: listed object 50... 2024-09-17T18:20:04.948 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_7_[36600]: saw 57 objects 2024-09-17T18:20:04.948 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_7_[36600]: shutting down. 2024-09-17T18:20:04.948 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: ******************************* 2024-09-17T18:20:04.948 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: ******************************* 2024-09-17T18:20:04.948 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_8_[36601]: starting. 2024-09-17T18:20:04.948 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_8_[36601]: added 25 objects... 2024-09-17T18:20:04.948 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_8_[36601]: added half of the objects 2024-09-17T18:20:04.948 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_8_[36601]: added 50 objects... 2024-09-17T18:20:04.948 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_8_[36601]: added 50 objects 2024-09-17T18:20:04.948 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_8_[36601]: shutting down. 2024-09-17T18:20:04.948 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: ******************************* 2024-09-17T18:20:04.949 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: ******************************* 2024-09-17T18:20:05.181 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: **********************admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi173-35030-16"} : dispatch 2024-09-17T18:20:05.181 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:03.350873+0000 mon.a [INF] from='client.? 172.21.15.173:0/2416687734' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi173-35030-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-17T18:20:05.181 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:03.353059+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi173-35349-7"} : dispatch 2024-09-17T18:20:05.181 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:03.354350+0000 mon.b [INF] from='client.? 172.21.15.173:0/4178523083' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi173-35511-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-17T18:20:05.181 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:03.357813+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi173-35511-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-17T18:20:05.181 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:03.508473+0000 mon.b [INF] from='client.? 172.21.15.173:0/3116255395' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi173-35349-7"} : dispatch 2024-09-17T18:20:05.181 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:03.508760+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi173-35349-7"} : dispatch 2024-09-17T18:20:05.181 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:03.883286+0000 mon.b [INF] from='client.? 172.21.15.173:0/3116255395' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi173-35349-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-17T18:20:05.182 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:03.883744+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi173-35349-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-17T18:20:05.182 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:04.137362+0000 mon.a [INF] from='client.? 172.21.15.173:0/1292664106' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi173-35399-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-17T18:20:05.182 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:04.137425+0000 mon.a [INF] from='client.? 172.21.15.173:0/2416687734' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi173-35030-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-17T18:20:05.182 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:04.137465+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi173-35511-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-17T18:20:05.182 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:04.137514+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi173-35349-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-17T18:20:05.182 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:04.142320+0000 mon.b [INF] from='client.? 172.21.15.173:0/3116255395' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi173-35349-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi173-35349-7"} : dispatch 2024-09-17T18:20:05.481 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:04.142429+0000 mon.b [INF] from='client.? 172.21.15.173:0/4178523083' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi173-35511-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofi********* 2024-09-17T18:20:05.481 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_9_[36883]: starting. 2024-09-17T18:20:05.481 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_9_[36883]: creating pool ceph_test_rados_list_parallel.smithi173-35767 2024-09-17T18:20:05.481 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_9_[36883]: created object 0... 2024-09-17T18:20:05.481 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_9_[36883]: created object 25... 2024-09-17T18:20:05.482 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_9_[36883]: created object 49... 2024-09-17T18:20:05.482 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_9_[36883]: finishing. 2024-09-17T18:20:05.482 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_9_[36883]: shutting down. 2024-09-17T18:20:05.482 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: ******************************* 2024-09-17T18:20:05.482 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: ******************************* 2024-09-17T18:20:05.482 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: ******************************* 2024-09-17T18:20:05.482 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_10_[36884]: starting. 2024-09-17T18:20:05.482 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_10_[36884]: listing objects. 2024-09-17T18:20:05.482 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_10_[36884]: listed object 0... 2024-09-17T18:20:05.482 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_10_[36884]: listed object 25... 2024-09-17T18:20:05.482 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_10_[36884]: listed object 50... 2024-09-17T18:20:05.482 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_10_[36884]: listed object 75... 2024-09-17T18:20:05.482 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_10_[36884]: listed object 100... 2024-09-17T18:20:05.482 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_10_[36884]: saw 103 objects 2024-09-17T18:20:05.483 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_10_[36884]: shutting down. 2024-09-17T18:20:05.483 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: ******************************* 2024-09-17T18:20:05.483 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: ******************************* 2024-09-17T18:20:05.483 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: ******************************* 2024-09-17T18:20:05.483 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_12_[36886]: starting. 2024-09-17T18:20:05.483 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_12_[36886]: added 25 objects... 2024-09-17T18:20:05.483 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_12_[36886]: added half of the objects 2024-09-17T18:20:05.483 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_12_[36886]: added 50 objects... 2024-09-17T18:20:05.483 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_12_[36886]: added 50 objects 2024-09-17T18:20:05.483 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_12_[36886]: shutting down. 2024-09-17T18:20:05.483 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: ******************************* 2024-09-17T18:20:05.483 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: ******************************* 2024-09-17T18:20:05.483 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: ******************************* 2024-09-17T18:20:05.483 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_13_[36887]: starting. 2024-09-17T18:20:05.484 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_13_[36887]: removed 25 objects... 2024-09-17T18:20:05.484 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_13_[36887]: removed half of the objects 2024-09-17T18:20:05.484 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_13_[36887]: removed 50 objects... 2024-09-17T18:20:05.484 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_13_[36887]: removed 50 objects 2024-09-17T18:20:05.484 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_13_[36887]: shutting down. 2024-09-17T18:20:05.484 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: ******************************* 2024-09-17T18:20:05.484 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: ******************************* 2024-09-17T18:20:05.484 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: ******************************* 2024-09-17T18:20:05.484 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_11_[36885]: starting. 2024-09-17T18:20:05.484 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_11_[36885]: added 25 objects... 2024-09-17T18:20:05.484 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_11_[36885]: added half of the objects 2024-09-17T18:20:05.484 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_11_[36885]: added 50 objects... 2024-09-17T18:20:05.484 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_11_[36885]: added 50 objects 2024-09-17T18:20:05.485 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_11_[36885]: shutting down. 2024-09-17T18:20:05.485 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: ******************************* 2024-09-17T18:20:05.485 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: ******************************* 2024-09-17T18:20:05.485 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: ******************************* 2024-09-17T18:20:05.485 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: ******************************* 2024-09-17T18:20:05.485 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_14_[37081]: starting. 2024-09-17T18:20:05.485 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_14_[37081]: creating pool ceph_test_rados_list_parallel.smithi173-35767 2024-09-17T18:20:05.485 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_14_[37081]: created object 0... 2024-09-17T18:20:05.485 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_14_[37081]: created object 25... 2024-09-17T18:20:05.485 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_14_[37081]: created object 49... 2024-09-17T18:20:05.485 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_14_[37081]: finishing. 2024-09-17T18:20:05.485 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_14_[37081]: shutting down. 2024-09-17T18:20:05.485 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: ******************************* 2024-09-17T18:20:05.485 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: ******************************* 2024-09-17T18:20:05.486 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: ******************************* 2024-09-17T18:20:05.486 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: ******************************* 2024-09-17T18:20:05.486 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_15_[37082]: starting. 2024-09-17T18:20:05.486 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_15_[37082]: listing objects. 2024-09-17T18:20:06.178 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_15_[37082]: listed object 0... 2024-09-17T18:20:06.178 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_15_[37082]: listed object 25... 2024-09-17T18:20:06.178 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_15_[37082]: listed object 50... 2024-09-17T18:20:06.178 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_15_[37082]: listed object 75... 2024-09-17T18:20:06.178 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_15_[37082]: listed object 100... 2024-09-17T18:20:06.179 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_15_[37082]: listed object 125... 2024-09-17T18:20:06.179 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_15_[37082]: saw 150 objects 2024-09-17T18:20:06.179 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_15_[37082]: shutting down. 2024-09-17T18:20:06.179 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: ******************************* 2024-09-17T18:20:06.179 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: ******************************* 2024-09-17T18:20:06.179 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: ******************************* 2024-09-17T18:20:06.179 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: ******************************* 2024-09-17T18:20:06.179 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_16_[37083]: starting. 2024-09-17T18:20:06.179 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_16_[37083]: added 25 objects... 2024-09-17T18:20:06.179 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_16_[37083]: added half of the objects 2024-09-17T18:20:06.179 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_16_[37083]: added 50 objects... 2024-09-17T18:20:06.179 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_16_[37083]: added 50 objects 2024-09-17T18:20:06.179 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: process_16_[37083]: shutting down. 2024-09-17T18:20:06.179 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: ******************************* 2024-09-17T18:20:06.180 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: ******************************* 2024-09-17T18:20:06.180 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: ******************************* 2024-09-17T18:20:06.180 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: ******************************* 2024-09-17T18:20:06.180 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: ******************************* 2024-09-17T18:20:06.180 INFO:tasks.workunit.client.0.smithi173.stdout: list_parallel: ******* SUCCESS ********** 2024-09-17T18:20:06.188 INFO:tasks.workunit.client.0.smithi173.stdout: delete_pools_parallel: process_1_[35946]: starting. 2024-09-17T18:20:06.188 INFO:tasks.workunit.client.0.smithi173.stdout: delete_pools_parallel: process_1_[35946]: creating pool ceph_test_rados_delete_pools_parallel.smithi173-35837 2024-09-17T18:20:06.189 INFO:tasks.workunit.client.0.smithi173.stdout: delete_pools_parallel: process_1_[35946]: created object 0... 2024-09-17T18:20:06.189 INFO:tasks.workunit.client.0.smithi173.stdout: delete_pools_parallel: process_1_[35946]: created object 25... 2024-09-17T18:20:06.189 INFO:tasks.workunit.client.0.smithi173.stdout: delete_pools_parallel: process_1_[35946]: created object 49... 2024-09-17T18:20:06.189 INFO:tasks.workunit.client.0.smithi173.stdout: delete_pools_parallel: process_1_[35946]: finishing. 2024-09-17T18:20:06.189 INFO:tasks.workunit.client.0.smithi173.stdout: delete_pools_parallel: process_1_[35946]: shutting down. 2024-09-17T18:20:06.189 INFO:tasks.workunit.client.0.smithi173.stdout: delete_pools_parallel: process_2_[35948]: starting. 2024-09-17T18:20:06.189 INFO:tasks.workunit.client.0.smithi173.stdout: delete_pools_parallel: process_2_[35948]: deleting pool ceph_test_rados_delete_pools_parallel.smithi173-35837 2024-09-17T18:20:06.189 INFO:tasks.workunit.client.0.smithi173.stdout: delete_pools_parallel: process_2_[35948]: shutting down. 2024-09-17T18:20:06.189 INFO:tasks.workunit.client.0.smithi173.stdout: delete_pools_parallel: ******************************* 2024-09-17T18:20:06.189 INFO:tasks.workunit.client.0.smithi173.stdout: delete_pools_parallel: process_3_[36650]: starting. 2024-09-17T18:20:06.189 INFO:tasks.workunit.client.0.smithi173.stdout: delete_pools_parallel: process_3_[36650]: creating pool ceph_test_rados_delete_pools_parallel.smithi173-35837 2024-09-17T18:20:06.189 INFO:tasks.workunit.client.0.smithi173.stdout: delete_pools_parallel: process_3_[36650]: created object 0... 2024-09-17T18:20:06.189 INFO:tasks.workunit.client.0.smithi173.stdout: delete_pools_parallel: process_3_[36650]: created object 25... 2024-09-17T18:20:06.189 INFO:tasks.workunit.client.0.smithi173.stdout: delete_pools_parallel: process_3_[36650]: created object 49... 2024-09-17T18:20:06.190 INFO:tasks.workunit.client.0.smithi173.stdout: delete_pools_parallel: process_3_[36650]: finishing. 2024-09-17T18:20:06.190 INFO:tasks.workunit.client.0.smithi173.stdout: delete_pools_parallel: process_3_[36650]: shutting down. 2024-09-17T18:20:06.190 INFO:tasks.workunit.client.0.smithi173.stdout: delete_pools_parallel: ******************************* 2024-09-17T18:20:06.190 INFO:tasks.workunit.client.0.smithi173.stdout: delete_pools_parallel: process_5_[36652]: starting. 2024-09-17T18:20:06.190 INFO:tasks.workunit.client.0.smithi173.stdout: delete_pools_parallel: process_5_[36652]: listing objects. 2024-09-17T18:20:06.190 INFO:tasks.workunit.client.0.smithi173.stdout: delete_pools_parallel: process_5_[36652]: listed object 0... 2024-09-17T18:20:06.190 INFO:tasks.workunit.client.0.smithi173.stdout: delete_pools_parallel: process_5_[36652]: listed object 25... 2024-09-17T18:20:06.190 INFO:tasks.workunit.client.0.smithi173.stdout: delete_pools_parallel: process_5_[36652]: saw 50 objects 2024-09-17T18:20:06.190 INFO:tasks.workunit.client.0.smithi173.stdout: delete_pools_parallel: process_5_[36652]: shutting down. 2024-09-17T18:20:06.190 INFO:tasks.workunit.client.0.smithi173.stdout: delete_pools_parallel: ******************************* 2024-09-17T18:20:06.190 INFO:tasks.workunit.client.0.smithi173.stdout: delete_pools_parallel: process_4_[36651]: starting. 2024-09-17T18:20:06.190 INFO:tasks.workunit.client.0.smithi173.stdout: delete_pools_parallel: process_4_[36651]: deleting pool ceph_test_rados_delete_pools_parallel.smithi173-35837 2024-09-17T18:20:06.190 INFO:tasks.workunit.client.0.smithi173.stdout: delete_pools_parallel: process_4_[36651]: shutting down. 2024-09-17T18:20:06.190 INFO:tasks.workunit.client.0.smithi173.stdout: delete_pools_parallel: ******************************* 2024-09-17T18:20:06.190 INFO:tasks.workunit.client.0.smithi173.stdout: delete_pools_parallel: ******************************* 2024-09-17T18:20:06.190 INFO:tasks.workunit.client.0.smithi173.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-09-17T18:20:06.208 INFO:tasks.workunit.client.0.smithi173.stdout: open_pools_parallel: process_1_[35906]: starting. 2024-09-17T18:20:06.208 INFO:tasks.workunit.client.0.smithi173.stdout: open_pools_parallel: process_1_[35906]: creating pool ceph_test_rados_open_pools_parallel.smithi173-35714 2024-09-17T18:20:06.208 INFO:tasks.workunit.client.0.smithi173.stdout: open_pools_parallel: process_1_[35906]: created object 0... 2024-09-17T18:20:06.208 INFO:tasks.workunit.client.0.smithi173.stdout: open_pools_parallel: process_1_[35906]: created object 25... 2024-09-17T18:20:06.208 INFO:tasks.workunit.client.0.smithi173.stdout: open_pools_parallel: process_1_[35906]: created object 49... 2024-09-17T18:20:06.208 INFO:tasks.workunit.client.0.smithi173.stdout: open_pools_parallel: process_1_[35906]: finishing. 2024-09-17T18:20:06.208 INFO:tasks.workunit.client.0.smithi173.stdout: open_pools_parallel: process_1_[35906]: shutting down. 2024-09-17T18:20:06.208 INFO:tasks.workunit.client.0.smithi173.stdout: open_pools_parallel: process_2_[35907]: starting. 2024-09-17T18:20:06.208 INFO:tasks.workunit.client.0.smithi173.stdout: open_pools_parallel: process_2_[35907]: rados_pool_create. 2024-09-17T18:20:06.209 INFO:tasks.workunit.client.0.smithi173.stdout: open_pools_parallel: process_2_[35907]: rados_ioctx_create. 2024-09-17T18:20:06.209 INFO:tasks.workunit.client.0.smithi173.stdout: open_pools_parallel: process_2_[35907]: shutting down. 2024-09-17T18:20:06.209 INFO:tasks.workunit.client.0.smithi173.stdout: open_pools_parallel: ******************************* 2024-09-17T18:20:06.209 INFO:tasks.workunit.client.0.smithi173.stdout: open_pools_parallel: process_3_[36653]: starting. 2024-09-17T18:20:06.209 INFO:tasks.workunit.client.0.smithi173.stdout: open_pools_parallel: process_3_[36653]: creating pool ceph_test_rados_open_pools_parallel.smithi173-35714 2024-09-17T18:20:06.209 INFO:tasks.workunit.client.0.smithi173.stdout: open_pools_parallel: process_3_[36653]: created object 0... 2024-09-17T18:20:06.209 INFO:tasks.workunit.client.0.smithi173.stdout: open_pools_parallel: process_3_[36653]: created object 25... 2024-09-17T18:20:06.209 INFO:tasks.workunit.client.0.smithi173.stdout: open_pools_parallel: process_3_[36653]: created object 49... 2024-09-17T18:20:06.209 INFO:tasks.workunit.client.0.smithi173.stdout: open_pools_parallel: process_3_[36653]: finishing. 2024-09-17T18:20:06.209 INFO:tasks.workunit.client.0.smithi173.stdout: open_pools_parallel: process_3_[36653]: shutting down. 2024-09-17T18:20:06.209 INFO:tasks.workunit.client.0.smithi173.stdout: open_pools_parallel: ******************************* 2024-09-17T18:20:06.209 INFO:tasks.workunit.client.0.smithi173.stdout: open_pools_parallel: process_4_[36654]: starting. 2024-09-17T18:20:06.209 INFO:tasks.workunit.client.0.smithi173.stdout: open_pools_parallel: process_4_[36654]: rados_pool_create. 2024-09-17T18:20:06.209 INFO:tasks.workunit.client.0.smithi173.stdout: open_pools_parallel: process_4_[36654]: rados_ioctx_create. 2024-09-17T18:20:06.209 INFO:tasks.workunit.client.0.smithi173.stdout: open_pools_parallel: process_4_[36654]: shutting down. 2024-09-17T18:20:06.210 INFO:tasks.workunit.client.0.smithi173.stdout: open_pools_parallel: ******************************* 2024-09-17T18:20:06.210 INFO:tasks.workunit.client.0.smithi173.stdout: open_pools_parallel: ******************************* 2024-09-17T18:20:06.210 INFO:tasks.workunit.client.0.smithi173.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-09-17T18:20:08.181 INFO:tasks.workunit.client.0.smithi173.stdout: cmd: Running main() from gmock_main.cc 2024-09-17T18:20:08.181 INFO:tasks.workunit.client.0.smithi173.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-09-17T18:20:08.181 INFO:tasks.workunit.client.0.smithi173.stdout: cmd: [----------] Global test environment set-up. 2024-09-17T18:20:08.181 INFO:tasks.workunit.client.0.smithi173.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-09-17T18:20:08.181 INFO:tasks.workunit.client.0.smithi173.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-09-17T18:20:08.181 INFO:tasks.workunit.client.0.smithi173.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (2148 ms) 2024-09-17T18:20:08.182 INFO:tasks.workunit.client.0.smithi173.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-09-17T18:20:08.182 INFO:tasks.workunit.client.0.smithi173.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (2027 ms) 2024-09-17T18:20:08.182 INFO:tasks.workunit.client.0.smithi173.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-09-17T18:20:08.182 INFO:tasks.workunit.client.0.smithi173.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (3001 ms) 2024-09-17T18:20:08.182 INFO:tasks.workunit.client.0.smithi173.stdout: cmd: [----------] 3 tests from NeoRadosCmd (7179 ms total) 2024-09-17T18:20:08.182 INFO:tasks.workunit.client.0.smithi173.stdout: cmd: 2024-09-17T18:20:08.182 INFO:tasks.workunit.client.0.smithi173.stdout: cmd: [----------] Global test environment tear-down 2024-09-17T18:20:08.182 INFO:tasks.workunit.client.0.smithi173.stdout: cmd: [==========] 3 tests from 1 test suite ran. (7180 ms total) 2024-09-17T18:20:08.182 INFO:tasks.workunit.client.0.smithi173.stdout: cmd: [ PASSED ] 3 tests. 2024-09-17T18:20:08.293 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: Running main() from gmock_main.cc 2024-09-17T18:20:08.294 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [==========] Running 40 tests from 2 test suites. 2024-09-17T18:20:08.294 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [----------] Global test environment set-up. 2024-09-17T18:20:08.294 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [----------] 22 tests from LibRadosIoPP 2024-09-17T18:20:08.294 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: seed 35046 2024-09-17T18:20:08.294 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-09-17T18:20:08.294 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-09-17T18:20:08.294 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-09-17T18:20:08.294 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (246 ms) 2024-09-17T18:20:08.294 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-09-17T18:20:08.294 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (20 ms) 2024-09-17T18:20:08.294 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-09-17T18:20:08.294 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (3 ms) 2024-09-17T18:20:08.294 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadExtentArrayOpPP 2024-09-17T18:20:08.294 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadExtentArrayOpPP (627 ms) 2024-09-17T18:20:08.295 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-09-17T18:20:08.295 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (4 ms) 2024-09-17T18:20:08.295 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-09-17T18:20:08.295 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (4 ms) 2024-09-17T18:20:08.295 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-09-17T18:20:08.295 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (9 ms) 2024-09-17T18:20:08.295 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-09-17T18:20:08.295 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (5 ms) 2024-09-17T18:20:08.295 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-09-17T18:20:08.295 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (13 ms) 2024-09-17T18:20:08.295 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-09-17T18:20:08.295 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (5 ms) 2024-09-17T18:20:08.295 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-09-17T18:20:08.295 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (2 ms) 2024-09-17T18:20:08.296 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-09-17T18:20:08.296 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (5 ms) 2024-09-17T18:20:08.296 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-09-17T18:20:08.296 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (4 ms) 2024-09-17T18:20:08.296 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-09-17T18:20:08.296 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (9 ms) 2024-09-17T18:20:08.296 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-09-17T18:20:08.296 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (3 ms) 2024-09-17T18:20:08.296 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-09-17T18:20:08.296 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (13 ms) 2024-09-17T18:20:08.296 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-09-17T18:20:08.296 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (8 ms) 2024-09-17T18:20:08.296 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CrcZeroWrite 2024-09-17T18:20:08.297 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ OK ] LibRadosIoPP.CrcZeroWrite (2 ms) 2024-09-17T18:20:08.297 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-09-17T18:20:08.297 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (3 ms) 2024-09-17T18:20:08.297 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-09-17T18:20:08.297 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (6 ms) 2024-09-17T18:20:08.297 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-09-17T18:20:08.297 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (8 ms) 2024-09-17T18:20:08.297 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [----------] 22 tests from LibRadosIoPP (1000 ms total) 2024-09-17T18:20:08.297 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: 2024-09-17T18:20:08.297 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [----------] 18 tests from LibRadosIoECPP 2024-09-17T18:20:08.297 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-09-17T18:20:08.297 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (1101 ms) 2024-09-17T18:20:08.297 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-09-17T18:20:08.297 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (15 ms) 2024-09-17T18:20:08.298 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-09-17T18:20:08.298 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (5 ms) 2024-09-17T18:20:09.200 INFO:tasks.workunit.client.0.smithi173.stdout: le-LibRadosStatECPP_smithi173-35511-7"} : dispatch 2024-09-17T18:20:09.201 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:04.144270+0000 mon.a [INF] from='client.? 172.21.15.173:0/2416687734' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi173-35030-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi173-35030-16"} : dispatch 2024-09-17T18:20:09.201 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:04.144462+0000 mon.b [INF] from='client.? 172.21.15.173:0/1532755706' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi173-36016-1"} : dispatch 2024-09-17T18:20:09.201 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:04.144671+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi173-35349-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi173-35349-7"} : dispatch 2024-09-17T18:20:09.201 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:04.145735+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi173-35511-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi173-35511-7"} : dispatch 2024-09-17T18:20:09.201 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:04.146398+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi173-36016-1"} : dispatch 2024-09-17T18:20:09.201 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:04.146697+0000 mon.b [INF] from='client.? 172.21.15.173:0/3333708651' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi173-35008-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-17T18:20:09.201 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:04.149599+0000 mon.b [INF] from='client.? 172.21.15.173:0/4076498538' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi173-35650-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-17T18:20:09.201 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:04.155875+0000 mon.a [INF] from='client.? 172.21.15.173:0/2431722596' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi173-35260-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-17T18:20:09.201 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:04.156117+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi173-35008-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-17T18:20:09.201 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:04.156463+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi173-35650-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-17T18:20:09.201 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:04.252404+0000 client.admin [INF] onexx 2024-09-17T18:20:09.201 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:04.259326+0000 mon.a [INF] from='client.? 172.21.15.173:0/3093620120' entity='client.admin' cmd={"prefix":"log", "logtext":["onexx"]} : dispatch 2024-09-17T18:20:09.202 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:04.259615+0000 mon.a [INF] from='client.? 172.21.15.173:0/458342829' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi173-35046-24"} : dispatch 2024-09-17T18:20:09.202 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:04.263018+0000 mon.a [INF] from='client.? 172.21.15.173:0/458342829' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi173-35046-24"} : dispatch 2024-09-17T18:20:09.202 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:04.263756+0000 mon.a [INF] from='client.? 172.21.15.173:0/458342829' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi173-35046-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-17T18:20:09.202 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:07.195049+0000 mon.a [INF] from='client.? 172.21.15.173:0/3093620120' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-09-17T18:20:09.202 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:07.208679+0000 mon.b [INF] from='client.? 172.21.15.173:0/3372932537' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi173-36188-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-17T18:20:09.202 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:07.209490+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi173-36188-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-17T18:20:09.202 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:08.153912+0000 mon.a [INF] from='client.? 172.21.15.173:0/4132430159' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi173-35110-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi173-35110-10"}]': finished 2024-09-17T18:20:09.202 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:08.153995+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi173-35098-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi173-35098-10"}]': finished 2024-09-17T18:20:09.202 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:08.154081+0000 mon.a [INF] from='client.? 172.21.15.173:0/1292664106' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi173-35399-4", "overlaypool": "test-rados-api-smithi173-35399-4-cache"}]': finished 2024-09-17T18:20:09.202 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:08.154139+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi173-35008-3", "field": "max_bytes", "val": "4096"}]': finished 2024-09-17T18:20:09.202 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:08.154218+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi173-35650-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-17T18:20:09.202 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:08.154291+0000 mon.a [INF] from='client.? 172.21.15.173:0/1286856817' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi173-35260-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-17T18:20:09.203 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:08.154354+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi173-36188-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-17T18:20:09.203 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:08.160735+0000 mon.b [INF] from='client.? 172.21.15.173:0/3372932537' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ListObjectsNSsmithi173-36188-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi173-36188-2"} : dispatch 2024-09-17T18:20:09.203 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:08.167692+0000 mon.b [INF] from='client.? 172.21.15.173:0/2574790038' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "WaitForSafe_smithi173-34988-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-17T18:20:09.203 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:08.167973+0000 mon.b [INF] from='client.? 172.21.15.173:0/4178523083' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi173-35511-7"} : dispatch 2024-09-17T18:20:09.203 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:08.168279+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ListObjectsNSsmithi173-36188-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi173-36188-2"} : dispatch 2024-09-17T18:20:09.208 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:08.168901+0000 mon.a [INF] from='client.? 172.21.15.173:0/1292664106' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi173-35399-4-cache", "mode": "writeback"} : dispatch 2024-09-17T18:20:09.208 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:08.169454+0000 mon.b [INF] from='client.? 172.21.15.173:0/3116255395' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi173-35349-7"} : dispatch 2024-09-17T18:20:09.208 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:08.170078+0000 mon.a [INF] from='client.? 172.21.15.173:0/2416687734' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi173-35030-16"} : dispatch 2024-09-17T18:20:09.208 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:08.173866+0000 mon.a [INF] from='client.? 172.21.15.173:0/1518295647' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi173-35451-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-17T18:20:09.209 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:08.184157+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "WaitForSafe_smithi173-34988-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-17T18:20:09.209 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:08.184530+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi173-35511-7"} : dispatch 2024-09-17T18:20:09.209 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:08.184771+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi173-35349-7"} : dispatch 2024-09-17T18:20:09.209 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:08.195085+0000 mon.a [WRN] Health check update: 15 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T18:20:09.209 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:08.406183+0000 mon.a [INF] from='client.? 172.21.15.173:0/458342829' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "LibRadosIoECPP_smithi173-35046-24", "var": "allow_ec_overwrites", "val": "true"} : dispatch 2024-09-17T18:20:09.209 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:09.155498+0000 mon.a [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-17T18:20:09.209 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:09.157225+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi173-36016-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi173-36016-2"}]': finished 2024-09-17T18:20:09.209 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:09.157291+0000 mon.a [INF] from='client.? 172.21.15.173:0/1292664106' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi173-35399-4-cache", "mode": "writeback"}]': finished 2024-09-17T18:20:09.209 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:09.157324+0000 mon.a [INF] from='client.? 172.21.15.173:0/2416687734' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi173-35030-16"}]': finished 2024-09-17T18:20:09.210 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:09.157358+0000 mon.a [INF] from='client.? 172.21.15.173:0/1518295647' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi173-35451-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-17T18:20:09.210 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:09.157389+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi173-34988-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-17T18:20:09.210 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:09.157423+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi173-35511-7"}]': finished 2024-09-17T18:20:10.190 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:09.157457+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi173-35349-7"}]': finish api_io: Running main() from gmock_main.cc 2024-09-17T18:20:10.191 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-09-17T18:20:10.191 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [----------] Global test environment set-up. 2024-09-17T18:20:10.191 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-09-17T18:20:10.191 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-09-17T18:20:10.191 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (208 ms) 2024-09-17T18:20:10.191 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-09-17T18:20:10.191 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-09-17T18:20:10.191 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-09-17T18:20:10.191 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: no timeout :/ 2024-09-17T18:20:10.191 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: no timeout :/ 2024-09-17T18:20:10.191 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: no timeout :/ 2024-09-17T18:20:10.191 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: no timeout :/ 2024-09-17T18:20:10.191 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: no timeout :/ 2024-09-17T18:20:10.191 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (64 ms) 2024-09-17T18:20:10.192 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-09-17T18:20:10.192 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (13 ms) 2024-09-17T18:20:10.192 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-09-17T18:20:10.192 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ OK ] LibRadosIo.Checksum (9 ms) 2024-09-17T18:20:10.192 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-09-17T18:20:10.192 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (18 ms) 2024-09-17T18:20:10.192 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-09-17T18:20:10.192 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (9 ms) 2024-09-17T18:20:10.192 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-09-17T18:20:10.192 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (12 ms) 2024-09-17T18:20:10.192 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-09-17T18:20:10.192 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (6 ms) 2024-09-17T18:20:10.193 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-09-17T18:20:10.193 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ OK ] LibRadosIo.TruncTest (8 ms) 2024-09-17T18:20:10.193 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-09-17T18:20:10.193 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (4 ms) 2024-09-17T18:20:10.193 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-09-17T18:20:10.193 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (12 ms) 2024-09-17T18:20:10.193 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-09-17T18:20:10.193 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ OK ] LibRadosIo.RmXattr (34 ms) 2024-09-17T18:20:10.193 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-09-17T18:20:10.193 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ OK ] LibRadosIo.XattrIter (13 ms) 2024-09-17T18:20:10.193 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [----------] 14 tests from LibRadosIo (410 ms total) 2024-09-17T18:20:10.193 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: 2024-09-17T18:20:10.193 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-09-17T18:20:10.193 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-09-17T18:20:10.193 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (1087 ms) 2024-09-17T18:20:10.194 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-09-17T18:20:10.194 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (216 ms) 2024-09-17T18:20:10.194 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-09-17T18:20:10.194 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (7 ms) 2024-09-17T18:20:10.194 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-09-17T18:20:10.194 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (10 ms) 2024-09-17T18:20:10.194 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-09-17T18:20:10.195 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (11 ms) 2024-09-17T18:20:10.195 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-09-17T18:20:10.195 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (3 ms) 2024-09-17T18:20:10.195 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-09-17T18:20:10.195 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (4 ms) 2024-09-17T18:20:10.195 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-09-17T18:20:10.195 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (4 ms) 2024-09-17T18:20:10.195 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-09-17T18:20:10.195 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (22 ms) 2024-09-17T18:20:10.195 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-09-17T18:20:10.195 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (16 ms) 2024-09-17T18:20:10.196 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [----------] 10 tests from LibRadosIoEC (1380 ms total) 2024-09-17T18:20:10.196 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: 2024-09-17T18:20:10.196 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [----------] Global test environment tear-down 2024-09-17T18:20:10.196 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [==========] 24 tests from 2 test suites ran. (9557 ms total) 2024-09-17T18:20:10.196 INFO:tasks.workunit.client.0.smithi173.stdout: api_io: [ PASSED ] 24 tests. 2024-09-17T18:20:10.211 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-09-17T18:20:10.211 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-09-17T18:20:10.211 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-09-17T18:20:10.211 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-09-17T18:20:10.211 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat_pp: seed 35511 2024-09-17T18:20:10.211 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-09-17T18:20:10.211 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (252 ms) 2024-09-17T18:20:10.211 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-09-17T18:20:10.211 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (13 ms) 2024-09-17T18:20:10.211 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-09-17T18:20:10.211 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (0 ms) 2024-09-17T18:20:10.211 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-09-17T18:20:10.212 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (5 ms) 2024-09-17T18:20:10.212 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-09-17T18:20:10.212 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (31 ms) 2024-09-17T18:20:10.212 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (301 ms total) 2024-09-17T18:20:10.212 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat_pp: 2024-09-17T18:20:10.212 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-09-17T18:20:10.212 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-09-17T18:20:10.212 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (1122 ms) 2024-09-17T18:20:10.212 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-09-17T18:20:10.212 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (0 ms) 2024-09-17T18:20:10.212 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-09-17T18:20:10.212 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (3 ms) 2024-09-17T18:20:10.212 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-09-17T18:20:10.212 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (16 ms) 2024-09-17T18:20:10.213 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (1141 ms total) 2024-09-17T18:20:10.213 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat_pp: 2024-09-17T18:20:10.213 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-09-17T18:20:10.213 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (9376 ms total) 2024-09-17T18:20:10.213 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-09-17T18:20:10.213 INFO:tasks.workunit.client.0.smithi173.stdout: watch_notify: Running main() from gmock_main.cc 2024-09-17T18:20:10.213 INFO:tasks.workunit.client.0.smithi173.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-09-17T18:20:10.213 INFO:tasks.workunit.client.0.smithi173.stdout: watch_notify: [----------] Global test environment set-up. 2024-09-17T18:20:10.213 INFO:tasks.workunit.client.0.smithi173.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-09-17T18:20:10.213 INFO:tasks.workunit.client.0.smithi173.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-09-17T18:20:10.214 INFO:tasks.workunit.client.0.smithi173.stdout: watch_notify: handle_notify cookie 94693729641280 notify_id 81604378624 notifier_gid 4648 2024-09-17T18:20:10.214 INFO:tasks.workunit.client.0.smithi173.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (2074 ms) 2024-09-17T18:20:10.214 INFO:tasks.workunit.client.0.smithi173.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-09-17T18:20:10.214 INFO:tasks.workunit.client.0.smithi173.stdout: watch_notify: Trying... 2024-09-17T18:20:10.214 INFO:tasks.workunit.client.0.smithi173.stdout: watch_notify: handle_notify cookie 94693741760752 notify_id 94489280512 notifier_gid 4714 2024-09-17T18:20:10.214 INFO:tasks.workunit.client.0.smithi173.stdout: watch_notify: Waiting for 3.000000000s 2024-09-17T18:20:10.214 INFO:tasks.workunit.client.0.smithi173.stdout: watch_notify: Timed out. 2024-09-17T18:20:10.214 INFO:tasks.workunit.client.0.smithi173.stdout: watch_notify: Flushing... 2024-09-17T18:20:10.214 INFO:tasks.workunit.client.0.smithi173.stdout: watch_notify: Flushed... 2024-09-17T18:20:10.214 INFO:tasks.workunit.client.0.smithi173.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (7035 ms) 2024-09-17T18:20:10.214 INFO:tasks.workunit.client.0.smithi173.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (9109 ms total) 2024-09-17T18:20:10.214 INFO:tasks.workunit.client.0.smithi173.stdout: watch_notify: 2024-09-17T18:20:10.214 INFO:tasks.workunit.client.0.smithi173.stdout: watch_notify: [----------] Global test environment tear-down 2024-09-17T18:20:10.214 INFO:tasks.workunit.client.0.smithi173.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (9109 ms total) 2024-09-17T18:20:10.215 INFO:tasks.workunit.client.0.smithi173.stdout: watch_notify: [ PASSED ] 2 tests. 2024-09-17T18:20:10.220 INFO:tasks.workunit.client.0.smithi173.stdout: list: Running main() from gmock_main.cc 2024-09-17T18:20:10.220 INFO:tasks.workunit.client.0.smithi173.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-09-17T18:20:10.220 INFO:tasks.workunit.client.0.smithi173.stdout: list: [----------] Global test environment set-up. 2024-09-17T18:20:10.220 INFO:tasks.workunit.client.0.smithi173.stdout: list: [----------] 3 tests from NeoradosList 2024-09-17T18:20:10.220 INFO:tasks.workunit.client.0.smithi173.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-09-17T18:20:10.220 INFO:tasks.workunit.client.0.smithi173.stdout: list: [ OK ] NeoradosList.ListObjects (2099 ms) 2024-09-17T18:20:10.220 INFO:tasks.workunit.client.0.smithi173.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-09-17T18:20:10.220 INFO:tasks.workunit.client.0.smithi173.stdout: list: [ OK ] NeoradosList.ListObjectsNS (3033 ms) 2024-09-17T18:20:10.220 INFO:tasks.workunit.client.0.smithi173.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-09-17T18:20:10.220 INFO:tasks.workunit.client.0.smithi173.stdout: list: [ OK ] NeoradosList.ListObjectsMany (4042 ms) 2024-09-17T18:20:10.220 INFO:tasks.workunit.client.0.smithi173.stdout: list: [----------] 3 tests from NeoradosList (9174 ms total) 2024-09-17T18:20:10.220 INFO:tasks.workunit.client.0.smithi173.stdout: list: 2024-09-17T18:20:10.220 INFO:tasks.workunit.client.0.smithi173.stdout: list: [----------] Global test environment tear-down 2024-09-17T18:20:10.221 INFO:tasks.workunit.client.0.smithi173.stdout: list: [==========] 3 tests from 1 test suite ran. (9174 ms total) 2024-09-17T18:20:10.221 INFO:tasks.workunit.client.0.smithi173.stdout: list: [ PASSED ] 3 tests. 2024-09-17T18:20:10.228 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat: Running main() from gmock_main.cc 2024-09-17T18:20:10.229 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-09-17T18:20:10.229 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat: [----------] Global test environment set-up. 2024-09-17T18:20:10.229 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-09-17T18:20:10.229 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-09-17T18:20:10.229 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat: [ OK ] LibRadosStat.Stat (210 ms) 2024-09-17T18:20:10.229 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-09-17T18:20:10.229 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (29 ms) 2024-09-17T18:20:10.229 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-09-17T18:20:10.229 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat: [ OK ] LibRadosStat.StatNS (31 ms) 2024-09-17T18:20:10.229 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-09-17T18:20:10.229 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (1 ms) 2024-09-17T18:20:10.229 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-09-17T18:20:10.229 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (2 ms) 2024-09-17T18:20:10.230 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat: [----------] 5 tests from LibRadosStat (273 ms total) 2024-09-17T18:20:10.230 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat: 2024-09-17T18:20:10.230 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-09-17T18:20:10.230 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-09-17T18:20:10.230 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (1111 ms) 2024-09-17T18:20:10.230 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-09-17T18:20:10.230 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (30 ms) 2024-09-17T18:20:10.230 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-09-17T18:20:10.230 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (0 ms) 2024-09-17T18:20:10.230 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-09-17T18:20:10.230 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (2 ms) 2024-09-17T18:20:10.230 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (1143 ms total) 2024-09-17T18:20:10.230 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat: 2024-09-17T18:20:10.230 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat: [----------] Global test environment tear-down 2024-09-17T18:20:10.230 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (9484 ms total) 2024-09-17T18:20:10.231 INFO:tasks.workunit.client.0.smithi173.stdout: api_stat: [ PASSED ] 9 tests. 2024-09-17T18:20:10.281 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2024-09-17T18:20:10.281 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: [----------] Global test environment set-up. 2024-09-17T18:20:10.281 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: [----------] 4 tests from LibRadosService 2024-09-17T18:20:10.281 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-09-17T18:20:10.281 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5115 ms) 2024-09-17T18:20:10.281 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-09-17T18:20:10.281 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: [ OK ] LibRadosService.RegisterLate (22 ms) 2024-09-17T18:20:10.281 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-09-17T18:20:10.282 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: cluster: 2024-09-17T18:20:10.282 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: id: a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:20:10.282 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: health: HEALTH_WARN 2024-09-17T18:20:10.282 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: 11 pool(s) do not have an application enabled 2024-09-17T18:20:10.282 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: 2024-09-17T18:20:10.282 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: services: 2024-09-17T18:20:10.282 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: mon: 3 daemons, quorum a,b (age 42s), out of quorum: c 2024-09-17T18:20:10.282 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: mgr: x(active, since 74s) 2024-09-17T18:20:10.282 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: osd: 6 osds: 6 up (since 68s), 6 in (since 76s) 2024-09-17T18:20:10.282 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-09-17T18:20:10.282 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: 2024-09-17T18:20:10.282 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: data: 2024-09-17T18:20:10.282 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: pools: 24 pools, 641 pgs 2024-09-17T18:20:10.282 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: objects: 4 objects, 577 KiB 2024-09-17T18:20:10.282 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: usage: 159 MiB used, 540 GiB / 540 GiB avail 2024-09-17T18:20:10.283 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: pgs: 60.842% pgs unknown 2024-09-17T18:20:10.283 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: 37.754% pgs not active 2024-09-17T18:20:10.283 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: 390 unknown 2024-09-17T18:20:10.283 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: 242 creating+peering 2024-09-17T18:20:10.283 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: 9 active+clean 2024-09-17T18:20:10.283 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: 2024-09-17T18:20:10.283 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: 2024-09-17T18:20:10.283 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: cluster: 2024-09-17T18:20:10.283 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: id: a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:20:10.283 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: health: HEALTH_WARN 2024-09-17T18:20:10.283 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: 14 pool(s) do not have an application enabled 2024-09-17T18:20:10.283 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: 2024-09-17T18:20:10.283 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: services: 2024-09-17T18:20:10.283 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: mon: 3 daemons, quorum a,b (age 44s), out of quorum: c 2024-09-17T18:20:10.283 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: mgr: x(active, since 76s) 2024-09-17T18:20:10.284 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: osd: 6 osds: 6 up (since 70s), 6 in (since 78s) 2024-09-17T18:20:10.284 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-09-17T18:20:10.284 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: 2024-09-17T18:20:10.284 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: data: 2024-09-17T18:20:10.284 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: pools: 26 pools, 777 pgs 2024-09-17T18:20:10.284 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: objects: 20 objects, 578 KiB 2024-09-17T18:20:10.284 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: usage: 164 MiB used, 540 GiB / 540 GiB avail 2024-09-17T18:20:10.284 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: pgs: 63.449% pgs unknown 2024-09-17T18:20:10.284 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: 23.810% pgs not active 2024-09-17T18:20:10.284 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: 493 unknown 2024-09-17T18:20:10.284 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: 185 creating+peering 2024-09-17T18:20:10.284 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: 99 active+clean 2024-09-17T18:20:10.284 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: 2024-09-17T18:20:10.284 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: io: 2024-09-17T18:20:10.285 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: client: 255 B/s rd, 18 KiB/s wr, 65 op/s rd, 95 op/s wr 2024-09-17T18:20:10.285 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: 2024-09-17T18:20:10.285 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: 2024-09-17T18:20:10.285 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: cluster: 2024-09-17T18:20:10.285 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: id: a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:20:10.285 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: health: HEALTH_WARN 2024-09-17T18:20:10.285 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: 1 cache pools are missing hit_sets 2024-09-17T18:20:10.285 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: 14 pool(s) do not have an application enabled 2024-09-17T18:20:10.285 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: 2024-09-17T18:20:10.285 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: services: 2024-09-17T18:20:10.285 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: mon: 3 daemons, quorum a,b (age 46s), out of quorum: c 2024-09-17T18:20:10.285 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: mgr: x(active, since 78s) 2024-09-17T18:20:10.285 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: osd: 6 osds: 6 up (since 72s), 6 in (since 81s) 2024-09-17T18:20:10.285 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-09-17T18:20:10.285 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-09-17T18:20:10.286 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: 2024-09-17T18:20:10.286 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: data: 2024-09-17T18:20:10.286 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: pools: 28 pools, 745 pgs 2024-09-17T18:20:13.069 INFO:tasks.workunit.client.0.smithi173.stdout: ap ec_io: Running main() from gmock_main.cc 2024-09-17T18:20:13.069 INFO:tasks.workunit.client.0.smithi173.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-09-17T18:20:13.070 INFO:tasks.workunit.client.0.smithi173.stdout: ec_io: [----------] Global test environment set-up. 2024-09-17T18:20:13.070 INFO:tasks.workunit.client.0.smithi173.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-09-17T18:20:13.070 INFO:tasks.workunit.client.0.smithi173.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-09-17T18:20:13.070 INFO:tasks.workunit.client.0.smithi173.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (5157 ms) 2024-09-17T18:20:13.070 INFO:tasks.workunit.client.0.smithi173.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-09-17T18:20:13.070 INFO:tasks.workunit.client.0.smithi173.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (6906 ms) 2024-09-17T18:20:13.070 INFO:tasks.workunit.client.0.smithi173.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (12063 ms total) 2024-09-17T18:20:13.070 INFO:tasks.workunit.client.0.smithi173.stdout: ec_io: 2024-09-17T18:20:13.070 INFO:tasks.workunit.client.0.smithi173.stdout: ec_io: [----------] Global test environment tear-down 2024-09-17T18:20:13.070 INFO:tasks.workunit.client.0.smithi173.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (12064 ms total) 2024-09-17T18:20:13.070 INFO:tasks.workunit.client.0.smithi173.stdout: ec_io: [ PASSED ] 2 tests. 2024-09-17T18:20:13.221 INFO:tasks.workunit.client.0.smithi173.stdout:ed 2024-09-17T18:20:13.221 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:09.157491+0000 mon.a [INF] from='client.? 172.21.15.173:0/458342829' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "LibRadosIoECPP_smithi173-35046-24", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-09-17T18:20:13.221 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:09.162257+0000 mon.b [INF] from='client.? 172.21.15.173:0/4178523083' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi173-35511-7"} : dispatch 2024-09-17T18:20:13.221 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:09.162558+0000 mon.b [INF] from='client.? 172.21.15.173:0/3116255395' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi173-35349-7"} : dispatch 2024-09-17T18:20:13.221 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:09.169852+0000 mon.a [INF] from='client.? 172.21.15.173:0/2416687734' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi173-35030-16"} : dispatch 2024-09-17T18:20:13.221 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:09.171623+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi173-35511-7"} : dispatch 2024-09-17T18:20:13.221 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:09.172167+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi173-35349-7"} : dispatch 2024-09-17T18:20:13.221 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:09.196982+0000 client.admin [INF] threexx 2024-09-17T18:20:13.221 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: got: 2024-09-17T18:20:09.197332+0000 mon.a [INF] from='client.? 172.21.15.173:0/3093620120' entity='client.admin' cmd={"prefix":"log", "logtext":["threexx"]} : dispatch 2024-09-17T18:20:13.221 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (9058 ms) 2024-09-17T18:20:13.221 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (12408 ms total) 2024-09-17T18:20:13.221 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: 2024-09-17T18:20:13.222 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: [----------] Global test environment tear-down 2024-09-17T18:20:13.222 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (12408 ms total) 2024-09-17T18:20:13.222 INFO:tasks.workunit.client.0.smithi173.stdout: api_cmd: [ PASSED ] 4 tests. 2024-09-17T18:20:13.230 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-09-17T18:20:13.230 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-09-17T18:20:13.230 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-09-17T18:20:13.230 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-09-17T18:20:13.230 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-09-17T18:20:13.230 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: notify 2024-09-17T18:20:13.230 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (1125 ms) 2024-09-17T18:20:13.230 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-09-17T18:20:13.230 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (21 ms) 2024-09-17T18:20:13.230 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (1146 ms total) 2024-09-17T18:20:13.231 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: 2024-09-17T18:20:13.231 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-09-17T18:20:13.231 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-09-17T18:20:13.231 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: notify 2024-09-17T18:20:13.231 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (86 ms) 2024-09-17T18:20:13.231 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-09-17T18:20:13.231 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: notify 2024-09-17T18:20:13.231 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (3872 ms) 2024-09-17T18:20:13.231 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-09-17T18:20:13.231 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (4 ms) 2024-09-17T18:20:13.231 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-09-17T18:20:13.231 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (9 ms) 2024-09-17T18:20:13.231 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-09-17T18:20:13.231 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: handle_notify cookie 94865721801200 notify_id 128849018884 notifier_gid 4974 2024-09-17T18:20:13.232 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (9 ms) 2024-09-17T18:20:13.232 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-09-17T18:20:13.232 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: handle_notify cookie 94865721796960 notify_id 128849018885 notifier_gid 4974 2024-09-17T18:20:13.232 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (14 ms) 2024-09-17T18:20:13.232 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-09-17T18:20:13.232 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: handle_notify cookie 94865721801200 notify_id 128849018886 notifier_gid 4974 2024-09-17T18:20:13.232 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (10 ms) 2024-09-17T18:20:13.232 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-09-17T18:20:13.232 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: handle_notify cookie 94865721801200 notify_id 128849018887 notifier_gid 4974 2024-09-17T18:20:13.232 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (15 ms) 2024-09-17T18:20:13.232 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-09-17T18:20:13.232 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: handle_notify cookie 94865721801200 notify_id 128849018888 notifier_gid 4974 2024-09-17T18:20:13.232 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: handle_notify cookie 94865721801200 notify_id 128849018889 notifier_gid 4974 2024-09-17T18:20:13.233 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (10 ms) 2024-09-17T18:20:13.233 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-09-17T18:20:13.233 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: handle_notify cookie 94865721801200 notify_id 128849018882 notifier_gid 4974 2024-09-17T18:20:13.233 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (15 ms) 2024-09-17T18:20:13.233 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-09-17T18:20:13.233 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: trying... 2024-09-17T18:20:14.052 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: handle_notify cookie 94865721801200 notify_id 128849018890 notif pool: Running main() from gmock_main.cc 2024-09-17T18:20:14.052 INFO:tasks.workunit.client.0.smithi173.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-09-17T18:20:14.052 INFO:tasks.workunit.client.0.smithi173.stdout: pool: [----------] Global test environment set-up. 2024-09-17T18:20:14.052 INFO:tasks.workunit.client.0.smithi173.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-09-17T18:20:14.052 INFO:tasks.workunit.client.0.smithi173.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-09-17T18:20:14.052 INFO:tasks.workunit.client.0.smithi173.stdout: pool: [ OK ] NeoRadosPools.PoolList (1090 ms) 2024-09-17T18:20:14.052 INFO:tasks.workunit.client.0.smithi173.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-09-17T18:20:14.053 INFO:tasks.workunit.client.0.smithi173.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (2026 ms) 2024-09-17T18:20:14.053 INFO:tasks.workunit.client.0.smithi173.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-09-17T18:20:14.053 INFO:tasks.workunit.client.0.smithi173.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (1995 ms) 2024-09-17T18:20:14.053 INFO:tasks.workunit.client.0.smithi173.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-09-17T18:20:14.053 INFO:tasks.workunit.client.0.smithi173.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (4017 ms) 2024-09-17T18:20:14.053 INFO:tasks.workunit.client.0.smithi173.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-09-17T18:20:14.053 INFO:tasks.workunit.client.0.smithi173.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (1861 ms) 2024-09-17T18:20:14.053 INFO:tasks.workunit.client.0.smithi173.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-09-17T18:20:14.053 INFO:tasks.workunit.client.0.smithi173.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (2000 ms) 2024-09-17T18:20:14.053 INFO:tasks.workunit.client.0.smithi173.stdout: pool: [----------] 6 tests from NeoRadosPools (12989 ms total) 2024-09-17T18:20:14.053 INFO:tasks.workunit.client.0.smithi173.stdout: pool: 2024-09-17T18:20:14.053 INFO:tasks.workunit.client.0.smithi173.stdout: pool: [----------] Global test environment tear-down 2024-09-17T18:20:14.053 INFO:tasks.workunit.client.0.smithi173.stdout: pool: [==========] 6 tests from 1 test suite ran. (12989 ms total) 2024-09-17T18:20:14.053 INFO:tasks.workunit.client.0.smithi173.stdout: pool: [ PASSED ] 6 tests. 2024-09-17T18:20:14.070 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: Running main() from gmock_main.cc 2024-09-17T18:20:14.070 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-09-17T18:20:14.070 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: [----------] Global test environment set-up. 2024-09-17T18:20:14.070 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-09-17T18:20:14.070 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-09-17T18:20:14.070 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (12 ms) 2024-09-17T18:20:14.071 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-09-17T18:20:14.071 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: [ OK ] LibRadosLock.LockShared (7 ms) 2024-09-17T18:20:14.071 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-09-17T18:20:14.071 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1011 ms) 2024-09-17T18:20:14.071 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-09-17T18:20:14.071 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1007 ms) 2024-09-17T18:20:14.071 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-09-17T18:20:14.071 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (8 ms) 2024-09-17T18:20:14.071 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-09-17T18:20:14.071 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: [ OK ] LibRadosLock.Unlock (16 ms) 2024-09-17T18:20:14.071 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-09-17T18:20:14.071 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (11 ms) 2024-09-17T18:20:14.071 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-09-17T18:20:14.071 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (4 ms) 2024-09-17T18:20:14.072 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: [----------] 8 tests from LibRadosLock (2076 ms total) 2024-09-17T18:20:14.072 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: 2024-09-17T18:20:14.072 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-09-17T18:20:14.072 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-09-17T18:20:14.072 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (1098 ms) 2024-09-17T18:20:14.072 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-09-17T18:20:14.072 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (12 ms) 2024-09-17T18:20:14.072 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-09-17T18:20:14.072 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1092 ms) 2024-09-17T18:20:14.072 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-09-17T18:20:14.072 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1008 ms) 2024-09-17T18:20:14.072 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-09-17T18:20:14.072 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (8 ms) 2024-09-17T18:20:14.072 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-09-17T18:20:14.073 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (12 ms) 2024-09-17T18:20:14.073 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-09-17T18:20:14.073 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (14 ms) 2024-09-17T18:20:14.073 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-09-17T18:20:14.073 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (4 ms) 2024-09-17T18:20:14.073 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (3249 ms total) 2024-09-17T18:20:14.073 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: 2024-09-17T18:20:14.073 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: [----------] Global test environment tear-down 2024-09-17T18:20:14.073 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (13400 ms total) 2024-09-17T18:20:14.073 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock: [ PASSED ] 16 tests. 2024-09-17T18:20:14.077 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-09-17T18:20:14.077 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-09-17T18:20:14.077 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-09-17T18:20:14.077 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-09-17T18:20:14.077 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: seed 35110 2024-09-17T18:20:14.077 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-09-17T18:20:14.077 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (225 ms) 2024-09-17T18:20:14.077 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-09-17T18:20:14.077 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (15 ms) 2024-09-17T18:20:14.077 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-09-17T18:20:14.077 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1068 ms) 2024-09-17T18:20:14.078 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-09-17T18:20:14.078 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1016 ms) 2024-09-17T18:20:14.078 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-09-17T18:20:14.078 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (8 ms) 2024-09-17T18:20:14.078 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-09-17T18:20:14.078 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (10 ms) 2024-09-17T18:20:14.078 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-09-17T18:20:14.078 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (4 ms) 2024-09-17T18:20:14.078 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-09-17T18:20:14.078 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (4 ms) 2024-09-17T18:20:14.078 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (2350 ms total) 2024-09-17T18:20:14.078 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: 2024-09-17T18:20:14.078 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-09-17T18:20:14.079 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-09-17T18:20:14.079 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (1073 ms) 2024-09-17T18:20:14.079 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-09-17T18:20:14.079 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (22 ms) 2024-09-17T18:20:14.079 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-09-17T18:20:14.079 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1101 ms) 2024-09-17T18:20:14.079 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-09-17T18:20:14.079 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1005 ms) 2024-09-17T18:20:14.079 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-09-17T18:20:14.079 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (10 ms) 2024-09-17T18:20:14.079 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-09-17T18:20:14.079 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (14 ms) 2024-09-17T18:20:14.079 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-09-17T18:20:14.079 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (12 ms) 2024-09-17T18:20:14.080 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-09-17T18:20:14.080 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (4 ms) 2024-09-17T18:20:14.080 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (3241 ms total) 2024-09-17T18:20:14.080 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: 2024-09-17T18:20:14.080 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-09-17T18:20:14.080 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (13395 ms total) 2024-09-17T18:20:14.080 INFO:tasks.workunit.client.0.smithi173.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-09-17T18:20:21.083 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-09-17T18:20:21.083 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (9 ms) 2024-09-17T18:20:21.083 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-09-17T18:20:21.083 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (9 ms) 2024-09-17T18:20:21.083 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-09-17T18:20:21.083 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (15 ms) 2024-09-17T18:20:21.083 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-09-17T18:20:21.083 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (8 ms) 2024-09-17T18:20:21.083 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-09-17T18:20:21.084 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (8 ms) 2024-09-17T18:20:21.084 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-09-17T18:20:21.084 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (12 ms) 2024-09-17T18:20:21.084 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-09-17T18:20:21.084 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (7 ms) 2024-09-17T18:20:21.084 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-09-17T18:20:21.084 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (8 ms) 2024-09-17T18:20:21.084 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-09-17T18:20:21.084 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (11 ms) 2024-09-17T18:20:21.084 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-09-17T18:20:21.084 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (26 ms) 2024-09-17T18:20:21.084 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CrcZeroWrite 2024-09-17T18:20:21.084 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CrcZeroWrite (8672 ms) 2024-09-17T18:20:21.084 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-09-17T18:20:21.085 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (1074 ms) 2024-09-17T18:20:21.085 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-09-17T18:20:21.085 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (18 ms) 2024-09-17T18:20:21.085 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-09-17T18:20:21.085 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (4 ms) 2024-09-17T18:20:21.085 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-09-17T18:20:21.085 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (222 ms) 2024-09-17T18:20:21.085 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [----------] 18 tests from LibRadosIoECPP (11224 ms total) 2024-09-17T18:20:21.085 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: 2024-09-17T18:20:21.085 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [----------] Global test environment tear-down 2024-09-17T18:20:21.085 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [==========] 40 tests from 2 test suites ran. (20441 ms total) 2024-09-17T18:20:21.085 INFO:tasks.workunit.client.0.smithi173.stdout: api_io_pp: [ PASSED ] 40 tests. 2024-09-17T18:20:21.086 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: Running main() from gmock_main.cc 2024-09-17T18:20:21.086 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-09-17T18:20:21.086 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-09-17T18:20:21.086 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-09-17T18:20:21.086 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-09-17T18:20:21.086 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: watch_notify_test_cb 2024-09-17T18:20:21.086 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (227 ms) 2024-09-17T18:20:21.086 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-09-17T18:20:21.086 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-09-17T18:20:21.087 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94381731007136 err -107 2024-09-17T18:20:21.087 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (1021 ms) 2024-09-17T18:20:21.087 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-09-17T18:20:21.087 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-09-17T18:20:21.087 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94381731019040 err -107 2024-09-17T18:20:21.087 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (1009 ms) 2024-09-17T18:20:21.087 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-09-17T18:20:21.087 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: watch_notify2_test_cb from 4470 notify_id 90194313217 cookie 94381731034864 2024-09-17T18:20:21.087 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (9 ms) 2024-09-17T18:20:21.087 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-09-17T18:20:21.087 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: watch_notify2_test_cb from 4470 notify_id 90194313218 cookie 94381731075952 2024-09-17T18:20:21.087 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (6 ms) 2024-09-17T18:20:21.087 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-09-17T18:20:21.087 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: watch_notify2_test_cb from 4470 notify_id 90194313217 cookie 94381731060768 2024-09-17T18:20:21.088 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (11 ms) 2024-09-17T18:20:21.088 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-09-17T18:20:21.088 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: watch_notify2_test_cb from 4470 notify_id 90194313217 cookie 94381731054944 2024-09-17T18:20:21.088 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: watch_notify2_test_cb from 4470 notify_id 90194313217 cookie 94381731060768 2024-09-17T18:20:21.088 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (21 ms) 2024-09-17T18:20:21.088 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-09-17T18:20:21.088 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: watch_notify2_test_cb from 4470 notify_id 90194313218 cookie 94381731056144 2024-09-17T18:20:21.088 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: watch_notify2_test_cb from 4470 notify_id 94489280515 cookie 94381731056144 2024-09-17T18:20:21.088 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3006 ms) 2024-09-17T18:20:21.088 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-09-17T18:20:21.088 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-09-17T18:20:21.088 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94381731056144 err -107 2024-09-17T18:20:21.088 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: watch_notify2_test_cb from 4470 notify_id 124554051585 cookie 94381731053888 2024-09-17T18:20:21.088 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5016 ms) 2024-09-17T18:20:21.089 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-09-17T18:20:21.089 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-09-17T18:20:21.089 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94381731053888 err -107 2024-09-17T18:20:21.089 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1005 ms) 2024-09-17T18:20:21.089 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (11331 ms total) 2024-09-17T18:20:21.089 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: 2024-09-17T18:20:21.089 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-09-17T18:20:21.089 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-09-17T18:20:21.089 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: watch_notify_test_cb 2024-09-17T18:20:21.089 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1089 ms) 2024-09-17T18:20:21.089 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1089 ms total) 2024-09-17T18:20:21.090 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: 2024-09-17T18:20:21.090 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-09-17T18:20:21.090 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (20283 ms total) 2024-09-17T18:20:21.090 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-09-17T18:20:22.091 INFO:tasks.workunit.client.0.smithi173.stdout: ec_list: Running main() from gmock_main.cc 2024-09-17T18:20:22.091 INFO:tasks.workunit.client.0.smithi173.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-09-17T18:20:22.091 INFO:tasks.workunit.client.0.smithi173.stdout: ec_list: [----------] Global test environment set-up. 2024-09-17T18:20:22.091 INFO:tasks.workunit.client.0.smithi173.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-09-17T18:20:22.091 INFO:tasks.workunit.client.0.smithi173.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-09-17T18:20:22.091 INFO:tasks.workunit.client.0.smithi173.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (6114 ms) 2024-09-17T18:20:22.091 INFO:tasks.workunit.client.0.smithi173.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-09-17T18:20:22.092 INFO:tasks.workunit.client.0.smithi173.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (6907 ms) 2024-09-17T18:20:22.092 INFO:tasks.workunit.client.0.smithi173.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-09-17T18:20:22.092 INFO:tasks.workunit.client.0.smithi173.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (8014 ms) 2024-09-17T18:20:22.092 INFO:tasks.workunit.client.0.smithi173.stdout: ec_list: [----------] 3 tests from NeoradosECList (21035 ms total) 2024-09-17T18:20:22.092 INFO:tasks.workunit.client.0.smithi173.stdout: ec_list: 2024-09-17T18:20:22.092 INFO:tasks.workunit.client.0.smithi173.stdout: ec_list: [----------] Global test environment tear-down 2024-09-17T18:20:22.092 INFO:tasks.workunit.client.0.smithi173.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (21035 ms total) 2024-09-17T18:20:22.092 INFO:tasks.workunit.client.0.smithi173.stdout: ec_list: [ PASSED ] 3 tests. 2024-09-17T18:20:23.201 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-09-17T18:20:23.201 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-09-17T18:20:23.202 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-09-17T18:20:23.202 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-09-17T18:20:23.202 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-09-17T18:20:23.202 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-09-17T18:20:23.202 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-09-17T18:20:23.202 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_write_operations: 2024-09-17T18:20:23.202 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-09-17T18:20:23.202 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-09-17T18:20:23.202 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (2307 ms) 2024-09-17T18:20:23.202 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-09-17T18:20:23.202 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (2992 ms) 2024-09-17T18:20:23.202 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-09-17T18:20:23.202 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (3011 ms) 2024-09-17T18:20:23.203 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-09-17T18:20:23.203 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (2880 ms) 2024-09-17T18:20:23.203 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-09-17T18:20:23.203 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (3009 ms) 2024-09-17T18:20:23.203 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-09-17T18:20:23.203 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (3008 ms) 2024-09-17T18:20:23.203 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-09-17T18:20:23.203 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (5028 ms) 2024-09-17T18:20:23.203 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (22235 ms total) 2024-09-17T18:20:23.203 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_write_operations: 2024-09-17T18:20:23.203 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-09-17T18:20:23.203 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (22235 ms total) 2024-09-17T18:20:23.203 INFO:tasks.workunit.client.0.smithi173.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-09-17T18:20:24.119 INFO:tasks.workunit.client.0.smithi173.stdout: write_operations: Running main() from gmock_main.cc 2024-09-17T18:20:24.119 INFO:tasks.workunit.client.0.smithi173.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-09-17T18:20:24.119 INFO:tasks.workunit.client.0.smithi173.stdout: write_operations: [----------] Global test environment set-up. 2024-09-17T18:20:24.119 INFO:tasks.workunit.client.0.smithi173.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-09-17T18:20:24.119 INFO:tasks.workunit.client.0.smithi173.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-09-17T18:20:24.119 INFO:tasks.workunit.client.0.smithi173.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (3078 ms) 2024-09-17T18:20:24.119 INFO:tasks.workunit.client.0.smithi173.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-09-17T18:20:24.119 INFO:tasks.workunit.client.0.smithi173.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (3016 ms) 2024-09-17T18:20:24.119 INFO:tasks.workunit.client.0.smithi173.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-09-17T18:20:24.119 INFO:tasks.workunit.client.0.smithi173.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (3005 ms) 2024-09-17T18:20:24.119 INFO:tasks.workunit.client.0.smithi173.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-09-17T18:20:24.119 INFO:tasks.workunit.client.0.smithi173.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (2876 ms) 2024-09-17T18:20:24.120 INFO:tasks.workunit.client.0.smithi173.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-09-17T18:20:24.120 INFO:tasks.workunit.client.0.smithi173.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (3019 ms) 2024-09-17T18:20:24.120 INFO:tasks.workunit.client.0.smithi173.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-09-17T18:20:24.120 INFO:tasks.workunit.client.0.smithi173.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (2999 ms) 2024-09-17T18:20:24.120 INFO:tasks.workunit.client.0.smithi173.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-09-17T18:20:24.120 INFO:tasks.workunit.client.0.smithi173.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (5022 ms) 2024-09-17T18:20:24.120 INFO:tasks.workunit.client.0.smithi173.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (23015 ms total) 2024-09-17T18:20:24.120 INFO:tasks.workunit.client.0.smithi173.stdout: write_operations: 2024-09-17T18:20:24.120 INFO:tasks.workunit.client.0.smithi173.stdout: write_operations: [----------] Global test environment tear-down 2024-09-17T18:20:24.120 INFO:tasks.workunit.client.0.smithi173.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (23015 ms total) 2024-09-17T18:20:24.120 INFO:tasks.workunit.client.0.smithi173.stdout: write_operations: [ PASSED ] 7 tests. 2024-09-17T18:20:29.080 INFO:tasks.workunit.client.0.smithi173.stdout:ier_gid 4974 2024-09-17T18:20:29.080 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: timed out 2024-09-17T18:20:29.080 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: flushing 2024-09-17T18:20:29.080 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: flushed 2024-09-17T18:20:29.081 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3010 ms) 2024-09-17T18:20:29.081 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-09-17T18:20:29.081 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: trying... 2024-09-17T18:20:29.081 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: handle_notify cookie 94865721801200 notify_id 141733920779 notifier_gid 4974 2024-09-17T18:20:29.081 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: timed out 2024-09-17T18:20:29.081 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: flushing 2024-09-17T18:20:29.081 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: flushed 2024-09-17T18:20:29.081 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3010 ms) 2024-09-17T18:20:29.081 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-09-17T18:20:29.081 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: List watches 2024-09-17T18:20:29.081 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: notify2 2024-09-17T18:20:29.081 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: handle_notify cookie 94865721801200 notify_id 154618822668 notifier_gid 4974 2024-09-17T18:20:29.082 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: notify2 done 2024-09-17T18:20:29.082 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: watch_check 2024-09-17T18:20:29.082 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: unwatch2 2024-09-17T18:20:29.082 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: flushing 2024-09-17T18:20:29.082 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: done 2024-09-17T18:20:29.082 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3018 ms) 2024-09-17T18:20:29.082 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-09-17T18:20:29.082 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: List watches 2024-09-17T18:20:29.082 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: notify2 2024-09-17T18:20:29.082 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: handle_notify cookie 94865721801200 notify_id 167503724547 notifier_gid 4974 2024-09-17T18:20:29.082 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: notify2 done 2024-09-17T18:20:29.082 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: watch_check 2024-09-17T18:20:29.083 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: unwatch2 2024-09-17T18:20:29.083 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: flushing 2024-09-17T18:20:29.083 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: done 2024-09-17T18:20:29.083 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3012 ms) 2024-09-17T18:20:29.083 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (16094 ms total) 2024-09-17T18:20:29.083 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: 2024-09-17T18:20:29.083 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-09-17T18:20:29.083 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (28277 ms total) 2024-09-17T18:20:29.083 INFO:tasks.workunit.client.0.smithi173.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-09-17T18:20:30.342 INFO:tasks.workunit.client.0.smithi173.stdout:i_service: objects: 112 objects, 145 MiB 2024-09-17T18:20:30.343 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: usage: 204 MiB used, 540 GiB / 540 GiB avail 2024-09-17T18:20:30.343 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: pgs: 32.483% pgs unknown 2024-09-17T18:20:30.343 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: 11.812% pgs not active 2024-09-17T18:20:30.343 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: 415 active+clean 2024-09-17T18:20:30.343 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: 242 unknown 2024-09-17T18:20:30.343 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: 88 creating+peering 2024-09-17T18:20:30.343 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: 2024-09-17T18:20:30.343 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: io: 2024-09-17T18:20:30.343 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: client: 12 MiB/s rd, 37 MiB/s wr, 194 op/s rd, 198 op/s wr 2024-09-17T18:20:30.343 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: 2024-09-17T18:20:30.343 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: 2024-09-17T18:20:30.343 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: [ OK ] LibRadosService.StatusFormat (4308 ms) 2024-09-17T18:20:30.344 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: [ RUN ] LibRadosService.Status 2024-09-17T18:20:30.344 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: [ OK ] LibRadosService.Status (20027 ms) 2024-09-17T18:20:30.344 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: [----------] 4 tests from LibRadosService (29472 ms total) 2024-09-17T18:20:30.344 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: 2024-09-17T18:20:30.344 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: [----------] Global test environment tear-down 2024-09-17T18:20:30.344 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: [==========] 4 tests from 1 test suite ran. (29472 ms total) 2024-09-17T18:20:30.344 INFO:tasks.workunit.client.0.smithi173.stdout: api_service: [ PASSED ] 4 tests. 2024-09-17T18:20:31.271 INFO:tasks.workunit.client.0.smithi173.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-09-17T18:20:31.271 INFO:tasks.workunit.client.0.smithi173.stdout: api_service_pp: [----------] Global test environment set-up. 2024-09-17T18:20:31.272 INFO:tasks.workunit.client.0.smithi173.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-09-17T18:20:31.272 INFO:tasks.workunit.client.0.smithi173.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-09-17T18:20:31.272 INFO:tasks.workunit.client.0.smithi173.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5078 ms) 2024-09-17T18:20:31.272 INFO:tasks.workunit.client.0.smithi173.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-09-17T18:20:31.272 INFO:tasks.workunit.client.0.smithi173.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (59 ms) 2024-09-17T18:20:31.272 INFO:tasks.workunit.client.0.smithi173.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-09-17T18:20:31.272 INFO:tasks.workunit.client.0.smithi173.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20055 ms) 2024-09-17T18:20:31.272 INFO:tasks.workunit.client.0.smithi173.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-09-17T18:20:31.272 INFO:tasks.workunit.client.0.smithi173.stdout: api_service_pp: attempt 0 of 20 2024-09-17T18:20:31.272 INFO:tasks.workunit.client.0.smithi173.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5138 ms) 2024-09-17T18:20:31.272 INFO:tasks.workunit.client.0.smithi173.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (30330 ms total) 2024-09-17T18:20:31.273 INFO:tasks.workunit.client.0.smithi173.stdout: api_service_pp: 2024-09-17T18:20:31.273 INFO:tasks.workunit.client.0.smithi173.stdout: api_service_pp: [----------] Global test environment tear-down 2024-09-17T18:20:31.273 INFO:tasks.workunit.client.0.smithi173.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (30331 ms total) 2024-09-17T18:20:31.273 INFO:tasks.workunit.client.0.smithi173.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-09-17T18:20:33.096 INFO:tasks.workunit.client.0.smithi173.stdout: misc: Running main() from gmock_main.cc 2024-09-17T18:20:33.096 INFO:tasks.workunit.client.0.smithi173.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-09-17T18:20:33.096 INFO:tasks.workunit.client.0.smithi173.stdout: misc: [----------] Global test environment set-up. 2024-09-17T18:20:33.097 INFO:tasks.workunit.client.0.smithi173.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-09-17T18:20:33.097 INFO:tasks.workunit.client.0.smithi173.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-09-17T18:20:33.097 INFO:tasks.workunit.client.0.smithi173.stdout: misc: [ OK ] NeoRadosMisc.Version (1100 ms) 2024-09-17T18:20:33.097 INFO:tasks.workunit.client.0.smithi173.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-09-17T18:20:33.097 INFO:tasks.workunit.client.0.smithi173.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (2025 ms) 2024-09-17T18:20:33.097 INFO:tasks.workunit.client.0.smithi173.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-09-17T18:20:33.097 INFO:tasks.workunit.client.0.smithi173.stdout: misc: [ OK ] NeoRadosMisc.LongName (3011 ms) 2024-09-17T18:20:33.097 INFO:tasks.workunit.client.0.smithi173.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-09-17T18:20:33.097 INFO:tasks.workunit.client.0.smithi173.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (2999 ms) 2024-09-17T18:20:33.097 INFO:tasks.workunit.client.0.smithi173.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-09-17T18:20:33.097 INFO:tasks.workunit.client.0.smithi173.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (2887 ms) 2024-09-17T18:20:33.097 INFO:tasks.workunit.client.0.smithi173.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-09-17T18:20:33.098 INFO:tasks.workunit.client.0.smithi173.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (3010 ms) 2024-09-17T18:20:33.098 INFO:tasks.workunit.client.0.smithi173.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-09-17T18:20:33.098 INFO:tasks.workunit.client.0.smithi173.stdout: misc: [ OK ] NeoRadosMisc.Exec (2997 ms) 2024-09-17T18:20:33.098 INFO:tasks.workunit.client.0.smithi173.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-09-17T18:20:33.098 INFO:tasks.workunit.client.0.smithi173.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (2991 ms) 2024-09-17T18:20:33.098 INFO:tasks.workunit.client.0.smithi173.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-09-17T18:20:33.098 INFO:tasks.workunit.client.0.smithi173.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (3045 ms) 2024-09-17T18:20:33.098 INFO:tasks.workunit.client.0.smithi173.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-09-17T18:20:33.098 INFO:tasks.workunit.client.0.smithi173.stdout: misc: [ OK ] NeoRadosMisc.BigObject (2941 ms) 2024-09-17T18:20:33.098 INFO:tasks.workunit.client.0.smithi173.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-09-17T18:20:33.098 INFO:tasks.workunit.client.0.smithi173.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (2033 ms) 2024-09-17T18:20:33.098 INFO:tasks.workunit.client.0.smithi173.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-09-17T18:20:33.098 INFO:tasks.workunit.client.0.smithi173.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (2972 ms) 2024-09-17T18:20:33.098 INFO:tasks.workunit.client.0.smithi173.stdout: misc: [----------] 12 tests from NeoRadosMisc (32011 ms total) 2024-09-17T18:20:33.099 INFO:tasks.workunit.client.0.smithi173.stdout: misc: 2024-09-17T18:20:33.099 INFO:tasks.workunit.client.0.smithi173.stdout: misc: [----------] Global test environment tear-down 2024-09-17T18:20:33.099 INFO:tasks.workunit.client.0.smithi173.stdout: misc: [==========] 12 tests from 1 test suite ran. (32011 ms total) 2024-09-17T18:20:33.099 INFO:tasks.workunit.client.0.smithi173.stdout: misc: [ PASSED ] 12 tests. 2024-09-17T18:20:39.084 INFO:tasks.workunit.client.0.smithi173.stdout: api_pool: Running main() from gmock_main.cc 2024-09-17T18:20:39.084 INFO:tasks.workunit.client.0.smithi173.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-09-17T18:20:39.084 INFO:tasks.workunit.client.0.smithi173.stdout: api_pool: [----------] Global test environment set-up. 2024-09-17T18:20:39.085 INFO:tasks.workunit.client.0.smithi173.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-09-17T18:20:39.085 INFO:tasks.workunit.client.0.smithi173.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-09-17T18:20:39.085 INFO:tasks.workunit.client.0.smithi173.stdout: api_pool: [ OK ] LibRadosPools.PoolList (2433 ms) 2024-09-17T18:20:39.085 INFO:tasks.workunit.client.0.smithi173.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-09-17T18:20:39.085 INFO:tasks.workunit.client.0.smithi173.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (3001 ms) 2024-09-17T18:20:39.085 INFO:tasks.workunit.client.0.smithi173.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-09-17T18:20:39.085 INFO:tasks.workunit.client.0.smithi173.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (3014 ms) 2024-09-17T18:20:39.085 INFO:tasks.workunit.client.0.smithi173.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-09-17T18:20:39.085 INFO:tasks.workunit.client.0.smithi173.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (2878 ms) 2024-09-17T18:20:39.085 INFO:tasks.workunit.client.0.smithi173.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-09-17T18:20:39.085 INFO:tasks.workunit.client.0.smithi173.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (3029 ms) 2024-09-17T18:20:39.085 INFO:tasks.workunit.client.0.smithi173.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-09-17T18:20:39.085 INFO:tasks.workunit.client.0.smithi173.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (5001 ms) 2024-09-17T18:20:39.086 INFO:tasks.workunit.client.0.smithi173.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-09-17T18:20:39.086 INFO:tasks.workunit.client.0.smithi173.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (5001 ms) 2024-09-17T18:20:39.086 INFO:tasks.workunit.client.0.smithi173.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-09-17T18:20:39.086 INFO:tasks.workunit.client.0.smithi173.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (5989 ms) 2024-09-17T18:20:39.086 INFO:tasks.workunit.client.0.smithi173.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-09-17T18:20:39.086 INFO:tasks.workunit.client.0.smithi173.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (7999 ms) 2024-09-17T18:20:39.086 INFO:tasks.workunit.client.0.smithi173.stdout: api_pool: [----------] 9 tests from LibRadosPools (38346 ms total) 2024-09-17T18:20:39.086 INFO:tasks.workunit.client.0.smithi173.stdout: api_pool: 2024-09-17T18:20:39.086 INFO:tasks.workunit.client.0.smithi173.stdout: api_pool: [----------] Global test environment tear-down 2024-09-17T18:20:39.086 INFO:tasks.workunit.client.0.smithi173.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (38347 ms total) 2024-09-17T18:20:39.086 INFO:tasks.workunit.client.0.smithi173.stdout: api_pool: [ PASSED ] 9 tests. 2024-09-17T18:20:40.116 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-09-17T18:20:40.116 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: [----------] Global test environment set-up. 2024-09-17T18:20:40.116 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-09-17T18:20:40.116 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-09-17T18:20:40.116 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-09-17T18:20:40.116 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-09-17T18:20:40.116 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: 2024-09-17T18:20:40.117 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-09-17T18:20:40.117 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-09-17T18:20:40.117 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-09-17T18:20:40.117 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: 2024-09-17T18:20:00.782+0000 7f3abc67b780 -1 failed for service _ceph-mon._tcp 2024-09-17T18:20:40.117 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: 2024-09-17T18:20:00.782+0000 7f3abc67b780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-09-17T18:20:40.117 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (83 ms) 2024-09-17T18:20:40.117 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-09-17T18:20:40.117 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (5014 ms) 2024-09-17T18:20:40.117 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (5097 ms total) 2024-09-17T18:20:40.117 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: 2024-09-17T18:20:40.117 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-09-17T18:20:40.118 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-09-17T18:20:40.118 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: started 0x5592a4192150 2024-09-17T18:20:40.118 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: started 0x5592a4317620 2024-09-17T18:20:40.118 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: started 2 aios 2024-09-17T18:20:40.118 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: waiting 0x5592a4192150 2024-09-17T18:20:40.118 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: waiting 0x5592a4317620 2024-09-17T18:20:40.118 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: done. 2024-09-17T18:20:40.118 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (5407 ms) 2024-09-17T18:20:40.118 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (5407 ms total) 2024-09-17T18:20:40.118 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: 2024-09-17T18:20:40.118 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-09-17T18:20:40.118 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-09-17T18:20:40.118 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-09-17T18:20:40.118 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-09-17T18:20:40.119 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: [ OK ] LibRadosMisc.Exec (38 ms) 2024-09-17T18:20:40.119 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-09-17T18:20:40.119 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (223 ms) 2024-09-17T18:20:40.119 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-09-17T18:20:40.119 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (3 ms) 2024-09-17T18:20:40.119 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-09-17T18:20:40.119 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: [ OK ] LibRadosMisc.Applications (4745 ms) 2024-09-17T18:20:40.119 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-09-17T18:20:40.119 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-09-17T18:20:40.119 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-09-17T18:20:40.119 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-09-17T18:20:40.119 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-09-17T18:20:40.119 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (21648 ms) 2024-09-17T18:20:40.119 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: [----------] 8 tests from LibRadosMisc (26657 ms total) 2024-09-17T18:20:40.120 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: 2024-09-17T18:20:40.120 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: [----------] Global test environment tear-down 2024-09-17T18:20:40.120 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (39377 ms total) 2024-09-17T18:20:40.120 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc: [ PASSED ] 12 tests. 2024-09-17T18:20:42.066 INFO:tasks.workunit.client.0.smithi173.stdout: io: Running main() from gmock_main.cc 2024-09-17T18:20:42.066 INFO:tasks.workunit.client.0.smithi173.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-09-17T18:20:42.066 INFO:tasks.workunit.client.0.smithi173.stdout: io: [----------] Global test environment set-up. 2024-09-17T18:20:42.066 INFO:tasks.workunit.client.0.smithi173.stdout: io: [----------] 14 tests from NeoRadosIo 2024-09-17T18:20:42.066 INFO:tasks.workunit.client.0.smithi173.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-09-17T18:20:42.066 INFO:tasks.workunit.client.0.smithi173.stdout: io: [ OK ] NeoRadosIo.Limits (2212 ms) 2024-09-17T18:20:42.066 INFO:tasks.workunit.client.0.smithi173.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-09-17T18:20:42.067 INFO:tasks.workunit.client.0.smithi173.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (3038 ms) 2024-09-17T18:20:42.067 INFO:tasks.workunit.client.0.smithi173.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-09-17T18:20:42.067 INFO:tasks.workunit.client.0.smithi173.stdout: io: [ OK ] NeoRadosIo.ReadOp (2993 ms) 2024-09-17T18:20:42.067 INFO:tasks.workunit.client.0.smithi173.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-09-17T18:20:42.067 INFO:tasks.workunit.client.0.smithi173.stdout: io: [ OK ] NeoRadosIo.SparseRead (2870 ms) 2024-09-17T18:20:42.067 INFO:tasks.workunit.client.0.smithi173.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-09-17T18:20:42.067 INFO:tasks.workunit.client.0.smithi173.stdout: io: [ OK ] NeoRadosIo.RoundTrip (3855 ms) 2024-09-17T18:20:42.067 INFO:tasks.workunit.client.0.smithi173.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-09-17T18:20:42.067 INFO:tasks.workunit.client.0.smithi173.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (2174 ms) 2024-09-17T18:20:42.067 INFO:tasks.workunit.client.0.smithi173.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-09-17T18:20:42.067 INFO:tasks.workunit.client.0.smithi173.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (3035 ms) 2024-09-17T18:20:42.067 INFO:tasks.workunit.client.0.smithi173.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-09-17T18:20:42.067 INFO:tasks.workunit.client.0.smithi173.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (2964 ms) 2024-09-17T18:20:42.068 INFO:tasks.workunit.client.0.smithi173.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-09-17T18:20:42.068 INFO:tasks.workunit.client.0.smithi173.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (2978 ms) 2024-09-17T18:20:42.068 INFO:tasks.workunit.client.0.smithi173.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-09-17T18:20:42.068 INFO:tasks.workunit.client.0.smithi173.stdout: io: [ OK ] NeoRadosIo.Trunc (3041 ms) 2024-09-17T18:20:42.068 INFO:tasks.workunit.client.0.smithi173.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-09-17T18:20:42.068 INFO:tasks.workunit.client.0.smithi173.stdout: io: [ OK ] NeoRadosIo.Remove (2958 ms) 2024-09-17T18:20:42.068 INFO:tasks.workunit.client.0.smithi173.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-09-17T18:20:42.068 INFO:tasks.workunit.client.0.smithi173.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (3011 ms) 2024-09-17T18:20:42.068 INFO:tasks.workunit.client.0.smithi173.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-09-17T18:20:42.068 INFO:tasks.workunit.client.0.smithi173.stdout: io: [ OK ] NeoRadosIo.RmXattr (3018 ms) 2024-09-17T18:20:42.068 INFO:tasks.workunit.client.0.smithi173.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-09-17T18:20:42.068 INFO:tasks.workunit.client.0.smithi173.stdout: io: [ OK ] NeoRadosIo.GetXattrs (2978 ms) 2024-09-17T18:20:42.068 INFO:tasks.workunit.client.0.smithi173.stdout: io: [----------] 14 tests from NeoRadosIo (41125 ms total) 2024-09-17T18:20:42.068 INFO:tasks.workunit.client.0.smithi173.stdout: io: 2024-09-17T18:20:42.069 INFO:tasks.workunit.client.0.smithi173.stdout: io: [----------] Global test environment tear-down 2024-09-17T18:20:42.069 INFO:tasks.workunit.client.0.smithi173.stdout: io: [==========] 14 tests from 1 test suite ran. (41125 ms total) 2024-09-17T18:20:42.069 INFO:tasks.workunit.client.0.smithi173.stdout: io: [ PASSED ] 14 tests. 2024-09-17T18:20:43.067 INFO:tasks.workunit.client.0.smithi173.stdout: read_operations: Running main() from gmock_main.cc 2024-09-17T18:20:43.067 INFO:tasks.workunit.client.0.smithi173.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-09-17T18:20:43.067 INFO:tasks.workunit.client.0.smithi173.stdout: read_operations: [----------] Global test environment set-up. 2024-09-17T18:20:43.067 INFO:tasks.workunit.client.0.smithi173.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-09-17T18:20:43.067 INFO:tasks.workunit.client.0.smithi173.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-09-17T18:20:43.067 INFO:tasks.workunit.client.0.smithi173.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (3104 ms) 2024-09-17T18:20:43.067 INFO:tasks.workunit.client.0.smithi173.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-09-17T18:20:43.067 INFO:tasks.workunit.client.0.smithi173.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (3005 ms) 2024-09-17T18:20:43.067 INFO:tasks.workunit.client.0.smithi173.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-09-17T18:20:43.067 INFO:tasks.workunit.client.0.smithi173.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (2994 ms) 2024-09-17T18:20:43.067 INFO:tasks.workunit.client.0.smithi173.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-09-17T18:20:43.068 INFO:tasks.workunit.client.0.smithi173.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (2895 ms) 2024-09-17T18:20:43.068 INFO:tasks.workunit.client.0.smithi173.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-09-17T18:20:43.068 INFO:tasks.workunit.client.0.smithi173.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (3018 ms) 2024-09-17T18:20:43.068 INFO:tasks.workunit.client.0.smithi173.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-09-17T18:20:43.068 INFO:tasks.workunit.client.0.smithi173.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (3007 ms) 2024-09-17T18:20:43.068 INFO:tasks.workunit.client.0.smithi173.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-09-17T18:20:43.068 INFO:tasks.workunit.client.0.smithi173.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (2976 ms) 2024-09-17T18:20:43.068 INFO:tasks.workunit.client.0.smithi173.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-09-17T18:20:43.068 INFO:tasks.workunit.client.0.smithi173.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (3062 ms) 2024-09-17T18:20:43.068 INFO:tasks.workunit.client.0.smithi173.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-09-17T18:20:43.068 INFO:tasks.workunit.client.0.smithi173.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (2924 ms) 2024-09-17T18:20:43.068 INFO:tasks.workunit.client.0.smithi173.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-09-17T18:20:43.069 INFO:tasks.workunit.client.0.smithi173.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (3013 ms) 2024-09-17T18:20:43.069 INFO:tasks.workunit.client.0.smithi173.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-09-17T18:20:43.069 INFO:tasks.workunit.client.0.smithi173.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (2992 ms) 2024-09-17T18:20:43.069 INFO:tasks.workunit.client.0.smithi173.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-09-17T18:20:43.069 INFO:tasks.workunit.client.0.smithi173.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (3035 ms) 2024-09-17T18:20:43.069 INFO:tasks.workunit.client.0.smithi173.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-09-17T18:20:43.069 INFO:tasks.workunit.client.0.smithi173.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (2972 ms) 2024-09-17T18:20:43.069 INFO:tasks.workunit.client.0.smithi173.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-09-17T18:20:43.069 INFO:tasks.workunit.client.0.smithi173.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (2999 ms) 2024-09-17T18:20:43.069 INFO:tasks.workunit.client.0.smithi173.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (41996 ms total) 2024-09-17T18:20:43.069 INFO:tasks.workunit.client.0.smithi173.stdout: read_operations: 2024-09-17T18:20:43.069 INFO:tasks.workunit.client.0.smithi173.stdout: read_operations: [----------] Global test environment tear-down 2024-09-17T18:20:43.069 INFO:tasks.workunit.client.0.smithi173.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (41996 ms total) 2024-09-17T18:20:43.069 INFO:tasks.workunit.client.0.smithi173.stdout: read_operations: [ PASSED ] 14 tests. 2024-09-17T18:20:54.629 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-17T18:20:54.629 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-17T18:20:54.629 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-17T18:20:54.632 INFO:tasks.ceph.mon.c:Started 2024-09-17T18:20:54.632 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T18:20:54.633 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:20:54.727 INFO:tasks.ceph.mon.c.smithi121.stderr:2024-09-17T18:20:54.724+0000 7f210c51ec80 -1 mon.c@-1(???).log v0 log_external_backlog rewinding external_log_to from 22 -> 0 (sync_force? mon rebuild?) 2024-09-17T18:20:54.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.738+0000 7f1700e73640 10 monclient: get_monmap_and_config 2024-09-17T18:20:54.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.738+0000 7f1700e73640 10 monclient: build_initial_monmap 2024-09-17T18:20:54.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.738+0000 7f1700e73640 10 monclient: monmap: 2024-09-17T18:20:54.741 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:20:54.741 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:20:54.741 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:20:54.739908+0000 2024-09-17T18:20:54.741 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:20:54.739908+0000 2024-09-17T18:20:54.741 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:20:54.741 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:20:54.741 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:20:54.742 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:20:54.742 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:20:54.742 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:20:54.742 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:20:54.742 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:20:54.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.738+0000 7f1700e73640 1 Processor -- start 2024-09-17T18:20:54.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.738+0000 7f1700e73640 1 -- start start 2024-09-17T18:20:54.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.738+0000 7f1700e73640 10 monclient: init 2024-09-17T18:20:54.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.739+0000 7f1700e73640 10 monclient: _reopen_session rank -1 2024-09-17T18:20:54.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.739+0000 7f1700e73640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-17T18:20:54.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.739+0000 7f1700e73640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f16fc0a5c00 0x7f16fc09d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:20:54.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.739+0000 7f1700e73640 10 monclient(hunting): picked mon.noname-c con 0x7f16fc0a5c00 addr v2:172.21.15.173:3300/0 2024-09-17T18:20:54.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.739+0000 7f1700e73640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f16fc0a5fe0 addr v1:172.21.15.173:6789/0 2024-09-17T18:20:54.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.739+0000 7f1700e73640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f16fc13dc00 addr v1:172.21.15.121:6789/0 2024-09-17T18:20:54.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.739+0000 7f1700e73640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16fc0a8a80 con 0x7f16fc13dc00 2024-09-17T18:20:54.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.739+0000 7f1700e73640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16fc09de00 con 0x7f16fc0a5fe0 2024-09-17T18:20:54.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.739+0000 7f1700e73640 10 monclient(hunting): start opening mon connection 2024-09-17T18:20:54.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.739+0000 7f1700e73640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f16fc0a7110 con 0x7f16fc0a5c00 2024-09-17T18:20:54.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.739+0000 7f1700e73640 10 monclient(hunting): _renew_subs 2024-09-17T18:20:54.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.739+0000 7f1700e73640 10 monclient(hunting): authenticate will time out at 858.456655s 2024-09-17T18:20:54.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.739+0000 7f16fad9d640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f16fc13dc00 0x7f16fc13fff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:33328/0 (socket says 172.21.15.121:33328) 2024-09-17T18:20:54.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.739+0000 7f16fad9d640 1 -- 172.21.15.121:0/3205416973 learned_addr learned my addr 172.21.15.121:0/3205416973 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:20:54.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.740+0000 7f16fa59c640 1 --2- 172.21.15.121:0/3205416973 >> v2:172.21.15.173:3300/0 conn(0x7f16fc0a5c00 0x7f16fc09d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:20:54.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.740+0000 7f16fa59c640 10 monclient(hunting): get_auth_request con 0x7f16fc0a5c00 auth_method 0 2024-09-17T18:20:54.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.740+0000 7f16fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:20:54.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.740+0000 7f16fa59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:20:54.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.740+0000 7f16fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:20:54.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.740+0000 7f16fa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:20:54.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.740+0000 7f16fa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:20:54.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.740+0000 7f16fa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:20:54.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.740+0000 7f16fa59c640 10 monclient(hunting): handle_auth_done global_id 24088 payload 274 2024-09-17T18:20:54.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.740+0000 7f16fa59c640 1 -- 172.21.15.121:0/3205416973 >> v1:172.21.15.173:6789/0 conn(0x7f16fc0a5fe0 legacy=0x7f16fc0a63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:20:54.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.740+0000 7f16fa59c640 1 -- 172.21.15.121:0/3205416973 >> v1:172.21.15.121:6789/0 conn(0x7f16fc13dc00 legacy=0x7f16fc13fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:20:54.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.740+0000 7f16fa59c640 10 monclient: _finish_hunting 0 2024-09-17T18:20:54.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.740+0000 7f16fa59c640 1 monclient: found mon.noname-c 2024-09-17T18:20:54.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.740+0000 7f16fa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:20:54.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.740+0000 7f16fa59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.173:3300/0 2024-09-17T18:20:54.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.740+0000 7f16fa59c640 1 -- 172.21.15.121:0/3205416973 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16fc0a74d0 con 0x7f16fc0a5c00 2024-09-17T18:20:54.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.743+0000 7f16fa59c640 1 -- 172.21.15.121:0/3205416973 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16fc0a5c00 msgr2=0x7f16fc09d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:20:54.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.743+0000 7f16fa59c640 1 --2- 172.21.15.121:0/3205416973 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16fc0a5c00 0x7f16fc09d8c0 secure :-1 s=READY pgs=7073 cs=0 l=1 rev1=1 crypto rx=0x7f16fc09dfd0 tx=0x7f16ec029fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47b503e15b509230 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:20:54.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.743+0000 7f16f959a640 1 -- 172.21.15.121:0/3205416973 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ec037070 con 0x7f16fc0a5c00 2024-09-17T18:20:54.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.743+0000 7f16f959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:20:54.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.743+0000 7f16f959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:20:54.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.743+0000 7f16f959a640 10 monclient: dump: 2024-09-17T18:20:54.755 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:20:54.755 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:20:54.755 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:20:54.755 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:20:54.755 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:20:54.755 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:20:54.755 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:20:54.755 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:20:54.755 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:20:54.755 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:20:54.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.743+0000 7f16f959a640 10 monclient: _finish_auth 0 2024-09-17T18:20:54.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.743+0000 7f16f959a640 10 monclient: _check_auth_tickets 2024-09-17T18:20:54.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.743+0000 7f16f959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:20:54.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.743+0000 7f16f959a640 1 -- 172.21.15.121:0/3205416973 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16ec032440 con 0x7f16fc0a5c00 2024-09-17T18:20:54.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.743+0000 7f16f959a640 10 monclient: handle_config config(0 keys) 2024-09-17T18:20:54.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.743+0000 7f16f959a640 1 -- 172.21.15.121:0/3205416973 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ec032810 con 0x7f16fc0a5c00 2024-09-17T18:20:54.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.743+0000 7f16f959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:20:54.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.743+0000 7f16f959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:20:54.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.743+0000 7f16f959a640 10 monclient: dump: 2024-09-17T18:20:54.756 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:20:54.756 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:20:54.756 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:20:54.756 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:20:54.756 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:20:54.757 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:20:54.757 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:20:54.757 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:20:54.757 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:20:54.757 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:20:54.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.743+0000 7f1700e73640 5 monclient: authenticate success, global_id 24088 2024-09-17T18:20:54.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.743+0000 7f1700e73640 10 monclient: get_monmap_and_config success 2024-09-17T18:20:54.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.743+0000 7f1700e73640 4 set_mon_vals no callback set 2024-09-17T18:20:54.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.744+0000 7f1700e73640 10 monclient: shutdown 2024-09-17T18:20:54.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.744+0000 7f1700e73640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:20:54.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.744+0000 7f1700e73640 1 -- 172.21.15.121:0/3205416973 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16fc0a5c00 msgr2=0x7f16fc09d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:20:54.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.744+0000 7f1700e73640 1 --2- 172.21.15.121:0/3205416973 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16fc0a5c00 0x7f16fc09d8c0 secure :-1 s=READY pgs=7073 cs=0 l=1 rev1=1 crypto rx=0x7f16fc09dfd0 tx=0x7f16ec029fb0 comp rx=0 tx=0).stop 2024-09-17T18:20:54.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.744+0000 7f1700e73640 1 -- 172.21.15.121:0/3205416973 shutdown_connections 2024-09-17T18:20:54.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.744+0000 7f1700e73640 1 --2- 172.21.15.121:0/3205416973 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16fc0a5c00 0x7f16fc09d8c0 unknown :-1 s=CLOSED pgs=7073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:20:54.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.744+0000 7f1700e73640 1 -- 172.21.15.121:0/3205416973 >> 172.21.15.121:0/3205416973 conn(0x7f16fc0ab9b0 msgr2=0x7f16fc0a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:20:54.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.744+0000 7f1700e73640 1 -- 172.21.15.121:0/3205416973 shutdown_connections 2024-09-17T18:20:54.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.744+0000 7f1700e73640 1 -- 172.21.15.121:0/3205416973 wait complete. 2024-09-17T18:20:54.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.745+0000 7f1700e73640 10 monclient: build_initial_monmap 2024-09-17T18:20:54.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.745+0000 7f1700e73640 10 monclient: monmap: 2024-09-17T18:20:54.758 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:20:54.758 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:20:54.758 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:20:54.758 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:20:54.758 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:20:54.758 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:20:54.758 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:20:54.759 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:20:54.759 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:20:54.759 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:20:54.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.745+0000 7f1700e73640 1 librados: starting msgr at 2024-09-17T18:20:54.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.745+0000 7f1700e73640 1 librados: starting objecter 2024-09-17T18:20:54.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.745+0000 7f1700e73640 1 Processor -- start 2024-09-17T18:20:54.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.745+0000 7f1700e73640 1 -- start start 2024-09-17T18:20:54.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.745+0000 7f1700e73640 1 librados: setting wanted keys 2024-09-17T18:20:54.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.745+0000 7f1700e73640 1 librados: calling monclient init 2024-09-17T18:20:54.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.745+0000 7f1700e73640 10 monclient: init 2024-09-17T18:20:54.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.745+0000 7f1700e73640 10 monclient: _reopen_session rank -1 2024-09-17T18:20:54.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.745+0000 7f1700e73640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T18:20:54.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.745+0000 7f1700e73640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16fc0a5c00 0x7f16fc13cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:20:54.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.745+0000 7f1700e73640 10 monclient(hunting): picked mon.noname-a con 0x7f16fc0a5c00 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:20:54.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.746+0000 7f1700e73640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16fc0a5fe0 0x7f16fc1cdcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:20:54.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.746+0000 7f1700e73640 10 monclient(hunting): picked mon.noname-c con 0x7f16fc0a5fe0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:20:54.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.746+0000 7f1700e73640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16fc13dc00 0x7f16fc13a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:20:54.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.746+0000 7f1700e73640 10 monclient(hunting): picked mon.noname-b con 0x7f16fc13dc00 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:20:54.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.746+0000 7f1700e73640 10 monclient(hunting): start opening mon connection 2024-09-17T18:20:54.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.746+0000 7f1700e73640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f16fc0703d0 con 0x7f16fc0a5c00 2024-09-17T18:20:54.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.746+0000 7f1700e73640 10 monclient(hunting): start opening mon connection 2024-09-17T18:20:54.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.746+0000 7f1700e73640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f16fc09ce70 con 0x7f16fc13dc00 2024-09-17T18:20:54.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.746+0000 7f1700e73640 10 monclient(hunting): start opening mon connection 2024-09-17T18:20:54.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.746+0000 7f1700e73640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f16fc070550 con 0x7f16fc0a5fe0 2024-09-17T18:20:54.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.746+0000 7f1700e73640 10 monclient(hunting): _renew_subs 2024-09-17T18:20:54.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.746+0000 7f1700e73640 10 monclient(hunting): authenticate will time out at 858.463056s 2024-09-17T18:20:54.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.746+0000 7f16fa59c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16fc0a5c00 0x7f16fc13cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:20:54.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.746+0000 7f16fa59c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16fc0a5c00 0x7f16fc13cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36154/0 (socket says 172.21.15.121:36154) 2024-09-17T18:20:54.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.746+0000 7f16fa59c640 1 -- 172.21.15.121:0/2852328308 learned_addr learned my addr 172.21.15.121:0/2852328308 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:20:54.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.746+0000 7f16fa59c640 10 monclient(hunting): get_auth_request con 0x7f16fc0a5c00 auth_method 0 2024-09-17T18:20:54.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.746+0000 7f16fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:20:54.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.746+0000 7f16fa59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:20:54.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.746+0000 7f16fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:20:54.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.746+0000 7f16f9d9b640 1 --2- 172.21.15.121:0/2852328308 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16fc0a5fe0 0x7f16fc1cdcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:20:54.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.746+0000 7f16f9d9b640 10 monclient(hunting): get_auth_request con 0x7f16fc0a5fe0 auth_method 0 2024-09-17T18:20:54.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.746+0000 7f16f9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:20:54.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.746+0000 7f16f9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:20:54.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.746+0000 7f16f9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:20:54.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.746+0000 7f16fa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:20:54.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.746+0000 7f16fa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:20:54.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.746+0000 7f16fa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:20:54.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.746+0000 7f16f9d9b640 1 -- 172.21.15.121:0/2852328308 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16fc0a5fe0 msgr2=0x7f16fc1cdcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T18:20:54.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.746+0000 7f16f9d9b640 1 -- 172.21.15.121:0/2852328308 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16fc0a5fe0 msgr2=0x7f16fc1cdcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:20:54.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.746+0000 7f16f9d9b640 1 --2- 172.21.15.121:0/2852328308 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16fc0a5fe0 0x7f16fc1cdcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:20:54.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.746+0000 7f16f9d9b640 1 --2- 172.21.15.121:0/2852328308 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16fc0a5fe0 0x7f16fc1cdcc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:20:54.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.746+0000 7f16fa59c640 10 monclient(hunting): handle_auth_done global_id 23640 payload 824 2024-09-17T18:20:54.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.746+0000 7f16fad9d640 1 --2- 172.21.15.121:0/2852328308 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16fc13dc00 0x7f16fc13a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:20:54.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.746+0000 7f16fa59c640 1 -- 172.21.15.121:0/2852328308 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16fc0a5fe0 msgr2=0x7f16fc1cdcc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:20:54.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.746+0000 7f16fa59c640 1 --2- 172.21.15.121:0/2852328308 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16fc0a5fe0 0x7f16fc1cdcc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:20:54.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.746+0000 7f16fa59c640 1 -- 172.21.15.121:0/2852328308 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16fc13dc00 msgr2=0x7f16fc13a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:20:54.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.746+0000 7f16fa59c640 1 --2- 172.21.15.121:0/2852328308 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16fc13dc00 0x7f16fc13a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:20:54.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.746+0000 7f16fa59c640 10 monclient: _finish_hunting 0 2024-09-17T18:20:54.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.746+0000 7f16fa59c640 1 monclient: found mon.noname-a 2024-09-17T18:20:54.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.746+0000 7f16fa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:20:54.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.746+0000 7f16fa59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:20:54.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.746+0000 7f16fa59c640 1 -- 172.21.15.121:0/2852328308 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16fc0a23f0 con 0x7f16fc0a5c00 2024-09-17T18:20:54.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.748+0000 7f16fa59c640 1 -- 172.21.15.121:0/2852328308 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16fc0a5c00 msgr2=0x7f16fc13cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:20:54.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.748+0000 7f16fa59c640 1 --2- 172.21.15.121:0/2852328308 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16fc0a5c00 0x7f16fc13cd00 secure :-1 s=READY pgs=6789 cs=0 l=1 rev1=1 crypto rx=0x7f16ec000c00 tx=0x7f16ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:20:54.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.748+0000 7f16eaffd640 10 client.?.objecter ms_handle_connect 0x7f16fc0a5c00 2024-09-17T18:20:54.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.748+0000 7f16eaffd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:20:54.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.748+0000 7f16eaffd640 1 -- 172.21.15.121:0/2852328308 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ec041070 con 0x7f16fc0a5c00 2024-09-17T18:20:54.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.748+0000 7f16eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:20:54.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.748+0000 7f16eaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:20:54.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.748+0000 7f16eaffd640 10 monclient: dump: 2024-09-17T18:20:54.765 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:20:54.765 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:20:54.765 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:20:54.765 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:20:54.765 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:20:54.765 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:20:54.765 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:20:54.765 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:20:54.765 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:20:54.766 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:20:54.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.748+0000 7f16eaffd640 10 monclient: _finish_auth 0 2024-09-17T18:20:54.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.748+0000 7f16eaffd640 10 monclient: _check_auth_tickets 2024-09-17T18:20:54.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.748+0000 7f16eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:20:54.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.748+0000 7f16eaffd640 1 -- 172.21.15.121:0/2852328308 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16ec03c890 con 0x7f16fc0a5c00 2024-09-17T18:20:54.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.748+0000 7f1700e73640 5 monclient: authenticate success, global_id 23640 2024-09-17T18:20:54.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.748+0000 7f1700e73640 10 monclient: _renew_subs 2024-09-17T18:20:54.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.748+0000 7f1700e73640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:20:54.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.748+0000 7f1700e73640 1 -- 172.21.15.121:0/2852328308 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16fc121e70 con 0x7f16fc0a5c00 2024-09-17T18:20:54.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.748+0000 7f16eaffd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:20:54.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.748+0000 7f16eaffd640 1 -- 172.21.15.121:0/2852328308 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ec037040 con 0x7f16fc0a5c00 2024-09-17T18:20:54.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.748+0000 7f16eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:20:54.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.748+0000 7f16eaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:20:54.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.748+0000 7f16eaffd640 10 monclient: dump: 2024-09-17T18:20:54.767 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:20:54.767 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:20:54.767 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:20:54.767 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:20:54.767 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:20:54.767 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:20:54.767 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:20:54.767 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:20:54.767 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:20:54.767 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:20:54.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.748+0000 7f16f8d99640 4 set_mon_vals no callback set 2024-09-17T18:20:54.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.748+0000 7f1700e73640 10 client.23640.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:20:54.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.748+0000 7f1700e73640 10 monclient: _renew_subs 2024-09-17T18:20:54.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.748+0000 7f1700e73640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:20:54.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.748+0000 7f1700e73640 1 -- 172.21.15.121:0/2852328308 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16fc1411a0 con 0x7f16fc0a5c00 2024-09-17T18:20:54.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.748+0000 7f1700e73640 1 librados: init done 2024-09-17T18:20:54.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.749+0000 7f16eaffd640 1 -- 172.21.15.121:0/2852328308 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f16ec044070 con 0x7f16fc0a5c00 2024-09-17T18:20:54.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.749+0000 7f16eaffd640 1 --2- 172.21.15.121:0/2852328308 >> v2:172.21.15.173:6812/2423786037 conn(0x7f16dc0401a0 0x7f16dc042660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:20:54.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.750+0000 7f1700e73640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:20:54.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.750+0000 7f1700e73640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:20:54.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.750+0000 7f1700e73640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:20:54.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.750+0000 7f16eaffd640 1 -- 172.21.15.121:0/2852328308 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7831+0+0 (secure 0 0 0) 0x7f16ec0747d0 con 0x7f16fc0a5c00 2024-09-17T18:20:54.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.750+0000 7f16eaffd640 10 client.23640.objecter ms_dispatch 0x7f16fc000b90 osd_map(71..71 src has 1..71) 2024-09-17T18:20:54.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.750+0000 7f16eaffd640 3 client.23640.objecter handle_osd_map got epochs [71,71] > 0 2024-09-17T18:20:54.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.750+0000 7f16eaffd640 3 client.23640.objecter handle_osd_map decoding full epoch 71 2024-09-17T18:20:54.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.750+0000 7f1700e73640 1 -- 172.21.15.121:0/2852328308 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16fc09d8c0 con 0x7f16fc0a5c00 2024-09-17T18:20:54.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.750+0000 7f16eaffd640 20 client.23640.objecter dump_active .. 0 homeless 2024-09-17T18:20:54.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.750+0000 7f16f9d9b640 1 --2- 172.21.15.121:0/2852328308 >> v2:172.21.15.173:6812/2423786037 conn(0x7f16dc0401a0 0x7f16dc042660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:20:54.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.750+0000 7f16f9d9b640 10 monclient: get_auth_request con 0x7f16dc0401a0 auth_method 0 2024-09-17T18:20:54.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.752+0000 7f16f9d9b640 1 --2- 172.21.15.121:0/2852328308 >> v2:172.21.15.173:6812/2423786037 conn(0x7f16dc0401a0 0x7f16dc042660 secure :-1 s=READY pgs=3849 cs=0 l=1 rev1=1 crypto rx=0x7f16f0007c50 tx=0x7f16f0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:20:54.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.752+0000 7f16eaffd640 10 client.23640.objecter ms_handle_connect 0x7f16dc0401a0 2024-09-17T18:20:54.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.755+0000 7f16eaffd640 1 -- 172.21.15.121:0/2852328308 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f16fc0a23f0 con 0x7f16fc0a5c00 2024-09-17T18:20:54.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.755+0000 7f16eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:20:54.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.755+0000 7f16eaffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:20:55.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.998+0000 7f1700e73640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:20:55.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.998+0000 7f1700e73640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:20:55.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.998+0000 7f1700e73640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:20:55.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.998+0000 7f1700e73640 1 -- 172.21.15.121:0/2852328308 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f16fc06cd40 con 0x7f16fc0a5c00 2024-09-17T18:20:55.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.999+0000 7f16eaffd640 1 -- 172.21.15.121:0/2852328308 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f16ec03be10 con 0x7f16fc0a5c00 2024-09-17T18:20:55.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.999+0000 7f16eaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:20:55.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:54.999+0000 7f16eaffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:20:55.002 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":8,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":90,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:20:55.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:55.000+0000 7f1700e73640 10 librados: watch_flush enter 2024-09-17T18:20:55.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:55.000+0000 7f1700e73640 10 librados: watch_flush exit 2024-09-17T18:20:55.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:55.000+0000 7f1700e73640 20 client.23640.objecter shutdown clearing up homeless session... 2024-09-17T18:20:55.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:55.000+0000 7f1700e73640 10 client.23640.objecter successfully canceled tick 2024-09-17T18:20:55.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:55.001+0000 7f1700e73640 1 -- 172.21.15.121:0/2852328308 >> v2:172.21.15.173:6812/2423786037 conn(0x7f16dc0401a0 msgr2=0x7f16dc042660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:20:55.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:55.001+0000 7f1700e73640 1 --2- 172.21.15.121:0/2852328308 >> v2:172.21.15.173:6812/2423786037 conn(0x7f16dc0401a0 0x7f16dc042660 secure :-1 s=READY pgs=3849 cs=0 l=1 rev1=1 crypto rx=0x7f16f0007c50 tx=0x7f16f0009040 comp rx=0 tx=0).stop 2024-09-17T18:20:55.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:55.001+0000 7f1700e73640 10 monclient: shutdown 2024-09-17T18:20:55.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:55.001+0000 7f1700e73640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:20:55.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:55.001+0000 7f1700e73640 1 -- 172.21.15.121:0/2852328308 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16fc0a5c00 msgr2=0x7f16fc13cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:20:55.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:55.001+0000 7f1700e73640 1 --2- 172.21.15.121:0/2852328308 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16fc0a5c00 0x7f16fc13cd00 secure :-1 s=READY pgs=6789 cs=0 l=1 rev1=1 crypto rx=0x7f16ec000c00 tx=0x7f16ec0046c0 comp rx=0 tx=0).stop 2024-09-17T18:20:55.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:55.001+0000 7f1700e73640 1 -- 172.21.15.121:0/2852328308 shutdown_connections 2024-09-17T18:20:55.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:55.001+0000 7f1700e73640 1 --2- 172.21.15.121:0/2852328308 >> v2:172.21.15.173:6812/2423786037 conn(0x7f16dc0401a0 0x7f16dc042660 unknown :-1 s=CLOSED pgs=3849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:20:55.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:55.001+0000 7f1700e73640 1 --2- 172.21.15.121:0/2852328308 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16fc13dc00 0x7f16fc13a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:20:55.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:55.001+0000 7f1700e73640 1 --2- 172.21.15.121:0/2852328308 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16fc0a5fe0 0x7f16fc1cdcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:20:55.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:55.001+0000 7f1700e73640 1 --2- 172.21.15.121:0/2852328308 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16fc0a5c00 0x7f16fc13cd00 unknown :-1 s=CLOSED pgs=6789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:20:55.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:55.001+0000 7f1700e73640 1 -- 172.21.15.121:0/2852328308 >> 172.21.15.121:0/2852328308 conn(0x7f16fc0ab9b0 msgr2=0x7f16fc0a99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:20:55.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:55.001+0000 7f1700e73640 1 -- 172.21.15.121:0/2852328308 shutdown_connections 2024-09-17T18:20:55.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:55.001+0000 7f1700e73640 1 -- 172.21.15.121:0/2852328308 wait complete. 2024-09-17T18:20:55.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:55.001+0000 7f1700e73640 1 librados: shutdown 2024-09-17T18:20:56.284 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: Running main() from gmock_main.cc 2024-09-17T18:20:56.284 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-09-17T18:20:56.284 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: [----------] Global test environment set-up. 2024-09-17T18:20:56.284 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-09-17T18:20:56.284 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-09-17T18:20:56.284 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (2000 ms) 2024-09-17T18:20:56.284 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-09-17T18:20:56.285 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (2023 ms) 2024-09-17T18:20:56.285 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-09-17T18:20:56.285 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (2002 ms) 2024-09-17T18:20:56.285 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-09-17T18:20:56.285 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (2001 ms) 2024-09-17T18:20:56.285 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (8026 ms total) 2024-09-17T18:20:56.285 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: 2024-09-17T18:20:56.285 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-09-17T18:20:56.285 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-09-17T18:20:56.285 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (4075 ms) 2024-09-17T18:20:56.285 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-09-17T18:20:56.285 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (4043 ms) 2024-09-17T18:20:56.285 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-09-17T18:20:56.286 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (4952 ms) 2024-09-17T18:20:56.286 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (13070 ms total) 2024-09-17T18:20:56.286 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: 2024-09-17T18:20:56.286 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-09-17T18:20:56.286 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-09-17T18:20:56.286 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (2962 ms) 2024-09-17T18:20:56.286 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-09-17T18:20:56.286 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (2004 ms) 2024-09-17T18:20:56.286 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-09-17T18:20:56.286 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (1999 ms) 2024-09-17T18:20:56.286 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-09-17T18:20:56.286 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (2005 ms) 2024-09-17T18:20:56.286 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (8970 ms total) 2024-09-17T18:20:56.287 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: 2024-09-17T18:20:56.287 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-09-17T18:20:56.287 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-09-17T18:20:56.287 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4024 ms) 2024-09-17T18:20:56.287 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-09-17T18:20:56.287 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (3991 ms) 2024-09-17T18:20:56.287 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (8015 ms total) 2024-09-17T18:20:56.287 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: 2024-09-17T18:20:56.287 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: [----------] Global test environment tear-down 2024-09-17T18:20:56.287 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (55527 ms total) 2024-09-17T18:20:56.287 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-09-17T18:20:58.014 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:20:58.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.119+0000 7fea7759e640 10 monclient: get_monmap_and_config 2024-09-17T18:20:58.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.119+0000 7fea7759e640 10 monclient: build_initial_monmap 2024-09-17T18:20:58.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.119+0000 7fea7759e640 10 monclient: monmap: 2024-09-17T18:20:58.122 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:20:58.122 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:20:58.122 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:20:58.121356+0000 2024-09-17T18:20:58.123 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:20:58.121356+0000 2024-09-17T18:20:58.123 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:20:58.123 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:20:58.123 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:20:58.123 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:20:58.123 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:20:58.123 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:20:58.123 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:20:58.123 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:20:58.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.120+0000 7fea7759e640 1 Processor -- start 2024-09-17T18:20:58.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.120+0000 7fea7759e640 1 -- start start 2024-09-17T18:20:58.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.120+0000 7fea7759e640 10 monclient: init 2024-09-17T18:20:58.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.120+0000 7fea7759e640 10 monclient: _reopen_session rank -1 2024-09-17T18:20:58.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.120+0000 7fea7759e640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-17T18:20:58.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.120+0000 7fea7759e640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fea780a1f20 0x7fea781386a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:20:58.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.120+0000 7fea7759e640 10 monclient(hunting): picked mon.noname-c con 0x7fea780a1f20 addr v2:172.21.15.173:3300/0 2024-09-17T18:20:58.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.120+0000 7fea7759e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fea78138be0 addr v1:172.21.15.173:6789/0 2024-09-17T18:20:58.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.120+0000 7fea7759e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fea78139030 addr v1:172.21.15.121:6789/0 2024-09-17T18:20:58.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.120+0000 7fea7759e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea781394a0 con 0x7fea78139030 2024-09-17T18:20:58.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.120+0000 7fea7759e640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea7809eb50 con 0x7fea78138be0 2024-09-17T18:20:58.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.120+0000 7fea7759e640 10 monclient(hunting): start opening mon connection 2024-09-17T18:20:58.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.120+0000 7fea7759e640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fea78074bc0 con 0x7fea780a1f20 2024-09-17T18:20:58.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.120+0000 7fea7759e640 10 monclient(hunting): _renew_subs 2024-09-17T18:20:58.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.120+0000 7fea7759e640 10 monclient(hunting): authenticate will time out at 861.837968s 2024-09-17T18:20:58.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.121+0000 7fea76d9d640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fea78139030 0x7fea780a2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:33332/0 (socket says 172.21.15.121:33332) 2024-09-17T18:20:58.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.121+0000 7fea76d9d640 1 -- 172.21.15.121:0/4086309082 learned_addr learned my addr 172.21.15.121:0/4086309082 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:20:58.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.121+0000 7fea7659c640 1 --2- 172.21.15.121:0/4086309082 >> v2:172.21.15.173:3300/0 conn(0x7fea780a1f20 0x7fea781386a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:20:58.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.121+0000 7fea7659c640 10 monclient(hunting): get_auth_request con 0x7fea780a1f20 auth_method 0 2024-09-17T18:20:58.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.121+0000 7fea7659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:20:58.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.121+0000 7fea7659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:20:58.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.121+0000 7fea7659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:20:58.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.121+0000 7fea7659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:20:58.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.121+0000 7fea7659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:20:58.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.121+0000 7fea7659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:20:58.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.121+0000 7fea7659c640 10 monclient(hunting): handle_auth_done global_id 24109 payload 274 2024-09-17T18:20:58.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.121+0000 7fea7659c640 1 -- 172.21.15.121:0/4086309082 >> v1:172.21.15.173:6789/0 conn(0x7fea78138be0 legacy=0x7fea780a2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:20:58.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.121+0000 7fea7659c640 1 -- 172.21.15.121:0/4086309082 >> v1:172.21.15.121:6789/0 conn(0x7fea78139030 legacy=0x7fea780a2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:20:58.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.121+0000 7fea7659c640 10 monclient: _finish_hunting 0 2024-09-17T18:20:58.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.121+0000 7fea7659c640 1 monclient: found mon.noname-c 2024-09-17T18:20:58.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.121+0000 7fea7659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:20:58.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.121+0000 7fea7659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.173:3300/0 2024-09-17T18:20:58.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.121+0000 7fea7659c640 1 -- 172.21.15.121:0/4086309082 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea7809ee30 con 0x7fea780a1f20 2024-09-17T18:20:58.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.124+0000 7fea7659c640 1 -- 172.21.15.121:0/4086309082 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea780a1f20 msgr2=0x7fea781386a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:20:58.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.124+0000 7fea7659c640 1 --2- 172.21.15.121:0/4086309082 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea780a1f20 0x7fea781386a0 secure :-1 s=READY pgs=7081 cs=0 l=1 rev1=1 crypto rx=0x7fea680097a0 tx=0x7fea6802a050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e87234cda20ebd0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:20:58.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.125+0000 7fea75d9b640 1 -- 172.21.15.121:0/4086309082 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea68037070 con 0x7fea780a1f20 2024-09-17T18:20:58.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.125+0000 7fea75d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:20:58.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.125+0000 7fea75d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:20:58.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.125+0000 7fea75d9b640 10 monclient: dump: 2024-09-17T18:20:58.129 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:20:58.129 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:20:58.129 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:20:58.129 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:20:58.129 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:20:58.129 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:20:58.129 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:20:58.129 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:20:58.129 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:20:58.129 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:20:58.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.125+0000 7fea75d9b640 10 monclient: _finish_auth 0 2024-09-17T18:20:58.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.125+0000 7fea75d9b640 10 monclient: _check_auth_tickets 2024-09-17T18:20:58.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.125+0000 7fea75d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:20:58.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.125+0000 7fea75d9b640 1 -- 172.21.15.121:0/4086309082 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea68032640 con 0x7fea780a1f20 2024-09-17T18:20:58.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.125+0000 7fea75d9b640 10 monclient: handle_config config(0 keys) 2024-09-17T18:20:58.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.125+0000 7fea75d9b640 1 -- 172.21.15.121:0/4086309082 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea68032a10 con 0x7fea780a1f20 2024-09-17T18:20:58.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.125+0000 7fea7759e640 5 monclient: authenticate success, global_id 24109 2024-09-17T18:20:58.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.125+0000 7fea7759e640 10 monclient: get_monmap_and_config success 2024-09-17T18:20:58.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.125+0000 7fea7759e640 4 set_mon_vals no callback set 2024-09-17T18:20:58.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.125+0000 7fea75d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:20:58.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.125+0000 7fea75d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:20:58.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.125+0000 7fea75d9b640 10 monclient: dump: 2024-09-17T18:20:58.132 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:20:58.132 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:20:58.132 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:20:58.132 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:20:58.132 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:20:58.132 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:20:58.132 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:20:58.132 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:20:58.133 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:20:58.133 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:20:58.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.125+0000 7fea7759e640 10 monclient: shutdown 2024-09-17T18:20:58.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.125+0000 7fea7759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:20:58.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.125+0000 7fea7759e640 1 -- 172.21.15.121:0/4086309082 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea780a1f20 msgr2=0x7fea781386a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:20:58.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.125+0000 7fea7759e640 1 --2- 172.21.15.121:0/4086309082 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea780a1f20 0x7fea781386a0 secure :-1 s=READY pgs=7081 cs=0 l=1 rev1=1 crypto rx=0x7fea680097a0 tx=0x7fea6802a050 comp rx=0 tx=0).stop 2024-09-17T18:20:58.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.125+0000 7fea7759e640 1 -- 172.21.15.121:0/4086309082 shutdown_connections 2024-09-17T18:20:58.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.125+0000 7fea7759e640 1 --2- 172.21.15.121:0/4086309082 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea780a1f20 0x7fea781386a0 unknown :-1 s=CLOSED pgs=7081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:20:58.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.125+0000 7fea7759e640 1 -- 172.21.15.121:0/4086309082 >> 172.21.15.121:0/4086309082 conn(0x7fea780a7700 msgr2=0x7fea780a7b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:20:58.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.126+0000 7fea7759e640 1 -- 172.21.15.121:0/4086309082 shutdown_connections 2024-09-17T18:20:58.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.126+0000 7fea7759e640 1 -- 172.21.15.121:0/4086309082 wait complete. 2024-09-17T18:20:58.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.126+0000 7fea7759e640 10 monclient: build_initial_monmap 2024-09-17T18:20:58.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.126+0000 7fea7759e640 10 monclient: monmap: 2024-09-17T18:20:58.138 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:20:58.138 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:20:58.138 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:20:58.138 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:20:58.138 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:20:58.139 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:20:58.139 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:20:58.139 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:20:58.139 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:20:58.139 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:20:58.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.126+0000 7fea7759e640 1 librados: starting msgr at 2024-09-17T18:20:58.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.126+0000 7fea7759e640 1 librados: starting objecter 2024-09-17T18:20:58.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.127+0000 7fea7759e640 1 Processor -- start 2024-09-17T18:20:58.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.127+0000 7fea7759e640 1 -- start start 2024-09-17T18:20:58.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.127+0000 7fea7759e640 1 librados: setting wanted keys 2024-09-17T18:20:58.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.127+0000 7fea7759e640 1 librados: calling monclient init 2024-09-17T18:20:58.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.127+0000 7fea7759e640 10 monclient: init 2024-09-17T18:20:58.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.127+0000 7fea7759e640 10 monclient: _reopen_session rank -1 2024-09-17T18:20:58.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.127+0000 7fea7759e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T18:20:58.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.127+0000 7fea7759e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea780a1f20 0x7fea781cd930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:20:58.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.127+0000 7fea7759e640 10 monclient(hunting): picked mon.noname-b con 0x7fea780a1f20 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:20:58.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.127+0000 7fea7759e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea78138be0 0x7fea781cde70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:20:58.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.127+0000 7fea7759e640 10 monclient(hunting): picked mon.noname-c con 0x7fea78138be0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:20:58.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.127+0000 7fea7759e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea78139030 0x7fea781327b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:20:58.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.127+0000 7fea7759e640 10 monclient(hunting): picked mon.noname-a con 0x7fea78139030 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:20:58.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.127+0000 7fea7759e640 10 monclient(hunting): start opening mon connection 2024-09-17T18:20:58.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.127+0000 7fea7759e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fea780a3120 con 0x7fea78139030 2024-09-17T18:20:58.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.127+0000 7fea7759e640 10 monclient(hunting): start opening mon connection 2024-09-17T18:20:58.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.127+0000 7fea7759e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fea780749d0 con 0x7fea780a1f20 2024-09-17T18:20:58.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.127+0000 7fea7759e640 10 monclient(hunting): start opening mon connection 2024-09-17T18:20:58.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.127+0000 7fea7759e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fea7809f9f0 con 0x7fea78138be0 2024-09-17T18:20:58.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.127+0000 7fea7759e640 10 monclient(hunting): _renew_subs 2024-09-17T18:20:58.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.127+0000 7fea7759e640 10 monclient(hunting): authenticate will time out at 861.844773s 2024-09-17T18:20:58.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.127+0000 7fea76d9d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea78139030 0x7fea781327b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:20:58.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.127+0000 7fea6dd9b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea78138be0 0x7fea781cde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:20:58.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.127+0000 7fea76d9d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea78139030 0x7fea781327b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36172/0 (socket says 172.21.15.121:36172) 2024-09-17T18:20:58.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.127+0000 7fea6dd9b640 10 monclient(hunting): get_auth_request con 0x7fea78138be0 auth_method 0 2024-09-17T18:20:58.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.127+0000 7fea76d9d640 1 -- 172.21.15.121:0/437792882 learned_addr learned my addr 172.21.15.121:0/437792882 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:20:58.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.127+0000 7fea6dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:20:58.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.127+0000 7fea6dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:20:58.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.127+0000 7fea6dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:20:58.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.127+0000 7fea76d9d640 10 monclient(hunting): get_auth_request con 0x7fea78139030 auth_method 0 2024-09-17T18:20:58.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.127+0000 7fea76d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:20:58.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.127+0000 7fea76d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:20:58.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.127+0000 7fea76d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:20:58.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.128+0000 7fea6dd9b640 1 -- 172.21.15.121:0/437792882 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea78138be0 msgr2=0x7fea781cde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T18:20:58.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.128+0000 7fea6dd9b640 1 -- 172.21.15.121:0/437792882 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea78138be0 msgr2=0x7fea781cde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:20:58.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.128+0000 7fea76d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:20:58.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.128+0000 7fea76d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:20:58.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.128+0000 7fea6dd9b640 1 --2- 172.21.15.121:0/437792882 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea78138be0 0x7fea781cde70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:20:58.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.128+0000 7fea76d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:20:58.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.128+0000 7fea7659c640 1 --2- 172.21.15.121:0/437792882 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea780a1f20 0x7fea781cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:20:58.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.128+0000 7fea6dd9b640 1 --2- 172.21.15.121:0/437792882 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea78138be0 0x7fea781cde70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:20:58.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.128+0000 7fea7659c640 10 monclient(hunting): get_auth_request con 0x7fea780a1f20 auth_method 0 2024-09-17T18:20:58.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.128+0000 7fea7659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:20:58.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.128+0000 7fea7659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:20:58.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.128+0000 7fea7659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:20:58.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.128+0000 7fea7659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:20:58.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.128+0000 7fea7659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:20:58.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.128+0000 7fea7659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:20:58.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.128+0000 7fea76d9d640 10 monclient(hunting): handle_auth_done global_id 23667 payload 824 2024-09-17T18:20:58.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.128+0000 7fea76d9d640 1 -- 172.21.15.121:0/437792882 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea78138be0 msgr2=0x7fea781cde70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:20:58.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.128+0000 7fea76d9d640 1 --2- 172.21.15.121:0/437792882 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea78138be0 0x7fea781cde70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:20:58.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.128+0000 7fea76d9d640 1 -- 172.21.15.121:0/437792882 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea780a1f20 msgr2=0x7fea781cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:20:58.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.128+0000 7fea76d9d640 1 --2- 172.21.15.121:0/437792882 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea780a1f20 0x7fea781cd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:20:58.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.128+0000 7fea76d9d640 10 monclient: _finish_hunting 0 2024-09-17T18:20:58.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.128+0000 7fea76d9d640 1 monclient: found mon.noname-a 2024-09-17T18:20:58.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.128+0000 7fea76d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:20:58.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.128+0000 7fea76d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:20:58.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.128+0000 7fea76d9d640 1 -- 172.21.15.121:0/437792882 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea78136340 con 0x7fea78139030 2024-09-17T18:20:58.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.130+0000 7fea76d9d640 1 -- 172.21.15.121:0/437792882 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea78139030 msgr2=0x7fea781327b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:20:58.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.130+0000 7fea76d9d640 1 --2- 172.21.15.121:0/437792882 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea78139030 0x7fea781327b0 secure :-1 s=READY pgs=6799 cs=0 l=1 rev1=1 crypto rx=0x7fea7000caf0 tx=0x7fea7000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:20:58.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.130+0000 7fea6f7fe640 10 client.?.objecter ms_handle_connect 0x7fea78139030 2024-09-17T18:20:58.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.130+0000 7fea6f7fe640 10 client.?.objecter resend_mon_ops 2024-09-17T18:20:58.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.130+0000 7fea6f7fe640 1 -- 172.21.15.121:0/437792882 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea70012400 con 0x7fea78139030 2024-09-17T18:20:58.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.130+0000 7fea6f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:20:58.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.130+0000 7fea6f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:20:58.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.130+0000 7fea6f7fe640 10 monclient: dump: 2024-09-17T18:20:58.144 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:20:58.144 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:20:58.144 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:20:58.144 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:20:58.144 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:20:58.145 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:20:58.145 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:20:58.145 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:20:58.145 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:20:58.145 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:20:58.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.130+0000 7fea6f7fe640 10 monclient: _finish_auth 0 2024-09-17T18:20:58.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.130+0000 7fea6f7fe640 10 monclient: _check_auth_tickets 2024-09-17T18:20:58.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.130+0000 7fea6f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:20:58.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.130+0000 7fea7759e640 5 monclient: authenticate success, global_id 23667 2024-09-17T18:20:58.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.130+0000 7fea6f7fe640 1 -- 172.21.15.121:0/437792882 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea700125c0 con 0x7fea78139030 2024-09-17T18:20:58.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.130+0000 7fea7759e640 10 monclient: _renew_subs 2024-09-17T18:20:58.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.130+0000 7fea7759e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:20:58.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.130+0000 7fea7759e640 1 -- 172.21.15.121:0/437792882 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea7809e710 con 0x7fea78139030 2024-09-17T18:20:58.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.130+0000 7fea6f7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:20:58.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.130+0000 7fea6f7fe640 1 -- 172.21.15.121:0/437792882 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea700129b0 con 0x7fea78139030 2024-09-17T18:20:58.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.130+0000 7fea6f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:20:58.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.130+0000 7fea6f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:20:58.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.130+0000 7fea6f7fe640 10 monclient: dump: 2024-09-17T18:20:58.146 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:20:58.146 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:20:58.146 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:20:58.146 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:20:58.146 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:20:58.146 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:20:58.146 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:20:58.146 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:20:58.146 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:20:58.146 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:20:58.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.130+0000 7fea7759e640 10 client.23667.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:20:58.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.130+0000 7fea7559a640 4 set_mon_vals no callback set 2024-09-17T18:20:58.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.130+0000 7fea7759e640 10 monclient: _renew_subs 2024-09-17T18:20:58.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.130+0000 7fea7759e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:20:58.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.130+0000 7fea7759e640 1 -- 172.21.15.121:0/437792882 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea78134b40 con 0x7fea78139030 2024-09-17T18:20:58.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.130+0000 7fea7759e640 1 librados: init done 2024-09-17T18:20:58.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.131+0000 7fea6f7fe640 1 -- 172.21.15.121:0/437792882 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fea70012bd0 con 0x7fea78139030 2024-09-17T18:20:58.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.131+0000 7fea7759e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:20:58.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.131+0000 7fea7759e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:20:58.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.131+0000 7fea7759e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:20:58.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.131+0000 7fea7759e640 1 -- 172.21.15.121:0/437792882 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea7806d7a0 con 0x7fea78139030 2024-09-17T18:20:58.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.131+0000 7fea6f7fe640 1 --2- 172.21.15.121:0/437792882 >> v2:172.21.15.173:6812/2423786037 conn(0x7fea58040170 0x7fea58042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:20:58.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.132+0000 7fea6f7fe640 1 -- 172.21.15.121:0/437792882 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7339+0+0 (secure 0 0 0) 0x7fea70054540 con 0x7fea78139030 2024-09-17T18:20:58.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.132+0000 7fea6f7fe640 10 client.23667.objecter ms_dispatch 0x7fea78000b90 osd_map(74..74 src has 1..74) 2024-09-17T18:20:58.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.132+0000 7fea6f7fe640 3 client.23667.objecter handle_osd_map got epochs [74,74] > 0 2024-09-17T18:20:58.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.132+0000 7fea6f7fe640 3 client.23667.objecter handle_osd_map decoding full epoch 74 2024-09-17T18:20:58.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.132+0000 7fea6f7fe640 20 client.23667.objecter dump_active .. 0 homeless 2024-09-17T18:20:58.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.132+0000 7fea7659c640 1 --2- 172.21.15.121:0/437792882 >> v2:172.21.15.173:6812/2423786037 conn(0x7fea58040170 0x7fea58042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:20:58.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.132+0000 7fea7659c640 10 monclient: get_auth_request con 0x7fea58040170 auth_method 0 2024-09-17T18:20:58.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.134+0000 7fea6f7fe640 10 client.23667.objecter ms_handle_connect 0x7fea58040170 2024-09-17T18:20:58.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.134+0000 7fea7659c640 1 --2- 172.21.15.121:0/437792882 >> v2:172.21.15.173:6812/2423786037 conn(0x7fea58040170 0x7fea58042630 secure :-1 s=READY pgs=3855 cs=0 l=1 rev1=1 crypto rx=0x7fea780a3d00 tx=0x7fea6802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:20:58.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.136+0000 7fea6f7fe640 1 -- 172.21.15.121:0/437792882 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fea70052540 con 0x7fea78139030 2024-09-17T18:20:58.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.136+0000 7fea6f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:20:58.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.136+0000 7fea6f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:20:58.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.410+0000 7fea7759e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:20:58.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.410+0000 7fea7759e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:20:58.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.410+0000 7fea7759e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:20:58.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.410+0000 7fea7759e640 1 -- 172.21.15.121:0/437792882 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fea7809dd50 con 0x7fea78139030 2024-09-17T18:20:58.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.410+0000 7fea6f7fe640 1 -- 172.21.15.121:0/437792882 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fea700566e0 con 0x7fea78139030 2024-09-17T18:20:58.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.410+0000 7fea6f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:20:58.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.410+0000 7fea6f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:20:58.414 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:20:58.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.413+0000 7fea7759e640 10 librados: watch_flush enter 2024-09-17T18:20:58.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.413+0000 7fea7759e640 10 librados: watch_flush exit 2024-09-17T18:20:58.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.413+0000 7fea7759e640 20 client.23667.objecter shutdown clearing up homeless session... 2024-09-17T18:20:58.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.413+0000 7fea7759e640 10 client.23667.objecter successfully canceled tick 2024-09-17T18:20:58.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.413+0000 7fea7759e640 1 -- 172.21.15.121:0/437792882 >> v2:172.21.15.173:6812/2423786037 conn(0x7fea58040170 msgr2=0x7fea58042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:20:58.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.413+0000 7fea7759e640 1 --2- 172.21.15.121:0/437792882 >> v2:172.21.15.173:6812/2423786037 conn(0x7fea58040170 0x7fea58042630 secure :-1 s=READY pgs=3855 cs=0 l=1 rev1=1 crypto rx=0x7fea780a3d00 tx=0x7fea6802e000 comp rx=0 tx=0).stop 2024-09-17T18:20:58.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.413+0000 7fea7759e640 10 monclient: shutdown 2024-09-17T18:20:58.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.413+0000 7fea7759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:20:58.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.413+0000 7fea7759e640 1 -- 172.21.15.121:0/437792882 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea78139030 msgr2=0x7fea781327b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:20:58.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.413+0000 7fea7759e640 1 --2- 172.21.15.121:0/437792882 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea78139030 0x7fea781327b0 secure :-1 s=READY pgs=6799 cs=0 l=1 rev1=1 crypto rx=0x7fea7000caf0 tx=0x7fea7000a060 comp rx=0 tx=0).stop 2024-09-17T18:20:58.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.413+0000 7fea7759e640 1 -- 172.21.15.121:0/437792882 shutdown_connections 2024-09-17T18:20:58.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.413+0000 7fea7759e640 1 --2- 172.21.15.121:0/437792882 >> v2:172.21.15.173:6812/2423786037 conn(0x7fea58040170 0x7fea58042630 unknown :-1 s=CLOSED pgs=3855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:20:58.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.413+0000 7fea7759e640 1 --2- 172.21.15.121:0/437792882 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea78139030 0x7fea781327b0 unknown :-1 s=CLOSED pgs=6799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:20:58.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.413+0000 7fea7759e640 1 --2- 172.21.15.121:0/437792882 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea78138be0 0x7fea781cde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:20:58.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.413+0000 7fea7759e640 1 --2- 172.21.15.121:0/437792882 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea780a1f20 0x7fea781cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:20:58.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.413+0000 7fea7759e640 1 -- 172.21.15.121:0/437792882 >> 172.21.15.121:0/437792882 conn(0x7fea780a7700 msgr2=0x7fea7809fc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:20:58.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.413+0000 7fea7759e640 1 -- 172.21.15.121:0/437792882 shutdown_connections 2024-09-17T18:20:58.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.413+0000 7fea7759e640 1 -- 172.21.15.121:0/437792882 wait complete. 2024-09-17T18:20:58.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.413+0000 7fea7759e640 1 librados: shutdown 2024-09-17T18:20:58.426 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T18:20:58.426 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T18:20:58.426 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T18:20:58.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.521+0000 7f4d00d52640 10 monclient: get_monmap_and_config 2024-09-17T18:20:58.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.521+0000 7f4d00d52640 10 monclient: build_initial_monmap 2024-09-17T18:20:58.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.521+0000 7f4d00d52640 10 monclient: monmap: 2024-09-17T18:20:58.524 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:20:58.524 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:20:58.524 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:20:58.523014+0000 2024-09-17T18:20:58.524 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:20:58.523014+0000 2024-09-17T18:20:58.524 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:20:58.524 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:20:58.525 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:20:58.525 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:20:58.525 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:20:58.525 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:20:58.525 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:20:58.525 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:20:58.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.521+0000 7f4d00d52640 1 Processor -- start 2024-09-17T18:20:58.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.522+0000 7f4d00d52640 1 -- start start 2024-09-17T18:20:58.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.522+0000 7f4d00d52640 10 monclient: init 2024-09-17T18:20:58.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.522+0000 7f4d00d52640 10 monclient: _reopen_session rank -1 2024-09-17T18:20:58.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.522+0000 7f4d00d52640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-17T18:20:58.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.522+0000 7f4d00d52640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4cfc073900 addr v1:172.21.15.173:6789/0 2024-09-17T18:20:58.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.522+0000 7f4d00d52640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4cfc13b6f0 0x7f4cfc13dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:20:58.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.522+0000 7f4d00d52640 10 monclient(hunting): picked mon.noname-a con 0x7f4cfc13b6f0 addr v2:172.21.15.121:3300/0 2024-09-17T18:20:58.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.522+0000 7f4d00d52640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4cfc13e020 addr v1:172.21.15.121:6789/0 2024-09-17T18:20:58.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.522+0000 7f4d00d52640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cfc1388b0 con 0x7f4cfc13e020 2024-09-17T18:20:58.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.522+0000 7f4d00d52640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cfc0701a0 con 0x7f4cfc073900 2024-09-17T18:20:58.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.522+0000 7f4d00d52640 10 monclient(hunting): start opening mon connection 2024-09-17T18:20:58.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.522+0000 7f4d00d52640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f4cfc06fbe0 con 0x7f4cfc13b6f0 2024-09-17T18:20:58.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.522+0000 7f4d00d52640 10 monclient(hunting): _renew_subs 2024-09-17T18:20:58.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.522+0000 7f4d00d52640 10 monclient(hunting): authenticate will time out at 862.239698s 2024-09-17T18:20:58.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.522+0000 7f4cf9d9b640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4cfc13b6f0 0x7f4cfc13dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:20:58.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.522+0000 7f4cf9d9b640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4cfc13b6f0 0x7f4cfc13dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36176/0 (socket says 172.21.15.121:36176) 2024-09-17T18:20:58.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.522+0000 7f4cf9d9b640 1 -- 172.21.15.121:0/455322649 learned_addr learned my addr 172.21.15.121:0/455322649 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:20:58.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.522+0000 7f4cf9d9b640 10 monclient(hunting): get_auth_request con 0x7f4cfc13b6f0 auth_method 0 2024-09-17T18:20:58.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.522+0000 7f4cf9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:20:58.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.522+0000 7f4cf9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:20:58.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.522+0000 7f4cf9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:20:58.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.525+0000 7f4cfad9d640 1 -- 172.21.15.121:0/455322649 >> v1:172.21.15.121:6789/0 conn(0x7f4cfc13e020 legacy=0x7f4cfc140410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:20:58.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.525+0000 7f4cf9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:20:58.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.525+0000 7f4cf9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:20:58.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.525+0000 7f4cf9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:20:58.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.525+0000 7f4cfa59c640 1 -- 172.21.15.121:0/455322649 >> v1:172.21.15.173:6789/0 conn(0x7f4cfc073900 legacy=0x7f4cfc073ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:20:58.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.525+0000 7f4cf9d9b640 10 monclient(hunting): handle_auth_done global_id 23685 payload 274 2024-09-17T18:20:58.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.525+0000 7f4cf959a640 1 -- 172.21.15.121:0/455322649 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1277002318 0 0) 0x7f4cfc0701a0 con 0x7f4cfc073900 2024-09-17T18:20:58.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.525+0000 7f4cf9d9b640 1 -- 172.21.15.121:0/455322649 >> v1:172.21.15.173:6789/0 conn(0x7f4cfc073900 legacy=0x7f4cfc073ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:20:58.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.525+0000 7f4cf9d9b640 1 -- 172.21.15.121:0/455322649 >> v1:172.21.15.121:6789/0 conn(0x7f4cfc13e020 legacy=0x7f4cfc140410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:20:58.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.525+0000 7f4cf9d9b640 10 monclient: _finish_hunting 0 2024-09-17T18:20:58.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.525+0000 7f4cf9d9b640 1 monclient: found mon.noname-a 2024-09-17T18:20:58.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.525+0000 7f4cf9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:20:58.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.525+0000 7f4cf9d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:20:58.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.525+0000 7f4cf9d9b640 1 -- 172.21.15.121:0/455322649 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cfc077290 con 0x7f4cfc13b6f0 2024-09-17T18:20:58.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.525+0000 7f4cf959a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-17T18:20:58.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.528+0000 7f4cf9d9b640 1 -- 172.21.15.121:0/455322649 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4cfc13b6f0 msgr2=0x7f4cfc13dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:20:58.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.528+0000 7f4cf9d9b640 1 --2- 172.21.15.121:0/455322649 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4cfc13b6f0 0x7f4cfc13dae0 secure :-1 s=READY pgs=6808 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4009970 tx=0x7f4ce402a2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4dccb67e3ec1cd6 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:20:58.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.528+0000 7f4cf959a640 1 -- 172.21.15.121:0/455322649 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce4037070 con 0x7f4cfc13b6f0 2024-09-17T18:20:58.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.528+0000 7f4cf959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:20:58.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.528+0000 7f4cf959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:20:58.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.528+0000 7f4cf959a640 10 monclient: dump: 2024-09-17T18:20:58.533 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:20:58.533 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:20:58.533 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:20:58.533 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:20:58.533 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:20:58.533 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:20:58.533 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:20:58.533 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:20:58.534 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:20:58.534 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:20:58.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.528+0000 7f4cf959a640 10 monclient: _finish_auth 0 2024-09-17T18:20:58.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.528+0000 7f4cf959a640 10 monclient: _check_auth_tickets 2024-09-17T18:20:58.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.528+0000 7f4cf959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:20:58.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.528+0000 7f4d00d52640 5 monclient: authenticate success, global_id 23685 2024-09-17T18:20:58.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.528+0000 7f4d00d52640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:20:58.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.528+0000 7f4cf959a640 1 -- 172.21.15.121:0/455322649 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ce4032640 con 0x7f4cfc13b6f0 2024-09-17T18:20:58.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.528+0000 7f4cf959a640 10 monclient: handle_config config(0 keys) 2024-09-17T18:20:58.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.528+0000 7f4d00d52640 10 monclient: get_monmap_and_config success 2024-09-17T18:20:58.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.528+0000 7f4d00d52640 4 set_mon_vals no callback set 2024-09-17T18:20:58.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.528+0000 7f4cf959a640 1 -- 172.21.15.121:0/455322649 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce4032a10 con 0x7f4cfc13b6f0 2024-09-17T18:20:58.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.528+0000 7f4cf959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:20:58.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.528+0000 7f4cf959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:20:58.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.528+0000 7f4cf959a640 10 monclient: dump: 2024-09-17T18:20:58.535 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:20:58.535 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:20:58.535 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:20:58.535 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:20:58.535 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:20:58.535 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:20:58.535 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:20:58.535 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:20:58.535 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:20:58.535 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:20:58.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.528+0000 7f4d00d52640 10 monclient: shutdown 2024-09-17T18:20:58.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.528+0000 7f4d00d52640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:20:58.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.528+0000 7f4d00d52640 1 -- 172.21.15.121:0/455322649 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4cfc13b6f0 msgr2=0x7f4cfc13dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:20:58.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.528+0000 7f4d00d52640 1 --2- 172.21.15.121:0/455322649 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4cfc13b6f0 0x7f4cfc13dae0 secure :-1 s=READY pgs=6808 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4009970 tx=0x7f4ce402a2e0 comp rx=0 tx=0).stop 2024-09-17T18:20:58.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.529+0000 7f4d00d52640 1 -- 172.21.15.121:0/455322649 shutdown_connections 2024-09-17T18:20:58.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.529+0000 7f4d00d52640 1 --2- 172.21.15.121:0/455322649 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4cfc13b6f0 0x7f4cfc13dae0 unknown :-1 s=CLOSED pgs=6808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:20:58.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.529+0000 7f4d00d52640 1 -- 172.21.15.121:0/455322649 >> 172.21.15.121:0/455322649 conn(0x7f4cfc0a49c0 msgr2=0x7f4cfc0768d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:20:58.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.529+0000 7f4d00d52640 1 -- 172.21.15.121:0/455322649 shutdown_connections 2024-09-17T18:20:58.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.529+0000 7f4d00d52640 1 -- 172.21.15.121:0/455322649 wait complete. 2024-09-17T18:20:58.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.529+0000 7f4d00d52640 10 monclient: build_initial_monmap 2024-09-17T18:20:58.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.529+0000 7f4d00d52640 10 monclient: monmap: 2024-09-17T18:20:58.543 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:20:58.543 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:20:58.543 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:20:58.543 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:20:58.543 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:20:58.543 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:20:58.543 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:20:58.543 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:20:58.543 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:20:58.543 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:20:58.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4d00d52640 1 librados: starting msgr at 2024-09-17T18:20:58.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4d00d52640 1 librados: starting objecter 2024-09-17T18:20:58.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4d00d52640 1 Processor -- start 2024-09-17T18:20:58.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4d00d52640 1 -- start start 2024-09-17T18:20:58.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4d00d52640 1 librados: setting wanted keys 2024-09-17T18:20:58.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4d00d52640 1 librados: calling monclient init 2024-09-17T18:20:58.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4d00d52640 10 monclient: init 2024-09-17T18:20:58.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4d00d52640 10 monclient: _reopen_session rank -1 2024-09-17T18:20:58.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4d00d52640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T18:20:58.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4d00d52640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4cfc073900 0x7f4cfc120330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:20:58.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4d00d52640 10 monclient(hunting): picked mon.noname-c con 0x7f4cfc073900 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:20:58.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4d00d52640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4cfc13b6f0 0x7f4cfc11e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:20:58.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4d00d52640 10 monclient(hunting): picked mon.noname-a con 0x7f4cfc13b6f0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:20:58.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4d00d52640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4cfc13e020 0x7f4cfc135180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:20:58.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4d00d52640 10 monclient(hunting): picked mon.noname-b con 0x7f4cfc13e020 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:20:58.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4d00d52640 10 monclient(hunting): start opening mon connection 2024-09-17T18:20:58.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4d00d52640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4cfc077770 con 0x7f4cfc13b6f0 2024-09-17T18:20:58.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4d00d52640 10 monclient(hunting): start opening mon connection 2024-09-17T18:20:58.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4d00d52640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4cfc0775f0 con 0x7f4cfc13e020 2024-09-17T18:20:58.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4d00d52640 10 monclient(hunting): start opening mon connection 2024-09-17T18:20:58.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4d00d52640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4cfc074450 con 0x7f4cfc073900 2024-09-17T18:20:58.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4d00d52640 10 monclient(hunting): _renew_subs 2024-09-17T18:20:58.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4d00d52640 10 monclient(hunting): authenticate will time out at 862.247632s 2024-09-17T18:20:58.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4cfa59c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4cfc073900 0x7f4cfc120330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:20:58.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4cf9d9b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4cfc13b6f0 0x7f4cfc11e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:20:58.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4cf9d9b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4cfc13b6f0 0x7f4cfc11e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36190/0 (socket says 172.21.15.121:36190) 2024-09-17T18:20:58.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4cfa59c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4cfc073900 0x7f4cfc120330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:45286/0 (socket says 172.21.15.121:45286) 2024-09-17T18:20:58.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4cf9d9b640 1 -- 172.21.15.121:0/835651127 learned_addr learned my addr 172.21.15.121:0/835651127 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:20:58.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4cf9d9b640 10 monclient(hunting): get_auth_request con 0x7f4cfc13b6f0 auth_method 0 2024-09-17T18:20:58.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4cf9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:20:58.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4cf9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:20:58.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4cf9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:20:58.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4cfa59c640 10 monclient(hunting): get_auth_request con 0x7f4cfc073900 auth_method 0 2024-09-17T18:20:58.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4cfa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:20:58.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4cfa59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:20:58.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4cfa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:20:58.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4cf9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:20:58.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4cf9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:20:58.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4cf9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:20:58.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4cfa59c640 1 -- 172.21.15.121:0/835651127 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4cfc073900 msgr2=0x7f4cfc120330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T18:20:58.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4cfa59c640 1 -- 172.21.15.121:0/835651127 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4cfc073900 msgr2=0x7f4cfc120330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:20:58.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4cfa59c640 1 --2- 172.21.15.121:0/835651127 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4cfc073900 0x7f4cfc120330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:20:58.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4cfa59c640 1 --2- 172.21.15.121:0/835651127 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4cfc073900 0x7f4cfc120330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:20:58.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4cfad9d640 1 --2- 172.21.15.121:0/835651127 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4cfc13e020 0x7f4cfc135180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:20:58.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4cfad9d640 10 monclient(hunting): get_auth_request con 0x7f4cfc13e020 auth_method 0 2024-09-17T18:20:58.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4cfad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:20:58.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4cfad9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:20:58.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4cfad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:20:58.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4cfad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:20:58.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4cfad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:20:58.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4cfad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:20:58.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.530+0000 7f4cf9d9b640 10 monclient(hunting): handle_auth_done global_id 23691 payload 824 2024-09-17T18:20:58.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.531+0000 7f4cf9d9b640 1 -- 172.21.15.121:0/835651127 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4cfc073900 msgr2=0x7f4cfc120330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:20:58.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.531+0000 7f4cf9d9b640 1 --2- 172.21.15.121:0/835651127 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4cfc073900 0x7f4cfc120330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:20:58.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.531+0000 7f4cf9d9b640 1 -- 172.21.15.121:0/835651127 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4cfc13e020 msgr2=0x7f4cfc135180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:20:58.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.531+0000 7f4cf9d9b640 1 --2- 172.21.15.121:0/835651127 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4cfc13e020 0x7f4cfc135180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:20:58.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.531+0000 7f4cf9d9b640 10 monclient: _finish_hunting 0 2024-09-17T18:20:58.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.531+0000 7f4cf9d9b640 1 monclient: found mon.noname-a 2024-09-17T18:20:58.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.531+0000 7f4cf9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:20:58.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.531+0000 7f4cf9d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:20:58.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.531+0000 7f4cf9d9b640 1 -- 172.21.15.121:0/835651127 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cfc11f7f0 con 0x7f4cfc13b6f0 2024-09-17T18:20:58.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.533+0000 7f4cf9d9b640 1 -- 172.21.15.121:0/835651127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4cfc13b6f0 msgr2=0x7f4cfc11e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:20:58.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.533+0000 7f4cf9d9b640 1 --2- 172.21.15.121:0/835651127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4cfc13b6f0 0x7f4cfc11e340 secure :-1 s=READY pgs=6809 cs=0 l=1 rev1=1 crypto rx=0x7f4ce40090b0 tx=0x7f4ce40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:20:58.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.533+0000 7f4cf2ffd640 10 client.?.objecter ms_handle_connect 0x7f4cfc13b6f0 2024-09-17T18:20:58.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.533+0000 7f4cf2ffd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:20:58.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.533+0000 7f4cf2ffd640 1 -- 172.21.15.121:0/835651127 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce4041020 con 0x7f4cfc13b6f0 2024-09-17T18:20:58.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.533+0000 7f4cf2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:20:58.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.533+0000 7f4cf2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:20:58.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.533+0000 7f4cf2ffd640 10 monclient: dump: 2024-09-17T18:20:58.548 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:20:58.549 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:20:58.549 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:20:58.549 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:20:58.549 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:20:58.549 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:20:58.549 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:20:58.549 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:20:58.549 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:20:58.549 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:20:58.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.533+0000 7f4cf2ffd640 10 monclient: _finish_auth 0 2024-09-17T18:20:58.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.533+0000 7f4cf2ffd640 10 monclient: _check_auth_tickets 2024-09-17T18:20:58.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.533+0000 7f4cf2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:20:58.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.533+0000 7f4cf2ffd640 1 -- 172.21.15.121:0/835651127 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ce402f980 con 0x7f4cfc13b6f0 2024-09-17T18:20:58.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.533+0000 7f4d00d52640 5 monclient: authenticate success, global_id 23691 2024-09-17T18:20:58.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.533+0000 7f4d00d52640 10 monclient: _renew_subs 2024-09-17T18:20:58.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.533+0000 7f4d00d52640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:20:58.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.533+0000 7f4d00d52640 1 -- 172.21.15.121:0/835651127 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cfc0a3540 con 0x7f4cfc13b6f0 2024-09-17T18:20:58.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.533+0000 7f4cf2ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:20:58.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.533+0000 7f4cf2ffd640 1 -- 172.21.15.121:0/835651127 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce4037040 con 0x7f4cfc13b6f0 2024-09-17T18:20:58.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.533+0000 7f4cf2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:20:58.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.533+0000 7f4cf2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:20:58.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.533+0000 7f4cf2ffd640 10 monclient: dump: 2024-09-17T18:20:58.550 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:20:58.550 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:20:58.550 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:20:58.550 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:20:58.550 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:20:58.550 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:20:58.551 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:20:58.551 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:20:58.551 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:20:58.551 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:20:58.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.533+0000 7f4cf8d99640 4 set_mon_vals no callback set 2024-09-17T18:20:58.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.533+0000 7f4d00d52640 10 client.23691.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:20:58.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.533+0000 7f4d00d52640 10 monclient: _renew_subs 2024-09-17T18:20:58.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.533+0000 7f4d00d52640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:20:58.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.533+0000 7f4d00d52640 1 -- 172.21.15.121:0/835651127 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cfc1356c0 con 0x7f4cfc13b6f0 2024-09-17T18:20:58.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.533+0000 7f4d00d52640 1 librados: init done 2024-09-17T18:20:58.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.534+0000 7f4d00d52640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:20:58.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.534+0000 7f4d00d52640 1 --2- 172.21.15.121:0/835651127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4cfc1cd320 0x7f4cfc1cd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:20:58.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.534+0000 7f4d00d52640 10 monclient(hunting): start opening mon connection 2024-09-17T18:20:58.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.534+0000 7f4d00d52640 1 -- 172.21.15.121:0/835651127 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4cfc073ce0 con 0x7f4cfc1cd320 2024-09-17T18:20:58.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.534+0000 7f4cfa59c640 1 --2- 172.21.15.121:0/835651127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4cfc1cd320 0x7f4cfc1cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:20:58.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.534+0000 7f4cfa59c640 10 monclient: get_auth_request con 0x7f4cfc1cd320 auth_method 0 2024-09-17T18:20:58.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.534+0000 7f4cfa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:20:58.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.534+0000 7f4cfa59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:20:58.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.534+0000 7f4cfa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:20:58.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.534+0000 7f4cfa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:20:58.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.534+0000 7f4cfa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:20:58.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.534+0000 7f4cfa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:20:58.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.534+0000 7f4cfa59c640 10 monclient(hunting): handle_auth_done global_id 23694 payload 824 2024-09-17T18:20:58.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.534+0000 7f4cfa59c640 1 -- 172.21.15.121:0/835651127 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4cfc0008d0 con 0x7f4cfc1cd320 2024-09-17T18:20:58.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.534+0000 7f4cf2ffd640 1 -- 172.21.15.121:0/835651127 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f4ce4044020 con 0x7f4cfc13b6f0 2024-09-17T18:20:58.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.534+0000 7f4cf2ffd640 1 --2- 172.21.15.121:0/835651127 >> v2:172.21.15.173:6812/2423786037 conn(0x7f4cdc040170 0x7f4cdc042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:20:58.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.534+0000 7f4cf2ffd640 1 -- 172.21.15.121:0/835651127 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 6984+0+0 (secure 0 0 0) 0x7f4ce40741e0 con 0x7f4cfc13b6f0 2024-09-17T18:20:58.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.534+0000 7f4cf2ffd640 10 client.23691.objecter ms_dispatch 0x7f4cfc000b90 osd_map(75..75 src has 1..75) 2024-09-17T18:20:58.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.534+0000 7f4cf2ffd640 3 client.23691.objecter handle_osd_map got epochs [75,75] > 0 2024-09-17T18:20:58.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.534+0000 7f4cf2ffd640 3 client.23691.objecter handle_osd_map decoding full epoch 75 2024-09-17T18:20:58.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.535+0000 7f4cf2ffd640 20 client.23691.objecter dump_active .. 0 homeless 2024-09-17T18:20:58.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.535+0000 7f4cfad9d640 1 --2- 172.21.15.121:0/835651127 >> v2:172.21.15.173:6812/2423786037 conn(0x7f4cdc040170 0x7f4cdc042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:20:58.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.535+0000 7f4cfad9d640 10 monclient: get_auth_request con 0x7f4cdc040170 auth_method 0 2024-09-17T18:20:58.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.536+0000 7f4cf2ffd640 10 client.23691.objecter ms_handle_connect 0x7f4cfc1cd320 2024-09-17T18:20:58.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.536+0000 7f4cf2ffd640 10 client.23691.objecter resend_mon_ops 2024-09-17T18:20:58.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.536+0000 7f4cfa59c640 1 -- 172.21.15.121:0/835651127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4cfc1cd320 msgr2=0x7f4cfc1cd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:20:58.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.536+0000 7f4cfa59c640 1 --2- 172.21.15.121:0/835651127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4cfc1cd320 0x7f4cfc1cd700 secure :-1 s=READY pgs=6810 cs=0 l=1 rev1=1 crypto rx=0x7f4cf40012e0 tx=0x7f4cf400ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:20:58.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.536+0000 7f4cf2ffd640 1 -- 172.21.15.121:0/835651127 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cf400ee70 con 0x7f4cfc1cd320 2024-09-17T18:20:58.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.536+0000 7f4cf2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:20:58.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.536+0000 7f4cf2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:20:58.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.536+0000 7f4cf2ffd640 10 monclient: dump: 2024-09-17T18:20:58.554 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:20:58.554 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:20:58.554 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:20:58.554 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:20:58.554 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:20:58.554 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:20:58.554 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:20:58.554 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:20:58.554 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:20:58.554 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:20:58.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.537+0000 7f4cf2ffd640 1 -- 172.21.15.121:0/835651127 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4cf4018070 con 0x7f4cfc1cd320 2024-09-17T18:20:58.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.537+0000 7f4cf2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:20:58.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.537+0000 7f4cf2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:20:58.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.537+0000 7f4cf2ffd640 1 -- 172.21.15.121:0/835651127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4cfc1cd320 msgr2=0x7f4cfc1cd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:20:58.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.537+0000 7f4cf2ffd640 1 --2- 172.21.15.121:0/835651127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4cfc1cd320 0x7f4cfc1cd700 secure :-1 s=READY pgs=6810 cs=0 l=1 rev1=1 crypto rx=0x7f4cf40012e0 tx=0x7f4cf400ec20 comp rx=0 tx=0).stop 2024-09-17T18:20:58.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.537+0000 7f4cf2ffd640 1 -- 172.21.15.121:0/835651127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4cfc1cd320 msgr2=0x7f4cfc1cd700 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:20:58.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.537+0000 7f4cf2ffd640 1 --2- 172.21.15.121:0/835651127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4cfc1cd320 0x7f4cfc1cd700 unknown :-1 s=CLOSED pgs=6810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:20:58.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.537+0000 7f4cf2ffd640 10 client.23691.objecter ms_handle_connect 0x7f4cdc040170 2024-09-17T18:20:58.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.537+0000 7f4cfad9d640 1 --2- 172.21.15.121:0/835651127 >> v2:172.21.15.173:6812/2423786037 conn(0x7f4cdc040170 0x7f4cdc042630 secure :-1 s=READY pgs=3857 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc11fc00 tx=0x7f4cec007c50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:20:58.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.543+0000 7f4d00d52640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:20:58.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.543+0000 7f4d00d52640 1 --2- 172.21.15.121:0/835651127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4cfc1cdc40 0x7f4cfc13bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:20:58.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.543+0000 7f4d00d52640 10 monclient(hunting): start opening mon connection 2024-09-17T18:20:58.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.543+0000 7f4d00d52640 1 -- 172.21.15.121:0/835651127 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4cfc1ce050 con 0x7f4cfc1cdc40 2024-09-17T18:20:58.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.543+0000 7f4cfa59c640 1 --2- 172.21.15.121:0/835651127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4cfc1cdc40 0x7f4cfc13bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:20:58.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.543+0000 7f4cfa59c640 10 monclient: get_auth_request con 0x7f4cfc1cdc40 auth_method 0 2024-09-17T18:20:58.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.543+0000 7f4cfa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:20:58.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.543+0000 7f4cfa59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:20:58.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.543+0000 7f4cfa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:20:58.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.543+0000 7f4cfa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:20:58.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.543+0000 7f4cfa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:20:58.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.543+0000 7f4cfa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:20:58.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.543+0000 7f4cfa59c640 10 monclient(hunting): handle_auth_done global_id 23697 payload 824 2024-09-17T18:20:58.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.543+0000 7f4cfa59c640 1 -- 172.21.15.121:0/835651127 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4cfc135950 con 0x7f4cfc1cdc40 2024-09-17T18:20:58.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.545+0000 7f4cf2ffd640 10 client.23691.objecter ms_handle_connect 0x7f4cfc1cdc40 2024-09-17T18:20:58.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.545+0000 7f4cf2ffd640 10 client.23691.objecter resend_mon_ops 2024-09-17T18:20:58.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.545+0000 7f4cfa59c640 1 -- 172.21.15.121:0/835651127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4cfc1cdc40 msgr2=0x7f4cfc13bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:20:58.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.545+0000 7f4cfa59c640 1 --2- 172.21.15.121:0/835651127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4cfc1cdc40 0x7f4cfc13bad0 secure :-1 s=READY pgs=6811 cs=0 l=1 rev1=1 crypto rx=0x7f4cf40012e0 tx=0x7f4cf4001790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:20:58.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.545+0000 7f4cf2ffd640 1 -- 172.21.15.121:0/835651127 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cf400ca80 con 0x7f4cfc1cdc40 2024-09-17T18:20:58.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.545+0000 7f4cf2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:20:58.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.545+0000 7f4cf2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:20:58.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.545+0000 7f4cf2ffd640 10 monclient: dump: 2024-09-17T18:20:58.558 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:20:58.558 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:20:58.558 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:20:58.558 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:20:58.559 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:20:58.559 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:20:58.559 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:20:58.559 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:20:58.559 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:20:58.559 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:20:58.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.545+0000 7f4cf2ffd640 1 -- 172.21.15.121:0/835651127 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7f4cf400ca80 con 0x7f4cfc1cdc40 2024-09-17T18:20:58.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.545+0000 7f4cf2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:20:58.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.545+0000 7f4cf2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:20:58.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.545+0000 7f4cf2ffd640 1 -- 172.21.15.121:0/835651127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4cfc1cdc40 msgr2=0x7f4cfc13bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:20:58.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.545+0000 7f4cf2ffd640 1 --2- 172.21.15.121:0/835651127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4cfc1cdc40 0x7f4cfc13bad0 secure :-1 s=READY pgs=6811 cs=0 l=1 rev1=1 crypto rx=0x7f4cf40012e0 tx=0x7f4cf4001790 comp rx=0 tx=0).stop 2024-09-17T18:20:58.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.545+0000 7f4cf2ffd640 1 -- 172.21.15.121:0/835651127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4cfc1cdc40 msgr2=0x7f4cfc13bad0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:20:58.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.545+0000 7f4cf2ffd640 1 --2- 172.21.15.121:0/835651127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4cfc1cdc40 0x7f4cfc13bad0 unknown :-1 s=CLOSED pgs=6811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:20:58.560 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:20:58.560 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:20:58.560 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:20:58.560 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "leader", 2024-09-17T18:20:58.560 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 10, 2024-09-17T18:20:58.560 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 131531, 2024-09-17T18:20:58.560 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:20:58.560 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:20:58.560 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:20:58.560 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:20:58.560 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:20:58.561 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 2, 2024-09-17T18:20:58.561 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:20:58.561 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:20:58.561 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:20:58.561 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:20:58.561 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:20:58.561 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:20:58.561 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:20:58.561 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:20:58.561 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:20:58.561 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:20:58.561 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:20:58.561 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:20:58.562 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:20:58.562 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:20:58.562 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:20:58.562 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:20:58.562 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:20:58.562 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:20:58.562 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:20:58.562 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:20:58.562 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:20:58.562 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:20:58.562 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:20:58.562 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:20:58.562 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:20:58.562 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:20:58.563 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:20:58.563 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:20:58.563 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:20:58.563 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:20:58.563 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:20:58.563 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:20:58.563 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:20:58.563 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:20:58.563 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:20:58.563 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:20:58.563 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:20:58.563 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:20:58.563 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:20:58.563 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:20:58.564 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:20:58.564 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:20:58.564 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:20:58.564 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:20:58.564 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:20:58.564 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:20:58.564 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:20:58.564 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:20:58.564 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:20:58.564 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:20:58.564 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:20:58.564 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:20:58.564 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:20:58.564 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:20:58.565 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:20:58.565 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:20:58.565 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:20:58.565 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:20:58.565 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:20:58.565 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:20:58.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:20:58.565 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:20:58.565 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:20:58.565 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:20:58.566 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:20:58.566 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:20:58.566 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:20:58.566 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:20:58.566 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:20:58.566 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:20:58.566 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:20:58.566 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:20:58.566 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:20:58.566 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:20:58.566 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:20:58.566 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:20:58.566 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:20:58.567 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:20:58.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:20:58.567 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:20:58.567 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:20:58.567 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:20:58.567 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:20:58.567 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:20:58.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:20:58.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:20:58.567 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:20:58.567 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:20:58.567 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:20:58.567 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:20:58.568 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:20:58.568 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:20:58.568 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:20:58.568 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:20:58.568 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:20:58.568 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:20:58.568 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:20:58.568 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:20:58.568 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:20:58.568 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:20:58.568 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:20:58.568 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:20:58.568 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:20:58.569 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:20:58.569 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:20:58.569 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:20:58.569 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:20:58.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:20:58.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:20:58.569 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:20:58.569 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:20:58.569 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:20:58.569 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:20:58.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:20:58.569 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:20:58.569 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:20:58.569 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:20:58.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:20:58.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:20:58.570 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:20:58.570 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:20:58.570 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:20:58.570 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:20:58.570 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:20:58.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:20:58.570 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:20:58.570 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:20:58.570 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:20:58.570 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:20:58.570 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:20:58.570 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:20:58.571 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:20:58.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:20:58.571 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:20:58.571 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:20:58.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:20:58.571 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:20:58.571 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:20:58.571 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:20:58.571 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:20:58.571 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:20:58.571 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:20:58.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:20:58.571 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:20:58.571 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:20:58.571 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-17T18:20:58.572 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:20:58.572 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:20:58.572 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:20:58.572 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:20:58.572 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:20:58.572 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:20:58.572 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 3 2024-09-17T18:20:58.572 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:20:58.572 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:20:58.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:20:58.572 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:20:58.572 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:20:58.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.546+0000 7f4d00d52640 10 librados: watch_flush enter 2024-09-17T18:20:58.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.546+0000 7f4d00d52640 10 librados: watch_flush exit 2024-09-17T18:20:58.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.546+0000 7f4d00d52640 20 client.23691.objecter shutdown clearing up homeless session... 2024-09-17T18:20:58.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.546+0000 7f4d00d52640 10 client.23691.objecter successfully canceled tick 2024-09-17T18:20:58.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.546+0000 7f4d00d52640 1 -- 172.21.15.121:0/835651127 >> v2:172.21.15.173:6812/2423786037 conn(0x7f4cdc040170 msgr2=0x7f4cdc042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:20:58.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.546+0000 7f4d00d52640 1 --2- 172.21.15.121:0/835651127 >> v2:172.21.15.173:6812/2423786037 conn(0x7f4cdc040170 0x7f4cdc042630 secure :-1 s=READY pgs=3857 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc11fc00 tx=0x7f4cec007c50 comp rx=0 tx=0).stop 2024-09-17T18:20:58.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.546+0000 7f4d00d52640 10 monclient: shutdown 2024-09-17T18:20:58.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.546+0000 7f4d00d52640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:20:58.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.546+0000 7f4d00d52640 1 -- 172.21.15.121:0/835651127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4cfc13b6f0 msgr2=0x7f4cfc11e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:20:58.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.546+0000 7f4cfad9d640 1 -- 172.21.15.121:0/835651127 reap_dead start 2024-09-17T18:20:58.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.546+0000 7f4d00d52640 1 --2- 172.21.15.121:0/835651127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4cfc13b6f0 0x7f4cfc11e340 secure :-1 s=READY pgs=6809 cs=0 l=1 rev1=1 crypto rx=0x7f4ce40090b0 tx=0x7f4ce40046c0 comp rx=0 tx=0).stop 2024-09-17T18:20:58.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.546+0000 7f4d00d52640 1 -- 172.21.15.121:0/835651127 shutdown_connections 2024-09-17T18:20:58.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.546+0000 7f4d00d52640 1 --2- 172.21.15.121:0/835651127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4cfc13b6f0 0x7f4cfc11e340 unknown :-1 s=CLOSED pgs=6809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:20:58.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.546+0000 7f4d00d52640 1 -- 172.21.15.121:0/835651127 >> 172.21.15.121:0/835651127 conn(0x7f4cfc0a49c0 msgr2=0x7f4cfc137d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:20:58.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.546+0000 7f4d00d52640 1 -- 172.21.15.121:0/835651127 shutdown_connections 2024-09-17T18:20:58.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.546+0000 7f4d00d52640 1 -- 172.21.15.121:0/835651127 wait complete. 2024-09-17T18:20:58.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.546+0000 7f4d00d52640 1 librados: shutdown 2024-09-17T18:20:58.576 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:20:58.576 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T18:20:58.576 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T18:20:58.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.671+0000 7fa5764b8640 10 monclient: get_monmap_and_config 2024-09-17T18:20:58.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.671+0000 7fa5764b8640 10 monclient: build_initial_monmap 2024-09-17T18:20:58.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.671+0000 7fa5764b8640 10 monclient: monmap: 2024-09-17T18:20:58.674 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:20:58.674 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:20:58.674 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:20:58.673229+0000 2024-09-17T18:20:58.674 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:20:58.673229+0000 2024-09-17T18:20:58.675 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:20:58.675 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:20:58.675 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:20:58.675 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:20:58.675 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:20:58.675 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:20:58.675 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:20:58.675 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:20:58.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.672+0000 7fa5764b8640 1 Processor -- start 2024-09-17T18:20:58.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.672+0000 7fa5764b8640 1 -- start start 2024-09-17T18:20:58.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.672+0000 7fa5764b8640 10 monclient: init 2024-09-17T18:20:58.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.672+0000 7fa5764b8640 10 monclient: _reopen_session rank -1 2024-09-17T18:20:58.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.672+0000 7fa5764b8640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-17T18:20:58.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.672+0000 7fa5764b8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa57009da70 addr v1:172.21.15.121:6789/0 2024-09-17T18:20:58.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.672+0000 7fa5764b8640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa5701368d0 0x7fa57013b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:20:58.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.672+0000 7fa5764b8640 10 monclient(hunting): picked mon.noname-a con 0x7fa5701368d0 addr v2:172.21.15.121:3300/0 2024-09-17T18:20:58.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.672+0000 7fa5764b8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa570136d00 addr v1:172.21.15.173:6789/0 2024-09-17T18:20:58.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.672+0000 7fa5764b8640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5700a2d40 con 0x7fa57009da70 2024-09-17T18:20:58.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.672+0000 7fa5764b8640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5700a3020 con 0x7fa570136d00 2024-09-17T18:20:58.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.672+0000 7fa5764b8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:20:58.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.672+0000 7fa5764b8640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fa57009de50 con 0x7fa5701368d0 2024-09-17T18:20:58.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.672+0000 7fa5764b8640 10 monclient(hunting): _renew_subs 2024-09-17T18:20:58.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.672+0000 7fa5764b8640 10 monclient(hunting): authenticate will time out at 862.389952s 2024-09-17T18:20:58.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.673+0000 7fa56f7fe640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa5701368d0 0x7fa57013b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:20:58.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.673+0000 7fa56f7fe640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa5701368d0 0x7fa57013b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36212/0 (socket says 172.21.15.121:36212) 2024-09-17T18:20:58.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.673+0000 7fa56f7fe640 1 -- 172.21.15.121:0/2675923583 learned_addr learned my addr 172.21.15.121:0/2675923583 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:20:58.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.673+0000 7fa56f7fe640 10 monclient(hunting): get_auth_request con 0x7fa5701368d0 auth_method 0 2024-09-17T18:20:58.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.673+0000 7fa56f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:20:58.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.673+0000 7fa56f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:20:58.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.673+0000 7fa56f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:20:58.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.673+0000 7fa56ffff640 1 --1- 172.21.15.121:0/2675923583 >> v1:172.21.15.121:6789/0 conn(0x7fa57009da70 0x7fa5701361c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:33356/0 (socket says 172.21.15.121:33356) 2024-09-17T18:20:58.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.673+0000 7fa56f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:20:58.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.673+0000 7fa56f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:20:58.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.673+0000 7fa56f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:20:58.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.675+0000 7fa56ffff640 1 -- 172.21.15.121:0/2675923583 >> v1:172.21.15.121:6789/0 conn(0x7fa57009da70 legacy=0x7fa5701361c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:20:58.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.675+0000 7fa56f7fe640 10 monclient(hunting): handle_auth_done global_id 23700 payload 274 2024-09-17T18:20:58.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.675+0000 7fa56f7fe640 1 -- 172.21.15.121:0/2675923583 >> v1:172.21.15.173:6789/0 conn(0x7fa570136d00 legacy=0x7fa57013da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:20:58.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.675+0000 7fa56f7fe640 1 -- 172.21.15.121:0/2675923583 >> v1:172.21.15.121:6789/0 conn(0x7fa57009da70 legacy=0x7fa5701361c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:20:58.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.675+0000 7fa56f7fe640 10 monclient: _finish_hunting 0 2024-09-17T18:20:58.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.675+0000 7fa56f7fe640 1 monclient: found mon.noname-a 2024-09-17T18:20:58.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.675+0000 7fa56f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:20:58.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.675+0000 7fa56f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:20:58.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.675+0000 7fa56f7fe640 1 -- 172.21.15.121:0/2675923583 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5700a3300 con 0x7fa5701368d0 2024-09-17T18:20:58.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.678+0000 7fa56f7fe640 1 -- 172.21.15.121:0/2675923583 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5701368d0 msgr2=0x7fa57013b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:20:58.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.678+0000 7fa56f7fe640 1 --2- 172.21.15.121:0/2675923583 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5701368d0 0x7fa57013b540 secure :-1 s=READY pgs=6813 cs=0 l=1 rev1=1 crypto rx=0x7fa5640098f0 tx=0x7fa56402a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afbea231da1d39ea server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:20:58.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.678+0000 7fa56effd640 1 -- 172.21.15.121:0/2675923583 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa564037070 con 0x7fa5701368d0 2024-09-17T18:20:58.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.678+0000 7fa56effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:20:58.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.678+0000 7fa56effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:20:58.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.678+0000 7fa56effd640 10 monclient: dump: 2024-09-17T18:20:58.682 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:20:58.682 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:20:58.682 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:20:58.682 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:20:58.682 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:20:58.682 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:20:58.683 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:20:58.683 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:20:58.683 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:20:58.683 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:20:58.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.678+0000 7fa56effd640 10 monclient: _finish_auth 0 2024-09-17T18:20:58.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.678+0000 7fa56effd640 10 monclient: _check_auth_tickets 2024-09-17T18:20:58.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.678+0000 7fa56effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:20:58.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.678+0000 7fa56effd640 1 -- 172.21.15.121:0/2675923583 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa564032640 con 0x7fa5701368d0 2024-09-17T18:20:58.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.678+0000 7fa5764b8640 5 monclient: authenticate success, global_id 23700 2024-09-17T18:20:58.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.678+0000 7fa5764b8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:20:58.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.678+0000 7fa56effd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:20:58.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.678+0000 7fa5764b8640 10 monclient: get_monmap_and_config success 2024-09-17T18:20:58.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.678+0000 7fa56effd640 1 -- 172.21.15.121:0/2675923583 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa564032a10 con 0x7fa5701368d0 2024-09-17T18:20:58.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.678+0000 7fa56effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:20:58.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.678+0000 7fa5764b8640 4 set_mon_vals no callback set 2024-09-17T18:20:58.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.678+0000 7fa56effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:20:58.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.678+0000 7fa56effd640 10 monclient: dump: 2024-09-17T18:20:58.684 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:20:58.684 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:20:58.684 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:20:58.684 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:20:58.684 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:20:58.684 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:20:58.684 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:20:58.684 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:20:58.684 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:20:58.684 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:20:58.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.678+0000 7fa5764b8640 10 monclient: shutdown 2024-09-17T18:20:58.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.678+0000 7fa5764b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:20:58.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.678+0000 7fa5764b8640 1 -- 172.21.15.121:0/2675923583 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5701368d0 msgr2=0x7fa57013b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:20:58.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.678+0000 7fa5764b8640 1 --2- 172.21.15.121:0/2675923583 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5701368d0 0x7fa57013b540 secure :-1 s=READY pgs=6813 cs=0 l=1 rev1=1 crypto rx=0x7fa5640098f0 tx=0x7fa56402a1c0 comp rx=0 tx=0).stop 2024-09-17T18:20:58.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.678+0000 7fa5764b8640 1 -- 172.21.15.121:0/2675923583 shutdown_connections 2024-09-17T18:20:58.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.678+0000 7fa5764b8640 1 --2- 172.21.15.121:0/2675923583 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5701368d0 0x7fa57013b540 unknown :-1 s=CLOSED pgs=6813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:20:58.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.678+0000 7fa5764b8640 1 -- 172.21.15.121:0/2675923583 >> 172.21.15.121:0/2675923583 conn(0x7fa5700a7830 msgr2=0x7fa5700a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:20:58.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.678+0000 7fa5764b8640 1 -- 172.21.15.121:0/2675923583 shutdown_connections 2024-09-17T18:20:58.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.678+0000 7fa5764b8640 1 -- 172.21.15.121:0/2675923583 wait complete. 2024-09-17T18:20:58.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.679+0000 7fa5764b8640 10 monclient: build_initial_monmap 2024-09-17T18:20:58.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.679+0000 7fa5764b8640 10 monclient: monmap: 2024-09-17T18:20:58.685 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:20:58.685 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:20:58.685 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:20:58.686 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:20:58.686 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:20:58.686 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:20:58.686 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:20:58.686 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:20:58.686 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:20:58.686 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:20:58.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.679+0000 7fa5764b8640 1 librados: starting msgr at 2024-09-17T18:20:58.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.679+0000 7fa5764b8640 1 librados: starting objecter 2024-09-17T18:20:58.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.679+0000 7fa5764b8640 1 Processor -- start 2024-09-17T18:20:58.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.679+0000 7fa5764b8640 1 -- start start 2024-09-17T18:20:58.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.679+0000 7fa5764b8640 1 librados: setting wanted keys 2024-09-17T18:20:58.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.679+0000 7fa5764b8640 1 librados: calling monclient init 2024-09-17T18:20:58.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.679+0000 7fa5764b8640 10 monclient: init 2024-09-17T18:20:58.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa5764b8640 10 monclient: _reopen_session rank -1 2024-09-17T18:20:58.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa5764b8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:20:58.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa5764b8640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa57009da70 0x7fa57013e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:20:58.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa5764b8640 10 monclient(hunting): picked mon.noname-a con 0x7fa57009da70 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:20:58.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa5764b8640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa5701368d0 0x7fa57013e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:20:58.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa5764b8640 10 monclient(hunting): picked mon.noname-b con 0x7fa5701368d0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:20:58.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa5764b8640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa570136d00 0x7fa57009e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:20:58.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa5764b8640 10 monclient(hunting): picked mon.noname-c con 0x7fa570136d00 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:20:58.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa5764b8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:20:58.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa5764b8640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa57013f1b0 con 0x7fa57009da70 2024-09-17T18:20:58.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa5764b8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:20:58.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa5764b8640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa5700a3930 con 0x7fa5701368d0 2024-09-17T18:20:58.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa5764b8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:20:58.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa5764b8640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa57013f330 con 0x7fa570136d00 2024-09-17T18:20:58.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa5764b8640 10 monclient(hunting): _renew_subs 2024-09-17T18:20:58.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa5764b8640 10 monclient(hunting): authenticate will time out at 862.397311s 2024-09-17T18:20:58.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa56ffff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa57009da70 0x7fa57013e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:20:58.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa574a55640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa570136d00 0x7fa57009e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:20:58.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa56ffff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa57009da70 0x7fa57013e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36220/0 (socket says 172.21.15.121:36220) 2024-09-17T18:20:58.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa56ffff640 1 -- 172.21.15.121:0/895483968 learned_addr learned my addr 172.21.15.121:0/895483968 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:20:58.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa56ffff640 10 monclient(hunting): get_auth_request con 0x7fa57009da70 auth_method 0 2024-09-17T18:20:58.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa56ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:20:58.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa56ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:20:58.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa56ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:20:58.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa574a55640 10 monclient(hunting): get_auth_request con 0x7fa570136d00 auth_method 0 2024-09-17T18:20:58.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa574a55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:20:58.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa574a55640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:20:58.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa574a55640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:20:58.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa56ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:20:58.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa56f7fe640 1 --2- 172.21.15.121:0/895483968 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa5701368d0 0x7fa57013e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:20:58.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa56ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:20:58.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa56ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:20:58.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa56f7fe640 10 monclient(hunting): get_auth_request con 0x7fa5701368d0 auth_method 0 2024-09-17T18:20:58.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa56f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:20:58.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa56f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:20:58.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa56f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:20:58.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa574a55640 1 -- 172.21.15.121:0/895483968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa570136d00 msgr2=0x7fa57009e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T18:20:58.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa574a55640 1 -- 172.21.15.121:0/895483968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa570136d00 msgr2=0x7fa57009e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:20:58.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa574a55640 1 --2- 172.21.15.121:0/895483968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa570136d00 0x7fa57009e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:20:58.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa574a55640 1 --2- 172.21.15.121:0/895483968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa570136d00 0x7fa57009e550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:20:58.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa56f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:20:58.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa56f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:20:58.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa56f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:20:58.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa56ffff640 10 monclient(hunting): handle_auth_done global_id 23703 payload 824 2024-09-17T18:20:58.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa56ffff640 1 -- 172.21.15.121:0/895483968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa570136d00 msgr2=0x7fa57009e550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:20:58.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa56ffff640 1 --2- 172.21.15.121:0/895483968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa570136d00 0x7fa57009e550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:20:58.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa56ffff640 1 -- 172.21.15.121:0/895483968 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa5701368d0 msgr2=0x7fa57013e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:20:58.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa56ffff640 1 --2- 172.21.15.121:0/895483968 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa5701368d0 0x7fa57013e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:20:58.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa56ffff640 10 monclient: _finish_hunting 0 2024-09-17T18:20:58.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa56ffff640 1 monclient: found mon.noname-a 2024-09-17T18:20:58.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa56ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:20:58.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa56ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:20:58.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.680+0000 7fa56ffff640 1 -- 172.21.15.121:0/895483968 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5701355b0 con 0x7fa57009da70 2024-09-17T18:20:58.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.683+0000 7fa56cff9640 10 client.?.objecter ms_handle_connect 0x7fa57009da70 2024-09-17T18:20:58.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.683+0000 7fa56cff9640 10 client.?.objecter resend_mon_ops 2024-09-17T18:20:58.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.682+0000 7fa56ffff640 1 -- 172.21.15.121:0/895483968 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa57009da70 msgr2=0x7fa57013e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:20:58.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.683+0000 7fa56ffff640 1 --2- 172.21.15.121:0/895483968 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa57009da70 0x7fa57013e2d0 secure :-1 s=READY pgs=6814 cs=0 l=1 rev1=1 crypto rx=0x7fa56000c450 tx=0x7fa56000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:20:58.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.683+0000 7fa56cff9640 1 -- 172.21.15.121:0/895483968 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa560007690 con 0x7fa57009da70 2024-09-17T18:20:58.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.683+0000 7fa56cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:20:58.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.683+0000 7fa56cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:20:58.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.683+0000 7fa56cff9640 10 monclient: dump: 2024-09-17T18:20:58.695 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:20:58.695 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:20:58.695 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:20:58.695 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:20:58.695 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:20:58.695 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:20:58.696 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:20:58.696 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:20:58.696 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:20:58.696 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:20:58.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.683+0000 7fa56cff9640 10 monclient: _finish_auth 0 2024-09-17T18:20:58.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.683+0000 7fa56cff9640 10 monclient: _check_auth_tickets 2024-09-17T18:20:58.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.683+0000 7fa56cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:20:58.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.683+0000 7fa56cff9640 1 -- 172.21.15.121:0/895483968 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5600047e0 con 0x7fa57009da70 2024-09-17T18:20:58.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.683+0000 7fa5764b8640 5 monclient: authenticate success, global_id 23703 2024-09-17T18:20:58.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.683+0000 7fa5764b8640 10 monclient: _renew_subs 2024-09-17T18:20:58.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.683+0000 7fa5764b8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:20:58.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.683+0000 7fa5764b8640 1 -- 172.21.15.121:0/895483968 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5700a6980 con 0x7fa57009da70 2024-09-17T18:20:58.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.683+0000 7fa56cff9640 10 monclient: handle_config config(0 keys) 2024-09-17T18:20:58.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.683+0000 7fa56cff9640 1 -- 172.21.15.121:0/895483968 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa560004b90 con 0x7fa57009da70 2024-09-17T18:20:58.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.683+0000 7fa56cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:20:58.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.683+0000 7fa56cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:20:58.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.683+0000 7fa56cff9640 10 monclient: dump: 2024-09-17T18:20:58.697 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:20:58.697 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:20:58.697 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:20:58.697 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:20:58.697 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:20:58.697 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:20:58.697 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:20:58.697 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:20:58.697 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:20:58.697 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:20:58.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.683+0000 7fa5764b8640 10 client.23703.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:20:58.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.683+0000 7fa5764b8640 10 monclient: _renew_subs 2024-09-17T18:20:58.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.683+0000 7fa5764b8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:20:58.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.683+0000 7fa56e7fc640 4 set_mon_vals no callback set 2024-09-17T18:20:58.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.683+0000 7fa5764b8640 1 -- 172.21.15.121:0/895483968 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa57013c7e0 con 0x7fa57009da70 2024-09-17T18:20:58.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.683+0000 7fa5764b8640 1 librados: init done 2024-09-17T18:20:58.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.683+0000 7fa5764b8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:20:58.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.683+0000 7fa5764b8640 1 --2- 172.21.15.121:0/895483968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5701cd850 0x7fa57002d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:20:58.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.683+0000 7fa5764b8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:20:58.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.683+0000 7fa5764b8640 1 -- 172.21.15.121:0/895483968 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa57009e240 con 0x7fa5701cd850 2024-09-17T18:20:58.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.684+0000 7fa56cff9640 1 -- 172.21.15.121:0/895483968 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fa56001a020 con 0x7fa57009da70 2024-09-17T18:20:58.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.684+0000 7fa56f7fe640 1 --2- 172.21.15.121:0/895483968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5701cd850 0x7fa57002d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:20:58.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.684+0000 7fa56f7fe640 10 monclient: get_auth_request con 0x7fa5701cd850 auth_method 0 2024-09-17T18:20:58.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.684+0000 7fa56f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:20:58.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.684+0000 7fa56f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:20:58.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.684+0000 7fa56f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:20:58.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.684+0000 7fa56f7fe640 1 -- 172.21.15.121:0/895483968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5701cd850 msgr2=0x7fa57002d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T18:20:58.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.684+0000 7fa56f7fe640 1 -- 172.21.15.121:0/895483968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5701cd850 msgr2=0x7fa57002d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:20:58.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.684+0000 7fa56f7fe640 1 --2- 172.21.15.121:0/895483968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5701cd850 0x7fa57002d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:20:58.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.684+0000 7fa56f7fe640 1 --2- 172.21.15.121:0/895483968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5701cd850 0x7fa57002d440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:20:58.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.684+0000 7fa56cff9640 1 --2- 172.21.15.121:0/895483968 >> v2:172.21.15.173:6812/2423786037 conn(0x7fa55c040170 0x7fa55c042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:20:58.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.684+0000 7fa56cff9640 1 -- 172.21.15.121:0/895483968 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 6984+0+0 (secure 0 0 0) 0x7fa560053430 con 0x7fa57009da70 2024-09-17T18:20:58.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.684+0000 7fa56cff9640 10 client.23703.objecter ms_dispatch 0x7fa570000b90 osd_map(75..75 src has 1..75) 2024-09-17T18:20:58.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.684+0000 7fa56cff9640 3 client.23703.objecter handle_osd_map got epochs [75,75] > 0 2024-09-17T18:20:58.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.684+0000 7fa56cff9640 3 client.23703.objecter handle_osd_map decoding full epoch 75 2024-09-17T18:20:58.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.684+0000 7fa56cff9640 20 client.23703.objecter dump_active .. 0 homeless 2024-09-17T18:20:58.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.684+0000 7fa574a55640 1 --2- 172.21.15.121:0/895483968 >> v2:172.21.15.173:6812/2423786037 conn(0x7fa55c040170 0x7fa55c042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:20:58.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.684+0000 7fa574a55640 10 monclient: get_auth_request con 0x7fa55c040170 auth_method 0 2024-09-17T18:20:58.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.686+0000 7fa56cff9640 10 client.23703.objecter ms_handle_connect 0x7fa55c040170 2024-09-17T18:20:58.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.686+0000 7fa574a55640 1 --2- 172.21.15.121:0/895483968 >> v2:172.21.15.173:6812/2423786037 conn(0x7fa55c040170 0x7fa55c042630 secure :-1 s=READY pgs=3858 cs=0 l=1 rev1=1 crypto rx=0x7fa57013ba90 tx=0x7fa568009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:20:58.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.885+0000 7fa56f7fe640 1 --2- 172.21.15.121:0/895483968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5701cd850 0x7fa57002d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:20:58.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.885+0000 7fa56f7fe640 10 monclient: get_auth_request con 0x7fa5701cd850 auth_method 0 2024-09-17T18:20:58.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.885+0000 7fa56f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:20:58.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.885+0000 7fa56f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:20:58.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.885+0000 7fa56f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:20:58.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.885+0000 7fa56f7fe640 1 -- 172.21.15.121:0/895483968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5701cd850 msgr2=0x7fa57002d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T18:20:58.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.885+0000 7fa56f7fe640 1 -- 172.21.15.121:0/895483968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5701cd850 msgr2=0x7fa57002d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:20:58.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.885+0000 7fa56f7fe640 1 --2- 172.21.15.121:0/895483968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5701cd850 0x7fa57002d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:20:58.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:58.885+0000 7fa56f7fe640 1 --2- 172.21.15.121:0/895483968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5701cd850 0x7fa57002d440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T18:20:59.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:59.289+0000 7fa56f7fe640 1 --2- 172.21.15.121:0/895483968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5701cd850 0x7fa57002d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:20:59.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:59.289+0000 7fa56f7fe640 10 monclient: get_auth_request con 0x7fa5701cd850 auth_method 0 2024-09-17T18:20:59.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:59.289+0000 7fa56f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:20:59.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:59.289+0000 7fa56f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:20:59.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:59.289+0000 7fa56f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:20:59.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:59.289+0000 7fa56f7fe640 1 -- 172.21.15.121:0/895483968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5701cd850 msgr2=0x7fa57002d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T18:20:59.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:59.289+0000 7fa56f7fe640 1 -- 172.21.15.121:0/895483968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5701cd850 msgr2=0x7fa57002d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:20:59.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:59.289+0000 7fa56f7fe640 1 --2- 172.21.15.121:0/895483968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5701cd850 0x7fa57002d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:20:59.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:20:59.289+0000 7fa56f7fe640 1 --2- 172.21.15.121:0/895483968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5701cd850 0x7fa57002d440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T18:21:00.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.091+0000 7fa56f7fe640 1 --2- 172.21.15.121:0/895483968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5701cd850 0x7fa57002d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:00.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.091+0000 7fa56f7fe640 10 monclient: get_auth_request con 0x7fa5701cd850 auth_method 0 2024-09-17T18:21:00.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.091+0000 7fa56f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:00.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.091+0000 7fa56f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:00.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.091+0000 7fa56f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:21:00.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.091+0000 7fa56f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:00.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.091+0000 7fa56f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:00.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.091+0000 7fa56f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:00.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.091+0000 7fa56f7fe640 10 monclient(hunting): handle_auth_done global_id 34100 payload 824 2024-09-17T18:21:00.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.091+0000 7fa56f7fe640 1 -- 172.21.15.121:0/895483968 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa5701361c0 con 0x7fa5701cd850 2024-09-17T18:21:00.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.093+0000 7fa56f7fe640 1 -- 172.21.15.121:0/895483968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5701cd850 msgr2=0x7fa57002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:21:00.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.093+0000 7fa56f7fe640 1 --2- 172.21.15.121:0/895483968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5701cd850 0x7fa57002d440 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa57009e070 tx=0x7fa5640046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:21:00.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.093+0000 7fa56cff9640 10 client.23703.objecter ms_handle_connect 0x7fa5701cd850 2024-09-17T18:21:00.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.093+0000 7fa56cff9640 10 client.23703.objecter resend_mon_ops 2024-09-17T18:21:00.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.094+0000 7fa56cff9640 1 -- 172.21.15.121:0/895483968 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa564037040 con 0x7fa5701cd850 2024-09-17T18:21:00.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.094+0000 7fa56cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:00.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.094+0000 7fa56cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:21:00.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.094+0000 7fa56cff9640 10 monclient: dump: 2024-09-17T18:21:00.103 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:00.103 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:00.103 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:00.103 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:00.103 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:00.103 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:00.103 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:00.103 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:00.103 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:00.103 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:00.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.094+0000 7fa56cff9640 1 -- 172.21.15.121:0/895483968 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa56403c820 con 0x7fa5701cd850 2024-09-17T18:21:00.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.094+0000 7fa56cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:21:00.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.094+0000 7fa56cff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:21:00.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.094+0000 7fa56cff9640 1 -- 172.21.15.121:0/895483968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5701cd850 msgr2=0x7fa57002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:00.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.094+0000 7fa56cff9640 1 --2- 172.21.15.121:0/895483968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5701cd850 0x7fa57002d440 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa57009e070 tx=0x7fa5640046c0 comp rx=0 tx=0).stop 2024-09-17T18:21:00.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.094+0000 7fa56cff9640 1 -- 172.21.15.121:0/895483968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5701cd850 msgr2=0x7fa57002d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:21:00.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.094+0000 7fa56cff9640 1 --2- 172.21.15.121:0/895483968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5701cd850 0x7fa57002d440 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:00.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.101+0000 7fa5764b8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:21:00.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.101+0000 7fa5764b8640 1 --2- 172.21.15.121:0/895483968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa57002d980 0x7fa57002dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:00.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.101+0000 7fa5764b8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:00.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.101+0000 7fa5764b8640 1 -- 172.21.15.121:0/895483968 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa5700709d0 con 0x7fa57002d980 2024-09-17T18:21:00.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.101+0000 7fa56f7fe640 1 --2- 172.21.15.121:0/895483968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa57002d980 0x7fa57002dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:00.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.101+0000 7fa56f7fe640 10 monclient: get_auth_request con 0x7fa57002d980 auth_method 0 2024-09-17T18:21:00.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.101+0000 7fa56f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:00.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.101+0000 7fa56f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:00.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.101+0000 7fa56f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:00.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.101+0000 7fa56f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:00.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.101+0000 7fa56f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:00.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.101+0000 7fa56f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:00.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.101+0000 7fa56f7fe640 10 monclient(hunting): handle_auth_done global_id 34103 payload 824 2024-09-17T18:21:00.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.101+0000 7fa56f7fe640 1 -- 172.21.15.121:0/895483968 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa57013b5a0 con 0x7fa57002d980 2024-09-17T18:21:00.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.103+0000 7fa56f7fe640 1 -- 172.21.15.121:0/895483968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa57002d980 msgr2=0x7fa57002dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:21:00.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.103+0000 7fa56f7fe640 1 --2- 172.21.15.121:0/895483968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa57002d980 0x7fa57002dd60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa56402a200 tx=0x7fa564008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:21:00.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.103+0000 7fa56cff9640 10 client.23703.objecter ms_handle_connect 0x7fa57002d980 2024-09-17T18:21:00.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.104+0000 7fa56cff9640 10 client.23703.objecter resend_mon_ops 2024-09-17T18:21:00.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.104+0000 7fa56cff9640 1 -- 172.21.15.121:0/895483968 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa564052030 con 0x7fa57002d980 2024-09-17T18:21:00.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.104+0000 7fa56cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:00.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.104+0000 7fa56cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:21:00.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.104+0000 7fa56cff9640 10 monclient: dump: 2024-09-17T18:21:00.108 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:00.108 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:00.108 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:00.108 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:00.108 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:00.108 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:00.108 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:00.108 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:00.108 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:00.108 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:00.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.104+0000 7fa56cff9640 1 -- 172.21.15.121:0/895483968 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4551 (secure 0 0 0) 0x7fa56404ea90 con 0x7fa57002d980 2024-09-17T18:21:00.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.104+0000 7fa56cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:21:00.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.104+0000 7fa56cff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:21:00.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.104+0000 7fa56cff9640 1 -- 172.21.15.121:0/895483968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa57002d980 msgr2=0x7fa57002dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:00.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.104+0000 7fa56cff9640 1 --2- 172.21.15.121:0/895483968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa57002d980 0x7fa57002dd60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa56402a200 tx=0x7fa564008030 comp rx=0 tx=0).stop 2024-09-17T18:21:00.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.104+0000 7fa56cff9640 1 -- 172.21.15.121:0/895483968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa57002d980 msgr2=0x7fa57002dd60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:21:00.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.104+0000 7fa56cff9640 1 --2- 172.21.15.121:0/895483968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa57002d980 0x7fa57002dd60 secure :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa56402a200 tx=0x7fa564008030 comp rx=0 tx=0).stop 2024-09-17T18:21:00.109 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:21:00.109 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:21:00.110 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:21:00.110 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:21:00.110 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 10, 2024-09-17T18:21:00.110 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 5381, 2024-09-17T18:21:00.110 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:21:00.110 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:21:00.110 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:21:00.110 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:21:00.110 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:21:00.110 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 4, 2024-09-17T18:21:00.110 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:21:00.110 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:21:00.110 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:21:00.110 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:21:00.111 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:21:00.111 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:21:00.111 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:21:00.111 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:21:00.111 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:21:00.111 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:21:00.111 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:21:00.111 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:21:00.111 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:21:00.111 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:21:00.111 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:21:00.111 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:21:00.111 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:21:00.111 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:21:00.112 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:21:00.112 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:21:00.112 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:21:00.112 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:21:00.112 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:21:00.112 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:21:00.112 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:21:00.112 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:21:00.112 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:21:00.112 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:21:00.112 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:21:00.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:00.112 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:21:00.112 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:21:00.113 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:21:00.113 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:21:00.113 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:21:00.113 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:21:00.113 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:21:00.113 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:21:00.113 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:21:00.113 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:21:00.113 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:21:00.113 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:21:00.113 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:21:00.113 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:21:00.113 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:21:00.114 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:21:00.114 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:21:00.114 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:21:00.114 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:21:00.114 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:21:00.114 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:21:00.114 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:21:00.114 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:21:00.114 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:21:00.114 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:21:00.114 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:21:00.114 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:21:00.114 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:21:00.114 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:21:00.114 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:21:00.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:00.115 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:21:00.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:00.115 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:21:00.115 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:21:00.116 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:21:00.116 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:21:00.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:00.116 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:21:00.116 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:21:00.116 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:21:00.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:00.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:00.116 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:21:00.116 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:21:00.116 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:21:00.117 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:00.117 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:21:00.117 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:00.117 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:21:00.117 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:21:00.117 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:21:00.117 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:21:00.117 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:21:00.117 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:00.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:00.117 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:21:00.117 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:21:00.117 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:21:00.117 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:21:00.118 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:00.118 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:21:00.118 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:21:00.118 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:21:00.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:00.118 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:00.118 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:21:00.118 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:21:00.118 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:21:00.118 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:00.118 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:21:00.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:00.118 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:21:00.118 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:21:00.119 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:21:00.119 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:21:00.119 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:21:00.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:00.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:00.119 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:21:00.119 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:21:00.119 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:21:00.119 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:21:00.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:00.119 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:21:00.119 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:21:00.119 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:21:00.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:00.120 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:00.120 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:21:00.120 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:21:00.120 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:21:00.120 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:00.120 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:21:00.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:00.121 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:21:00.121 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:21:00.121 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:21:00.121 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:21:00.121 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:21:00.121 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:00.121 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:21:00.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:00.121 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:21:00.121 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:21:00.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:00.121 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:21:00.122 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:21:00.122 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:21:00.122 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:00.122 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:21:00.122 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:21:00.122 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:00.122 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:21:00.122 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:21:00.122 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:21:00.122 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:00.122 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:21:00.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:00.122 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:21:00.122 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:21:00.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.104+0000 7fa5764b8640 10 librados: watch_flush enter 2024-09-17T18:21:00.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.104+0000 7fa5764b8640 10 librados: watch_flush exit 2024-09-17T18:21:00.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.104+0000 7fa5764b8640 20 client.23703.objecter shutdown clearing up homeless session... 2024-09-17T18:21:00.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.104+0000 7fa5764b8640 10 client.23703.objecter successfully canceled tick 2024-09-17T18:21:00.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.105+0000 7fa5764b8640 1 -- 172.21.15.121:0/895483968 >> v2:172.21.15.173:6812/2423786037 conn(0x7fa55c040170 msgr2=0x7fa55c042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:00.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.105+0000 7fa5764b8640 1 --2- 172.21.15.121:0/895483968 >> v2:172.21.15.173:6812/2423786037 conn(0x7fa55c040170 0x7fa55c042630 secure :-1 s=READY pgs=3858 cs=0 l=1 rev1=1 crypto rx=0x7fa57013ba90 tx=0x7fa568009040 comp rx=0 tx=0).stop 2024-09-17T18:21:00.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.105+0000 7fa5764b8640 10 monclient: shutdown 2024-09-17T18:21:00.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.105+0000 7fa5764b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:21:00.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.105+0000 7fa5764b8640 1 -- 172.21.15.121:0/895483968 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa57009da70 msgr2=0x7fa57013e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:00.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.105+0000 7fa5764b8640 1 --2- 172.21.15.121:0/895483968 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa57009da70 0x7fa57013e2d0 secure :-1 s=READY pgs=6814 cs=0 l=1 rev1=1 crypto rx=0x7fa56000c450 tx=0x7fa56000c910 comp rx=0 tx=0).stop 2024-09-17T18:21:00.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.105+0000 7fa574a55640 1 -- 172.21.15.121:0/895483968 reap_dead start 2024-09-17T18:21:00.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.105+0000 7fa574a55640 1 -- 172.21.15.121:0/895483968 reap_dead start 2024-09-17T18:21:00.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.105+0000 7fa5764b8640 1 -- 172.21.15.121:0/895483968 shutdown_connections 2024-09-17T18:21:00.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.105+0000 7fa5764b8640 1 -- 172.21.15.121:0/895483968 >> 172.21.15.121:0/895483968 conn(0x7fa5700a7830 msgr2=0x7fa570132bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:21:00.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.105+0000 7fa5764b8640 1 -- 172.21.15.121:0/895483968 shutdown_connections 2024-09-17T18:21:00.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.105+0000 7fa5764b8640 1 -- 172.21.15.121:0/895483968 wait complete. 2024-09-17T18:21:00.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.105+0000 7fa5764b8640 1 librados: shutdown 2024-09-17T18:21:00.126 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:21:00.126 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T18:21:00.126 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T18:21:00.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.230+0000 7f4a3e050640 10 monclient: get_monmap_and_config 2024-09-17T18:21:00.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.230+0000 7f4a3e050640 10 monclient: build_initial_monmap 2024-09-17T18:21:00.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.230+0000 7f4a3e050640 10 monclient: monmap: 2024-09-17T18:21:00.233 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:21:00.233 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:00.233 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:21:00.232245+0000 2024-09-17T18:21:00.233 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:21:00.232245+0000 2024-09-17T18:21:00.234 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:21:00.234 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:00.234 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:21:00.234 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:21:00.234 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:21:00.234 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:21:00.234 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:21:00.234 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:00.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.231+0000 7f4a3e050640 1 Processor -- start 2024-09-17T18:21:00.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.231+0000 7f4a3e050640 1 -- start start 2024-09-17T18:21:00.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.231+0000 7f4a3e050640 10 monclient: init 2024-09-17T18:21:00.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.231+0000 7f4a3e050640 10 monclient: _reopen_session rank -1 2024-09-17T18:21:00.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.231+0000 7f4a3e050640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-17T18:21:00.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.232+0000 7f4a3e050640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4a3809e550 addr v1:172.21.15.121:6789/0 2024-09-17T18:21:00.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.232+0000 7f4a3e050640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a3809f040 0x7f4a3813a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:00.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.232+0000 7f4a3e050640 10 monclient(hunting): picked mon.noname-b con 0x7f4a3809f040 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:21:00.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.232+0000 7f4a3e050640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4a3813ad90 0x7f4a3813d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:00.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.232+0000 7f4a3e050640 10 monclient(hunting): picked mon.noname-a con 0x7f4a3813ad90 addr v2:172.21.15.121:3300/0 2024-09-17T18:21:00.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.232+0000 7f4a3e050640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a380a4460 con 0x7f4a3809e550 2024-09-17T18:21:00.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.232+0000 7f4a3e050640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:00.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.232+0000 7f4a3e050640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f4a380a4950 con 0x7f4a3813ad90 2024-09-17T18:21:00.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.232+0000 7f4a3e050640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:00.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.232+0000 7f4a3e050640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4a380a2b60 con 0x7f4a3809f040 2024-09-17T18:21:00.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.232+0000 7f4a3e050640 10 monclient(hunting): _renew_subs 2024-09-17T18:21:00.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.232+0000 7f4a3e050640 10 monclient(hunting): authenticate will time out at 863.949067s 2024-09-17T18:21:00.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.232+0000 7f4a37fff640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4a3813ad90 0x7f4a3813d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:00.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.232+0000 7f4a36ffd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a3809f040 0x7f4a3813a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:00.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.232+0000 7f4a377fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f4a3809e550 0x7f4a3809e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:51358/0 (socket says 172.21.15.121:51358) 2024-09-17T18:21:00.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.232+0000 7f4a377fe640 1 -- 172.21.15.121:0/533157638 learned_addr learned my addr 172.21.15.121:0/533157638 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:21:00.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.232+0000 7f4a36ffd640 10 monclient(hunting): get_auth_request con 0x7f4a3809f040 auth_method 0 2024-09-17T18:21:00.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.232+0000 7f4a36ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:00.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.232+0000 7f4a36ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:00.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.232+0000 7f4a36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:00.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.232+0000 7f4a37fff640 10 monclient(hunting): get_auth_request con 0x7f4a3813ad90 auth_method 0 2024-09-17T18:21:00.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.232+0000 7f4a37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:00.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.232+0000 7f4a37fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:00.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.232+0000 7f4a37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:00.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.232+0000 7f4a36ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:00.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.232+0000 7f4a36ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:00.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.232+0000 7f4a36ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:00.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.232+0000 7f4a37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:00.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.232+0000 7f4a37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:00.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.232+0000 7f4a37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:00.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.232+0000 7f4a37fff640 10 monclient(hunting): handle_auth_done global_id 23721 payload 274 2024-09-17T18:21:00.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.232+0000 7f4a37fff640 1 -- 172.21.15.121:0/533157638 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a3809f040 msgr2=0x7f4a3813a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:21:00.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.232+0000 7f4a37fff640 1 --2- 172.21.15.121:0/533157638 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a3809f040 0x7f4a3813a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:00.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.232+0000 7f4a37fff640 1 -- 172.21.15.121:0/533157638 >> v1:172.21.15.121:6789/0 conn(0x7f4a3809e550 legacy=0x7f4a3809e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:21:00.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.232+0000 7f4a37fff640 10 monclient: _finish_hunting 0 2024-09-17T18:21:00.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.232+0000 7f4a37fff640 1 monclient: found mon.noname-a 2024-09-17T18:21:00.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.232+0000 7f4a37fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:21:00.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.232+0000 7f4a37fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:21:00.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.232+0000 7f4a37fff640 1 -- 172.21.15.121:0/533157638 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a3809f840 con 0x7f4a3813ad90 2024-09-17T18:21:00.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.232+0000 7f4a36ffd640 1 --2- 172.21.15.121:0/533157638 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a3809f040 0x7f4a3813a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:21:00.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.235+0000 7f4a37fff640 1 -- 172.21.15.121:0/533157638 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a3813ad90 msgr2=0x7f4a3813d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:21:00.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.235+0000 7f4a37fff640 1 --2- 172.21.15.121:0/533157638 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a3813ad90 0x7f4a3813d180 secure :-1 s=READY pgs=6819 cs=0 l=1 rev1=1 crypto rx=0x7f4a300097a0 tx=0x7f4a30029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ab69550e7305bf0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:21:00.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.235+0000 7f4a367fc640 1 -- 172.21.15.121:0/533157638 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a30037070 con 0x7f4a3813ad90 2024-09-17T18:21:00.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.235+0000 7f4a367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:00.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.235+0000 7f4a367fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:21:00.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.235+0000 7f4a367fc640 10 monclient: dump: 2024-09-17T18:21:00.253 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:00.253 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:00.253 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:00.253 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:00.253 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:00.253 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:00.253 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:00.253 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:00.253 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:00.253 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:00.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.235+0000 7f4a367fc640 10 monclient: _finish_auth 0 2024-09-17T18:21:00.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.235+0000 7f4a367fc640 10 monclient: _check_auth_tickets 2024-09-17T18:21:00.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.235+0000 7f4a367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:21:00.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.235+0000 7f4a3e050640 5 monclient: authenticate success, global_id 23721 2024-09-17T18:21:00.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.235+0000 7f4a3e050640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:21:00.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.235+0000 7f4a367fc640 1 -- 172.21.15.121:0/533157638 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a30032440 con 0x7f4a3813ad90 2024-09-17T18:21:00.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.235+0000 7f4a367fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:21:00.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.235+0000 7f4a367fc640 1 -- 172.21.15.121:0/533157638 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a30032810 con 0x7f4a3813ad90 2024-09-17T18:21:00.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.235+0000 7f4a367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:00.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.235+0000 7f4a367fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:21:00.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.235+0000 7f4a367fc640 10 monclient: dump: 2024-09-17T18:21:00.254 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:00.254 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:00.255 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:00.255 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:00.255 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:00.255 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:00.255 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:00.255 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:00.255 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:00.255 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:00.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.235+0000 7f4a3e050640 10 monclient: get_monmap_and_config success 2024-09-17T18:21:00.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.235+0000 7f4a3e050640 4 set_mon_vals no callback set 2024-09-17T18:21:00.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.235+0000 7f4a3e050640 10 monclient: shutdown 2024-09-17T18:21:00.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.235+0000 7f4a3e050640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:21:00.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.235+0000 7f4a3e050640 1 -- 172.21.15.121:0/533157638 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a3813ad90 msgr2=0x7f4a3813d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:00.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.235+0000 7f4a3e050640 1 --2- 172.21.15.121:0/533157638 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a3813ad90 0x7f4a3813d180 secure :-1 s=READY pgs=6819 cs=0 l=1 rev1=1 crypto rx=0x7f4a300097a0 tx=0x7f4a30029fd0 comp rx=0 tx=0).stop 2024-09-17T18:21:00.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.236+0000 7f4a3e050640 1 -- 172.21.15.121:0/533157638 shutdown_connections 2024-09-17T18:21:00.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.236+0000 7f4a3e050640 1 --2- 172.21.15.121:0/533157638 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a3813ad90 0x7f4a3813d180 unknown :-1 s=CLOSED pgs=6819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:00.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.236+0000 7f4a3e050640 1 --2- 172.21.15.121:0/533157638 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a3809f040 0x7f4a3813a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:00.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.236+0000 7f4a3e050640 1 -- 172.21.15.121:0/533157638 >> 172.21.15.121:0/533157638 conn(0x7f4a380a7830 msgr2=0x7f4a380a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:21:00.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.236+0000 7f4a3e050640 1 -- 172.21.15.121:0/533157638 shutdown_connections 2024-09-17T18:21:00.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.236+0000 7f4a3e050640 1 -- 172.21.15.121:0/533157638 wait complete. 2024-09-17T18:21:00.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.236+0000 7f4a3e050640 10 monclient: build_initial_monmap 2024-09-17T18:21:00.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.236+0000 7f4a3e050640 10 monclient: monmap: 2024-09-17T18:21:00.256 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:21:00.256 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:21:00.256 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:21:00.256 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:21:00.256 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:21:00.256 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:00.257 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:21:00.257 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:21:00.257 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:21:00.257 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:00.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.236+0000 7f4a3e050640 1 librados: starting msgr at 2024-09-17T18:21:00.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.236+0000 7f4a3e050640 1 librados: starting objecter 2024-09-17T18:21:00.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.236+0000 7f4a3e050640 1 Processor -- start 2024-09-17T18:21:00.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.237+0000 7f4a3e050640 1 -- start start 2024-09-17T18:21:00.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.237+0000 7f4a3e050640 1 librados: setting wanted keys 2024-09-17T18:21:00.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.237+0000 7f4a3e050640 1 librados: calling monclient init 2024-09-17T18:21:00.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.237+0000 7f4a3e050640 10 monclient: init 2024-09-17T18:21:00.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.237+0000 7f4a3e050640 10 monclient: _reopen_session rank -1 2024-09-17T18:21:00.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.237+0000 7f4a3e050640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T18:21:00.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.237+0000 7f4a3e050640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a3809e550 0x7f4a38135dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:00.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.237+0000 7f4a3e050640 10 monclient(hunting): picked mon.noname-c con 0x7f4a3809e550 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:21:00.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.237+0000 7f4a3e050640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a3809f040 0x7f4a381ca090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:00.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.237+0000 7f4a3e050640 10 monclient(hunting): picked mon.noname-a con 0x7f4a3809f040 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:21:00.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.237+0000 7f4a3e050640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a3813ad90 0x7f4a381ca5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:00.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.237+0000 7f4a3e050640 10 monclient(hunting): picked mon.noname-b con 0x7f4a3813ad90 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:21:00.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.237+0000 7f4a3e050640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:00.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.237+0000 7f4a3e050640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3809dce0 con 0x7f4a3809f040 2024-09-17T18:21:00.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.237+0000 7f4a3e050640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:00.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.237+0000 7f4a3e050640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3809f4d0 con 0x7f4a3813ad90 2024-09-17T18:21:00.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.237+0000 7f4a3e050640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:00.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.237+0000 7f4a3e050640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4a3809de60 con 0x7f4a3809e550 2024-09-17T18:21:00.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.237+0000 7f4a3e050640 10 monclient(hunting): _renew_subs 2024-09-17T18:21:00.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.237+0000 7f4a3e050640 10 monclient(hunting): authenticate will time out at 863.954727s 2024-09-17T18:21:00.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.237+0000 7f4a377fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a3809e550 0x7f4a38135dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:00.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.237+0000 7f4a377fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a3809e550 0x7f4a38135dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:56398/0 (socket says 172.21.15.121:56398) 2024-09-17T18:21:00.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.237+0000 7f4a36ffd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a3809f040 0x7f4a381ca090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:00.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.237+0000 7f4a377fe640 1 -- 172.21.15.121:0/40509502 learned_addr learned my addr 172.21.15.121:0/40509502 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:21:00.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.237+0000 7f4a377fe640 10 monclient(hunting): get_auth_request con 0x7f4a3809e550 auth_method 0 2024-09-17T18:21:00.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.237+0000 7f4a377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:00.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.237+0000 7f4a377fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:00.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.237+0000 7f4a377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:00.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.237+0000 7f4a36ffd640 10 monclient(hunting): get_auth_request con 0x7f4a3809f040 auth_method 0 2024-09-17T18:21:00.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.237+0000 7f4a36ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:00.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.237+0000 7f4a36ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:00.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.237+0000 7f4a36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:00.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.237+0000 7f4a377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:00.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.237+0000 7f4a377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:00.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.237+0000 7f4a377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:00.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.237+0000 7f4a36ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:00.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.237+0000 7f4a36ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:00.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.237+0000 7f4a36ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:00.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.238+0000 7f4a377fe640 10 monclient(hunting): handle_auth_done global_id 34109 payload 824 2024-09-17T18:21:00.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.238+0000 7f4a377fe640 1 -- 172.21.15.121:0/40509502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a3813ad90 msgr2=0x7f4a381ca5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:00.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.238+0000 7f4a377fe640 1 --2- 172.21.15.121:0/40509502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a3813ad90 0x7f4a381ca5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:00.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.238+0000 7f4a377fe640 1 -- 172.21.15.121:0/40509502 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a3809f040 msgr2=0x7f4a381ca090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:00.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.238+0000 7f4a377fe640 1 --2- 172.21.15.121:0/40509502 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a3809f040 0x7f4a381ca090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:00.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.238+0000 7f4a377fe640 10 monclient: _finish_hunting 0 2024-09-17T18:21:00.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.238+0000 7f4a377fe640 1 monclient: found mon.noname-c 2024-09-17T18:21:00.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.238+0000 7f4a377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:21:00.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.238+0000 7f4a377fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:21:00.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.238+0000 7f4a377fe640 1 -- 172.21.15.121:0/40509502 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a381257d0 con 0x7f4a3809e550 2024-09-17T18:21:00.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.238+0000 7f4a36ffd640 1 --2- 172.21.15.121:0/40509502 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a3809f040 0x7f4a381ca090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:21:00.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.240+0000 7f4a377fe640 1 -- 172.21.15.121:0/40509502 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a3809e550 msgr2=0x7f4a38135dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:21:00.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.240+0000 7f4a377fe640 1 --2- 172.21.15.121:0/40509502 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a3809e550 0x7f4a38135dd0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4a2800cb90 tx=0x7f4a2800a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:21:00.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.240+0000 7f4a1ffff640 10 client.?.objecter ms_handle_connect 0x7f4a3809e550 2024-09-17T18:21:00.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.240+0000 7f4a1ffff640 10 client.?.objecter resend_mon_ops 2024-09-17T18:21:00.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.240+0000 7f4a1ffff640 1 -- 172.21.15.121:0/40509502 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a28012400 con 0x7f4a3809e550 2024-09-17T18:21:00.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.240+0000 7f4a1ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:00.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.240+0000 7f4a1ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:21:00.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.240+0000 7f4a1ffff640 10 monclient: dump: 2024-09-17T18:21:00.261 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:00.261 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:00.261 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:00.261 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:00.262 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:00.262 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:00.262 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:00.262 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:00.262 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:00.262 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:00.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.240+0000 7f4a1ffff640 10 monclient: _finish_auth 0 2024-09-17T18:21:00.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.240+0000 7f4a1ffff640 10 monclient: _check_auth_tickets 2024-09-17T18:21:00.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.240+0000 7f4a1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:21:00.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.240+0000 7f4a1ffff640 1 -- 172.21.15.121:0/40509502 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a280125a0 con 0x7f4a3809e550 2024-09-17T18:21:00.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.240+0000 7f4a3e050640 5 monclient: authenticate success, global_id 34109 2024-09-17T18:21:00.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.240+0000 7f4a3e050640 10 monclient: _renew_subs 2024-09-17T18:21:00.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.240+0000 7f4a3e050640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:21:00.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.240+0000 7f4a3e050640 1 -- 172.21.15.121:0/40509502 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a380a5ed0 con 0x7f4a3809e550 2024-09-17T18:21:00.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.240+0000 7f4a1ffff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:21:00.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.240+0000 7f4a1ffff640 1 -- 172.21.15.121:0/40509502 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a28012990 con 0x7f4a3809e550 2024-09-17T18:21:00.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.240+0000 7f4a1ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:00.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.240+0000 7f4a1ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:21:00.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.240+0000 7f4a1ffff640 10 monclient: dump: 2024-09-17T18:21:00.264 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:00.264 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:00.264 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:00.264 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:00.264 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:00.264 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:00.264 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:00.264 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:00.264 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:00.265 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:00.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.240+0000 7f4a35ffb640 4 set_mon_vals no callback set 2024-09-17T18:21:00.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.240+0000 7f4a3e050640 10 client.34109.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:21:00.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.240+0000 7f4a3e050640 10 monclient: _renew_subs 2024-09-17T18:21:00.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.240+0000 7f4a3e050640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:21:00.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.240+0000 7f4a3e050640 1 -- 172.21.15.121:0/40509502 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a381254f0 con 0x7f4a3809e550 2024-09-17T18:21:00.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.240+0000 7f4a3e050640 1 librados: init done 2024-09-17T18:21:00.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.241+0000 7f4a3e050640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:21:00.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.241+0000 7f4a3e050640 1 --2- 172.21.15.121:0/40509502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a38136310 0x7f4a3802e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:00.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.241+0000 7f4a3e050640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:00.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.241+0000 7f4a3e050640 1 -- 172.21.15.121:0/40509502 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4a38125170 con 0x7f4a38136310 2024-09-17T18:21:00.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.241+0000 7f4a1ffff640 1 -- 172.21.15.121:0/40509502 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f4a28012bf0 con 0x7f4a3809e550 2024-09-17T18:21:00.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.241+0000 7f4a1ffff640 1 --2- 172.21.15.121:0/40509502 >> v2:172.21.15.173:6812/2423786037 conn(0x7f4a240401a0 0x7f4a24042660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:00.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.241+0000 7f4a36ffd640 1 --2- 172.21.15.121:0/40509502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a38136310 0x7f4a3802e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:00.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.241+0000 7f4a1ffff640 1 -- 172.21.15.121:0/40509502 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(76..76 src has 1..76) ==== 6963+0+0 (secure 0 0 0) 0x7f4a280543f0 con 0x7f4a3809e550 2024-09-17T18:21:00.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.241+0000 7f4a1ffff640 10 client.34109.objecter ms_dispatch 0x7f4a38000b90 osd_map(76..76 src has 1..76) 2024-09-17T18:21:00.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.241+0000 7f4a1ffff640 3 client.34109.objecter handle_osd_map got epochs [76,76] > 0 2024-09-17T18:21:00.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.241+0000 7f4a1ffff640 3 client.34109.objecter handle_osd_map decoding full epoch 76 2024-09-17T18:21:00.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.241+0000 7f4a36ffd640 10 monclient: get_auth_request con 0x7f4a38136310 auth_method 0 2024-09-17T18:21:00.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.241+0000 7f4a36ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:00.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.241+0000 7f4a36ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:00.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.241+0000 7f4a36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:00.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.241+0000 7f4a36ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:00.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.241+0000 7f4a36ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:00.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.241+0000 7f4a36ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:00.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.241+0000 7f4a1ffff640 20 client.34109.objecter dump_active .. 0 homeless 2024-09-17T18:21:00.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.241+0000 7f4a37fff640 1 --2- 172.21.15.121:0/40509502 >> v2:172.21.15.173:6812/2423786037 conn(0x7f4a240401a0 0x7f4a24042660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:00.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.241+0000 7f4a37fff640 10 monclient: get_auth_request con 0x7f4a240401a0 auth_method 0 2024-09-17T18:21:00.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.241+0000 7f4a36ffd640 10 monclient(hunting): handle_auth_done global_id 24139 payload 824 2024-09-17T18:21:00.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.242+0000 7f4a36ffd640 1 -- 172.21.15.121:0/40509502 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4a3809e930 con 0x7f4a38136310 2024-09-17T18:21:00.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.244+0000 7f4a1ffff640 10 client.34109.objecter ms_handle_connect 0x7f4a240401a0 2024-09-17T18:21:00.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.244+0000 7f4a37fff640 1 --2- 172.21.15.121:0/40509502 >> v2:172.21.15.173:6812/2423786037 conn(0x7f4a240401a0 0x7f4a24042660 secure :-1 s=READY pgs=3860 cs=0 l=1 rev1=1 crypto rx=0x7f4a3809e0a0 tx=0x7f4a3002c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:21:00.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.244+0000 7f4a1ffff640 10 client.34109.objecter ms_handle_connect 0x7f4a38136310 2024-09-17T18:21:00.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.244+0000 7f4a1ffff640 10 client.34109.objecter resend_mon_ops 2024-09-17T18:21:00.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.244+0000 7f4a36ffd640 1 -- 172.21.15.121:0/40509502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a38136310 msgr2=0x7f4a3802e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:21:00.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.244+0000 7f4a36ffd640 1 --2- 172.21.15.121:0/40509502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a38136310 0x7f4a3802e5c0 secure :-1 s=READY pgs=7094 cs=0 l=1 rev1=1 crypto rx=0x7f4a381259e0 tx=0x7f4a2c00f610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:21:00.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.244+0000 7f4a1ffff640 1 -- 172.21.15.121:0/40509502 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c019070 con 0x7f4a38136310 2024-09-17T18:21:00.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.244+0000 7f4a1ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:00.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.244+0000 7f4a1ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:21:00.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.244+0000 7f4a1ffff640 10 monclient: dump: 2024-09-17T18:21:00.268 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:00.268 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:00.268 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:00.268 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:00.268 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:00.268 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:00.268 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:00.268 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:00.268 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:00.268 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:00.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.244+0000 7f4a1ffff640 1 -- 172.21.15.121:0/40509502 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4a2c004240 con 0x7f4a38136310 2024-09-17T18:21:00.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.244+0000 7f4a1ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:21:00.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.244+0000 7f4a1ffff640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:21:00.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.244+0000 7f4a1ffff640 1 -- 172.21.15.121:0/40509502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a38136310 msgr2=0x7f4a3802e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:00.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.244+0000 7f4a1ffff640 1 --2- 172.21.15.121:0/40509502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a38136310 0x7f4a3802e5c0 secure :-1 s=READY pgs=7094 cs=0 l=1 rev1=1 crypto rx=0x7f4a381259e0 tx=0x7f4a2c00f610 comp rx=0 tx=0).stop 2024-09-17T18:21:00.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.245+0000 7f4a1ffff640 1 -- 172.21.15.121:0/40509502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a38136310 msgr2=0x7f4a3802e5c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:21:00.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.245+0000 7f4a1ffff640 1 --2- 172.21.15.121:0/40509502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a38136310 0x7f4a3802e5c0 unknown :-1 s=CLOSED pgs=7094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:00.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.251+0000 7f4a3e050640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:21:00.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.251+0000 7f4a3e050640 1 --2- 172.21.15.121:0/40509502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a381cab10 0x7f4a38063d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:00.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.251+0000 7f4a3e050640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:00.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.251+0000 7f4a3e050640 1 -- 172.21.15.121:0/40509502 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4a381caf20 con 0x7f4a381cab10 2024-09-17T18:21:00.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.251+0000 7f4a36ffd640 1 --2- 172.21.15.121:0/40509502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a381cab10 0x7f4a38063d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:00.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.251+0000 7f4a36ffd640 10 monclient: get_auth_request con 0x7f4a381cab10 auth_method 0 2024-09-17T18:21:00.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.251+0000 7f4a36ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:00.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.251+0000 7f4a36ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:00.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.251+0000 7f4a36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:00.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.251+0000 7f4a36ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:00.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.251+0000 7f4a36ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:00.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.251+0000 7f4a36ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:00.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.251+0000 7f4a36ffd640 10 monclient(hunting): handle_auth_done global_id 24142 payload 824 2024-09-17T18:21:00.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.251+0000 7f4a36ffd640 1 -- 172.21.15.121:0/40509502 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4a380707e0 con 0x7f4a381cab10 2024-09-17T18:21:00.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.254+0000 7f4a36ffd640 1 -- 172.21.15.121:0/40509502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a381cab10 msgr2=0x7f4a38063d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:21:00.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.254+0000 7f4a36ffd640 1 --2- 172.21.15.121:0/40509502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a381cab10 0x7f4a38063d50 secure :-1 s=READY pgs=7095 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c01b000 tx=0x7f4a2c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:21:00.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.254+0000 7f4a1ffff640 10 client.34109.objecter ms_handle_connect 0x7f4a381cab10 2024-09-17T18:21:00.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.254+0000 7f4a1ffff640 10 client.34109.objecter resend_mon_ops 2024-09-17T18:21:00.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.254+0000 7f4a1ffff640 1 -- 172.21.15.121:0/40509502 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c019040 con 0x7f4a381cab10 2024-09-17T18:21:00.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.254+0000 7f4a1ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:00.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.254+0000 7f4a1ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:21:00.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.254+0000 7f4a1ffff640 10 monclient: dump: 2024-09-17T18:21:00.272 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:00.272 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:00.272 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:00.272 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:00.272 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:00.273 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:00.273 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:00.273 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:00.273 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:00.273 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:00.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.254+0000 7f4a1ffff640 1 -- 172.21.15.121:0/40509502 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4888 (secure 0 0 0) 0x7f4a2c01e950 con 0x7f4a381cab10 2024-09-17T18:21:00.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.254+0000 7f4a1ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:21:00.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.254+0000 7f4a1ffff640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:21:00.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.254+0000 7f4a1ffff640 1 -- 172.21.15.121:0/40509502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a381cab10 msgr2=0x7f4a38063d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:00.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.254+0000 7f4a1ffff640 1 --2- 172.21.15.121:0/40509502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a381cab10 0x7f4a38063d50 secure :-1 s=READY pgs=7095 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c01b000 tx=0x7f4a2c0046c0 comp rx=0 tx=0).stop 2024-09-17T18:21:00.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.254+0000 7f4a1ffff640 1 -- 172.21.15.121:0/40509502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a381cab10 msgr2=0x7f4a38063d50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:21:00.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.254+0000 7f4a1ffff640 1 --2- 172.21.15.121:0/40509502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a381cab10 0x7f4a38063d50 secure :-1 s=CLOSED pgs=7095 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c01b000 tx=0x7f4a2c0046c0 comp rx=0 tx=0).stop 2024-09-17T18:21:00.274 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:21:00.274 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:21:00.274 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:21:00.274 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:21:00.274 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 10, 2024-09-17T18:21:00.274 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 133201, 2024-09-17T18:21:00.274 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:21:00.274 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:21:00.274 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:21:00.275 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:21:00.275 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:21:00.275 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 4, 2024-09-17T18:21:00.275 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:21:00.275 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:21:00.275 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:21:00.275 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:21:00.275 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:21:00.275 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:21:00.275 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:21:00.275 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:21:00.275 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:21:00.275 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:21:00.275 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:21:00.275 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:21:00.276 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:21:00.276 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:21:00.276 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:21:00.276 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:21:00.276 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:21:00.276 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:21:00.276 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:21:00.276 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:21:00.276 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:21:00.276 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:21:00.276 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:21:00.276 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:21:00.276 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:21:00.276 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:21:00.277 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:21:00.277 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:21:00.277 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:21:00.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:00.277 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:21:00.277 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:21:00.277 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:21:00.277 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:21:00.277 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:21:00.277 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:21:00.277 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:21:00.277 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:21:00.277 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:21:00.277 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:21:00.278 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:21:00.278 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:21:00.278 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:21:00.278 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:21:00.278 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:21:00.278 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:21:00.278 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:21:00.278 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:21:00.278 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:21:00.278 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:21:00.278 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:21:00.278 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:21:00.278 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:21:00.279 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:21:00.279 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:21:00.279 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:21:00.279 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:21:00.279 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:21:00.279 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:21:00.279 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:21:00.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:00.279 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:21:00.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:00.279 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:21:00.279 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:21:00.279 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:21:00.279 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:21:00.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:00.280 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:21:00.280 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:21:00.280 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:21:00.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:00.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:00.280 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:21:00.280 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:21:00.280 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:21:00.280 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:21:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:21:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:21:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:21:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:21:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:21:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:21:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:21:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:21:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:21:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:00.282 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:21:00.282 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:21:00.282 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:21:00.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:00.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:00.282 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:21:00.282 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:21:00.282 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:21:00.282 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:00.282 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:21:00.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:00.282 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:21:00.282 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:21:00.282 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:21:00.283 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:21:00.283 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:21:00.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:00.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:00.283 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:21:00.283 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:21:00.283 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:21:00.283 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:21:00.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:00.283 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:21:00.283 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:21:00.283 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:21:00.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:00.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:00.284 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:21:00.284 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:21:00.284 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:21:00.284 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:00.284 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:21:00.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:00.284 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:21:00.284 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:21:00.284 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:21:00.284 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:21:00.284 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:21:00.284 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:00.284 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:21:00.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:00.285 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:21:00.285 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:21:00.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:00.285 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:21:00.285 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:21:00.285 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:21:00.285 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:00.285 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:21:00.285 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:21:00.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:00.285 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:21:00.285 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:21:00.285 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 4 2024-09-17T18:21:00.285 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:00.286 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:21:00.286 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:21:00.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:00.286 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:21:00.286 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:21:00.286 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 11 2024-09-17T18:21:00.286 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:00.286 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:21:00.286 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": [ 2024-09-17T18:21:00.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:00.286 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:21:00.287 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:21:00.287 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:21:00.287 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:00.287 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:21:00.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:00.287 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:21:00.287 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:21:00.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.255+0000 7f4a3e050640 10 librados: watch_flush enter 2024-09-17T18:21:00.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.255+0000 7f4a3e050640 10 librados: watch_flush exit 2024-09-17T18:21:00.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.255+0000 7f4a3e050640 20 client.34109.objecter shutdown clearing up homeless session... 2024-09-17T18:21:00.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.255+0000 7f4a3e050640 10 client.34109.objecter successfully canceled tick 2024-09-17T18:21:00.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.255+0000 7f4a3e050640 1 -- 172.21.15.121:0/40509502 >> v2:172.21.15.173:6812/2423786037 conn(0x7f4a240401a0 msgr2=0x7f4a24042660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:00.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.255+0000 7f4a3e050640 1 --2- 172.21.15.121:0/40509502 >> v2:172.21.15.173:6812/2423786037 conn(0x7f4a240401a0 0x7f4a24042660 secure :-1 s=READY pgs=3860 cs=0 l=1 rev1=1 crypto rx=0x7f4a3809e0a0 tx=0x7f4a3002c000 comp rx=0 tx=0).stop 2024-09-17T18:21:00.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.255+0000 7f4a3e050640 10 monclient: shutdown 2024-09-17T18:21:00.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.255+0000 7f4a3e050640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:21:00.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.255+0000 7f4a3e050640 1 -- 172.21.15.121:0/40509502 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a3809e550 msgr2=0x7f4a38135dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:00.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.255+0000 7f4a3e050640 1 --2- 172.21.15.121:0/40509502 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a3809e550 0x7f4a38135dd0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4a2800cb90 tx=0x7f4a2800a430 comp rx=0 tx=0).stop 2024-09-17T18:21:00.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.255+0000 7f4a37fff640 1 -- 172.21.15.121:0/40509502 reap_dead start 2024-09-17T18:21:00.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.255+0000 7f4a37fff640 1 -- 172.21.15.121:0/40509502 reap_dead start 2024-09-17T18:21:00.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.255+0000 7f4a3e050640 1 -- 172.21.15.121:0/40509502 shutdown_connections 2024-09-17T18:21:00.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.255+0000 7f4a3e050640 1 -- 172.21.15.121:0/40509502 >> 172.21.15.121:0/40509502 conn(0x7f4a380a7830 msgr2=0x7f4a3813a630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:21:00.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.255+0000 7f4a3e050640 1 -- 172.21.15.121:0/40509502 shutdown_connections 2024-09-17T18:21:00.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.255+0000 7f4a3e050640 1 -- 172.21.15.121:0/40509502 wait complete. 2024-09-17T18:21:00.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.255+0000 7f4a3e050640 1 librados: shutdown 2024-09-17T18:21:00.294 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:21:00.294 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-17T18:21:00.294 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-17T18:21:00.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.388+0000 7fc965bc6640 10 monclient: get_monmap_and_config 2024-09-17T18:21:00.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.388+0000 7fc965bc6640 10 monclient: build_initial_monmap 2024-09-17T18:21:00.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.389+0000 7fc965bc6640 10 monclient: monmap: 2024-09-17T18:21:00.392 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:21:00.392 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:00.392 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:21:00.390604+0000 2024-09-17T18:21:00.392 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:21:00.390604+0000 2024-09-17T18:21:00.392 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:21:00.392 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:00.393 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:21:00.393 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:21:00.393 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:21:00.393 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:21:00.393 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:21:00.393 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:00.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.389+0000 7fc965bc6640 1 Processor -- start 2024-09-17T18:21:00.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.389+0000 7fc965bc6640 1 -- start start 2024-09-17T18:21:00.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.389+0000 7fc965bc6640 10 monclient: init 2024-09-17T18:21:00.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.390+0000 7fc965bc6640 10 monclient: _reopen_session rank -1 2024-09-17T18:21:00.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.390+0000 7fc965bc6640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-17T18:21:00.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.390+0000 7fc965bc6640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9600738a0 0x7fc960138a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:00.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.390+0000 7fc965bc6640 10 monclient(hunting): picked mon.noname-b con 0x7fc9600738a0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:21:00.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.390+0000 7fc965bc6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc960073c80 addr v1:172.21.15.173:6789/0 2024-09-17T18:21:00.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.390+0000 7fc965bc6640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fc9600740c0 0x7fc9600a1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:00.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.390+0000 7fc965bc6640 10 monclient(hunting): picked mon.noname-c con 0x7fc9600740c0 addr v2:172.21.15.173:3300/0 2024-09-17T18:21:00.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.390+0000 7fc965bc6640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc96006fd90 con 0x7fc960073c80 2024-09-17T18:21:00.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.390+0000 7fc965bc6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:00.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.390+0000 7fc965bc6640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fc960070350 con 0x7fc9600740c0 2024-09-17T18:21:00.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.390+0000 7fc965bc6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:00.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.390+0000 7fc965bc6640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc960070070 con 0x7fc9600738a0 2024-09-17T18:21:00.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.390+0000 7fc965bc6640 10 monclient(hunting): _renew_subs 2024-09-17T18:21:00.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.390+0000 7fc965bc6640 10 monclient(hunting): authenticate will time out at 864.107301s 2024-09-17T18:21:00.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.390+0000 7fc95f7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9600738a0 0x7fc960138a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:00.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.390+0000 7fc95f7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9600738a0 0x7fc960138a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:56414/0 (socket says 172.21.15.121:56414) 2024-09-17T18:21:00.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.390+0000 7fc95f7fe640 1 -- 172.21.15.121:0/3686065006 learned_addr learned my addr 172.21.15.121:0/3686065006 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:21:00.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.390+0000 7fc95f7fe640 10 monclient(hunting): get_auth_request con 0x7fc9600738a0 auth_method 0 2024-09-17T18:21:00.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.390+0000 7fc95f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:00.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.390+0000 7fc95f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:00.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.390+0000 7fc95f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:00.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.390+0000 7fc95f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:00.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.390+0000 7fc95f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:00.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.390+0000 7fc95f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:00.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.390+0000 7fc95ffff640 1 --2- 172.21.15.121:0/3686065006 >> v2:172.21.15.173:3300/0 conn(0x7fc9600740c0 0x7fc9600a1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:00.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.390+0000 7fc95f7fe640 10 monclient(hunting): handle_auth_done global_id 34112 payload 274 2024-09-17T18:21:00.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.390+0000 7fc95f7fe640 1 -- 172.21.15.121:0/3686065006 >> v2:172.21.15.173:3300/0 conn(0x7fc9600740c0 msgr2=0x7fc9600a1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:21:00.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.390+0000 7fc95f7fe640 1 --2- 172.21.15.121:0/3686065006 >> v2:172.21.15.173:3300/0 conn(0x7fc9600740c0 0x7fc9600a1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:00.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.390+0000 7fc95f7fe640 1 -- 172.21.15.121:0/3686065006 >> v1:172.21.15.173:6789/0 conn(0x7fc960073c80 legacy=0x7fc9600a0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:21:00.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.390+0000 7fc95f7fe640 10 monclient: _finish_hunting 0 2024-09-17T18:21:00.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.390+0000 7fc95f7fe640 1 monclient: found mon.noname-b 2024-09-17T18:21:00.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.390+0000 7fc95f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:21:00.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.390+0000 7fc95f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:21:00.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.390+0000 7fc95f7fe640 1 -- 172.21.15.121:0/3686065006 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9600a4020 con 0x7fc9600738a0 2024-09-17T18:21:00.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.390+0000 7fc95ffff640 10 monclient: get_auth_request con 0x7fc9600740c0 auth_method 0 2024-09-17T18:21:00.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.390+0000 7fc95ffff640 1 --2- 172.21.15.121:0/3686065006 >> v2:172.21.15.173:3300/0 conn(0x7fc9600740c0 0x7fc9600a1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T18:21:00.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.393+0000 7fc95f7fe640 1 -- 172.21.15.121:0/3686065006 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9600738a0 msgr2=0x7fc960138a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:21:00.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.393+0000 7fc95f7fe640 1 --2- 172.21.15.121:0/3686065006 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9600738a0 0x7fc960138a40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc950009e50 tx=0x7fc95002a560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d10739a51e997f98 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:21:00.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.393+0000 7fc95e7fc640 1 -- 172.21.15.121:0/3686065006 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc950036070 con 0x7fc9600738a0 2024-09-17T18:21:00.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.393+0000 7fc95e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:00.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.393+0000 7fc95e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:21:00.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.393+0000 7fc95e7fc640 10 monclient: dump: 2024-09-17T18:21:00.406 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:00.407 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:00.407 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:00.407 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:00.407 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:00.407 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:00.407 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:00.407 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:00.407 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:00.407 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:00.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.393+0000 7fc95e7fc640 10 monclient: _finish_auth 0 2024-09-17T18:21:00.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.393+0000 7fc95e7fc640 10 monclient: _check_auth_tickets 2024-09-17T18:21:00.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.393+0000 7fc95e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:21:00.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.393+0000 7fc95e7fc640 1 -- 172.21.15.121:0/3686065006 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc950007800 con 0x7fc9600738a0 2024-09-17T18:21:00.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.393+0000 7fc965bc6640 5 monclient: authenticate success, global_id 34112 2024-09-17T18:21:00.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.393+0000 7fc965bc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:21:00.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.393+0000 7fc95e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:21:00.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.393+0000 7fc95e7fc640 1 -- 172.21.15.121:0/3686065006 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc950007bd0 con 0x7fc9600738a0 2024-09-17T18:21:00.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.393+0000 7fc95e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:00.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.393+0000 7fc95e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:21:00.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.393+0000 7fc95e7fc640 10 monclient: dump: 2024-09-17T18:21:00.408 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:00.408 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:00.409 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:00.409 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:00.409 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:00.409 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:00.409 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:00.409 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:00.409 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:00.409 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:00.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.393+0000 7fc965bc6640 10 monclient: get_monmap_and_config success 2024-09-17T18:21:00.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.393+0000 7fc965bc6640 4 set_mon_vals no callback set 2024-09-17T18:21:00.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.394+0000 7fc965bc6640 10 monclient: shutdown 2024-09-17T18:21:00.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.394+0000 7fc965bc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:21:00.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.394+0000 7fc965bc6640 1 -- 172.21.15.121:0/3686065006 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9600738a0 msgr2=0x7fc960138a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:00.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.394+0000 7fc965bc6640 1 --2- 172.21.15.121:0/3686065006 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9600738a0 0x7fc960138a40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc950009e50 tx=0x7fc95002a560 comp rx=0 tx=0).stop 2024-09-17T18:21:00.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.394+0000 7fc965bc6640 1 -- 172.21.15.121:0/3686065006 shutdown_connections 2024-09-17T18:21:00.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.394+0000 7fc965bc6640 1 --2- 172.21.15.121:0/3686065006 >> v2:172.21.15.173:3300/0 conn(0x7fc9600740c0 0x7fc9600a1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:00.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.394+0000 7fc965bc6640 1 --2- 172.21.15.121:0/3686065006 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9600738a0 0x7fc960138a40 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:00.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.394+0000 7fc965bc6640 1 -- 172.21.15.121:0/3686065006 >> 172.21.15.121:0/3686065006 conn(0x7fc960076f00 msgr2=0x7fc960077310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:21:00.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.394+0000 7fc965bc6640 1 -- 172.21.15.121:0/3686065006 shutdown_connections 2024-09-17T18:21:00.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.394+0000 7fc965bc6640 1 -- 172.21.15.121:0/3686065006 wait complete. 2024-09-17T18:21:00.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.394+0000 7fc965bc6640 10 monclient: build_initial_monmap 2024-09-17T18:21:00.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.394+0000 7fc965bc6640 10 monclient: monmap: 2024-09-17T18:21:00.410 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:21:00.410 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:21:00.410 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:21:00.410 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:21:00.410 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:21:00.410 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:00.411 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:21:00.411 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:21:00.411 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:21:00.411 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:00.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.395+0000 7fc965bc6640 1 librados: starting msgr at 2024-09-17T18:21:00.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.395+0000 7fc965bc6640 1 librados: starting objecter 2024-09-17T18:21:00.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.395+0000 7fc965bc6640 1 Processor -- start 2024-09-17T18:21:00.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.395+0000 7fc965bc6640 1 -- start start 2024-09-17T18:21:00.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.395+0000 7fc965bc6640 1 librados: setting wanted keys 2024-09-17T18:21:00.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.395+0000 7fc965bc6640 1 librados: calling monclient init 2024-09-17T18:21:00.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.395+0000 7fc965bc6640 10 monclient: init 2024-09-17T18:21:00.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.395+0000 7fc965bc6640 10 monclient: _reopen_session rank -1 2024-09-17T18:21:00.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.395+0000 7fc965bc6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T18:21:00.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.395+0000 7fc965bc6640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9600738a0 0x7fc9601cd740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:00.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.395+0000 7fc965bc6640 10 monclient(hunting): picked mon.noname-c con 0x7fc9600738a0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:21:00.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.395+0000 7fc965bc6640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc960073c80 0x7fc9601351f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:00.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.395+0000 7fc965bc6640 10 monclient(hunting): picked mon.noname-b con 0x7fc960073c80 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:21:00.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.395+0000 7fc965bc6640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc9600740c0 0x7fc960135730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:00.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.395+0000 7fc965bc6640 10 monclient(hunting): picked mon.noname-a con 0x7fc9600740c0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:21:00.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.395+0000 7fc965bc6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:00.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.395+0000 7fc965bc6640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc96006d820 con 0x7fc9600740c0 2024-09-17T18:21:00.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.395+0000 7fc965bc6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:00.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.395+0000 7fc965bc6640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc96006d6a0 con 0x7fc960073c80 2024-09-17T18:21:00.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.395+0000 7fc965bc6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:00.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.395+0000 7fc965bc6640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc9600a1ac0 con 0x7fc9600738a0 2024-09-17T18:21:00.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.395+0000 7fc965bc6640 10 monclient(hunting): _renew_subs 2024-09-17T18:21:00.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.395+0000 7fc965bc6640 10 monclient(hunting): authenticate will time out at 864.112862s 2024-09-17T18:21:00.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.395+0000 7fc95f7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9600738a0 0x7fc9601cd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:00.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.395+0000 7fc95f7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9600738a0 0x7fc9601cd740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:56416/0 (socket says 172.21.15.121:56416) 2024-09-17T18:21:00.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.395+0000 7fc95f7fe640 1 -- 172.21.15.121:0/3008156301 learned_addr learned my addr 172.21.15.121:0/3008156301 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:21:00.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.395+0000 7fc95f7fe640 10 monclient(hunting): get_auth_request con 0x7fc9600738a0 auth_method 0 2024-09-17T18:21:00.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.395+0000 7fc95f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:00.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.395+0000 7fc95f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:00.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.395+0000 7fc95f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:00.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.395+0000 7fc95ffff640 1 --2- 172.21.15.121:0/3008156301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc9600740c0 0x7fc960135730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:00.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.396+0000 7fc95ffff640 10 monclient(hunting): get_auth_request con 0x7fc9600740c0 auth_method 0 2024-09-17T18:21:00.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.396+0000 7fc95ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:00.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.396+0000 7fc95ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:00.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.396+0000 7fc95ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:00.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.396+0000 7fc95f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:00.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.396+0000 7fc95f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:00.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.396+0000 7fc95f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:00.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.396+0000 7fc95ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:00.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.396+0000 7fc95ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:00.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.396+0000 7fc95ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:00.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.396+0000 7fc95f7fe640 10 monclient(hunting): handle_auth_done global_id 34115 payload 824 2024-09-17T18:21:00.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.396+0000 7fc95effd640 1 --2- 172.21.15.121:0/3008156301 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc960073c80 0x7fc9601351f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:00.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.396+0000 7fc95f7fe640 1 -- 172.21.15.121:0/3008156301 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc960073c80 msgr2=0x7fc9601351f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:00.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.396+0000 7fc95f7fe640 1 --2- 172.21.15.121:0/3008156301 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc960073c80 0x7fc9601351f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:00.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.396+0000 7fc95f7fe640 1 -- 172.21.15.121:0/3008156301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc9600740c0 msgr2=0x7fc960135730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:00.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.396+0000 7fc95f7fe640 1 --2- 172.21.15.121:0/3008156301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc9600740c0 0x7fc960135730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:00.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.396+0000 7fc95f7fe640 10 monclient: _finish_hunting 0 2024-09-17T18:21:00.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.396+0000 7fc95f7fe640 1 monclient: found mon.noname-c 2024-09-17T18:21:00.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.396+0000 7fc95f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:21:00.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.396+0000 7fc95f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:21:00.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.396+0000 7fc95f7fe640 1 -- 172.21.15.121:0/3008156301 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc960122bb0 con 0x7fc9600738a0 2024-09-17T18:21:00.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.396+0000 7fc95effd640 10 monclient: get_auth_request con 0x7fc960073c80 auth_method 0 2024-09-17T18:21:00.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.396+0000 7fc95effd640 1 --2- 172.21.15.121:0/3008156301 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc960073c80 0x7fc9601351f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T18:21:00.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.398+0000 7fc95f7fe640 1 -- 172.21.15.121:0/3008156301 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9600738a0 msgr2=0x7fc9601cd740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:21:00.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.398+0000 7fc95f7fe640 1 --2- 172.21.15.121:0/3008156301 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9600738a0 0x7fc9601cd740 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc9500093e0 tx=0x7fc950004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:21:00.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.398+0000 7fc947fff640 10 client.?.objecter ms_handle_connect 0x7fc9600738a0 2024-09-17T18:21:00.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.398+0000 7fc947fff640 10 client.?.objecter resend_mon_ops 2024-09-17T18:21:00.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.398+0000 7fc947fff640 1 -- 172.21.15.121:0/3008156301 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc950036050 con 0x7fc9600738a0 2024-09-17T18:21:00.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.398+0000 7fc947fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:00.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.398+0000 7fc947fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:21:00.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.398+0000 7fc947fff640 10 monclient: dump: 2024-09-17T18:21:00.415 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:00.415 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:00.415 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:00.416 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:00.416 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:00.416 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:00.416 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:00.416 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:00.416 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:00.416 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:00.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.398+0000 7fc947fff640 10 monclient: _finish_auth 0 2024-09-17T18:21:00.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.398+0000 7fc947fff640 10 monclient: _check_auth_tickets 2024-09-17T18:21:00.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.398+0000 7fc947fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:21:00.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.398+0000 7fc947fff640 1 -- 172.21.15.121:0/3008156301 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc950042070 con 0x7fc9600738a0 2024-09-17T18:21:00.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.398+0000 7fc965bc6640 5 monclient: authenticate success, global_id 34115 2024-09-17T18:21:00.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.398+0000 7fc947fff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:21:00.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.398+0000 7fc947fff640 1 -- 172.21.15.121:0/3008156301 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc950031e10 con 0x7fc9600738a0 2024-09-17T18:21:00.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.398+0000 7fc965bc6640 10 monclient: _renew_subs 2024-09-17T18:21:00.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.398+0000 7fc965bc6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:21:00.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.398+0000 7fc965bc6640 1 -- 172.21.15.121:0/3008156301 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc960120850 con 0x7fc9600738a0 2024-09-17T18:21:00.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.398+0000 7fc947fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:00.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.398+0000 7fc947fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:21:00.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.398+0000 7fc947fff640 10 monclient: dump: 2024-09-17T18:21:00.417 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:00.417 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:00.417 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:00.417 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:00.417 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:00.417 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:00.417 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:00.417 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:00.417 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:00.418 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:00.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.398+0000 7fc95dffb640 4 set_mon_vals no callback set 2024-09-17T18:21:00.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.398+0000 7fc965bc6640 10 client.34115.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:21:00.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.398+0000 7fc965bc6640 10 monclient: _renew_subs 2024-09-17T18:21:00.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.398+0000 7fc965bc6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:21:00.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.398+0000 7fc965bc6640 1 -- 172.21.15.121:0/3008156301 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc960122310 con 0x7fc9600738a0 2024-09-17T18:21:00.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.398+0000 7fc965bc6640 1 librados: init done 2024-09-17T18:21:00.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.399+0000 7fc947fff640 1 -- 172.21.15.121:0/3008156301 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fc950047460 con 0x7fc9600738a0 2024-09-17T18:21:00.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.399+0000 7fc965bc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:21:00.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.399+0000 7fc965bc6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:21:00.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.399+0000 7fc965bc6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:21:00.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.399+0000 7fc965bc6640 1 -- 172.21.15.121:0/3008156301 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc960138a40 con 0x7fc9600738a0 2024-09-17T18:21:00.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.399+0000 7fc947fff640 1 --2- 172.21.15.121:0/3008156301 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc94c040170 0x7fc94c042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:00.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.399+0000 7fc947fff640 1 -- 172.21.15.121:0/3008156301 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(76..76 src has 1..76) ==== 6963+0+0 (secure 0 0 0) 0x7fc950074f80 con 0x7fc9600738a0 2024-09-17T18:21:00.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.399+0000 7fc947fff640 10 client.34115.objecter ms_dispatch 0x7fc960000b90 osd_map(76..76 src has 1..76) 2024-09-17T18:21:00.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.399+0000 7fc947fff640 3 client.34115.objecter handle_osd_map got epochs [76,76] > 0 2024-09-17T18:21:00.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.399+0000 7fc947fff640 3 client.34115.objecter handle_osd_map decoding full epoch 76 2024-09-17T18:21:00.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.399+0000 7fc947fff640 20 client.34115.objecter dump_active .. 0 homeless 2024-09-17T18:21:00.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.400+0000 7fc95effd640 1 --2- 172.21.15.121:0/3008156301 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc94c040170 0x7fc94c042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:00.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.400+0000 7fc95effd640 10 monclient: get_auth_request con 0x7fc94c040170 auth_method 0 2024-09-17T18:21:00.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.402+0000 7fc947fff640 10 client.34115.objecter ms_handle_connect 0x7fc94c040170 2024-09-17T18:21:00.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.402+0000 7fc95effd640 1 --2- 172.21.15.121:0/3008156301 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc94c040170 0x7fc94c042630 secure :-1 s=READY pgs=3861 cs=0 l=1 rev1=1 crypto rx=0x7fc960122240 tx=0x7fc954009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:21:00.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.404+0000 7fc947fff640 1 -- 172.21.15.121:0/3008156301 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc9500730f0 con 0x7fc9600738a0 2024-09-17T18:21:00.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.404+0000 7fc947fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:21:00.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.404+0000 7fc947fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:21:00.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.663+0000 7fc965bc6640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-17T18:21:00.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.663+0000 7fc965bc6640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-17T18:21:00.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.663+0000 7fc965bc6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:21:00.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.663+0000 7fc965bc6640 1 -- 172.21.15.121:0/3008156301 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fc960122830 con 0x7fc9600738a0 2024-09-17T18:21:00.668 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.665+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:21:00.668 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.665+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=12,kv=13,logm=63} crc {auth=3842090736,config=2586702516,health=77907301,kv=2079569364,logm=397903827}) 2024-09-17T18:21:00.668 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.665+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=9,config=2,health=12,kv=14,logm=63} crc {auth=1742902937,config=2586702516,health=77907301,kv=2790417503,logm=2453737521}) 2024-09-17T18:21:00.668 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.665+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:21:00.668 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.665+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=12,kv=13,logm=63} crc {auth=3842090736,config=2586702516,health=77907301,kv=2079569364,logm=397903827}) 2024-09-17T18:21:00.668 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.665+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=10,config=2,health=12,kv=12,logm=64} crc {auth=3842090736,config=2586702516,health=69248957,kv=1513842021,logm=3176205272}) 2024-09-17T18:21:00.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.665+0000 7fc947fff640 1 -- 172.21.15.121:0/3008156301 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fc95003d450 con 0x7fc9600738a0 2024-09-17T18:21:00.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.665+0000 7fc947fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-17T18:21:00.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.665+0000 7fc947fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:21:00.669 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.666+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:21:00.670 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.666+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2177250928}) 2024-09-17T18:21:00.670 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.666+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4162627325}) 2024-09-17T18:21:00.670 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.666+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:21:00.670 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.666+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2177250928}) 2024-09-17T18:21:00.670 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.666+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=103390036}) 2024-09-17T18:21:00.670 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.667+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:21:00.670 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.667+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2278011733}) 2024-09-17T18:21:00.670 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.667+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1939635684}) 2024-09-17T18:21:00.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.667+0000 7fc965bc6640 10 librados: watch_flush enter 2024-09-17T18:21:00.671 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.667+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:21:00.671 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.667+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2278011733}) 2024-09-17T18:21:00.671 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.667+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4178874506}) 2024-09-17T18:21:00.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.667+0000 7fc965bc6640 10 librados: watch_flush exit 2024-09-17T18:21:00.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.667+0000 7fc965bc6640 20 client.34115.objecter shutdown clearing up homeless session... 2024-09-17T18:21:00.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.667+0000 7fc965bc6640 10 client.34115.objecter successfully canceled tick 2024-09-17T18:21:00.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.667+0000 7fc965bc6640 1 -- 172.21.15.121:0/3008156301 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc94c040170 msgr2=0x7fc94c042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:00.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.667+0000 7fc965bc6640 1 --2- 172.21.15.121:0/3008156301 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc94c040170 0x7fc94c042630 secure :-1 s=READY pgs=3861 cs=0 l=1 rev1=1 crypto rx=0x7fc960122240 tx=0x7fc954009040 comp rx=0 tx=0).stop 2024-09-17T18:21:00.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.667+0000 7fc965bc6640 10 monclient: shutdown 2024-09-17T18:21:00.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.667+0000 7fc965bc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:21:00.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.667+0000 7fc965bc6640 1 -- 172.21.15.121:0/3008156301 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9600738a0 msgr2=0x7fc9601cd740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:00.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.667+0000 7fc965bc6640 1 --2- 172.21.15.121:0/3008156301 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9600738a0 0x7fc9601cd740 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc9500093e0 tx=0x7fc950004a40 comp rx=0 tx=0).stop 2024-09-17T18:21:00.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.667+0000 7fc965bc6640 1 -- 172.21.15.121:0/3008156301 shutdown_connections 2024-09-17T18:21:00.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.667+0000 7fc965bc6640 1 --2- 172.21.15.121:0/3008156301 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc94c040170 0x7fc94c042630 unknown :-1 s=CLOSED pgs=3861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:00.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.667+0000 7fc965bc6640 1 --2- 172.21.15.121:0/3008156301 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc960073c80 0x7fc9601351f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:00.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.667+0000 7fc965bc6640 1 --2- 172.21.15.121:0/3008156301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc9600740c0 0x7fc960135730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:00.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.667+0000 7fc965bc6640 1 --2- 172.21.15.121:0/3008156301 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9600738a0 0x7fc9601cd740 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:00.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.667+0000 7fc965bc6640 1 -- 172.21.15.121:0/3008156301 >> 172.21.15.121:0/3008156301 conn(0x7fc960076f00 msgr2=0x7fc960072810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:21:00.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.668+0000 7fc965bc6640 1 -- 172.21.15.121:0/3008156301 shutdown_connections 2024-09-17T18:21:00.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.668+0000 7fc965bc6640 1 -- 172.21.15.121:0/3008156301 wait complete. 2024-09-17T18:21:00.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:00.668+0000 7fc965bc6640 1 librados: shutdown 2024-09-17T18:21:00.674 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.668+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:21:00.675 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.668+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1733646318}) 2024-09-17T18:21:00.675 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.668+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2004631530}) 2024-09-17T18:21:00.675 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.668+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:21:00.675 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.668+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1733646318}) 2024-09-17T18:21:00.675 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.668+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1212162951}) 2024-09-17T18:21:00.675 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.669+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:21:00.675 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.669+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=215199872}) 2024-09-17T18:21:00.675 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.669+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1411199462}) 2024-09-17T18:21:00.675 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.669+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:21:00.675 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.669+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=215199872}) 2024-09-17T18:21:00.675 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.669+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1837645955}) 2024-09-17T18:21:00.679 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.676+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:21:00.679 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.676+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=723628676}) 2024-09-17T18:21:00.679 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.676+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3421301143}) 2024-09-17T18:21:00.679 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.676+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:21:00.679 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.676+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=723628676}) 2024-09-17T18:21:00.679 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.676+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2097715788}) 2024-09-17T18:21:00.680 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.677+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:21:00.680 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.677+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=154460176}) 2024-09-17T18:21:00.680 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.677+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1765753946}) 2024-09-17T18:21:00.680 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.677+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:21:00.680 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.677+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=154460176}) 2024-09-17T18:21:00.681 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.677+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1235201900}) 2024-09-17T18:21:00.681 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-17T18:21:00.683 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.678+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:21:00.683 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.678+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3694915875}) 2024-09-17T18:21:00.683 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.678+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1589630349}) 2024-09-17T18:21:00.683 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.678+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:21:00.683 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.678+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3694915875}) 2024-09-17T18:21:00.683 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.678+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1419690273}) 2024-09-17T18:21:00.683 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.679+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:21:00.683 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.679+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=769342888}) 2024-09-17T18:21:00.684 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.679+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1655846773}) 2024-09-17T18:21:00.684 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.679+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:21:00.684 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.679+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=769342888}) 2024-09-17T18:21:00.684 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.679+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=515115406}) 2024-09-17T18:21:00.684 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.680+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:21:00.684 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.680+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=30971819}) 2024-09-17T18:21:00.684 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.680+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=138535022}) 2024-09-17T18:21:00.684 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.680+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:21:00.684 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.680+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=30971819}) 2024-09-17T18:21:00.684 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.680+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=282075219}) 2024-09-17T18:21:00.685 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.681+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:21:00.685 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.681+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=491721849}) 2024-09-17T18:21:00.685 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.681+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2861909839}) 2024-09-17T18:21:00.685 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.681+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:21:00.685 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.681+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=491721849}) 2024-09-17T18:21:00.686 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.681+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2661232712}) 2024-09-17T18:21:00.686 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.682+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:21:00.686 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.682+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=355781650}) 2024-09-17T18:21:00.686 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.682+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4015006978}) 2024-09-17T18:21:00.686 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.682+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:21:00.686 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.682+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=355781650}) 2024-09-17T18:21:00.686 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.682+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=157553604}) 2024-09-17T18:21:00.697 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.694+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:21:00.697 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.694+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=10,mdsmap=3,mgr=7,mgr_command_descs=1,mgrstat=14,mon_config_key=17,monmap=2,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=13,osdmap=24} crc {logm=1419330333,mdsmap=761013554,mgr=1719742396,mgr_command_descs=1957073259,mgrstat=4223738316,mon_config_key=1582663684,monmap=344458555,nvmeofgw=598337686,osd_metadata=794661280,osd_pg_creating=568343529,osd_snap=3007519311,osdmap=1142583968}) 2024-09-17T18:21:00.697 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.694+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=12,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=13,osdmap=18} crc {logm=241549394,mdsmap=761013554,mgr=3361498112,mgr_command_descs=1957073259,mgr_metadata=402068389,mgrstat=4223738316,mon_config_key=1582663684,monmap=2277444362,nvmeofgw=598337686,osd_metadata=121115626,osd_pg_creating=568343529,osd_snap=2276565254,osdmap=3941498669}) 2024-09-17T18:21:00.697 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.694+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:21:00.697 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.694+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=10,mdsmap=3,mgr=7,mgr_command_descs=1,mgrstat=14,mon_config_key=17,monmap=2,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=13,osdmap=24} crc {logm=1419330333,mdsmap=761013554,mgr=1719742396,mgr_command_descs=1957073259,mgrstat=4223738316,mon_config_key=1582663684,monmap=344458555,nvmeofgw=598337686,osd_metadata=794661280,osd_pg_creating=568343529,osd_snap=3007519311,osdmap=1142583968}) 2024-09-17T18:21:00.697 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.694+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=11,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=16,monmap=2,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=13,osdmap=25} crc {logm=3574196078,mdsmap=3140514009,mgr=3361498112,mgr_command_descs=1957073259,mgr_metadata=402068389,mgrstat=2238467101,mon_config_key=3478702618,monmap=344458555,nvmeofgw=598337686,osd_metadata=794661279,osd_pg_creating=568343529,osd_snap=2934943733,osdmap=1265577492}) 2024-09-17T18:21:00.699 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.696+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:21:00.699 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.696+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=999989917}) 2024-09-17T18:21:00.699 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.696+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2171057726}) 2024-09-17T18:21:00.699 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.696+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:21:00.699 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.696+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=999989917}) 2024-09-17T18:21:00.699 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:21:00.696+0000 7fb3404d6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2280287167}) 2024-09-17T18:21:01.683 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T18:21:01.683 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:21:01.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.781+0000 7f5f1c823640 10 monclient: get_monmap_and_config 2024-09-17T18:21:01.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.781+0000 7f5f1c823640 10 monclient: build_initial_monmap 2024-09-17T18:21:01.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.781+0000 7f5f1c823640 10 monclient: monmap: 2024-09-17T18:21:01.785 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:21:01.785 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:01.785 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:21:01.783566+0000 2024-09-17T18:21:01.785 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:21:01.783566+0000 2024-09-17T18:21:01.785 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:21:01.785 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:01.785 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:21:01.785 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:21:01.785 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:21:01.785 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:21:01.785 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:21:01.786 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:01.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.782+0000 7f5f1c823640 1 Processor -- start 2024-09-17T18:21:01.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.782+0000 7f5f1c823640 1 -- start start 2024-09-17T18:21:01.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.782+0000 7f5f1c823640 10 monclient: init 2024-09-17T18:21:01.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.783+0000 7f5f1c823640 10 monclient: _reopen_session rank -1 2024-09-17T18:21:01.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.783+0000 7f5f1c823640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-17T18:21:01.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.783+0000 7f5f1c823640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f5f1809a600 0x7f5f1809c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:01.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.783+0000 7f5f1c823640 10 monclient(hunting): picked mon.noname-a con 0x7f5f1809a600 addr v2:172.21.15.121:3300/0 2024-09-17T18:21:01.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.783+0000 7f5f1c823640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5f1809a9e0 addr v1:172.21.15.173:6789/0 2024-09-17T18:21:01.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.783+0000 7f5f1c823640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f1809ae10 0x7f5f180a3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:01.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.783+0000 7f5f1c823640 10 monclient(hunting): picked mon.noname-b con 0x7f5f1809ae10 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:21:01.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.783+0000 7f5f1c823640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f1809b850 con 0x7f5f1809a9e0 2024-09-17T18:21:01.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.783+0000 7f5f1c823640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:01.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.783+0000 7f5f1c823640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f5f18077240 con 0x7f5f1809a600 2024-09-17T18:21:01.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.783+0000 7f5f1c823640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:01.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.783+0000 7f5f1c823640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5f1809b570 con 0x7f5f1809ae10 2024-09-17T18:21:01.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.783+0000 7f5f1c823640 10 monclient(hunting): _renew_subs 2024-09-17T18:21:01.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.783+0000 7f5f1c823640 10 monclient(hunting): authenticate will time out at 865.500315s 2024-09-17T18:21:01.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.783+0000 7f5f1659c640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f5f1809a600 0x7f5f1809c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:01.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.783+0000 7f5f16d9d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f1809ae10 0x7f5f180a3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:01.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.783+0000 7f5f1659c640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f5f1809a600 0x7f5f1809c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:55530/0 (socket says 172.21.15.121:55530) 2024-09-17T18:21:01.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.783+0000 7f5f1659c640 1 -- 172.21.15.121:0/1301105995 learned_addr learned my addr 172.21.15.121:0/1301105995 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:21:01.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.783+0000 7f5f1659c640 10 monclient(hunting): get_auth_request con 0x7f5f1809a600 auth_method 0 2024-09-17T18:21:01.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.783+0000 7f5f1659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:01.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.783+0000 7f5f1659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:01.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.783+0000 7f5f1659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:01.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.783+0000 7f5f16d9d640 10 monclient(hunting): get_auth_request con 0x7f5f1809ae10 auth_method 0 2024-09-17T18:21:01.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.783+0000 7f5f16d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:01.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.783+0000 7f5f16d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:01.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.783+0000 7f5f16d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:01.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.783+0000 7f5f1659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:01.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.783+0000 7f5f1659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:01.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.783+0000 7f5f1659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:01.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.783+0000 7f5f16d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:01.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.783+0000 7f5f16d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:01.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.783+0000 7f5f16d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:01.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.783+0000 7f5f16d9d640 10 monclient(hunting): handle_auth_done global_id 34133 payload 274 2024-09-17T18:21:01.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.783+0000 7f5f16d9d640 1 -- 172.21.15.121:0/1301105995 >> v2:172.21.15.121:3300/0 conn(0x7f5f1809a600 msgr2=0x7f5f1809c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:21:01.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.783+0000 7f5f16d9d640 1 --2- 172.21.15.121:0/1301105995 >> v2:172.21.15.121:3300/0 conn(0x7f5f1809a600 0x7f5f1809c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:01.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.783+0000 7f5f16d9d640 1 -- 172.21.15.121:0/1301105995 >> v1:172.21.15.173:6789/0 conn(0x7f5f1809a9e0 legacy=0x7f5f180a2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:21:01.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.783+0000 7f5f16d9d640 10 monclient: _finish_hunting 0 2024-09-17T18:21:01.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.783+0000 7f5f16d9d640 1 monclient: found mon.noname-b 2024-09-17T18:21:01.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.783+0000 7f5f16d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:21:01.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.783+0000 7f5f16d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:21:01.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.783+0000 7f5f16d9d640 1 -- 172.21.15.121:0/1301105995 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f1809bb30 con 0x7f5f1809ae10 2024-09-17T18:21:01.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.783+0000 7f5f1659c640 1 --2- 172.21.15.121:0/1301105995 >> v2:172.21.15.121:3300/0 conn(0x7f5f1809a600 0x7f5f1809c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:21:01.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.786+0000 7f5f16d9d640 1 -- 172.21.15.121:0/1301105995 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f1809ae10 msgr2=0x7f5f180a3560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:21:01.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.786+0000 7f5f16d9d640 1 --2- 172.21.15.121:0/1301105995 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f1809ae10 0x7f5f180a3560 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5f100097a0 tx=0x7f5f10029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc6febf6be1c68e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:21:01.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.786+0000 7f5f1559a640 1 -- 172.21.15.121:0/1301105995 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f10037070 con 0x7f5f1809ae10 2024-09-17T18:21:01.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.786+0000 7f5f1559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:01.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.786+0000 7f5f1559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:21:01.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.786+0000 7f5f1559a640 10 monclient: dump: 2024-09-17T18:21:01.799 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:01.799 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:01.799 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:01.799 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:01.799 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:01.799 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:01.800 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:01.800 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:01.800 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:01.800 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:01.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.786+0000 7f5f1559a640 10 monclient: _finish_auth 0 2024-09-17T18:21:01.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.786+0000 7f5f1559a640 10 monclient: _check_auth_tickets 2024-09-17T18:21:01.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.786+0000 7f5f1559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:21:01.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.786+0000 7f5f1559a640 1 -- 172.21.15.121:0/1301105995 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f10032440 con 0x7f5f1809ae10 2024-09-17T18:21:01.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.786+0000 7f5f1c823640 5 monclient: authenticate success, global_id 34133 2024-09-17T18:21:01.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.786+0000 7f5f1c823640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:21:01.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.786+0000 7f5f1559a640 10 monclient: handle_config config(0 keys) 2024-09-17T18:21:01.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.786+0000 7f5f1c823640 10 monclient: get_monmap_and_config success 2024-09-17T18:21:01.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.786+0000 7f5f1559a640 1 -- 172.21.15.121:0/1301105995 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f10032810 con 0x7f5f1809ae10 2024-09-17T18:21:01.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.786+0000 7f5f1559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:01.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.786+0000 7f5f1c823640 4 set_mon_vals no callback set 2024-09-17T18:21:01.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.786+0000 7f5f1559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:21:01.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.786+0000 7f5f1559a640 10 monclient: dump: 2024-09-17T18:21:01.801 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:01.801 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:01.801 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:01.801 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:01.801 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:01.801 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:01.801 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:01.801 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:01.801 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:01.801 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:01.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.787+0000 7f5f1c823640 10 monclient: shutdown 2024-09-17T18:21:01.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.787+0000 7f5f1c823640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:21:01.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.787+0000 7f5f1c823640 1 -- 172.21.15.121:0/1301105995 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f1809ae10 msgr2=0x7f5f180a3560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:01.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.787+0000 7f5f1c823640 1 --2- 172.21.15.121:0/1301105995 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f1809ae10 0x7f5f180a3560 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5f100097a0 tx=0x7f5f10029fd0 comp rx=0 tx=0).stop 2024-09-17T18:21:01.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.787+0000 7f5f1c823640 1 -- 172.21.15.121:0/1301105995 shutdown_connections 2024-09-17T18:21:01.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.787+0000 7f5f1c823640 1 --2- 172.21.15.121:0/1301105995 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f1809ae10 0x7f5f180a3560 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:01.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.787+0000 7f5f1c823640 1 --2- 172.21.15.121:0/1301105995 >> v2:172.21.15.121:3300/0 conn(0x7f5f1809a600 0x7f5f1809c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:01.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.787+0000 7f5f1c823640 1 -- 172.21.15.121:0/1301105995 >> 172.21.15.121:0/1301105995 conn(0x7f5f180a3f40 msgr2=0x7f5f180a4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:21:01.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.787+0000 7f5f1c823640 1 -- 172.21.15.121:0/1301105995 shutdown_connections 2024-09-17T18:21:01.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.787+0000 7f5f1c823640 1 -- 172.21.15.121:0/1301105995 wait complete. 2024-09-17T18:21:01.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.788+0000 7f5f1c823640 10 monclient: build_initial_monmap 2024-09-17T18:21:01.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.788+0000 7f5f1c823640 10 monclient: monmap: 2024-09-17T18:21:01.802 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:21:01.802 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:21:01.802 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:21:01.803 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:21:01.803 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:21:01.803 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:01.803 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:21:01.803 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:21:01.803 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:21:01.803 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:01.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.788+0000 7f5f1c823640 1 librados: starting msgr at 2024-09-17T18:21:01.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.788+0000 7f5f1c823640 1 librados: starting objecter 2024-09-17T18:21:01.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.788+0000 7f5f1c823640 1 Processor -- start 2024-09-17T18:21:01.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.788+0000 7f5f1c823640 1 -- start start 2024-09-17T18:21:01.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.788+0000 7f5f1c823640 1 librados: setting wanted keys 2024-09-17T18:21:01.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.788+0000 7f5f1c823640 1 librados: calling monclient init 2024-09-17T18:21:01.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.788+0000 7f5f1c823640 10 monclient: init 2024-09-17T18:21:01.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.788+0000 7f5f1c823640 10 monclient: _reopen_session rank -1 2024-09-17T18:21:01.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.788+0000 7f5f1c823640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T18:21:01.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.788+0000 7f5f1c823640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f1809a600 0x7f5f180982e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:01.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.788+0000 7f5f1c823640 10 monclient(hunting): picked mon.noname-c con 0x7f5f1809a600 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:21:01.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.788+0000 7f5f1c823640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f1809a9e0 0x7f5f18098820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:01.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.789+0000 7f5f1c823640 10 monclient(hunting): picked mon.noname-a con 0x7f5f1809a9e0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:21:01.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.789+0000 7f5f1c823640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5f1809ae10 0x7f5f18092fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:01.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.789+0000 7f5f1c823640 10 monclient(hunting): picked mon.noname-b con 0x7f5f1809ae10 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:21:01.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.789+0000 7f5f1c823640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:01.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.789+0000 7f5f1c823640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5f180a4e50 con 0x7f5f1809a9e0 2024-09-17T18:21:01.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.789+0000 7f5f1c823640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:01.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.789+0000 7f5f1c823640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5f180a4cd0 con 0x7f5f1809ae10 2024-09-17T18:21:01.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.789+0000 7f5f1c823640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:01.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.789+0000 7f5f1c823640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5f180a3aa0 con 0x7f5f1809a600 2024-09-17T18:21:01.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.789+0000 7f5f1c823640 10 monclient(hunting): _renew_subs 2024-09-17T18:21:01.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.789+0000 7f5f1c823640 10 monclient(hunting): authenticate will time out at 865.506059s 2024-09-17T18:21:01.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.789+0000 7f5f1659c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f1809a600 0x7f5f180982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:01.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.789+0000 7f5f1659c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f1809a600 0x7f5f180982e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:56442/0 (socket says 172.21.15.121:56442) 2024-09-17T18:21:01.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.789+0000 7f5f1659c640 1 -- 172.21.15.121:0/60628529 learned_addr learned my addr 172.21.15.121:0/60628529 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:21:01.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.789+0000 7f5f1659c640 10 monclient(hunting): get_auth_request con 0x7f5f1809a600 auth_method 0 2024-09-17T18:21:01.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.789+0000 7f5f1659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:01.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.789+0000 7f5f1659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:01.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.789+0000 7f5f1659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:01.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.789+0000 7f5f15d9b640 1 --2- 172.21.15.121:0/60628529 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f1809a9e0 0x7f5f18098820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:01.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.789+0000 7f5f15d9b640 10 monclient(hunting): get_auth_request con 0x7f5f1809a9e0 auth_method 0 2024-09-17T18:21:01.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.789+0000 7f5f15d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:01.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.789+0000 7f5f15d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:01.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.789+0000 7f5f15d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:01.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.789+0000 7f5f1659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:01.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.789+0000 7f5f1659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:01.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.789+0000 7f5f1659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:01.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.789+0000 7f5f15d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:01.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.789+0000 7f5f15d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:01.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.789+0000 7f5f15d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:01.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.789+0000 7f5f1659c640 10 monclient(hunting): handle_auth_done global_id 34136 payload 824 2024-09-17T18:21:01.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.789+0000 7f5f16d9d640 1 --2- 172.21.15.121:0/60628529 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5f1809ae10 0x7f5f18092fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:01.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.789+0000 7f5f1659c640 1 -- 172.21.15.121:0/60628529 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5f1809ae10 msgr2=0x7f5f18092fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:01.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.789+0000 7f5f1659c640 1 --2- 172.21.15.121:0/60628529 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5f1809ae10 0x7f5f18092fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:01.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.789+0000 7f5f1659c640 1 -- 172.21.15.121:0/60628529 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f1809a9e0 msgr2=0x7f5f18098820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:01.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.789+0000 7f5f1659c640 1 --2- 172.21.15.121:0/60628529 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f1809a9e0 0x7f5f18098820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:01.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.789+0000 7f5f1659c640 10 monclient: _finish_hunting 0 2024-09-17T18:21:01.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.789+0000 7f5f1659c640 1 monclient: found mon.noname-c 2024-09-17T18:21:01.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.789+0000 7f5f1659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:21:01.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.789+0000 7f5f1659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:21:01.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.789+0000 7f5f1659c640 1 -- 172.21.15.121:0/60628529 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f18096db0 con 0x7f5f1809a600 2024-09-17T18:21:01.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.789+0000 7f5f16d9d640 10 monclient: get_auth_request con 0x7f5f1809ae10 auth_method 0 2024-09-17T18:21:01.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.789+0000 7f5f16d9d640 1 --2- 172.21.15.121:0/60628529 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5f1809ae10 0x7f5f18092fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T18:21:01.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.791+0000 7f5f1659c640 1 -- 172.21.15.121:0/60628529 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f1809a600 msgr2=0x7f5f180982e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:21:01.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.791+0000 7f5f1659c640 1 --2- 172.21.15.121:0/60628529 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f1809a600 0x7f5f180982e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5f0800efc0 tx=0x7f5f0800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:21:01.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.791+0000 7f5f06ffd640 10 client.?.objecter ms_handle_connect 0x7f5f1809a600 2024-09-17T18:21:01.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.791+0000 7f5f06ffd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:21:01.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.792+0000 7f5f06ffd640 1 -- 172.21.15.121:0/60628529 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f08014680 con 0x7f5f1809a600 2024-09-17T18:21:01.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.792+0000 7f5f06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:01.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.792+0000 7f5f06ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:21:01.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.792+0000 7f5f06ffd640 10 monclient: dump: 2024-09-17T18:21:01.807 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:01.808 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:01.808 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:01.808 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:01.808 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:01.808 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:01.808 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:01.808 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:01.808 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:01.808 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:01.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.792+0000 7f5f06ffd640 10 monclient: _finish_auth 0 2024-09-17T18:21:01.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.792+0000 7f5f06ffd640 10 monclient: _check_auth_tickets 2024-09-17T18:21:01.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.792+0000 7f5f06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:21:01.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.792+0000 7f5f06ffd640 1 -- 172.21.15.121:0/60628529 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f08019070 con 0x7f5f1809a600 2024-09-17T18:21:01.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.792+0000 7f5f1c823640 5 monclient: authenticate success, global_id 34136 2024-09-17T18:21:01.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.792+0000 7f5f06ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:21:01.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.792+0000 7f5f06ffd640 1 -- 172.21.15.121:0/60628529 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f08014a20 con 0x7f5f1809a600 2024-09-17T18:21:01.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.792+0000 7f5f1c823640 10 monclient: _renew_subs 2024-09-17T18:21:01.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.792+0000 7f5f1c823640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:21:01.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.792+0000 7f5f1c823640 1 -- 172.21.15.121:0/60628529 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f18098d60 con 0x7f5f1809a600 2024-09-17T18:21:01.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.792+0000 7f5f06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:01.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.792+0000 7f5f14d99640 4 set_mon_vals no callback set 2024-09-17T18:21:01.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.792+0000 7f5f06ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:21:01.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.792+0000 7f5f06ffd640 10 monclient: dump: 2024-09-17T18:21:01.809 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:01.810 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:01.810 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:01.810 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:01.810 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:01.810 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:01.810 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:01.810 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:01.810 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:01.810 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:01.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.792+0000 7f5f1c823640 10 client.34136.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:21:01.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.792+0000 7f5f1c823640 10 monclient: _renew_subs 2024-09-17T18:21:01.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.792+0000 7f5f1c823640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:21:01.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.792+0000 7f5f1c823640 1 -- 172.21.15.121:0/60628529 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f18094a50 con 0x7f5f1809a600 2024-09-17T18:21:01.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.792+0000 7f5f1c823640 1 librados: init done 2024-09-17T18:21:01.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.793+0000 7f5f06ffd640 1 -- 172.21.15.121:0/60628529 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f5f08005d10 con 0x7f5f1809a600 2024-09-17T18:21:01.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.793+0000 7f5f1c823640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:21:01.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.793+0000 7f5f1c823640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:21:01.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.793+0000 7f5f1c823640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:21:01.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.793+0000 7f5f1c823640 1 -- 172.21.15.121:0/60628529 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f1809c800 con 0x7f5f1809a600 2024-09-17T18:21:01.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.793+0000 7f5f06ffd640 1 --2- 172.21.15.121:0/60628529 >> v2:172.21.15.173:6812/2423786037 conn(0x7f5ef80401a0 0x7f5ef8042660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:01.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.793+0000 7f5f06ffd640 1 -- 172.21.15.121:0/60628529 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(78..78 src has 1..78) ==== 6656+0+0 (secure 0 0 0) 0x7f5f08054280 con 0x7f5f1809a600 2024-09-17T18:21:01.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.793+0000 7f5f06ffd640 10 client.34136.objecter ms_dispatch 0x7f5f18000b90 osd_map(78..78 src has 1..78) 2024-09-17T18:21:01.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.793+0000 7f5f06ffd640 3 client.34136.objecter handle_osd_map got epochs [78,78] > 0 2024-09-17T18:21:01.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.793+0000 7f5f06ffd640 3 client.34136.objecter handle_osd_map decoding full epoch 78 2024-09-17T18:21:01.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.793+0000 7f5f06ffd640 20 client.34136.objecter dump_active .. 0 homeless 2024-09-17T18:21:01.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.793+0000 7f5f15d9b640 1 --2- 172.21.15.121:0/60628529 >> v2:172.21.15.173:6812/2423786037 conn(0x7f5ef80401a0 0x7f5ef8042660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:01.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.793+0000 7f5f15d9b640 10 monclient: get_auth_request con 0x7f5ef80401a0 auth_method 0 2024-09-17T18:21:01.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.796+0000 7f5f06ffd640 10 client.34136.objecter ms_handle_connect 0x7f5ef80401a0 2024-09-17T18:21:01.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.796+0000 7f5f15d9b640 1 --2- 172.21.15.121:0/60628529 >> v2:172.21.15.173:6812/2423786037 conn(0x7f5ef80401a0 0x7f5ef8042660 secure :-1 s=READY pgs=3864 cs=0 l=1 rev1=1 crypto rx=0x7f5f180972a0 tx=0x7f5f0c007e30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:21:01.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.798+0000 7f5f06ffd640 1 -- 172.21.15.121:0/60628529 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f5f08010040 con 0x7f5f1809a600 2024-09-17T18:21:01.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.798+0000 7f5f06ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:21:01.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:01.798+0000 7f5f06ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:21:02.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.040+0000 7f5f1c823640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:21:02.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.040+0000 7f5f1c823640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:21:02.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.040+0000 7f5f1c823640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:21:02.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.040+0000 7f5f1c823640 1 -- 172.21.15.121:0/60628529 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5f180955b0 con 0x7f5f1809a600 2024-09-17T18:21:02.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.041+0000 7f5f06ffd640 1 -- 172.21.15.121:0/60628529 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f5f0801c370 con 0x7f5f1809a600 2024-09-17T18:21:02.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.041+0000 7f5f06ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:21:02.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.041+0000 7f5f06ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:21:02.044 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:21:02.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.043+0000 7f5f1c823640 10 librados: watch_flush enter 2024-09-17T18:21:02.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.043+0000 7f5f1c823640 10 librados: watch_flush exit 2024-09-17T18:21:02.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.043+0000 7f5f1c823640 20 client.34136.objecter shutdown clearing up homeless session... 2024-09-17T18:21:02.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.043+0000 7f5f1c823640 10 client.34136.objecter successfully canceled tick 2024-09-17T18:21:02.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.043+0000 7f5f1c823640 1 -- 172.21.15.121:0/60628529 >> v2:172.21.15.173:6812/2423786037 conn(0x7f5ef80401a0 msgr2=0x7f5ef8042660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:02.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.043+0000 7f5f1c823640 1 --2- 172.21.15.121:0/60628529 >> v2:172.21.15.173:6812/2423786037 conn(0x7f5ef80401a0 0x7f5ef8042660 secure :-1 s=READY pgs=3864 cs=0 l=1 rev1=1 crypto rx=0x7f5f180972a0 tx=0x7f5f0c007e30 comp rx=0 tx=0).stop 2024-09-17T18:21:02.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.043+0000 7f5f1c823640 10 monclient: shutdown 2024-09-17T18:21:02.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.043+0000 7f5f1c823640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:21:02.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.043+0000 7f5f1c823640 1 -- 172.21.15.121:0/60628529 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f1809a600 msgr2=0x7f5f180982e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:02.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.043+0000 7f5f1c823640 1 --2- 172.21.15.121:0/60628529 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f1809a600 0x7f5f180982e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5f0800efc0 tx=0x7f5f0800c490 comp rx=0 tx=0).stop 2024-09-17T18:21:02.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.043+0000 7f5f1c823640 1 -- 172.21.15.121:0/60628529 shutdown_connections 2024-09-17T18:21:02.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.043+0000 7f5f1c823640 1 --2- 172.21.15.121:0/60628529 >> v2:172.21.15.173:6812/2423786037 conn(0x7f5ef80401a0 0x7f5ef8042660 unknown :-1 s=CLOSED pgs=3864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:02.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.043+0000 7f5f1c823640 1 --2- 172.21.15.121:0/60628529 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5f1809ae10 0x7f5f18092fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:02.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.043+0000 7f5f1c823640 1 --2- 172.21.15.121:0/60628529 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f1809a9e0 0x7f5f18098820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:02.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.043+0000 7f5f1c823640 1 --2- 172.21.15.121:0/60628529 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f1809a600 0x7f5f180982e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:02.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.043+0000 7f5f1c823640 1 -- 172.21.15.121:0/60628529 >> 172.21.15.121:0/60628529 conn(0x7f5f180a3f40 msgr2=0x7f5f180a0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:21:02.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.043+0000 7f5f1c823640 1 -- 172.21.15.121:0/60628529 shutdown_connections 2024-09-17T18:21:02.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.043+0000 7f5f1c823640 1 -- 172.21.15.121:0/60628529 wait complete. 2024-09-17T18:21:02.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.043+0000 7f5f1c823640 1 librados: shutdown 2024-09-17T18:21:02.056 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T18:21:02.056 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-17T18:21:02.057 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T18:21:02.057 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T18:21:02.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.160+0000 7fcd5c2b6640 10 monclient: get_monmap_and_config 2024-09-17T18:21:02.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.160+0000 7fcd5c2b6640 10 monclient: build_initial_monmap 2024-09-17T18:21:02.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.160+0000 7fcd5c2b6640 10 monclient: monmap: 2024-09-17T18:21:02.164 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:21:02.164 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:02.164 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:21:02.161764+0000 2024-09-17T18:21:02.164 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:21:02.161764+0000 2024-09-17T18:21:02.164 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:21:02.164 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:02.164 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:21:02.164 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:21:02.164 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:21:02.164 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:21:02.164 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:21:02.164 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:02.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.162+0000 7fcd5c2b6640 1 Processor -- start 2024-09-17T18:21:02.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.162+0000 7fcd5c2b6640 1 -- start start 2024-09-17T18:21:02.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.162+0000 7fcd5c2b6640 10 monclient: init 2024-09-17T18:21:02.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.163+0000 7fcd5c2b6640 10 monclient: _reopen_session rank -1 2024-09-17T18:21:02.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.163+0000 7fcd5c2b6640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-17T18:21:02.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.163+0000 7fcd5c2b6640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fcd540a1a50 0x7fcd540a1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:02.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.163+0000 7fcd5c2b6640 10 monclient(hunting): picked mon.noname-c con 0x7fcd540a1a50 addr v2:172.21.15.173:3300/0 2024-09-17T18:21:02.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.163+0000 7fcd5c2b6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcd5409dce0 addr v1:172.21.15.121:6789/0 2024-09-17T18:21:02.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.163+0000 7fcd5c2b6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcd540a4670 addr v1:172.21.15.173:6789/0 2024-09-17T18:21:02.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.163+0000 7fcd5c2b6640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd540a2f20 con 0x7fcd5409dce0 2024-09-17T18:21:02.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.163+0000 7fcd5c2b6640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd5409e7a0 con 0x7fcd540a4670 2024-09-17T18:21:02.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.163+0000 7fcd5c2b6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:02.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.163+0000 7fcd5c2b6640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fcd54077640 con 0x7fcd540a1a50 2024-09-17T18:21:02.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.163+0000 7fcd5c2b6640 10 monclient(hunting): _renew_subs 2024-09-17T18:21:02.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.163+0000 7fcd5c2b6640 10 monclient(hunting): authenticate will time out at 865.880653s 2024-09-17T18:21:02.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.163+0000 7fcd59851640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fcd5409dce0 0x7fcd54134810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:51366/0 (socket says 172.21.15.121:51366) 2024-09-17T18:21:02.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.163+0000 7fcd59851640 1 -- 172.21.15.121:0/3212621988 learned_addr learned my addr 172.21.15.121:0/3212621988 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:21:02.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.163+0000 7fcd5a052640 1 --2- 172.21.15.121:0/3212621988 >> v2:172.21.15.173:3300/0 conn(0x7fcd540a1a50 0x7fcd540a1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:02.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.163+0000 7fcd5a052640 10 monclient(hunting): get_auth_request con 0x7fcd540a1a50 auth_method 0 2024-09-17T18:21:02.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.163+0000 7fcd5a052640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:02.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.163+0000 7fcd5a052640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:02.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.163+0000 7fcd5a052640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:02.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.164+0000 7fcd5a052640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:02.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.164+0000 7fcd5a052640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:02.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.164+0000 7fcd5a052640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:02.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.164+0000 7fcd5a052640 10 monclient(hunting): handle_auth_done global_id 24160 payload 274 2024-09-17T18:21:02.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.164+0000 7fcd5a052640 1 -- 172.21.15.121:0/3212621988 >> v1:172.21.15.173:6789/0 conn(0x7fcd540a4670 legacy=0x7fcd5413cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:21:02.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.164+0000 7fcd5a052640 1 -- 172.21.15.121:0/3212621988 >> v1:172.21.15.121:6789/0 conn(0x7fcd5409dce0 legacy=0x7fcd54134810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:21:02.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.164+0000 7fcd5a052640 10 monclient: _finish_hunting 0 2024-09-17T18:21:02.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.164+0000 7fcd5a052640 1 monclient: found mon.noname-c 2024-09-17T18:21:02.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.164+0000 7fcd5a052640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:21:02.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.164+0000 7fcd5a052640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.173:3300/0 2024-09-17T18:21:02.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.164+0000 7fcd5a052640 1 -- 172.21.15.121:0/3212621988 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd5409ef10 con 0x7fcd540a1a50 2024-09-17T18:21:02.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.167+0000 7fcd5a052640 1 -- 172.21.15.121:0/3212621988 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd540a1a50 msgr2=0x7fcd540a1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:21:02.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.167+0000 7fcd5a052640 1 --2- 172.21.15.121:0/3212621988 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd540a1a50 0x7fcd540a1e30 secure :-1 s=READY pgs=7109 cs=0 l=1 rev1=1 crypto rx=0x7fcd500097a0 tx=0x7fcd5002a050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8fd3af6e65f8a084 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:21:02.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.167+0000 7fcd59050640 1 -- 172.21.15.121:0/3212621988 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd50037070 con 0x7fcd540a1a50 2024-09-17T18:21:02.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.167+0000 7fcd59050640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:02.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.167+0000 7fcd59050640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:21:02.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.167+0000 7fcd59050640 10 monclient: dump: 2024-09-17T18:21:02.175 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:02.175 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:02.175 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:02.175 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:02.175 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:02.175 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:02.175 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:02.175 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:02.175 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:02.175 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:02.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.167+0000 7fcd59050640 10 monclient: _finish_auth 0 2024-09-17T18:21:02.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.167+0000 7fcd59050640 10 monclient: _check_auth_tickets 2024-09-17T18:21:02.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.167+0000 7fcd59050640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:21:02.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.167+0000 7fcd59050640 1 -- 172.21.15.121:0/3212621988 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd50032640 con 0x7fcd540a1a50 2024-09-17T18:21:02.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.167+0000 7fcd59050640 10 monclient: handle_config config(0 keys) 2024-09-17T18:21:02.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.167+0000 7fcd59050640 1 -- 172.21.15.121:0/3212621988 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd50032a10 con 0x7fcd540a1a50 2024-09-17T18:21:02.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.167+0000 7fcd5c2b6640 5 monclient: authenticate success, global_id 24160 2024-09-17T18:21:02.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.167+0000 7fcd5c2b6640 10 monclient: get_monmap_and_config success 2024-09-17T18:21:02.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.167+0000 7fcd5c2b6640 4 set_mon_vals no callback set 2024-09-17T18:21:02.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.167+0000 7fcd59050640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:02.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.167+0000 7fcd59050640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:21:02.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.167+0000 7fcd59050640 10 monclient: dump: 2024-09-17T18:21:02.176 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:02.176 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:02.176 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:02.176 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:02.176 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:02.176 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:02.177 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:02.177 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:02.177 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:02.177 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:02.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.168+0000 7fcd5c2b6640 10 monclient: shutdown 2024-09-17T18:21:02.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.168+0000 7fcd5c2b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:21:02.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.168+0000 7fcd5c2b6640 1 -- 172.21.15.121:0/3212621988 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd540a1a50 msgr2=0x7fcd540a1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:02.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.168+0000 7fcd5c2b6640 1 --2- 172.21.15.121:0/3212621988 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd540a1a50 0x7fcd540a1e30 secure :-1 s=READY pgs=7109 cs=0 l=1 rev1=1 crypto rx=0x7fcd500097a0 tx=0x7fcd5002a050 comp rx=0 tx=0).stop 2024-09-17T18:21:02.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.168+0000 7fcd5c2b6640 1 -- 172.21.15.121:0/3212621988 shutdown_connections 2024-09-17T18:21:02.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.168+0000 7fcd5c2b6640 1 --2- 172.21.15.121:0/3212621988 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd540a1a50 0x7fcd540a1e30 unknown :-1 s=CLOSED pgs=7109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:02.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.168+0000 7fcd5c2b6640 1 -- 172.21.15.121:0/3212621988 >> 172.21.15.121:0/3212621988 conn(0x7fcd540a3b50 msgr2=0x7fcd540a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:21:02.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.168+0000 7fcd5c2b6640 1 -- 172.21.15.121:0/3212621988 shutdown_connections 2024-09-17T18:21:02.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.168+0000 7fcd5c2b6640 1 -- 172.21.15.121:0/3212621988 wait complete. 2024-09-17T18:21:02.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.169+0000 7fcd5c2b6640 10 monclient: build_initial_monmap 2024-09-17T18:21:02.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.169+0000 7fcd5c2b6640 10 monclient: monmap: 2024-09-17T18:21:02.178 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:21:02.178 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:21:02.178 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:21:02.178 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:21:02.178 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:21:02.178 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:02.178 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:21:02.178 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:21:02.178 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:21:02.178 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:02.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.169+0000 7fcd5c2b6640 1 librados: starting msgr at 2024-09-17T18:21:02.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.169+0000 7fcd5c2b6640 1 librados: starting objecter 2024-09-17T18:21:02.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.169+0000 7fcd5c2b6640 1 Processor -- start 2024-09-17T18:21:02.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.169+0000 7fcd5c2b6640 1 -- start start 2024-09-17T18:21:02.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.169+0000 7fcd5c2b6640 1 librados: setting wanted keys 2024-09-17T18:21:02.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.169+0000 7fcd5c2b6640 1 librados: calling monclient init 2024-09-17T18:21:02.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.169+0000 7fcd5c2b6640 10 monclient: init 2024-09-17T18:21:02.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5c2b6640 10 monclient: _reopen_session rank -1 2024-09-17T18:21:02.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5c2b6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T18:21:02.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5c2b6640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd5409dce0 0x7fcd54135310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:02.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5c2b6640 10 monclient(hunting): picked mon.noname-b con 0x7fcd5409dce0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:21:02.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5c2b6640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd540a1a50 0x7fcd5412fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:02.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5c2b6640 10 monclient(hunting): picked mon.noname-c con 0x7fcd540a1a50 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:21:02.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5c2b6640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd540a4670 0x7fcd54130140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:02.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5c2b6640 10 monclient(hunting): picked mon.noname-a con 0x7fcd540a4670 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:21:02.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5c2b6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:02.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5c2b6640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcd540a2370 con 0x7fcd540a4670 2024-09-17T18:21:02.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5c2b6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:02.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5c2b6640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcd540a24f0 con 0x7fcd5409dce0 2024-09-17T18:21:02.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5c2b6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:02.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5c2b6640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcd540a2670 con 0x7fcd540a1a50 2024-09-17T18:21:02.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5c2b6640 10 monclient(hunting): _renew_subs 2024-09-17T18:21:02.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5c2b6640 10 monclient(hunting): authenticate will time out at 865.887302s 2024-09-17T18:21:02.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5a853640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd540a4670 0x7fcd54130140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:02.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5a853640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd540a4670 0x7fcd54130140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:55554/0 (socket says 172.21.15.121:55554) 2024-09-17T18:21:02.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5a853640 1 -- 172.21.15.121:0/668225795 learned_addr learned my addr 172.21.15.121:0/668225795 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:21:02.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5a853640 10 monclient(hunting): get_auth_request con 0x7fcd540a4670 auth_method 0 2024-09-17T18:21:02.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5a853640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:02.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5a853640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:02.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5a853640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:02.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd59851640 1 --2- 172.21.15.121:0/668225795 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd540a1a50 0x7fcd5412fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:02.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd59851640 10 monclient(hunting): get_auth_request con 0x7fcd540a1a50 auth_method 0 2024-09-17T18:21:02.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd59851640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:02.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd59851640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:02.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd59851640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:02.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5a052640 1 --2- 172.21.15.121:0/668225795 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd5409dce0 0x7fcd54135310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:02.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5a853640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:02.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5a853640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:02.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5a853640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:02.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5a052640 10 monclient(hunting): get_auth_request con 0x7fcd5409dce0 auth_method 0 2024-09-17T18:21:02.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5a052640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:02.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5a052640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:02.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5a052640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:02.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd59851640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:02.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd59851640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:02.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd59851640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:02.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5a052640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:02.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5a052640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:02.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5a052640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:02.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5a853640 10 monclient(hunting): handle_auth_done global_id 23751 payload 824 2024-09-17T18:21:02.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5a853640 1 -- 172.21.15.121:0/668225795 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd540a1a50 msgr2=0x7fcd5412fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:02.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5a853640 1 --2- 172.21.15.121:0/668225795 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd540a1a50 0x7fcd5412fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:02.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5a853640 1 -- 172.21.15.121:0/668225795 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd5409dce0 msgr2=0x7fcd54135310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:02.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5a853640 1 --2- 172.21.15.121:0/668225795 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd5409dce0 0x7fcd54135310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:02.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5a853640 10 monclient: _finish_hunting 0 2024-09-17T18:21:02.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5a853640 1 monclient: found mon.noname-a 2024-09-17T18:21:02.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5a853640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:21:02.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5a853640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:21:02.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd5a853640 1 -- 172.21.15.121:0/668225795 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd54135850 con 0x7fcd540a4670 2024-09-17T18:21:02.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.170+0000 7fcd59851640 1 --2- 172.21.15.121:0/668225795 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd540a1a50 0x7fcd5412fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:21:02.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.172+0000 7fcd5a853640 1 -- 172.21.15.121:0/668225795 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd540a4670 msgr2=0x7fcd54130140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:21:02.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.172+0000 7fcd5a853640 1 --2- 172.21.15.121:0/668225795 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd540a4670 0x7fcd54130140 secure :-1 s=READY pgs=6831 cs=0 l=1 rev1=1 crypto rx=0x7fcd4c00c780 tx=0x7fcd4c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:21:02.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.172+0000 7fcd4affd640 10 client.?.objecter ms_handle_connect 0x7fcd540a4670 2024-09-17T18:21:02.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.172+0000 7fcd4affd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:21:02.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.173+0000 7fcd4affd640 1 -- 172.21.15.121:0/668225795 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd4c00ac60 con 0x7fcd540a4670 2024-09-17T18:21:02.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.173+0000 7fcd4affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:02.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.173+0000 7fcd4affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:21:02.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.173+0000 7fcd4affd640 10 monclient: dump: 2024-09-17T18:21:02.188 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:02.188 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:02.188 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:02.188 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:02.188 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:02.188 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:02.189 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:02.189 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:02.189 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:02.189 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:02.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.173+0000 7fcd4affd640 10 monclient: _finish_auth 0 2024-09-17T18:21:02.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.173+0000 7fcd4affd640 10 monclient: _check_auth_tickets 2024-09-17T18:21:02.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.173+0000 7fcd4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:21:02.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.173+0000 7fcd5c2b6640 5 monclient: authenticate success, global_id 23751 2024-09-17T18:21:02.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.173+0000 7fcd4affd640 1 -- 172.21.15.121:0/668225795 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd4c00ae00 con 0x7fcd540a4670 2024-09-17T18:21:02.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.173+0000 7fcd5c2b6640 10 monclient: _renew_subs 2024-09-17T18:21:02.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.173+0000 7fcd5c2b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:21:02.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.173+0000 7fcd5c2b6640 1 -- 172.21.15.121:0/668225795 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd54133890 con 0x7fcd540a4670 2024-09-17T18:21:02.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.173+0000 7fcd4affd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:21:02.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.173+0000 7fcd4affd640 1 -- 172.21.15.121:0/668225795 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd4c004ba0 con 0x7fcd540a4670 2024-09-17T18:21:02.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.173+0000 7fcd4affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:02.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.173+0000 7fcd4affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:21:02.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.173+0000 7fcd4affd640 10 monclient: dump: 2024-09-17T18:21:02.190 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:02.190 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:02.190 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:02.190 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:02.190 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:02.190 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:02.190 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:02.190 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:02.190 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:02.190 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:02.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.173+0000 7fcd5c2b6640 10 client.23751.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:21:02.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.173+0000 7fcd5c2b6640 10 monclient: _renew_subs 2024-09-17T18:21:02.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.173+0000 7fcd5884f640 4 set_mon_vals no callback set 2024-09-17T18:21:02.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.173+0000 7fcd5c2b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:21:02.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.173+0000 7fcd5c2b6640 1 -- 172.21.15.121:0/668225795 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd54132ff0 con 0x7fcd540a4670 2024-09-17T18:21:02.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.173+0000 7fcd5c2b6640 1 librados: init done 2024-09-17T18:21:02.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.174+0000 7fcd5c2b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:21:02.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.174+0000 7fcd5c2b6640 1 --2- 172.21.15.121:0/668225795 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd5412b020 0x7fcd54130680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:02.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.174+0000 7fcd5c2b6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:02.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.174+0000 7fcd5c2b6640 1 -- 172.21.15.121:0/668225795 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcd540a0990 con 0x7fcd5412b020 2024-09-17T18:21:02.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.174+0000 7fcd4affd640 1 -- 172.21.15.121:0/668225795 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fcd4c006b50 con 0x7fcd540a4670 2024-09-17T18:21:02.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.174+0000 7fcd5a052640 1 --2- 172.21.15.121:0/668225795 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd5412b020 0x7fcd54130680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:02.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.174+0000 7fcd5a052640 10 monclient: get_auth_request con 0x7fcd5412b020 auth_method 0 2024-09-17T18:21:02.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.174+0000 7fcd5a052640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:02.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.174+0000 7fcd5a052640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:02.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.174+0000 7fcd5a052640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:02.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.174+0000 7fcd5a052640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:02.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.174+0000 7fcd5a052640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:02.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.174+0000 7fcd5a052640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:02.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.174+0000 7fcd4affd640 1 --2- 172.21.15.121:0/668225795 >> v2:172.21.15.173:6812/2423786037 conn(0x7fcd38040170 0x7fcd38042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:02.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.174+0000 7fcd4affd640 1 -- 172.21.15.121:0/668225795 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6635+0+0 (secure 0 0 0) 0x7fcd4c053e70 con 0x7fcd540a4670 2024-09-17T18:21:02.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.174+0000 7fcd4affd640 10 client.23751.objecter ms_dispatch 0x7fcd54000b90 osd_map(79..79 src has 1..79) 2024-09-17T18:21:02.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.174+0000 7fcd4affd640 3 client.23751.objecter handle_osd_map got epochs [79,79] > 0 2024-09-17T18:21:02.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.174+0000 7fcd4affd640 3 client.23751.objecter handle_osd_map decoding full epoch 79 2024-09-17T18:21:02.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.174+0000 7fcd5a052640 10 monclient(hunting): handle_auth_done global_id 23754 payload 824 2024-09-17T18:21:02.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.174+0000 7fcd5a052640 1 -- 172.21.15.121:0/668225795 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcd54132150 con 0x7fcd5412b020 2024-09-17T18:21:02.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.174+0000 7fcd4affd640 20 client.23751.objecter dump_active .. 0 homeless 2024-09-17T18:21:02.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.174+0000 7fcd59851640 1 --2- 172.21.15.121:0/668225795 >> v2:172.21.15.173:6812/2423786037 conn(0x7fcd38040170 0x7fcd38042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:02.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.174+0000 7fcd59851640 10 monclient: get_auth_request con 0x7fcd38040170 auth_method 0 2024-09-17T18:21:02.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.176+0000 7fcd4affd640 10 client.23751.objecter ms_handle_connect 0x7fcd5412b020 2024-09-17T18:21:02.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.176+0000 7fcd4affd640 10 client.23751.objecter resend_mon_ops 2024-09-17T18:21:02.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.176+0000 7fcd5a052640 1 -- 172.21.15.121:0/668225795 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd5412b020 msgr2=0x7fcd54130680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:21:02.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.176+0000 7fcd5a052640 1 --2- 172.21.15.121:0/668225795 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd5412b020 0x7fcd54130680 secure :-1 s=READY pgs=6832 cs=0 l=1 rev1=1 crypto rx=0x7fcd54077150 tx=0x7fcd500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:21:02.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.177+0000 7fcd4affd640 1 -- 172.21.15.121:0/668225795 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd50037040 con 0x7fcd5412b020 2024-09-17T18:21:02.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.177+0000 7fcd4affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:02.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.177+0000 7fcd4affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:21:02.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.177+0000 7fcd4affd640 10 monclient: dump: 2024-09-17T18:21:02.193 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:02.193 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:02.193 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:02.193 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:02.194 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:02.194 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:02.194 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:02.194 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:02.194 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:02.194 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:02.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.177+0000 7fcd4affd640 10 client.23751.objecter ms_handle_connect 0x7fcd38040170 2024-09-17T18:21:02.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.177+0000 7fcd59851640 1 --2- 172.21.15.121:0/668225795 >> v2:172.21.15.173:6812/2423786037 conn(0x7fcd38040170 0x7fcd38042630 secure :-1 s=READY pgs=3865 cs=0 l=1 rev1=1 crypto rx=0x7fcd44007c90 tx=0x7fcd44007be0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:21:02.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.177+0000 7fcd4affd640 1 -- 172.21.15.121:0/668225795 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcd50032e20 con 0x7fcd5412b020 2024-09-17T18:21:02.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.177+0000 7fcd4affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:21:02.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.177+0000 7fcd4affd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:21:02.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.177+0000 7fcd4affd640 1 -- 172.21.15.121:0/668225795 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd5412b020 msgr2=0x7fcd54130680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:02.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.177+0000 7fcd4affd640 1 --2- 172.21.15.121:0/668225795 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd5412b020 0x7fcd54130680 secure :-1 s=READY pgs=6832 cs=0 l=1 rev1=1 crypto rx=0x7fcd54077150 tx=0x7fcd500046c0 comp rx=0 tx=0).stop 2024-09-17T18:21:02.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.177+0000 7fcd4affd640 1 -- 172.21.15.121:0/668225795 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd5412b020 msgr2=0x7fcd54130680 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:21:02.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.177+0000 7fcd4affd640 1 --2- 172.21.15.121:0/668225795 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd5412b020 0x7fcd54130680 secure :-1 s=CLOSED pgs=6832 cs=0 l=1 rev1=1 crypto rx=0x7fcd54077150 tx=0x7fcd500046c0 comp rx=0 tx=0).stop 2024-09-17T18:21:02.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.183+0000 7fcd5c2b6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:21:02.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.183+0000 7fcd5c2b6640 1 --2- 172.21.15.121:0/668225795 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd5412b990 0x7fcd5412bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:02.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.183+0000 7fcd5c2b6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:02.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.183+0000 7fcd5c2b6640 1 -- 172.21.15.121:0/668225795 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcd5409e240 con 0x7fcd5412b990 2024-09-17T18:21:02.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.183+0000 7fcd5a052640 1 --2- 172.21.15.121:0/668225795 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd5412b990 0x7fcd5412bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:02.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.183+0000 7fcd5a052640 10 monclient: get_auth_request con 0x7fcd5412b990 auth_method 0 2024-09-17T18:21:02.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.183+0000 7fcd5a052640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:02.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.183+0000 7fcd5a052640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:02.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.183+0000 7fcd5a052640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:02.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.183+0000 7fcd5a052640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:02.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.183+0000 7fcd5a052640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:02.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.183+0000 7fcd5a052640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:02.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.183+0000 7fcd5a052640 10 monclient(hunting): handle_auth_done global_id 23763 payload 824 2024-09-17T18:21:02.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.183+0000 7fcd5a052640 1 -- 172.21.15.121:0/668225795 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcd540a1e30 con 0x7fcd5412b990 2024-09-17T18:21:02.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.186+0000 7fcd5a052640 1 -- 172.21.15.121:0/668225795 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd5412b990 msgr2=0x7fcd5412bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:21:02.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.186+0000 7fcd5a052640 1 --2- 172.21.15.121:0/668225795 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd5412b990 0x7fcd5412bda0 secure :-1 s=READY pgs=6834 cs=0 l=1 rev1=1 crypto rx=0x7fcd500097a0 tx=0x7fcd50008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:21:02.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.186+0000 7fcd4affd640 10 client.23751.objecter ms_handle_connect 0x7fcd5412b990 2024-09-17T18:21:02.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.186+0000 7fcd4affd640 10 client.23751.objecter resend_mon_ops 2024-09-17T18:21:02.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.189+0000 7fcd4affd640 1 -- 172.21.15.121:0/668225795 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd50037070 con 0x7fcd5412b990 2024-09-17T18:21:02.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.189+0000 7fcd4affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:02.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.189+0000 7fcd4affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:21:02.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.189+0000 7fcd4affd640 10 monclient: dump: 2024-09-17T18:21:02.197 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:02.197 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:02.197 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:02.197 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:02.197 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:02.197 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:02.198 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:02.198 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:02.198 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:02.198 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:02.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.189+0000 7fcd4affd640 1 -- 172.21.15.121:0/668225795 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7fcd50052410 con 0x7fcd5412b990 2024-09-17T18:21:02.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.189+0000 7fcd4affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:21:02.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.189+0000 7fcd4affd640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:21:02.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.189+0000 7fcd4affd640 1 -- 172.21.15.121:0/668225795 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd5412b990 msgr2=0x7fcd5412bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:02.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.189+0000 7fcd4affd640 1 --2- 172.21.15.121:0/668225795 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd5412b990 0x7fcd5412bda0 secure :-1 s=READY pgs=6834 cs=0 l=1 rev1=1 crypto rx=0x7fcd500097a0 tx=0x7fcd50008030 comp rx=0 tx=0).stop 2024-09-17T18:21:02.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.189+0000 7fcd4affd640 1 -- 172.21.15.121:0/668225795 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd5412b990 msgr2=0x7fcd5412bda0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:21:02.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.189+0000 7fcd4affd640 1 --2- 172.21.15.121:0/668225795 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd5412b990 0x7fcd5412bda0 secure :-1 s=CLOSED pgs=6834 cs=0 l=1 rev1=1 crypto rx=0x7fcd500097a0 tx=0x7fcd50008030 comp rx=0 tx=0).stop 2024-09-17T18:21:02.199 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:21:02.199 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:21:02.199 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:21:02.199 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "leader", 2024-09-17T18:21:02.199 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 10, 2024-09-17T18:21:02.199 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 135175, 2024-09-17T18:21:02.199 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:21:02.199 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:21:02.199 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:21:02.199 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:21:02.199 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:21:02.199 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 6, 2024-09-17T18:21:02.199 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:21:02.199 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:21:02.200 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:21:02.200 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:21:02.200 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:21:02.200 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:21:02.200 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:21:02.200 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:21:02.200 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:21:02.200 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:21:02.200 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:21:02.200 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:21:02.200 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:21:02.200 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:21:02.200 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:21:02.200 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:21:02.201 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:21:02.201 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:21:02.201 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:21:02.201 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:21:02.201 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:21:02.201 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:21:02.201 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:21:02.201 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:21:02.201 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:21:02.201 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:21:02.201 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:21:02.201 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:21:02.201 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:21:02.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:02.201 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:21:02.202 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:21:02.202 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:21:02.202 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:21:02.202 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:21:02.202 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:21:02.202 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:21:02.202 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:21:02.202 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:21:02.202 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:21:02.202 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:21:02.202 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:21:02.202 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:21:02.202 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:21:02.202 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:21:02.203 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:21:02.203 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:21:02.203 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:21:02.203 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:21:02.203 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:21:02.203 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:21:02.203 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:21:02.203 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:21:02.203 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:21:02.203 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:21:02.203 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:21:02.203 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:21:02.203 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:21:02.203 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:21:02.203 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:21:02.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:02.204 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:21:02.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:02.204 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:21:02.204 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:21:02.204 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:21:02.204 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:21:02.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:02.204 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:21:02.204 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:21:02.204 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:21:02.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:02.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:02.204 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:21:02.205 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:21:02.205 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:21:02.205 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:02.205 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:21:02.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:02.205 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:21:02.205 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:21:02.205 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:21:02.205 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:21:02.205 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:21:02.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:02.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:02.205 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:21:02.205 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:21:02.206 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:21:02.206 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:21:02.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:02.206 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:21:02.206 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:21:02.206 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:21:02.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:02.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:02.206 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:21:02.206 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:21:02.206 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:21:02.206 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:02.206 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:21:02.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:02.207 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:21:02.207 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:21:02.207 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:21:02.207 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:21:02.207 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:21:02.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:02.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:02.207 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:21:02.207 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:21:02.207 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:21:02.207 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:21:02.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:02.207 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:21:02.207 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:21:02.207 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:21:02.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:02.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:02.208 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:21:02.208 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:21:02.208 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:21:02.208 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:02.208 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:21:02.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:02.208 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:21:02.208 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:21:02.208 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:21:02.208 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:21:02.208 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:21:02.208 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:02.209 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:21:02.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:02.209 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:21:02.209 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:21:02.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:02.209 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:21:02.209 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:21:02.209 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:21:02.209 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:02.209 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:21:02.209 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:21:02.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:02.209 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:21:02.210 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:21:02.210 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-17T18:21:02.210 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:02.210 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:21:02.210 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:21:02.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:02.210 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:21:02.210 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:21:02.210 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 4 2024-09-17T18:21:02.210 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:02.210 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:21:02.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:02.210 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:21:02.210 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:21:02.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.190+0000 7fcd5c2b6640 10 librados: watch_flush enter 2024-09-17T18:21:02.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.190+0000 7fcd5c2b6640 10 librados: watch_flush exit 2024-09-17T18:21:02.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.190+0000 7fcd5c2b6640 20 client.23751.objecter shutdown clearing up homeless session... 2024-09-17T18:21:02.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.190+0000 7fcd5c2b6640 10 client.23751.objecter successfully canceled tick 2024-09-17T18:21:02.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.190+0000 7fcd5c2b6640 1 -- 172.21.15.121:0/668225795 >> v2:172.21.15.173:6812/2423786037 conn(0x7fcd38040170 msgr2=0x7fcd38042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:02.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.190+0000 7fcd5c2b6640 1 --2- 172.21.15.121:0/668225795 >> v2:172.21.15.173:6812/2423786037 conn(0x7fcd38040170 0x7fcd38042630 secure :-1 s=READY pgs=3865 cs=0 l=1 rev1=1 crypto rx=0x7fcd44007c90 tx=0x7fcd44007be0 comp rx=0 tx=0).stop 2024-09-17T18:21:02.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.190+0000 7fcd5c2b6640 10 monclient: shutdown 2024-09-17T18:21:02.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.190+0000 7fcd5c2b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:21:02.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.190+0000 7fcd5c2b6640 1 -- 172.21.15.121:0/668225795 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd540a4670 msgr2=0x7fcd54130140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:02.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.190+0000 7fcd5c2b6640 1 --2- 172.21.15.121:0/668225795 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd540a4670 0x7fcd54130140 secure :-1 s=READY pgs=6831 cs=0 l=1 rev1=1 crypto rx=0x7fcd4c00c780 tx=0x7fcd4c00cc40 comp rx=0 tx=0).stop 2024-09-17T18:21:02.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.190+0000 7fcd5a853640 1 -- 172.21.15.121:0/668225795 reap_dead start 2024-09-17T18:21:02.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.190+0000 7fcd5a853640 1 -- 172.21.15.121:0/668225795 reap_dead start 2024-09-17T18:21:02.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.190+0000 7fcd5c2b6640 1 -- 172.21.15.121:0/668225795 shutdown_connections 2024-09-17T18:21:02.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.190+0000 7fcd5c2b6640 1 -- 172.21.15.121:0/668225795 >> 172.21.15.121:0/668225795 conn(0x7fcd540a3b50 msgr2=0x7fcd54070550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:21:02.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.190+0000 7fcd5c2b6640 1 -- 172.21.15.121:0/668225795 shutdown_connections 2024-09-17T18:21:02.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.190+0000 7fcd5c2b6640 1 -- 172.21.15.121:0/668225795 wait complete. 2024-09-17T18:21:02.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.190+0000 7fcd5c2b6640 1 librados: shutdown 2024-09-17T18:21:02.214 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:21:02.214 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T18:21:02.214 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T18:21:02.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.310+0000 7fda4ad6b640 10 monclient: get_monmap_and_config 2024-09-17T18:21:02.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.310+0000 7fda4ad6b640 10 monclient: build_initial_monmap 2024-09-17T18:21:02.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.310+0000 7fda4ad6b640 10 monclient: monmap: 2024-09-17T18:21:02.313 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:21:02.313 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:02.313 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:21:02.312190+0000 2024-09-17T18:21:02.313 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:21:02.312190+0000 2024-09-17T18:21:02.314 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:21:02.314 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:02.314 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:21:02.314 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:21:02.314 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:21:02.315 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:21:02.315 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:21:02.315 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:02.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.311+0000 7fda4ad6b640 1 Processor -- start 2024-09-17T18:21:02.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.311+0000 7fda4ad6b640 1 -- start start 2024-09-17T18:21:02.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.311+0000 7fda4ad6b640 10 monclient: init 2024-09-17T18:21:02.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.311+0000 7fda4ad6b640 10 monclient: _reopen_session rank -1 2024-09-17T18:21:02.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.311+0000 7fda4ad6b640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-17T18:21:02.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.311+0000 7fda4ad6b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda440a4a10 0x7fda440a4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:02.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.311+0000 7fda4ad6b640 10 monclient(hunting): picked mon.noname-b con 0x7fda440a4a10 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:21:02.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.311+0000 7fda4ad6b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fda4406fcf0 addr v1:172.21.15.173:6789/0 2024-09-17T18:21:02.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.311+0000 7fda4ad6b640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fda440a5330 0x7fda44140300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:02.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.311+0000 7fda4ad6b640 10 monclient(hunting): picked mon.noname-c con 0x7fda440a5330 addr v2:172.21.15.173:3300/0 2024-09-17T18:21:02.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.311+0000 7fda4ad6b640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda4406ef60 con 0x7fda4406fcf0 2024-09-17T18:21:02.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.311+0000 7fda4ad6b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:02.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.311+0000 7fda4ad6b640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fda44070480 con 0x7fda440a5330 2024-09-17T18:21:02.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.311+0000 7fda4ad6b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:02.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.311+0000 7fda4ad6b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fda4406f470 con 0x7fda440a4a10 2024-09-17T18:21:02.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.311+0000 7fda4ad6b640 10 monclient(hunting): _renew_subs 2024-09-17T18:21:02.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.311+0000 7fda4ad6b640 10 monclient(hunting): authenticate will time out at 866.028864s 2024-09-17T18:21:02.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.311+0000 7fda48b07640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda440a4a10 0x7fda440a4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:02.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.312+0000 7fda48b07640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda440a4a10 0x7fda440a4df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:56458/0 (socket says 172.21.15.121:56458) 2024-09-17T18:21:02.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.312+0000 7fda48b07640 1 -- 172.21.15.121:0/4158211728 learned_addr learned my addr 172.21.15.121:0/4158211728 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:21:02.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.312+0000 7fda48b07640 10 monclient(hunting): get_auth_request con 0x7fda440a4a10 auth_method 0 2024-09-17T18:21:02.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.312+0000 7fda48b07640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:02.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.312+0000 7fda48b07640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:02.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.312+0000 7fda48b07640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:02.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.312+0000 7fda48b07640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:02.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.312+0000 7fda48b07640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:02.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.312+0000 7fda48b07640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:02.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.312+0000 7fda48b07640 10 monclient(hunting): handle_auth_done global_id 34148 payload 274 2024-09-17T18:21:02.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.312+0000 7fda49308640 1 --2- 172.21.15.121:0/4158211728 >> v2:172.21.15.173:3300/0 conn(0x7fda440a5330 0x7fda44140300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:02.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.312+0000 7fda48b07640 1 -- 172.21.15.121:0/4158211728 >> v2:172.21.15.173:3300/0 conn(0x7fda440a5330 msgr2=0x7fda44140300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:21:02.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.312+0000 7fda48b07640 1 --2- 172.21.15.121:0/4158211728 >> v2:172.21.15.173:3300/0 conn(0x7fda440a5330 0x7fda44140300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:02.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.312+0000 7fda48b07640 1 -- 172.21.15.121:0/4158211728 >> v1:172.21.15.173:6789/0 conn(0x7fda4406fcf0 legacy=0x7fda4413cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:21:02.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.312+0000 7fda48b07640 10 monclient: _finish_hunting 0 2024-09-17T18:21:02.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.312+0000 7fda48b07640 1 monclient: found mon.noname-b 2024-09-17T18:21:02.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.312+0000 7fda48b07640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:21:02.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.312+0000 7fda48b07640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:21:02.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.312+0000 7fda48b07640 1 -- 172.21.15.121:0/4158211728 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda4406f160 con 0x7fda440a4a10 2024-09-17T18:21:02.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.315+0000 7fda48b07640 1 -- 172.21.15.121:0/4158211728 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda440a4a10 msgr2=0x7fda440a4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:21:02.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.315+0000 7fda48b07640 1 --2- 172.21.15.121:0/4158211728 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda440a4a10 0x7fda440a4df0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fda40009870 tx=0x7fda4002a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=795445dab21d0701 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:21:02.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.315+0000 7fda3b7fe640 1 -- 172.21.15.121:0/4158211728 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda40037070 con 0x7fda440a4a10 2024-09-17T18:21:02.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.315+0000 7fda3b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:02.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.315+0000 7fda3b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:21:02.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.315+0000 7fda3b7fe640 10 monclient: dump: 2024-09-17T18:21:02.325 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:02.325 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:02.325 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:02.325 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:02.325 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:02.325 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:02.325 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:02.325 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:02.325 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:02.325 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:02.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.315+0000 7fda3b7fe640 10 monclient: _finish_auth 0 2024-09-17T18:21:02.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.315+0000 7fda3b7fe640 10 monclient: _check_auth_tickets 2024-09-17T18:21:02.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.315+0000 7fda3b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:21:02.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.315+0000 7fda3b7fe640 1 -- 172.21.15.121:0/4158211728 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda40032640 con 0x7fda440a4a10 2024-09-17T18:21:02.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.315+0000 7fda4ad6b640 5 monclient: authenticate success, global_id 34148 2024-09-17T18:21:02.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.315+0000 7fda4ad6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:21:02.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.315+0000 7fda3b7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:21:02.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.315+0000 7fda4ad6b640 10 monclient: get_monmap_and_config success 2024-09-17T18:21:02.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.315+0000 7fda3b7fe640 1 -- 172.21.15.121:0/4158211728 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda40032a10 con 0x7fda440a4a10 2024-09-17T18:21:02.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.315+0000 7fda4ad6b640 4 set_mon_vals no callback set 2024-09-17T18:21:02.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.315+0000 7fda3b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:02.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.315+0000 7fda3b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:21:02.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.315+0000 7fda3b7fe640 10 monclient: dump: 2024-09-17T18:21:02.326 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:02.326 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:02.326 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:02.327 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:02.327 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:02.327 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:02.327 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:02.327 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:02.327 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:02.327 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:02.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.315+0000 7fda4ad6b640 10 monclient: shutdown 2024-09-17T18:21:02.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.315+0000 7fda4ad6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:21:02.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.315+0000 7fda4ad6b640 1 -- 172.21.15.121:0/4158211728 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda440a4a10 msgr2=0x7fda440a4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:02.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.315+0000 7fda4ad6b640 1 --2- 172.21.15.121:0/4158211728 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda440a4a10 0x7fda440a4df0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fda40009870 tx=0x7fda4002a0a0 comp rx=0 tx=0).stop 2024-09-17T18:21:02.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.315+0000 7fda4ad6b640 1 -- 172.21.15.121:0/4158211728 shutdown_connections 2024-09-17T18:21:02.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.315+0000 7fda4ad6b640 1 --2- 172.21.15.121:0/4158211728 >> v2:172.21.15.173:3300/0 conn(0x7fda440a5330 0x7fda44140300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:02.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.315+0000 7fda4ad6b640 1 --2- 172.21.15.121:0/4158211728 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda440a4a10 0x7fda440a4df0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:02.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.315+0000 7fda4ad6b640 1 -- 172.21.15.121:0/4158211728 >> 172.21.15.121:0/4158211728 conn(0x7fda4406db20 msgr2=0x7fda440a5b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:21:02.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.315+0000 7fda4ad6b640 1 -- 172.21.15.121:0/4158211728 shutdown_connections 2024-09-17T18:21:02.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.315+0000 7fda4ad6b640 1 -- 172.21.15.121:0/4158211728 wait complete. 2024-09-17T18:21:02.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.316+0000 7fda4ad6b640 10 monclient: build_initial_monmap 2024-09-17T18:21:02.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.316+0000 7fda4ad6b640 10 monclient: monmap: 2024-09-17T18:21:02.328 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:21:02.328 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:21:02.328 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:21:02.328 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:21:02.328 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:21:02.328 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:02.328 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:21:02.328 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:21:02.328 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:21:02.329 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:02.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.316+0000 7fda4ad6b640 1 librados: starting msgr at 2024-09-17T18:21:02.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.316+0000 7fda4ad6b640 1 librados: starting objecter 2024-09-17T18:21:02.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.316+0000 7fda4ad6b640 1 Processor -- start 2024-09-17T18:21:02.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.316+0000 7fda4ad6b640 1 -- start start 2024-09-17T18:21:02.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.316+0000 7fda4ad6b640 1 librados: setting wanted keys 2024-09-17T18:21:02.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.316+0000 7fda4ad6b640 1 librados: calling monclient init 2024-09-17T18:21:02.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.316+0000 7fda4ad6b640 10 monclient: init 2024-09-17T18:21:02.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda4ad6b640 10 monclient: _reopen_session rank -1 2024-09-17T18:21:02.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda4ad6b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T18:21:02.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda4ad6b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda4406fcf0 0x7fda441cd540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:02.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda4ad6b640 10 monclient(hunting): picked mon.noname-b con 0x7fda4406fcf0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:21:02.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda4ad6b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda440a4a10 0x7fda441bcc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:02.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda4ad6b640 10 monclient(hunting): picked mon.noname-a con 0x7fda440a4a10 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:21:02.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda4ad6b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda440a5330 0x7fda4409dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:02.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda4ad6b640 10 monclient(hunting): picked mon.noname-c con 0x7fda440a5330 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:21:02.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda4ad6b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:02.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda4ad6b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fda440a5760 con 0x7fda440a4a10 2024-09-17T18:21:02.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda4ad6b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:02.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda4ad6b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fda440a0e70 con 0x7fda4406fcf0 2024-09-17T18:21:02.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda4ad6b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:02.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda4ad6b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fda440a6480 con 0x7fda440a5330 2024-09-17T18:21:02.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda4ad6b640 10 monclient(hunting): _renew_subs 2024-09-17T18:21:02.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda4ad6b640 10 monclient(hunting): authenticate will time out at 866.034320s 2024-09-17T18:21:02.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda49308640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda440a5330 0x7fda4409dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:02.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda3bfff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda440a4a10 0x7fda441bcc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:02.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda49308640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda440a5330 0x7fda4409dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:56460/0 (socket says 172.21.15.121:56460) 2024-09-17T18:21:02.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda49308640 1 -- 172.21.15.121:0/3638173840 learned_addr learned my addr 172.21.15.121:0/3638173840 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:21:02.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda49308640 10 monclient(hunting): get_auth_request con 0x7fda440a5330 auth_method 0 2024-09-17T18:21:02.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda49308640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:02.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda49308640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:02.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda49308640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:02.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda3bfff640 10 monclient(hunting): get_auth_request con 0x7fda440a4a10 auth_method 0 2024-09-17T18:21:02.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:02.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:02.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:02.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda48b07640 1 --2- 172.21.15.121:0/3638173840 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda4406fcf0 0x7fda441cd540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:02.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda48b07640 10 monclient(hunting): get_auth_request con 0x7fda4406fcf0 auth_method 0 2024-09-17T18:21:02.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda48b07640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:02.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda48b07640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:02.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda48b07640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:02.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda49308640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:02.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda49308640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:02.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda49308640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:02.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:02.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:02.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:02.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda48b07640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:02.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda48b07640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:02.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda48b07640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:02.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda49308640 10 monclient(hunting): handle_auth_done global_id 34151 payload 824 2024-09-17T18:21:02.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda49308640 1 -- 172.21.15.121:0/3638173840 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda4406fcf0 msgr2=0x7fda441cd540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:02.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda49308640 1 --2- 172.21.15.121:0/3638173840 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda4406fcf0 0x7fda441cd540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:02.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda49308640 1 -- 172.21.15.121:0/3638173840 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda440a4a10 msgr2=0x7fda441bcc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:02.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda49308640 1 --2- 172.21.15.121:0/3638173840 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda440a4a10 0x7fda441bcc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:02.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda49308640 10 monclient: _finish_hunting 0 2024-09-17T18:21:02.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda49308640 1 monclient: found mon.noname-c 2024-09-17T18:21:02.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda49308640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:21:02.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda49308640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:21:02.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda49308640 1 -- 172.21.15.121:0/3638173840 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda441aa5f0 con 0x7fda440a5330 2024-09-17T18:21:02.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.317+0000 7fda3bfff640 1 --2- 172.21.15.121:0/3638173840 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda440a4a10 0x7fda441bcc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:21:02.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.319+0000 7fda49308640 1 -- 172.21.15.121:0/3638173840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda440a5330 msgr2=0x7fda4409dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:21:02.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.319+0000 7fda49308640 1 --2- 172.21.15.121:0/3638173840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda440a5330 0x7fda4409dae0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fda3c00cd70 tx=0x7fda3c00a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:21:02.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.319+0000 7fda397fa640 10 client.?.objecter ms_handle_connect 0x7fda440a5330 2024-09-17T18:21:02.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.319+0000 7fda397fa640 10 client.?.objecter resend_mon_ops 2024-09-17T18:21:02.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.320+0000 7fda397fa640 1 -- 172.21.15.121:0/3638173840 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda3c003b90 con 0x7fda440a5330 2024-09-17T18:21:02.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.320+0000 7fda397fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:02.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.320+0000 7fda397fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:21:02.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.320+0000 7fda397fa640 10 monclient: dump: 2024-09-17T18:21:02.339 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:02.339 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:02.339 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:02.339 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:02.339 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:02.339 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:02.339 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:02.339 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:02.339 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:02.339 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:02.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.320+0000 7fda397fa640 10 monclient: _finish_auth 0 2024-09-17T18:21:02.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.320+0000 7fda397fa640 10 monclient: _check_auth_tickets 2024-09-17T18:21:02.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.320+0000 7fda397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:21:02.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.320+0000 7fda397fa640 1 -- 172.21.15.121:0/3638173840 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda3c003d30 con 0x7fda440a5330 2024-09-17T18:21:02.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.320+0000 7fda4ad6b640 5 monclient: authenticate success, global_id 34151 2024-09-17T18:21:02.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.320+0000 7fda4ad6b640 10 monclient: _renew_subs 2024-09-17T18:21:02.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.320+0000 7fda4ad6b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:21:02.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.320+0000 7fda4ad6b640 1 -- 172.21.15.121:0/3638173840 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda441ce420 con 0x7fda440a5330 2024-09-17T18:21:02.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.320+0000 7fda397fa640 10 monclient: handle_config config(0 keys) 2024-09-17T18:21:02.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.320+0000 7fda397fa640 1 -- 172.21.15.121:0/3638173840 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda3c004240 con 0x7fda440a5330 2024-09-17T18:21:02.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.320+0000 7fda397fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:02.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.320+0000 7fda397fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:21:02.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.320+0000 7fda397fa640 10 monclient: dump: 2024-09-17T18:21:02.340 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:02.340 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:02.340 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:02.340 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:02.341 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:02.341 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:02.341 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:02.341 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:02.341 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:02.341 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:02.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.320+0000 7fda4ad6b640 10 client.34151.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:21:02.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.320+0000 7fda4ad6b640 10 monclient: _renew_subs 2024-09-17T18:21:02.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.320+0000 7fda4ad6b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:21:02.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.320+0000 7fda4ad6b640 1 -- 172.21.15.121:0/3638173840 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fda44073420 con 0x7fda440a5330 2024-09-17T18:21:02.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.320+0000 7fda3affd640 4 set_mon_vals no callback set 2024-09-17T18:21:02.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.320+0000 7fda4ad6b640 1 librados: init done 2024-09-17T18:21:02.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.320+0000 7fda4ad6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:21:02.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.320+0000 7fda4ad6b640 1 --2- 172.21.15.121:0/3638173840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda441bd170 0x7fda4409e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:02.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.321+0000 7fda4ad6b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:02.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.321+0000 7fda4ad6b640 1 -- 172.21.15.121:0/3638173840 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fda440a4df0 con 0x7fda441bd170 2024-09-17T18:21:02.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.321+0000 7fda397fa640 1 -- 172.21.15.121:0/3638173840 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fda3c004460 con 0x7fda440a5330 2024-09-17T18:21:02.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.321+0000 7fda48b07640 1 --2- 172.21.15.121:0/3638173840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda441bd170 0x7fda4409e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:02.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.321+0000 7fda48b07640 10 monclient: get_auth_request con 0x7fda441bd170 auth_method 0 2024-09-17T18:21:02.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.321+0000 7fda48b07640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:02.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.321+0000 7fda48b07640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:02.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.321+0000 7fda48b07640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:02.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.321+0000 7fda48b07640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:02.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.321+0000 7fda48b07640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:02.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.321+0000 7fda48b07640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:02.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.321+0000 7fda48b07640 10 monclient(hunting): handle_auth_done global_id 34154 payload 824 2024-09-17T18:21:02.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.321+0000 7fda397fa640 1 --2- 172.21.15.121:0/3638173840 >> v2:172.21.15.173:6812/2423786037 conn(0x7fda30040170 0x7fda30042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:02.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.321+0000 7fda48b07640 1 -- 172.21.15.121:0/3638173840 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fda441bd5a0 con 0x7fda441bd170 2024-09-17T18:21:02.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.321+0000 7fda397fa640 1 -- 172.21.15.121:0/3638173840 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(79..79 src has 1..79) ==== 6635+0+0 (secure 0 0 0) 0x7fda3c054800 con 0x7fda440a5330 2024-09-17T18:21:02.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.321+0000 7fda397fa640 10 client.34151.objecter ms_dispatch 0x7fda44000b90 osd_map(79..79 src has 1..79) 2024-09-17T18:21:02.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.321+0000 7fda397fa640 3 client.34151.objecter handle_osd_map got epochs [79,79] > 0 2024-09-17T18:21:02.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.321+0000 7fda397fa640 3 client.34151.objecter handle_osd_map decoding full epoch 79 2024-09-17T18:21:02.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.321+0000 7fda3bfff640 1 --2- 172.21.15.121:0/3638173840 >> v2:172.21.15.173:6812/2423786037 conn(0x7fda30040170 0x7fda30042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:02.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.321+0000 7fda397fa640 20 client.34151.objecter dump_active .. 0 homeless 2024-09-17T18:21:02.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.321+0000 7fda3bfff640 10 monclient: get_auth_request con 0x7fda30040170 auth_method 0 2024-09-17T18:21:02.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.323+0000 7fda397fa640 10 client.34151.objecter ms_handle_connect 0x7fda441bd170 2024-09-17T18:21:02.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.323+0000 7fda397fa640 10 client.34151.objecter resend_mon_ops 2024-09-17T18:21:02.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.323+0000 7fda48b07640 1 -- 172.21.15.121:0/3638173840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda441bd170 msgr2=0x7fda4409e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:21:02.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.323+0000 7fda48b07640 1 --2- 172.21.15.121:0/3638173840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda441bd170 0x7fda4409e020 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fda40034000 tx=0x7fda400046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:21:02.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.323+0000 7fda397fa640 1 -- 172.21.15.121:0/3638173840 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda40037040 con 0x7fda441bd170 2024-09-17T18:21:02.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.323+0000 7fda397fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:02.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.323+0000 7fda397fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:21:02.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.323+0000 7fda397fa640 10 monclient: dump: 2024-09-17T18:21:02.344 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:02.344 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:02.344 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:02.344 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:02.344 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:02.344 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:02.344 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:02.344 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:02.344 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:02.344 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:02.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.323+0000 7fda397fa640 1 -- 172.21.15.121:0/3638173840 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fda4003c850 con 0x7fda441bd170 2024-09-17T18:21:02.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.323+0000 7fda397fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:21:02.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.323+0000 7fda397fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:21:02.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.323+0000 7fda397fa640 1 -- 172.21.15.121:0/3638173840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda441bd170 msgr2=0x7fda4409e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:02.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.323+0000 7fda397fa640 1 --2- 172.21.15.121:0/3638173840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda441bd170 0x7fda4409e020 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fda40034000 tx=0x7fda400046c0 comp rx=0 tx=0).stop 2024-09-17T18:21:02.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.324+0000 7fda397fa640 1 -- 172.21.15.121:0/3638173840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda441bd170 msgr2=0x7fda4409e020 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:21:02.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.324+0000 7fda397fa640 1 --2- 172.21.15.121:0/3638173840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda441bd170 0x7fda4409e020 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fda40034000 tx=0x7fda400046c0 comp rx=0 tx=0).stop 2024-09-17T18:21:02.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.324+0000 7fda397fa640 10 client.34151.objecter ms_handle_connect 0x7fda30040170 2024-09-17T18:21:02.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.324+0000 7fda3bfff640 1 --2- 172.21.15.121:0/3638173840 >> v2:172.21.15.173:6812/2423786037 conn(0x7fda30040170 0x7fda30042630 secure :-1 s=READY pgs=3867 cs=0 l=1 rev1=1 crypto rx=0x7fda441a9c80 tx=0x7fda34009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:21:02.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.329+0000 7fda4ad6b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:21:02.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.329+0000 7fda4ad6b640 1 --2- 172.21.15.121:0/3638173840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda4409e560 0x7fda440a1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:02.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.330+0000 7fda4ad6b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:02.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.330+0000 7fda4ad6b640 1 -- 172.21.15.121:0/3638173840 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fda4409e970 con 0x7fda4409e560 2024-09-17T18:21:02.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.330+0000 7fda48b07640 1 --2- 172.21.15.121:0/3638173840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda4409e560 0x7fda440a1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:02.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.330+0000 7fda48b07640 10 monclient: get_auth_request con 0x7fda4409e560 auth_method 0 2024-09-17T18:21:02.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.330+0000 7fda48b07640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:02.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.330+0000 7fda48b07640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:02.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.330+0000 7fda48b07640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:02.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.330+0000 7fda48b07640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:02.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.330+0000 7fda48b07640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:02.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.330+0000 7fda48b07640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:02.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.330+0000 7fda48b07640 10 monclient(hunting): handle_auth_done global_id 34157 payload 824 2024-09-17T18:21:02.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.330+0000 7fda48b07640 1 -- 172.21.15.121:0/3638173840 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fda4406cd40 con 0x7fda4409e560 2024-09-17T18:21:02.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.332+0000 7fda48b07640 1 -- 172.21.15.121:0/3638173840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda4409e560 msgr2=0x7fda440a1080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:21:02.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.332+0000 7fda397fa640 10 client.34151.objecter ms_handle_connect 0x7fda4409e560 2024-09-17T18:21:02.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.332+0000 7fda397fa640 10 client.34151.objecter resend_mon_ops 2024-09-17T18:21:02.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.332+0000 7fda48b07640 1 --2- 172.21.15.121:0/3638173840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda4409e560 0x7fda440a1080 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fda440a51e0 tx=0x7fda40008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:21:02.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.332+0000 7fda397fa640 1 -- 172.21.15.121:0/3638173840 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda40037070 con 0x7fda4409e560 2024-09-17T18:21:02.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.332+0000 7fda397fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:02.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.332+0000 7fda397fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:21:02.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.332+0000 7fda397fa640 10 monclient: dump: 2024-09-17T18:21:02.347 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:02.347 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:02.347 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:02.347 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:02.347 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:02.347 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:02.347 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:02.347 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:02.347 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:02.347 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:02.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.333+0000 7fda397fa640 1 -- 172.21.15.121:0/3638173840 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4551 (secure 0 0 0) 0x7fda40003820 con 0x7fda4409e560 2024-09-17T18:21:02.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.333+0000 7fda397fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:21:02.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.333+0000 7fda397fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:21:02.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.333+0000 7fda397fa640 1 -- 172.21.15.121:0/3638173840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda4409e560 msgr2=0x7fda440a1080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:02.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.333+0000 7fda397fa640 1 --2- 172.21.15.121:0/3638173840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda4409e560 0x7fda440a1080 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fda440a51e0 tx=0x7fda40008030 comp rx=0 tx=0).stop 2024-09-17T18:21:02.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.333+0000 7fda397fa640 1 -- 172.21.15.121:0/3638173840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda4409e560 msgr2=0x7fda440a1080 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:21:02.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.333+0000 7fda397fa640 1 --2- 172.21.15.121:0/3638173840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda4409e560 0x7fda440a1080 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fda440a51e0 tx=0x7fda40008030 comp rx=0 tx=0).stop 2024-09-17T18:21:02.348 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:21:02.348 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:21:02.348 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:21:02.348 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:21:02.349 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 10, 2024-09-17T18:21:02.349 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 7609, 2024-09-17T18:21:02.349 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:21:02.349 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:21:02.349 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:21:02.349 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:21:02.349 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:21:02.349 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 6, 2024-09-17T18:21:02.349 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:21:02.349 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:21:02.349 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:21:02.349 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:21:02.349 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:21:02.350 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:21:02.350 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:21:02.350 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:21:02.350 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:21:02.350 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:21:02.350 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:21:02.350 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:21:02.350 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:21:02.350 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:21:02.350 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:21:02.350 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:21:02.350 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:21:02.350 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:21:02.350 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:21:02.351 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:21:02.351 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:21:02.351 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:21:02.351 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:21:02.351 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:21:02.351 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:21:02.351 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:21:02.351 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:21:02.351 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:21:02.351 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:21:02.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:02.351 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:21:02.351 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:21:02.351 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:21:02.352 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:21:02.352 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:21:02.352 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:21:02.352 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:21:02.352 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:21:02.352 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:21:02.352 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:21:02.352 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:21:02.352 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:21:02.352 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:21:02.352 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:21:02.352 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:21:02.352 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:21:02.352 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:21:02.353 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:21:02.353 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:21:02.353 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:21:02.353 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:21:02.353 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:21:02.353 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:21:02.353 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:21:02.353 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:21:02.353 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:21:02.353 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:21:02.353 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:21:02.353 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:21:02.353 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:21:02.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:02.354 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:21:02.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:02.354 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:21:02.354 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:21:02.354 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:21:02.354 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:21:02.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:02.354 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:21:02.354 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:21:02.354 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:21:02.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:02.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:02.354 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:21:02.354 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:21:02.355 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:21:02.355 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:02.355 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:21:02.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:02.355 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:21:02.355 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:21:02.355 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:21:02.355 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:21:02.355 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:21:02.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:02.355 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:02.356 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:21:02.356 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:21:02.356 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:21:02.356 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:21:02.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:02.356 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:21:02.356 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:21:02.356 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:21:02.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:02.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:02.356 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:21:02.356 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:21:02.356 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:21:02.356 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:02.357 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:21:02.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:02.357 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:21:02.357 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:21:02.357 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:21:02.357 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:21:02.357 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:21:02.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:02.357 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:02.357 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:21:02.357 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:21:02.357 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:21:02.357 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:21:02.358 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:02.358 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:21:02.358 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:21:02.358 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:21:02.358 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:02.358 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:02.358 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:21:02.358 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:21:02.358 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:21:02.358 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:02.358 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:21:02.358 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:02.358 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:21:02.358 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:21:02.359 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:21:02.359 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:21:02.359 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:21:02.359 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:02.359 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:21:02.359 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:02.359 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:21:02.359 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:21:02.359 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:02.359 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:21:02.359 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:21:02.359 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:21:02.359 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:02.359 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:21:02.360 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:21:02.360 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:02.360 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:21:02.360 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:21:02.360 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-17T18:21:02.360 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:02.360 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:21:02.360 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:02.360 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:21:02.360 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:21:02.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.333+0000 7fda4ad6b640 10 librados: watch_flush enter 2024-09-17T18:21:02.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.333+0000 7fda4ad6b640 10 librados: watch_flush exit 2024-09-17T18:21:02.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.333+0000 7fda4ad6b640 20 client.34151.objecter shutdown clearing up homeless session... 2024-09-17T18:21:02.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.333+0000 7fda4ad6b640 10 client.34151.objecter successfully canceled tick 2024-09-17T18:21:02.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.333+0000 7fda4ad6b640 1 -- 172.21.15.121:0/3638173840 >> v2:172.21.15.173:6812/2423786037 conn(0x7fda30040170 msgr2=0x7fda30042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:02.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.333+0000 7fda4ad6b640 1 --2- 172.21.15.121:0/3638173840 >> v2:172.21.15.173:6812/2423786037 conn(0x7fda30040170 0x7fda30042630 secure :-1 s=READY pgs=3867 cs=0 l=1 rev1=1 crypto rx=0x7fda441a9c80 tx=0x7fda34009060 comp rx=0 tx=0).stop 2024-09-17T18:21:02.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.333+0000 7fda4ad6b640 10 monclient: shutdown 2024-09-17T18:21:02.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.333+0000 7fda4ad6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:21:02.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.333+0000 7fda4ad6b640 1 -- 172.21.15.121:0/3638173840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda440a5330 msgr2=0x7fda4409dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:02.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.333+0000 7fda49308640 1 -- 172.21.15.121:0/3638173840 reap_dead start 2024-09-17T18:21:02.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.333+0000 7fda4ad6b640 1 --2- 172.21.15.121:0/3638173840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda440a5330 0x7fda4409dae0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fda3c00cd70 tx=0x7fda3c00a660 comp rx=0 tx=0).stop 2024-09-17T18:21:02.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.333+0000 7fda4ad6b640 1 -- 172.21.15.121:0/3638173840 shutdown_connections 2024-09-17T18:21:02.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.333+0000 7fda4ad6b640 1 --2- 172.21.15.121:0/3638173840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda440a5330 0x7fda4409dae0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:02.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.333+0000 7fda4ad6b640 1 -- 172.21.15.121:0/3638173840 >> 172.21.15.121:0/3638173840 conn(0x7fda4406db20 msgr2=0x7fda440a26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:21:02.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.333+0000 7fda4ad6b640 1 -- 172.21.15.121:0/3638173840 shutdown_connections 2024-09-17T18:21:02.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.333+0000 7fda4ad6b640 1 -- 172.21.15.121:0/3638173840 wait complete. 2024-09-17T18:21:02.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.333+0000 7fda4ad6b640 1 librados: shutdown 2024-09-17T18:21:02.363 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:21:02.363 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T18:21:02.364 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T18:21:02.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.458+0000 7fe7aaa6e640 10 monclient: get_monmap_and_config 2024-09-17T18:21:02.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.458+0000 7fe7aaa6e640 10 monclient: build_initial_monmap 2024-09-17T18:21:02.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.458+0000 7fe7aaa6e640 10 monclient: monmap: 2024-09-17T18:21:02.461 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:21:02.461 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:02.461 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:21:02.459851+0000 2024-09-17T18:21:02.461 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:21:02.459851+0000 2024-09-17T18:21:02.461 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:21:02.461 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:02.461 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:21:02.461 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:21:02.461 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:21:02.461 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:21:02.462 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:21:02.462 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:02.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.458+0000 7fe7aaa6e640 1 Processor -- start 2024-09-17T18:21:02.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.459+0000 7fe7aaa6e640 1 -- start start 2024-09-17T18:21:02.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.459+0000 7fe7aaa6e640 10 monclient: init 2024-09-17T18:21:02.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.459+0000 7fe7aaa6e640 10 monclient: _reopen_session rank -1 2024-09-17T18:21:02.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.459+0000 7fe7aaa6e640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-17T18:21:02.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.459+0000 7fe7aaa6e640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe7a409e550 0x7fe7a409e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:02.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.459+0000 7fe7aaa6e640 10 monclient(hunting): picked mon.noname-a con 0x7fe7a409e550 addr v2:172.21.15.121:3300/0 2024-09-17T18:21:02.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.459+0000 7fe7aaa6e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe7a409ee70 addr v1:172.21.15.173:6789/0 2024-09-17T18:21:02.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.459+0000 7fe7aaa6e640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fe7a409f2c0 0x7fe7a413cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:02.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.459+0000 7fe7aaa6e640 10 monclient(hunting): picked mon.noname-c con 0x7fe7a409f2c0 addr v2:172.21.15.173:3300/0 2024-09-17T18:21:02.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.459+0000 7fe7aaa6e640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7a40a4460 con 0x7fe7a409ee70 2024-09-17T18:21:02.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.459+0000 7fe7aaa6e640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:02.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.459+0000 7fe7aaa6e640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fe7a40a4950 con 0x7fe7a409e550 2024-09-17T18:21:02.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.459+0000 7fe7aaa6e640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:02.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.459+0000 7fe7aaa6e640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fe7a40a2b60 con 0x7fe7a409f2c0 2024-09-17T18:21:02.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.459+0000 7fe7aaa6e640 10 monclient(hunting): _renew_subs 2024-09-17T18:21:02.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.459+0000 7fe7aaa6e640 10 monclient(hunting): authenticate will time out at 866.176503s 2024-09-17T18:21:02.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.459+0000 7fe7a880a640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe7a409e550 0x7fe7a409e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:02.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.459+0000 7fe7a880a640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe7a409e550 0x7fe7a409e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:55600/0 (socket says 172.21.15.121:55600) 2024-09-17T18:21:02.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.459+0000 7fe7a880a640 1 -- 172.21.15.121:0/3367155139 learned_addr learned my addr 172.21.15.121:0/3367155139 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:21:02.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.459+0000 7fe7a880a640 10 monclient(hunting): get_auth_request con 0x7fe7a409e550 auth_method 0 2024-09-17T18:21:02.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.459+0000 7fe7a880a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:02.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.459+0000 7fe7a880a640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:02.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.459+0000 7fe7a880a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:02.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.459+0000 7fe7a880a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:02.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.459+0000 7fe7a880a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:02.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.459+0000 7fe7a880a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:02.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.459+0000 7fe7a900b640 1 --2- 172.21.15.121:0/3367155139 >> v2:172.21.15.173:3300/0 conn(0x7fe7a409f2c0 0x7fe7a413cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:02.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.459+0000 7fe7a900b640 10 monclient(hunting): get_auth_request con 0x7fe7a409f2c0 auth_method 0 2024-09-17T18:21:02.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.459+0000 7fe7a900b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:02.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.459+0000 7fe7a900b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:02.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.459+0000 7fe7a900b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:02.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.459+0000 7fe7a880a640 10 monclient(hunting): handle_auth_done global_id 23769 payload 274 2024-09-17T18:21:02.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.459+0000 7fe7a880a640 1 -- 172.21.15.121:0/3367155139 >> v2:172.21.15.173:3300/0 conn(0x7fe7a409f2c0 msgr2=0x7fe7a413cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:21:02.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.459+0000 7fe7a880a640 1 --2- 172.21.15.121:0/3367155139 >> v2:172.21.15.173:3300/0 conn(0x7fe7a409f2c0 0x7fe7a413cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:02.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.459+0000 7fe7a880a640 1 -- 172.21.15.121:0/3367155139 >> v1:172.21.15.173:6789/0 conn(0x7fe7a409ee70 legacy=0x7fe7a4139840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:21:02.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.459+0000 7fe7a880a640 10 monclient: _finish_hunting 0 2024-09-17T18:21:02.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.459+0000 7fe7a880a640 1 monclient: found mon.noname-a 2024-09-17T18:21:02.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.459+0000 7fe7a880a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:21:02.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.459+0000 7fe7a880a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:21:02.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.459+0000 7fe7a880a640 1 -- 172.21.15.121:0/3367155139 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7a409f840 con 0x7fe7a409e550 2024-09-17T18:21:02.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.462+0000 7fe7a880a640 1 -- 172.21.15.121:0/3367155139 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7a409e550 msgr2=0x7fe7a409e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:21:02.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.462+0000 7fe7a880a640 1 --2- 172.21.15.121:0/3367155139 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7a409e550 0x7fe7a409e930 secure :-1 s=READY pgs=6835 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4077910 tx=0x7fe79802a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4bfaa0cd36e955a server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:21:02.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.462+0000 7fe7a37fe640 1 -- 172.21.15.121:0/3367155139 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe798037070 con 0x7fe7a409e550 2024-09-17T18:21:02.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.462+0000 7fe7a37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:02.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.462+0000 7fe7a37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:21:02.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.462+0000 7fe7a37fe640 10 monclient: dump: 2024-09-17T18:21:02.474 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:02.474 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:02.474 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:02.474 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:02.474 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:02.474 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:02.474 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:02.474 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:02.474 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:02.474 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:02.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.462+0000 7fe7a37fe640 10 monclient: _finish_auth 0 2024-09-17T18:21:02.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.462+0000 7fe7a37fe640 10 monclient: _check_auth_tickets 2024-09-17T18:21:02.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.462+0000 7fe7a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:21:02.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.462+0000 7fe7a37fe640 1 -- 172.21.15.121:0/3367155139 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe798032640 con 0x7fe7a409e550 2024-09-17T18:21:02.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.462+0000 7fe7aaa6e640 5 monclient: authenticate success, global_id 23769 2024-09-17T18:21:02.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.462+0000 7fe7aaa6e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:21:02.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.462+0000 7fe7a37fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:21:02.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.462+0000 7fe7aaa6e640 10 monclient: get_monmap_and_config success 2024-09-17T18:21:02.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.462+0000 7fe7aaa6e640 4 set_mon_vals no callback set 2024-09-17T18:21:02.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.462+0000 7fe7a37fe640 1 -- 172.21.15.121:0/3367155139 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe798032a10 con 0x7fe7a409e550 2024-09-17T18:21:02.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.462+0000 7fe7a37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:02.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.462+0000 7fe7a37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:21:02.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.462+0000 7fe7a37fe640 10 monclient: dump: 2024-09-17T18:21:02.475 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:02.475 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:02.475 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:02.476 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:02.476 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:02.476 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:02.476 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:02.476 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:02.476 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:02.476 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:02.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.463+0000 7fe7aaa6e640 10 monclient: shutdown 2024-09-17T18:21:02.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.463+0000 7fe7aaa6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:21:02.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.463+0000 7fe7aaa6e640 1 -- 172.21.15.121:0/3367155139 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7a409e550 msgr2=0x7fe7a409e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:02.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.463+0000 7fe7aaa6e640 1 --2- 172.21.15.121:0/3367155139 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7a409e550 0x7fe7a409e930 secure :-1 s=READY pgs=6835 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4077910 tx=0x7fe79802a0b0 comp rx=0 tx=0).stop 2024-09-17T18:21:02.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.463+0000 7fe7aaa6e640 1 -- 172.21.15.121:0/3367155139 shutdown_connections 2024-09-17T18:21:02.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.463+0000 7fe7aaa6e640 1 --2- 172.21.15.121:0/3367155139 >> v2:172.21.15.173:3300/0 conn(0x7fe7a409f2c0 0x7fe7a413cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:02.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.463+0000 7fe7aaa6e640 1 --2- 172.21.15.121:0/3367155139 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7a409e550 0x7fe7a409e930 unknown :-1 s=CLOSED pgs=6835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:02.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.463+0000 7fe7aaa6e640 1 -- 172.21.15.121:0/3367155139 >> 172.21.15.121:0/3367155139 conn(0x7fe7a40a7830 msgr2=0x7fe7a40a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:21:02.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.463+0000 7fe7aaa6e640 1 -- 172.21.15.121:0/3367155139 shutdown_connections 2024-09-17T18:21:02.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.463+0000 7fe7aaa6e640 1 -- 172.21.15.121:0/3367155139 wait complete. 2024-09-17T18:21:02.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.464+0000 7fe7aaa6e640 10 monclient: build_initial_monmap 2024-09-17T18:21:02.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.464+0000 7fe7aaa6e640 10 monclient: monmap: 2024-09-17T18:21:02.477 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:21:02.477 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:21:02.477 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:21:02.477 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:21:02.477 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:21:02.477 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:02.477 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:21:02.477 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:21:02.477 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:21:02.478 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:02.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.464+0000 7fe7aaa6e640 1 librados: starting msgr at 2024-09-17T18:21:02.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.464+0000 7fe7aaa6e640 1 librados: starting objecter 2024-09-17T18:21:02.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.464+0000 7fe7aaa6e640 1 Processor -- start 2024-09-17T18:21:02.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.464+0000 7fe7aaa6e640 1 -- start start 2024-09-17T18:21:02.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.464+0000 7fe7aaa6e640 1 librados: setting wanted keys 2024-09-17T18:21:02.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.464+0000 7fe7aaa6e640 1 librados: calling monclient init 2024-09-17T18:21:02.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.464+0000 7fe7aaa6e640 10 monclient: init 2024-09-17T18:21:02.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7aaa6e640 10 monclient: _reopen_session rank -1 2024-09-17T18:21:02.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7aaa6e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T18:21:02.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7aaa6e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7a409e550 0x7fe7a41ab2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:02.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7aaa6e640 10 monclient(hunting): picked mon.noname-c con 0x7fe7a409e550 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:21:02.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7aaa6e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7a409ee70 0x7fe7a41c5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:02.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7aaa6e640 10 monclient(hunting): picked mon.noname-a con 0x7fe7a409ee70 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:21:02.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7aaa6e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7a409f2c0 0x7fe7a41c7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:02.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7aaa6e640 10 monclient(hunting): picked mon.noname-b con 0x7fe7a409f2c0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:21:02.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7aaa6e640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:02.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7aaa6e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe7a409de60 con 0x7fe7a409ee70 2024-09-17T18:21:02.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7aaa6e640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:02.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7aaa6e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe7a409dce0 con 0x7fe7a409f2c0 2024-09-17T18:21:02.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7aaa6e640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:02.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7aaa6e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe7a40747e0 con 0x7fe7a409e550 2024-09-17T18:21:02.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7aaa6e640 10 monclient(hunting): _renew_subs 2024-09-17T18:21:02.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7aaa6e640 10 monclient(hunting): authenticate will time out at 866.182182s 2024-09-17T18:21:02.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7a3fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7a409ee70 0x7fe7a41c5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:02.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7a3fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7a409ee70 0x7fe7a41c5050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:55606/0 (socket says 172.21.15.121:55606) 2024-09-17T18:21:02.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7a3fff640 1 -- 172.21.15.121:0/3045004958 learned_addr learned my addr 172.21.15.121:0/3045004958 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:21:02.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7a3fff640 10 monclient(hunting): get_auth_request con 0x7fe7a409ee70 auth_method 0 2024-09-17T18:21:02.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:02.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:02.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:02.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7a880a640 1 --2- 172.21.15.121:0/3045004958 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7a409e550 0x7fe7a41ab2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:02.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7a900b640 1 --2- 172.21.15.121:0/3045004958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7a409f2c0 0x7fe7a41c7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:02.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7a880a640 10 monclient(hunting): get_auth_request con 0x7fe7a409e550 auth_method 0 2024-09-17T18:21:02.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7a880a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:02.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7a880a640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:02.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7a880a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:02.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7a900b640 10 monclient(hunting): get_auth_request con 0x7fe7a409f2c0 auth_method 0 2024-09-17T18:21:02.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7a900b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:02.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7a900b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:02.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7a900b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:02.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:02.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:02.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:02.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7a880a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:02.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7a880a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:02.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7a880a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:02.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7a900b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:02.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7a900b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:02.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7a900b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:02.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7a3fff640 10 monclient(hunting): handle_auth_done global_id 23772 payload 824 2024-09-17T18:21:02.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7a3fff640 1 -- 172.21.15.121:0/3045004958 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7a409e550 msgr2=0x7fe7a41ab2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:02.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7a3fff640 1 --2- 172.21.15.121:0/3045004958 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7a409e550 0x7fe7a41ab2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:02.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7a3fff640 1 -- 172.21.15.121:0/3045004958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7a409f2c0 msgr2=0x7fe7a41c7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:02.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7a3fff640 1 --2- 172.21.15.121:0/3045004958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7a409f2c0 0x7fe7a41c7c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:02.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7a3fff640 10 monclient: _finish_hunting 0 2024-09-17T18:21:02.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7a3fff640 1 monclient: found mon.noname-a 2024-09-17T18:21:02.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:21:02.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7a3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:21:02.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7a3fff640 1 -- 172.21.15.121:0/3045004958 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7a41abc40 con 0x7fe7a409ee70 2024-09-17T18:21:02.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.465+0000 7fe7a880a640 1 --2- 172.21.15.121:0/3045004958 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7a409e550 0x7fe7a41ab2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:21:02.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.467+0000 7fe7a3fff640 1 -- 172.21.15.121:0/3045004958 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7a409ee70 msgr2=0x7fe7a41c5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:21:02.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.467+0000 7fe7a3fff640 1 --2- 172.21.15.121:0/3045004958 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7a409ee70 0x7fe7a41c5050 secure :-1 s=READY pgs=6836 cs=0 l=1 rev1=1 crypto rx=0x7fe79000c460 tx=0x7fe79000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:21:02.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.467+0000 7fe7a17fa640 10 client.?.objecter ms_handle_connect 0x7fe7a409ee70 2024-09-17T18:21:02.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.467+0000 7fe7a17fa640 10 client.?.objecter resend_mon_ops 2024-09-17T18:21:02.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.468+0000 7fe7a17fa640 1 -- 172.21.15.121:0/3045004958 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe790016020 con 0x7fe7a409ee70 2024-09-17T18:21:02.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.468+0000 7fe7a17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:02.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.468+0000 7fe7a17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:21:02.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.468+0000 7fe7a17fa640 10 monclient: dump: 2024-09-17T18:21:02.483 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:02.483 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:02.483 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:02.483 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:02.483 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:02.483 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:02.483 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:02.483 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:02.483 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:02.483 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:02.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.468+0000 7fe7a17fa640 10 monclient: _finish_auth 0 2024-09-17T18:21:02.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.468+0000 7fe7a17fa640 10 monclient: _check_auth_tickets 2024-09-17T18:21:02.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.468+0000 7fe7a17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:21:02.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.468+0000 7fe7a17fa640 1 -- 172.21.15.121:0/3045004958 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe790012a80 con 0x7fe7a409ee70 2024-09-17T18:21:02.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.468+0000 7fe7aaa6e640 5 monclient: authenticate success, global_id 23772 2024-09-17T18:21:02.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.468+0000 7fe7aaa6e640 10 monclient: _renew_subs 2024-09-17T18:21:02.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.468+0000 7fe7aaa6e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:21:02.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.468+0000 7fe7aaa6e640 1 -- 172.21.15.121:0/3045004958 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7a41cba10 con 0x7fe7a409ee70 2024-09-17T18:21:02.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.468+0000 7fe7a17fa640 10 monclient: handle_config config(0 keys) 2024-09-17T18:21:02.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.468+0000 7fe7a17fa640 1 -- 172.21.15.121:0/3045004958 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe790011380 con 0x7fe7a409ee70 2024-09-17T18:21:02.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.468+0000 7fe7a17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:02.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.468+0000 7fe7a17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:21:02.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.468+0000 7fe7a17fa640 10 monclient: dump: 2024-09-17T18:21:02.484 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:02.484 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:02.484 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:02.484 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:02.484 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:02.485 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:02.485 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:02.485 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:02.485 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:02.485 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:02.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.468+0000 7fe7a2ffd640 4 set_mon_vals no callback set 2024-09-17T18:21:02.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.468+0000 7fe7aaa6e640 10 client.23772.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:21:02.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.468+0000 7fe7aaa6e640 10 monclient: _renew_subs 2024-09-17T18:21:02.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.468+0000 7fe7aaa6e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:21:02.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.468+0000 7fe7aaa6e640 1 -- 172.21.15.121:0/3045004958 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7a41acab0 con 0x7fe7a409ee70 2024-09-17T18:21:02.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.468+0000 7fe7aaa6e640 1 librados: init done 2024-09-17T18:21:02.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.469+0000 7fe7aaa6e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:21:02.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.469+0000 7fe7aaa6e640 1 --2- 172.21.15.121:0/3045004958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7a41c5590 0x7fe7a41c8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:02.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.469+0000 7fe7aaa6e640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:02.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.469+0000 7fe7aaa6e640 1 -- 172.21.15.121:0/3045004958 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe7a409e930 con 0x7fe7a41c5590 2024-09-17T18:21:02.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.469+0000 7fe7a17fa640 1 -- 172.21.15.121:0/3045004958 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fe790019020 con 0x7fe7a409ee70 2024-09-17T18:21:02.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.469+0000 7fe7a880a640 1 --2- 172.21.15.121:0/3045004958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7a41c5590 0x7fe7a41c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:02.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.469+0000 7fe7a880a640 10 monclient: get_auth_request con 0x7fe7a41c5590 auth_method 0 2024-09-17T18:21:02.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.469+0000 7fe7a880a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:02.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.469+0000 7fe7a880a640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:02.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.469+0000 7fe7a880a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:02.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.469+0000 7fe7a880a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:02.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.469+0000 7fe7a880a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:02.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.469+0000 7fe7a880a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:02.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.469+0000 7fe7a17fa640 1 --2- 172.21.15.121:0/3045004958 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe7880401f0 0x7fe7880426b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:02.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.469+0000 7fe7a17fa640 1 -- 172.21.15.121:0/3045004958 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6635+0+0 (secure 0 0 0) 0x7fe790053620 con 0x7fe7a409ee70 2024-09-17T18:21:02.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.469+0000 7fe7a17fa640 10 client.23772.objecter ms_dispatch 0x7fe7a4000b90 osd_map(79..79 src has 1..79) 2024-09-17T18:21:02.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.469+0000 7fe7a17fa640 3 client.23772.objecter handle_osd_map got epochs [79,79] > 0 2024-09-17T18:21:02.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.469+0000 7fe7a17fa640 3 client.23772.objecter handle_osd_map decoding full epoch 79 2024-09-17T18:21:02.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.469+0000 7fe7a880a640 10 monclient(hunting): handle_auth_done global_id 24178 payload 824 2024-09-17T18:21:02.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.469+0000 7fe7a880a640 1 -- 172.21.15.121:0/3045004958 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe7a41ac270 con 0x7fe7a41c5590 2024-09-17T18:21:02.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.469+0000 7fe7a17fa640 20 client.23772.objecter dump_active .. 0 homeless 2024-09-17T18:21:02.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.469+0000 7fe7a900b640 1 --2- 172.21.15.121:0/3045004958 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe7880401f0 0x7fe7880426b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:02.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.469+0000 7fe7a900b640 10 monclient: get_auth_request con 0x7fe7880401f0 auth_method 0 2024-09-17T18:21:02.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.472+0000 7fe7a17fa640 10 client.23772.objecter ms_handle_connect 0x7fe7a41c5590 2024-09-17T18:21:02.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.472+0000 7fe7a17fa640 10 client.23772.objecter resend_mon_ops 2024-09-17T18:21:02.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.472+0000 7fe7a880a640 1 -- 172.21.15.121:0/3045004958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7a41c5590 msgr2=0x7fe7a41c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:21:02.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.472+0000 7fe7a880a640 1 --2- 172.21.15.121:0/3045004958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7a41c5590 0x7fe7a41c8170 secure :-1 s=READY pgs=7113 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4074790 tx=0x7fe7980046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:21:02.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.472+0000 7fe7a900b640 1 --2- 172.21.15.121:0/3045004958 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe7880401f0 0x7fe7880426b0 secure :-1 s=READY pgs=3868 cs=0 l=1 rev1=1 crypto rx=0x7fe79c003aa0 tx=0x7fe79c009820 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:21:02.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.472+0000 7fe7a17fa640 10 client.23772.objecter ms_handle_connect 0x7fe7880401f0 2024-09-17T18:21:02.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.472+0000 7fe7a17fa640 1 -- 172.21.15.121:0/3045004958 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe798043070 con 0x7fe7a41c5590 2024-09-17T18:21:02.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.472+0000 7fe7a17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:02.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.472+0000 7fe7a17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:21:02.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.472+0000 7fe7a17fa640 10 monclient: dump: 2024-09-17T18:21:02.491 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:02.491 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:02.491 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:02.491 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:02.491 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:02.491 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:02.491 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:02.491 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:02.491 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:02.491 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:02.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.472+0000 7fe7a17fa640 1 -- 172.21.15.121:0/3045004958 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe79803ca60 con 0x7fe7a41c5590 2024-09-17T18:21:02.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.472+0000 7fe7a17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:21:02.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.472+0000 7fe7a17fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:21:02.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.472+0000 7fe7a17fa640 1 -- 172.21.15.121:0/3045004958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7a41c5590 msgr2=0x7fe7a41c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:02.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.472+0000 7fe7a17fa640 1 --2- 172.21.15.121:0/3045004958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7a41c5590 0x7fe7a41c8170 secure :-1 s=READY pgs=7113 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4074790 tx=0x7fe7980046c0 comp rx=0 tx=0).stop 2024-09-17T18:21:02.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.472+0000 7fe7a17fa640 1 -- 172.21.15.121:0/3045004958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7a41c5590 msgr2=0x7fe7a41c8170 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:21:02.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.472+0000 7fe7a17fa640 1 --2- 172.21.15.121:0/3045004958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7a41c5590 0x7fe7a41c8170 secure :-1 s=CLOSED pgs=7113 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4074790 tx=0x7fe7980046c0 comp rx=0 tx=0).stop 2024-09-17T18:21:02.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.479+0000 7fe7aaa6e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:21:02.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.479+0000 7fe7aaa6e640 1 --2- 172.21.15.121:0/3045004958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7a41c86b0 0x7fe7a41a43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:02.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.479+0000 7fe7aaa6e640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:02.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.479+0000 7fe7aaa6e640 1 -- 172.21.15.121:0/3045004958 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe7a41c8ac0 con 0x7fe7a41c86b0 2024-09-17T18:21:02.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.479+0000 7fe7a880a640 1 --2- 172.21.15.121:0/3045004958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7a41c86b0 0x7fe7a41a43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:02.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.479+0000 7fe7a880a640 10 monclient: get_auth_request con 0x7fe7a41c86b0 auth_method 0 2024-09-17T18:21:02.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.479+0000 7fe7a880a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:02.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.479+0000 7fe7a880a640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:02.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.479+0000 7fe7a880a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:02.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.479+0000 7fe7a880a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:02.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.479+0000 7fe7a880a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:02.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.479+0000 7fe7a880a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:02.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.479+0000 7fe7a880a640 10 monclient(hunting): handle_auth_done global_id 24181 payload 824 2024-09-17T18:21:02.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.479+0000 7fe7a880a640 1 -- 172.21.15.121:0/3045004958 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe7a406cd40 con 0x7fe7a41c86b0 2024-09-17T18:21:02.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.482+0000 7fe7a880a640 1 -- 172.21.15.121:0/3045004958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7a41c86b0 msgr2=0x7fe7a41a43c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:21:02.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.482+0000 7fe7a17fa640 10 client.23772.objecter ms_handle_connect 0x7fe7a41c86b0 2024-09-17T18:21:02.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.482+0000 7fe7a17fa640 10 client.23772.objecter resend_mon_ops 2024-09-17T18:21:02.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.482+0000 7fe7a880a640 1 --2- 172.21.15.121:0/3045004958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7a41c86b0 0x7fe7a41a43c0 secure :-1 s=READY pgs=7114 cs=0 l=1 rev1=1 crypto rx=0x7fe79802a0f0 tx=0x7fe798008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:21:02.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.482+0000 7fe7a17fa640 1 -- 172.21.15.121:0/3045004958 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe798037040 con 0x7fe7a41c86b0 2024-09-17T18:21:02.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.482+0000 7fe7a17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:02.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.482+0000 7fe7a17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:21:02.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.482+0000 7fe7a17fa640 10 monclient: dump: 2024-09-17T18:21:02.493 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:02.493 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:02.494 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:02.494 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:02.494 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:02.494 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:02.494 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:02.494 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:02.494 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:02.494 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:02.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.482+0000 7fe7a17fa640 1 -- 172.21.15.121:0/3045004958 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4888 (secure 0 0 0) 0x7fe798047e10 con 0x7fe7a41c86b0 2024-09-17T18:21:02.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.482+0000 7fe7a17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:21:02.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.482+0000 7fe7a17fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:21:02.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.482+0000 7fe7a17fa640 1 -- 172.21.15.121:0/3045004958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7a41c86b0 msgr2=0x7fe7a41a43c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:02.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.482+0000 7fe7a17fa640 1 --2- 172.21.15.121:0/3045004958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7a41c86b0 0x7fe7a41a43c0 secure :-1 s=READY pgs=7114 cs=0 l=1 rev1=1 crypto rx=0x7fe79802a0f0 tx=0x7fe798008030 comp rx=0 tx=0).stop 2024-09-17T18:21:02.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.482+0000 7fe7a17fa640 1 -- 172.21.15.121:0/3045004958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7a41c86b0 msgr2=0x7fe7a41a43c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:21:02.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.482+0000 7fe7a17fa640 1 --2- 172.21.15.121:0/3045004958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7a41c86b0 0x7fe7a41a43c0 unknown :-1 s=CLOSED pgs=7114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:02.495 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:21:02.495 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:21:02.495 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:21:02.495 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:21:02.495 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 10, 2024-09-17T18:21:02.495 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 135429, 2024-09-17T18:21:02.495 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:21:02.495 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:21:02.495 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:21:02.496 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:21:02.496 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:21:02.496 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 6, 2024-09-17T18:21:02.496 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:21:02.496 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:21:02.496 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:21:02.496 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:21:02.496 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:21:02.496 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:21:02.496 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:21:02.496 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:21:02.496 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:21:02.496 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:21:02.496 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:21:02.497 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:21:02.497 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:21:02.497 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:21:02.497 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:21:02.497 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:21:02.497 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:21:02.497 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:21:02.497 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:21:02.497 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:21:02.497 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:21:02.497 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:21:02.497 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:21:02.497 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:21:02.497 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:21:02.497 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:21:02.498 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:21:02.498 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:21:02.498 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:21:02.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:02.498 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:21:02.498 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:21:02.498 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:21:02.498 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:21:02.498 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:21:02.498 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:21:02.498 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:21:02.498 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:21:02.498 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:21:02.498 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:21:02.499 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:21:02.499 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:21:02.499 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:21:02.499 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:21:02.499 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:21:02.499 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:21:02.499 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:21:02.499 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:21:02.499 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:21:02.499 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:21:02.499 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:21:02.499 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:21:02.499 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:21:02.499 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:21:02.499 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:21:02.500 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:21:02.500 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:21:02.500 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:21:02.500 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:21:02.500 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:21:02.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:02.500 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:21:02.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:02.500 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:21:02.500 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:21:02.500 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:21:02.500 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:21:02.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:02.500 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:21:02.501 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:21:02.501 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:21:02.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:02.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:02.501 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:21:02.501 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:21:02.501 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:21:02.501 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:02.501 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:21:02.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:02.501 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:21:02.501 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:21:02.501 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:21:02.501 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:21:02.502 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:21:02.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:02.502 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:02.502 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:21:02.502 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:21:02.502 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:21:02.502 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:21:02.502 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:02.502 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:21:02.502 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:21:02.502 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:21:02.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:02.502 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:02.502 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:21:02.503 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:21:02.503 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:21:02.503 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:02.503 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:21:02.503 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:02.503 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:21:02.503 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:21:02.503 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:21:02.503 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:21:02.503 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:21:02.503 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:02.503 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:02.503 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:21:02.503 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:21:02.504 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:21:02.504 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:21:02.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:02.504 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:21:02.504 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:21:02.504 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:21:02.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:02.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:02.504 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:21:02.504 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:21:02.504 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:21:02.504 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:02.504 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:21:02.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:02.504 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:21:02.505 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:21:02.505 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:21:02.505 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:21:02.505 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:21:02.505 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:02.505 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:21:02.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:02.505 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:21:02.505 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:21:02.505 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:02.505 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:21:02.505 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:21:02.505 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:21:02.505 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:02.506 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:21:02.506 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:21:02.506 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:02.506 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:21:02.506 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:21:02.506 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 4 2024-09-17T18:21:02.506 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:02.506 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:21:02.506 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:21:02.506 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:02.506 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:21:02.506 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:21:02.506 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 10 2024-09-17T18:21:02.506 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:02.506 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:21:02.507 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": [ 2024-09-17T18:21:02.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:02.507 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:21:02.507 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:21:02.507 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:21:02.507 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:02.507 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:21:02.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:02.507 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:21:02.507 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:21:02.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.482+0000 7fe7aaa6e640 10 librados: watch_flush enter 2024-09-17T18:21:02.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.482+0000 7fe7aaa6e640 10 librados: watch_flush exit 2024-09-17T18:21:02.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.482+0000 7fe7aaa6e640 20 client.23772.objecter shutdown clearing up homeless session... 2024-09-17T18:21:02.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.482+0000 7fe7aaa6e640 10 client.23772.objecter successfully canceled tick 2024-09-17T18:21:02.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.482+0000 7fe7aaa6e640 1 -- 172.21.15.121:0/3045004958 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe7880401f0 msgr2=0x7fe7880426b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:02.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.482+0000 7fe7aaa6e640 1 --2- 172.21.15.121:0/3045004958 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe7880401f0 0x7fe7880426b0 secure :-1 s=READY pgs=3868 cs=0 l=1 rev1=1 crypto rx=0x7fe79c003aa0 tx=0x7fe79c009820 comp rx=0 tx=0).stop 2024-09-17T18:21:02.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.483+0000 7fe7aaa6e640 10 monclient: shutdown 2024-09-17T18:21:02.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.483+0000 7fe7aaa6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:21:02.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.483+0000 7fe7aaa6e640 1 -- 172.21.15.121:0/3045004958 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7a409ee70 msgr2=0x7fe7a41c5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:02.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.483+0000 7fe7aaa6e640 1 --2- 172.21.15.121:0/3045004958 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7a409ee70 0x7fe7a41c5050 secure :-1 s=READY pgs=6836 cs=0 l=1 rev1=1 crypto rx=0x7fe79000c460 tx=0x7fe79000c920 comp rx=0 tx=0).stop 2024-09-17T18:21:02.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.483+0000 7fe7a900b640 1 -- 172.21.15.121:0/3045004958 reap_dead start 2024-09-17T18:21:02.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.483+0000 7fe7a900b640 1 -- 172.21.15.121:0/3045004958 reap_dead start 2024-09-17T18:21:02.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.483+0000 7fe7aaa6e640 1 -- 172.21.15.121:0/3045004958 shutdown_connections 2024-09-17T18:21:02.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.483+0000 7fe7aaa6e640 1 -- 172.21.15.121:0/3045004958 >> 172.21.15.121:0/3045004958 conn(0x7fe7a40a7830 msgr2=0x7fe7a4071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:21:02.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.483+0000 7fe7aaa6e640 1 -- 172.21.15.121:0/3045004958 shutdown_connections 2024-09-17T18:21:02.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.483+0000 7fe7aaa6e640 1 -- 172.21.15.121:0/3045004958 wait complete. 2024-09-17T18:21:02.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:02.483+0000 7fe7aaa6e640 1 librados: shutdown 2024-09-17T18:21:02.510 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:21:02.511 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-17T18:21:02.511 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2024-09-17T18:21:02.511 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-17T18:21:02.511 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-17T18:21:02.511 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-17T18:21:02.511 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T18:21:02.524 INFO:tasks.ceph.mon.a:Stopped 2024-09-17T18:21:02.524 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-17T18:21:02.524 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-17T18:21:02.524 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-17T18:21:17.525 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-17T18:21:17.526 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-17T18:21:17.526 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-17T18:21:17.527 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:21:17.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.626+0000 7fa817f52640 10 monclient: get_monmap_and_config 2024-09-17T18:21:17.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.626+0000 7fa817f52640 10 monclient: build_initial_monmap 2024-09-17T18:21:17.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.626+0000 7fa817f52640 10 monclient: monmap: 2024-09-17T18:21:17.630 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:21:17.630 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:17.630 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:21:17.628115+0000 2024-09-17T18:21:17.630 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:21:17.628115+0000 2024-09-17T18:21:17.630 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:21:17.630 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:17.630 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:21:17.630 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:21:17.630 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:21:17.630 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:21:17.631 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:21:17.631 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:17.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.627+0000 7fa817f52640 1 Processor -- start 2024-09-17T18:21:17.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.627+0000 7fa817f52640 1 -- start start 2024-09-17T18:21:17.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.627+0000 7fa817f52640 10 monclient: init 2024-09-17T18:21:17.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.627+0000 7fa817f52640 10 monclient: _reopen_session rank -1 2024-09-17T18:21:17.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.627+0000 7fa817f52640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-17T18:21:17.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.627+0000 7fa817f52640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8100a1a50 0x7fa8100a1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:17.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.627+0000 7fa817f52640 10 monclient(hunting): picked mon.noname-b con 0x7fa8100a1a50 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:21:17.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.627+0000 7fa817f52640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa8100a2370 0x7fa810141040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:17.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.627+0000 7fa817f52640 10 monclient(hunting): picked mon.noname-a con 0x7fa8100a2370 addr v2:172.21.15.121:3300/0 2024-09-17T18:21:17.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.627+0000 7fa817f52640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa8100a2750 addr v1:172.21.15.173:6789/0 2024-09-17T18:21:17.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.627+0000 7fa817f52640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8100a0960 con 0x7fa8100a2750 2024-09-17T18:21:17.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.627+0000 7fa817f52640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:17.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.627+0000 7fa817f52640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fa8100a3000 con 0x7fa8100a2370 2024-09-17T18:21:17.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.627+0000 7fa817f52640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:17.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.627+0000 7fa817f52640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa810077730 con 0x7fa8100a1a50 2024-09-17T18:21:17.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.627+0000 7fa817f52640 10 monclient(hunting): _renew_subs 2024-09-17T18:21:17.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.627+0000 7fa817f52640 10 monclient(hunting): authenticate will time out at 881.344855s 2024-09-17T18:21:17.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.627+0000 7fa8154ed640 1 -- >> v2:172.21.15.121:3300/0 conn(0x7fa8100a2370 msgr2=0x7fa810141040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:21:17.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.627+0000 7fa8154ed640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa8100a2370 0x7fa810141040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:21:17.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.627+0000 7fa815cee640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8100a1a50 0x7fa8100a1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:17.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.628+0000 7fa815cee640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8100a1a50 0x7fa8100a1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:35600/0 (socket says 172.21.15.121:35600) 2024-09-17T18:21:17.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.628+0000 7fa815cee640 1 -- 172.21.15.121:0/125984064 learned_addr learned my addr 172.21.15.121:0/125984064 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:21:17.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.628+0000 7fa815cee640 10 monclient(hunting): get_auth_request con 0x7fa8100a1a50 auth_method 0 2024-09-17T18:21:17.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.628+0000 7fa815cee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:17.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.628+0000 7fa815cee640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:17.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.628+0000 7fa815cee640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:17.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.628+0000 7fa815cee640 1 -- 172.21.15.121:0/125984064 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8100a1a50 msgr2=0x7fa8100a1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:21:17.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.628+0000 7fa815cee640 1 -- 172.21.15.121:0/125984064 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8100a1a50 msgr2=0x7fa8100a1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:21:17.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.628+0000 7fa815cee640 1 --2- 172.21.15.121:0/125984064 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8100a1a50 0x7fa8100a1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:21:17.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.628+0000 7fa815cee640 1 --2- 172.21.15.121:0/125984064 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8100a1a50 0x7fa8100a1e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:21:17.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.630+0000 7fa8164ef640 1 -- 172.21.15.121:0/125984064 >> v1:172.21.15.173:6789/0 conn(0x7fa8100a2750 legacy=0x7fa810143590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:21:17.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.829+0000 7fa8154ed640 1 -- 172.21.15.121:0/125984064 >> v2:172.21.15.121:3300/0 conn(0x7fa8100a2370 msgr2=0x7fa810141040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:21:17.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.829+0000 7fa8154ed640 1 --2- 172.21.15.121:0/125984064 >> v2:172.21.15.121:3300/0 conn(0x7fa8100a2370 0x7fa810141040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T18:21:17.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.829+0000 7fa815cee640 1 --2- 172.21.15.121:0/125984064 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8100a1a50 0x7fa8100a1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:17.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.829+0000 7fa815cee640 10 monclient(hunting): get_auth_request con 0x7fa8100a1a50 auth_method 0 2024-09-17T18:21:17.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.829+0000 7fa815cee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:17.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.829+0000 7fa815cee640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:17.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.829+0000 7fa815cee640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:21:17.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.829+0000 7fa815cee640 1 -- 172.21.15.121:0/125984064 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8100a1a50 msgr2=0x7fa8100a1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:21:17.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.829+0000 7fa815cee640 1 -- 172.21.15.121:0/125984064 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8100a1a50 msgr2=0x7fa8100a1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:21:17.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.829+0000 7fa815cee640 1 --2- 172.21.15.121:0/125984064 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8100a1a50 0x7fa8100a1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:21:17.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:17.829+0000 7fa815cee640 1 --2- 172.21.15.121:0/125984064 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8100a1a50 0x7fa8100a1e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T18:21:18.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:18.230+0000 7fa815cee640 1 --2- 172.21.15.121:0/125984064 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8100a1a50 0x7fa8100a1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:18.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:18.230+0000 7fa815cee640 10 monclient(hunting): get_auth_request con 0x7fa8100a1a50 auth_method 0 2024-09-17T18:21:18.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:18.230+0000 7fa815cee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:18.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:18.230+0000 7fa815cee640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:18.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:18.230+0000 7fa815cee640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:21:18.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:18.230+0000 7fa815cee640 1 -- 172.21.15.121:0/125984064 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8100a1a50 msgr2=0x7fa8100a1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:21:18.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:18.230+0000 7fa815cee640 1 -- 172.21.15.121:0/125984064 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8100a1a50 msgr2=0x7fa8100a1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:21:18.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:18.230+0000 7fa815cee640 1 --2- 172.21.15.121:0/125984064 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8100a1a50 0x7fa8100a1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:21:18.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:18.230+0000 7fa815cee640 1 --2- 172.21.15.121:0/125984064 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8100a1a50 0x7fa8100a1e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T18:21:18.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:18.230+0000 7fa8154ed640 1 -- 172.21.15.121:0/125984064 >> v2:172.21.15.121:3300/0 conn(0x7fa8100a2370 msgr2=0x7fa810141040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:21:18.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:18.231+0000 7fa8154ed640 1 --2- 172.21.15.121:0/125984064 >> v2:172.21.15.121:3300/0 conn(0x7fa8100a2370 0x7fa810141040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T18:21:19.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:19.032+0000 7fa815cee640 1 --2- 172.21.15.121:0/125984064 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8100a1a50 0x7fa8100a1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:19.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:19.032+0000 7fa815cee640 10 monclient(hunting): get_auth_request con 0x7fa8100a1a50 auth_method 0 2024-09-17T18:21:19.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:19.032+0000 7fa815cee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:19.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:19.032+0000 7fa815cee640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:19.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:19.032+0000 7fa815cee640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:21:19.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:19.032+0000 7fa815cee640 1 -- 172.21.15.121:0/125984064 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8100a1a50 msgr2=0x7fa8100a1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:21:19.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:19.032+0000 7fa815cee640 1 -- 172.21.15.121:0/125984064 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8100a1a50 msgr2=0x7fa8100a1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:21:19.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:19.032+0000 7fa815cee640 1 --2- 172.21.15.121:0/125984064 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8100a1a50 0x7fa8100a1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:21:19.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:19.032+0000 7fa815cee640 1 --2- 172.21.15.121:0/125984064 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8100a1a50 0x7fa8100a1e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T18:21:19.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:19.032+0000 7fa8154ed640 1 -- 172.21.15.121:0/125984064 >> v2:172.21.15.121:3300/0 conn(0x7fa8100a2370 msgr2=0x7fa810141040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:21:19.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:19.032+0000 7fa8154ed640 1 --2- 172.21.15.121:0/125984064 >> v2:172.21.15.121:3300/0 conn(0x7fa8100a2370 0x7fa810141040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T18:21:20.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:20.628+0000 7fa8077fe640 10 monclient(hunting): tick 2024-09-17T18:21:20.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:20.628+0000 7fa8077fe640 10 monclient(hunting): _check_auth_tickets 2024-09-17T18:21:20.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:20.628+0000 7fa8077fe640 1 monclient(hunting): continuing hunt 2024-09-17T18:21:20.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:20.628+0000 7fa8077fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T18:21:20.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:20.628+0000 7fa8077fe640 1 -- 172.21.15.121:0/125984064 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8100a1a50 msgr2=0x7fa8100a1e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:21:20.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:20.628+0000 7fa8077fe640 1 --2- 172.21.15.121:0/125984064 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8100a1a50 0x7fa8100a1e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:20.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:20.628+0000 7fa8077fe640 1 -- 172.21.15.121:0/125984064 >> v2:172.21.15.121:3300/0 conn(0x7fa8100a2370 msgr2=0x7fa810141040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:21:20.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:20.628+0000 7fa8077fe640 1 --2- 172.21.15.121:0/125984064 >> v2:172.21.15.121:3300/0 conn(0x7fa8100a2370 0x7fa810141040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:20.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:20.628+0000 7fa8077fe640 1 -- 172.21.15.121:0/125984064 >> v1:172.21.15.173:6789/0 conn(0x7fa8100a2750 legacy=0x7fa810143590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:20.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:20.628+0000 7fa8077fe640 10 monclient: _add_conns ranks=[0,4] 2024-09-17T18:21:20.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:20.628+0000 7fa8077fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa7f4003740 addr v1:172.21.15.121:6789/0 2024-09-17T18:21:20.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:20.628+0000 7fa8077fe640 1 --2- 172.21.15.121:0/125984064 >> v2:172.21.15.173:3300/0 conn(0x7fa7f4007490 0x7fa7f4009950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:20.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:20.628+0000 7fa8077fe640 10 monclient(hunting): picked mon.noname-c con 0x7fa7f4007490 addr v2:172.21.15.173:3300/0 2024-09-17T18:21:20.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:20.628+0000 7fa8077fe640 1 -- 172.21.15.121:0/125984064 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa81009e6f0 con 0x7fa7f4003740 2024-09-17T18:21:20.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:20.628+0000 7fa8077fe640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:20.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:20.628+0000 7fa8077fe640 1 -- 172.21.15.121:0/125984064 --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fa8100a3000 con 0x7fa7f4007490 2024-09-17T18:21:20.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:20.628+0000 7fa8077fe640 10 monclient(hunting): _renew_subs 2024-09-17T18:21:20.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:20.628+0000 7fa815cee640 1 -- 172.21.15.121:0/125984064 >> v1:172.21.15.121:6789/0 conn(0x7fa7f4003740 legacy=0x7fa7f4005bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.121:6789/0 2024-09-17T18:21:20.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:20.628+0000 7fa8154ed640 1 --2- 172.21.15.121:0/125984064 >> v2:172.21.15.173:3300/0 conn(0x7fa7f4007490 0x7fa7f4009950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:20.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:20.628+0000 7fa8154ed640 10 monclient(hunting): get_auth_request con 0x7fa7f4007490 auth_method 0 2024-09-17T18:21:20.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:20.628+0000 7fa8154ed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:20.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:20.628+0000 7fa8154ed640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:20.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:20.628+0000 7fa8154ed640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:20.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:20.628+0000 7fa8154ed640 1 -- 172.21.15.121:0/125984064 >> v2:172.21.15.173:3300/0 conn(0x7fa7f4007490 msgr2=0x7fa7f4009950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T18:21:20.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:20.628+0000 7fa8154ed640 1 -- 172.21.15.121:0/125984064 >> v2:172.21.15.173:3300/0 conn(0x7fa7f4007490 msgr2=0x7fa7f4009950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:21:20.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:20.628+0000 7fa8154ed640 1 --2- 172.21.15.121:0/125984064 >> v2:172.21.15.173:3300/0 conn(0x7fa7f4007490 0x7fa7f4009950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:21:20.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:20.628+0000 7fa8154ed640 1 --2- 172.21.15.121:0/125984064 >> v2:172.21.15.173:3300/0 conn(0x7fa7f4007490 0x7fa7f4009950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:21:20.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:20.829+0000 7fa815cee640 1 -- 172.21.15.121:0/125984064 >> v1:172.21.15.121:6789/0 conn(0x7fa7f4003740 legacy=0x7fa7f4005bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.121:6789/0 2024-09-17T18:21:20.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:20.829+0000 7fa8154ed640 1 --2- 172.21.15.121:0/125984064 >> v2:172.21.15.173:3300/0 conn(0x7fa7f4007490 0x7fa7f4009950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:20.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:20.829+0000 7fa8154ed640 10 monclient(hunting): get_auth_request con 0x7fa7f4007490 auth_method 0 2024-09-17T18:21:20.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:20.829+0000 7fa8154ed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:20.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:20.829+0000 7fa8154ed640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:20.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:20.829+0000 7fa8154ed640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:21:20.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:20.829+0000 7fa8154ed640 1 -- 172.21.15.121:0/125984064 >> v2:172.21.15.173:3300/0 conn(0x7fa7f4007490 msgr2=0x7fa7f4009950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:21:20.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:20.829+0000 7fa8154ed640 1 -- 172.21.15.121:0/125984064 >> v2:172.21.15.173:3300/0 conn(0x7fa7f4007490 msgr2=0x7fa7f4009950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:21:20.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:20.829+0000 7fa8154ed640 1 --2- 172.21.15.121:0/125984064 >> v2:172.21.15.173:3300/0 conn(0x7fa7f4007490 0x7fa7f4009950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:21:20.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:20.829+0000 7fa8154ed640 1 --2- 172.21.15.121:0/125984064 >> v2:172.21.15.173:3300/0 conn(0x7fa7f4007490 0x7fa7f4009950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T18:21:21.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:21.230+0000 7fa815cee640 1 -- 172.21.15.121:0/125984064 >> v1:172.21.15.121:6789/0 conn(0x7fa7f4003740 legacy=0x7fa7f4005bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.121:6789/0 2024-09-17T18:21:21.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:21.231+0000 7fa8154ed640 1 --2- 172.21.15.121:0/125984064 >> v2:172.21.15.173:3300/0 conn(0x7fa7f4007490 0x7fa7f4009950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:21.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:21.231+0000 7fa8154ed640 10 monclient(hunting): get_auth_request con 0x7fa7f4007490 auth_method 0 2024-09-17T18:21:21.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:21.231+0000 7fa8154ed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:21.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:21.231+0000 7fa8154ed640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:21.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:21.231+0000 7fa8154ed640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:21:21.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:21.231+0000 7fa8154ed640 1 -- 172.21.15.121:0/125984064 >> v2:172.21.15.173:3300/0 conn(0x7fa7f4007490 msgr2=0x7fa7f4009950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:21:21.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:21.231+0000 7fa8154ed640 1 -- 172.21.15.121:0/125984064 >> v2:172.21.15.173:3300/0 conn(0x7fa7f4007490 msgr2=0x7fa7f4009950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:21:21.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:21.231+0000 7fa8154ed640 1 --2- 172.21.15.121:0/125984064 >> v2:172.21.15.173:3300/0 conn(0x7fa7f4007490 0x7fa7f4009950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:21:21.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:21.231+0000 7fa8154ed640 1 --2- 172.21.15.121:0/125984064 >> v2:172.21.15.173:3300/0 conn(0x7fa7f4007490 0x7fa7f4009950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T18:21:22.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:22.031+0000 7fa815cee640 1 -- 172.21.15.121:0/125984064 >> v1:172.21.15.121:6789/0 conn(0x7fa7f4003740 legacy=0x7fa7f4005bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.121:6789/0 2024-09-17T18:21:22.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:22.032+0000 7fa8154ed640 1 --2- 172.21.15.121:0/125984064 >> v2:172.21.15.173:3300/0 conn(0x7fa7f4007490 0x7fa7f4009950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:22.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:22.032+0000 7fa8154ed640 10 monclient(hunting): get_auth_request con 0x7fa7f4007490 auth_method 0 2024-09-17T18:21:22.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:22.032+0000 7fa8154ed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:22.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:22.032+0000 7fa8154ed640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:22.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:22.032+0000 7fa8154ed640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:21:22.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:22.032+0000 7fa8154ed640 1 -- 172.21.15.121:0/125984064 >> v2:172.21.15.173:3300/0 conn(0x7fa7f4007490 msgr2=0x7fa7f4009950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:21:22.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:22.032+0000 7fa8154ed640 1 -- 172.21.15.121:0/125984064 >> v2:172.21.15.173:3300/0 conn(0x7fa7f4007490 msgr2=0x7fa7f4009950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:21:22.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:22.032+0000 7fa8154ed640 1 --2- 172.21.15.121:0/125984064 >> v2:172.21.15.173:3300/0 conn(0x7fa7f4007490 0x7fa7f4009950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:21:22.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:22.032+0000 7fa8154ed640 1 --2- 172.21.15.121:0/125984064 >> v2:172.21.15.173:3300/0 conn(0x7fa7f4007490 0x7fa7f4009950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T18:21:22.619 INFO:tasks.workunit.client.0.smithi173.stdout: snapshots: Running main() from gmock_main.cc 2024-09-17T18:21:22.619 INFO:tasks.workunit.client.0.smithi173.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-09-17T18:21:22.619 INFO:tasks.workunit.client.0.smithi173.stdout: snapshots: [----------] Global test environment set-up. 2024-09-17T18:21:22.619 INFO:tasks.workunit.client.0.smithi173.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-09-17T18:21:22.619 INFO:tasks.workunit.client.0.smithi173.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-09-17T18:21:22.619 INFO:tasks.workunit.client.0.smithi173.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (5092 ms) 2024-09-17T18:21:22.619 INFO:tasks.workunit.client.0.smithi173.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-09-17T18:21:22.619 INFO:tasks.workunit.client.0.smithi173.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (5019 ms) 2024-09-17T18:21:22.619 INFO:tasks.workunit.client.0.smithi173.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-09-17T18:21:22.619 INFO:tasks.workunit.client.0.smithi173.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (3875 ms) 2024-09-17T18:21:22.619 INFO:tasks.workunit.client.0.smithi173.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-09-17T18:21:22.620 INFO:tasks.workunit.client.0.smithi173.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (5030 ms) 2024-09-17T18:21:22.620 INFO:tasks.workunit.client.0.smithi173.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-09-17T18:21:22.620 INFO:tasks.workunit.client.0.smithi173.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (6953 ms) 2024-09-17T18:21:22.620 INFO:tasks.workunit.client.0.smithi173.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (25969 ms total) 2024-09-17T18:21:22.620 INFO:tasks.workunit.client.0.smithi173.stdout: snapshots: 2024-09-17T18:21:22.620 INFO:tasks.workunit.client.0.smithi173.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-09-17T18:21:22.620 INFO:tasks.workunit.client.0.smithi173.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-09-17T18:21:22.620 INFO:tasks.workunit.client.0.smithi173.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (5000 ms) 2024-09-17T18:21:22.620 INFO:tasks.workunit.client.0.smithi173.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-09-17T18:21:22.620 INFO:tasks.workunit.client.0.smithi173.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (6003 ms) 2024-09-17T18:21:22.620 INFO:tasks.workunit.client.0.smithi173.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-09-17T18:21:22.620 INFO:tasks.workunit.client.0.smithi173.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (8011 ms) 2024-09-17T18:21:22.620 INFO:tasks.workunit.client.0.smithi173.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-09-17T18:21:22.620 INFO:tasks.workunit.client.0.smithi173.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (5996 ms) 2024-09-17T18:21:22.621 INFO:tasks.workunit.client.0.smithi173.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-09-17T18:21:22.621 INFO:tasks.workunit.client.0.smithi173.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (4227 ms) 2024-09-17T18:21:22.621 INFO:tasks.workunit.client.0.smithi173.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-09-17T18:21:22.621 INFO:tasks.workunit.client.0.smithi173.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi173-36361-11. 2024-09-17T18:21:22.621 INFO:tasks.workunit.client.0.smithi173.stdout: snapshots: Waiting for snaps to purge. 2024-09-17T18:21:22.621 INFO:tasks.workunit.client.0.smithi173.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (26329 ms) 2024-09-17T18:21:22.621 INFO:tasks.workunit.client.0.smithi173.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (55566 ms total) 2024-09-17T18:21:22.621 INFO:tasks.workunit.client.0.smithi173.stdout: snapshots: 2024-09-17T18:21:22.621 INFO:tasks.workunit.client.0.smithi173.stdout: snapshots: [----------] Global test environment tear-down 2024-09-17T18:21:22.621 INFO:tasks.workunit.client.0.smithi173.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (81535 ms total) 2024-09-17T18:21:22.621 INFO:tasks.workunit.client.0.smithi173.stdout: snapshots: [ PASSED ] 11 tests. 2024-09-17T18:21:23.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.628+0000 7fa8077fe640 10 monclient(hunting): tick 2024-09-17T18:21:23.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.628+0000 7fa8077fe640 10 monclient(hunting): _check_auth_tickets 2024-09-17T18:21:23.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.628+0000 7fa8077fe640 1 monclient(hunting): continuing hunt 2024-09-17T18:21:23.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.628+0000 7fa8077fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T18:21:23.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.628+0000 7fa8077fe640 1 -- 172.21.15.121:0/125984064 >> v2:172.21.15.173:3300/0 conn(0x7fa7f4007490 msgr2=0x7fa7f4009950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:21:23.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.628+0000 7fa8077fe640 1 --2- 172.21.15.121:0/125984064 >> v2:172.21.15.173:3300/0 conn(0x7fa7f4007490 0x7fa7f4009950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:23.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.628+0000 7fa8077fe640 1 -- 172.21.15.121:0/125984064 >> v1:172.21.15.121:6789/0 conn(0x7fa7f4003740 legacy=0x7fa7f4005bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:21:23.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.628+0000 7fa8164ef640 1 -- 172.21.15.121:0/125984064 reap_dead start 2024-09-17T18:21:23.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.628+0000 7fa8077fe640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-17T18:21:23.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.628+0000 7fa8077fe640 1 --2- 172.21.15.121:0/125984064 >> v2:172.21.15.173:3300/0 conn(0x7fa7f4009fc0 0x7fa7f4009950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:23.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.628+0000 7fa8077fe640 10 monclient(hunting): picked mon.noname-c con 0x7fa7f4009fc0 addr v2:172.21.15.173:3300/0 2024-09-17T18:21:23.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.628+0000 7fa8154ed640 1 -- 172.21.15.121:0/125984064 >> v1:172.21.15.121:6789/0 conn(0x7fa7f400a470 legacy=0x7fa7f400d500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.121:6789/0 2024-09-17T18:21:23.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.628+0000 7fa8077fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa7f400a470 addr v1:172.21.15.121:6789/0 2024-09-17T18:21:23.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.628+0000 7fa8077fe640 1 --2- 172.21.15.121:0/125984064 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7f400aa10 0x7fa7f4005bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:23.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.628+0000 7fa815cee640 1 --2- 172.21.15.121:0/125984064 >> v2:172.21.15.173:3300/0 conn(0x7fa7f4009fc0 0x7fa7f4009950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:23.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.628+0000 7fa8077fe640 10 monclient(hunting): picked mon.noname-b con 0x7fa7f400aa10 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:21:23.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.628+0000 7fa8077fe640 1 -- 172.21.15.121:0/125984064 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7f400b0c0 con 0x7fa7f400a470 2024-09-17T18:21:23.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.629+0000 7fa8077fe640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:23.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.629+0000 7fa8077fe640 1 -- 172.21.15.121:0/125984064 --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fa8100a3000 con 0x7fa7f4009fc0 2024-09-17T18:21:23.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.629+0000 7fa8077fe640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:23.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.629+0000 7fa8077fe640 1 -- 172.21.15.121:0/125984064 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa810077730 con 0x7fa7f400aa10 2024-09-17T18:21:23.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.629+0000 7fa8077fe640 10 monclient(hunting): _renew_subs 2024-09-17T18:21:23.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.629+0000 7fa815cee640 10 monclient(hunting): get_auth_request con 0x7fa7f4009fc0 auth_method 0 2024-09-17T18:21:23.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.629+0000 7fa815cee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:23.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.629+0000 7fa815cee640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:23.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.629+0000 7fa815cee640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:23.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.629+0000 7fa8164ef640 1 --2- 172.21.15.121:0/125984064 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7f400aa10 0x7fa7f4005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:23.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.629+0000 7fa815cee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:23.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.629+0000 7fa815cee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:23.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.629+0000 7fa815cee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:23.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.629+0000 7fa8164ef640 10 monclient(hunting): get_auth_request con 0x7fa7f400aa10 auth_method 0 2024-09-17T18:21:23.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.629+0000 7fa8164ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:23.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.629+0000 7fa8164ef640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:23.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.629+0000 7fa8164ef640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:23.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.629+0000 7fa8164ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:23.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.629+0000 7fa8164ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:23.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.629+0000 7fa8164ef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:23.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.629+0000 7fa8164ef640 10 monclient(hunting): handle_auth_done global_id 34163 payload 274 2024-09-17T18:21:23.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.629+0000 7fa8164ef640 1 -- 172.21.15.121:0/125984064 >> v2:172.21.15.173:3300/0 conn(0x7fa7f4009fc0 msgr2=0x7fa7f4009950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:21:23.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.629+0000 7fa8164ef640 1 --2- 172.21.15.121:0/125984064 >> v2:172.21.15.173:3300/0 conn(0x7fa7f4009fc0 0x7fa7f4009950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:23.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.629+0000 7fa8164ef640 1 -- 172.21.15.121:0/125984064 >> v1:172.21.15.121:6789/0 conn(0x7fa7f400a470 legacy=0x7fa7f400d500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:21:23.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.629+0000 7fa8164ef640 10 monclient: _finish_hunting 0 2024-09-17T18:21:23.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.629+0000 7fa8164ef640 1 monclient: found mon.noname-b 2024-09-17T18:21:23.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.629+0000 7fa8164ef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:21:23.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.629+0000 7fa8164ef640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:21:23.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.629+0000 7fa8164ef640 1 -- 172.21.15.121:0/125984064 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa81009e6f0 con 0x7fa7f400aa10 2024-09-17T18:21:23.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.629+0000 7fa815cee640 1 --2- 172.21.15.121:0/125984064 >> v2:172.21.15.173:3300/0 conn(0x7fa7f4009fc0 0x7fa7f4009950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:21:23.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.632+0000 7fa8164ef640 1 -- 172.21.15.121:0/125984064 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7f400aa10 msgr2=0x7fa7f4005bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:21:23.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.632+0000 7fa8164ef640 1 --2- 172.21.15.121:0/125984064 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7f400aa10 0x7fa7f4005bc0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa8100a2b30 tx=0x7fa80802a2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9bd5794949020bb server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:21:23.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.633+0000 7fa814cec640 1 -- 172.21.15.121:0/125984064 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa808005b40 con 0x7fa7f400aa10 2024-09-17T18:21:23.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.633+0000 7fa814cec640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:23.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.633+0000 7fa814cec640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:21:23.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.633+0000 7fa814cec640 10 monclient: dump: 2024-09-17T18:21:23.638 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:23.638 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:23.638 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:23.638 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:23.638 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:23.638 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:23.638 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:23.638 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:23.638 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:23.638 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:23.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.633+0000 7fa814cec640 10 monclient: _finish_auth 0 2024-09-17T18:21:23.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.633+0000 7fa814cec640 10 monclient: _check_auth_tickets 2024-09-17T18:21:23.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.633+0000 7fa814cec640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:21:23.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.633+0000 7fa814cec640 1 -- 172.21.15.121:0/125984064 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa80802e070 con 0x7fa7f400aa10 2024-09-17T18:21:23.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.633+0000 7fa814cec640 10 monclient: handle_config config(0 keys) 2024-09-17T18:21:23.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.633+0000 7fa814cec640 1 -- 172.21.15.121:0/125984064 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa808007070 con 0x7fa7f400aa10 2024-09-17T18:21:23.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.633+0000 7fa814cec640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:23.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.633+0000 7fa814cec640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:21:23.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.633+0000 7fa814cec640 10 monclient: dump: 2024-09-17T18:21:23.639 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:23.639 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:23.639 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:23.639 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:23.639 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:23.640 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:23.640 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:23.640 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:23.640 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:23.640 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:23.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.633+0000 7fa817f52640 5 monclient: authenticate success, global_id 34163 2024-09-17T18:21:23.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.633+0000 7fa817f52640 10 monclient: get_monmap_and_config success 2024-09-17T18:21:23.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.633+0000 7fa817f52640 4 set_mon_vals no callback set 2024-09-17T18:21:23.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.634+0000 7fa817f52640 10 monclient: shutdown 2024-09-17T18:21:23.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.634+0000 7fa817f52640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:21:23.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.634+0000 7fa817f52640 1 -- 172.21.15.121:0/125984064 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7f400aa10 msgr2=0x7fa7f4005bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:23.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.634+0000 7fa817f52640 1 --2- 172.21.15.121:0/125984064 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7f400aa10 0x7fa7f4005bc0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa8100a2b30 tx=0x7fa80802a2d0 comp rx=0 tx=0).stop 2024-09-17T18:21:23.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.634+0000 7fa817f52640 1 -- 172.21.15.121:0/125984064 shutdown_connections 2024-09-17T18:21:23.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.634+0000 7fa817f52640 1 --2- 172.21.15.121:0/125984064 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7f400aa10 0x7fa7f4005bc0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:23.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.634+0000 7fa817f52640 1 --2- 172.21.15.121:0/125984064 >> v2:172.21.15.173:3300/0 conn(0x7fa7f4009fc0 0x7fa7f4009950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:23.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.634+0000 7fa817f52640 1 -- 172.21.15.121:0/125984064 >> 172.21.15.121:0/125984064 conn(0x7fa8100a3b50 msgr2=0x7fa8100a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:21:23.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.634+0000 7fa817f52640 1 -- 172.21.15.121:0/125984064 shutdown_connections 2024-09-17T18:21:23.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.634+0000 7fa817f52640 1 -- 172.21.15.121:0/125984064 wait complete. 2024-09-17T18:21:23.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.635+0000 7fa817f52640 10 monclient: build_initial_monmap 2024-09-17T18:21:23.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.635+0000 7fa817f52640 10 monclient: monmap: 2024-09-17T18:21:23.645 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:21:23.645 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:21:23.645 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:21:23.645 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:21:23.645 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:21:23.645 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:23.645 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:21:23.645 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:21:23.645 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:21:23.646 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:23.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.635+0000 7fa817f52640 1 librados: starting msgr at 2024-09-17T18:21:23.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.635+0000 7fa817f52640 1 librados: starting objecter 2024-09-17T18:21:23.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.635+0000 7fa817f52640 1 Processor -- start 2024-09-17T18:21:23.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.636+0000 7fa817f52640 1 -- start start 2024-09-17T18:21:23.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.636+0000 7fa817f52640 1 librados: setting wanted keys 2024-09-17T18:21:23.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.636+0000 7fa817f52640 1 librados: calling monclient init 2024-09-17T18:21:23.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.636+0000 7fa817f52640 10 monclient: init 2024-09-17T18:21:23.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.636+0000 7fa817f52640 10 monclient: _reopen_session rank -1 2024-09-17T18:21:23.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.636+0000 7fa817f52640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T18:21:23.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.636+0000 7fa817f52640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7f4009fc0 0x7fa8101d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:23.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.636+0000 7fa817f52640 10 monclient(hunting): picked mon.noname-c con 0x7fa7f4009fc0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:21:23.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.636+0000 7fa817f52640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7f400a470 0x7fa8101c13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:23.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.636+0000 7fa817f52640 10 monclient(hunting): picked mon.noname-b con 0x7fa7f400a470 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:21:23.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.636+0000 7fa817f52640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7f400aa10 0x7fa8101c1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:23.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.636+0000 7fa8164ef640 1 -- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7f400aa10 msgr2=0x7fa8101c1900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:21:23.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.636+0000 7fa8164ef640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7f400aa10 0x7fa8101c1900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:21:23.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.637+0000 7fa817f52640 10 monclient(hunting): picked mon.noname-a con 0x7fa7f400aa10 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:21:23.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.637+0000 7fa817f52640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:23.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.637+0000 7fa817f52640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa81009de60 con 0x7fa7f400aa10 2024-09-17T18:21:23.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.637+0000 7fa817f52640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:23.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.637+0000 7fa817f52640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa81009dce0 con 0x7fa7f400a470 2024-09-17T18:21:23.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.637+0000 7fa817f52640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:23.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.637+0000 7fa817f52640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa8100749b0 con 0x7fa7f4009fc0 2024-09-17T18:21:23.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.637+0000 7fa817f52640 10 monclient(hunting): _renew_subs 2024-09-17T18:21:23.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.637+0000 7fa815cee640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7f4009fc0 0x7fa8101d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:23.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.637+0000 7fa817f52640 10 monclient(hunting): authenticate will time out at 887.354044s 2024-09-17T18:21:23.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.637+0000 7fa815cee640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7f4009fc0 0x7fa8101d1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:54310/0 (socket says 172.21.15.121:54310) 2024-09-17T18:21:23.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.637+0000 7fa815cee640 1 -- 172.21.15.121:0/3108032685 learned_addr learned my addr 172.21.15.121:0/3108032685 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:21:23.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.637+0000 7fa815cee640 10 monclient(hunting): get_auth_request con 0x7fa7f4009fc0 auth_method 0 2024-09-17T18:21:23.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.637+0000 7fa815cee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:23.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.637+0000 7fa815cee640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:23.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.637+0000 7fa815cee640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:23.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.637+0000 7fa8154ed640 1 --2- 172.21.15.121:0/3108032685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7f400a470 0x7fa8101c13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:23.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.637+0000 7fa8154ed640 10 monclient(hunting): get_auth_request con 0x7fa7f400a470 auth_method 0 2024-09-17T18:21:23.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.637+0000 7fa8154ed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:23.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.637+0000 7fa8154ed640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:23.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.637+0000 7fa8154ed640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:23.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.637+0000 7fa815cee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:23.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.637+0000 7fa815cee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:23.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.637+0000 7fa815cee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:23.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.637+0000 7fa8154ed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:23.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.637+0000 7fa8154ed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:23.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.637+0000 7fa8154ed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:23.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.637+0000 7fa815cee640 10 monclient(hunting): handle_auth_done global_id 34166 payload 824 2024-09-17T18:21:23.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.637+0000 7fa815cee640 1 -- 172.21.15.121:0/3108032685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7f400a470 msgr2=0x7fa8101c13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:23.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.637+0000 7fa815cee640 1 --2- 172.21.15.121:0/3108032685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7f400a470 0x7fa8101c13c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:23.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.637+0000 7fa815cee640 1 -- 172.21.15.121:0/3108032685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7f400aa10 msgr2=0x7fa8101c1900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:21:23.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.637+0000 7fa815cee640 1 --2- 172.21.15.121:0/3108032685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7f400aa10 0x7fa8101c1900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:23.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.637+0000 7fa815cee640 10 monclient: _finish_hunting 0 2024-09-17T18:21:23.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.637+0000 7fa815cee640 1 monclient: found mon.noname-c 2024-09-17T18:21:23.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.637+0000 7fa815cee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:21:23.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.637+0000 7fa815cee640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:21:23.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.637+0000 7fa815cee640 1 -- 172.21.15.121:0/3108032685 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8101aee90 con 0x7fa7f4009fc0 2024-09-17T18:21:23.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.640+0000 7fa815cee640 1 -- 172.21.15.121:0/3108032685 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7f4009fc0 msgr2=0x7fa8101d1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:21:23.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.640+0000 7fa815cee640 1 --2- 172.21.15.121:0/3108032685 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7f4009fc0 0x7fa8101d1da0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa80c009100 tx=0x7fa80c017ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:21:23.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.640+0000 7fa8067fc640 10 client.?.objecter ms_handle_connect 0x7fa7f4009fc0 2024-09-17T18:21:23.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.640+0000 7fa8067fc640 10 client.?.objecter resend_mon_ops 2024-09-17T18:21:23.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.641+0000 7fa8067fc640 1 -- 172.21.15.121:0/3108032685 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80c00f040 con 0x7fa7f4009fc0 2024-09-17T18:21:23.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.641+0000 7fa8067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:23.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.641+0000 7fa8067fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:21:23.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.641+0000 7fa8067fc640 10 monclient: dump: 2024-09-17T18:21:23.651 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:23.651 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:23.651 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:23.651 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:23.651 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:23.651 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:23.651 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:23.651 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:23.651 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:23.651 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:23.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.641+0000 7fa8067fc640 10 monclient: _finish_auth 0 2024-09-17T18:21:23.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.641+0000 7fa8067fc640 10 monclient: _check_auth_tickets 2024-09-17T18:21:23.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.641+0000 7fa8067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:21:23.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.641+0000 7fa8067fc640 1 -- 172.21.15.121:0/3108032685 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa80c015cf0 con 0x7fa7f4009fc0 2024-09-17T18:21:23.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.641+0000 7fa8067fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:21:23.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.641+0000 7fa807fff640 4 set_mon_vals no callback set 2024-09-17T18:21:23.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.641+0000 7fa8067fc640 1 -- 172.21.15.121:0/3108032685 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80c018670 con 0x7fa7f4009fc0 2024-09-17T18:21:23.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.641+0000 7fa8067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:23.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.641+0000 7fa8067fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:21:23.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.641+0000 7fa8067fc640 10 monclient: dump: 2024-09-17T18:21:23.652 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:23.652 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:23.652 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:23.652 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:23.652 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:23.653 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:23.653 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:23.653 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:23.653 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:23.653 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:23.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.641+0000 7fa817f52640 5 monclient: authenticate success, global_id 34166 2024-09-17T18:21:23.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.641+0000 7fa817f52640 10 monclient: _renew_subs 2024-09-17T18:21:23.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.641+0000 7fa817f52640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:21:23.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.641+0000 7fa817f52640 1 -- 172.21.15.121:0/3108032685 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8101ada30 con 0x7fa7f4009fc0 2024-09-17T18:21:23.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.641+0000 7fa817f52640 10 client.34166.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:21:23.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.641+0000 7fa817f52640 10 monclient: _renew_subs 2024-09-17T18:21:23.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.641+0000 7fa817f52640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:21:23.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.641+0000 7fa817f52640 1 -- 172.21.15.121:0/3108032685 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8101d22e0 con 0x7fa7f4009fc0 2024-09-17T18:21:23.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.641+0000 7fa817f52640 1 librados: init done 2024-09-17T18:21:23.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.643+0000 7fa817f52640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:21:23.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.643+0000 7fa817f52640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:21:23.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.643+0000 7fa817f52640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:21:23.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.643+0000 7fa817f52640 1 -- 172.21.15.121:0/3108032685 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8100a1e30 con 0x7fa7f4009fc0 2024-09-17T18:21:23.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.644+0000 7fa8067fc640 1 -- 172.21.15.121:0/3108032685 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fa80c0188d0 con 0x7fa7f4009fc0 2024-09-17T18:21:23.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.644+0000 7fa8067fc640 1 --2- 172.21.15.121:0/3108032685 >> v2:172.21.15.173:6812/2423786037 conn(0x7fa7f4041df0 0x7fa7f40442b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:23.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.644+0000 7fa8067fc640 1 -- 172.21.15.121:0/3108032685 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(81..81 src has 1..81) ==== 6871+0+0 (secure 0 0 0) 0x7fa80c055130 con 0x7fa7f4009fc0 2024-09-17T18:21:23.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.644+0000 7fa8067fc640 10 client.34166.objecter ms_dispatch 0x7fa810000b90 osd_map(81..81 src has 1..81) 2024-09-17T18:21:23.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.644+0000 7fa8067fc640 3 client.34166.objecter handle_osd_map got epochs [81,81] > 0 2024-09-17T18:21:23.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.644+0000 7fa8067fc640 3 client.34166.objecter handle_osd_map decoding full epoch 81 2024-09-17T18:21:23.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.644+0000 7fa8067fc640 20 client.34166.objecter dump_active .. 0 homeless 2024-09-17T18:21:23.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.644+0000 7fa8154ed640 1 --2- 172.21.15.121:0/3108032685 >> v2:172.21.15.173:6812/2423786037 conn(0x7fa7f4041df0 0x7fa7f40442b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:23.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.644+0000 7fa8154ed640 10 monclient: get_auth_request con 0x7fa7f4041df0 auth_method 0 2024-09-17T18:21:23.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.648+0000 7fa8154ed640 1 --2- 172.21.15.121:0/3108032685 >> v2:172.21.15.173:6812/2423786037 conn(0x7fa7f4041df0 0x7fa7f40442b0 secure :-1 s=READY pgs=3869 cs=0 l=1 rev1=1 crypto rx=0x7fa8101d1850 tx=0x7fa800005040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:21:23.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.648+0000 7fa8067fc640 10 client.34166.objecter ms_handle_connect 0x7fa7f4041df0 2024-09-17T18:21:23.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.652+0000 7fa8067fc640 1 -- 172.21.15.121:0/3108032685 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fa80c018da0 con 0x7fa7f4009fc0 2024-09-17T18:21:23.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.652+0000 7fa8067fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:21:23.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.652+0000 7fa8067fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:21:23.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.911+0000 7fa817f52640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:21:23.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.911+0000 7fa817f52640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:21:23.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.911+0000 7fa817f52640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:21:23.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.911+0000 7fa817f52640 1 -- 172.21.15.121:0/3108032685 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa81006ce60 con 0x7fa7f4009fc0 2024-09-17T18:21:23.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.911+0000 7fa8067fc640 1 -- 172.21.15.121:0/3108032685 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7fa80c01fda0 con 0x7fa7f4009fc0 2024-09-17T18:21:23.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.911+0000 7fa8067fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:21:23.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.911+0000 7fa8067fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:21:23.915 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":12,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:21:23.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.913+0000 7fa817f52640 10 librados: watch_flush enter 2024-09-17T18:21:23.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.913+0000 7fa817f52640 10 librados: watch_flush exit 2024-09-17T18:21:23.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.913+0000 7fa817f52640 20 client.34166.objecter shutdown clearing up homeless session... 2024-09-17T18:21:23.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.913+0000 7fa817f52640 10 client.34166.objecter successfully canceled tick 2024-09-17T18:21:23.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.913+0000 7fa817f52640 1 -- 172.21.15.121:0/3108032685 >> v2:172.21.15.173:6812/2423786037 conn(0x7fa7f4041df0 msgr2=0x7fa7f40442b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:23.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.913+0000 7fa817f52640 1 --2- 172.21.15.121:0/3108032685 >> v2:172.21.15.173:6812/2423786037 conn(0x7fa7f4041df0 0x7fa7f40442b0 secure :-1 s=READY pgs=3869 cs=0 l=1 rev1=1 crypto rx=0x7fa8101d1850 tx=0x7fa800005040 comp rx=0 tx=0).stop 2024-09-17T18:21:23.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.913+0000 7fa817f52640 10 monclient: shutdown 2024-09-17T18:21:23.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.913+0000 7fa817f52640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:21:23.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.913+0000 7fa817f52640 1 -- 172.21.15.121:0/3108032685 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7f4009fc0 msgr2=0x7fa8101d1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:23.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.913+0000 7fa817f52640 1 --2- 172.21.15.121:0/3108032685 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7f4009fc0 0x7fa8101d1da0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa80c009100 tx=0x7fa80c017ee0 comp rx=0 tx=0).stop 2024-09-17T18:21:23.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.913+0000 7fa817f52640 1 -- 172.21.15.121:0/3108032685 shutdown_connections 2024-09-17T18:21:23.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.913+0000 7fa817f52640 1 --2- 172.21.15.121:0/3108032685 >> v2:172.21.15.173:6812/2423786037 conn(0x7fa7f4041df0 0x7fa7f40442b0 unknown :-1 s=CLOSED pgs=3869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:23.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.913+0000 7fa817f52640 1 --2- 172.21.15.121:0/3108032685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7f400a470 0x7fa8101c13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:23.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.913+0000 7fa817f52640 1 --2- 172.21.15.121:0/3108032685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7f400aa10 0x7fa8101c1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:23.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.913+0000 7fa817f52640 1 --2- 172.21.15.121:0/3108032685 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7f4009fc0 0x7fa8101d1da0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:23.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.913+0000 7fa817f52640 1 -- 172.21.15.121:0/3108032685 >> 172.21.15.121:0/3108032685 conn(0x7fa8100a3b50 msgr2=0x7fa8101c3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:21:23.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.913+0000 7fa817f52640 1 -- 172.21.15.121:0/3108032685 shutdown_connections 2024-09-17T18:21:23.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.913+0000 7fa817f52640 1 -- 172.21.15.121:0/3108032685 wait complete. 2024-09-17T18:21:23.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:23.913+0000 7fa817f52640 1 librados: shutdown 2024-09-17T18:21:23.927 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-17T18:21:23.927 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-17T18:21:23.928 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T18:21:24.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.022+0000 7f672f14b640 10 monclient: get_monmap_and_config 2024-09-17T18:21:24.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.022+0000 7f672f14b640 10 monclient: build_initial_monmap 2024-09-17T18:21:24.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.022+0000 7f672f14b640 10 monclient: monmap: 2024-09-17T18:21:24.025 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:21:24.025 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:24.025 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:21:24.023950+0000 2024-09-17T18:21:24.025 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:21:24.023950+0000 2024-09-17T18:21:24.025 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:21:24.025 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:24.026 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:21:24.026 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:21:24.026 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:21:24.026 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:21:24.026 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:21:24.026 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:24.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.023+0000 7f672f14b640 1 Processor -- start 2024-09-17T18:21:24.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.023+0000 7f672f14b640 1 -- start start 2024-09-17T18:21:24.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.023+0000 7f672f14b640 10 monclient: init 2024-09-17T18:21:24.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.023+0000 7f672f14b640 10 monclient: _reopen_session rank -1 2024-09-17T18:21:24.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.023+0000 7f672f14b640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-17T18:21:24.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.023+0000 7f672f14b640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f67280a0a00 0x7f67280a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:24.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.023+0000 7f672f14b640 10 monclient(hunting): picked mon.noname-a con 0x7f67280a0a00 addr v2:172.21.15.121:3300/0 2024-09-17T18:21:24.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.023+0000 7f672f14b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f67280a1320 addr v1:172.21.15.121:6789/0 2024-09-17T18:21:24.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.023+0000 7f672f14b640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f6728135090 0x7f672813ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:24.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.023+0000 7f672f14b640 10 monclient(hunting): picked mon.noname-c con 0x7f6728135090 addr v2:172.21.15.173:3300/0 2024-09-17T18:21:24.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.023+0000 7f672f14b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67280a3820 con 0x7f67280a1320 2024-09-17T18:21:24.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.023+0000 7f672f14b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:24.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.023+0000 7f672f14b640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f67280a2ea0 con 0x7f67280a0a00 2024-09-17T18:21:24.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.023+0000 7f672f14b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:24.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.023+0000 7f672f14b640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f672809e800 con 0x7f6728135090 2024-09-17T18:21:24.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.023+0000 7f672f14b640 10 monclient(hunting): _renew_subs 2024-09-17T18:21:24.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.023+0000 7f672f14b640 10 monclient(hunting): authenticate will time out at 887.740606s 2024-09-17T18:21:24.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.023+0000 7f672cee7640 1 -- >> v2:172.21.15.121:3300/0 conn(0x7f67280a0a00 msgr2=0x7f67280a0de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:21:24.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.023+0000 7f6727fff640 1 -- >> v1:172.21.15.121:6789/0 conn(0x7f67280a1320 legacy=0x7f6728134980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.121:6789/0 2024-09-17T18:21:24.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.023+0000 7f672cee7640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f67280a0a00 0x7f67280a0de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:21:24.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.023+0000 7f672d6e8640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f6728135090 0x7f672813ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:24.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.023+0000 7f672d6e8640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f6728135090 0x7f672813ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.121:38330/0 (socket says 172.21.15.121:38330) 2024-09-17T18:21:24.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.023+0000 7f672d6e8640 1 -- 172.21.15.121:0/1454955473 learned_addr learned my addr 172.21.15.121:0/1454955473 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:21:24.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.023+0000 7f672d6e8640 10 monclient(hunting): get_auth_request con 0x7f6728135090 auth_method 0 2024-09-17T18:21:24.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.023+0000 7f672d6e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:24.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.023+0000 7f672d6e8640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:24.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.023+0000 7f672d6e8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:24.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.024+0000 7f672d6e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:24.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.024+0000 7f672d6e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:24.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.024+0000 7f672d6e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:24.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.024+0000 7f672d6e8640 10 monclient(hunting): handle_auth_done global_id 24196 payload 274 2024-09-17T18:21:24.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.024+0000 7f672d6e8640 1 -- 172.21.15.121:0/1454955473 >> v2:172.21.15.121:3300/0 conn(0x7f67280a0a00 msgr2=0x7f67280a0de0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:21:24.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.024+0000 7f672d6e8640 1 --2- 172.21.15.121:0/1454955473 >> v2:172.21.15.121:3300/0 conn(0x7f67280a0a00 0x7f67280a0de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:24.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.024+0000 7f672d6e8640 1 -- 172.21.15.121:0/1454955473 >> v1:172.21.15.121:6789/0 conn(0x7f67280a1320 legacy=0x7f6728134980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:21:24.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.024+0000 7f672d6e8640 10 monclient: _finish_hunting 0 2024-09-17T18:21:24.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.024+0000 7f672d6e8640 1 monclient: found mon.noname-c 2024-09-17T18:21:24.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.024+0000 7f672d6e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:21:24.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.024+0000 7f672d6e8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.173:3300/0 2024-09-17T18:21:24.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.024+0000 7f672d6e8640 1 -- 172.21.15.121:0/1454955473 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67280a2c30 con 0x7f6728135090 2024-09-17T18:21:24.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.027+0000 7f672d6e8640 1 -- 172.21.15.121:0/1454955473 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6728135090 msgr2=0x7f672813ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:21:24.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.027+0000 7f672d6e8640 1 --2- 172.21.15.121:0/1454955473 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6728135090 0x7f672813ccc0 secure :-1 s=READY pgs=7126 cs=0 l=1 rev1=1 crypto rx=0x7f6728077930 tx=0x7f672002a590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9afec7c6ade2ff0e server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:21:24.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.027+0000 7f67277fe640 1 -- 172.21.15.121:0/1454955473 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6720036070 con 0x7f6728135090 2024-09-17T18:21:24.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.027+0000 7f67277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:24.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.027+0000 7f67277fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:21:24.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.027+0000 7f67277fe640 10 monclient: dump: 2024-09-17T18:21:24.034 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:24.034 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:24.034 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:24.034 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:24.034 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:24.034 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:24.034 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:24.034 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:24.034 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:24.034 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:24.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.027+0000 7f67277fe640 10 monclient: _finish_auth 0 2024-09-17T18:21:24.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.027+0000 7f67277fe640 10 monclient: _check_auth_tickets 2024-09-17T18:21:24.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.027+0000 7f67277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:21:24.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.027+0000 7f67277fe640 1 -- 172.21.15.121:0/1454955473 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6720007800 con 0x7f6728135090 2024-09-17T18:21:24.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.027+0000 7f672f14b640 5 monclient: authenticate success, global_id 24196 2024-09-17T18:21:24.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.027+0000 7f672f14b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:21:24.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.027+0000 7f67277fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:21:24.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.027+0000 7f672f14b640 10 monclient: get_monmap_and_config success 2024-09-17T18:21:24.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.027+0000 7f672f14b640 4 set_mon_vals no callback set 2024-09-17T18:21:24.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.027+0000 7f67277fe640 1 -- 172.21.15.121:0/1454955473 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6720007bd0 con 0x7f6728135090 2024-09-17T18:21:24.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.027+0000 7f67277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:24.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.027+0000 7f67277fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:21:24.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.027+0000 7f67277fe640 10 monclient: dump: 2024-09-17T18:21:24.035 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:24.035 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:24.035 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:24.036 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:24.036 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:24.036 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:24.036 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:24.036 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:24.036 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:24.036 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:24.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.027+0000 7f672f14b640 10 monclient: shutdown 2024-09-17T18:21:24.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.027+0000 7f672f14b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:21:24.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.027+0000 7f672f14b640 1 -- 172.21.15.121:0/1454955473 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6728135090 msgr2=0x7f672813ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:24.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.027+0000 7f672f14b640 1 --2- 172.21.15.121:0/1454955473 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6728135090 0x7f672813ccc0 secure :-1 s=READY pgs=7126 cs=0 l=1 rev1=1 crypto rx=0x7f6728077930 tx=0x7f672002a590 comp rx=0 tx=0).stop 2024-09-17T18:21:24.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.027+0000 7f672f14b640 1 -- 172.21.15.121:0/1454955473 shutdown_connections 2024-09-17T18:21:24.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.027+0000 7f672f14b640 1 --2- 172.21.15.121:0/1454955473 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6728135090 0x7f672813ccc0 unknown :-1 s=CLOSED pgs=7126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:24.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.027+0000 7f672f14b640 1 --2- 172.21.15.121:0/1454955473 >> v2:172.21.15.121:3300/0 conn(0x7f67280a0a00 0x7f67280a0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:24.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.027+0000 7f672f14b640 1 -- 172.21.15.121:0/1454955473 >> 172.21.15.121:0/1454955473 conn(0x7f67280a7830 msgr2=0x7f67280a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:21:24.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.027+0000 7f672f14b640 1 -- 172.21.15.121:0/1454955473 shutdown_connections 2024-09-17T18:21:24.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.027+0000 7f672f14b640 1 -- 172.21.15.121:0/1454955473 wait complete. 2024-09-17T18:21:24.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.028+0000 7f672f14b640 10 monclient: build_initial_monmap 2024-09-17T18:21:24.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.028+0000 7f672f14b640 10 monclient: monmap: 2024-09-17T18:21:24.043 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:21:24.044 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:21:24.044 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:21:24.044 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:21:24.044 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:21:24.044 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:24.044 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:21:24.044 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:21:24.044 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:21:24.044 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:24.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.028+0000 7f672f14b640 1 librados: starting msgr at 2024-09-17T18:21:24.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.028+0000 7f672f14b640 1 librados: starting objecter 2024-09-17T18:21:24.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.028+0000 7f672f14b640 1 Processor -- start 2024-09-17T18:21:24.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.028+0000 7f672f14b640 1 -- start start 2024-09-17T18:21:24.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.028+0000 7f672f14b640 1 librados: setting wanted keys 2024-09-17T18:21:24.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.028+0000 7f672f14b640 1 librados: calling monclient init 2024-09-17T18:21:24.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.028+0000 7f672f14b640 10 monclient: init 2024-09-17T18:21:24.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672f14b640 10 monclient: _reopen_session rank -1 2024-09-17T18:21:24.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672f14b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T18:21:24.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672f14b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67280a0a00 0x7f6728134410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:24.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672f14b640 10 monclient(hunting): picked mon.noname-b con 0x7f67280a0a00 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:21:24.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672f14b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67280a1320 0x7f6728134950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:24.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672f14b640 10 monclient(hunting): picked mon.noname-a con 0x7f67280a1320 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:21:24.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672f14b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6728135090 0x7f67281393f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:24.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672f14b640 10 monclient(hunting): picked mon.noname-c con 0x7f6728135090 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:21:24.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672f14b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:24.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672f14b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f672809de60 con 0x7f67280a1320 2024-09-17T18:21:24.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672f14b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:24.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672f14b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f672809dce0 con 0x7f67280a0a00 2024-09-17T18:21:24.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672f14b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:24.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672f14b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f67280a3120 con 0x7f6728135090 2024-09-17T18:21:24.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672f14b640 10 monclient(hunting): _renew_subs 2024-09-17T18:21:24.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672f14b640 10 monclient(hunting): authenticate will time out at 887.746373s 2024-09-17T18:21:24.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f6727fff640 1 -- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67280a1320 msgr2=0x7f6728134950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:21:24.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f6727fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67280a1320 0x7f6728134950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:21:24.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672d6e8640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6728135090 0x7f67281393f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:24.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672d6e8640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6728135090 0x7f67281393f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:54314/0 (socket says 172.21.15.121:54314) 2024-09-17T18:21:24.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672d6e8640 1 -- 172.21.15.121:0/1246085110 learned_addr learned my addr 172.21.15.121:0/1246085110 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:21:24.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672d6e8640 10 monclient(hunting): get_auth_request con 0x7f6728135090 auth_method 0 2024-09-17T18:21:24.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672d6e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:24.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672d6e8640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:24.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672d6e8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:24.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672d6e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:24.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672d6e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:24.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672d6e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:24.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672cee7640 1 --2- 172.21.15.121:0/1246085110 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67280a0a00 0x7f6728134410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:24.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672cee7640 10 monclient(hunting): get_auth_request con 0x7f67280a0a00 auth_method 0 2024-09-17T18:21:24.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672cee7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:24.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672cee7640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:24.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672cee7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:24.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672cee7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:24.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672cee7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:24.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672cee7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:24.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672d6e8640 10 monclient(hunting): handle_auth_done global_id 34172 payload 824 2024-09-17T18:21:24.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672d6e8640 1 -- 172.21.15.121:0/1246085110 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67280a0a00 msgr2=0x7f6728134410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:24.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672d6e8640 1 --2- 172.21.15.121:0/1246085110 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67280a0a00 0x7f6728134410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:24.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672d6e8640 1 -- 172.21.15.121:0/1246085110 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67280a1320 msgr2=0x7f6728134950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:21:24.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672d6e8640 1 --2- 172.21.15.121:0/1246085110 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67280a1320 0x7f6728134950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:24.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672d6e8640 10 monclient: _finish_hunting 0 2024-09-17T18:21:24.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672d6e8640 1 monclient: found mon.noname-c 2024-09-17T18:21:24.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672d6e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:21:24.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672d6e8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:21:24.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.029+0000 7f672d6e8640 1 -- 172.21.15.121:0/1246085110 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f672813be50 con 0x7f6728135090 2024-09-17T18:21:24.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.031+0000 7f672d6e8640 1 -- 172.21.15.121:0/1246085110 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6728135090 msgr2=0x7f67281393f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:21:24.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.031+0000 7f672d6e8640 1 --2- 172.21.15.121:0/1246085110 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6728135090 0x7f67281393f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6720034040 tx=0x7f6720004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:21:24.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.031+0000 7f67257fa640 10 client.?.objecter ms_handle_connect 0x7f6728135090 2024-09-17T18:21:24.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.031+0000 7f67257fa640 10 client.?.objecter resend_mon_ops 2024-09-17T18:21:24.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.032+0000 7f67257fa640 1 -- 172.21.15.121:0/1246085110 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6720041030 con 0x7f6728135090 2024-09-17T18:21:24.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.032+0000 7f67257fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:24.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.032+0000 7f67257fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:21:24.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.032+0000 7f67257fa640 10 monclient: dump: 2024-09-17T18:21:24.049 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:24.050 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:24.050 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:24.050 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:24.050 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:24.050 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:24.050 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:24.050 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:24.050 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:24.050 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:24.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.032+0000 7f67257fa640 10 monclient: _finish_auth 0 2024-09-17T18:21:24.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.032+0000 7f67257fa640 10 monclient: _check_auth_tickets 2024-09-17T18:21:24.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.032+0000 7f67257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:21:24.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.032+0000 7f672f14b640 5 monclient: authenticate success, global_id 34172 2024-09-17T18:21:24.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.032+0000 7f672f14b640 10 monclient: _renew_subs 2024-09-17T18:21:24.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.032+0000 7f672f14b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:21:24.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.032+0000 7f672f14b640 1 -- 172.21.15.121:0/1246085110 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67280738d0 con 0x7f6728135090 2024-09-17T18:21:24.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.032+0000 7f67257fa640 1 -- 172.21.15.121:0/1246085110 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6720007600 con 0x7f6728135090 2024-09-17T18:21:24.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.032+0000 7f67257fa640 10 monclient: handle_config config(0 keys) 2024-09-17T18:21:24.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.032+0000 7f67257fa640 1 -- 172.21.15.121:0/1246085110 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6720036040 con 0x7f6728135090 2024-09-17T18:21:24.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.032+0000 7f67257fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:24.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.032+0000 7f67257fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:21:24.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.032+0000 7f67257fa640 10 monclient: dump: 2024-09-17T18:21:24.051 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:24.052 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:24.052 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:24.052 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:24.052 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:24.052 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:24.052 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:24.052 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:24.052 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:24.052 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:24.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.032+0000 7f6726ffd640 4 set_mon_vals no callback set 2024-09-17T18:21:24.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.032+0000 7f672f14b640 10 client.34172.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:21:24.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.032+0000 7f672f14b640 10 monclient: _renew_subs 2024-09-17T18:21:24.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.032+0000 7f672f14b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:21:24.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.032+0000 7f672f14b640 1 -- 172.21.15.121:0/1246085110 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f672813aeb0 con 0x7f6728135090 2024-09-17T18:21:24.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.032+0000 7f672f14b640 1 librados: init done 2024-09-17T18:21:24.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.032+0000 7f672f14b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:21:24.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.032+0000 7f672f14b640 1 --2- 172.21.15.121:0/1246085110 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f672813ba10 0x7f6728139930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:24.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.032+0000 7f672f14b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:24.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.032+0000 7f672f14b640 1 -- 172.21.15.121:0/1246085110 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6728135580 con 0x7f672813ba10 2024-09-17T18:21:24.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.033+0000 7f67257fa640 1 -- 172.21.15.121:0/1246085110 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f6720044070 con 0x7f6728135090 2024-09-17T18:21:24.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.033+0000 7f672cee7640 1 --2- 172.21.15.121:0/1246085110 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f672813ba10 0x7f6728139930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:24.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.033+0000 7f672cee7640 10 monclient: get_auth_request con 0x7f672813ba10 auth_method 0 2024-09-17T18:21:24.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.033+0000 7f672cee7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:24.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.033+0000 7f672cee7640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:24.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.033+0000 7f672cee7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:24.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.033+0000 7f672cee7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:24.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.033+0000 7f672cee7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:24.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.033+0000 7f672cee7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:24.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.033+0000 7f672cee7640 10 monclient(hunting): handle_auth_done global_id 34175 payload 824 2024-09-17T18:21:24.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.033+0000 7f67257fa640 1 --2- 172.21.15.121:0/1246085110 >> v2:172.21.15.173:6812/2423786037 conn(0x7f670c040150 0x7f670c042610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:24.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.033+0000 7f672cee7640 1 -- 172.21.15.121:0/1246085110 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f67280a2030 con 0x7f672813ba10 2024-09-17T18:21:24.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.033+0000 7f67257fa640 1 -- 172.21.15.121:0/1246085110 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(81..81 src has 1..81) ==== 6871+0+0 (secure 0 0 0) 0x7f6720074210 con 0x7f6728135090 2024-09-17T18:21:24.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.033+0000 7f67257fa640 10 client.34172.objecter ms_dispatch 0x7f6728000b90 osd_map(81..81 src has 1..81) 2024-09-17T18:21:24.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.033+0000 7f67257fa640 3 client.34172.objecter handle_osd_map got epochs [81,81] > 0 2024-09-17T18:21:24.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.033+0000 7f67257fa640 3 client.34172.objecter handle_osd_map decoding full epoch 81 2024-09-17T18:21:24.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.033+0000 7f6727fff640 1 --2- 172.21.15.121:0/1246085110 >> v2:172.21.15.173:6812/2423786037 conn(0x7f670c040150 0x7f670c042610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:24.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.033+0000 7f67257fa640 20 client.34172.objecter dump_active .. 0 homeless 2024-09-17T18:21:24.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.033+0000 7f6727fff640 10 monclient: get_auth_request con 0x7f670c040150 auth_method 0 2024-09-17T18:21:24.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.035+0000 7f67257fa640 10 client.34172.objecter ms_handle_connect 0x7f672813ba10 2024-09-17T18:21:24.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.035+0000 7f672cee7640 1 -- 172.21.15.121:0/1246085110 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f672813ba10 msgr2=0x7f6728139930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:21:24.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.035+0000 7f67257fa640 10 client.34172.objecter resend_mon_ops 2024-09-17T18:21:24.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.035+0000 7f672cee7640 1 --2- 172.21.15.121:0/1246085110 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f672813ba10 0x7f6728139930 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f671c009610 tx=0x7f671c009ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:21:24.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.035+0000 7f67257fa640 1 -- 172.21.15.121:0/1246085110 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f671c016070 con 0x7f672813ba10 2024-09-17T18:21:24.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.035+0000 7f67257fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:24.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.035+0000 7f67257fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:21:24.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.035+0000 7f67257fa640 10 monclient: dump: 2024-09-17T18:21:24.055 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:24.055 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:24.055 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:24.055 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:24.055 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:24.055 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:24.055 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:24.055 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:24.055 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:24.056 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:24.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.035+0000 7f67257fa640 1 -- 172.21.15.121:0/1246085110 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f671c005a40 con 0x7f672813ba10 2024-09-17T18:21:24.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.035+0000 7f67257fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:21:24.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.035+0000 7f67257fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:21:24.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.035+0000 7f67257fa640 1 -- 172.21.15.121:0/1246085110 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f672813ba10 msgr2=0x7f6728139930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:24.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.035+0000 7f67257fa640 1 --2- 172.21.15.121:0/1246085110 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f672813ba10 0x7f6728139930 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f671c009610 tx=0x7f671c009ad0 comp rx=0 tx=0).stop 2024-09-17T18:21:24.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.035+0000 7f67257fa640 1 -- 172.21.15.121:0/1246085110 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f672813ba10 msgr2=0x7f6728139930 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:21:24.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.035+0000 7f67257fa640 1 --2- 172.21.15.121:0/1246085110 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f672813ba10 0x7f6728139930 secure :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f671c009610 tx=0x7f671c009ad0 comp rx=0 tx=0).stop 2024-09-17T18:21:24.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.035+0000 7f6727fff640 1 --2- 172.21.15.121:0/1246085110 >> v2:172.21.15.173:6812/2423786037 conn(0x7f670c040150 0x7f670c042610 secure :-1 s=READY pgs=3871 cs=0 l=1 rev1=1 crypto rx=0x7f672813c710 tx=0x7f6714006d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:21:24.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.036+0000 7f67257fa640 10 client.34172.objecter ms_handle_connect 0x7f670c040150 2024-09-17T18:21:24.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.041+0000 7f672f14b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:21:24.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.042+0000 7f672f14b640 1 --2- 172.21.15.121:0/1246085110 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f67281cb7f0 0x7f6728139e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:24.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.042+0000 7f672f14b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:24.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.042+0000 7f672f14b640 1 -- 172.21.15.121:0/1246085110 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f672806cd40 con 0x7f67281cb7f0 2024-09-17T18:21:24.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.042+0000 7f672cee7640 1 --2- 172.21.15.121:0/1246085110 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f67281cb7f0 0x7f6728139e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:24.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.042+0000 7f672cee7640 10 monclient: get_auth_request con 0x7f67281cb7f0 auth_method 0 2024-09-17T18:21:24.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.042+0000 7f672cee7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:24.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.042+0000 7f672cee7640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:24.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.042+0000 7f672cee7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:24.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.042+0000 7f672cee7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:24.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.042+0000 7f672cee7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:24.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.042+0000 7f672cee7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:24.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.042+0000 7f672cee7640 10 monclient(hunting): handle_auth_done global_id 34178 payload 824 2024-09-17T18:21:24.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.042+0000 7f672cee7640 1 -- 172.21.15.121:0/1246085110 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f672811dcf0 con 0x7f67281cb7f0 2024-09-17T18:21:24.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.044+0000 7f672cee7640 1 -- 172.21.15.121:0/1246085110 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f67281cb7f0 msgr2=0x7f6728139e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:21:24.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.044+0000 7f67257fa640 10 client.34172.objecter ms_handle_connect 0x7f67281cb7f0 2024-09-17T18:21:24.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.044+0000 7f67257fa640 10 client.34172.objecter resend_mon_ops 2024-09-17T18:21:24.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.044+0000 7f672cee7640 1 --2- 172.21.15.121:0/1246085110 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f67281cb7f0 0x7f6728139e70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f671c017000 tx=0x7f671c0059d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:21:24.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.044+0000 7f67257fa640 1 -- 172.21.15.121:0/1246085110 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f671c016050 con 0x7f67281cb7f0 2024-09-17T18:21:24.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.044+0000 7f67257fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:24.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.044+0000 7f67257fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:21:24.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.044+0000 7f67257fa640 10 monclient: dump: 2024-09-17T18:21:24.059 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:24.060 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:24.060 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:24.060 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:24.060 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:24.060 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:24.060 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:24.060 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:24.061 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:24.061 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:24.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.044+0000 7f67257fa640 1 -- 172.21.15.121:0/1246085110 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4708 (secure 0 0 0) 0x7f671c00e3f0 con 0x7f67281cb7f0 2024-09-17T18:21:24.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.044+0000 7f67257fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:21:24.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.044+0000 7f67257fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:21:24.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.044+0000 7f67257fa640 1 -- 172.21.15.121:0/1246085110 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f67281cb7f0 msgr2=0x7f6728139e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:24.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.044+0000 7f67257fa640 1 --2- 172.21.15.121:0/1246085110 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f67281cb7f0 0x7f6728139e70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f671c017000 tx=0x7f671c0059d0 comp rx=0 tx=0).stop 2024-09-17T18:21:24.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.044+0000 7f67257fa640 1 -- 172.21.15.121:0/1246085110 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f67281cb7f0 msgr2=0x7f6728139e70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:21:24.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.044+0000 7f67257fa640 1 --2- 172.21.15.121:0/1246085110 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f67281cb7f0 0x7f6728139e70 secure :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f671c017000 tx=0x7f671c0059d0 comp rx=0 tx=0).stop 2024-09-17T18:21:24.062 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:21:24.062 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:21:24.062 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:21:24.062 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:21:24.062 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 12, 2024-09-17T18:21:24.062 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 29321, 2024-09-17T18:21:24.062 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:21:24.062 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:21:24.062 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:21:24.062 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:21:24.062 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 1, 2024-09-17T18:21:24.062 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:21:24.062 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:21:24.063 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:21:24.063 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:21:24.063 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:21:24.063 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:21:24.063 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:21:24.063 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:21:24.063 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:21:24.063 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:21:24.063 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:21:24.063 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:21:24.063 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:21:24.063 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:21:24.063 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:21:24.063 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:21:24.064 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:21:24.064 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:21:24.064 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:21:24.064 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:21:24.064 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:21:24.064 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:21:24.064 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:21:24.064 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:21:24.064 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:21:24.064 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:21:24.064 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:21:24.064 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:21:24.064 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:21:24.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:24.065 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:21:24.065 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:21:24.065 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:21:24.065 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:21:24.065 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:21:24.065 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:21:24.065 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:21:24.065 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:21:24.065 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:21:24.065 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:21:24.065 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:21:24.066 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:21:24.066 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:21:24.066 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:21:24.066 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:21:24.066 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:21:24.066 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:21:24.066 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:21:24.066 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:21:24.066 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:21:24.067 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:21:24.067 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:21:24.067 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:21:24.067 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:21:24.067 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:21:24.067 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:21:24.067 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:21:24.068 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:21:24.068 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:21:24.068 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:21:24.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:24.068 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:21:24.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:24.068 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:21:24.068 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:21:24.068 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:21:24.068 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:21:24.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:24.068 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:21:24.068 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:21:24.068 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:21:24.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:24.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:24.069 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:21:24.069 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:21:24.069 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:21:24.069 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:24.069 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:21:24.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:24.069 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:21:24.069 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:21:24.069 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:21:24.069 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:21:24.069 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:21:24.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:24.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:24.070 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:21:24.070 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:21:24.070 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:21:24.070 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:21:24.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:24.070 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:21:24.070 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:21:24.070 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:21:24.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:24.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:24.070 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:21:24.070 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:21:24.071 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:21:24.071 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:24.071 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:21:24.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:24.071 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:21:24.071 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:21:24.071 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:21:24.071 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:21:24.071 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:21:24.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:24.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:24.071 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:21:24.071 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:21:24.071 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:21:24.072 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:21:24.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:24.072 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:21:24.072 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:21:24.072 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:21:24.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:24.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:24.072 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:21:24.072 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:21:24.072 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:21:24.072 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:24.072 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:21:24.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:24.072 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:21:24.072 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:21:24.073 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:21:24.073 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:21:24.073 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:21:24.073 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:24.073 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:21:24.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:24.073 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:21:24.073 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:21:24.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:24.073 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:21:24.073 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:21:24.073 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:21:24.073 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:24.073 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:21:24.073 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:21:24.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:24.074 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:21:24.074 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:21:24.074 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:21:24.074 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:24.074 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:21:24.074 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:21:24.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:24.074 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:21:24.074 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:21:24.074 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-17T18:21:24.074 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:24.074 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:21:24.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:24.075 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:21:24.075 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:21:24.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.045+0000 7f672f14b640 10 librados: watch_flush enter 2024-09-17T18:21:24.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.045+0000 7f672f14b640 10 librados: watch_flush exit 2024-09-17T18:21:24.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.045+0000 7f672f14b640 20 client.34172.objecter shutdown clearing up homeless session... 2024-09-17T18:21:24.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.045+0000 7f672f14b640 10 client.34172.objecter successfully canceled tick 2024-09-17T18:21:24.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.045+0000 7f672f14b640 1 -- 172.21.15.121:0/1246085110 >> v2:172.21.15.173:6812/2423786037 conn(0x7f670c040150 msgr2=0x7f670c042610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:24.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.045+0000 7f672f14b640 1 --2- 172.21.15.121:0/1246085110 >> v2:172.21.15.173:6812/2423786037 conn(0x7f670c040150 0x7f670c042610 secure :-1 s=READY pgs=3871 cs=0 l=1 rev1=1 crypto rx=0x7f672813c710 tx=0x7f6714006d30 comp rx=0 tx=0).stop 2024-09-17T18:21:24.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.045+0000 7f672f14b640 10 monclient: shutdown 2024-09-17T18:21:24.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.045+0000 7f672f14b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:21:24.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.045+0000 7f672f14b640 1 -- 172.21.15.121:0/1246085110 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6728135090 msgr2=0x7f67281393f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:24.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.045+0000 7f672d6e8640 1 -- 172.21.15.121:0/1246085110 reap_dead start 2024-09-17T18:21:24.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.045+0000 7f672f14b640 1 --2- 172.21.15.121:0/1246085110 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6728135090 0x7f67281393f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6720034040 tx=0x7f6720004a40 comp rx=0 tx=0).stop 2024-09-17T18:21:24.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.045+0000 7f672f14b640 1 -- 172.21.15.121:0/1246085110 shutdown_connections 2024-09-17T18:21:24.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.045+0000 7f672f14b640 1 --2- 172.21.15.121:0/1246085110 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6728135090 0x7f67281393f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:24.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.045+0000 7f672f14b640 1 -- 172.21.15.121:0/1246085110 >> 172.21.15.121:0/1246085110 conn(0x7f67280a7830 msgr2=0x7f67280731b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:21:24.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.045+0000 7f672f14b640 1 -- 172.21.15.121:0/1246085110 shutdown_connections 2024-09-17T18:21:24.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.045+0000 7f672f14b640 1 -- 172.21.15.121:0/1246085110 wait complete. 2024-09-17T18:21:24.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.045+0000 7f672f14b640 1 librados: shutdown 2024-09-17T18:21:24.079 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T18:21:24.079 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-17T18:21:24.079 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T18:21:24.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.173+0000 7f946a394640 10 monclient: get_monmap_and_config 2024-09-17T18:21:24.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.173+0000 7f946a394640 10 monclient: build_initial_monmap 2024-09-17T18:21:24.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.173+0000 7f946a394640 10 monclient: monmap: 2024-09-17T18:21:24.177 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:21:24.177 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:24.177 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:21:24.175478+0000 2024-09-17T18:21:24.177 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:21:24.175478+0000 2024-09-17T18:21:24.177 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:21:24.177 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:24.177 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:21:24.177 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:21:24.177 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:21:24.177 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:21:24.177 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:21:24.177 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:24.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.174+0000 7f946a394640 1 Processor -- start 2024-09-17T18:21:24.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.174+0000 7f946a394640 1 -- start start 2024-09-17T18:21:24.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.174+0000 7f946a394640 10 monclient: init 2024-09-17T18:21:24.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.175+0000 7f946a394640 10 monclient: _reopen_session rank -1 2024-09-17T18:21:24.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.175+0000 7f946a394640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-17T18:21:24.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.175+0000 7f946a394640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f94640700f0 0x7f9464134e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:24.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.175+0000 7f946a394640 10 monclient(hunting): picked mon.noname-a con 0x7f94640700f0 addr v2:172.21.15.121:3300/0 2024-09-17T18:21:24.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.175+0000 7f946a394640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f946409e810 0x7f9464138360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:24.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.175+0000 7f946a394640 10 monclient(hunting): picked mon.noname-c con 0x7f946409e810 addr v2:172.21.15.173:3300/0 2024-09-17T18:21:24.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.175+0000 7f946a394640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f946409ec60 addr v1:172.21.15.173:6789/0 2024-09-17T18:21:24.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.175+0000 7f946a394640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f946409ca60 con 0x7f946409ec60 2024-09-17T18:21:24.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.175+0000 7f946a394640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:24.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.175+0000 7f946a394640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f94640704d0 con 0x7f94640700f0 2024-09-17T18:21:24.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.175+0000 7f946a394640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:24.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.175+0000 7f946a394640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f946409cd20 con 0x7f946409e810 2024-09-17T18:21:24.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.175+0000 7f946a394640 10 monclient(hunting): _renew_subs 2024-09-17T18:21:24.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.175+0000 7f946a394640 10 monclient(hunting): authenticate will time out at 887.892211s 2024-09-17T18:21:24.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.175+0000 7f9463fff640 1 -- >> v2:172.21.15.121:3300/0 conn(0x7f94640700f0 msgr2=0x7f9464134e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:21:24.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.175+0000 7f9463fff640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f94640700f0 0x7f9464134e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:21:24.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.175+0000 7f94637fe640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f946409e810 0x7f9464138360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:24.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.175+0000 7f94637fe640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f946409e810 0x7f9464138360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.121:38348/0 (socket says 172.21.15.121:38348) 2024-09-17T18:21:24.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.175+0000 7f94637fe640 1 -- 172.21.15.121:0/2212220991 learned_addr learned my addr 172.21.15.121:0/2212220991 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:21:24.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.175+0000 7f94637fe640 10 monclient(hunting): get_auth_request con 0x7f946409e810 auth_method 0 2024-09-17T18:21:24.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.175+0000 7f94637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:24.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.175+0000 7f94637fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:24.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.175+0000 7f94637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:24.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.175+0000 7f9468931640 1 --1- 172.21.15.121:0/2212220991 >> v1:172.21.15.173:6789/0 conn(0x7f946409ec60 0x7f94641388a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.121:45554/0 (socket says 172.21.15.121:45554) 2024-09-17T18:21:24.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.175+0000 7f94637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:24.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.175+0000 7f94637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:24.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.175+0000 7f94637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:24.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.175+0000 7f94637fe640 10 monclient(hunting): handle_auth_done global_id 24202 payload 274 2024-09-17T18:21:24.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.175+0000 7f94637fe640 1 -- 172.21.15.121:0/2212220991 >> v2:172.21.15.121:3300/0 conn(0x7f94640700f0 msgr2=0x7f9464134e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:21:24.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.175+0000 7f94637fe640 1 --2- 172.21.15.121:0/2212220991 >> v2:172.21.15.121:3300/0 conn(0x7f94640700f0 0x7f9464134e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:24.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.175+0000 7f94637fe640 1 -- 172.21.15.121:0/2212220991 >> v1:172.21.15.173:6789/0 conn(0x7f946409ec60 legacy=0x7f94641388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:21:24.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.175+0000 7f94637fe640 10 monclient: _finish_hunting 0 2024-09-17T18:21:24.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.175+0000 7f94637fe640 1 monclient: found mon.noname-c 2024-09-17T18:21:24.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.175+0000 7f94637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:21:24.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.175+0000 7f94637fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.173:3300/0 2024-09-17T18:21:24.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.175+0000 7f94637fe640 1 -- 172.21.15.121:0/2212220991 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f946409d000 con 0x7f946409e810 2024-09-17T18:21:24.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.178+0000 7f94637fe640 1 -- 172.21.15.121:0/2212220991 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f946409e810 msgr2=0x7f9464138360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:21:24.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.178+0000 7f94637fe640 1 --2- 172.21.15.121:0/2212220991 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f946409e810 0x7f9464138360 secure :-1 s=READY pgs=7128 cs=0 l=1 rev1=1 crypto rx=0x7f94640a89f0 tx=0x7f945802a0d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6e37192c0dfd648 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:21:24.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.179+0000 7f9462ffd640 1 -- 172.21.15.121:0/2212220991 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9458037070 con 0x7f946409e810 2024-09-17T18:21:24.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.179+0000 7f9462ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:24.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.179+0000 7f9462ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:21:24.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.179+0000 7f9462ffd640 10 monclient: dump: 2024-09-17T18:21:24.195 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:24.195 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:24.195 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:24.195 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:24.195 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:24.195 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:24.195 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:24.196 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:24.196 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:24.196 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:24.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.179+0000 7f9462ffd640 10 monclient: _finish_auth 0 2024-09-17T18:21:24.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.179+0000 7f9462ffd640 10 monclient: _check_auth_tickets 2024-09-17T18:21:24.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.179+0000 7f9462ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:21:24.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.179+0000 7f946a394640 5 monclient: authenticate success, global_id 24202 2024-09-17T18:21:24.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.179+0000 7f9462ffd640 1 -- 172.21.15.121:0/2212220991 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9458032640 con 0x7f946409e810 2024-09-17T18:21:24.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.179+0000 7f946a394640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:21:24.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.179+0000 7f9462ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:21:24.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.179+0000 7f946a394640 10 monclient: get_monmap_and_config success 2024-09-17T18:21:24.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.179+0000 7f9462ffd640 1 -- 172.21.15.121:0/2212220991 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9458032a10 con 0x7f946409e810 2024-09-17T18:21:24.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.179+0000 7f9462ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:24.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.179+0000 7f946a394640 4 set_mon_vals no callback set 2024-09-17T18:21:24.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.179+0000 7f9462ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:21:24.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.179+0000 7f9462ffd640 10 monclient: dump: 2024-09-17T18:21:24.197 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:24.197 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:24.197 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:24.197 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:24.197 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:24.197 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:24.197 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:24.198 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:24.198 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:24.198 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:24.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.179+0000 7f946a394640 10 monclient: shutdown 2024-09-17T18:21:24.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.179+0000 7f946a394640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:21:24.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.179+0000 7f946a394640 1 -- 172.21.15.121:0/2212220991 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f946409e810 msgr2=0x7f9464138360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:24.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.179+0000 7f946a394640 1 --2- 172.21.15.121:0/2212220991 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f946409e810 0x7f9464138360 secure :-1 s=READY pgs=7128 cs=0 l=1 rev1=1 crypto rx=0x7f94640a89f0 tx=0x7f945802a0d0 comp rx=0 tx=0).stop 2024-09-17T18:21:24.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.179+0000 7f946a394640 1 -- 172.21.15.121:0/2212220991 shutdown_connections 2024-09-17T18:21:24.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.179+0000 7f946a394640 1 --2- 172.21.15.121:0/2212220991 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f946409e810 0x7f9464138360 unknown :-1 s=CLOSED pgs=7128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:24.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.179+0000 7f946a394640 1 --2- 172.21.15.121:0/2212220991 >> v2:172.21.15.121:3300/0 conn(0x7f94640700f0 0x7f9464134e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:24.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.179+0000 7f946a394640 1 -- 172.21.15.121:0/2212220991 >> 172.21.15.121:0/2212220991 conn(0x7f94640ab830 msgr2=0x7f94640a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:21:24.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.179+0000 7f946a394640 1 -- 172.21.15.121:0/2212220991 shutdown_connections 2024-09-17T18:21:24.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.179+0000 7f946a394640 1 -- 172.21.15.121:0/2212220991 wait complete. 2024-09-17T18:21:24.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.180+0000 7f946a394640 10 monclient: build_initial_monmap 2024-09-17T18:21:24.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.180+0000 7f946a394640 10 monclient: monmap: 2024-09-17T18:21:24.199 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:21:24.200 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:21:24.200 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:21:24.200 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:21:24.200 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:21:24.200 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:24.200 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:21:24.201 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:21:24.201 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:21:24.201 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:24.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.180+0000 7f946a394640 1 librados: starting msgr at 2024-09-17T18:21:24.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.180+0000 7f946a394640 1 librados: starting objecter 2024-09-17T18:21:24.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.180+0000 7f946a394640 1 Processor -- start 2024-09-17T18:21:24.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.180+0000 7f946a394640 1 -- start start 2024-09-17T18:21:24.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.180+0000 7f946a394640 1 librados: setting wanted keys 2024-09-17T18:21:24.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.180+0000 7f946a394640 1 librados: calling monclient init 2024-09-17T18:21:24.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.180+0000 7f946a394640 10 monclient: init 2024-09-17T18:21:24.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f946a394640 10 monclient: _reopen_session rank -1 2024-09-17T18:21:24.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f946a394640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T18:21:24.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f946a394640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94640700f0 0x7f9464131bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:24.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f946a394640 10 monclient(hunting): picked mon.noname-a con 0x7f94640700f0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:21:24.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f946a394640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f946409e810 0x7f9464132100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:24.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f946a394640 10 monclient(hunting): picked mon.noname-c con 0x7f946409e810 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:21:24.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f946a394640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f946409ec60 0x7f94640a3320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:24.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f946a394640 10 monclient(hunting): picked mon.noname-b con 0x7f946409ec60 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:21:24.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f946a394640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:24.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f946a394640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9464135510 con 0x7f94640700f0 2024-09-17T18:21:24.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f946a394640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:24.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f946a394640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9464135390 con 0x7f946409ec60 2024-09-17T18:21:24.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f946a394640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:24.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f946a394640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9464138fb0 con 0x7f946409e810 2024-09-17T18:21:24.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f946a394640 10 monclient(hunting): _renew_subs 2024-09-17T18:21:24.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f9463fff640 1 -- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94640700f0 msgr2=0x7f9464131bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:21:24.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f946a394640 10 monclient(hunting): authenticate will time out at 887.898222s 2024-09-17T18:21:24.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f9463fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94640700f0 0x7f9464131bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:21:24.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f94637fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f946409e810 0x7f9464132100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:24.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f94637fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f946409e810 0x7f9464132100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:54338/0 (socket says 172.21.15.121:54338) 2024-09-17T18:21:24.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f94637fe640 1 -- 172.21.15.121:0/2490995267 learned_addr learned my addr 172.21.15.121:0/2490995267 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:21:24.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f94637fe640 10 monclient(hunting): get_auth_request con 0x7f946409e810 auth_method 0 2024-09-17T18:21:24.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f94637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:24.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f94637fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:24.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f94637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:24.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f94637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:24.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f94637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:24.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f94637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:24.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f9468931640 1 --2- 172.21.15.121:0/2490995267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f946409ec60 0x7f94640a3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:24.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f9468931640 10 monclient(hunting): get_auth_request con 0x7f946409ec60 auth_method 0 2024-09-17T18:21:24.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f9468931640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:24.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f9468931640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:24.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f9468931640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:24.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f94637fe640 10 monclient(hunting): handle_auth_done global_id 34181 payload 824 2024-09-17T18:21:24.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f94637fe640 1 -- 172.21.15.121:0/2490995267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f946409ec60 msgr2=0x7f94640a3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:24.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f94637fe640 1 --2- 172.21.15.121:0/2490995267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f946409ec60 0x7f94640a3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:24.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f94637fe640 1 -- 172.21.15.121:0/2490995267 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94640700f0 msgr2=0x7f9464131bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:21:24.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f94637fe640 1 --2- 172.21.15.121:0/2490995267 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94640700f0 0x7f9464131bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:24.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f94637fe640 10 monclient: _finish_hunting 0 2024-09-17T18:21:24.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f94637fe640 1 monclient: found mon.noname-c 2024-09-17T18:21:24.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f94637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:21:24.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f94637fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:21:24.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.181+0000 7f94637fe640 1 -- 172.21.15.121:0/2490995267 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94641bf8e0 con 0x7f946409e810 2024-09-17T18:21:24.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.183+0000 7f94637fe640 1 -- 172.21.15.121:0/2490995267 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f946409e810 msgr2=0x7f9464132100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:21:24.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.183+0000 7f94637fe640 1 --2- 172.21.15.121:0/2490995267 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f946409e810 0x7f9464132100 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9458000c00 tx=0x7f94580046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:21:24.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.183+0000 7f9460ff9640 10 client.?.objecter ms_handle_connect 0x7f946409e810 2024-09-17T18:21:24.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.183+0000 7f9460ff9640 10 client.?.objecter resend_mon_ops 2024-09-17T18:21:24.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.183+0000 7f94637fe640 1 --2- 172.21.15.121:0/2490995267 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f946409e810 0x7f9464132100 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9458000c00 tx=0x7f94580046c0 comp rx=0 tx=0).queue_received will delay after 1.7266e+09 on 0x7f9458041070 mon_map magic: 0 2024-09-17T18:21:24.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.183+0000 7f9460ff9640 1 -- 172.21.15.121:0/2490995267 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9458041070 con 0x7f946409e810 2024-09-17T18:21:24.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.184+0000 7f9460ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:24.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.184+0000 7f9460ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:21:24.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.184+0000 7f9460ff9640 10 monclient: dump: 2024-09-17T18:21:24.212 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:24.212 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:24.213 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:24.213 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:24.213 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:24.213 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:24.213 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:24.214 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:24.214 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:24.214 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:24.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.184+0000 7f9460ff9640 10 monclient: _finish_auth 0 2024-09-17T18:21:24.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.184+0000 7f9460ff9640 10 monclient: _check_auth_tickets 2024-09-17T18:21:24.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.184+0000 7f9460ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:21:24.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.184+0000 7f9460ff9640 1 -- 172.21.15.121:0/2490995267 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f945803c880 con 0x7f946409e810 2024-09-17T18:21:24.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.184+0000 7f946a394640 5 monclient: authenticate success, global_id 34181 2024-09-17T18:21:24.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.184+0000 7f9460ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T18:21:24.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.184+0000 7f946a394640 10 monclient: _renew_subs 2024-09-17T18:21:24.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.184+0000 7f946a394640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:21:24.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.184+0000 7f946a394640 1 -- 172.21.15.121:0/2490995267 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94640a99b0 con 0x7f946409e810 2024-09-17T18:21:24.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.184+0000 7f94627fc640 4 set_mon_vals no callback set 2024-09-17T18:21:24.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.184+0000 7f946a394640 10 client.34181.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:21:24.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.184+0000 7f946a394640 10 monclient: _renew_subs 2024-09-17T18:21:24.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.184+0000 7f946a394640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:21:24.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.184+0000 7f946a394640 1 -- 172.21.15.121:0/2490995267 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f94641be3a0 con 0x7f946409e810 2024-09-17T18:21:24.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.184+0000 7f946a394640 1 librados: init done 2024-09-17T18:21:24.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.184+0000 7f946a394640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:21:24.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.184+0000 7f946a394640 1 --2- 172.21.15.121:0/2490995267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9464135f30 0x7f9464136310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:24.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.184+0000 7f946a394640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:24.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.184+0000 7f946a394640 1 -- 172.21.15.121:0/2490995267 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f94641bf080 con 0x7f9464135f30 2024-09-17T18:21:24.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.184+0000 7f9460ff9640 1 -- 172.21.15.121:0/2490995267 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9458037040 con 0x7f946409e810 2024-09-17T18:21:24.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.184+0000 7f9460ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:24.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.184+0000 7f9460ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:21:24.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.184+0000 7f9460ff9640 10 monclient: dump: 2024-09-17T18:21:24.219 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:24.219 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:24.219 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:24.219 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:24.219 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:24.219 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:24.220 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:24.220 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:24.220 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:24.220 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:24.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.184+0000 7f9460ff9640 1 -- 172.21.15.121:0/2490995267 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f9458044070 con 0x7f946409e810 2024-09-17T18:21:24.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.185+0000 7f9460ff9640 1 --2- 172.21.15.121:0/2490995267 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9450040120 0x7f94500425e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:24.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.185+0000 7f9463fff640 1 --2- 172.21.15.121:0/2490995267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9464135f30 0x7f9464136310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:24.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.185+0000 7f9463fff640 10 monclient: get_auth_request con 0x7f9464135f30 auth_method 0 2024-09-17T18:21:24.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.185+0000 7f9463fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:24.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.185+0000 7f9463fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:24.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.185+0000 7f9463fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:24.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.185+0000 7f9463fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:24.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.185+0000 7f9463fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:24.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.185+0000 7f9463fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:24.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.185+0000 7f9468931640 1 --2- 172.21.15.121:0/2490995267 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9450040120 0x7f94500425e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:24.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.185+0000 7f9468931640 10 monclient: get_auth_request con 0x7f9450040120 auth_method 0 2024-09-17T18:21:24.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.185+0000 7f9463fff640 10 monclient(hunting): handle_auth_done global_id 24208 payload 824 2024-09-17T18:21:24.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.185+0000 7f9463fff640 1 -- 172.21.15.121:0/2490995267 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9464134e50 con 0x7f9464135f30 2024-09-17T18:21:24.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.187+0000 7f9463fff640 1 -- 172.21.15.121:0/2490995267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9464135f30 msgr2=0x7f9464136310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:21:24.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.187+0000 7f9460ff9640 10 client.34181.objecter ms_handle_connect 0x7f9464135f30 2024-09-17T18:21:24.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.187+0000 7f9460ff9640 10 client.34181.objecter resend_mon_ops 2024-09-17T18:21:24.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.187+0000 7f9463fff640 1 --2- 172.21.15.121:0/2490995267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9464135f30 0x7f9464136310 secure :-1 s=READY pgs=7129 cs=0 l=1 rev1=1 crypto rx=0x7f945400b450 tx=0x7f945400b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:21:24.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.187+0000 7f9463fff640 1 -- 172.21.15.121:0/2490995267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9464135f30 msgr2=0x7f9464136310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T18:21:24.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.187+0000 7f9463fff640 1 -- 172.21.15.121:0/2490995267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9464135f30 msgr2=0x7f9464136310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:21:24.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.187+0000 7f9463fff640 1 --2- 172.21.15.121:0/2490995267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9464135f30 0x7f9464136310 secure :-1 s=READY pgs=7129 cs=0 l=1 rev1=1 crypto rx=0x7f945400b450 tx=0x7f945400b910 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:21:24.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.187+0000 7f9463fff640 1 --2- 172.21.15.121:0/2490995267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9464135f30 0x7f9464136310 secure :-1 s=READY pgs=7129 cs=0 l=1 rev1=1 crypto rx=0x7f945400b450 tx=0x7f945400b910 comp rx=0 tx=0).stop 2024-09-17T18:21:24.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.187+0000 7f9460ff9640 1 -- 172.21.15.121:0/2490995267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9464135f30 msgr2=0x7f9464136310 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:21:24.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.187+0000 7f9460ff9640 1 --2- 172.21.15.121:0/2490995267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9464135f30 0x7f9464136310 unknown :-1 s=CLOSED pgs=7129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:24.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.187+0000 7f9460ff9640 1 --2- 172.21.15.121:0/2490995267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9450042cb0 0x7f94500450a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:24.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.187+0000 7f9460ff9640 1 -- 172.21.15.121:0/2490995267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9464135f30 msgr2=0x7f9464136310 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:21:24.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.187+0000 7f9460ff9640 1 --2- 172.21.15.121:0/2490995267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9464135f30 0x7f9464136310 unknown :-1 s=CLOSED pgs=7129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:24.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.187+0000 7f9460ff9640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:24.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.187+0000 7f9460ff9640 1 -- 172.21.15.121:0/2490995267 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9450045680 con 0x7f9450042cb0 2024-09-17T18:21:24.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.187+0000 7f9460ff9640 10 client.34181.objecter ms_handle_connect 0x7f9450040120 2024-09-17T18:21:24.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.187+0000 7f9468931640 1 --2- 172.21.15.121:0/2490995267 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9450040120 0x7f94500425e0 secure :-1 s=READY pgs=3872 cs=0 l=1 rev1=1 crypto rx=0x7f945c0076f0 tx=0x7f945c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:21:24.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.188+0000 7f9463fff640 1 --2- 172.21.15.121:0/2490995267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9450042cb0 0x7f94500450a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:24.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.188+0000 7f9463fff640 10 monclient: get_auth_request con 0x7f9450042cb0 auth_method 0 2024-09-17T18:21:24.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.188+0000 7f9463fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:24.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.188+0000 7f9463fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:24.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.188+0000 7f9463fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:24.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.188+0000 7f9463fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:24.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.188+0000 7f9463fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:24.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.188+0000 7f9463fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:24.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.188+0000 7f9463fff640 10 monclient(hunting): handle_auth_done global_id 24211 payload 824 2024-09-17T18:21:24.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.188+0000 7f9463fff640 1 -- 172.21.15.121:0/2490995267 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9458032640 con 0x7f9450042cb0 2024-09-17T18:21:24.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.190+0000 7f9460ff9640 10 client.34181.objecter ms_handle_connect 0x7f9450042cb0 2024-09-17T18:21:24.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.190+0000 7f9460ff9640 10 client.34181.objecter resend_mon_ops 2024-09-17T18:21:24.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.190+0000 7f9463fff640 1 -- 172.21.15.121:0/2490995267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9450042cb0 msgr2=0x7f94500450a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:21:24.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.190+0000 7f9463fff640 1 --2- 172.21.15.121:0/2490995267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9450042cb0 0x7f94500450a0 secure :-1 s=READY pgs=7130 cs=0 l=1 rev1=1 crypto rx=0x7f945400beb0 tx=0x7f94540059d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:21:24.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.190+0000 7f9460ff9640 1 -- 172.21.15.121:0/2490995267 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f945401cbc0 con 0x7f9450042cb0 2024-09-17T18:21:24.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.190+0000 7f9460ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:24.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.190+0000 7f9460ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:21:24.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.190+0000 7f9460ff9640 10 monclient: dump: 2024-09-17T18:21:24.225 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:24.225 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:24.225 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:24.225 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:24.225 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:24.225 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:24.225 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:24.225 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:24.225 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:24.225 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:24.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.191+0000 7f9460ff9640 1 -- 172.21.15.121:0/2490995267 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9454015070 con 0x7f9450042cb0 2024-09-17T18:21:24.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.191+0000 7f9460ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:21:24.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.191+0000 7f9460ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:21:24.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.191+0000 7f9460ff9640 1 -- 172.21.15.121:0/2490995267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9450042cb0 msgr2=0x7f94500450a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:24.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.191+0000 7f9460ff9640 1 --2- 172.21.15.121:0/2490995267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9450042cb0 0x7f94500450a0 secure :-1 s=READY pgs=7130 cs=0 l=1 rev1=1 crypto rx=0x7f945400beb0 tx=0x7f94540059d0 comp rx=0 tx=0).stop 2024-09-17T18:21:24.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.191+0000 7f9460ff9640 1 -- 172.21.15.121:0/2490995267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9450042cb0 msgr2=0x7f94500450a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:21:24.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.191+0000 7f9460ff9640 1 --2- 172.21.15.121:0/2490995267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9450042cb0 0x7f94500450a0 secure :-1 s=CLOSED pgs=7130 cs=0 l=1 rev1=1 crypto rx=0x7f945400beb0 tx=0x7f94540059d0 comp rx=0 tx=0).stop 2024-09-17T18:21:24.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.197+0000 7f946a394640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:21:24.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.197+0000 7f946a394640 1 --2- 172.21.15.121:0/2490995267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9464136850 0x7f94640a3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:21:24.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.197+0000 7f946a394640 10 monclient(hunting): start opening mon connection 2024-09-17T18:21:24.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.197+0000 7f946a394640 1 -- 172.21.15.121:0/2490995267 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f946406a660 con 0x7f9464136850 2024-09-17T18:21:24.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.197+0000 7f9463fff640 1 --2- 172.21.15.121:0/2490995267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9464136850 0x7f94640a3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:21:24.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.197+0000 7f9463fff640 10 monclient: get_auth_request con 0x7f9464136850 auth_method 0 2024-09-17T18:21:24.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.197+0000 7f9463fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:21:24.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.197+0000 7f9463fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:21:24.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.197+0000 7f9463fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:21:24.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.197+0000 7f9463fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:21:24.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.197+0000 7f9463fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:21:24.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.197+0000 7f9463fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:21:24.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.197+0000 7f9463fff640 10 monclient(hunting): handle_auth_done global_id 24214 payload 824 2024-09-17T18:21:24.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.197+0000 7f9463fff640 1 -- 172.21.15.121:0/2490995267 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9464132640 con 0x7f9464136850 2024-09-17T18:21:24.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.200+0000 7f9463fff640 1 -- 172.21.15.121:0/2490995267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9464136850 msgr2=0x7f94640a3860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:21:24.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.200+0000 7f9460ff9640 10 client.34181.objecter ms_handle_connect 0x7f9464136850 2024-09-17T18:21:24.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.200+0000 7f9460ff9640 10 client.34181.objecter resend_mon_ops 2024-09-17T18:21:24.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.200+0000 7f9463fff640 1 --2- 172.21.15.121:0/2490995267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9464136850 0x7f94640a3860 secure :-1 s=READY pgs=7131 cs=0 l=1 rev1=1 crypto rx=0x7f9454012040 tx=0x7f9454015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:21:24.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.200+0000 7f9460ff9640 1 -- 172.21.15.121:0/2490995267 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f945402aa30 con 0x7f9464136850 2024-09-17T18:21:24.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.200+0000 7f9460ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:21:24.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.200+0000 7f9460ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:21:24.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.200+0000 7f9460ff9640 10 monclient: dump: 2024-09-17T18:21:24.230 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:21:24.230 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:21:24.230 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:24.230 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:21:24.230 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:21:24.230 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:21:24.230 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:21:24.230 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:21:24.230 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:21:24.230 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:21:24.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.200+0000 7f9460ff9640 1 -- 172.21.15.121:0/2490995267 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7f945401a480 con 0x7f9464136850 2024-09-17T18:21:24.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.200+0000 7f9460ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:21:24.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.200+0000 7f9460ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:21:24.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.200+0000 7f9460ff9640 1 -- 172.21.15.121:0/2490995267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9464136850 msgr2=0x7f94640a3860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:24.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.200+0000 7f9460ff9640 1 --2- 172.21.15.121:0/2490995267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9464136850 0x7f94640a3860 secure :-1 s=READY pgs=7131 cs=0 l=1 rev1=1 crypto rx=0x7f9454012040 tx=0x7f9454015000 comp rx=0 tx=0).stop 2024-09-17T18:21:24.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.200+0000 7f9460ff9640 1 -- 172.21.15.121:0/2490995267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9464136850 msgr2=0x7f94640a3860 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:21:24.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.200+0000 7f9460ff9640 1 --2- 172.21.15.121:0/2490995267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9464136850 0x7f94640a3860 unknown :-1 s=CLOSED pgs=7131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:24.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.200+0000 7f9468931640 1 -- 172.21.15.121:0/2490995267 reap_dead start 2024-09-17T18:21:24.231 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:21:24.231 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:21:24.231 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:21:24.231 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "leader", 2024-09-17T18:21:24.231 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 12, 2024-09-17T18:21:24.231 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 157146, 2024-09-17T18:21:24.231 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:21:24.231 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:21:24.231 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:21:24.232 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:21:24.232 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 1, 2024-09-17T18:21:24.232 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:21:24.232 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:21:24.232 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:21:24.232 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:21:24.232 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:21:24.232 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:21:24.232 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:21:24.232 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:21:24.232 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:21:24.232 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:21:24.232 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:21:24.232 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:21:24.233 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:21:24.233 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:21:24.233 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:21:24.233 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:21:24.233 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:21:24.233 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:21:24.233 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:21:24.233 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:21:24.233 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:21:24.233 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:21:24.233 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:21:24.233 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:21:24.233 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:21:24.233 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:21:24.234 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:21:24.234 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:21:24.234 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:21:24.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:24.234 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:21:24.234 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:21:24.234 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:21:24.234 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:21:24.234 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:21:24.234 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:21:24.234 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:21:24.234 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:21:24.234 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:21:24.235 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:21:24.235 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:21:24.235 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:21:24.235 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:21:24.235 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:21:24.235 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:21:24.235 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:21:24.235 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:21:24.235 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:21:24.235 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:21:24.235 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:21:24.235 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:21:24.235 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:21:24.235 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:21:24.236 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:21:24.236 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:21:24.236 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:21:24.236 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:21:24.236 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:21:24.236 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:21:24.236 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:21:24.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:24.236 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:21:24.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:24.236 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:21:24.236 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:21:24.237 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:21:24.237 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:21:24.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:24.237 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:21:24.237 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:21:24.237 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:21:24.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:24.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:24.237 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:21:24.237 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:21:24.237 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:21:24.237 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:24.237 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:21:24.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:24.238 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:21:24.238 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:21:24.238 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:21:24.238 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:21:24.238 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:21:24.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:24.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:24.238 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:21:24.238 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:21:24.238 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:21:24.238 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:21:24.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:24.238 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:21:24.238 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:21:24.239 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:21:24.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:24.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:24.239 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:21:24.239 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:21:24.239 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:21:24.239 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:24.239 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:21:24.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:24.239 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:21:24.239 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:21:24.239 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:21:24.239 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:21:24.239 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:21:24.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:24.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:24.240 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:21:24.240 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:21:24.240 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:21:24.240 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:21:24.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:24.240 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:21:24.240 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:21:24.240 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:21:24.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:24.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:24.240 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:21:24.240 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:21:24.240 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:21:24.241 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:24.241 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:21:24.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:24.241 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:21:24.241 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:21:24.241 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:21:24.241 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:21:24.241 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:21:24.241 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:24.241 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:21:24.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:24.241 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:21:24.241 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:21:24.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:24.242 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:21:24.242 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:21:24.242 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:21:24.242 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:24.242 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:21:24.242 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:21:24.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:24.242 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:21:24.242 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:21:24.242 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 5 2024-09-17T18:21:24.242 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:24.242 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:21:24.242 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:21:24.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:24.243 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:21:24.243 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:21:24.243 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 11 2024-09-17T18:21:24.243 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:24.243 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:21:24.243 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": [ 2024-09-17T18:21:24.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:21:24.243 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:21:24.243 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:21:24.243 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:21:24.244 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:21:24.244 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:21:24.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:21:24.244 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:21:24.244 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:21:24.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.200+0000 7f946a394640 10 librados: watch_flush enter 2024-09-17T18:21:24.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.200+0000 7f946a394640 10 librados: watch_flush exit 2024-09-17T18:21:24.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.200+0000 7f946a394640 20 client.34181.objecter shutdown clearing up homeless session... 2024-09-17T18:21:24.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.200+0000 7f946a394640 10 client.34181.objecter successfully canceled tick 2024-09-17T18:21:24.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.200+0000 7f946a394640 1 -- 172.21.15.121:0/2490995267 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9450040120 msgr2=0x7f94500425e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:24.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.200+0000 7f946a394640 1 --2- 172.21.15.121:0/2490995267 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9450040120 0x7f94500425e0 secure :-1 s=READY pgs=3872 cs=0 l=1 rev1=1 crypto rx=0x7f945c0076f0 tx=0x7f945c009040 comp rx=0 tx=0).stop 2024-09-17T18:21:24.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.200+0000 7f946a394640 10 monclient: shutdown 2024-09-17T18:21:24.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.200+0000 7f946a394640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:21:24.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.200+0000 7f946a394640 1 -- 172.21.15.121:0/2490995267 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f946409e810 msgr2=0x7f9464132100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:21:24.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.200+0000 7f946a394640 1 --2- 172.21.15.121:0/2490995267 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f946409e810 0x7f9464132100 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9458000c00 tx=0x7f94580046c0 comp rx=0 tx=0).stop 2024-09-17T18:21:24.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.200+0000 7f9460ff9640 1 -- 172.21.15.121:0/2490995267 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(81..81 src has 1..81) ==== 6871+0+0 (unknown 0 0 0) 0x7f94580745d0 con 0x7f946409e810 2024-09-17T18:21:24.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.200+0000 7f9460ff9640 10 client.34181.objecter ms_dispatch 0x7f9464000b90 osd_map(81..81 src has 1..81) 2024-09-17T18:21:24.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.200+0000 7f9460ff9640 10 librados: disconnected, discarding osd_map(81..81 src has 1..81) 2024-09-17T18:21:24.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.200+0000 7f946a394640 1 -- 172.21.15.121:0/2490995267 shutdown_connections 2024-09-17T18:21:24.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.200+0000 7f946a394640 1 --2- 172.21.15.121:0/2490995267 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9450040120 0x7f94500425e0 unknown :-1 s=CLOSED pgs=3872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:24.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.200+0000 7f946a394640 1 --2- 172.21.15.121:0/2490995267 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f946409e810 0x7f9464132100 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:21:24.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.200+0000 7f946a394640 1 -- 172.21.15.121:0/2490995267 >> 172.21.15.121:0/2490995267 conn(0x7f94640ab830 msgr2=0x7f9464133b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:21:24.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.200+0000 7f946a394640 1 -- 172.21.15.121:0/2490995267 shutdown_connections 2024-09-17T18:21:24.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.200+0000 7f946a394640 1 -- 172.21.15.121:0/2490995267 wait complete. 2024-09-17T18:21:24.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:21:24.201+0000 7f946a394640 1 librados: shutdown 2024-09-17T18:21:24.248 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T18:21:24.248 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-17T18:21:27.093 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-09-17T18:21:27.093 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-09-17T18:21:27.093 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-09-17T18:21:27.093 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-09-17T18:21:27.093 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-09-17T18:21:27.094 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-09-17T18:21:27.094 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: 2024-09-17T18:21:27.094 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-09-17T18:21:27.094 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: seed 35220 2024-09-17T18:21:27.094 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-09-17T18:21:27.094 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (0 ms) 2024-09-17T18:21:27.094 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-09-17T18:21:27.094 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (227 ms) 2024-09-17T18:21:27.094 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-09-17T18:21:27.094 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (21 ms) 2024-09-17T18:21:27.094 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-09-17T18:21:27.094 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (11 ms) 2024-09-17T18:21:27.094 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-09-17T18:21:27.094 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (18 ms) 2024-09-17T18:21:27.095 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-09-17T18:21:27.095 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (4 ms) 2024-09-17T18:21:27.095 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-09-17T18:21:27.095 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (8 ms) 2024-09-17T18:21:27.095 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-09-17T18:21:27.095 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (11 ms) 2024-09-17T18:21:27.095 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-09-17T18:21:27.095 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (3 ms) 2024-09-17T18:21:27.095 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-09-17T18:21:27.095 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (27 ms) 2024-09-17T18:21:27.095 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-09-17T18:21:27.095 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (2 ms) 2024-09-17T18:21:27.095 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-09-17T18:21:27.095 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (18 ms) 2024-09-17T18:21:27.096 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-09-17T18:21:27.096 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (5 ms) 2024-09-17T18:21:27.096 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-09-17T18:21:27.096 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: osd_max_attr_size = 0 2024-09-17T18:21:27.096 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-09-17T18:21:27.096 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (2982 ms) 2024-09-17T18:21:27.096 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-09-17T18:21:27.096 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (254 ms) 2024-09-17T18:21:27.096 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-09-17T18:21:27.096 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-09-17T18:21:27.096 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: done waiting, doing copies 2024-09-17T18:21:27.096 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: waiting for final deep scrubs... 2024-09-17T18:21:27.096 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: done waiting 2024-09-17T18:21:27.097 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (60583 ms) 2024-09-17T18:21:27.097 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-09-17T18:21:27.097 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (7 ms) 2024-09-17T18:21:27.097 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-09-17T18:21:27.097 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (6 ms) 2024-09-17T18:21:27.097 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-09-17T18:21:27.097 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (20309 ms) 2024-09-17T18:21:27.097 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-09-17T18:21:27.097 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-09-17T18:21:27.097 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-09-17T18:21:27.097 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-09-17T18:21:27.098 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-09-17T18:21:27.098 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-09-17T18:21:29.105 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [----------] 22 tests from LibRadosMis api_snapshots_pp: Running main() from gmock_main.cc 2024-09-17T18:21:29.105 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-09-17T18:21:29.105 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-09-17T18:21:29.105 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-09-17T18:21:29.105 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: seed 35395 2024-09-17T18:21:29.105 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-09-17T18:21:29.105 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (2017 ms) 2024-09-17T18:21:29.105 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-09-17T18:21:29.106 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (2022 ms) 2024-09-17T18:21:29.106 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-09-17T18:21:29.106 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (1999 ms) 2024-09-17T18:21:29.106 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-09-17T18:21:29.106 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (2006 ms) 2024-09-17T18:21:29.106 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-09-17T18:21:29.106 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (3873 ms) 2024-09-17T18:21:29.106 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (11917 ms total) 2024-09-17T18:21:29.106 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: 2024-09-17T18:21:29.106 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-09-17T18:21:29.106 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-09-17T18:21:29.106 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (4090 ms) 2024-09-17T18:21:29.106 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-09-17T18:21:29.106 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (3916 ms) 2024-09-17T18:21:29.107 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-09-17T18:21:29.107 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (5996 ms) 2024-09-17T18:21:29.107 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-09-17T18:21:29.107 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (3987 ms) 2024-09-17T18:21:29.107 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-09-17T18:21:29.107 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (2050 ms) 2024-09-17T18:21:29.107 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-09-17T18:21:29.107 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4975-g1889a165/rpm/el9/BUILD/ceph-19.3.0-4975-g1889a165/src/test/librados/snapshots_cxx.cc:460: Skipped 2024-09-17T18:21:29.107 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: 2024-09-17T18:21:29.107 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-09-17T18:21:29.107 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-09-17T18:21:29.107 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi173-35395-7 2024-09-17T18:21:29.107 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: waiting for snaps to purge 2024-09-17T18:21:29.107 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (17965 ms) 2024-09-17T18:21:29.108 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (38004 ms total) 2024-09-17T18:21:29.108 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: 2024-09-17T18:21:29.108 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-09-17T18:21:29.108 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-09-17T18:21:29.108 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (5 ms) 2024-09-17T18:21:29.108 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-09-17T18:21:29.108 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (4370 ms) 2024-09-17T18:21:29.108 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (4375 ms total) 2024-09-17T18:21:29.108 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: 2024-09-17T18:21:29.108 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-09-17T18:21:29.108 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-09-17T18:21:29.108 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (2478 ms) 2024-09-17T18:21:29.108 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-09-17T18:21:45.099 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ OK ] LibRadosS api_aio: Running main() from gmock_main.cc 2024-09-17T18:21:45.100 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-09-17T18:21:45.100 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [----------] Global test environment set-up. 2024-09-17T18:21:45.100 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-09-17T18:21:45.100 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-09-17T18:21:45.100 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ OK ] LibRadosAio.TooBig (3526 ms) 2024-09-17T18:21:45.100 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-09-17T18:21:45.100 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (3049 ms) 2024-09-17T18:21:45.100 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-09-17T18:21:45.100 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (2983 ms) 2024-09-17T18:21:45.100 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-09-17T18:21:45.100 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (2875 ms) 2024-09-17T18:21:45.101 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-09-17T18:21:45.101 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (3020 ms) 2024-09-17T18:21:45.101 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-09-17T18:21:45.101 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (3010 ms) 2024-09-17T18:21:45.101 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-09-17T18:21:45.101 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (2970 ms) 2024-09-17T18:21:45.101 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-09-17T18:21:45.101 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (3085 ms) 2024-09-17T18:21:45.101 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-09-17T18:21:45.101 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (2900 ms) 2024-09-17T18:21:45.101 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-09-17T18:21:45.101 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (3007 ms) 2024-09-17T18:21:45.101 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-09-17T18:21:45.101 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (3023 ms) 2024-09-17T18:21:45.102 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-09-17T18:21:45.102 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (3022 ms) 2024-09-17T18:21:45.102 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-09-17T18:21:45.102 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (2969 ms) 2024-09-17T18:21:45.102 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-09-17T18:21:45.102 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (2999 ms) 2024-09-17T18:21:45.102 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-09-17T18:21:45.102 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ OK ] LibRadosAio.Flush (2996 ms) 2024-09-17T18:21:45.102 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-09-17T18:21:45.102 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (3003 ms) 2024-09-17T18:21:45.102 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-09-17T18:21:45.102 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (2989 ms) 2024-09-17T18:21:45.102 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-09-17T18:21:45.102 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (3008 ms) 2024-09-17T18:21:45.103 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-09-17T18:21:45.103 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (3225 ms) 2024-09-17T18:21:45.103 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-09-17T18:21:45.103 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (3189 ms) 2024-09-17T18:21:45.103 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-09-17T18:21:45.103 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (22139 ms) 2024-09-17T18:21:45.103 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-09-17T18:21:45.103 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (3015 ms) 2024-09-17T18:21:45.103 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-09-17T18:21:45.103 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (2477 ms) 2024-09-17T18:21:45.103 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-09-17T18:21:45.103 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (2988 ms) 2024-09-17T18:21:45.103 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-09-17T18:21:45.104 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3002 ms) 2024-09-17T18:21:45.104 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-09-17T18:21:45.104 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3009 ms) 2024-09-17T18:21:45.104 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [----------] 26 tests from LibRadosAio (97478 ms total) 2024-09-17T18:21:45.104 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: 2024-09-17T18:21:45.104 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-09-17T18:21:45.104 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWrite 2024-09-17T18:21:53.111 INFO:tasks.workunit.client.0.smithi173.stdout:cPP (84497 ms total) 2024-09-17T18:21:53.111 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: 2024-09-17T18:21:53.111 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-09-17T18:21:53.111 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-09-17T18:21:53.111 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (114 ms) 2024-09-17T18:21:53.111 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (114 ms total) 2024-09-17T18:21:53.111 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: 2024-09-17T18:21:53.111 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-09-17T18:21:53.111 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-09-17T18:21:53.112 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (17 ms) 2024-09-17T18:21:53.112 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-09-17T18:21:53.112 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (7 ms) 2024-09-17T18:21:53.112 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (24 ms total) 2024-09-17T18:21:53.112 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: 2024-09-17T18:21:53.112 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-09-17T18:21:53.112 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-09-17T18:21:53.112 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (36 ms) 2024-09-17T18:21:53.112 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-09-17T18:21:53.112 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (9 ms) 2024-09-17T18:21:53.112 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (45 ms total) 2024-09-17T18:21:53.112 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: 2024-09-17T18:21:53.113 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-09-17T18:21:53.113 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-09-17T18:21:53.113 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (24 ms) 2024-09-17T18:21:53.113 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-09-17T18:21:53.113 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (9 ms) 2024-09-17T18:21:53.113 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (33 ms total) 2024-09-17T18:21:53.113 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: 2024-09-17T18:21:53.113 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-09-17T18:21:53.113 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-09-17T18:21:53.113 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1039 ms) 2024-09-17T18:21:53.113 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1039 ms total) 2024-09-17T18:21:53.113 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: 2024-09-17T18:21:53.114 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-09-17T18:21:53.114 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (112364 ms total) 2024-09-17T18:21:53.114 INFO:tasks.workunit.client.0.smithi173.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-09-17T18:21:54.105 INFO:tasks.workunit.client.0.smithi173.stdout:napshotsECPP.SnapRemovePP (2016 ms) 2024-09-17T18:21:54.106 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-09-17T18:21:54.106 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (2393 ms) 2024-09-17T18:21:54.106 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-09-17T18:21:54.106 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (1594 ms) 2024-09-17T18:21:54.106 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (8481 ms total) 2024-09-17T18:21:54.106 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: 2024-09-17T18:21:54.106 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-09-17T18:21:54.106 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-09-17T18:21:54.106 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (4012 ms) 2024-09-17T18:21:54.106 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-09-17T18:21:54.106 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (3989 ms) 2024-09-17T18:21:54.106 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-09-17T18:21:54.106 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (4000 ms) 2024-09-17T18:21:54.107 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (12002 ms total) 2024-09-17T18:21:54.107 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: 2024-09-17T18:21:54.107 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-09-17T18:21:54.107 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (113346 ms total) 2024-09-17T18:21:54.107 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-09-17T18:21:54.107 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-09-17T18:21:54.107 INFO:tasks.workunit.client.0.smithi173.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-09-17T18:21:57.098 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-09-17T18:21:57.098 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-09-17T18:21:57.098 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-09-17T18:21:57.099 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-09-17T18:21:57.099 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-09-17T18:21:57.099 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (2520 ms) 2024-09-17T18:21:57.099 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-09-17T18:21:57.099 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (15921 ms) 2024-09-17T18:21:57.099 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-09-17T18:21:57.099 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (6112 ms) 2024-09-17T18:21:57.099 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-09-17T18:21:57.099 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (2869 ms) 2024-09-17T18:21:57.099 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-09-17T18:21:57.099 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (4021 ms) 2024-09-17T18:21:57.099 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-09-17T18:21:57.099 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (2993 ms) 2024-09-17T18:21:57.100 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-09-17T18:21:57.100 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (3002 ms) 2024-09-17T18:21:57.100 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-09-17T18:21:57.100 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (3004 ms) 2024-09-17T18:21:57.100 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-09-17T18:21:57.100 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (2993 ms) 2024-09-17T18:21:57.100 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-09-17T18:21:57.100 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (2987 ms) 2024-09-17T18:21:57.100 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-09-17T18:21:57.100 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (3020 ms) 2024-09-17T18:21:57.100 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-09-17T18:21:57.100 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (2987 ms) 2024-09-17T18:21:57.100 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-09-17T18:21:57.100 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3227 ms) 2024-09-17T18:21:57.101 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-09-17T18:21:57.101 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (3001 ms) 2024-09-17T18:21:57.101 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-09-17T18:21:57.101 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (2878 ms) 2024-09-17T18:21:57.101 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-09-17T18:21:57.101 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (22450 ms) 2024-09-17T18:21:57.101 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-09-17T18:21:57.101 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (2478 ms) 2024-09-17T18:21:57.101 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-09-17T18:21:57.101 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (3383 ms) 2024-09-17T18:21:57.101 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-09-17T18:21:57.101 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (2619 ms) 2024-09-17T18:21:57.101 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-09-17T18:21:57.101 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3007 ms) 2024-09-17T18:21:57.102 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-09-17T18:21:57.102 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (3008 ms) 2024-09-17T18:21:57.102 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-09-17T18:21:57.102 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (2984 ms) 2024-09-17T18:21:57.102 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-09-17T18:21:57.102 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3012 ms) 2024-09-17T18:21:57.102 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-09-17T18:21:57.102 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (3004 ms) 2024-09-17T18:21:57.102 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-09-17T18:21:57.102 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (2997 ms) 2024-09-17T18:21:57.102 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-09-17T18:21:57.102 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3001 ms) 2024-09-17T18:21:57.102 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-09-17T18:22:51.298 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAi : cursor()=8:f905c69b:::2:head expected=8:f905c69b:::2:head 2024-09-17T18:22:51.298 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > 8:f905c69b:::2:head -> 2 2024-09-17T18:22:51.298 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : entry=2 expected=2 2024-09-17T18:22:51.299 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : seek to 8:e960b815:::9:head 2024-09-17T18:22:51.299 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : cursor()=8:e960b815:::9:head expected=8:e960b815:::9:head 2024-09-17T18:22:51.299 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > 8:e960b815:::9:head -> 9 2024-09-17T18:22:51.299 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : entry=9 expected=9 2024-09-17T18:22:51.299 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : seek to 8:bd63b0f1:::8:head 2024-09-17T18:22:51.299 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : cursor()=8:bd63b0f1:::8:head expected=8:bd63b0f1:::8:head 2024-09-17T18:22:51.299 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > 8:bd63b0f1:::8:head -> 8 2024-09-17T18:22:51.299 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : entry=8 expected=8 2024-09-17T18:22:51.299 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : seek to 8:566253c9:::13:head 2024-09-17T18:22:51.299 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : cursor()=8:566253c9:::13:head expected=8:566253c9:::13:head 2024-09-17T18:22:51.299 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > 8:566253c9:::13:head -> 13 2024-09-17T18:22:51.299 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : entry=13 expected=13 2024-09-17T18:22:51.300 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : seek to 8:89d3ae78:::11:head 2024-09-17T18:22:51.300 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : cursor()=8:89d3ae78:::11:head expected=8:89d3ae78:::11:head 2024-09-17T18:22:51.300 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > 8:89d3ae78:::11:head -> 11 2024-09-17T18:22:51.300 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : entry=11 expected=11 2024-09-17T18:22:51.300 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : seek to 8:863748b0:::15:head 2024-09-17T18:22:51.300 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : cursor()=8:863748b0:::15:head expected=8:863748b0:::15:head 2024-09-17T18:22:51.300 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > 8:863748b0:::15:head -> 15 2024-09-17T18:22:51.300 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : entry=15 expected=15 2024-09-17T18:22:51.300 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : seek to 8:6cac518f:::0:head 2024-09-17T18:22:51.300 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : cursor()=8:6cac518f:::0:head expected=8:6cac518f:::0:head 2024-09-17T18:22:51.300 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > 8:6cac518f:::0:head -> 0 2024-09-17T18:22:51.300 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : entry=0 expected=0 2024-09-17T18:22:51.300 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : seek to 8:62a1935d:::14:head 2024-09-17T18:22:51.300 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : cursor()=8:62a1935d:::14:head expected=8:62a1935d:::14:head 2024-09-17T18:22:51.301 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: > 8:62a1935d:::14:head -> 14 2024-09-17T18:22:51.301 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: : entry=14 expected=14 2024-09-17T18:22:51.301 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (303 ms) 2024-09-17T18:22:51.301 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-09-17T18:22:51.301 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (159153 ms) 2024-09-17T18:22:51.301 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-09-17T18:22:51.301 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: split 0/5 -> MIN 8:33333333::::head 2024-09-17T18:22:51.301 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: split 1/5 -> 8:33333333::::head 8:66666666::::head 2024-09-17T18:22:51.301 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: split 2/5 -> 8:66666666::::head 8:99999999::::head 2024-09-17T18:22:51.301 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: split 3/5 -> 8:99999999::::head 8:cccccccc::::head 2024-09-17T18:22:51.301 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: split 4/5 -> 8:cccccccc::::head MAX 2024-09-17T18:22:51.301 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (3997 ms) 2024-09-17T18:22:51.301 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: [----------] 7 tests from LibRadosList (164065 ms total) 2024-09-17T18:22:51.301 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: 2024-09-17T18:22:51.302 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-09-17T18:22:51.302 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-09-17T18:22:51.302 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1036 ms) 2024-09-17T18:22:51.302 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-09-17T18:22:51.302 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: myset foo1,foo2,foo3 2024-09-17T18:22:51.302 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: foo1 2024-09-17T18:22:51.302 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: foo2 2024-09-17T18:22:51.302 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: foo3 2024-09-17T18:22:51.302 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: myset foo1,foo4,foo5 2024-09-17T18:22:51.302 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: foo4 2024-09-17T18:22:51.302 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: foo5 2024-09-17T18:22:51.302 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: foo1 2024-09-17T18:22:51.302 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: myset foo6,foo7 2024-09-17T18:22:51.302 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: foo7 2024-09-17T18:22:51.303 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: foo6 2024-09-17T18:22:51.303 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-09-17T18:22:51.303 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: ns1:foo4 2024-09-17T18:22:51.303 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: ns1:foo5 2024-09-17T18:22:51.303 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: ns2:foo7 2024-09-17T18:22:51.303 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: ns2:foo6 2024-09-17T18:22:51.303 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: ns1:foo1 2024-09-17T18:22:51.303 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: :foo1 2024-09-17T18:22:51.303 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: :foo2 2024-09-17T18:22:51.303 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: :foo3 2024-09-17T18:22:51.303 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (82 ms) 2024-09-17T18:22:51.303 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-09-17T18:22:51.303 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: 1 0 2024-09-17T18:22:51.304 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: 10 0 2024-09-17T18:22:51.304 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: 13 0 2024-09-17T18:22:54.248 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-17T18:22:54.249 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-17T18:22:54.249 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-17T18:22:54.252 INFO:tasks.ceph.mon.a:Started 2024-09-17T18:22:54.252 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-17T18:22:54.253 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-17T18:22:54.253 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-17T18:23:09.253 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-17T18:23:09.254 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-17T18:23:09.254 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T18:23:09.254 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:23:09.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.360+0000 7f392c887640 10 monclient: get_monmap_and_config 2024-09-17T18:23:09.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.360+0000 7f392c887640 10 monclient: build_initial_monmap 2024-09-17T18:23:09.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.360+0000 7f392c887640 10 monclient: monmap: 2024-09-17T18:23:09.362 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:23:09.362 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:09.362 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:23:09.360736+0000 2024-09-17T18:23:09.362 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:23:09.360736+0000 2024-09-17T18:23:09.362 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:23:09.362 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:09.362 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:23:09.362 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:23:09.362 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:23:09.363 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:23:09.363 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:23:09.363 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:09.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.360+0000 7f392c887640 1 Processor -- start 2024-09-17T18:23:09.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.360+0000 7f392c887640 1 -- start start 2024-09-17T18:23:09.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.360+0000 7f392c887640 10 monclient: init 2024-09-17T18:23:09.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.361+0000 7f392c887640 10 monclient: _reopen_session rank -1 2024-09-17T18:23:09.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.361+0000 7f392c887640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-17T18:23:09.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.361+0000 7f392c887640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f39280a8620 0x7f39280a5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:09.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.361+0000 7f392c887640 10 monclient(hunting): picked mon.noname-c con 0x7f39280a8620 addr v2:172.21.15.173:3300/0 2024-09-17T18:23:09.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.361+0000 7f392c887640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f39280a6350 addr v1:172.21.15.173:6789/0 2024-09-17T18:23:09.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.361+0000 7f392c887640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39280a6730 0x7f3928140320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:09.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.361+0000 7f392c887640 10 monclient(hunting): picked mon.noname-b con 0x7f39280a6730 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:23:09.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.361+0000 7f392c887640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f392809f0e0 con 0x7f39280a6350 2024-09-17T18:23:09.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.361+0000 7f392c887640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:09.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.361+0000 7f392c887640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f392809cf50 con 0x7f39280a8620 2024-09-17T18:23:09.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.361+0000 7f392c887640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:09.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.361+0000 7f392c887640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3928070900 con 0x7f39280a6730 2024-09-17T18:23:09.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.361+0000 7f392c887640 10 monclient(hunting): _renew_subs 2024-09-17T18:23:09.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.361+0000 7f392c887640 10 monclient(hunting): authenticate will time out at 993.077394s 2024-09-17T18:23:09.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.361+0000 7f3926d9d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39280a6730 0x7f3928140320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:09.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.361+0000 7f3926d9d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39280a6730 0x7f3928140320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:52644/0 (socket says 172.21.15.121:52644) 2024-09-17T18:23:09.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.361+0000 7f3926d9d640 1 -- 172.21.15.121:0/2475010090 learned_addr learned my addr 172.21.15.121:0/2475010090 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:23:09.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.361+0000 7f3926d9d640 10 monclient(hunting): get_auth_request con 0x7f39280a6730 auth_method 0 2024-09-17T18:23:09.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.361+0000 7f3926d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:09.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.361+0000 7f3926d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:09.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.361+0000 7f3926d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:09.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.361+0000 7f3926d9d640 1 -- 172.21.15.121:0/2475010090 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39280a6730 msgr2=0x7f3928140320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T18:23:09.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.361+0000 7f3926d9d640 1 -- 172.21.15.121:0/2475010090 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39280a6730 msgr2=0x7f3928140320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:23:09.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.361+0000 7f3926d9d640 1 --2- 172.21.15.121:0/2475010090 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39280a6730 0x7f3928140320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:23:09.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.361+0000 7f3926d9d640 1 --2- 172.21.15.121:0/2475010090 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39280a6730 0x7f3928140320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:23:09.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.361+0000 7f392659c640 1 --2- 172.21.15.121:0/2475010090 >> v2:172.21.15.173:3300/0 conn(0x7f39280a8620 0x7f39280a5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:09.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.361+0000 7f392659c640 10 monclient(hunting): get_auth_request con 0x7f39280a8620 auth_method 0 2024-09-17T18:23:09.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.361+0000 7f392659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:09.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.361+0000 7f392659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:09.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.361+0000 7f392659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:09.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.361+0000 7f392659c640 1 -- 172.21.15.121:0/2475010090 >> v2:172.21.15.173:3300/0 conn(0x7f39280a8620 msgr2=0x7f39280a5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:23:09.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.361+0000 7f392659c640 1 -- 172.21.15.121:0/2475010090 >> v2:172.21.15.173:3300/0 conn(0x7f39280a8620 msgr2=0x7f39280a5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:23:09.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.361+0000 7f392659c640 1 --2- 172.21.15.121:0/2475010090 >> v2:172.21.15.173:3300/0 conn(0x7f39280a8620 0x7f39280a5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:23:09.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.361+0000 7f392659c640 1 --2- 172.21.15.121:0/2475010090 >> v2:172.21.15.173:3300/0 conn(0x7f39280a8620 0x7f39280a5e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:23:09.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.363+0000 7f391dd9b640 1 -- 172.21.15.121:0/2475010090 >> v1:172.21.15.173:6789/0 conn(0x7f39280a6350 legacy=0x7f392813cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:09.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.562+0000 7f3926d9d640 1 --2- 172.21.15.121:0/2475010090 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39280a6730 0x7f3928140320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:09.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.562+0000 7f3926d9d640 10 monclient(hunting): get_auth_request con 0x7f39280a6730 auth_method 0 2024-09-17T18:23:09.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.562+0000 7f3926d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:09.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.562+0000 7f3926d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:09.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.562+0000 7f3926d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:23:09.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.562+0000 7f3926d9d640 1 -- 172.21.15.121:0/2475010090 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39280a6730 msgr2=0x7f3928140320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:23:09.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.562+0000 7f3926d9d640 1 -- 172.21.15.121:0/2475010090 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39280a6730 msgr2=0x7f3928140320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:23:09.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.562+0000 7f3926d9d640 1 --2- 172.21.15.121:0/2475010090 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39280a6730 0x7f3928140320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:23:09.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.562+0000 7f3926d9d640 1 --2- 172.21.15.121:0/2475010090 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39280a6730 0x7f3928140320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T18:23:09.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.562+0000 7f392659c640 1 --2- 172.21.15.121:0/2475010090 >> v2:172.21.15.173:3300/0 conn(0x7f39280a8620 0x7f39280a5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:09.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.562+0000 7f392659c640 10 monclient(hunting): get_auth_request con 0x7f39280a8620 auth_method 0 2024-09-17T18:23:09.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.562+0000 7f392659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:09.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.562+0000 7f392659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:09.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.562+0000 7f392659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:23:09.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.562+0000 7f392659c640 1 -- 172.21.15.121:0/2475010090 >> v2:172.21.15.173:3300/0 conn(0x7f39280a8620 msgr2=0x7f39280a5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T18:23:09.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.562+0000 7f392659c640 1 -- 172.21.15.121:0/2475010090 >> v2:172.21.15.173:3300/0 conn(0x7f39280a8620 msgr2=0x7f39280a5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:23:09.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.562+0000 7f392659c640 1 --2- 172.21.15.121:0/2475010090 >> v2:172.21.15.173:3300/0 conn(0x7f39280a8620 0x7f39280a5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:23:09.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.562+0000 7f392659c640 1 --2- 172.21.15.121:0/2475010090 >> v2:172.21.15.173:3300/0 conn(0x7f39280a8620 0x7f39280a5e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T18:23:09.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.671+0000 7f3925d9b640 1 -- 172.21.15.121:0/2475010090 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2701257308 0 0) 0x7f392809f0e0 con 0x7f39280a6350 2024-09-17T18:23:09.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.671+0000 7f3925d9b640 10 monclient(hunting): _negotiate 2024-09-17T18:23:09.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.671+0000 7f3925d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:09.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.671+0000 7f3925d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:09.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.671+0000 7f3925d9b640 10 monclient(hunting): my global_id is 24475 2024-09-17T18:23:09.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.671+0000 7f3925d9b640 1 -- 172.21.15.121:0/2475010090 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3914003520 con 0x7f39280a6350 2024-09-17T18:23:09.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f3925d9b640 1 -- 172.21.15.121:0/2475010090 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 394619598 0 0) 0x7f3914003520 con 0x7f39280a6350 2024-09-17T18:23:09.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f3925d9b640 1 -- 172.21.15.121:0/2475010090 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39280a6730 msgr2=0x7f3928140320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:23:09.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f3925d9b640 1 --2- 172.21.15.121:0/2475010090 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39280a6730 0x7f3928140320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:09.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f3925d9b640 1 -- 172.21.15.121:0/2475010090 >> v2:172.21.15.173:3300/0 conn(0x7f39280a8620 msgr2=0x7f39280a5e10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:23:09.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f3925d9b640 1 --2- 172.21.15.121:0/2475010090 >> v2:172.21.15.173:3300/0 conn(0x7f39280a8620 0x7f39280a5e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:09.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f3925d9b640 10 monclient: _finish_hunting 0 2024-09-17T18:23:09.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f3925d9b640 1 monclient: found mon.noname-c-legacy 2024-09-17T18:23:09.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f3925d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:23:09.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f3925d9b640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.173:6789/0 2024-09-17T18:23:09.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f3925d9b640 1 -- 172.21.15.121:0/2475010090 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f392809cc70 con 0x7f39280a6350 2024-09-17T18:23:09.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f3925d9b640 10 monclient: _finish_auth 0 2024-09-17T18:23:09.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f3925d9b640 10 monclient: _check_auth_tickets 2024-09-17T18:23:09.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f3925d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:23:09.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f3925d9b640 1 -- 172.21.15.121:0/2475010090 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4002019445 0 0) 0x7f39100019e0 con 0x7f39280a6350 2024-09-17T18:23:09.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f3925d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:09.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f3925d9b640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-17T18:23:09.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f3925d9b640 10 monclient: dump: 2024-09-17T18:23:09.677 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:09.677 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:09.677 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:09.677 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:09.677 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:09.677 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:09.677 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:09.677 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:09.677 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:09.678 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:09.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f3925d9b640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] but i'm connected to v1:172.21.15.173:6789/0, reconnecting 2024-09-17T18:23:09.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f3925d9b640 10 monclient: _reopen_session rank -1 2024-09-17T18:23:09.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f3925d9b640 1 -- 172.21.15.121:0/2475010090 >> v1:172.21.15.173:6789/0 conn(0x7f39280a6350 legacy=0x7f392813cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:09.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f3925d9b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:23:09.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f3925d9b640 1 --2- 172.21.15.121:0/2475010090 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3914003a80 0x7f3914003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:09.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f3925d9b640 10 monclient(hunting): picked mon.a con 0x7f3914003a80 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:23:09.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f3925d9b640 1 --2- 172.21.15.121:0/2475010090 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3914004800 0x7f3914004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:09.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f3925d9b640 10 monclient(hunting): picked mon.b con 0x7f3914004800 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:23:09.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f3925d9b640 1 --2- 172.21.15.121:0/2475010090 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39280a6730 0x7f3914005270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:09.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f3925d9b640 10 monclient(hunting): picked mon.c con 0x7f39280a6730 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:23:09.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f3925d9b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:09.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f3925d9b640 1 -- 172.21.15.121:0/2475010090 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f392809cf50 con 0x7f3914003a80 2024-09-17T18:23:09.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f3925d9b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:09.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f3925d9b640 1 -- 172.21.15.121:0/2475010090 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3928070900 con 0x7f3914004800 2024-09-17T18:23:09.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f3925d9b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:09.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f3925d9b640 1 -- 172.21.15.121:0/2475010090 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f391400bc30 con 0x7f39280a6730 2024-09-17T18:23:09.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f3925d9b640 10 monclient(hunting): _renew_subs 2024-09-17T18:23:09.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f3925d9b640 10 monclient(hunting): _finish_auth 0 2024-09-17T18:23:09.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f392659c640 1 --2- 172.21.15.121:0/2475010090 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3914003a80 0x7f3914003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:09.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f392659c640 10 monclient(hunting): get_auth_request con 0x7f3914003a80 auth_method 0 2024-09-17T18:23:09.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f392659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:09.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f392659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:09.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f392659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:23:09.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f391dd9b640 1 --2- 172.21.15.121:0/2475010090 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3914004800 0x7f3914004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:09.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f391dd9b640 10 monclient(hunting): get_auth_request con 0x7f3914004800 auth_method 0 2024-09-17T18:23:09.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f391dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:09.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f391dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:09.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.672+0000 7f391dd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:23:09.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.673+0000 7f3926d9d640 1 --2- 172.21.15.121:0/2475010090 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39280a6730 0x7f3914005270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:09.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.673+0000 7f392659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:09.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.673+0000 7f392659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:09.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.673+0000 7f392659c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:23:09.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.673+0000 7f3926d9d640 10 monclient(hunting): get_auth_request con 0x7f39280a6730 auth_method 0 2024-09-17T18:23:09.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.673+0000 7f3926d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:09.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.673+0000 7f3926d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:09.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.673+0000 7f3926d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:23:09.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.673+0000 7f391dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:09.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.673+0000 7f391dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:09.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.673+0000 7f391dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:23:09.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.673+0000 7f3926d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:09.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.673+0000 7f3926d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:09.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.673+0000 7f3926d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:23:09.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.673+0000 7f391dd9b640 10 monclient(hunting): handle_auth_done global_id 24475 payload 293 2024-09-17T18:23:09.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.673+0000 7f391dd9b640 1 -- 172.21.15.121:0/2475010090 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39280a6730 msgr2=0x7f3914005270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:23:09.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.673+0000 7f391dd9b640 1 --2- 172.21.15.121:0/2475010090 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39280a6730 0x7f3914005270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:09.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.673+0000 7f391dd9b640 1 -- 172.21.15.121:0/2475010090 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3914003a80 msgr2=0x7f3914003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:23:09.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.673+0000 7f391dd9b640 1 --2- 172.21.15.121:0/2475010090 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3914003a80 0x7f3914003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:09.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.673+0000 7f391dd9b640 10 monclient: _finish_hunting 0 2024-09-17T18:23:09.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.673+0000 7f391dd9b640 1 monclient: found mon.b 2024-09-17T18:23:09.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.673+0000 7f391dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:23:09.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.673+0000 7f391dd9b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:23:09.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.673+0000 7f391dd9b640 1 -- 172.21.15.121:0/2475010090 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f392809cc70 con 0x7f3914004800 2024-09-17T18:23:09.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.673+0000 7f391dd9b640 10 monclient: _finish_auth 0 2024-09-17T18:23:09.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.673+0000 7f391dd9b640 10 monclient: _check_auth_tickets 2024-09-17T18:23:09.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.673+0000 7f391dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:23:09.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.673+0000 7f392c887640 5 monclient: authenticate success, global_id 24475 2024-09-17T18:23:09.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.673+0000 7f392c887640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:23:09.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.673+0000 7f392659c640 1 --2- 172.21.15.121:0/2475010090 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3914003a80 0x7f3914003ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:23:09.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.676+0000 7f391dd9b640 1 -- 172.21.15.121:0/2475010090 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3914004800 msgr2=0x7f3914004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:09.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.676+0000 7f391dd9b640 1 --2- 172.21.15.121:0/2475010090 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3914004800 0x7f3914004c80 secure :-1 s=READY pgs=7267 cs=0 l=1 rev1=1 crypto rx=0x7f3910001eb0 tx=0x7f3914005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a84c3c1928cce40c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:09.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.678+0000 7f3925d9b640 1 -- 172.21.15.121:0/2475010090 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3910036070 con 0x7f3914004800 2024-09-17T18:23:09.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.678+0000 7f3925d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:09.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.678+0000 7f3925d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:23:09.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.678+0000 7f3925d9b640 10 monclient: dump: 2024-09-17T18:23:09.688 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:09.689 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:09.689 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:09.689 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:09.689 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:09.689 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:09.689 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:09.689 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:09.689 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:09.689 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:09.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.678+0000 7f3925d9b640 1 -- 172.21.15.121:0/2475010090 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3910007290 con 0x7f3914004800 2024-09-17T18:23:09.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.678+0000 7f3925d9b640 10 monclient: handle_config config(0 keys) 2024-09-17T18:23:09.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.678+0000 7f3925d9b640 1 -- 172.21.15.121:0/2475010090 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f391000c0e0 con 0x7f3914004800 2024-09-17T18:23:09.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.678+0000 7f392c887640 10 monclient: get_monmap_and_config success 2024-09-17T18:23:09.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.678+0000 7f392c887640 4 set_mon_vals no callback set 2024-09-17T18:23:09.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.678+0000 7f3925d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:09.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.678+0000 7f3925d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:23:09.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.678+0000 7f3925d9b640 10 monclient: dump: 2024-09-17T18:23:09.690 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:09.690 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:09.690 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:09.690 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:09.690 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:09.690 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:09.690 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:09.690 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:09.690 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:09.690 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:09.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.679+0000 7f392c887640 10 monclient: shutdown 2024-09-17T18:23:09.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.679+0000 7f392c887640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:23:09.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.679+0000 7f392c887640 1 -- 172.21.15.121:0/2475010090 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3914004800 msgr2=0x7f3914004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:09.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.679+0000 7f392c887640 1 --2- 172.21.15.121:0/2475010090 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3914004800 0x7f3914004c80 secure :-1 s=READY pgs=7267 cs=0 l=1 rev1=1 crypto rx=0x7f3910001eb0 tx=0x7f3914005830 comp rx=0 tx=0).stop 2024-09-17T18:23:09.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.679+0000 7f3926d9d640 1 -- 172.21.15.121:0/2475010090 reap_dead start 2024-09-17T18:23:09.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.679+0000 7f392c887640 1 -- 172.21.15.121:0/2475010090 shutdown_connections 2024-09-17T18:23:09.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.679+0000 7f392c887640 1 -- 172.21.15.121:0/2475010090 >> 172.21.15.121:0/2475010090 conn(0x7f39280abcb0 msgr2=0x7f39280a7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:23:09.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.679+0000 7f392c887640 1 -- 172.21.15.121:0/2475010090 shutdown_connections 2024-09-17T18:23:09.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.679+0000 7f392c887640 1 -- 172.21.15.121:0/2475010090 wait complete. 2024-09-17T18:23:09.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.680+0000 7f392c887640 10 monclient: build_initial_monmap 2024-09-17T18:23:09.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.680+0000 7f392c887640 10 monclient: monmap: 2024-09-17T18:23:09.691 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:23:09.691 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:23:09.692 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:23:09.692 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:23:09.692 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:23:09.692 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:09.692 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:23:09.692 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:23:09.692 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:23:09.692 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:09.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.680+0000 7f392c887640 1 librados: starting msgr at 2024-09-17T18:23:09.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.680+0000 7f392c887640 1 librados: starting objecter 2024-09-17T18:23:09.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.680+0000 7f392c887640 1 Processor -- start 2024-09-17T18:23:09.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.680+0000 7f392c887640 1 -- start start 2024-09-17T18:23:09.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.680+0000 7f392c887640 1 librados: setting wanted keys 2024-09-17T18:23:09.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.680+0000 7f392c887640 1 librados: calling monclient init 2024-09-17T18:23:09.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.680+0000 7f392c887640 10 monclient: init 2024-09-17T18:23:09.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f392c887640 10 monclient: _reopen_session rank -1 2024-09-17T18:23:09.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f392c887640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T18:23:09.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f392c887640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3914004800 0x7f39281ad4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:09.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f392c887640 10 monclient(hunting): picked mon.noname-c con 0x7f3914004800 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:23:09.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f392c887640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f39281cd8b0 0x7f39281c7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:09.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f392c887640 10 monclient(hunting): picked mon.noname-b con 0x7f39281cd8b0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:23:09.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f392c887640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39281c77d0 0x7f39281c9e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:09.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f392c887640 10 monclient(hunting): picked mon.noname-a con 0x7f39281c77d0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:23:09.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f392c887640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:09.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f392c887640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f39280a6b80 con 0x7f39281c77d0 2024-09-17T18:23:09.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f392c887640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:09.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f392c887640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f392809d930 con 0x7f39281cd8b0 2024-09-17T18:23:09.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f392c887640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:09.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f392c887640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f39280a7490 con 0x7f3914004800 2024-09-17T18:23:09.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f392c887640 10 monclient(hunting): _renew_subs 2024-09-17T18:23:09.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f392c887640 10 monclient(hunting): authenticate will time out at 993.397308s 2024-09-17T18:23:09.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f392659c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3914004800 0x7f39281ad4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:09.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f3926d9d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39281c77d0 0x7f39281c9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:09.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f392659c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3914004800 0x7f39281ad4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:52684/0 (socket says 172.21.15.121:52684) 2024-09-17T18:23:09.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f392659c640 1 -- 172.21.15.121:0/1004994212 learned_addr learned my addr 172.21.15.121:0/1004994212 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:23:09.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f392659c640 10 monclient(hunting): get_auth_request con 0x7f3914004800 auth_method 0 2024-09-17T18:23:09.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f392659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:09.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f392659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:09.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f392659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:09.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f3926d9d640 10 monclient(hunting): get_auth_request con 0x7f39281c77d0 auth_method 0 2024-09-17T18:23:09.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f3926d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:09.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f3926d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:09.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f3926d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:09.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f392659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:09.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f392659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:09.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f392659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:09.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f3926d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:09.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f3926d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:09.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f391dd9b640 1 --2- 172.21.15.121:0/1004994212 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f39281cd8b0 0x7f39281c7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:09.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f3926d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:09.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f391dd9b640 10 monclient(hunting): get_auth_request con 0x7f39281cd8b0 auth_method 0 2024-09-17T18:23:09.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f391dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:09.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f391dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:09.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f391dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:09.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f391dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:09.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f391dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:09.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f391dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:09.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f3926d9d640 10 monclient(hunting): handle_auth_done global_id 44097 payload 824 2024-09-17T18:23:09.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f3926d9d640 1 -- 172.21.15.121:0/1004994212 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3914004800 msgr2=0x7f39281ad4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:09.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f3926d9d640 1 --2- 172.21.15.121:0/1004994212 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3914004800 0x7f39281ad4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:09.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f3926d9d640 1 -- 172.21.15.121:0/1004994212 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f39281cd8b0 msgr2=0x7f39281c7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:09.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f3926d9d640 1 --2- 172.21.15.121:0/1004994212 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f39281cd8b0 0x7f39281c7290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:09.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f3926d9d640 10 monclient: _finish_hunting 0 2024-09-17T18:23:09.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f3926d9d640 1 monclient: found mon.noname-a 2024-09-17T18:23:09.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f3926d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:23:09.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f3926d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:23:09.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f3926d9d640 1 -- 172.21.15.121:0/1004994212 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39281ade80 con 0x7f39281c77d0 2024-09-17T18:23:09.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.681+0000 7f392659c640 1 --2- 172.21.15.121:0/1004994212 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3914004800 0x7f39281ad4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:23:09.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.684+0000 7f3926d9d640 1 -- 172.21.15.121:0/1004994212 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39281c77d0 msgr2=0x7f39281c9e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:09.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.684+0000 7f3926d9d640 1 --2- 172.21.15.121:0/1004994212 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39281c77d0 0x7f39281c9e70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3928076f20 tx=0x7f3920010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:09.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.684+0000 7f391f7fe640 10 client.?.objecter ms_handle_connect 0x7f39281c77d0 2024-09-17T18:23:09.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.684+0000 7f391f7fe640 10 client.?.objecter resend_mon_ops 2024-09-17T18:23:09.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.684+0000 7f391f7fe640 1 -- 172.21.15.121:0/1004994212 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f391800d690 con 0x7f39281c77d0 2024-09-17T18:23:09.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.684+0000 7f391f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:09.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.684+0000 7f391f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:23:09.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.684+0000 7f391f7fe640 10 monclient: dump: 2024-09-17T18:23:09.697 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:09.697 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:09.697 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:09.698 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:09.698 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:09.698 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:09.698 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:09.698 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:09.698 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:09.698 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:09.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.684+0000 7f391f7fe640 10 monclient: _finish_auth 0 2024-09-17T18:23:09.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.684+0000 7f391f7fe640 10 monclient: _check_auth_tickets 2024-09-17T18:23:09.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.684+0000 7f391f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:23:09.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.684+0000 7f392c887640 5 monclient: authenticate success, global_id 44097 2024-09-17T18:23:09.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.684+0000 7f392c887640 10 monclient: _renew_subs 2024-09-17T18:23:09.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.684+0000 7f392c887640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:23:09.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.684+0000 7f392c887640 1 -- 172.21.15.121:0/1004994212 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39280a2600 con 0x7f39281c77d0 2024-09-17T18:23:09.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.684+0000 7f391f7fe640 1 -- 172.21.15.121:0/1004994212 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f392001a070 con 0x7f39281c77d0 2024-09-17T18:23:09.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.684+0000 7f391f7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:23:09.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.684+0000 7f391f7fe640 1 -- 172.21.15.121:0/1004994212 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39200156c0 con 0x7f39281c77d0 2024-09-17T18:23:09.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.684+0000 7f391f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:09.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.684+0000 7f391f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:23:09.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.684+0000 7f391f7fe640 10 monclient: dump: 2024-09-17T18:23:09.699 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:09.699 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:09.699 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:09.699 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:09.699 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:09.699 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:09.699 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:09.699 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:09.700 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:09.700 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:09.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.684+0000 7f392559a640 4 set_mon_vals no callback set 2024-09-17T18:23:09.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.684+0000 7f392c887640 10 client.44097.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:23:09.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.684+0000 7f392c887640 10 monclient: _renew_subs 2024-09-17T18:23:09.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.684+0000 7f392c887640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:23:09.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.684+0000 7f392c887640 1 -- 172.21.15.121:0/1004994212 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39281aecf0 con 0x7f39281c77d0 2024-09-17T18:23:09.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.684+0000 7f392c887640 1 librados: init done 2024-09-17T18:23:09.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.685+0000 7f391f7fe640 1 -- 172.21.15.121:0/1004994212 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f39200158a0 con 0x7f39281c77d0 2024-09-17T18:23:09.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.685+0000 7f392c887640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:23:09.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.685+0000 7f392c887640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:23:09.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.685+0000 7f392c887640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:23:09.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.685+0000 7f392c887640 1 -- 172.21.15.121:0/1004994212 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39281cdc90 con 0x7f39281c77d0 2024-09-17T18:23:09.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.685+0000 7f391f7fe640 1 --2- 172.21.15.121:0/1004994212 >> v2:172.21.15.173:6812/2423786037 conn(0x7f39080401a0 0x7f3908042660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:09.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.685+0000 7f391f7fe640 1 -- 172.21.15.121:0/1004994212 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(175..175 src has 1..175) ==== 6557+0+0 (secure 0 0 0) 0x7f3920054010 con 0x7f39281c77d0 2024-09-17T18:23:09.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.685+0000 7f391f7fe640 10 client.44097.objecter ms_dispatch 0x7f3928000b90 osd_map(175..175 src has 1..175) 2024-09-17T18:23:09.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.685+0000 7f391f7fe640 3 client.44097.objecter handle_osd_map got epochs [175,175] > 0 2024-09-17T18:23:09.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.685+0000 7f391f7fe640 3 client.44097.objecter handle_osd_map decoding full epoch 175 2024-09-17T18:23:09.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.685+0000 7f391f7fe640 20 client.44097.objecter dump_active .. 0 homeless 2024-09-17T18:23:09.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.685+0000 7f392659c640 1 --2- 172.21.15.121:0/1004994212 >> v2:172.21.15.173:6812/2423786037 conn(0x7f39080401a0 0x7f3908042660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:09.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.685+0000 7f392659c640 10 monclient: get_auth_request con 0x7f39080401a0 auth_method 0 2024-09-17T18:23:09.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.688+0000 7f392659c640 1 --2- 172.21.15.121:0/1004994212 >> v2:172.21.15.173:6812/2423786037 conn(0x7f39080401a0 0x7f3908042660 secure :-1 s=READY pgs=3928 cs=0 l=1 rev1=1 crypto rx=0x7f391800dcc0 tx=0x7f3918008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:09.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.688+0000 7f391f7fe640 10 client.44097.objecter ms_handle_connect 0x7f39080401a0 2024-09-17T18:23:09.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.690+0000 7f391f7fe640 1 -- 172.21.15.121:0/1004994212 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f39281cdc90 con 0x7f39281c77d0 2024-09-17T18:23:09.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.690+0000 7f391f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:23:09.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.690+0000 7f391f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:23:09.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.938+0000 7f392c887640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:23:09.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.938+0000 7f392c887640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:23:09.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.938+0000 7f392c887640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:23:09.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.938+0000 7f392c887640 1 -- 172.21.15.121:0/1004994212 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f39281ae970 con 0x7f39281c77d0 2024-09-17T18:23:09.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.938+0000 7f391f7fe640 1 -- 172.21.15.121:0/1004994212 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f39281ae970 con 0x7f39281c77d0 2024-09-17T18:23:09.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.938+0000 7f391f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:23:09.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.938+0000 7f391f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:23:09.941 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:23:09.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.940+0000 7f392c887640 10 librados: watch_flush enter 2024-09-17T18:23:09.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.940+0000 7f392c887640 10 librados: watch_flush exit 2024-09-17T18:23:09.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.940+0000 7f392c887640 20 client.44097.objecter shutdown clearing up homeless session... 2024-09-17T18:23:09.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.940+0000 7f392c887640 10 client.44097.objecter successfully canceled tick 2024-09-17T18:23:09.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.940+0000 7f392c887640 1 -- 172.21.15.121:0/1004994212 >> v2:172.21.15.173:6812/2423786037 conn(0x7f39080401a0 msgr2=0x7f3908042660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:09.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.940+0000 7f392c887640 1 --2- 172.21.15.121:0/1004994212 >> v2:172.21.15.173:6812/2423786037 conn(0x7f39080401a0 0x7f3908042660 secure :-1 s=READY pgs=3928 cs=0 l=1 rev1=1 crypto rx=0x7f391800dcc0 tx=0x7f3918008040 comp rx=0 tx=0).stop 2024-09-17T18:23:09.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.940+0000 7f392c887640 10 monclient: shutdown 2024-09-17T18:23:09.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.940+0000 7f392c887640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:23:09.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.940+0000 7f392c887640 1 -- 172.21.15.121:0/1004994212 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39281c77d0 msgr2=0x7f39281c9e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:09.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.940+0000 7f392c887640 1 --2- 172.21.15.121:0/1004994212 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39281c77d0 0x7f39281c9e70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3928076f20 tx=0x7f3920010590 comp rx=0 tx=0).stop 2024-09-17T18:23:09.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.941+0000 7f392c887640 1 -- 172.21.15.121:0/1004994212 shutdown_connections 2024-09-17T18:23:09.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.941+0000 7f392c887640 1 --2- 172.21.15.121:0/1004994212 >> v2:172.21.15.173:6812/2423786037 conn(0x7f39080401a0 0x7f3908042660 unknown :-1 s=CLOSED pgs=3928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:09.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.941+0000 7f392c887640 1 --2- 172.21.15.121:0/1004994212 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f39281cd8b0 0x7f39281c7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:09.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.941+0000 7f392c887640 1 --2- 172.21.15.121:0/1004994212 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39281c77d0 0x7f39281c9e70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:09.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.941+0000 7f392c887640 1 --2- 172.21.15.121:0/1004994212 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3914004800 0x7f39281ad4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:09.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.941+0000 7f392c887640 1 -- 172.21.15.121:0/1004994212 >> 172.21.15.121:0/1004994212 conn(0x7f39280abcb0 msgr2=0x7f39280a1cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:23:09.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.941+0000 7f392c887640 1 -- 172.21.15.121:0/1004994212 shutdown_connections 2024-09-17T18:23:09.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.941+0000 7f392c887640 1 -- 172.21.15.121:0/1004994212 wait complete. 2024-09-17T18:23:09.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:09.941+0000 7f392c887640 1 librados: shutdown 2024-09-17T18:23:09.953 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T18:23:09.953 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T18:23:09.953 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T18:23:10.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.048+0000 7feebde85640 10 monclient: get_monmap_and_config 2024-09-17T18:23:10.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.048+0000 7feebde85640 10 monclient: build_initial_monmap 2024-09-17T18:23:10.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.049+0000 7feebde85640 10 monclient: monmap: 2024-09-17T18:23:10.051 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:23:10.051 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:10.051 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:23:10.049596+0000 2024-09-17T18:23:10.051 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:23:10.049596+0000 2024-09-17T18:23:10.051 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:23:10.051 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:10.051 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:23:10.051 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:23:10.051 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:23:10.051 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:23:10.052 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:23:10.052 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:10.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.049+0000 7feebde85640 1 Processor -- start 2024-09-17T18:23:10.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.049+0000 7feebde85640 1 -- start start 2024-09-17T18:23:10.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.049+0000 7feebde85640 10 monclient: init 2024-09-17T18:23:10.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.050+0000 7feebde85640 10 monclient: _reopen_session rank -1 2024-09-17T18:23:10.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.050+0000 7feebde85640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-17T18:23:10.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.050+0000 7feebde85640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feeb809e550 0x7feeb809e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:10.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.050+0000 7feebde85640 10 monclient(hunting): picked mon.noname-b con 0x7feeb809e550 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:23:10.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.050+0000 7feebde85640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7feeb809ee70 0x7feeb8139840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:10.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.050+0000 7feebde85640 10 monclient(hunting): picked mon.noname-c con 0x7feeb809ee70 addr v2:172.21.15.173:3300/0 2024-09-17T18:23:10.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.050+0000 7feebde85640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feeb809f290 addr v1:172.21.15.121:6789/0 2024-09-17T18:23:10.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.050+0000 7feebde85640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feeb80a4460 con 0x7feeb809f290 2024-09-17T18:23:10.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.050+0000 7feebde85640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:10.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.050+0000 7feebde85640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7feeb80a4950 con 0x7feeb809ee70 2024-09-17T18:23:10.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.050+0000 7feebde85640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:10.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.050+0000 7feebde85640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7feeb80a2b60 con 0x7feeb809e550 2024-09-17T18:23:10.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.050+0000 7feebde85640 10 monclient(hunting): _renew_subs 2024-09-17T18:23:10.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.050+0000 7feebde85640 10 monclient(hunting): authenticate will time out at 993.766331s 2024-09-17T18:23:10.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.050+0000 7feeb77fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feeb809e550 0x7feeb809e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:10.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.050+0000 7feeb77fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feeb809e550 0x7feeb809e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:52700/0 (socket says 172.21.15.121:52700) 2024-09-17T18:23:10.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.050+0000 7feeb7fff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7feeb809f290 0x7feeb813bd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:50806/0 (socket says 172.21.15.121:50806) 2024-09-17T18:23:10.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.050+0000 7feeb77fe640 1 -- 172.21.15.121:0/583669139 learned_addr learned my addr 172.21.15.121:0/583669139 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:23:10.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.050+0000 7feeb77fe640 10 monclient(hunting): get_auth_request con 0x7feeb809e550 auth_method 0 2024-09-17T18:23:10.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.050+0000 7feeb77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:10.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.050+0000 7feeb77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:10.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.050+0000 7feeb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:10.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.050+0000 7feeb77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:10.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.050+0000 7feeb77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:10.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.050+0000 7feeb77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:10.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.050+0000 7feeaeffd640 1 --2- 172.21.15.121:0/583669139 >> v2:172.21.15.173:3300/0 conn(0x7feeb809ee70 0x7feeb8139840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:10.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.050+0000 7feeaeffd640 10 monclient(hunting): get_auth_request con 0x7feeb809ee70 auth_method 0 2024-09-17T18:23:10.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.050+0000 7feeaeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:10.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.050+0000 7feeaeffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:10.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.050+0000 7feeaeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:10.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.050+0000 7feeb77fe640 10 monclient(hunting): handle_auth_done global_id 34430 payload 274 2024-09-17T18:23:10.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.050+0000 7feeb77fe640 1 -- 172.21.15.121:0/583669139 >> v2:172.21.15.173:3300/0 conn(0x7feeb809ee70 msgr2=0x7feeb8139840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:23:10.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.050+0000 7feeb77fe640 1 --2- 172.21.15.121:0/583669139 >> v2:172.21.15.173:3300/0 conn(0x7feeb809ee70 0x7feeb8139840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:10.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.050+0000 7feeb77fe640 1 -- 172.21.15.121:0/583669139 >> v1:172.21.15.121:6789/0 conn(0x7feeb809f290 legacy=0x7feeb813bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:23:10.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.050+0000 7feeb77fe640 10 monclient: _finish_hunting 0 2024-09-17T18:23:10.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.050+0000 7feeb77fe640 1 monclient: found mon.noname-b 2024-09-17T18:23:10.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.050+0000 7feeb77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:23:10.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.050+0000 7feeb77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:23:10.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.050+0000 7feeb77fe640 1 -- 172.21.15.121:0/583669139 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feeb809f840 con 0x7feeb809e550 2024-09-17T18:23:10.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.050+0000 7feeaeffd640 1 --2- 172.21.15.121:0/583669139 >> v2:172.21.15.173:3300/0 conn(0x7feeb809ee70 0x7feeb8139840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T18:23:10.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.053+0000 7feeb77fe640 1 -- 172.21.15.121:0/583669139 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feeb809e550 msgr2=0x7feeb809e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:10.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.053+0000 7feeb77fe640 1 --2- 172.21.15.121:0/583669139 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feeb809e550 0x7feeb809e930 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7feeb809e1a0 tx=0x7feea802c720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9280539383df04f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:10.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.053+0000 7feeb6ffd640 1 -- 172.21.15.121:0/583669139 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea8031920 con 0x7feeb809e550 2024-09-17T18:23:10.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.053+0000 7feeb6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:10.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.053+0000 7feeb6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:23:10.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.053+0000 7feeb6ffd640 10 monclient: dump: 2024-09-17T18:23:10.066 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:10.066 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:10.066 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.066 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.066 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:10.066 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:10.066 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:10.066 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:10.066 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:10.066 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:10.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.053+0000 7feeb6ffd640 10 monclient: _finish_auth 0 2024-09-17T18:23:10.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.053+0000 7feeb6ffd640 10 monclient: _check_auth_tickets 2024-09-17T18:23:10.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.053+0000 7feeb6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:23:10.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.053+0000 7feebde85640 5 monclient: authenticate success, global_id 34430 2024-09-17T18:23:10.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.053+0000 7feebde85640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:23:10.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.053+0000 7feeb6ffd640 1 -- 172.21.15.121:0/583669139 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feea8031ac0 con 0x7feeb809e550 2024-09-17T18:23:10.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.053+0000 7feeb6ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:23:10.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.053+0000 7feebde85640 10 monclient: get_monmap_and_config success 2024-09-17T18:23:10.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.053+0000 7feeb6ffd640 1 -- 172.21.15.121:0/583669139 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea8033400 con 0x7feeb809e550 2024-09-17T18:23:10.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.053+0000 7feebde85640 4 set_mon_vals no callback set 2024-09-17T18:23:10.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.053+0000 7feeb6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:10.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.053+0000 7feeb6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:23:10.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.053+0000 7feeb6ffd640 10 monclient: dump: 2024-09-17T18:23:10.067 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:10.067 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:10.067 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.067 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.068 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:10.068 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:10.068 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:10.068 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:10.068 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:10.068 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:10.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.054+0000 7feebde85640 10 monclient: shutdown 2024-09-17T18:23:10.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.054+0000 7feebde85640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:23:10.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.054+0000 7feebde85640 1 -- 172.21.15.121:0/583669139 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feeb809e550 msgr2=0x7feeb809e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:10.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.054+0000 7feebde85640 1 --2- 172.21.15.121:0/583669139 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feeb809e550 0x7feeb809e930 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7feeb809e1a0 tx=0x7feea802c720 comp rx=0 tx=0).stop 2024-09-17T18:23:10.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.054+0000 7feebde85640 1 -- 172.21.15.121:0/583669139 shutdown_connections 2024-09-17T18:23:10.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.054+0000 7feebde85640 1 --2- 172.21.15.121:0/583669139 >> v2:172.21.15.173:3300/0 conn(0x7feeb809ee70 0x7feeb8139840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:10.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.054+0000 7feebde85640 1 --2- 172.21.15.121:0/583669139 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feeb809e550 0x7feeb809e930 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:10.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.054+0000 7feebde85640 1 -- 172.21.15.121:0/583669139 >> 172.21.15.121:0/583669139 conn(0x7feeb80a7830 msgr2=0x7feeb80a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:23:10.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.054+0000 7feebde85640 1 -- 172.21.15.121:0/583669139 shutdown_connections 2024-09-17T18:23:10.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.054+0000 7feebde85640 1 -- 172.21.15.121:0/583669139 wait complete. 2024-09-17T18:23:10.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.054+0000 7feebde85640 10 monclient: build_initial_monmap 2024-09-17T18:23:10.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.054+0000 7feebde85640 10 monclient: monmap: 2024-09-17T18:23:10.069 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:23:10.069 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:23:10.069 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:23:10.069 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:23:10.069 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:23:10.069 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:10.069 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:23:10.069 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:23:10.069 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:23:10.069 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:10.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.055+0000 7feebde85640 1 librados: starting msgr at 2024-09-17T18:23:10.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.055+0000 7feebde85640 1 librados: starting objecter 2024-09-17T18:23:10.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.055+0000 7feebde85640 1 Processor -- start 2024-09-17T18:23:10.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.055+0000 7feebde85640 1 -- start start 2024-09-17T18:23:10.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.055+0000 7feebde85640 1 librados: setting wanted keys 2024-09-17T18:23:10.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.055+0000 7feebde85640 1 librados: calling monclient init 2024-09-17T18:23:10.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.055+0000 7feebde85640 10 monclient: init 2024-09-17T18:23:10.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.055+0000 7feebde85640 10 monclient: _reopen_session rank -1 2024-09-17T18:23:10.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.055+0000 7feebde85640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T18:23:10.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.055+0000 7feebde85640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feeb809e550 0x7feeb81b9190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:10.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.055+0000 7feebde85640 10 monclient(hunting): picked mon.noname-a con 0x7feeb809e550 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:23:10.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.055+0000 7feebde85640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feeb809ee70 0x7feeb81a7a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:10.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.055+0000 7feebde85640 10 monclient(hunting): picked mon.noname-c con 0x7feeb809ee70 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:23:10.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.055+0000 7feebde85640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feeb809f290 0x7feeb81a7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:10.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.055+0000 7feebde85640 10 monclient(hunting): picked mon.noname-b con 0x7feeb809f290 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:23:10.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.055+0000 7feebde85640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:10.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.055+0000 7feebde85640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7feeb809de60 con 0x7feeb809e550 2024-09-17T18:23:10.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.055+0000 7feebde85640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:10.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.055+0000 7feebde85640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7feeb809dce0 con 0x7feeb809f290 2024-09-17T18:23:10.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.055+0000 7feebde85640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:10.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.055+0000 7feebde85640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7feeb80747e0 con 0x7feeb809ee70 2024-09-17T18:23:10.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.055+0000 7feebde85640 10 monclient(hunting): _renew_subs 2024-09-17T18:23:10.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.055+0000 7feebde85640 10 monclient(hunting): authenticate will time out at 993.771941s 2024-09-17T18:23:10.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.056+0000 7feeaeffd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feeb809ee70 0x7feeb81a7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:10.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.056+0000 7feeaeffd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feeb809ee70 0x7feeb81a7a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:52706/0 (socket says 172.21.15.121:52706) 2024-09-17T18:23:10.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.056+0000 7feeaeffd640 1 -- 172.21.15.121:0/62158345 learned_addr learned my addr 172.21.15.121:0/62158345 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:23:10.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.056+0000 7feeaeffd640 10 monclient(hunting): get_auth_request con 0x7feeb809ee70 auth_method 0 2024-09-17T18:23:10.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.056+0000 7feeaeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:10.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.056+0000 7feeaeffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:10.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.056+0000 7feeaeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:10.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.056+0000 7feeb77fe640 1 --2- 172.21.15.121:0/62158345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feeb809e550 0x7feeb81b9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:10.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.056+0000 7feeb77fe640 10 monclient(hunting): get_auth_request con 0x7feeb809e550 auth_method 0 2024-09-17T18:23:10.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.056+0000 7feeb77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:10.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.056+0000 7feeb77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:10.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.056+0000 7feeb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:10.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.056+0000 7feeb7fff640 1 --2- 172.21.15.121:0/62158345 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feeb809f290 0x7feeb81a7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:10.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.056+0000 7feeb7fff640 10 monclient(hunting): get_auth_request con 0x7feeb809f290 auth_method 0 2024-09-17T18:23:10.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.056+0000 7feeb7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:10.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.056+0000 7feeb7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:10.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.056+0000 7feeb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:10.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.056+0000 7feeb77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:10.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.056+0000 7feeb77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:10.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.056+0000 7feeb77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:10.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.056+0000 7feeaeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:10.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.056+0000 7feeaeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:10.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.056+0000 7feeaeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:10.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.056+0000 7feeb7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:10.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.056+0000 7feeb7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:10.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.056+0000 7feeb7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:10.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.056+0000 7feeaeffd640 10 monclient(hunting): handle_auth_done global_id 34433 payload 824 2024-09-17T18:23:10.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.056+0000 7feeaeffd640 1 -- 172.21.15.121:0/62158345 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feeb809f290 msgr2=0x7feeb81a7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:10.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.056+0000 7feeaeffd640 1 --2- 172.21.15.121:0/62158345 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feeb809f290 0x7feeb81a7f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:10.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.056+0000 7feeaeffd640 1 -- 172.21.15.121:0/62158345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feeb809e550 msgr2=0x7feeb81b9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:10.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.056+0000 7feeaeffd640 1 --2- 172.21.15.121:0/62158345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feeb809e550 0x7feeb81b9190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:10.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.056+0000 7feeaeffd640 10 monclient: _finish_hunting 0 2024-09-17T18:23:10.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.056+0000 7feeaeffd640 1 monclient: found mon.noname-c 2024-09-17T18:23:10.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.056+0000 7feeaeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:23:10.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.056+0000 7feeaeffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:23:10.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.056+0000 7feeaeffd640 1 -- 172.21.15.121:0/62158345 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feeb81aafd0 con 0x7feeb809ee70 2024-09-17T18:23:10.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.056+0000 7feeb77fe640 1 --2- 172.21.15.121:0/62158345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feeb809e550 0x7feeb81b9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:23:10.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.058+0000 7feeaeffd640 1 -- 172.21.15.121:0/62158345 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feeb809ee70 msgr2=0x7feeb81a7a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:10.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.058+0000 7feeaeffd640 1 --2- 172.21.15.121:0/62158345 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feeb809ee70 0x7feeb81a7a50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7feea000e760 tx=0x7feea000ec20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:10.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.058+0000 7feeb4ff9640 10 client.?.objecter ms_handle_connect 0x7feeb809ee70 2024-09-17T18:23:10.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.058+0000 7feeb4ff9640 10 client.?.objecter resend_mon_ops 2024-09-17T18:23:10.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.058+0000 7feeb4ff9640 1 -- 172.21.15.121:0/62158345 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea000f040 con 0x7feeb809ee70 2024-09-17T18:23:10.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.058+0000 7feeb4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:10.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.058+0000 7feeb4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:23:10.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.058+0000 7feeb4ff9640 10 monclient: dump: 2024-09-17T18:23:10.074 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:10.075 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:10.075 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.075 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.075 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:10.075 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:10.075 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:10.075 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:10.075 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:10.075 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:10.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.058+0000 7feeb4ff9640 10 monclient: _finish_auth 0 2024-09-17T18:23:10.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.058+0000 7feeb4ff9640 10 monclient: _check_auth_tickets 2024-09-17T18:23:10.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.058+0000 7feeb4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:23:10.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.058+0000 7feeb4ff9640 1 -- 172.21.15.121:0/62158345 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feea0004230 con 0x7feeb809ee70 2024-09-17T18:23:10.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.058+0000 7feeb4ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T18:23:10.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.058+0000 7feeb4ff9640 1 -- 172.21.15.121:0/62158345 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea000ee70 con 0x7feeb809ee70 2024-09-17T18:23:10.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.058+0000 7feebde85640 5 monclient: authenticate success, global_id 34433 2024-09-17T18:23:10.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.058+0000 7feebde85640 10 monclient: _renew_subs 2024-09-17T18:23:10.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.058+0000 7feebde85640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:23:10.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.058+0000 7feebde85640 1 -- 172.21.15.121:0/62158345 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feeb809e240 con 0x7feeb809ee70 2024-09-17T18:23:10.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.058+0000 7feeb4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:10.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.058+0000 7feeb4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:23:10.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.058+0000 7feeb67fc640 4 set_mon_vals no callback set 2024-09-17T18:23:10.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.058+0000 7feeb4ff9640 10 monclient: dump: 2024-09-17T18:23:10.076 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:10.076 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:10.076 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.076 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.077 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:10.077 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:10.077 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:10.077 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:10.077 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:10.077 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:10.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.058+0000 7feebde85640 10 client.34433.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:23:10.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.058+0000 7feebde85640 10 monclient: _renew_subs 2024-09-17T18:23:10.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.058+0000 7feebde85640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:23:10.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.058+0000 7feebde85640 1 -- 172.21.15.121:0/62158345 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7feeb81ab870 con 0x7feeb809ee70 2024-09-17T18:23:10.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.058+0000 7feebde85640 1 librados: init done 2024-09-17T18:23:10.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.059+0000 7feebde85640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:23:10.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.059+0000 7feebde85640 1 --2- 172.21.15.121:0/62158345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feeb81a84d0 0x7feeb8070970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:10.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.059+0000 7feebde85640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:10.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.059+0000 7feebde85640 1 -- 172.21.15.121:0/62158345 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7feeb809e930 con 0x7feeb81a84d0 2024-09-17T18:23:10.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.059+0000 7feeaeffd640 1 --2- 172.21.15.121:0/62158345 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feeb809ee70 0x7feeb81a7a50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7feea000e760 tx=0x7feea000ec20 comp rx=0 tx=0).queue_received will delay after 1.7266e+09 on 0x7feea0053a60 osd_map(176..176 src has 1..176) 2024-09-17T18:23:10.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.059+0000 7feeb4ff9640 1 -- 172.21.15.121:0/62158345 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7feea000c8e0 con 0x7feeb809ee70 2024-09-17T18:23:10.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.059+0000 7feeb77fe640 1 --2- 172.21.15.121:0/62158345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feeb81a84d0 0x7feeb8070970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:10.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.059+0000 7feeb77fe640 10 monclient: get_auth_request con 0x7feeb81a84d0 auth_method 0 2024-09-17T18:23:10.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.059+0000 7feeb77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:10.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.059+0000 7feeb77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:10.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.059+0000 7feeb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:10.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.059+0000 7feeb77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:10.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.059+0000 7feeb77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:10.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.059+0000 7feeb77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:10.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.060+0000 7feeb4ff9640 1 --2- 172.21.15.121:0/62158345 >> v2:172.21.15.173:6812/2423786037 conn(0x7fee980401a0 0x7fee98042660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:10.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.060+0000 7feeb77fe640 10 monclient(hunting): handle_auth_done global_id 44109 payload 824 2024-09-17T18:23:10.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.060+0000 7feeb77fe640 1 -- 172.21.15.121:0/62158345 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feeb80008d0 con 0x7feeb81a84d0 2024-09-17T18:23:10.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.060+0000 7feeb7fff640 1 --2- 172.21.15.121:0/62158345 >> v2:172.21.15.173:6812/2423786037 conn(0x7fee980401a0 0x7fee98042660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:10.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.060+0000 7feeb7fff640 10 monclient: get_auth_request con 0x7fee980401a0 auth_method 0 2024-09-17T18:23:10.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.062+0000 7feeb77fe640 1 -- 172.21.15.121:0/62158345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feeb81a84d0 msgr2=0x7feeb8070970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:10.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.062+0000 7feeb4ff9640 10 client.34433.objecter ms_handle_connect 0x7feeb81a84d0 2024-09-17T18:23:10.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.062+0000 7feeb4ff9640 10 client.34433.objecter resend_mon_ops 2024-09-17T18:23:10.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.062+0000 7feeb77fe640 1 --2- 172.21.15.121:0/62158345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feeb81a84d0 0x7feeb8070970 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7feeb809fbb0 tx=0x7feea8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:10.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.062+0000 7feeb4ff9640 1 -- 172.21.15.121:0/62158345 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea8047410 con 0x7feeb81a84d0 2024-09-17T18:23:10.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.062+0000 7feeb4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:10.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.062+0000 7feeb4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:23:10.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.062+0000 7feeb4ff9640 10 monclient: dump: 2024-09-17T18:23:10.079 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:10.079 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:10.080 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.080 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.080 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:10.080 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:10.080 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:10.080 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:10.080 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:10.080 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:10.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.062+0000 7feeb4ff9640 1 -- 172.21.15.121:0/62158345 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7feea803d410 con 0x7feeb81a84d0 2024-09-17T18:23:10.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.062+0000 7feeb4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:23:10.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.062+0000 7feeb4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:23:10.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.062+0000 7feeb4ff9640 1 -- 172.21.15.121:0/62158345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feeb81a84d0 msgr2=0x7feeb8070970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:10.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.062+0000 7feeb4ff9640 1 --2- 172.21.15.121:0/62158345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feeb81a84d0 0x7feeb8070970 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7feeb809fbb0 tx=0x7feea8004730 comp rx=0 tx=0).stop 2024-09-17T18:23:10.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.062+0000 7feeb4ff9640 1 -- 172.21.15.121:0/62158345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feeb81a84d0 msgr2=0x7feeb8070970 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:23:10.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.062+0000 7feeb4ff9640 1 --2- 172.21.15.121:0/62158345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feeb81a84d0 0x7feeb8070970 secure :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7feeb809fbb0 tx=0x7feea8004730 comp rx=0 tx=0).stop 2024-09-17T18:23:10.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.063+0000 7feeb4ff9640 10 client.34433.objecter ms_handle_connect 0x7fee980401a0 2024-09-17T18:23:10.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.063+0000 7feeb7fff640 1 --2- 172.21.15.121:0/62158345 >> v2:172.21.15.173:6812/2423786037 conn(0x7fee980401a0 0x7fee98042660 secure :-1 s=READY pgs=3930 cs=0 l=1 rev1=1 crypto rx=0x7feeb81a9aa0 tx=0x7feeb0004ee0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:10.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.068+0000 7feebde85640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:23:10.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.068+0000 7feebde85640 1 --2- 172.21.15.121:0/62158345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feeb81a88b0 0x7feeb8070eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:10.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.068+0000 7feebde85640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:10.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.068+0000 7feebde85640 1 -- 172.21.15.121:0/62158345 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7feeb81d8d80 con 0x7feeb81a88b0 2024-09-17T18:23:10.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.068+0000 7feeb77fe640 1 --2- 172.21.15.121:0/62158345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feeb81a88b0 0x7feeb8070eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:10.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.068+0000 7feeb77fe640 10 monclient: get_auth_request con 0x7feeb81a88b0 auth_method 0 2024-09-17T18:23:10.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.068+0000 7feeb77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:10.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.068+0000 7feeb77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:10.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.068+0000 7feeb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:10.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.069+0000 7feeb77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:10.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.069+0000 7feeb77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:10.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.069+0000 7feeb77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:10.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.069+0000 7feeb77fe640 10 monclient(hunting): handle_auth_done global_id 44112 payload 824 2024-09-17T18:23:10.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.069+0000 7feeb77fe640 1 -- 172.21.15.121:0/62158345 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7feeb81d8f30 con 0x7feeb81a88b0 2024-09-17T18:23:10.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.071+0000 7feeb77fe640 1 -- 172.21.15.121:0/62158345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feeb81a88b0 msgr2=0x7feeb8070eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:10.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.071+0000 7feeb77fe640 1 --2- 172.21.15.121:0/62158345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feeb81a88b0 0x7feeb8070eb0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7feeb809eab0 tx=0x7feea80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:10.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.071+0000 7feeb4ff9640 10 client.34433.objecter ms_handle_connect 0x7feeb81a88b0 2024-09-17T18:23:10.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.071+0000 7feeb4ff9640 10 client.34433.objecter resend_mon_ops 2024-09-17T18:23:10.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.071+0000 7feeb4ff9640 1 -- 172.21.15.121:0/62158345 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea803d5c0 con 0x7feeb81a88b0 2024-09-17T18:23:10.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.071+0000 7feeb4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:10.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.071+0000 7feeb4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:23:10.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.071+0000 7feeb4ff9640 10 monclient: dump: 2024-09-17T18:23:10.084 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:10.084 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:10.084 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.084 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.084 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:10.084 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:10.084 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:10.085 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:10.085 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:10.085 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:10.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.071+0000 7feeb4ff9640 1 -- 172.21.15.121:0/62158345 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4554 (secure 0 0 0) 0x7feea803d5c0 con 0x7feeb81a88b0 2024-09-17T18:23:10.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.071+0000 7feeb4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:23:10.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.071+0000 7feeb4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:23:10.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.071+0000 7feeb4ff9640 1 -- 172.21.15.121:0/62158345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feeb81a88b0 msgr2=0x7feeb8070eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:10.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.071+0000 7feeb4ff9640 1 --2- 172.21.15.121:0/62158345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feeb81a88b0 0x7feeb8070eb0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7feeb809eab0 tx=0x7feea80046c0 comp rx=0 tx=0).stop 2024-09-17T18:23:10.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.071+0000 7feeb4ff9640 1 -- 172.21.15.121:0/62158345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feeb81a88b0 msgr2=0x7feeb8070eb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:23:10.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.071+0000 7feeb4ff9640 1 --2- 172.21.15.121:0/62158345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feeb81a88b0 0x7feeb8070eb0 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7feeb809eab0 tx=0x7feea80046c0 comp rx=0 tx=0).stop 2024-09-17T18:23:10.085 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:23:10.086 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:23:10.086 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:23:10.086 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "leader", 2024-09-17T18:23:10.086 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 20, 2024-09-17T18:23:10.086 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 15735, 2024-09-17T18:23:10.086 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:23:10.086 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:23:10.086 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:23:10.086 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:23:10.086 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:23:10.086 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 0, 2024-09-17T18:23:10.086 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:23:10.087 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:23:10.087 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:23:10.087 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:23:10.087 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:23:10.087 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:23:10.087 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:23:10.087 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:23:10.087 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:23:10.087 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:23:10.087 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:23:10.087 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:23:10.087 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:23:10.087 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:23:10.087 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:23:10.088 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:23:10.088 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:23:10.088 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:23:10.088 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:23:10.088 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:23:10.088 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:23:10.088 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:23:10.088 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:23:10.088 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:23:10.088 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:23:10.088 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:23:10.088 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:23:10.088 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:23:10.089 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:10.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:10.089 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:23:10.089 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:23:10.089 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:23:10.089 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:23:10.089 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:23:10.089 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:23:10.089 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:23:10.089 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:23:10.089 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:23:10.089 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:23:10.089 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:23:10.089 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:23:10.090 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:23:10.090 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:23:10.090 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:23:10.090 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:23:10.090 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:23:10.090 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:23:10.090 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:23:10.090 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:23:10.090 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:23:10.090 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:23:10.090 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:23:10.090 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:23:10.090 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:23:10.090 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:23:10.091 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:23:10.091 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:23:10.091 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:23:10.091 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:23:10.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:10.091 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:23:10.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:10.091 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:23:10.091 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:23:10.091 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:23:10.091 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:23:10.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:10.091 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:23:10.091 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:23:10.092 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:10.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:10.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:10.092 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:23:10.092 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:23:10.092 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:10.092 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:10.092 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:10.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:10.092 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:23:10.092 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:23:10.092 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:23:10.092 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:23:10.093 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:23:10.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:10.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:10.093 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:23:10.093 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:23:10.093 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:23:10.093 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:23:10.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:10.093 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:23:10.093 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:23:10.093 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:10.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:10.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:10.094 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:23:10.094 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:23:10.094 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:10.094 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:10.094 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:10.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:10.094 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:23:10.094 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:23:10.094 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:23:10.094 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:23:10.094 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:23:10.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:10.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:10.094 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:23:10.095 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:23:10.095 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:23:10.095 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:23:10.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:10.095 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:23:10.095 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:23:10.095 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:10.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:10.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:10.095 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:23:10.095 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:23:10.095 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:10.095 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:10.095 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:10.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:10.096 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:23:10.096 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:23:10.096 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:23:10.096 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:23:10.096 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:23:10.096 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:10.096 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:10.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:10.096 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:23:10.096 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:23:10.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:10.096 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:23:10.096 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:23:10.097 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:23:10.097 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:10.097 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:23:10.097 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:23:10.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:10.097 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:23:10.097 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:23:10.097 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-17T18:23:10.097 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:10.097 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:10.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:10.097 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:23:10.097 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:23:10.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.071+0000 7feebde85640 10 librados: watch_flush enter 2024-09-17T18:23:10.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.071+0000 7feebde85640 10 librados: watch_flush exit 2024-09-17T18:23:10.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.071+0000 7feebde85640 20 client.34433.objecter shutdown clearing up homeless session... 2024-09-17T18:23:10.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.071+0000 7feebde85640 10 client.34433.objecter successfully canceled tick 2024-09-17T18:23:10.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.072+0000 7feebde85640 1 -- 172.21.15.121:0/62158345 >> v2:172.21.15.173:6812/2423786037 conn(0x7fee980401a0 msgr2=0x7fee98042660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:10.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.072+0000 7feebde85640 1 --2- 172.21.15.121:0/62158345 >> v2:172.21.15.173:6812/2423786037 conn(0x7fee980401a0 0x7fee98042660 secure :-1 s=READY pgs=3930 cs=0 l=1 rev1=1 crypto rx=0x7feeb81a9aa0 tx=0x7feeb0004ee0 comp rx=0 tx=0).stop 2024-09-17T18:23:10.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.072+0000 7feebde85640 10 monclient: shutdown 2024-09-17T18:23:10.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.072+0000 7feebde85640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:23:10.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.072+0000 7feebde85640 1 -- 172.21.15.121:0/62158345 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feeb809ee70 msgr2=0x7feeb81a7a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:10.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.072+0000 7feebde85640 1 --2- 172.21.15.121:0/62158345 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feeb809ee70 0x7feeb81a7a50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7feea000e760 tx=0x7feea000ec20 comp rx=0 tx=0).stop 2024-09-17T18:23:10.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.072+0000 7feeb7fff640 1 -- 172.21.15.121:0/62158345 reap_dead start 2024-09-17T18:23:10.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.072+0000 7feeb4ff9640 1 -- 172.21.15.121:0/62158345 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(176..176 src has 1..176) ==== 6557+0+0 (unknown 0 0 0) 0x7feea0053a60 con 0x7feeb809ee70 2024-09-17T18:23:10.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.072+0000 7feeb4ff9640 10 client.34433.objecter ms_dispatch 0x7feeb8000b90 osd_map(176..176 src has 1..176) 2024-09-17T18:23:10.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.072+0000 7feeb4ff9640 10 librados: disconnected, discarding osd_map(176..176 src has 1..176) 2024-09-17T18:23:10.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.072+0000 7feeb7fff640 1 -- 172.21.15.121:0/62158345 reap_dead start 2024-09-17T18:23:10.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.072+0000 7feebde85640 1 -- 172.21.15.121:0/62158345 shutdown_connections 2024-09-17T18:23:10.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.072+0000 7feebde85640 1 -- 172.21.15.121:0/62158345 >> 172.21.15.121:0/62158345 conn(0x7feeb80a7830 msgr2=0x7feeb80a4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:23:10.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.072+0000 7feebde85640 1 -- 172.21.15.121:0/62158345 shutdown_connections 2024-09-17T18:23:10.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.072+0000 7feebde85640 1 -- 172.21.15.121:0/62158345 wait complete. 2024-09-17T18:23:10.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.072+0000 7feebde85640 1 librados: shutdown 2024-09-17T18:23:10.101 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:23:10.101 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T18:23:10.101 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T18:23:10.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.198+0000 7f933ffcd640 10 monclient: get_monmap_and_config 2024-09-17T18:23:10.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.198+0000 7f933ffcd640 10 monclient: build_initial_monmap 2024-09-17T18:23:10.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.198+0000 7f933ffcd640 10 monclient: monmap: 2024-09-17T18:23:10.201 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:23:10.201 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:10.201 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:23:10.199552+0000 2024-09-17T18:23:10.201 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:23:10.199552+0000 2024-09-17T18:23:10.201 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:23:10.202 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:10.202 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:23:10.202 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:23:10.202 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:23:10.202 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:23:10.202 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:23:10.202 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:10.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.199+0000 7f933ffcd640 1 Processor -- start 2024-09-17T18:23:10.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.199+0000 7f933ffcd640 1 -- start start 2024-09-17T18:23:10.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.199+0000 7f933ffcd640 10 monclient: init 2024-09-17T18:23:10.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.200+0000 7f933ffcd640 10 monclient: _reopen_session rank -1 2024-09-17T18:23:10.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.200+0000 7f933ffcd640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-17T18:23:10.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.200+0000 7f933ffcd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f93380a0a00 addr v1:172.21.15.121:6789/0 2024-09-17T18:23:10.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.200+0000 7f933ffcd640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f9338137c90 0x7f933813a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:10.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.200+0000 7f933ffcd640 10 monclient(hunting): picked mon.noname-c con 0x7f9338137c90 addr v2:172.21.15.173:3300/0 2024-09-17T18:23:10.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.200+0000 7f933ffcd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f933813a5c0 addr v1:172.21.15.173:6789/0 2024-09-17T18:23:10.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.200+0000 7f933ffcd640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93380a3820 con 0x7f93380a0a00 2024-09-17T18:23:10.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.200+0000 7f933ffcd640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93380a2ea0 con 0x7f933813a5c0 2024-09-17T18:23:10.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.200+0000 7f933ffcd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:10.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.200+0000 7f933ffcd640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f933809e800 con 0x7f9338137c90 2024-09-17T18:23:10.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.200+0000 7f933ffcd640 10 monclient(hunting): _renew_subs 2024-09-17T18:23:10.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.200+0000 7f933ffcd640 10 monclient(hunting): authenticate will time out at 993.916371s 2024-09-17T18:23:10.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.200+0000 7f933dd69640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f93380a0a00 0x7f93380a0de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:50814/0 (socket says 172.21.15.121:50814) 2024-09-17T18:23:10.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.200+0000 7f933dd69640 1 -- 172.21.15.121:0/1613859038 learned_addr learned my addr 172.21.15.121:0/1613859038 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:23:10.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.200+0000 7f933d568640 1 --2- 172.21.15.121:0/1613859038 >> v2:172.21.15.173:3300/0 conn(0x7f9338137c90 0x7f933813a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:10.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.200+0000 7f933d568640 10 monclient(hunting): get_auth_request con 0x7f9338137c90 auth_method 0 2024-09-17T18:23:10.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.200+0000 7f933d568640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:10.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.200+0000 7f933d568640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:10.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.200+0000 7f933d568640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:10.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.200+0000 7f933d568640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:10.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.200+0000 7f933d568640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:10.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.200+0000 7f933d568640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:10.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.200+0000 7f933d568640 10 monclient(hunting): handle_auth_done global_id 24490 payload 274 2024-09-17T18:23:10.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.201+0000 7f933d568640 1 -- 172.21.15.121:0/1613859038 >> v1:172.21.15.173:6789/0 conn(0x7f933813a5c0 legacy=0x7f933813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:23:10.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.201+0000 7f933d568640 1 -- 172.21.15.121:0/1613859038 >> v1:172.21.15.121:6789/0 conn(0x7f93380a0a00 legacy=0x7f93380a0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:23:10.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.201+0000 7f933d568640 10 monclient: _finish_hunting 0 2024-09-17T18:23:10.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.201+0000 7f933d568640 1 monclient: found mon.noname-c 2024-09-17T18:23:10.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.201+0000 7f933d568640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:23:10.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.201+0000 7f933d568640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.173:3300/0 2024-09-17T18:23:10.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.201+0000 7f933d568640 1 -- 172.21.15.121:0/1613859038 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93380a2c30 con 0x7f9338137c90 2024-09-17T18:23:10.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.203+0000 7f933d568640 1 -- 172.21.15.121:0/1613859038 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9338137c90 msgr2=0x7f933813a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:10.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.203+0000 7f933d568640 1 --2- 172.21.15.121:0/1613859038 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9338137c90 0x7f933813a080 secure :-1 s=READY pgs=7270 cs=0 l=1 rev1=1 crypto rx=0x7f932c0097d0 tx=0x7f932c02b110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f139810f27adc4e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:10.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.204+0000 7f933cd67640 1 -- 172.21.15.121:0/1613859038 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f932c038070 con 0x7f9338137c90 2024-09-17T18:23:10.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.204+0000 7f933cd67640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:10.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.204+0000 7f933cd67640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:23:10.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.204+0000 7f933cd67640 10 monclient: dump: 2024-09-17T18:23:10.214 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:10.214 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:10.214 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.214 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.215 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:10.215 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:10.215 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:10.215 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:10.215 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:10.215 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:10.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.204+0000 7f933cd67640 10 monclient: _finish_auth 0 2024-09-17T18:23:10.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.204+0000 7f933cd67640 10 monclient: _check_auth_tickets 2024-09-17T18:23:10.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.204+0000 7f933cd67640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:23:10.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.204+0000 7f933ffcd640 5 monclient: authenticate success, global_id 24490 2024-09-17T18:23:10.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.204+0000 7f933ffcd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:23:10.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.204+0000 7f933cd67640 1 -- 172.21.15.121:0/1613859038 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f932c033640 con 0x7f9338137c90 2024-09-17T18:23:10.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.204+0000 7f933cd67640 10 monclient: handle_config config(0 keys) 2024-09-17T18:23:10.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.204+0000 7f933ffcd640 10 monclient: get_monmap_and_config success 2024-09-17T18:23:10.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.204+0000 7f933cd67640 1 -- 172.21.15.121:0/1613859038 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f932c033a10 con 0x7f9338137c90 2024-09-17T18:23:10.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.204+0000 7f933cd67640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:10.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.204+0000 7f933ffcd640 4 set_mon_vals no callback set 2024-09-17T18:23:10.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.204+0000 7f933cd67640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:23:10.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.204+0000 7f933cd67640 10 monclient: dump: 2024-09-17T18:23:10.216 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:10.216 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:10.216 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.216 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.216 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:10.216 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:10.216 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:10.216 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:10.217 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:10.217 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:10.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.204+0000 7f933ffcd640 10 monclient: shutdown 2024-09-17T18:23:10.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.204+0000 7f933ffcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:23:10.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.204+0000 7f933ffcd640 1 -- 172.21.15.121:0/1613859038 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9338137c90 msgr2=0x7f933813a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:10.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.204+0000 7f933ffcd640 1 --2- 172.21.15.121:0/1613859038 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9338137c90 0x7f933813a080 secure :-1 s=READY pgs=7270 cs=0 l=1 rev1=1 crypto rx=0x7f932c0097d0 tx=0x7f932c02b110 comp rx=0 tx=0).stop 2024-09-17T18:23:10.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.204+0000 7f933ffcd640 1 -- 172.21.15.121:0/1613859038 shutdown_connections 2024-09-17T18:23:10.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.204+0000 7f933ffcd640 1 --2- 172.21.15.121:0/1613859038 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9338137c90 0x7f933813a080 unknown :-1 s=CLOSED pgs=7270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:10.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.204+0000 7f933ffcd640 1 -- 172.21.15.121:0/1613859038 >> 172.21.15.121:0/1613859038 conn(0x7f93380a7830 msgr2=0x7f93380a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:23:10.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.205+0000 7f933ffcd640 1 -- 172.21.15.121:0/1613859038 shutdown_connections 2024-09-17T18:23:10.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.205+0000 7f933ffcd640 1 -- 172.21.15.121:0/1613859038 wait complete. 2024-09-17T18:23:10.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.205+0000 7f933ffcd640 10 monclient: build_initial_monmap 2024-09-17T18:23:10.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.205+0000 7f933ffcd640 10 monclient: monmap: 2024-09-17T18:23:10.218 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:23:10.218 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:23:10.218 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:23:10.218 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:23:10.218 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:23:10.218 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:10.218 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:23:10.218 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:23:10.218 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:23:10.218 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:10.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.205+0000 7f933ffcd640 1 librados: starting msgr at 2024-09-17T18:23:10.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.205+0000 7f933ffcd640 1 librados: starting objecter 2024-09-17T18:23:10.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.205+0000 7f933ffcd640 1 Processor -- start 2024-09-17T18:23:10.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.205+0000 7f933ffcd640 1 -- start start 2024-09-17T18:23:10.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.205+0000 7f933ffcd640 1 librados: setting wanted keys 2024-09-17T18:23:10.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.205+0000 7f933ffcd640 1 librados: calling monclient init 2024-09-17T18:23:10.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.205+0000 7f933ffcd640 10 monclient: init 2024-09-17T18:23:10.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933ffcd640 10 monclient: _reopen_session rank -1 2024-09-17T18:23:10.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933ffcd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T18:23:10.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933ffcd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93380a0a00 0x7f93381c99c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:10.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933ffcd640 10 monclient(hunting): picked mon.noname-a con 0x7f93380a0a00 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:23:10.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933ffcd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9338137c90 0x7f93381c9f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:10.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933ffcd640 10 monclient(hunting): picked mon.noname-c con 0x7f9338137c90 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:23:10.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933ffcd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f933813a5c0 0x7f933811d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:10.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933ffcd640 10 monclient(hunting): picked mon.noname-b con 0x7f933813a5c0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:23:10.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933ffcd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:10.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933ffcd640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f933809de60 con 0x7f93380a0a00 2024-09-17T18:23:10.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933ffcd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:10.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933ffcd640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f933809dce0 con 0x7f933813a5c0 2024-09-17T18:23:10.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933ffcd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:10.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933ffcd640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f93380a3120 con 0x7f9338137c90 2024-09-17T18:23:10.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933ffcd640 10 monclient(hunting): _renew_subs 2024-09-17T18:23:10.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933ffcd640 10 monclient(hunting): authenticate will time out at 993.922363s 2024-09-17T18:23:10.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933d568640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9338137c90 0x7f93381c9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:10.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933dd69640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93380a0a00 0x7f93381c99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:10.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933d568640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9338137c90 0x7f93381c9f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:52708/0 (socket says 172.21.15.121:52708) 2024-09-17T18:23:10.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933d568640 1 -- 172.21.15.121:0/2402047237 learned_addr learned my addr 172.21.15.121:0/2402047237 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:23:10.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933dd69640 10 monclient(hunting): get_auth_request con 0x7f93380a0a00 auth_method 0 2024-09-17T18:23:10.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933dd69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:10.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933dd69640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:10.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933dd69640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:10.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933d568640 10 monclient(hunting): get_auth_request con 0x7f9338137c90 auth_method 0 2024-09-17T18:23:10.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933d568640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:10.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933d568640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:10.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933d568640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:10.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933dd69640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:10.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933dd69640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:10.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933dd69640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:10.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933e56a640 1 --2- 172.21.15.121:0/2402047237 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f933813a5c0 0x7f933811d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:10.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933d568640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:10.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933d568640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:10.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933d568640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:10.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933e56a640 10 monclient(hunting): get_auth_request con 0x7f933813a5c0 auth_method 0 2024-09-17T18:23:10.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933e56a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:10.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933e56a640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:10.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933e56a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:10.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933e56a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:10.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933e56a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:10.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933e56a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:10.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933dd69640 10 monclient(hunting): handle_auth_done global_id 44115 payload 824 2024-09-17T18:23:10.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933dd69640 1 -- 172.21.15.121:0/2402047237 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9338137c90 msgr2=0x7f93381c9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:10.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933dd69640 1 --2- 172.21.15.121:0/2402047237 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9338137c90 0x7f93381c9f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:10.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933dd69640 1 -- 172.21.15.121:0/2402047237 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f933813a5c0 msgr2=0x7f933811d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:10.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933dd69640 1 --2- 172.21.15.121:0/2402047237 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f933813a5c0 0x7f933811d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:10.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933dd69640 10 monclient: _finish_hunting 0 2024-09-17T18:23:10.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933dd69640 1 monclient: found mon.noname-a 2024-09-17T18:23:10.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933dd69640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:23:10.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933dd69640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:23:10.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933dd69640 1 -- 172.21.15.121:0/2402047237 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f933811f2d0 con 0x7f93380a0a00 2024-09-17T18:23:10.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.206+0000 7f933d568640 1 --2- 172.21.15.121:0/2402047237 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9338137c90 0x7f93381c9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:23:10.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.208+0000 7f933dd69640 1 -- 172.21.15.121:0/2402047237 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93380a0a00 msgr2=0x7f93381c99c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:10.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.208+0000 7f933dd69640 1 --2- 172.21.15.121:0/2402047237 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93380a0a00 0x7f93381c99c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f932800caf0 tx=0x7f932800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:10.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.208+0000 7f93367fc640 10 client.?.objecter ms_handle_connect 0x7f93380a0a00 2024-09-17T18:23:10.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.208+0000 7f93367fc640 10 client.?.objecter resend_mon_ops 2024-09-17T18:23:10.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.209+0000 7f93367fc640 1 -- 172.21.15.121:0/2402047237 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9328012400 con 0x7f93380a0a00 2024-09-17T18:23:10.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.209+0000 7f93367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:10.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.209+0000 7f93367fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:23:10.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.209+0000 7f93367fc640 10 monclient: dump: 2024-09-17T18:23:10.224 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:10.224 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:10.224 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.224 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.224 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:10.225 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:10.225 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:10.225 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:10.225 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:10.225 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:10.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.209+0000 7f93367fc640 10 monclient: _finish_auth 0 2024-09-17T18:23:10.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.209+0000 7f93367fc640 10 monclient: _check_auth_tickets 2024-09-17T18:23:10.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.209+0000 7f93367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:23:10.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.209+0000 7f93367fc640 1 -- 172.21.15.121:0/2402047237 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93280125a0 con 0x7f93380a0a00 2024-09-17T18:23:10.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.209+0000 7f933ffcd640 5 monclient: authenticate success, global_id 44115 2024-09-17T18:23:10.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.209+0000 7f933ffcd640 10 monclient: _renew_subs 2024-09-17T18:23:10.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.209+0000 7f933ffcd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:23:10.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.209+0000 7f933ffcd640 1 -- 172.21.15.121:0/2402047237 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9338120990 con 0x7f93380a0a00 2024-09-17T18:23:10.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.209+0000 7f93367fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:23:10.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.209+0000 7f93367fc640 1 -- 172.21.15.121:0/2402047237 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9328012990 con 0x7f93380a0a00 2024-09-17T18:23:10.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.209+0000 7f93367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:10.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.209+0000 7f93367fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:23:10.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.209+0000 7f93367fc640 10 monclient: dump: 2024-09-17T18:23:10.226 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:10.226 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:10.226 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.226 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.226 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:10.226 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:10.226 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:10.226 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:10.227 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:10.227 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:10.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.209+0000 7f933ffcd640 10 client.44115.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:23:10.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.209+0000 7f9337fff640 4 set_mon_vals no callback set 2024-09-17T18:23:10.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.209+0000 7f933ffcd640 10 monclient: _renew_subs 2024-09-17T18:23:10.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.209+0000 7f933ffcd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:23:10.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.209+0000 7f933ffcd640 1 -- 172.21.15.121:0/2402047237 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93380a56e0 con 0x7f93380a0a00 2024-09-17T18:23:10.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.209+0000 7f933ffcd640 1 librados: init done 2024-09-17T18:23:10.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.210+0000 7f93367fc640 1 -- 172.21.15.121:0/2402047237 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f9328012bf0 con 0x7f93380a0a00 2024-09-17T18:23:10.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.210+0000 7f933ffcd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:23:10.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.210+0000 7f933ffcd640 1 --2- 172.21.15.121:0/2402047237 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f933811e3a0 0x7f933802d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:10.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.210+0000 7f933ffcd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:10.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.210+0000 7f933ffcd640 1 -- 172.21.15.121:0/2402047237 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9338137580 con 0x7f933811e3a0 2024-09-17T18:23:10.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.210+0000 7f933d568640 1 --2- 172.21.15.121:0/2402047237 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f933811e3a0 0x7f933802d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:10.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.210+0000 7f933d568640 10 monclient: get_auth_request con 0x7f933811e3a0 auth_method 0 2024-09-17T18:23:10.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.210+0000 7f933d568640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:10.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.210+0000 7f933d568640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:10.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.210+0000 7f933d568640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:10.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.210+0000 7f93367fc640 1 --2- 172.21.15.121:0/2402047237 >> v2:172.21.15.173:6812/2423786037 conn(0x7f93240401a0 0x7f9324042660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:10.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.210+0000 7f933d568640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:10.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.210+0000 7f933d568640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:10.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.210+0000 7f933d568640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:10.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.210+0000 7f93367fc640 1 -- 172.21.15.121:0/2402047237 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(176..176 src has 1..176) ==== 6557+0+0 (secure 0 0 0) 0x7f9328054260 con 0x7f93380a0a00 2024-09-17T18:23:10.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.210+0000 7f93367fc640 10 client.44115.objecter ms_dispatch 0x7f9338000b90 osd_map(176..176 src has 1..176) 2024-09-17T18:23:10.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.210+0000 7f93367fc640 3 client.44115.objecter handle_osd_map got epochs [176,176] > 0 2024-09-17T18:23:10.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.210+0000 7f93367fc640 3 client.44115.objecter handle_osd_map decoding full epoch 176 2024-09-17T18:23:10.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.210+0000 7f933d568640 10 monclient(hunting): handle_auth_done global_id 34439 payload 824 2024-09-17T18:23:10.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.210+0000 7f933d568640 1 -- 172.21.15.121:0/2402047237 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f93380a1fa0 con 0x7f933811e3a0 2024-09-17T18:23:10.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.210+0000 7f93367fc640 20 client.44115.objecter dump_active .. 0 homeless 2024-09-17T18:23:10.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.210+0000 7f933e56a640 1 --2- 172.21.15.121:0/2402047237 >> v2:172.21.15.173:6812/2423786037 conn(0x7f93240401a0 0x7f9324042660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:10.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.210+0000 7f933e56a640 10 monclient: get_auth_request con 0x7f93240401a0 auth_method 0 2024-09-17T18:23:10.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.212+0000 7f93367fc640 10 client.44115.objecter ms_handle_connect 0x7f933811e3a0 2024-09-17T18:23:10.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.212+0000 7f933d568640 1 -- 172.21.15.121:0/2402047237 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f933811e3a0 msgr2=0x7f933802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:10.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.212+0000 7f93367fc640 10 client.44115.objecter resend_mon_ops 2024-09-17T18:23:10.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.212+0000 7f933d568640 1 --2- 172.21.15.121:0/2402047237 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f933811e3a0 0x7f933802d440 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f932c036040 tx=0x7f932c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:10.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.213+0000 7f93367fc640 10 client.44115.objecter ms_handle_connect 0x7f93240401a0 2024-09-17T18:23:10.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.213+0000 7f933e56a640 1 --2- 172.21.15.121:0/2402047237 >> v2:172.21.15.173:6812/2423786037 conn(0x7f93240401a0 0x7f9324042660 secure :-1 s=READY pgs=3931 cs=0 l=1 rev1=1 crypto rx=0x7f9330004ee0 tx=0x7f9330007db0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:10.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.214+0000 7f93367fc640 1 -- 172.21.15.121:0/2402047237 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f932c044070 con 0x7f933811e3a0 2024-09-17T18:23:10.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.214+0000 7f93367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:10.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.214+0000 7f93367fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:23:10.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.214+0000 7f93367fc640 10 monclient: dump: 2024-09-17T18:23:10.232 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:10.232 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:10.232 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.232 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.232 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:10.232 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:10.232 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:10.232 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:10.232 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:10.232 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:10.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.214+0000 7f93367fc640 1 -- 172.21.15.121:0/2402047237 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f932c03da60 con 0x7f933811e3a0 2024-09-17T18:23:10.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.214+0000 7f93367fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:23:10.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.214+0000 7f93367fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:23:10.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.214+0000 7f93367fc640 1 -- 172.21.15.121:0/2402047237 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f933811e3a0 msgr2=0x7f933802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:10.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.214+0000 7f93367fc640 1 --2- 172.21.15.121:0/2402047237 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f933811e3a0 0x7f933802d440 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f932c036040 tx=0x7f932c0046c0 comp rx=0 tx=0).stop 2024-09-17T18:23:10.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.215+0000 7f93367fc640 1 -- 172.21.15.121:0/2402047237 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f933811e3a0 msgr2=0x7f933802d440 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:23:10.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.215+0000 7f93367fc640 1 --2- 172.21.15.121:0/2402047237 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f933811e3a0 0x7f933802d440 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:10.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.220+0000 7f933ffcd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:23:10.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.220+0000 7f933ffcd640 1 --2- 172.21.15.121:0/2402047237 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f933811e780 0x7f933802d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:10.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.221+0000 7f933ffcd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:10.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.221+0000 7f933ffcd640 1 -- 172.21.15.121:0/2402047237 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9338120eb0 con 0x7f933811e780 2024-09-17T18:23:10.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.221+0000 7f933d568640 1 --2- 172.21.15.121:0/2402047237 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f933811e780 0x7f933802d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:10.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.221+0000 7f933d568640 10 monclient: get_auth_request con 0x7f933811e780 auth_method 0 2024-09-17T18:23:10.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.221+0000 7f933d568640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:10.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.221+0000 7f933d568640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:10.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.221+0000 7f933d568640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:10.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.221+0000 7f933d568640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:10.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.221+0000 7f933d568640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:10.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.221+0000 7f933d568640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:10.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.221+0000 7f933d568640 10 monclient(hunting): handle_auth_done global_id 34442 payload 824 2024-09-17T18:23:10.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.221+0000 7f933d568640 1 -- 172.21.15.121:0/2402047237 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f93380709d0 con 0x7f933811e780 2024-09-17T18:23:10.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.223+0000 7f933d568640 1 -- 172.21.15.121:0/2402047237 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f933811e780 msgr2=0x7f933802d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:10.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.223+0000 7f933d568640 1 --2- 172.21.15.121:0/2402047237 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f933811e780 0x7f933802d980 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f932c02b150 tx=0x7f932c009190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:10.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.223+0000 7f93367fc640 10 client.44115.objecter ms_handle_connect 0x7f933811e780 2024-09-17T18:23:10.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.223+0000 7f93367fc640 10 client.44115.objecter resend_mon_ops 2024-09-17T18:23:10.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.223+0000 7f93367fc640 1 -- 172.21.15.121:0/2402047237 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f932c038040 con 0x7f933811e780 2024-09-17T18:23:10.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.223+0000 7f93367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:10.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.223+0000 7f93367fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:23:10.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.223+0000 7f93367fc640 10 monclient: dump: 2024-09-17T18:23:10.235 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:10.235 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:10.235 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.235 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.235 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:10.235 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:10.235 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:10.235 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:10.235 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:10.235 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:10.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.223+0000 7f93367fc640 1 -- 172.21.15.121:0/2402047237 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7f932c03c7c0 con 0x7f933811e780 2024-09-17T18:23:10.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.223+0000 7f93367fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:23:10.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.223+0000 7f93367fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:23:10.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.223+0000 7f93367fc640 1 -- 172.21.15.121:0/2402047237 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f933811e780 msgr2=0x7f933802d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:10.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.223+0000 7f93367fc640 1 --2- 172.21.15.121:0/2402047237 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f933811e780 0x7f933802d980 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f932c02b150 tx=0x7f932c009190 comp rx=0 tx=0).stop 2024-09-17T18:23:10.236 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:23:10.236 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:23:10.236 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:23:10.236 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:23:10.237 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 20, 2024-09-17T18:23:10.237 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 135500, 2024-09-17T18:23:10.237 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:23:10.237 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:23:10.237 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:23:10.237 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:23:10.237 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:23:10.237 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 0, 2024-09-17T18:23:10.237 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:23:10.237 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:23:10.237 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:23:10.237 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:23:10.237 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:23:10.238 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:23:10.238 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:23:10.238 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:23:10.238 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:23:10.238 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:23:10.238 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:23:10.238 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:23:10.238 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:23:10.238 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:23:10.238 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:23:10.238 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:23:10.238 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:23:10.238 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:23:10.238 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:23:10.239 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:23:10.239 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:23:10.239 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:23:10.239 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:23:10.239 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:23:10.239 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:23:10.239 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:23:10.239 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:23:10.239 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:23:10.239 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:10.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:10.239 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:23:10.239 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:23:10.240 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:23:10.240 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:23:10.240 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:23:10.240 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:23:10.240 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:23:10.240 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:23:10.240 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:23:10.240 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:23:10.240 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:23:10.240 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:23:10.240 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:23:10.240 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:23:10.240 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:23:10.240 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:23:10.240 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:23:10.241 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:23:10.241 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:23:10.241 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:23:10.241 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:23:10.241 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:23:10.241 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:23:10.241 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:23:10.241 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:23:10.241 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:23:10.241 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:23:10.241 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:23:10.241 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:23:10.241 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:23:10.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:10.242 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:23:10.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:10.242 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:23:10.242 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:23:10.242 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:23:10.242 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:23:10.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:10.242 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:23:10.242 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:23:10.242 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:10.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:10.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:10.242 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:23:10.242 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:23:10.243 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:10.243 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:10.243 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:10.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:10.243 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:23:10.243 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:23:10.243 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:23:10.243 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:23:10.243 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:23:10.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:10.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:10.243 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:23:10.243 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:23:10.243 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:23:10.244 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:23:10.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:10.244 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:23:10.244 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:23:10.244 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:10.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:10.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:10.244 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:23:10.244 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:23:10.244 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:10.244 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:10.244 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:10.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:10.245 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:23:10.245 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:23:10.245 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:23:10.245 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:23:10.245 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:23:10.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:10.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:10.245 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:23:10.245 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:23:10.245 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:23:10.245 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:23:10.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:10.245 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:23:10.246 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:23:10.246 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:10.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:10.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:10.246 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:23:10.246 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:23:10.246 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:10.246 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:10.246 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:10.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:10.246 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:23:10.246 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:23:10.247 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:23:10.247 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:23:10.247 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:23:10.247 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:10.247 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:10.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:10.247 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:23:10.247 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:23:10.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:10.247 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:23:10.247 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:23:10.247 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:23:10.247 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:10.248 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:23:10.248 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:23:10.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:10.248 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:23:10.248 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:23:10.248 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 3 2024-09-17T18:23:10.248 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:10.248 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:23:10.248 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:23:10.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:10.248 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:23:10.248 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:23:10.249 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:23:10.249 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:10.249 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:10.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:10.249 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:23:10.249 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:23:10.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.223+0000 7f93367fc640 1 -- 172.21.15.121:0/2402047237 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f933811e780 msgr2=0x7f933802d980 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:23:10.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.223+0000 7f93367fc640 1 --2- 172.21.15.121:0/2402047237 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f933811e780 0x7f933802d980 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:10.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.224+0000 7f933ffcd640 10 librados: watch_flush enter 2024-09-17T18:23:10.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.224+0000 7f933ffcd640 10 librados: watch_flush exit 2024-09-17T18:23:10.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.224+0000 7f933ffcd640 20 client.44115.objecter shutdown clearing up homeless session... 2024-09-17T18:23:10.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.224+0000 7f933ffcd640 10 client.44115.objecter successfully canceled tick 2024-09-17T18:23:10.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.224+0000 7f933ffcd640 1 -- 172.21.15.121:0/2402047237 >> v2:172.21.15.173:6812/2423786037 conn(0x7f93240401a0 msgr2=0x7f9324042660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:10.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.224+0000 7f933ffcd640 1 --2- 172.21.15.121:0/2402047237 >> v2:172.21.15.173:6812/2423786037 conn(0x7f93240401a0 0x7f9324042660 secure :-1 s=READY pgs=3931 cs=0 l=1 rev1=1 crypto rx=0x7f9330004ee0 tx=0x7f9330007db0 comp rx=0 tx=0).stop 2024-09-17T18:23:10.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.224+0000 7f933ffcd640 10 monclient: shutdown 2024-09-17T18:23:10.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.224+0000 7f933ffcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:23:10.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.224+0000 7f933ffcd640 1 -- 172.21.15.121:0/2402047237 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93380a0a00 msgr2=0x7f93381c99c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:10.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.224+0000 7f933ffcd640 1 --2- 172.21.15.121:0/2402047237 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93380a0a00 0x7f93381c99c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f932800caf0 tx=0x7f932800a060 comp rx=0 tx=0).stop 2024-09-17T18:23:10.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.224+0000 7f933e56a640 1 -- 172.21.15.121:0/2402047237 reap_dead start 2024-09-17T18:23:10.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.224+0000 7f933e56a640 1 -- 172.21.15.121:0/2402047237 reap_dead start 2024-09-17T18:23:10.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.224+0000 7f933ffcd640 1 -- 172.21.15.121:0/2402047237 shutdown_connections 2024-09-17T18:23:10.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.224+0000 7f933ffcd640 1 -- 172.21.15.121:0/2402047237 >> 172.21.15.121:0/2402047237 conn(0x7f93380a7830 msgr2=0x7f9338136400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:23:10.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.224+0000 7f933ffcd640 1 -- 172.21.15.121:0/2402047237 shutdown_connections 2024-09-17T18:23:10.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.224+0000 7f933ffcd640 1 -- 172.21.15.121:0/2402047237 wait complete. 2024-09-17T18:23:10.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.224+0000 7f933ffcd640 1 librados: shutdown 2024-09-17T18:23:10.253 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:23:10.253 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T18:23:10.253 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T18:23:10.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.350+0000 7fed655e2640 10 monclient: get_monmap_and_config 2024-09-17T18:23:10.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.350+0000 7fed655e2640 10 monclient: build_initial_monmap 2024-09-17T18:23:10.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.350+0000 7fed655e2640 10 monclient: monmap: 2024-09-17T18:23:10.352 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:23:10.352 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:10.352 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:23:10.350645+0000 2024-09-17T18:23:10.352 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:23:10.350645+0000 2024-09-17T18:23:10.352 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:23:10.352 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:10.352 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:23:10.352 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:23:10.352 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:23:10.353 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:23:10.353 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:23:10.353 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:10.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.350+0000 7fed655e2640 1 Processor -- start 2024-09-17T18:23:10.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.350+0000 7fed655e2640 1 -- start start 2024-09-17T18:23:10.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.350+0000 7fed655e2640 10 monclient: init 2024-09-17T18:23:10.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.351+0000 7fed655e2640 10 monclient: _reopen_session rank -1 2024-09-17T18:23:10.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.351+0000 7fed655e2640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-17T18:23:10.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.351+0000 7fed655e2640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed600a5ca0 0x7fed600a6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:10.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.351+0000 7fed655e2640 10 monclient(hunting): picked mon.noname-b con 0x7fed600a5ca0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:23:10.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.351+0000 7fed655e2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fed600a65c0 addr v1:172.21.15.173:6789/0 2024-09-17T18:23:10.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.351+0000 7fed655e2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fed6013e1a0 addr v1:172.21.15.121:6789/0 2024-09-17T18:23:10.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.351+0000 7fed655e2640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed600a7b50 con 0x7fed6013e1a0 2024-09-17T18:23:10.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.351+0000 7fed655e2640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed600a2a50 con 0x7fed600a65c0 2024-09-17T18:23:10.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.351+0000 7fed655e2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:10.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.351+0000 7fed655e2640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fed600704c0 con 0x7fed600a5ca0 2024-09-17T18:23:10.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.351+0000 7fed655e2640 10 monclient(hunting): _renew_subs 2024-09-17T18:23:10.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.351+0000 7fed655e2640 10 monclient(hunting): authenticate will time out at 994.067295s 2024-09-17T18:23:10.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.351+0000 7fed5effd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed600a5ca0 0x7fed600a6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:10.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.351+0000 7fed5f7fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fed6013e1a0 0x7fed60140590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:50828/0 (socket says 172.21.15.121:50828) 2024-09-17T18:23:10.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.351+0000 7fed5f7fe640 1 -- 172.21.15.121:0/4139903959 learned_addr learned my addr 172.21.15.121:0/4139903959 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:23:10.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.351+0000 7fed5effd640 10 monclient(hunting): get_auth_request con 0x7fed600a5ca0 auth_method 0 2024-09-17T18:23:10.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.351+0000 7fed5effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:10.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.351+0000 7fed5effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:10.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.351+0000 7fed5effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:10.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.351+0000 7fed5effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:10.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.351+0000 7fed5effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:10.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.351+0000 7fed5effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:10.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.351+0000 7fed5effd640 10 monclient(hunting): handle_auth_done global_id 34445 payload 274 2024-09-17T18:23:10.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.351+0000 7fed5effd640 1 -- 172.21.15.121:0/4139903959 >> v1:172.21.15.173:6789/0 conn(0x7fed600a65c0 legacy=0x7fed6013ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:23:10.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.351+0000 7fed5effd640 1 -- 172.21.15.121:0/4139903959 >> v1:172.21.15.121:6789/0 conn(0x7fed6013e1a0 legacy=0x7fed60140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:23:10.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.351+0000 7fed5effd640 10 monclient: _finish_hunting 0 2024-09-17T18:23:10.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.351+0000 7fed5effd640 1 monclient: found mon.noname-b 2024-09-17T18:23:10.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.351+0000 7fed5effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:23:10.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.351+0000 7fed5effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:23:10.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.351+0000 7fed5effd640 1 -- 172.21.15.121:0/4139903959 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed600a2d30 con 0x7fed600a5ca0 2024-09-17T18:23:10.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.354+0000 7fed5effd640 1 -- 172.21.15.121:0/4139903959 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed600a5ca0 msgr2=0x7fed600a6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:10.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.354+0000 7fed5effd640 1 --2- 172.21.15.121:0/4139903959 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed600a5ca0 0x7fed600a6080 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fed50009dc0 tx=0x7fed5002a480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf60870d3705c3e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:10.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.354+0000 7fed5dffb640 1 -- 172.21.15.121:0/4139903959 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed50007a30 con 0x7fed600a5ca0 2024-09-17T18:23:10.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.354+0000 7fed5dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:10.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.354+0000 7fed5dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:23:10.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.354+0000 7fed5dffb640 10 monclient: dump: 2024-09-17T18:23:10.364 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:10.364 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:10.364 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.364 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.364 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:10.365 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:10.365 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:10.365 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:10.365 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:10.365 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:10.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.354+0000 7fed5dffb640 10 monclient: _finish_auth 0 2024-09-17T18:23:10.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.354+0000 7fed5dffb640 10 monclient: _check_auth_tickets 2024-09-17T18:23:10.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.354+0000 7fed5dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:23:10.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.354+0000 7fed5dffb640 1 -- 172.21.15.121:0/4139903959 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed50007bd0 con 0x7fed600a5ca0 2024-09-17T18:23:10.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.354+0000 7fed655e2640 5 monclient: authenticate success, global_id 34445 2024-09-17T18:23:10.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.354+0000 7fed655e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:23:10.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.354+0000 7fed5dffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:23:10.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.354+0000 7fed5dffb640 1 -- 172.21.15.121:0/4139903959 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed50031400 con 0x7fed600a5ca0 2024-09-17T18:23:10.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.354+0000 7fed655e2640 10 monclient: get_monmap_and_config success 2024-09-17T18:23:10.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.354+0000 7fed655e2640 4 set_mon_vals no callback set 2024-09-17T18:23:10.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.354+0000 7fed5dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:10.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.354+0000 7fed5dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:23:10.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.354+0000 7fed5dffb640 10 monclient: dump: 2024-09-17T18:23:10.366 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:10.366 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:10.366 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.366 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.366 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:10.366 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:10.366 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:10.366 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:10.367 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:10.367 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:10.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.354+0000 7fed655e2640 10 monclient: shutdown 2024-09-17T18:23:10.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.354+0000 7fed655e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:23:10.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.354+0000 7fed655e2640 1 -- 172.21.15.121:0/4139903959 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed600a5ca0 msgr2=0x7fed600a6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:10.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.355+0000 7fed655e2640 1 --2- 172.21.15.121:0/4139903959 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed600a5ca0 0x7fed600a6080 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fed50009dc0 tx=0x7fed5002a480 comp rx=0 tx=0).stop 2024-09-17T18:23:10.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.355+0000 7fed655e2640 1 -- 172.21.15.121:0/4139903959 shutdown_connections 2024-09-17T18:23:10.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.355+0000 7fed655e2640 1 --2- 172.21.15.121:0/4139903959 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed600a5ca0 0x7fed600a6080 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:10.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.355+0000 7fed655e2640 1 -- 172.21.15.121:0/4139903959 >> 172.21.15.121:0/4139903959 conn(0x7fed6006dd10 msgr2=0x7fed600a7da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:23:10.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.355+0000 7fed655e2640 1 -- 172.21.15.121:0/4139903959 shutdown_connections 2024-09-17T18:23:10.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.355+0000 7fed655e2640 1 -- 172.21.15.121:0/4139903959 wait complete. 2024-09-17T18:23:10.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.355+0000 7fed655e2640 10 monclient: build_initial_monmap 2024-09-17T18:23:10.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.355+0000 7fed655e2640 10 monclient: monmap: 2024-09-17T18:23:10.367 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:23:10.368 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:23:10.368 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:23:10.368 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:23:10.368 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:23:10.368 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:10.368 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:23:10.368 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:23:10.368 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:23:10.368 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:10.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.355+0000 7fed655e2640 1 librados: starting msgr at 2024-09-17T18:23:10.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.355+0000 7fed655e2640 1 librados: starting objecter 2024-09-17T18:23:10.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed655e2640 1 Processor -- start 2024-09-17T18:23:10.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed655e2640 1 -- start start 2024-09-17T18:23:10.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed655e2640 1 librados: setting wanted keys 2024-09-17T18:23:10.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed655e2640 1 librados: calling monclient init 2024-09-17T18:23:10.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed655e2640 10 monclient: init 2024-09-17T18:23:10.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed655e2640 10 monclient: _reopen_session rank -1 2024-09-17T18:23:10.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed655e2640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:23:10.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed655e2640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed600a5ca0 0x7fed600a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:10.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed655e2640 10 monclient(hunting): picked mon.noname-a con 0x7fed600a5ca0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:23:10.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed655e2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed600a65c0 0x7fed601cd970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:10.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed655e2640 10 monclient(hunting): picked mon.noname-b con 0x7fed600a65c0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:23:10.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed655e2640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed6013e1a0 0x7fed60138070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:10.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed655e2640 10 monclient(hunting): picked mon.noname-c con 0x7fed6013e1a0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:23:10.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed655e2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:10.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed655e2640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fed600a5670 con 0x7fed600a5ca0 2024-09-17T18:23:10.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed655e2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:10.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed655e2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fed600a5490 con 0x7fed600a65c0 2024-09-17T18:23:10.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed655e2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:10.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed655e2640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fed600a69a0 con 0x7fed6013e1a0 2024-09-17T18:23:10.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed655e2640 10 monclient(hunting): _renew_subs 2024-09-17T18:23:10.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed655e2640 10 monclient(hunting): authenticate will time out at 994.072665s 2024-09-17T18:23:10.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed5effd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed600a5ca0 0x7fed600a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:10.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed5effd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed600a5ca0 0x7fed600a05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60136/0 (socket says 172.21.15.121:60136) 2024-09-17T18:23:10.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed5f7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed6013e1a0 0x7fed60138070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:10.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed5effd640 1 -- 172.21.15.121:0/1518278406 learned_addr learned my addr 172.21.15.121:0/1518278406 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:23:10.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed5effd640 10 monclient(hunting): get_auth_request con 0x7fed600a5ca0 auth_method 0 2024-09-17T18:23:10.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed5effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:10.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed5effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:10.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed5effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:10.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed5f7fe640 10 monclient(hunting): get_auth_request con 0x7fed6013e1a0 auth_method 0 2024-09-17T18:23:10.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed5f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:10.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed5f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:10.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:10.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed5effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:10.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed5effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:10.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed5effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:10.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed5e7fc640 1 --2- 172.21.15.121:0/1518278406 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed600a65c0 0x7fed601cd970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:10.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed5f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:10.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed5f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:10.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed5f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:10.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed5e7fc640 10 monclient(hunting): get_auth_request con 0x7fed600a65c0 auth_method 0 2024-09-17T18:23:10.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed5e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:10.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed5e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:10.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:10.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed5e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:10.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed5e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:10.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.356+0000 7fed5e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:10.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.357+0000 7fed5f7fe640 10 monclient(hunting): handle_auth_done global_id 34448 payload 824 2024-09-17T18:23:10.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.357+0000 7fed5f7fe640 1 -- 172.21.15.121:0/1518278406 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed600a65c0 msgr2=0x7fed601cd970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:10.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.357+0000 7fed5f7fe640 1 --2- 172.21.15.121:0/1518278406 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed600a65c0 0x7fed601cd970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:10.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.357+0000 7fed5f7fe640 1 -- 172.21.15.121:0/1518278406 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed600a5ca0 msgr2=0x7fed600a05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:10.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.357+0000 7fed5f7fe640 1 --2- 172.21.15.121:0/1518278406 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed600a5ca0 0x7fed600a05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:10.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.357+0000 7fed5f7fe640 10 monclient: _finish_hunting 0 2024-09-17T18:23:10.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.357+0000 7fed5f7fe640 1 monclient: found mon.noname-c 2024-09-17T18:23:10.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.357+0000 7fed5f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:23:10.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.357+0000 7fed5f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:23:10.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.357+0000 7fed5f7fe640 1 -- 172.21.15.121:0/1518278406 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed601ce030 con 0x7fed6013e1a0 2024-09-17T18:23:10.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.357+0000 7fed5effd640 1 --2- 172.21.15.121:0/1518278406 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed600a5ca0 0x7fed600a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:23:10.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.359+0000 7fed5f7fe640 1 -- 172.21.15.121:0/1518278406 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed6013e1a0 msgr2=0x7fed60138070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:10.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.359+0000 7fed5f7fe640 1 --2- 172.21.15.121:0/1518278406 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed6013e1a0 0x7fed60138070 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fed5800c3e0 tx=0x7fed5800c7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:10.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.359+0000 7fed577fe640 10 client.?.objecter ms_handle_connect 0x7fed6013e1a0 2024-09-17T18:23:10.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.359+0000 7fed577fe640 10 client.?.objecter resend_mon_ops 2024-09-17T18:23:10.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.359+0000 7fed577fe640 1 -- 172.21.15.121:0/1518278406 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed58017070 con 0x7fed6013e1a0 2024-09-17T18:23:10.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.359+0000 7fed577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:10.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.359+0000 7fed577fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:23:10.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.359+0000 7fed577fe640 10 monclient: dump: 2024-09-17T18:23:10.373 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:10.373 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:10.374 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.374 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.374 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:10.374 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:10.374 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:10.374 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:10.374 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:10.374 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:10.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.359+0000 7fed577fe640 10 monclient: _finish_auth 0 2024-09-17T18:23:10.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.359+0000 7fed577fe640 10 monclient: _check_auth_tickets 2024-09-17T18:23:10.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.359+0000 7fed577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:23:10.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.359+0000 7fed577fe640 1 -- 172.21.15.121:0/1518278406 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed580070d0 con 0x7fed6013e1a0 2024-09-17T18:23:10.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.359+0000 7fed655e2640 5 monclient: authenticate success, global_id 34448 2024-09-17T18:23:10.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.359+0000 7fed655e2640 10 monclient: _renew_subs 2024-09-17T18:23:10.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.359+0000 7fed655e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:23:10.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.359+0000 7fed655e2640 1 -- 172.21.15.121:0/1518278406 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed601241e0 con 0x7fed6013e1a0 2024-09-17T18:23:10.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.359+0000 7fed577fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:23:10.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.359+0000 7fed577fe640 1 -- 172.21.15.121:0/1518278406 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed580074c0 con 0x7fed6013e1a0 2024-09-17T18:23:10.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.359+0000 7fed577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:10.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.359+0000 7fed577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:23:10.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.359+0000 7fed577fe640 10 monclient: dump: 2024-09-17T18:23:10.375 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:10.375 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:10.375 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.375 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.375 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:10.375 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:10.375 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:10.375 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:10.376 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:10.376 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:10.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.359+0000 7fed5d7fa640 4 set_mon_vals no callback set 2024-09-17T18:23:10.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.359+0000 7fed655e2640 10 client.34448.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:23:10.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.359+0000 7fed655e2640 10 monclient: _renew_subs 2024-09-17T18:23:10.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.359+0000 7fed655e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:23:10.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.359+0000 7fed655e2640 1 -- 172.21.15.121:0/1518278406 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fed601385b0 con 0x7fed6013e1a0 2024-09-17T18:23:10.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.359+0000 7fed655e2640 1 librados: init done 2024-09-17T18:23:10.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.360+0000 7fed655e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:23:10.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.360+0000 7fed655e2640 1 --2- 172.21.15.121:0/1518278406 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed60138870 0x7fed601218f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:10.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.360+0000 7fed655e2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:10.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.360+0000 7fed655e2640 1 -- 172.21.15.121:0/1518278406 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fed60124b50 con 0x7fed60138870 2024-09-17T18:23:10.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.360+0000 7fed577fe640 1 -- 172.21.15.121:0/1518278406 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fed5801a070 con 0x7fed6013e1a0 2024-09-17T18:23:10.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.360+0000 7fed5effd640 1 --2- 172.21.15.121:0/1518278406 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed60138870 0x7fed601218f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:10.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.360+0000 7fed5effd640 10 monclient: get_auth_request con 0x7fed60138870 auth_method 0 2024-09-17T18:23:10.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.360+0000 7fed5effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:10.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.360+0000 7fed5effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:10.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.360+0000 7fed5effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:10.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.360+0000 7fed5effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:10.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.360+0000 7fed5effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:10.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.360+0000 7fed5effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:10.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.360+0000 7fed577fe640 1 --2- 172.21.15.121:0/1518278406 >> v2:172.21.15.173:6812/2423786037 conn(0x7fed400401a0 0x7fed40042660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:10.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.360+0000 7fed577fe640 1 -- 172.21.15.121:0/1518278406 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(176..176 src has 1..176) ==== 6557+0+0 (secure 0 0 0) 0x7fed58052f40 con 0x7fed6013e1a0 2024-09-17T18:23:10.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.360+0000 7fed577fe640 10 client.34448.objecter ms_dispatch 0x7fed60000b90 osd_map(176..176 src has 1..176) 2024-09-17T18:23:10.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.360+0000 7fed577fe640 3 client.34448.objecter handle_osd_map got epochs [176,176] > 0 2024-09-17T18:23:10.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.360+0000 7fed577fe640 3 client.34448.objecter handle_osd_map decoding full epoch 176 2024-09-17T18:23:10.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.360+0000 7fed5effd640 10 monclient(hunting): handle_auth_done global_id 24499 payload 824 2024-09-17T18:23:10.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.360+0000 7fed5effd640 1 -- 172.21.15.121:0/1518278406 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fed60078540 con 0x7fed60138870 2024-09-17T18:23:10.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.360+0000 7fed577fe640 20 client.34448.objecter dump_active .. 0 homeless 2024-09-17T18:23:10.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.360+0000 7fed5e7fc640 1 --2- 172.21.15.121:0/1518278406 >> v2:172.21.15.173:6812/2423786037 conn(0x7fed400401a0 0x7fed40042660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:10.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.360+0000 7fed5e7fc640 10 monclient: get_auth_request con 0x7fed400401a0 auth_method 0 2024-09-17T18:23:10.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.363+0000 7fed577fe640 10 client.34448.objecter ms_handle_connect 0x7fed60138870 2024-09-17T18:23:10.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.363+0000 7fed577fe640 10 client.34448.objecter resend_mon_ops 2024-09-17T18:23:10.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.363+0000 7fed5effd640 1 -- 172.21.15.121:0/1518278406 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed60138870 msgr2=0x7fed601218f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:10.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.363+0000 7fed5effd640 1 --2- 172.21.15.121:0/1518278406 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed60138870 0x7fed601218f0 secure :-1 s=READY pgs=7272 cs=0 l=1 rev1=1 crypto rx=0x7fed50000f80 tx=0x7fed50004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:10.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.363+0000 7fed577fe640 1 -- 172.21.15.121:0/1518278406 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed50046440 con 0x7fed60138870 2024-09-17T18:23:10.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.363+0000 7fed577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:10.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.363+0000 7fed577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:23:10.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.363+0000 7fed577fe640 10 monclient: dump: 2024-09-17T18:23:10.381 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:10.381 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:10.381 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.381 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.381 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:10.381 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:10.381 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:10.381 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:10.381 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:10.381 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:10.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.363+0000 7fed577fe640 10 client.34448.objecter ms_handle_connect 0x7fed400401a0 2024-09-17T18:23:10.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.363+0000 7fed5e7fc640 1 --2- 172.21.15.121:0/1518278406 >> v2:172.21.15.173:6812/2423786037 conn(0x7fed400401a0 0x7fed40042660 secure :-1 s=READY pgs=3932 cs=0 l=1 rev1=1 crypto rx=0x7fed60123ca0 tx=0x7fed48007e30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:10.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.363+0000 7fed577fe640 1 -- 172.21.15.121:0/1518278406 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fed50046440 con 0x7fed60138870 2024-09-17T18:23:10.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.363+0000 7fed577fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:23:10.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.363+0000 7fed577fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:23:10.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.363+0000 7fed577fe640 1 -- 172.21.15.121:0/1518278406 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed60138870 msgr2=0x7fed601218f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:10.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.363+0000 7fed577fe640 1 --2- 172.21.15.121:0/1518278406 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed60138870 0x7fed601218f0 secure :-1 s=READY pgs=7272 cs=0 l=1 rev1=1 crypto rx=0x7fed50000f80 tx=0x7fed50004a40 comp rx=0 tx=0).stop 2024-09-17T18:23:10.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.363+0000 7fed577fe640 1 -- 172.21.15.121:0/1518278406 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed60138870 msgr2=0x7fed601218f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:23:10.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.363+0000 7fed577fe640 1 --2- 172.21.15.121:0/1518278406 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed60138870 0x7fed601218f0 unknown :-1 s=CLOSED pgs=7272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:10.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.369+0000 7fed655e2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:23:10.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.369+0000 7fed655e2640 1 --2- 172.21.15.121:0/1518278406 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed60121e30 0x7fed60122240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:10.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.369+0000 7fed655e2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:10.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.369+0000 7fed655e2640 1 -- 172.21.15.121:0/1518278406 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fed60122780 con 0x7fed60121e30 2024-09-17T18:23:10.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.370+0000 7fed5effd640 1 --2- 172.21.15.121:0/1518278406 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed60121e30 0x7fed60122240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:10.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.370+0000 7fed5effd640 10 monclient: get_auth_request con 0x7fed60121e30 auth_method 0 2024-09-17T18:23:10.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.370+0000 7fed5effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:10.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.370+0000 7fed5effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:10.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.370+0000 7fed5effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:10.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.370+0000 7fed5effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:10.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.370+0000 7fed5effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:10.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.370+0000 7fed5effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:10.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.370+0000 7fed5effd640 10 monclient(hunting): handle_auth_done global_id 24502 payload 824 2024-09-17T18:23:10.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.370+0000 7fed5effd640 1 -- 172.21.15.121:0/1518278406 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fed600a87a0 con 0x7fed60121e30 2024-09-17T18:23:10.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.372+0000 7fed5effd640 1 -- 172.21.15.121:0/1518278406 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed60121e30 msgr2=0x7fed60122240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:10.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.372+0000 7fed577fe640 10 client.34448.objecter ms_handle_connect 0x7fed60121e30 2024-09-17T18:23:10.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.372+0000 7fed577fe640 10 client.34448.objecter resend_mon_ops 2024-09-17T18:23:10.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.372+0000 7fed5effd640 1 --2- 172.21.15.121:0/1518278406 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed60121e30 0x7fed60122240 secure :-1 s=READY pgs=7273 cs=0 l=1 rev1=1 crypto rx=0x7fed5002a4c0 tx=0x7fed50030000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:10.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.373+0000 7fed577fe640 1 -- 172.21.15.121:0/1518278406 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed50004240 con 0x7fed60121e30 2024-09-17T18:23:10.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.373+0000 7fed577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:10.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.373+0000 7fed577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:23:10.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.373+0000 7fed577fe640 10 monclient: dump: 2024-09-17T18:23:10.384 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:10.384 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:10.384 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.384 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.384 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:10.384 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:10.384 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:10.384 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:10.384 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:10.384 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:10.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.373+0000 7fed577fe640 1 -- 172.21.15.121:0/1518278406 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7fed5004c040 con 0x7fed60121e30 2024-09-17T18:23:10.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.373+0000 7fed577fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:23:10.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.373+0000 7fed577fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:23:10.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.373+0000 7fed577fe640 1 -- 172.21.15.121:0/1518278406 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed60121e30 msgr2=0x7fed60122240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:10.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.373+0000 7fed577fe640 1 --2- 172.21.15.121:0/1518278406 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed60121e30 0x7fed60122240 secure :-1 s=READY pgs=7273 cs=0 l=1 rev1=1 crypto rx=0x7fed5002a4c0 tx=0x7fed50030000 comp rx=0 tx=0).stop 2024-09-17T18:23:10.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.373+0000 7fed577fe640 1 -- 172.21.15.121:0/1518278406 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed60121e30 msgr2=0x7fed60122240 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:23:10.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.373+0000 7fed577fe640 1 --2- 172.21.15.121:0/1518278406 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed60121e30 0x7fed60122240 secure :-1 s=CLOSED pgs=7273 cs=0 l=1 rev1=1 crypto rx=0x7fed5002a4c0 tx=0x7fed50030000 comp rx=0 tx=0).stop 2024-09-17T18:23:10.385 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:23:10.385 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:23:10.385 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:23:10.385 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:23:10.386 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 20, 2024-09-17T18:23:10.386 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 263319, 2024-09-17T18:23:10.386 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:23:10.386 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:23:10.386 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:23:10.386 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:23:10.386 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:23:10.386 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 0, 2024-09-17T18:23:10.386 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:23:10.386 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:23:10.386 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:23:10.386 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:23:10.386 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:23:10.386 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:23:10.387 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:23:10.387 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:23:10.387 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:23:10.387 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:23:10.387 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:23:10.387 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:23:10.387 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:23:10.387 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:23:10.387 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:23:10.387 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:23:10.387 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:23:10.387 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:23:10.387 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:23:10.388 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:23:10.388 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:23:10.388 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:23:10.388 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:23:10.388 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:23:10.388 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:23:10.388 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:23:10.388 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:23:10.388 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:23:10.388 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:10.388 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:10.388 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:23:10.388 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:23:10.389 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:23:10.389 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:23:10.389 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:23:10.389 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:23:10.389 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:23:10.389 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:23:10.389 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:23:10.389 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:23:10.389 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:23:10.389 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:23:10.389 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:23:10.389 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:23:10.389 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:23:10.390 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:23:10.390 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:23:10.390 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:23:10.390 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:23:10.390 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:23:10.390 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:23:10.390 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:23:10.390 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:23:10.390 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:23:10.390 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:23:10.390 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:23:10.390 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:23:10.390 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:23:10.391 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:23:10.391 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:23:10.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:10.391 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:23:10.391 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:10.391 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:23:10.391 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:23:10.391 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:23:10.391 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:23:10.391 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:10.391 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:23:10.391 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:23:10.391 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:10.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:10.392 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:10.392 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:23:10.392 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:23:10.392 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:10.392 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:10.392 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:10.392 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:10.392 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:23:10.392 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:23:10.392 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:23:10.392 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:23:10.392 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:23:10.392 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:10.393 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:10.393 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:23:10.393 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:23:10.393 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:23:10.393 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:23:10.393 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:10.393 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:23:10.393 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:23:10.393 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:10.393 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:10.393 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:10.393 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:23:10.393 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:23:10.393 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:10.394 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:10.394 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:10.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:10.394 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:23:10.394 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:23:10.394 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:23:10.394 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:23:10.394 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:23:10.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:10.394 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:10.394 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:23:10.394 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:23:10.394 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:23:10.394 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:23:10.395 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:10.395 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:23:10.395 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:23:10.395 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:10.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:10.395 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:10.395 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:23:10.395 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:23:10.395 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:10.395 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:10.395 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:10.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:10.395 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:23:10.395 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:23:10.396 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:23:10.396 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:23:10.396 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:23:10.396 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:10.396 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:10.396 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:10.396 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:23:10.396 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:23:10.396 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:10.396 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:23:10.396 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:23:10.396 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:23:10.396 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:10.397 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:23:10.397 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:23:10.397 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:10.397 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:23:10.397 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:23:10.397 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 3 2024-09-17T18:23:10.397 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:10.397 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:23:10.397 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:23:10.397 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:10.397 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:23:10.397 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:23:10.397 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 8 2024-09-17T18:23:10.397 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:10.398 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:23:10.398 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": [ 2024-09-17T18:23:10.398 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:10.398 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:23:10.398 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:23:10.398 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:23:10.398 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:10.398 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:10.398 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:10.398 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:23:10.398 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:23:10.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.373+0000 7fed655e2640 10 librados: watch_flush enter 2024-09-17T18:23:10.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.374+0000 7fed655e2640 10 librados: watch_flush exit 2024-09-17T18:23:10.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.374+0000 7fed655e2640 20 client.34448.objecter shutdown clearing up homeless session... 2024-09-17T18:23:10.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.374+0000 7fed655e2640 10 client.34448.objecter successfully canceled tick 2024-09-17T18:23:10.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.374+0000 7fed655e2640 1 -- 172.21.15.121:0/1518278406 >> v2:172.21.15.173:6812/2423786037 conn(0x7fed400401a0 msgr2=0x7fed40042660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:10.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.374+0000 7fed655e2640 1 --2- 172.21.15.121:0/1518278406 >> v2:172.21.15.173:6812/2423786037 conn(0x7fed400401a0 0x7fed40042660 secure :-1 s=READY pgs=3932 cs=0 l=1 rev1=1 crypto rx=0x7fed60123ca0 tx=0x7fed48007e30 comp rx=0 tx=0).stop 2024-09-17T18:23:10.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.374+0000 7fed655e2640 10 monclient: shutdown 2024-09-17T18:23:10.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.374+0000 7fed655e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:23:10.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.374+0000 7fed655e2640 1 -- 172.21.15.121:0/1518278406 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed6013e1a0 msgr2=0x7fed60138070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:10.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.374+0000 7fed5f7fe640 1 -- 172.21.15.121:0/1518278406 reap_dead start 2024-09-17T18:23:10.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.374+0000 7fed655e2640 1 --2- 172.21.15.121:0/1518278406 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed6013e1a0 0x7fed60138070 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fed5800c3e0 tx=0x7fed5800c7e0 comp rx=0 tx=0).stop 2024-09-17T18:23:10.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.374+0000 7fed655e2640 1 -- 172.21.15.121:0/1518278406 shutdown_connections 2024-09-17T18:23:10.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.374+0000 7fed655e2640 1 --2- 172.21.15.121:0/1518278406 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed6013e1a0 0x7fed60138070 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:10.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.374+0000 7fed655e2640 1 -- 172.21.15.121:0/1518278406 >> 172.21.15.121:0/1518278406 conn(0x7fed6006dd10 msgr2=0x7fed600a91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:23:10.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.374+0000 7fed655e2640 1 -- 172.21.15.121:0/1518278406 shutdown_connections 2024-09-17T18:23:10.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.374+0000 7fed655e2640 1 -- 172.21.15.121:0/1518278406 wait complete. 2024-09-17T18:23:10.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.374+0000 7fed655e2640 1 librados: shutdown 2024-09-17T18:23:10.401 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:23:10.402 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-17T18:23:10.402 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-17T18:23:10.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.496+0000 7fab42fad640 10 monclient: get_monmap_and_config 2024-09-17T18:23:10.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.496+0000 7fab42fad640 10 monclient: build_initial_monmap 2024-09-17T18:23:10.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.496+0000 7fab42fad640 10 monclient: monmap: 2024-09-17T18:23:10.499 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:23:10.499 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:10.499 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:23:10.497441+0000 2024-09-17T18:23:10.499 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:23:10.497441+0000 2024-09-17T18:23:10.499 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:23:10.499 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:10.499 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:23:10.499 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:23:10.499 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:23:10.499 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:23:10.499 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:23:10.499 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:10.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.497+0000 7fab42fad640 1 Processor -- start 2024-09-17T18:23:10.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.497+0000 7fab42fad640 1 -- start start 2024-09-17T18:23:10.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.497+0000 7fab42fad640 10 monclient: init 2024-09-17T18:23:10.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.498+0000 7fab42fad640 10 monclient: _reopen_session rank -1 2024-09-17T18:23:10.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.498+0000 7fab42fad640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-17T18:23:10.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.498+0000 7fab42fad640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fab3c0a33f0 addr v1:172.21.15.121:6789/0 2024-09-17T18:23:10.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.498+0000 7fab42fad640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fab3c09da70 0x7fab3c09ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:10.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.498+0000 7fab42fad640 10 monclient(hunting): picked mon.noname-a con 0x7fab3c09da70 addr v2:172.21.15.121:3300/0 2024-09-17T18:23:10.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.498+0000 7fab42fad640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab3c13a850 0x7fab3c13cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:10.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.498+0000 7fab42fad640 10 monclient(hunting): picked mon.noname-b con 0x7fab3c13a850 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:23:10.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.498+0000 7fab42fad640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab3c074b40 con 0x7fab3c0a33f0 2024-09-17T18:23:10.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.498+0000 7fab42fad640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:10.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.498+0000 7fab42fad640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fab3c074550 con 0x7fab3c09da70 2024-09-17T18:23:10.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.498+0000 7fab42fad640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:10.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.498+0000 7fab42fad640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fab3c0a37d0 con 0x7fab3c13a850 2024-09-17T18:23:10.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.498+0000 7fab42fad640 10 monclient(hunting): _renew_subs 2024-09-17T18:23:10.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.498+0000 7fab42fad640 10 monclient(hunting): authenticate will time out at 994.214191s 2024-09-17T18:23:10.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.498+0000 7fab4154a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab3c13a850 0x7fab3c13cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:10.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.498+0000 7fab3bfff640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fab3c09da70 0x7fab3c09ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:10.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.498+0000 7fab4154a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab3c13a850 0x7fab3c13cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:52768/0 (socket says 172.21.15.121:52768) 2024-09-17T18:23:10.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.498+0000 7fab4154a640 1 -- 172.21.15.121:0/1382964186 learned_addr learned my addr 172.21.15.121:0/1382964186 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:23:10.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.498+0000 7fab4154a640 10 monclient(hunting): get_auth_request con 0x7fab3c13a850 auth_method 0 2024-09-17T18:23:10.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.498+0000 7fab4154a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:10.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.498+0000 7fab4154a640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:10.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.498+0000 7fab4154a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:10.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.498+0000 7fab3bfff640 10 monclient(hunting): get_auth_request con 0x7fab3c09da70 auth_method 0 2024-09-17T18:23:10.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.498+0000 7fab3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:10.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.498+0000 7fab3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:10.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.498+0000 7fab3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:10.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.498+0000 7fab3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:10.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.498+0000 7fab3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:10.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.498+0000 7fab3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:10.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.498+0000 7fab4154a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:10.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.498+0000 7fab4154a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:10.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.498+0000 7fab4154a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:10.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.498+0000 7fab4154a640 10 monclient(hunting): handle_auth_done global_id 34451 payload 274 2024-09-17T18:23:10.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.498+0000 7fab4154a640 1 -- 172.21.15.121:0/1382964186 >> v2:172.21.15.121:3300/0 conn(0x7fab3c09da70 msgr2=0x7fab3c09ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:23:10.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.498+0000 7fab4154a640 1 --2- 172.21.15.121:0/1382964186 >> v2:172.21.15.121:3300/0 conn(0x7fab3c09da70 0x7fab3c09ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:10.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.498+0000 7fab4154a640 1 -- 172.21.15.121:0/1382964186 >> v1:172.21.15.121:6789/0 conn(0x7fab3c0a33f0 legacy=0x7fab3c09e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:23:10.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.498+0000 7fab4154a640 10 monclient: _finish_hunting 0 2024-09-17T18:23:10.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.498+0000 7fab4154a640 1 monclient: found mon.noname-b 2024-09-17T18:23:10.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.498+0000 7fab4154a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:23:10.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.498+0000 7fab4154a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:23:10.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.498+0000 7fab4154a640 1 -- 172.21.15.121:0/1382964186 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab3c0a13c0 con 0x7fab3c13a850 2024-09-17T18:23:10.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.498+0000 7fab3bfff640 1 --2- 172.21.15.121:0/1382964186 >> v2:172.21.15.121:3300/0 conn(0x7fab3c09da70 0x7fab3c09ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:23:10.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.501+0000 7fab4154a640 1 -- 172.21.15.121:0/1382964186 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab3c13a850 msgr2=0x7fab3c13cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:10.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.501+0000 7fab4154a640 1 --2- 172.21.15.121:0/1382964186 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab3c13a850 0x7fab3c13cc40 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fab34009e50 tx=0x7fab3402a560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=540bf7a1eb2ed7a3 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:10.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.501+0000 7fab3b7fe640 1 -- 172.21.15.121:0/1382964186 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab34036070 con 0x7fab3c13a850 2024-09-17T18:23:10.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.501+0000 7fab3b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:10.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.501+0000 7fab3b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:23:10.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.501+0000 7fab3b7fe640 10 monclient: dump: 2024-09-17T18:23:10.511 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:10.511 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:10.512 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.512 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.512 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:10.512 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:10.512 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:10.512 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:10.512 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:10.512 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:10.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.501+0000 7fab3b7fe640 10 monclient: _finish_auth 0 2024-09-17T18:23:10.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.501+0000 7fab3b7fe640 10 monclient: _check_auth_tickets 2024-09-17T18:23:10.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.501+0000 7fab3b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:23:10.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.501+0000 7fab3b7fe640 1 -- 172.21.15.121:0/1382964186 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab34007800 con 0x7fab3c13a850 2024-09-17T18:23:10.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.501+0000 7fab42fad640 5 monclient: authenticate success, global_id 34451 2024-09-17T18:23:10.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.501+0000 7fab42fad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:23:10.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.501+0000 7fab3b7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:23:10.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.501+0000 7fab42fad640 10 monclient: get_monmap_and_config success 2024-09-17T18:23:10.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.501+0000 7fab3b7fe640 1 -- 172.21.15.121:0/1382964186 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab34007bd0 con 0x7fab3c13a850 2024-09-17T18:23:10.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.501+0000 7fab3b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:10.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.501+0000 7fab42fad640 4 set_mon_vals no callback set 2024-09-17T18:23:10.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.501+0000 7fab3b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:23:10.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.501+0000 7fab3b7fe640 10 monclient: dump: 2024-09-17T18:23:10.513 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:10.513 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:10.513 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.513 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.513 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:10.513 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:10.513 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:10.514 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:10.514 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:10.514 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:10.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.501+0000 7fab42fad640 10 monclient: shutdown 2024-09-17T18:23:10.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.501+0000 7fab42fad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:23:10.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.501+0000 7fab42fad640 1 -- 172.21.15.121:0/1382964186 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab3c13a850 msgr2=0x7fab3c13cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:10.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.501+0000 7fab42fad640 1 --2- 172.21.15.121:0/1382964186 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab3c13a850 0x7fab3c13cc40 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fab34009e50 tx=0x7fab3402a560 comp rx=0 tx=0).stop 2024-09-17T18:23:10.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.501+0000 7fab42fad640 1 -- 172.21.15.121:0/1382964186 shutdown_connections 2024-09-17T18:23:10.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.501+0000 7fab42fad640 1 --2- 172.21.15.121:0/1382964186 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab3c13a850 0x7fab3c13cc40 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:10.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.501+0000 7fab42fad640 1 --2- 172.21.15.121:0/1382964186 >> v2:172.21.15.121:3300/0 conn(0x7fab3c09da70 0x7fab3c09ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:10.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.501+0000 7fab42fad640 1 -- 172.21.15.121:0/1382964186 >> 172.21.15.121:0/1382964186 conn(0x7fab3c0a3b50 msgr2=0x7fab3c0a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:23:10.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.502+0000 7fab42fad640 1 -- 172.21.15.121:0/1382964186 shutdown_connections 2024-09-17T18:23:10.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.502+0000 7fab42fad640 1 -- 172.21.15.121:0/1382964186 wait complete. 2024-09-17T18:23:10.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.502+0000 7fab42fad640 10 monclient: build_initial_monmap 2024-09-17T18:23:10.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.502+0000 7fab42fad640 10 monclient: monmap: 2024-09-17T18:23:10.515 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:23:10.515 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:23:10.515 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:23:10.515 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:23:10.515 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:23:10.515 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:10.515 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:23:10.515 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:23:10.515 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:23:10.515 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:10.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.502+0000 7fab42fad640 1 librados: starting msgr at 2024-09-17T18:23:10.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.502+0000 7fab42fad640 1 librados: starting objecter 2024-09-17T18:23:10.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.502+0000 7fab42fad640 1 Processor -- start 2024-09-17T18:23:10.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.502+0000 7fab42fad640 1 -- start start 2024-09-17T18:23:10.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.502+0000 7fab42fad640 1 librados: setting wanted keys 2024-09-17T18:23:10.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.502+0000 7fab42fad640 1 librados: calling monclient init 2024-09-17T18:23:10.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.502+0000 7fab42fad640 10 monclient: init 2024-09-17T18:23:10.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab42fad640 10 monclient: _reopen_session rank -1 2024-09-17T18:23:10.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab42fad640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T18:23:10.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab42fad640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab3c0a33f0 0x7fab3c11e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:10.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab42fad640 10 monclient(hunting): picked mon.noname-c con 0x7fab3c0a33f0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:23:10.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab42fad640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab3c13a850 0x7fab3c11e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:10.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab42fad640 10 monclient(hunting): picked mon.noname-b con 0x7fab3c13a850 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:23:10.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab42fad640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab3c11eb30 0x7fab3c1c99e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:10.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab42fad640 10 monclient(hunting): picked mon.noname-a con 0x7fab3c11eb30 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:23:10.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab42fad640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:10.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab42fad640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fab3c09f320 con 0x7fab3c11eb30 2024-09-17T18:23:10.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab42fad640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:10.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab42fad640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fab3c09f1a0 con 0x7fab3c13a850 2024-09-17T18:23:10.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab42fad640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:10.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab42fad640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fab3c0a22e0 con 0x7fab3c0a33f0 2024-09-17T18:23:10.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab42fad640 10 monclient(hunting): _renew_subs 2024-09-17T18:23:10.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab42fad640 10 monclient(hunting): authenticate will time out at 994.219631s 2024-09-17T18:23:10.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab40d49640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab3c0a33f0 0x7fab3c11e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:10.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab4154a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab3c11eb30 0x7fab3c1c99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:10.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab40d49640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab3c0a33f0 0x7fab3c11e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:52772/0 (socket says 172.21.15.121:52772) 2024-09-17T18:23:10.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab4154a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab3c11eb30 0x7fab3c1c99e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60164/0 (socket says 172.21.15.121:60164) 2024-09-17T18:23:10.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab40d49640 1 -- 172.21.15.121:0/2222781548 learned_addr learned my addr 172.21.15.121:0/2222781548 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:23:10.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab40d49640 10 monclient(hunting): get_auth_request con 0x7fab3c0a33f0 auth_method 0 2024-09-17T18:23:10.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab40d49640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:10.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab40d49640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:10.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab40d49640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:10.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab4154a640 10 monclient(hunting): get_auth_request con 0x7fab3c11eb30 auth_method 0 2024-09-17T18:23:10.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab4154a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:10.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab4154a640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:10.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab4154a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:10.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab40d49640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:10.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab40d49640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:10.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab40d49640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:10.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab4154a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:10.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab4154a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:10.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab4154a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:10.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab40d49640 10 monclient(hunting): handle_auth_done global_id 34454 payload 824 2024-09-17T18:23:10.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab3bfff640 1 --2- 172.21.15.121:0/2222781548 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab3c13a850 0x7fab3c11e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:10.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab40d49640 1 -- 172.21.15.121:0/2222781548 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab3c13a850 msgr2=0x7fab3c11e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:10.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab40d49640 1 --2- 172.21.15.121:0/2222781548 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab3c13a850 0x7fab3c11e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:10.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab40d49640 1 -- 172.21.15.121:0/2222781548 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab3c11eb30 msgr2=0x7fab3c1c99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:10.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab40d49640 1 --2- 172.21.15.121:0/2222781548 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab3c11eb30 0x7fab3c1c99e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:10.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab40d49640 10 monclient: _finish_hunting 0 2024-09-17T18:23:10.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab40d49640 1 monclient: found mon.noname-c 2024-09-17T18:23:10.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.503+0000 7fab40d49640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:23:10.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.504+0000 7fab40d49640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:23:10.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.504+0000 7fab40d49640 1 -- 172.21.15.121:0/2222781548 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab3c1223a0 con 0x7fab3c0a33f0 2024-09-17T18:23:10.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.504+0000 7fab4154a640 1 --2- 172.21.15.121:0/2222781548 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab3c11eb30 0x7fab3c1c99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:23:10.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.506+0000 7fab40d49640 1 -- 172.21.15.121:0/2222781548 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab3c0a33f0 msgr2=0x7fab3c11e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:10.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.506+0000 7fab40d49640 1 --2- 172.21.15.121:0/2222781548 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab3c0a33f0 0x7fab3c11e0b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fab2c00c430 tx=0x7fab2c00c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:10.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.506+0000 7fab397fa640 10 client.?.objecter ms_handle_connect 0x7fab3c0a33f0 2024-09-17T18:23:10.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.506+0000 7fab397fa640 10 client.?.objecter resend_mon_ops 2024-09-17T18:23:10.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.506+0000 7fab397fa640 1 -- 172.21.15.121:0/2222781548 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab2c0119a0 con 0x7fab3c0a33f0 2024-09-17T18:23:10.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.506+0000 7fab397fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:10.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.506+0000 7fab397fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:23:10.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.506+0000 7fab397fa640 10 monclient: dump: 2024-09-17T18:23:10.520 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:10.520 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:10.520 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.520 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.521 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:10.521 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:10.521 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:10.521 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:10.521 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:10.521 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:10.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.506+0000 7fab397fa640 10 monclient: _finish_auth 0 2024-09-17T18:23:10.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.506+0000 7fab397fa640 10 monclient: _check_auth_tickets 2024-09-17T18:23:10.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.506+0000 7fab397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:23:10.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.506+0000 7fab397fa640 1 -- 172.21.15.121:0/2222781548 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab2c017070 con 0x7fab3c0a33f0 2024-09-17T18:23:10.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.506+0000 7fab42fad640 5 monclient: authenticate success, global_id 34454 2024-09-17T18:23:10.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.506+0000 7fab42fad640 10 monclient: _renew_subs 2024-09-17T18:23:10.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.506+0000 7fab42fad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:23:10.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.506+0000 7fab42fad640 1 -- 172.21.15.121:0/2222781548 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab3c09e240 con 0x7fab3c0a33f0 2024-09-17T18:23:10.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.506+0000 7fab397fa640 10 monclient: handle_config config(0 keys) 2024-09-17T18:23:10.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.506+0000 7fab397fa640 1 -- 172.21.15.121:0/2222781548 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab2c011d80 con 0x7fab3c0a33f0 2024-09-17T18:23:10.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.506+0000 7fab397fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:10.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.506+0000 7fab397fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:23:10.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.506+0000 7fab397fa640 10 monclient: dump: 2024-09-17T18:23:10.522 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:10.522 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:10.522 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.522 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:10.522 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:10.522 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:10.522 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:10.522 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:10.522 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:10.523 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:10.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.506+0000 7fab42fad640 10 client.34454.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:23:10.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.506+0000 7fab42fad640 10 monclient: _renew_subs 2024-09-17T18:23:10.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.506+0000 7fab42fad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:23:10.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.506+0000 7fab42fad640 1 -- 172.21.15.121:0/2222781548 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fab3c121ed0 con 0x7fab3c0a33f0 2024-09-17T18:23:10.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.506+0000 7fab3affd640 4 set_mon_vals no callback set 2024-09-17T18:23:10.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.506+0000 7fab42fad640 1 librados: init done 2024-09-17T18:23:10.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.507+0000 7fab42fad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:23:10.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.507+0000 7fab42fad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:23:10.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.507+0000 7fab42fad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:23:10.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.507+0000 7fab42fad640 1 -- 172.21.15.121:0/2222781548 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab3c0647b0 con 0x7fab3c0a33f0 2024-09-17T18:23:10.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.507+0000 7fab397fa640 1 -- 172.21.15.121:0/2222781548 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fab2c01c460 con 0x7fab3c0a33f0 2024-09-17T18:23:10.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.507+0000 7fab397fa640 1 --2- 172.21.15.121:0/2222781548 >> v2:172.21.15.173:6812/2423786037 conn(0x7fab1c040170 0x7fab1c042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:10.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.507+0000 7fab397fa640 1 -- 172.21.15.121:0/2222781548 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(176..176 src has 1..176) ==== 6557+0+0 (secure 0 0 0) 0x7fab2c053a60 con 0x7fab3c0a33f0 2024-09-17T18:23:10.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.507+0000 7fab397fa640 10 client.34454.objecter ms_dispatch 0x7fab3c000b90 osd_map(176..176 src has 1..176) 2024-09-17T18:23:10.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.507+0000 7fab397fa640 3 client.34454.objecter handle_osd_map got epochs [176,176] > 0 2024-09-17T18:23:10.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.507+0000 7fab397fa640 3 client.34454.objecter handle_osd_map decoding full epoch 176 2024-09-17T18:23:10.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.508+0000 7fab3bfff640 1 --2- 172.21.15.121:0/2222781548 >> v2:172.21.15.173:6812/2423786037 conn(0x7fab1c040170 0x7fab1c042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:10.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.508+0000 7fab3bfff640 10 monclient: get_auth_request con 0x7fab1c040170 auth_method 0 2024-09-17T18:23:10.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.508+0000 7fab397fa640 20 client.34454.objecter dump_active .. 0 homeless 2024-09-17T18:23:10.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.510+0000 7fab3bfff640 1 --2- 172.21.15.121:0/2222781548 >> v2:172.21.15.173:6812/2423786037 conn(0x7fab1c040170 0x7fab1c042630 secure :-1 s=READY pgs=3933 cs=0 l=1 rev1=1 crypto rx=0x7fab3c1220e0 tx=0x7fab24008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:10.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.510+0000 7fab397fa640 10 client.34454.objecter ms_handle_connect 0x7fab1c040170 2024-09-17T18:23:10.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.512+0000 7fab397fa640 1 -- 172.21.15.121:0/2222781548 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fab3c0647b0 con 0x7fab3c0a33f0 2024-09-17T18:23:10.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.512+0000 7fab397fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:23:10.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.512+0000 7fab397fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:23:10.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.753+0000 7fab42fad640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-17T18:23:10.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.753+0000 7fab42fad640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-17T18:23:10.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.753+0000 7fab42fad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:23:10.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.753+0000 7fab42fad640 1 -- 172.21.15.121:0/2222781548 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fab3c0a2810 con 0x7fab3c0a33f0 2024-09-17T18:23:10.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.754+0000 7fab397fa640 1 -- 172.21.15.121:0/2222781548 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fab2c055e20 con 0x7fab3c0a33f0 2024-09-17T18:23:10.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.754+0000 7fab397fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-17T18:23:10.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.754+0000 7fab397fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:23:10.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.756+0000 7fab42fad640 10 librados: watch_flush enter 2024-09-17T18:23:10.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.756+0000 7fab42fad640 10 librados: watch_flush exit 2024-09-17T18:23:10.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.756+0000 7fab42fad640 20 client.34454.objecter shutdown clearing up homeless session... 2024-09-17T18:23:10.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.756+0000 7fab42fad640 10 client.34454.objecter successfully canceled tick 2024-09-17T18:23:10.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.756+0000 7fab42fad640 1 -- 172.21.15.121:0/2222781548 >> v2:172.21.15.173:6812/2423786037 conn(0x7fab1c040170 msgr2=0x7fab1c042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:10.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.756+0000 7fab42fad640 1 --2- 172.21.15.121:0/2222781548 >> v2:172.21.15.173:6812/2423786037 conn(0x7fab1c040170 0x7fab1c042630 secure :-1 s=READY pgs=3933 cs=0 l=1 rev1=1 crypto rx=0x7fab3c1220e0 tx=0x7fab24008040 comp rx=0 tx=0).stop 2024-09-17T18:23:10.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.756+0000 7fab42fad640 10 monclient: shutdown 2024-09-17T18:23:10.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.756+0000 7fab42fad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:23:10.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.756+0000 7fab42fad640 1 -- 172.21.15.121:0/2222781548 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab3c0a33f0 msgr2=0x7fab3c11e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:10.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.756+0000 7fab42fad640 1 --2- 172.21.15.121:0/2222781548 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab3c0a33f0 0x7fab3c11e0b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fab2c00c430 tx=0x7fab2c00c8f0 comp rx=0 tx=0).stop 2024-09-17T18:23:10.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.757+0000 7fab42fad640 1 -- 172.21.15.121:0/2222781548 shutdown_connections 2024-09-17T18:23:10.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.757+0000 7fab42fad640 1 --2- 172.21.15.121:0/2222781548 >> v2:172.21.15.173:6812/2423786037 conn(0x7fab1c040170 0x7fab1c042630 unknown :-1 s=CLOSED pgs=3933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:10.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.757+0000 7fab42fad640 1 --2- 172.21.15.121:0/2222781548 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab3c13a850 0x7fab3c11e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:10.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.757+0000 7fab42fad640 1 --2- 172.21.15.121:0/2222781548 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab3c11eb30 0x7fab3c1c99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:10.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.757+0000 7fab42fad640 1 --2- 172.21.15.121:0/2222781548 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab3c0a33f0 0x7fab3c11e0b0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:10.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.757+0000 7fab42fad640 1 -- 172.21.15.121:0/2222781548 >> 172.21.15.121:0/2222781548 conn(0x7fab3c0a3b50 msgr2=0x7fab3c139b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:23:10.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.757+0000 7fab42fad640 1 -- 172.21.15.121:0/2222781548 shutdown_connections 2024-09-17T18:23:10.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.757+0000 7fab42fad640 1 -- 172.21.15.121:0/2222781548 wait complete. 2024-09-17T18:23:10.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:10.757+0000 7fab42fad640 1 librados: shutdown 2024-09-17T18:23:10.761 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.760+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.762 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.760+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=11,config=2,health=11,kv=13,logm=63} crc {auth=195597266,config=2586702516,health=3476149336,kv=2423095029,logm=1557685387}) 2024-09-17T18:23:10.762 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.760+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=10,config=2,health=12,kv=14,logm=62} crc {auth=1876193646,config=2586702516,health=1082688103,kv=2790417503,logm=1979417474}) 2024-09-17T18:23:10.762 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.760+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.762 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.760+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=11,config=2,health=11,kv=13,logm=63} crc {auth=195597266,config=2586702516,health=3476149336,kv=2423095029,logm=1557685387}) 2024-09-17T18:23:10.762 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.760+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=10,config=2,health=10,kv=13,logm=65} crc {auth=2441666368,config=2586702516,health=1863317135,kv=3128395109,logm=268572959}) 2024-09-17T18:23:10.770 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-17T18:23:10.774 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.772+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.774 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.772+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3156733399}) 2024-09-17T18:23:10.774 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.772+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1339824647}) 2024-09-17T18:23:10.774 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.772+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.774 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.772+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3156733399}) 2024-09-17T18:23:10.775 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.772+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4287600403}) 2024-09-17T18:23:10.775 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.774+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.776 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.774+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1290736671}) 2024-09-17T18:23:10.776 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.774+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2367774072}) 2024-09-17T18:23:10.776 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.774+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.776 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.774+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1290736671}) 2024-09-17T18:23:10.777 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.774+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1356941686}) 2024-09-17T18:23:10.781 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.778+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.781 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.778+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3337978080}) 2024-09-17T18:23:10.781 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.778+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3648385667}) 2024-09-17T18:23:10.781 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.778+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.781 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.778+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3337978080}) 2024-09-17T18:23:10.781 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.778+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1279559768}) 2024-09-17T18:23:10.813 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.812+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.814 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.812+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=25461597}) 2024-09-17T18:23:10.814 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.812+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2527526521}) 2024-09-17T18:23:10.814 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.812+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.814 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.812+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=25461597}) 2024-09-17T18:23:10.814 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.812+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2664079870}) 2024-09-17T18:23:10.815 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.812+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.815 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.812+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1390572391}) 2024-09-17T18:23:10.816 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.812+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1147764323}) 2024-09-17T18:23:10.816 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.812+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.816 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.812+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1390572391}) 2024-09-17T18:23:10.816 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.812+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3781480890}) 2024-09-17T18:23:10.816 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.813+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.816 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.813+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1749630183}) 2024-09-17T18:23:10.816 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.813+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3368280763}) 2024-09-17T18:23:10.816 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.813+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.816 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.813+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1749630183}) 2024-09-17T18:23:10.816 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.813+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3631280638}) 2024-09-17T18:23:10.816 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.814+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.816 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.814+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2214323542}) 2024-09-17T18:23:10.816 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.814+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3515067965}) 2024-09-17T18:23:10.816 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.814+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.817 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.814+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2214323542}) 2024-09-17T18:23:10.817 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.814+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=327515}) 2024-09-17T18:23:10.818 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.814+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.818 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.814+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=98117108}) 2024-09-17T18:23:10.819 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.814+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1456688907}) 2024-09-17T18:23:10.819 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.814+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.819 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.814+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=98117108}) 2024-09-17T18:23:10.819 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.814+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=975945614}) 2024-09-17T18:23:10.819 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.815+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.819 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.815+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=871537279}) 2024-09-17T18:23:10.819 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.815+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2332424808}) 2024-09-17T18:23:10.819 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.815+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.819 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.815+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=871537279}) 2024-09-17T18:23:10.819 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.815+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=650898250}) 2024-09-17T18:23:10.819 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.816+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.819 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.816+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2790493777}) 2024-09-17T18:23:10.819 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.816+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=562959955}) 2024-09-17T18:23:10.819 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.816+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.819 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.816+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2790493777}) 2024-09-17T18:23:10.820 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.816+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2761766382}) 2024-09-17T18:23:10.820 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.817+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.820 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.817+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2642123607}) 2024-09-17T18:23:10.820 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.817+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=119479300}) 2024-09-17T18:23:10.820 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.817+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.820 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.817+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2642123607}) 2024-09-17T18:23:10.820 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.817+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2056842138}) 2024-09-17T18:23:10.822 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.817+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.822 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.817+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2911236179}) 2024-09-17T18:23:10.822 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.817+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4131704252}) 2024-09-17T18:23:10.822 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.817+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.822 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.817+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2911236179}) 2024-09-17T18:23:10.822 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.817+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3947656132}) 2024-09-17T18:23:10.822 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.818+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.822 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.818+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2952665663}) 2024-09-17T18:23:10.823 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.818+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2193090240}) 2024-09-17T18:23:10.823 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.818+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.823 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.818+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2952665663}) 2024-09-17T18:23:10.823 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.818+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=798772873}) 2024-09-17T18:23:10.823 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.819+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.823 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.819+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1470079987}) 2024-09-17T18:23:10.823 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.819+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3921148511}) 2024-09-17T18:23:10.823 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.819+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.823 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.819+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1470079987}) 2024-09-17T18:23:10.823 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.819+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3661245897}) 2024-09-17T18:23:10.823 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.819+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.823 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.819+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2039717335}) 2024-09-17T18:23:10.823 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.819+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3073150156}) 2024-09-17T18:23:10.823 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.819+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.824 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.819+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2039717335}) 2024-09-17T18:23:10.824 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.819+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4236059415}) 2024-09-17T18:23:10.824 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.820+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.824 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.820+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3124366065}) 2024-09-17T18:23:10.824 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.820+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1041853925}) 2024-09-17T18:23:10.824 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.820+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.824 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.820+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3124366065}) 2024-09-17T18:23:10.824 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.820+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=110525762}) 2024-09-17T18:23:10.827 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.821+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.828 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.821+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2879750018}) 2024-09-17T18:23:10.828 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.821+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2491702643}) 2024-09-17T18:23:10.828 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.821+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.828 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.821+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2879750018}) 2024-09-17T18:23:10.828 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.821+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3850085085}) 2024-09-17T18:23:10.828 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.821+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.828 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.821+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1878941875}) 2024-09-17T18:23:10.828 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.821+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=830719165}) 2024-09-17T18:23:10.828 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.821+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.828 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.821+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1878941875}) 2024-09-17T18:23:10.828 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.821+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1679461717}) 2024-09-17T18:23:10.828 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.822+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.828 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.822+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3500520133}) 2024-09-17T18:23:10.829 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.822+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2688438935}) 2024-09-17T18:23:10.829 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.822+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.829 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.822+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3500520133}) 2024-09-17T18:23:10.829 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.822+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=819722096}) 2024-09-17T18:23:10.829 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.823+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.829 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.823+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3935971135}) 2024-09-17T18:23:10.829 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.823+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1931507439}) 2024-09-17T18:23:10.829 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.823+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.829 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.823+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3935971135}) 2024-09-17T18:23:10.829 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.823+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=371904216}) 2024-09-17T18:23:10.829 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.823+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.829 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.823+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=91,mdsmap=2,mgr=7} crc {logm=1373742169,mdsmap=2213087478,mgr=834655845}) 2024-09-17T18:23:10.829 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.823+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=87,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1} crc {logm=2657468809,mdsmap=761013554,mgr=3361498112,mgr_command_descs=1957073259,mgr_metadata=402068389}) 2024-09-17T18:23:10.829 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.823+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.830 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.823+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=91,mdsmap=2,mgr=7} crc {logm=1373742169,mdsmap=2213087478,mgr=834655845}) 2024-09-17T18:23:10.830 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.823+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=90,mdsmap=2,mgr=8} crc {logm=3142029338,mdsmap=2213087478,mgr=3361498112}) 2024-09-17T18:23:10.830 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.824+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.830 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.824+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=21,osdmap=35} crc {mgr_command_descs=1957073259,mgr_metadata=402068389,mgrstat=2251403128,mon_config_key=3728294312,monmap=2277444362,nvmeofgw=3517310312,osd_metadata=121115626,osd_pg_creating=562626564,osd_snap=3593790141,osdmap=249694129}) 2024-09-17T18:23:10.830 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.824+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=22,osdmap=34} crc {mgr_command_descs=1957073259,mgr_metadata=402068389,mgrstat=3741741516,mon_config_key=1582663684,monmap=2277444362,nvmeofgw=598337686,osd_metadata=121115626,osd_pg_creating=562626564,osd_snap=3893480431,osdmap=2886385645}) 2024-09-17T18:23:10.830 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.824+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.830 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.824+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=21,osdmap=35} crc {mgr_command_descs=1957073259,mgr_metadata=402068389,mgrstat=2251403128,mon_config_key=3728294312,monmap=2277444362,nvmeofgw=3517310312,osd_metadata=121115626,osd_pg_creating=562626564,osd_snap=3593790141,osdmap=249694129}) 2024-09-17T18:23:10.830 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.824+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=21,osdmap=36} crc {mgr_command_descs=1957073259,mgr_metadata=402068389,mgrstat=1498718944,mon_config_key=1582663684,monmap=2277444362,nvmeofgw=598337686,osd_metadata=121115626,osd_pg_creating=562626564,osd_snap=2367954316,osdmap=3326574840}) 2024-09-17T18:23:10.830 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.825+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.830 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.825+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2921954145}) 2024-09-17T18:23:10.830 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.825+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2171095741}) 2024-09-17T18:23:10.830 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.825+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.830 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.825+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2921954145}) 2024-09-17T18:23:10.830 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.825+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=748677649}) 2024-09-17T18:23:10.837 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.836+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.838 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.836+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2177335348}) 2024-09-17T18:23:10.838 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.836+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3535730315}) 2024-09-17T18:23:10.838 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.836+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.838 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.836+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2177335348}) 2024-09-17T18:23:10.838 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.836+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=964969098}) 2024-09-17T18:23:10.843 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.841+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.843 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.841+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=40} crc {osdmap=1032951717}) 2024-09-17T18:23:10.843 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.841+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=37} crc {osdmap=3840396511}) 2024-09-17T18:23:10.843 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.841+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:23:10.843 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.841+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=40} crc {osdmap=1032951717}) 2024-09-17T18:23:10.843 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:23:10.841+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=37} crc {osdmap=1963066687}) 2024-09-17T18:23:11.771 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T18:23:11.771 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:23:11.783 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: 7 0 2024-09-17T18:23:11.783 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: 14 0 2024-09-17T18:23:11.783 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: 0 0 2024-09-17T18:23:11.784 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: 15 0 2024-09-17T18:23:11.784 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: 11 0 2024-09-17T18:23:11.784 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: 5 0 2024-09-17T18:23:11.784 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: 8 0 2024-09-17T18:23:11.784 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: 6 0 2024-09-17T18:23:11.784 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: 3 0 2024-09-17T18:23:11.784 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: 4 0 2024-09-17T18:23:11.784 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: 12 0 2024-09-17T18:23:11.784 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: 9 0 2024-09-17T18:23:11.784 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: 2 0 2024-09-17T18:23:11.784 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-09-17T18:23:11.784 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (110 ms) 2024-09-17T18:23:11.784 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: [----------] 3 tests from LibRadosListEC (1228 ms total) 2024-09-17T18:23:11.785 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: 2024-09-17T18:23:11.785 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-09-17T18:23:11.785 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-09-17T18:23:11.785 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (17653 ms) 2024-09-17T18:23:11.785 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: [----------] 1 test from LibRadosListNP (17653 ms total) 2024-09-17T18:23:11.785 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: 2024-09-17T18:23:11.785 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: [----------] Global test environment tear-down 2024-09-17T18:23:11.785 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: [==========] 11 tests from 3 test suites ran. (191066 ms total) 2024-09-17T18:23:11.785 INFO:tasks.workunit.client.0.smithi173.stdout: api_list: [ PASSED ] 11 tests. 2024-09-17T18:23:11.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.870+0000 7f7602fec640 10 monclient: get_monmap_and_config 2024-09-17T18:23:11.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.870+0000 7f7602fec640 10 monclient: build_initial_monmap 2024-09-17T18:23:11.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.870+0000 7f7602fec640 10 monclient: monmap: 2024-09-17T18:23:11.872 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:23:11.872 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:11.872 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:23:11.871104+0000 2024-09-17T18:23:11.872 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:23:11.871104+0000 2024-09-17T18:23:11.873 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:23:11.873 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:11.873 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:23:11.873 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:23:11.873 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:23:11.873 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:23:11.873 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:23:11.873 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:11.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.871+0000 7f7602fec640 1 Processor -- start 2024-09-17T18:23:11.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.871+0000 7f7602fec640 1 -- start start 2024-09-17T18:23:11.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.871+0000 7f7602fec640 10 monclient: init 2024-09-17T18:23:11.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.871+0000 7f7602fec640 10 monclient: _reopen_session rank -1 2024-09-17T18:23:11.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.871+0000 7f7602fec640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-17T18:23:11.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.871+0000 7f7602fec640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f75fc0a0a70 addr v1:172.21.15.121:6789/0 2024-09-17T18:23:11.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.871+0000 7f7602fec640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f75fc1348f0 addr v1:172.21.15.173:6789/0 2024-09-17T18:23:11.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.871+0000 7f7602fec640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f75fc13b330 0x7f75fc13d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:11.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.871+0000 7f7602fec640 10 monclient(hunting): picked mon.noname-a con 0x7f75fc13b330 addr v2:172.21.15.121:3300/0 2024-09-17T18:23:11.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.871+0000 7f7602fec640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75fc09f5b0 con 0x7f75fc0a0a70 2024-09-17T18:23:11.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.871+0000 7f7602fec640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75fc0a1560 con 0x7f75fc1348f0 2024-09-17T18:23:11.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.871+0000 7f7602fec640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:11.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.871+0000 7f7602fec640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f75fc09ef80 con 0x7f75fc13b330 2024-09-17T18:23:11.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.871+0000 7f7602fec640 10 monclient(hunting): _renew_subs 2024-09-17T18:23:11.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.871+0000 7f7602fec640 10 monclient(hunting): authenticate will time out at 995.587779s 2024-09-17T18:23:11.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.871+0000 7f7601589640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f75fc13b330 0x7f75fc13d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:11.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.871+0000 7f7601589640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f75fc13b330 0x7f75fc13d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60194/0 (socket says 172.21.15.121:60194) 2024-09-17T18:23:11.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.871+0000 7f7601589640 1 -- 172.21.15.121:0/3110168530 learned_addr learned my addr 172.21.15.121:0/3110168530 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:23:11.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.871+0000 7f7601589640 10 monclient(hunting): get_auth_request con 0x7f75fc13b330 auth_method 0 2024-09-17T18:23:11.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.871+0000 7f7601589640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:11.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.871+0000 7f7601589640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:11.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.871+0000 7f7601589640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:11.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.872+0000 7f7601589640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:11.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.872+0000 7f7601589640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:11.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.872+0000 7f7601589640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:11.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.874+0000 7f7600d88640 1 -- 172.21.15.121:0/3110168530 >> v1:172.21.15.121:6789/0 conn(0x7f75fc0a0a70 legacy=0x7f75fc0a0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:11.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.874+0000 7f7601589640 10 monclient(hunting): handle_auth_done global_id 44127 payload 274 2024-09-17T18:23:11.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.874+0000 7f7601589640 1 -- 172.21.15.121:0/3110168530 >> v1:172.21.15.173:6789/0 conn(0x7f75fc1348f0 legacy=0x7f75fc139c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:23:11.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.874+0000 7f7601589640 1 -- 172.21.15.121:0/3110168530 >> v1:172.21.15.121:6789/0 conn(0x7f75fc0a0a70 legacy=0x7f75fc0a0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:11.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.874+0000 7f7601589640 10 monclient: _finish_hunting 0 2024-09-17T18:23:11.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.874+0000 7f7601589640 1 monclient: found mon.noname-a 2024-09-17T18:23:11.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.874+0000 7f7601589640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:23:11.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.874+0000 7f7601589640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:23:11.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.874+0000 7f7601589640 1 -- 172.21.15.121:0/3110168530 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75fc09eca0 con 0x7f75fc13b330 2024-09-17T18:23:11.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.877+0000 7f7601589640 1 -- 172.21.15.121:0/3110168530 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75fc13b330 msgr2=0x7f75fc13d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:11.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.877+0000 7f7601589640 1 --2- 172.21.15.121:0/3110168530 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75fc13b330 0x7f75fc13d720 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f75f400d200 tx=0x7f75f402c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d0d628f3f50589f server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:11.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.877+0000 7f75f37fe640 1 -- 172.21.15.121:0/3110168530 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f4039070 con 0x7f75fc13b330 2024-09-17T18:23:11.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.877+0000 7f75f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:11.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.877+0000 7f75f37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:23:11.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.877+0000 7f75f37fe640 10 monclient: dump: 2024-09-17T18:23:11.882 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:11.882 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:11.882 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:11.883 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:11.883 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:11.883 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:11.883 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:11.883 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:11.883 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:11.883 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:11.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.877+0000 7f75f37fe640 10 monclient: _finish_auth 0 2024-09-17T18:23:11.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.877+0000 7f75f37fe640 10 monclient: _check_auth_tickets 2024-09-17T18:23:11.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.877+0000 7f75f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:23:11.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.877+0000 7f7602fec640 5 monclient: authenticate success, global_id 44127 2024-09-17T18:23:11.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.877+0000 7f7602fec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:23:11.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.877+0000 7f75f37fe640 1 -- 172.21.15.121:0/3110168530 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75f4009800 con 0x7f75fc13b330 2024-09-17T18:23:11.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.877+0000 7f75f37fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:23:11.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.877+0000 7f7602fec640 10 monclient: get_monmap_and_config success 2024-09-17T18:23:11.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.877+0000 7f7602fec640 4 set_mon_vals no callback set 2024-09-17T18:23:11.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.877+0000 7f75f37fe640 1 -- 172.21.15.121:0/3110168530 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f4009bd0 con 0x7f75fc13b330 2024-09-17T18:23:11.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.877+0000 7f75f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:11.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.877+0000 7f75f37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:23:11.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.877+0000 7f75f37fe640 10 monclient: dump: 2024-09-17T18:23:11.884 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:11.884 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:11.884 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:11.884 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:11.884 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:11.884 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:11.884 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:11.884 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:11.885 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:11.885 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:11.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.877+0000 7f7602fec640 10 monclient: shutdown 2024-09-17T18:23:11.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.877+0000 7f7602fec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:23:11.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.877+0000 7f7602fec640 1 -- 172.21.15.121:0/3110168530 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75fc13b330 msgr2=0x7f75fc13d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:11.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.877+0000 7f7602fec640 1 --2- 172.21.15.121:0/3110168530 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75fc13b330 0x7f75fc13d720 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f75f400d200 tx=0x7f75f402c8b0 comp rx=0 tx=0).stop 2024-09-17T18:23:11.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.877+0000 7f7602fec640 1 -- 172.21.15.121:0/3110168530 shutdown_connections 2024-09-17T18:23:11.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.877+0000 7f7602fec640 1 --2- 172.21.15.121:0/3110168530 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75fc13b330 0x7f75fc13d720 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:11.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.877+0000 7f7602fec640 1 -- 172.21.15.121:0/3110168530 >> 172.21.15.121:0/3110168530 conn(0x7f75fc0a3b50 msgr2=0x7f75fc0a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:23:11.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.877+0000 7f7602fec640 1 -- 172.21.15.121:0/3110168530 shutdown_connections 2024-09-17T18:23:11.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.877+0000 7f7602fec640 1 -- 172.21.15.121:0/3110168530 wait complete. 2024-09-17T18:23:11.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.878+0000 7f7602fec640 10 monclient: build_initial_monmap 2024-09-17T18:23:11.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.878+0000 7f7602fec640 10 monclient: monmap: 2024-09-17T18:23:11.885 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:23:11.886 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:23:11.886 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:23:11.886 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:23:11.886 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:23:11.886 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:11.886 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:23:11.886 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:23:11.886 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:23:11.886 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:11.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.878+0000 7f7602fec640 1 librados: starting msgr at 2024-09-17T18:23:11.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.878+0000 7f7602fec640 1 librados: starting objecter 2024-09-17T18:23:11.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.878+0000 7f7602fec640 1 Processor -- start 2024-09-17T18:23:11.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.878+0000 7f7602fec640 1 -- start start 2024-09-17T18:23:11.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.878+0000 7f7602fec640 1 librados: setting wanted keys 2024-09-17T18:23:11.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.878+0000 7f7602fec640 1 librados: calling monclient init 2024-09-17T18:23:11.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.878+0000 7f7602fec640 10 monclient: init 2024-09-17T18:23:11.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f7602fec640 10 monclient: _reopen_session rank -1 2024-09-17T18:23:11.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f7602fec640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T18:23:11.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f7602fec640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f75fc0a0a70 0x7f75fc1cd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:11.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f7602fec640 10 monclient(hunting): picked mon.noname-b con 0x7f75fc0a0a70 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:23:11.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f7602fec640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75fc1348f0 0x7f75fc133980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:11.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f7602fec640 10 monclient(hunting): picked mon.noname-a con 0x7f75fc1348f0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:23:11.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f7602fec640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75fc13b330 0x7f75fc072090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:11.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f7602fec640 10 monclient(hunting): picked mon.noname-c con 0x7f75fc13b330 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:23:11.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f7602fec640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:11.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f7602fec640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f75fc09de60 con 0x7f75fc1348f0 2024-09-17T18:23:11.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f7602fec640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:11.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f7602fec640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f75fc09dce0 con 0x7f75fc0a0a70 2024-09-17T18:23:11.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f7602fec640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:11.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f7602fec640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f75fc0a1a50 con 0x7f75fc13b330 2024-09-17T18:23:11.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f7602fec640 10 monclient(hunting): _renew_subs 2024-09-17T18:23:11.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f7602fec640 10 monclient(hunting): authenticate will time out at 995.595349s 2024-09-17T18:23:11.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f75f3fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75fc1348f0 0x7f75fc133980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:11.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f75f3fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75fc1348f0 0x7f75fc133980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60210/0 (socket says 172.21.15.121:60210) 2024-09-17T18:23:11.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f75f3fff640 1 -- 172.21.15.121:0/1889451306 learned_addr learned my addr 172.21.15.121:0/1889451306 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:23:11.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f75f3fff640 10 monclient(hunting): get_auth_request con 0x7f75fc1348f0 auth_method 0 2024-09-17T18:23:11.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f75f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:11.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f75f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:11.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f75f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:11.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f7601589640 1 --2- 172.21.15.121:0/1889451306 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75fc13b330 0x7f75fc072090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:11.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f7601589640 10 monclient(hunting): get_auth_request con 0x7f75fc13b330 auth_method 0 2024-09-17T18:23:11.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f7601589640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:11.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f7601589640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:11.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f7601589640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:11.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f75f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:11.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f75f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:11.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f75f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:11.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f7601589640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:11.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f7600d88640 1 --2- 172.21.15.121:0/1889451306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f75fc0a0a70 0x7f75fc1cd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:11.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f7601589640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:11.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f7601589640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:11.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f7600d88640 10 monclient(hunting): get_auth_request con 0x7f75fc0a0a70 auth_method 0 2024-09-17T18:23:11.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f7600d88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:11.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f7600d88640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:11.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f7600d88640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:11.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f75f3fff640 10 monclient(hunting): handle_auth_done global_id 44130 payload 824 2024-09-17T18:23:11.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f75f3fff640 1 -- 172.21.15.121:0/1889451306 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75fc13b330 msgr2=0x7f75fc072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:11.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f75f3fff640 1 --2- 172.21.15.121:0/1889451306 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75fc13b330 0x7f75fc072090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:11.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f75f3fff640 1 -- 172.21.15.121:0/1889451306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f75fc0a0a70 msgr2=0x7f75fc1cd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:11.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f75f3fff640 1 --2- 172.21.15.121:0/1889451306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f75fc0a0a70 0x7f75fc1cd5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:11.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f75f3fff640 10 monclient: _finish_hunting 0 2024-09-17T18:23:11.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f75f3fff640 1 monclient: found mon.noname-a 2024-09-17T18:23:11.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f75f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:23:11.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f75f3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:23:11.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f75f3fff640 1 -- 172.21.15.121:0/1889451306 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75fc09e240 con 0x7f75fc1348f0 2024-09-17T18:23:11.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.879+0000 7f7600d88640 1 --2- 172.21.15.121:0/1889451306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f75fc0a0a70 0x7f75fc1cd5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T18:23:11.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.881+0000 7f75f3fff640 1 -- 172.21.15.121:0/1889451306 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75fc1348f0 msgr2=0x7f75fc133980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:11.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.881+0000 7f75f3fff640 1 --2- 172.21.15.121:0/1889451306 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75fc1348f0 0x7f75fc133980 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f75ec00c430 tx=0x7f75ec00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:11.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.881+0000 7f75f17fa640 10 client.?.objecter ms_handle_connect 0x7f75fc1348f0 2024-09-17T18:23:11.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.881+0000 7f75f17fa640 10 client.?.objecter resend_mon_ops 2024-09-17T18:23:11.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.882+0000 7f75f17fa640 1 -- 172.21.15.121:0/1889451306 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75ec0119a0 con 0x7f75fc1348f0 2024-09-17T18:23:11.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.882+0000 7f75f17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:11.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.882+0000 7f75f17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:23:11.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.882+0000 7f75f17fa640 10 monclient: dump: 2024-09-17T18:23:11.893 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:11.894 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:11.894 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:11.894 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:11.894 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:11.894 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:11.894 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:11.894 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:11.894 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:11.894 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:11.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.882+0000 7f75f17fa640 10 monclient: _finish_auth 0 2024-09-17T18:23:11.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.882+0000 7f75f17fa640 10 monclient: _check_auth_tickets 2024-09-17T18:23:11.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.882+0000 7f75f17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:23:11.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.882+0000 7f75f17fa640 1 -- 172.21.15.121:0/1889451306 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75ec017070 con 0x7f75fc1348f0 2024-09-17T18:23:11.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.882+0000 7f7602fec640 5 monclient: authenticate success, global_id 44130 2024-09-17T18:23:11.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.882+0000 7f75f17fa640 10 monclient: handle_config config(0 keys) 2024-09-17T18:23:11.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.882+0000 7f75f17fa640 1 -- 172.21.15.121:0/1889451306 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75ec011d80 con 0x7f75fc1348f0 2024-09-17T18:23:11.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.882+0000 7f7602fec640 10 monclient: _renew_subs 2024-09-17T18:23:11.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.882+0000 7f7602fec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:23:11.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.882+0000 7f7602fec640 1 -- 172.21.15.121:0/1889451306 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75fc1ce4f0 con 0x7f75fc1348f0 2024-09-17T18:23:11.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.882+0000 7f75f17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:11.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.882+0000 7f75f17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:23:11.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.882+0000 7f75f17fa640 10 monclient: dump: 2024-09-17T18:23:11.895 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:11.895 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:11.895 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:11.895 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:11.895 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:11.896 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:11.896 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:11.896 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:11.896 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:11.896 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:11.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.882+0000 7f75f27fc640 4 set_mon_vals no callback set 2024-09-17T18:23:11.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.882+0000 7f7602fec640 10 client.44130.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:23:11.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.882+0000 7f7602fec640 10 monclient: _renew_subs 2024-09-17T18:23:11.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.882+0000 7f7602fec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:23:11.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.882+0000 7f7602fec640 1 -- 172.21.15.121:0/1889451306 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75fc123080 con 0x7f75fc1348f0 2024-09-17T18:23:11.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.882+0000 7f7602fec640 1 librados: init done 2024-09-17T18:23:11.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.883+0000 7f7602fec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:23:11.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.883+0000 7f7602fec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:23:11.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.883+0000 7f7602fec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:23:11.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.883+0000 7f7602fec640 1 -- 172.21.15.121:0/1889451306 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75fc121b60 con 0x7f75fc1348f0 2024-09-17T18:23:11.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.883+0000 7f75f17fa640 1 -- 172.21.15.121:0/1889451306 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f75ec01c460 con 0x7f75fc1348f0 2024-09-17T18:23:11.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.883+0000 7f75f17fa640 1 --2- 172.21.15.121:0/1889451306 >> v2:172.21.15.173:6812/2423786037 conn(0x7f75e8040170 0x7f75e8042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:11.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.883+0000 7f75f17fa640 1 -- 172.21.15.121:0/1889451306 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(178..178 src has 1..178) ==== 5795+0+0 (secure 0 0 0) 0x7f75ec053750 con 0x7f75fc1348f0 2024-09-17T18:23:11.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.883+0000 7f75f17fa640 10 client.44130.objecter ms_dispatch 0x7f75fc000b90 osd_map(178..178 src has 1..178) 2024-09-17T18:23:11.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.883+0000 7f75f17fa640 3 client.44130.objecter handle_osd_map got epochs [178,178] > 0 2024-09-17T18:23:11.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.883+0000 7f75f17fa640 3 client.44130.objecter handle_osd_map decoding full epoch 178 2024-09-17T18:23:11.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.883+0000 7f75f17fa640 20 client.44130.objecter dump_active .. 0 homeless 2024-09-17T18:23:11.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.883+0000 7f7600d88640 1 --2- 172.21.15.121:0/1889451306 >> v2:172.21.15.173:6812/2423786037 conn(0x7f75e8040170 0x7f75e8042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:11.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.884+0000 7f7600d88640 10 monclient: get_auth_request con 0x7f75e8040170 auth_method 0 2024-09-17T18:23:11.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.886+0000 7f75f17fa640 10 client.44130.objecter ms_handle_connect 0x7f75e8040170 2024-09-17T18:23:11.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.886+0000 7f7600d88640 1 --2- 172.21.15.121:0/1889451306 >> v2:172.21.15.173:6812/2423786037 conn(0x7f75e8040170 0x7f75e8042630 secure :-1 s=READY pgs=3934 cs=0 l=1 rev1=1 crypto rx=0x7f75f8000f60 tx=0x7f75f8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:11.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.888+0000 7f75f17fa640 1 -- 172.21.15.121:0/1889451306 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f75fc09e240 con 0x7f75fc1348f0 2024-09-17T18:23:11.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.888+0000 7f75f17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:23:11.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:11.888+0000 7f75f17fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:23:12.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.141+0000 7f7602fec640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:23:12.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.141+0000 7f7602fec640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:23:12.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.141+0000 7f7602fec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:23:12.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.141+0000 7f7602fec640 1 -- 172.21.15.121:0/1889451306 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f75fc0a5130 con 0x7f75fc1348f0 2024-09-17T18:23:12.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.142+0000 7f75f17fa640 1 -- 172.21.15.121:0/1889451306 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f75ec051d50 con 0x7f75fc1348f0 2024-09-17T18:23:12.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.142+0000 7f75f17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:23:12.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.142+0000 7f75f17fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:23:12.144 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:23:12.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.144+0000 7f7602fec640 10 librados: watch_flush enter 2024-09-17T18:23:12.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.144+0000 7f7602fec640 10 librados: watch_flush exit 2024-09-17T18:23:12.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.144+0000 7f7602fec640 20 client.44130.objecter shutdown clearing up homeless session... 2024-09-17T18:23:12.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.144+0000 7f7602fec640 10 client.44130.objecter successfully canceled tick 2024-09-17T18:23:12.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.144+0000 7f7602fec640 1 -- 172.21.15.121:0/1889451306 >> v2:172.21.15.173:6812/2423786037 conn(0x7f75e8040170 msgr2=0x7f75e8042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:12.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.144+0000 7f7602fec640 1 --2- 172.21.15.121:0/1889451306 >> v2:172.21.15.173:6812/2423786037 conn(0x7f75e8040170 0x7f75e8042630 secure :-1 s=READY pgs=3934 cs=0 l=1 rev1=1 crypto rx=0x7f75f8000f60 tx=0x7f75f8009040 comp rx=0 tx=0).stop 2024-09-17T18:23:12.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.144+0000 7f7602fec640 10 monclient: shutdown 2024-09-17T18:23:12.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.144+0000 7f7602fec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:23:12.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.144+0000 7f7602fec640 1 -- 172.21.15.121:0/1889451306 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75fc1348f0 msgr2=0x7f75fc133980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:12.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.144+0000 7f7602fec640 1 --2- 172.21.15.121:0/1889451306 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75fc1348f0 0x7f75fc133980 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f75ec00c430 tx=0x7f75ec00c8f0 comp rx=0 tx=0).stop 2024-09-17T18:23:12.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.144+0000 7f7602fec640 1 -- 172.21.15.121:0/1889451306 shutdown_connections 2024-09-17T18:23:12.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.144+0000 7f7602fec640 1 --2- 172.21.15.121:0/1889451306 >> v2:172.21.15.173:6812/2423786037 conn(0x7f75e8040170 0x7f75e8042630 unknown :-1 s=CLOSED pgs=3934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:12.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.144+0000 7f7602fec640 1 --2- 172.21.15.121:0/1889451306 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75fc13b330 0x7f75fc072090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:12.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.144+0000 7f7602fec640 1 --2- 172.21.15.121:0/1889451306 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75fc1348f0 0x7f75fc133980 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:12.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.144+0000 7f7602fec640 1 --2- 172.21.15.121:0/1889451306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f75fc0a0a70 0x7f75fc1cd5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:12.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.144+0000 7f7602fec640 1 -- 172.21.15.121:0/1889451306 >> 172.21.15.121:0/1889451306 conn(0x7f75fc0a3b50 msgr2=0x7f75fc13aae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:23:12.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.144+0000 7f7602fec640 1 -- 172.21.15.121:0/1889451306 shutdown_connections 2024-09-17T18:23:12.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.144+0000 7f7602fec640 1 -- 172.21.15.121:0/1889451306 wait complete. 2024-09-17T18:23:12.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.145+0000 7f7602fec640 1 librados: shutdown 2024-09-17T18:23:12.157 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T18:23:12.157 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-17T18:23:12.157 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T18:23:12.157 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T18:23:12.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.253+0000 7f645bf73640 10 monclient: get_monmap_and_config 2024-09-17T18:23:12.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.253+0000 7f645bf73640 10 monclient: build_initial_monmap 2024-09-17T18:23:12.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.253+0000 7f645bf73640 10 monclient: monmap: 2024-09-17T18:23:12.255 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:23:12.255 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:12.255 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:23:12.253875+0000 2024-09-17T18:23:12.255 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:23:12.253875+0000 2024-09-17T18:23:12.255 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:23:12.255 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:12.255 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:23:12.256 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:23:12.256 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:23:12.256 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:23:12.256 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:23:12.256 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:12.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.253+0000 7f645bf73640 1 Processor -- start 2024-09-17T18:23:12.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.254+0000 7f645bf73640 1 -- start start 2024-09-17T18:23:12.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.254+0000 7f645bf73640 10 monclient: init 2024-09-17T18:23:12.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.254+0000 7f645bf73640 10 monclient: _reopen_session rank -1 2024-09-17T18:23:12.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.254+0000 7f645bf73640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-17T18:23:12.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.254+0000 7f645bf73640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f64540a0a00 0x7f64540a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:12.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.254+0000 7f645bf73640 10 monclient(hunting): picked mon.noname-a con 0x7f64540a0a00 addr v2:172.21.15.121:3300/0 2024-09-17T18:23:12.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.254+0000 7f645bf73640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f64540a1320 0x7f6454134980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:12.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.254+0000 7f645bf73640 10 monclient(hunting): picked mon.noname-c con 0x7f64540a1320 addr v2:172.21.15.173:3300/0 2024-09-17T18:23:12.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.254+0000 7f645bf73640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6454134ec0 addr v1:172.21.15.173:6789/0 2024-09-17T18:23:12.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.254+0000 7f645bf73640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64540a3820 con 0x7f6454134ec0 2024-09-17T18:23:12.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.254+0000 7f645bf73640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:12.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.254+0000 7f645bf73640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f64540a2ea0 con 0x7f64540a0a00 2024-09-17T18:23:12.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.254+0000 7f645bf73640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:12.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.254+0000 7f645bf73640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f645409e800 con 0x7f64540a1320 2024-09-17T18:23:12.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.254+0000 7f645bf73640 10 monclient(hunting): _renew_subs 2024-09-17T18:23:12.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.254+0000 7f645bf73640 10 monclient(hunting): authenticate will time out at 995.970530s 2024-09-17T18:23:12.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.254+0000 7f6459d0f640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f64540a0a00 0x7f64540a0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:12.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.254+0000 7f6459d0f640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f64540a0a00 0x7f64540a0de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60224/0 (socket says 172.21.15.121:60224) 2024-09-17T18:23:12.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.254+0000 7f6459d0f640 1 -- 172.21.15.121:0/2238596510 learned_addr learned my addr 172.21.15.121:0/2238596510 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:23:12.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.254+0000 7f6459d0f640 10 monclient(hunting): get_auth_request con 0x7f64540a0a00 auth_method 0 2024-09-17T18:23:12.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.254+0000 7f6459d0f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:12.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.254+0000 7f6459d0f640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:12.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.254+0000 7f6459d0f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:12.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.254+0000 7f6459d0f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:12.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.254+0000 7f6459d0f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:12.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.254+0000 7f6459d0f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:12.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.254+0000 7f645950e640 1 --2- 172.21.15.121:0/2238596510 >> v2:172.21.15.173:3300/0 conn(0x7f64540a1320 0x7f6454134980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:12.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.254+0000 7f645950e640 10 monclient(hunting): get_auth_request con 0x7f64540a1320 auth_method 0 2024-09-17T18:23:12.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.254+0000 7f645950e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:12.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.254+0000 7f645950e640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:12.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.254+0000 7f645950e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:12.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.254+0000 7f6459d0f640 10 monclient(hunting): handle_auth_done global_id 44133 payload 274 2024-09-17T18:23:12.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.254+0000 7f6459d0f640 1 -- 172.21.15.121:0/2238596510 >> v2:172.21.15.173:3300/0 conn(0x7f64540a1320 msgr2=0x7f6454134980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:23:12.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.254+0000 7f6459d0f640 1 --2- 172.21.15.121:0/2238596510 >> v2:172.21.15.173:3300/0 conn(0x7f64540a1320 0x7f6454134980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:12.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.254+0000 7f6459d0f640 1 -- 172.21.15.121:0/2238596510 >> v1:172.21.15.173:6789/0 conn(0x7f6454134ec0 legacy=0x7f645413bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:23:12.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.254+0000 7f6459d0f640 10 monclient: _finish_hunting 0 2024-09-17T18:23:12.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.254+0000 7f6459d0f640 1 monclient: found mon.noname-a 2024-09-17T18:23:12.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.254+0000 7f6459d0f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:23:12.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.254+0000 7f6459d0f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:23:12.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.254+0000 7f6459d0f640 1 -- 172.21.15.121:0/2238596510 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64540a2c30 con 0x7f64540a0a00 2024-09-17T18:23:12.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.257+0000 7f6459d0f640 1 -- 172.21.15.121:0/2238596510 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64540a0a00 msgr2=0x7f64540a0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:12.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.257+0000 7f6459d0f640 1 --2- 172.21.15.121:0/2238596510 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64540a0a00 0x7f64540a0de0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f645409e1d0 tx=0x7f644802a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad8d3de5f18355b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:12.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.257+0000 7f6458d0d640 1 -- 172.21.15.121:0/2238596510 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6448037070 con 0x7f64540a0a00 2024-09-17T18:23:12.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.257+0000 7f6458d0d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:12.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.257+0000 7f6458d0d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:23:12.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.257+0000 7f6458d0d640 10 monclient: dump: 2024-09-17T18:23:12.269 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:12.270 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:12.270 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.271 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.271 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:12.271 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:12.271 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:12.271 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:12.271 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:12.271 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:12.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.257+0000 7f6458d0d640 10 monclient: _finish_auth 0 2024-09-17T18:23:12.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.257+0000 7f6458d0d640 10 monclient: _check_auth_tickets 2024-09-17T18:23:12.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.257+0000 7f6458d0d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:23:12.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.257+0000 7f645bf73640 5 monclient: authenticate success, global_id 44133 2024-09-17T18:23:12.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.257+0000 7f6458d0d640 1 -- 172.21.15.121:0/2238596510 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6448032640 con 0x7f64540a0a00 2024-09-17T18:23:12.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.257+0000 7f645bf73640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:23:12.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.257+0000 7f6458d0d640 10 monclient: handle_config config(0 keys) 2024-09-17T18:23:12.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.257+0000 7f645bf73640 10 monclient: get_monmap_and_config success 2024-09-17T18:23:12.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.257+0000 7f6458d0d640 1 -- 172.21.15.121:0/2238596510 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6448032a10 con 0x7f64540a0a00 2024-09-17T18:23:12.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.257+0000 7f6458d0d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:12.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.257+0000 7f645bf73640 4 set_mon_vals no callback set 2024-09-17T18:23:12.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.257+0000 7f6458d0d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:23:12.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.257+0000 7f6458d0d640 10 monclient: dump: 2024-09-17T18:23:12.272 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:12.272 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:12.272 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.272 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.272 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:12.272 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:12.273 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:12.273 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:12.273 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:12.273 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:12.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.258+0000 7f645bf73640 10 monclient: shutdown 2024-09-17T18:23:12.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.258+0000 7f645bf73640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:23:12.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.258+0000 7f645bf73640 1 -- 172.21.15.121:0/2238596510 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64540a0a00 msgr2=0x7f64540a0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:12.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.258+0000 7f645bf73640 1 --2- 172.21.15.121:0/2238596510 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64540a0a00 0x7f64540a0de0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f645409e1d0 tx=0x7f644802a0b0 comp rx=0 tx=0).stop 2024-09-17T18:23:12.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.258+0000 7f645bf73640 1 -- 172.21.15.121:0/2238596510 shutdown_connections 2024-09-17T18:23:12.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.258+0000 7f645bf73640 1 --2- 172.21.15.121:0/2238596510 >> v2:172.21.15.173:3300/0 conn(0x7f64540a1320 0x7f6454134980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:12.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.258+0000 7f645bf73640 1 --2- 172.21.15.121:0/2238596510 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64540a0a00 0x7f64540a0de0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:12.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.258+0000 7f645bf73640 1 -- 172.21.15.121:0/2238596510 >> 172.21.15.121:0/2238596510 conn(0x7f64540a7830 msgr2=0x7f64540a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:23:12.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.258+0000 7f645bf73640 1 -- 172.21.15.121:0/2238596510 shutdown_connections 2024-09-17T18:23:12.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.258+0000 7f645bf73640 1 -- 172.21.15.121:0/2238596510 wait complete. 2024-09-17T18:23:12.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.258+0000 7f645bf73640 10 monclient: build_initial_monmap 2024-09-17T18:23:12.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.258+0000 7f645bf73640 10 monclient: monmap: 2024-09-17T18:23:12.274 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:23:12.274 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:23:12.274 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:23:12.274 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:23:12.274 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:23:12.274 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:12.274 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:23:12.274 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:23:12.274 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:23:12.275 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:12.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.259+0000 7f645bf73640 1 librados: starting msgr at 2024-09-17T18:23:12.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.259+0000 7f645bf73640 1 librados: starting objecter 2024-09-17T18:23:12.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.259+0000 7f645bf73640 1 Processor -- start 2024-09-17T18:23:12.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.259+0000 7f645bf73640 1 -- start start 2024-09-17T18:23:12.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.259+0000 7f645bf73640 1 librados: setting wanted keys 2024-09-17T18:23:12.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.259+0000 7f645bf73640 1 librados: calling monclient init 2024-09-17T18:23:12.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.259+0000 7f645bf73640 10 monclient: init 2024-09-17T18:23:12.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.259+0000 7f645bf73640 10 monclient: _reopen_session rank -1 2024-09-17T18:23:12.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.259+0000 7f645bf73640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T18:23:12.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.259+0000 7f645bf73640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64540a0a00 0x7f645413dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:12.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.259+0000 7f645bf73640 10 monclient(hunting): picked mon.noname-b con 0x7f64540a0a00 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:23:12.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.259+0000 7f645bf73640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64540a1320 0x7f6454131cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:12.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.259+0000 7f645bf73640 10 monclient(hunting): picked mon.noname-a con 0x7f64540a1320 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:23:12.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.259+0000 7f645bf73640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6454134ec0 0x7f6454132200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:12.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.259+0000 7f645bf73640 10 monclient(hunting): picked mon.noname-c con 0x7f6454134ec0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:23:12.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.259+0000 7f645bf73640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:12.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.259+0000 7f645bf73640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f645409de60 con 0x7f64540a1320 2024-09-17T18:23:12.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.259+0000 7f645bf73640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:12.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.259+0000 7f645bf73640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f645409dce0 con 0x7f64540a0a00 2024-09-17T18:23:12.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.259+0000 7f645bf73640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:12.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.259+0000 7f645bf73640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f64540a3120 con 0x7f6454134ec0 2024-09-17T18:23:12.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.259+0000 7f645bf73640 10 monclient(hunting): _renew_subs 2024-09-17T18:23:12.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.259+0000 7f645bf73640 10 monclient(hunting): authenticate will time out at 995.975774s 2024-09-17T18:23:12.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.259+0000 7f645950e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64540a1320 0x7f6454131cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:12.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.259+0000 7f645950e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64540a1320 0x7f6454131cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60240/0 (socket says 172.21.15.121:60240) 2024-09-17T18:23:12.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.259+0000 7f645950e640 1 -- 172.21.15.121:0/2391312737 learned_addr learned my addr 172.21.15.121:0/2391312737 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:23:12.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.259+0000 7f645950e640 10 monclient(hunting): get_auth_request con 0x7f64540a1320 auth_method 0 2024-09-17T18:23:12.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.259+0000 7f645950e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:12.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.259+0000 7f645950e640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:12.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.259+0000 7f645950e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:12.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.259+0000 7f645a510640 1 --2- 172.21.15.121:0/2391312737 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6454134ec0 0x7f6454132200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:12.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.259+0000 7f645a510640 10 monclient(hunting): get_auth_request con 0x7f6454134ec0 auth_method 0 2024-09-17T18:23:12.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.259+0000 7f645a510640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:12.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.259+0000 7f645a510640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:12.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.259+0000 7f645a510640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:12.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.259+0000 7f645950e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:12.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.259+0000 7f645950e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:12.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.259+0000 7f645950e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:12.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.259+0000 7f645a510640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:12.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.259+0000 7f645a510640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:12.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.260+0000 7f645a510640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:12.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.260+0000 7f6459d0f640 1 --2- 172.21.15.121:0/2391312737 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64540a0a00 0x7f645413dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:12.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.260+0000 7f6459d0f640 10 monclient(hunting): get_auth_request con 0x7f64540a0a00 auth_method 0 2024-09-17T18:23:12.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.260+0000 7f6459d0f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:12.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.260+0000 7f6459d0f640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:12.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.260+0000 7f6459d0f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:12.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.260+0000 7f645950e640 10 monclient(hunting): handle_auth_done global_id 44136 payload 824 2024-09-17T18:23:12.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.260+0000 7f645950e640 1 -- 172.21.15.121:0/2391312737 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6454134ec0 msgr2=0x7f6454132200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:12.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.260+0000 7f645950e640 1 --2- 172.21.15.121:0/2391312737 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6454134ec0 0x7f6454132200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:12.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.260+0000 7f645950e640 1 -- 172.21.15.121:0/2391312737 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64540a0a00 msgr2=0x7f645413dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:12.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.260+0000 7f645950e640 1 --2- 172.21.15.121:0/2391312737 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64540a0a00 0x7f645413dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:12.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.260+0000 7f645950e640 10 monclient: _finish_hunting 0 2024-09-17T18:23:12.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.260+0000 7f645950e640 1 monclient: found mon.noname-a 2024-09-17T18:23:12.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.260+0000 7f645950e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:23:12.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.260+0000 7f645950e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:23:12.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.260+0000 7f645950e640 1 -- 172.21.15.121:0/2391312737 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6454133540 con 0x7f64540a1320 2024-09-17T18:23:12.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.260+0000 7f6459d0f640 1 --2- 172.21.15.121:0/2391312737 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64540a0a00 0x7f645413dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T18:23:12.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.262+0000 7f645950e640 1 -- 172.21.15.121:0/2391312737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64540a1320 msgr2=0x7f6454131cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:12.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.262+0000 7f645950e640 1 --2- 172.21.15.121:0/2391312737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64540a1320 0x7f6454131cc0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f645000d800 tx=0x7f645000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:12.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.262+0000 7f64467fc640 10 client.?.objecter ms_handle_connect 0x7f64540a1320 2024-09-17T18:23:12.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.262+0000 7f64467fc640 10 client.?.objecter resend_mon_ops 2024-09-17T18:23:12.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.262+0000 7f64467fc640 1 -- 172.21.15.121:0/2391312737 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f645000b8d0 con 0x7f64540a1320 2024-09-17T18:23:12.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.262+0000 7f64467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:12.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.262+0000 7f64467fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:23:12.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.262+0000 7f64467fc640 10 monclient: dump: 2024-09-17T18:23:12.279 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:12.280 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:12.280 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.280 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.280 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:12.280 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:12.280 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:12.280 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:12.280 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:12.280 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:12.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.262+0000 7f64467fc640 10 monclient: _finish_auth 0 2024-09-17T18:23:12.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.262+0000 7f64467fc640 10 monclient: _check_auth_tickets 2024-09-17T18:23:12.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.262+0000 7f64467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:23:12.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.262+0000 7f64467fc640 1 -- 172.21.15.121:0/2391312737 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6450019070 con 0x7f64540a1320 2024-09-17T18:23:12.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.262+0000 7f645bf73640 5 monclient: authenticate success, global_id 44136 2024-09-17T18:23:12.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.262+0000 7f645bf73640 10 monclient: _renew_subs 2024-09-17T18:23:12.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.262+0000 7f645bf73640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:23:12.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.262+0000 7f645bf73640 1 -- 172.21.15.121:0/2391312737 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64541c68f0 con 0x7f64540a1320 2024-09-17T18:23:12.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.262+0000 7f64467fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:23:12.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.262+0000 7f64467fc640 1 -- 172.21.15.121:0/2391312737 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f645000bc90 con 0x7f64540a1320 2024-09-17T18:23:12.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.262+0000 7f64467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:12.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.262+0000 7f64467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:23:12.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.262+0000 7f64467fc640 10 monclient: dump: 2024-09-17T18:23:12.281 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:12.281 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:12.281 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.281 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.282 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:12.282 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:12.282 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:12.282 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:12.282 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:12.282 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:12.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.262+0000 7f645bf73640 10 client.44136.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:23:12.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.262+0000 7f6447fff640 4 set_mon_vals no callback set 2024-09-17T18:23:12.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.262+0000 7f645bf73640 10 monclient: _renew_subs 2024-09-17T18:23:12.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.262+0000 7f645bf73640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:23:12.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.262+0000 7f645bf73640 1 -- 172.21.15.121:0/2391312737 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6454135be0 con 0x7f64540a1320 2024-09-17T18:23:12.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.262+0000 7f645bf73640 1 librados: init done 2024-09-17T18:23:12.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.263+0000 7f645bf73640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:23:12.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.263+0000 7f645bf73640 1 --2- 172.21.15.121:0/2391312737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f645413e1c0 0x7f6454132740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:12.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.263+0000 7f645bf73640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:12.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.263+0000 7f645bf73640 1 -- 172.21.15.121:0/2391312737 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f64540a2030 con 0x7f645413e1c0 2024-09-17T18:23:12.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.263+0000 7f64467fc640 1 -- 172.21.15.121:0/2391312737 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f645001e460 con 0x7f64540a1320 2024-09-17T18:23:12.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.263+0000 7f6459d0f640 1 --2- 172.21.15.121:0/2391312737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f645413e1c0 0x7f6454132740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:12.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.263+0000 7f6459d0f640 10 monclient: get_auth_request con 0x7f645413e1c0 auth_method 0 2024-09-17T18:23:12.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.263+0000 7f6459d0f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:12.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.263+0000 7f6459d0f640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:12.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.263+0000 7f6459d0f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:12.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.263+0000 7f6459d0f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:12.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.263+0000 7f6459d0f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:12.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.263+0000 7f6459d0f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:12.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.263+0000 7f64467fc640 1 --2- 172.21.15.121:0/2391312737 >> v2:172.21.15.173:6812/2423786037 conn(0x7f64380401a0 0x7f6438042660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:12.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.263+0000 7f6459d0f640 10 monclient(hunting): handle_auth_done global_id 44139 payload 824 2024-09-17T18:23:12.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.263+0000 7f6459d0f640 1 -- 172.21.15.121:0/2391312737 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f64540008d0 con 0x7f645413e1c0 2024-09-17T18:23:12.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.263+0000 7f64467fc640 1 -- 172.21.15.121:0/2391312737 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(178..178 src has 1..178) ==== 5795+0+0 (secure 0 0 0) 0x7f6450053dd0 con 0x7f64540a1320 2024-09-17T18:23:12.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.263+0000 7f64467fc640 10 client.44136.objecter ms_dispatch 0x7f6454000b90 osd_map(178..178 src has 1..178) 2024-09-17T18:23:12.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.263+0000 7f64467fc640 3 client.44136.objecter handle_osd_map got epochs [178,178] > 0 2024-09-17T18:23:12.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.263+0000 7f64467fc640 3 client.44136.objecter handle_osd_map decoding full epoch 178 2024-09-17T18:23:12.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.263+0000 7f64467fc640 20 client.44136.objecter dump_active .. 0 homeless 2024-09-17T18:23:12.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.263+0000 7f645a510640 1 --2- 172.21.15.121:0/2391312737 >> v2:172.21.15.173:6812/2423786037 conn(0x7f64380401a0 0x7f6438042660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:12.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.264+0000 7f645a510640 10 monclient: get_auth_request con 0x7f64380401a0 auth_method 0 2024-09-17T18:23:12.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.265+0000 7f64467fc640 10 client.44136.objecter ms_handle_connect 0x7f645413e1c0 2024-09-17T18:23:12.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.265+0000 7f64467fc640 10 client.44136.objecter resend_mon_ops 2024-09-17T18:23:12.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.265+0000 7f6459d0f640 1 -- 172.21.15.121:0/2391312737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f645413e1c0 msgr2=0x7f6454132740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:12.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.265+0000 7f6459d0f640 1 --2- 172.21.15.121:0/2391312737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f645413e1c0 0x7f6454132740 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6454077930 tx=0x7f64480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:12.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.266+0000 7f64467fc640 1 -- 172.21.15.121:0/2391312737 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6448042070 con 0x7f645413e1c0 2024-09-17T18:23:12.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.266+0000 7f64467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:12.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.266+0000 7f64467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:23:12.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.266+0000 7f64467fc640 10 monclient: dump: 2024-09-17T18:23:12.285 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:12.285 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:12.285 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.285 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.285 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:12.285 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:12.285 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:12.285 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:12.285 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:12.285 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:12.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.266+0000 7f64467fc640 1 -- 172.21.15.121:0/2391312737 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f644803e660 con 0x7f645413e1c0 2024-09-17T18:23:12.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.266+0000 7f64467fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:23:12.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.266+0000 7f64467fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:23:12.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.266+0000 7f64467fc640 1 -- 172.21.15.121:0/2391312737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f645413e1c0 msgr2=0x7f6454132740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:12.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.266+0000 7f64467fc640 1 --2- 172.21.15.121:0/2391312737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f645413e1c0 0x7f6454132740 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6454077930 tx=0x7f64480046c0 comp rx=0 tx=0).stop 2024-09-17T18:23:12.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.266+0000 7f64467fc640 1 -- 172.21.15.121:0/2391312737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f645413e1c0 msgr2=0x7f6454132740 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:23:12.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.266+0000 7f64467fc640 1 --2- 172.21.15.121:0/2391312737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f645413e1c0 0x7f6454132740 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:12.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.266+0000 7f64467fc640 10 client.44136.objecter ms_handle_connect 0x7f64380401a0 2024-09-17T18:23:12.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.266+0000 7f645a510640 1 --2- 172.21.15.121:0/2391312737 >> v2:172.21.15.173:6812/2423786037 conn(0x7f64380401a0 0x7f6438042660 secure :-1 s=READY pgs=3935 cs=0 l=1 rev1=1 crypto rx=0x7f644c0055e0 tx=0x7f644c003d10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:12.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.272+0000 7f645bf73640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:23:12.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.272+0000 7f645bf73640 1 --2- 172.21.15.121:0/2391312737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6454132c80 0x7f64540728f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:12.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.272+0000 7f645bf73640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:12.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.272+0000 7f645bf73640 1 -- 172.21.15.121:0/2391312737 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6454133090 con 0x7f6454132c80 2024-09-17T18:23:12.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.272+0000 7f6459d0f640 1 --2- 172.21.15.121:0/2391312737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6454132c80 0x7f64540728f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:12.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.272+0000 7f6459d0f640 10 monclient: get_auth_request con 0x7f6454132c80 auth_method 0 2024-09-17T18:23:12.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.272+0000 7f6459d0f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:12.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.272+0000 7f6459d0f640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:12.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.272+0000 7f6459d0f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:12.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.272+0000 7f6459d0f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:12.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.272+0000 7f6459d0f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:12.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.272+0000 7f6459d0f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:12.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.272+0000 7f6459d0f640 10 monclient(hunting): handle_auth_done global_id 44142 payload 824 2024-09-17T18:23:12.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.272+0000 7f6459d0f640 1 -- 172.21.15.121:0/2391312737 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f64541341f0 con 0x7f6454132c80 2024-09-17T18:23:12.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.275+0000 7f6459d0f640 1 -- 172.21.15.121:0/2391312737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6454132c80 msgr2=0x7f64540728f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:12.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.275+0000 7f64467fc640 10 client.44136.objecter ms_handle_connect 0x7f6454132c80 2024-09-17T18:23:12.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.275+0000 7f64467fc640 10 client.44136.objecter resend_mon_ops 2024-09-17T18:23:12.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.275+0000 7f6459d0f640 1 --2- 172.21.15.121:0/2391312737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6454132c80 0x7f64540728f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f64480090b0 tx=0x7f6448008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:12.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.275+0000 7f64467fc640 1 -- 172.21.15.121:0/2391312737 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6448037040 con 0x7f6454132c80 2024-09-17T18:23:12.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.275+0000 7f64467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:12.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.275+0000 7f64467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:23:12.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.275+0000 7f64467fc640 10 monclient: dump: 2024-09-17T18:23:12.289 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:12.290 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:12.290 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.290 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.290 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:12.290 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:12.290 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:12.290 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:12.290 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:12.290 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:12.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.275+0000 7f64467fc640 1 -- 172.21.15.121:0/2391312737 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4554 (secure 0 0 0) 0x7f644804c070 con 0x7f6454132c80 2024-09-17T18:23:12.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.275+0000 7f64467fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:23:12.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.275+0000 7f64467fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:23:12.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.275+0000 7f64467fc640 1 -- 172.21.15.121:0/2391312737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6454132c80 msgr2=0x7f64540728f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:12.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.275+0000 7f64467fc640 1 --2- 172.21.15.121:0/2391312737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6454132c80 0x7f64540728f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f64480090b0 tx=0x7f6448008030 comp rx=0 tx=0).stop 2024-09-17T18:23:12.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.275+0000 7f64467fc640 1 -- 172.21.15.121:0/2391312737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6454132c80 msgr2=0x7f64540728f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:23:12.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.275+0000 7f64467fc640 1 --2- 172.21.15.121:0/2391312737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6454132c80 0x7f64540728f0 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f64480090b0 tx=0x7f6448008030 comp rx=0 tx=0).stop 2024-09-17T18:23:12.291 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:23:12.291 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:23:12.291 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:23:12.291 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "leader", 2024-09-17T18:23:12.291 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 20, 2024-09-17T18:23:12.291 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 17939, 2024-09-17T18:23:12.291 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:23:12.291 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:23:12.292 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:23:12.292 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:23:12.292 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:23:12.292 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 2, 2024-09-17T18:23:12.292 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:23:12.292 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:23:12.292 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:23:12.292 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:23:12.292 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:23:12.292 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:23:12.292 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:23:12.292 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:23:12.292 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:23:12.292 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:23:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:23:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:23:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:23:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:23:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:23:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:23:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:23:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:23:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:23:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:23:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:23:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:23:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:23:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:23:12.294 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:23:12.294 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:23:12.294 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:23:12.294 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:23:12.294 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:12.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:12.294 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:23:12.294 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:23:12.294 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:23:12.294 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:23:12.295 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:23:12.295 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:23:12.295 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:23:12.295 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:23:12.295 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:23:12.295 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:23:12.295 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:23:12.295 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:23:12.295 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:23:12.295 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:23:12.295 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:23:12.295 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:23:12.295 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:23:12.296 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:23:12.296 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:23:12.296 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:23:12.296 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:23:12.296 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:23:12.296 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:23:12.296 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:23:12.296 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:23:12.296 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:23:12.296 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:23:12.296 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:23:12.296 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:23:12.296 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:23:12.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:12.297 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:23:12.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:12.297 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:23:12.297 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:23:12.297 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:23:12.297 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:23:12.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:12.297 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:23:12.297 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:23:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:23:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:23:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:23:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:23:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:23:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:23:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:23:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:23:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:23:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:23:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:23:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:23:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:23:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:23:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:23:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:23:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:23:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:23:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:23:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:23:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:23:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:23:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:23:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:23:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:23:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:23:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:23:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:23:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:23:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:23:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:23:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:23:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:23:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:23:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:23:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:23:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:23:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:23:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:23:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:23:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:23:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:23:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-17T18:23:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:23:12.304 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:23:12.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.275+0000 7f645bf73640 10 librados: watch_flush enter 2024-09-17T18:23:12.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.275+0000 7f645bf73640 10 librados: watch_flush exit 2024-09-17T18:23:12.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.275+0000 7f645bf73640 20 client.44136.objecter shutdown clearing up homeless session... 2024-09-17T18:23:12.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.275+0000 7f645bf73640 10 client.44136.objecter successfully canceled tick 2024-09-17T18:23:12.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.275+0000 7f645bf73640 1 -- 172.21.15.121:0/2391312737 >> v2:172.21.15.173:6812/2423786037 conn(0x7f64380401a0 msgr2=0x7f6438042660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:12.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.275+0000 7f645bf73640 1 --2- 172.21.15.121:0/2391312737 >> v2:172.21.15.173:6812/2423786037 conn(0x7f64380401a0 0x7f6438042660 secure :-1 s=READY pgs=3935 cs=0 l=1 rev1=1 crypto rx=0x7f644c0055e0 tx=0x7f644c003d10 comp rx=0 tx=0).stop 2024-09-17T18:23:12.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.275+0000 7f645bf73640 10 monclient: shutdown 2024-09-17T18:23:12.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.275+0000 7f645bf73640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:23:12.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.275+0000 7f645bf73640 1 -- 172.21.15.121:0/2391312737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64540a1320 msgr2=0x7f6454131cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:12.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.275+0000 7f645bf73640 1 --2- 172.21.15.121:0/2391312737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64540a1320 0x7f6454131cc0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f645000d800 tx=0x7f645000dcc0 comp rx=0 tx=0).stop 2024-09-17T18:23:12.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.275+0000 7f645a510640 1 -- 172.21.15.121:0/2391312737 reap_dead start 2024-09-17T18:23:12.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.275+0000 7f645a510640 1 -- 172.21.15.121:0/2391312737 reap_dead start 2024-09-17T18:23:12.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.275+0000 7f645bf73640 1 -- 172.21.15.121:0/2391312737 shutdown_connections 2024-09-17T18:23:12.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.275+0000 7f645bf73640 1 -- 172.21.15.121:0/2391312737 >> 172.21.15.121:0/2391312737 conn(0x7f64540a7830 msgr2=0x7f64540a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:23:12.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.275+0000 7f645bf73640 1 -- 172.21.15.121:0/2391312737 shutdown_connections 2024-09-17T18:23:12.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.275+0000 7f645bf73640 1 -- 172.21.15.121:0/2391312737 wait complete. 2024-09-17T18:23:12.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.275+0000 7f645bf73640 1 librados: shutdown 2024-09-17T18:23:12.307 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:23:12.307 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T18:23:12.307 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T18:23:12.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.403+0000 7f9be206e640 10 monclient: get_monmap_and_config 2024-09-17T18:23:12.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.403+0000 7f9be206e640 10 monclient: build_initial_monmap 2024-09-17T18:23:12.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.403+0000 7f9be206e640 10 monclient: monmap: 2024-09-17T18:23:12.405 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:23:12.405 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:12.405 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:23:12.404103+0000 2024-09-17T18:23:12.406 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:23:12.404103+0000 2024-09-17T18:23:12.406 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:23:12.406 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:12.406 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:23:12.406 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:23:12.406 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:23:12.406 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:23:12.406 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:23:12.406 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:12.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.404+0000 7f9be206e640 1 Processor -- start 2024-09-17T18:23:12.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.404+0000 7f9be206e640 1 -- start start 2024-09-17T18:23:12.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.404+0000 7f9be206e640 10 monclient: init 2024-09-17T18:23:12.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.404+0000 7f9be206e640 10 monclient: _reopen_session rank -1 2024-09-17T18:23:12.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.404+0000 7f9be206e640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-17T18:23:12.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.404+0000 7f9be206e640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f9bdc09e550 0x7f9bdc09e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:12.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.404+0000 7f9be206e640 10 monclient(hunting): picked mon.noname-c con 0x7f9bdc09e550 addr v2:172.21.15.173:3300/0 2024-09-17T18:23:12.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.404+0000 7f9be206e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9bdc09ee70 addr v1:172.21.15.173:6789/0 2024-09-17T18:23:12.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.404+0000 7f9be206e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9bdc09f2c0 addr v1:172.21.15.121:6789/0 2024-09-17T18:23:12.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.404+0000 7f9be206e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9bdc0a4460 con 0x7f9bdc09f2c0 2024-09-17T18:23:12.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.404+0000 7f9be206e640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9bdc0a2b60 con 0x7f9bdc09ee70 2024-09-17T18:23:12.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.404+0000 7f9be206e640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:12.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.404+0000 7f9be206e640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f9bdc0a4950 con 0x7f9bdc09e550 2024-09-17T18:23:12.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.404+0000 7f9be206e640 10 monclient(hunting): _renew_subs 2024-09-17T18:23:12.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.404+0000 7f9be206e640 10 monclient(hunting): authenticate will time out at 996.120754s 2024-09-17T18:23:12.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.404+0000 7f9bdbfff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f9bdc09f2c0 0x7f9bdc13cf70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:50854/0 (socket says 172.21.15.121:50854) 2024-09-17T18:23:12.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.404+0000 7f9bdbfff640 1 -- 172.21.15.121:0/3602330424 learned_addr learned my addr 172.21.15.121:0/3602330424 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:23:12.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.404+0000 7f9bdb7fe640 1 --2- 172.21.15.121:0/3602330424 >> v2:172.21.15.173:3300/0 conn(0x7f9bdc09e550 0x7f9bdc09e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:12.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.405+0000 7f9bdb7fe640 10 monclient(hunting): get_auth_request con 0x7f9bdc09e550 auth_method 0 2024-09-17T18:23:12.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.405+0000 7f9bdb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:12.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.405+0000 7f9bdb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:12.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.405+0000 7f9bdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:12.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.405+0000 7f9bdb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:12.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.405+0000 7f9bdb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:12.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.405+0000 7f9bdb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:12.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bdbfff640 1 -- 172.21.15.121:0/3602330424 >> v1:172.21.15.121:6789/0 conn(0x7f9bdc09f2c0 legacy=0x7f9bdc13cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:12.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bda7fc640 1 -- 172.21.15.121:0/3602330424 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3864660115 0 0) 0x7f9bdc0a4460 con 0x7f9bdc09f2c0 2024-09-17T18:23:12.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bda7fc640 10 monclient(hunting): _negotiate 2024-09-17T18:23:12.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bda7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:12.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:12.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bda7fc640 10 monclient(hunting): my global_id is 44145 2024-09-17T18:23:12.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bda7fc640 1 -- 172.21.15.121:0/3602330424 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9bc4003520 con 0x7f9bdc09f2c0 2024-09-17T18:23:12.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bdaffd640 1 -- 172.21.15.121:0/3602330424 >> v1:172.21.15.173:6789/0 conn(0x7f9bdc09ee70 legacy=0x7f9bdc139840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:12.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bda7fc640 1 -- 172.21.15.121:0/3602330424 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2120021789 0 0) 0x7f9bc4003520 con 0x7f9bdc09f2c0 2024-09-17T18:23:12.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bda7fc640 1 -- 172.21.15.121:0/3602330424 >> v2:172.21.15.173:3300/0 conn(0x7f9bdc09e550 msgr2=0x7f9bdc09e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:23:12.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bda7fc640 1 --2- 172.21.15.121:0/3602330424 >> v2:172.21.15.173:3300/0 conn(0x7f9bdc09e550 0x7f9bdc09e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:12.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bda7fc640 1 -- 172.21.15.121:0/3602330424 >> v1:172.21.15.173:6789/0 conn(0x7f9bdc09ee70 legacy=0x7f9bdc139840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:12.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bda7fc640 10 monclient: _finish_hunting 0 2024-09-17T18:23:12.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bda7fc640 1 monclient: found mon.noname-a-legacy 2024-09-17T18:23:12.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bda7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:23:12.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bda7fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.121:6789/0 2024-09-17T18:23:12.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bda7fc640 1 -- 172.21.15.121:0/3602330424 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bdc09f840 con 0x7f9bdc09f2c0 2024-09-17T18:23:12.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bda7fc640 10 monclient: _finish_auth 0 2024-09-17T18:23:12.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bda7fc640 10 monclient: _check_auth_tickets 2024-09-17T18:23:12.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bda7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:23:12.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bda7fc640 1 -- 172.21.15.121:0/3602330424 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4002019445 0 0) 0x7f9bd4001de0 con 0x7f9bdc09f2c0 2024-09-17T18:23:12.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9be206e640 5 monclient: authenticate success, global_id 44145 2024-09-17T18:23:12.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9be206e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:23:12.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bda7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:12.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bda7fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T18:23:12.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bda7fc640 10 monclient: dump: 2024-09-17T18:23:12.415 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:12.415 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:12.415 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.415 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.415 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:12.416 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:12.416 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:12.416 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:12.416 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:12.416 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:12.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bda7fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] but i'm connected to v1:172.21.15.121:6789/0, reconnecting 2024-09-17T18:23:12.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bda7fc640 10 monclient: _reopen_session rank -1 2024-09-17T18:23:12.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bda7fc640 1 -- 172.21.15.121:0/3602330424 >> v1:172.21.15.121:6789/0 conn(0x7f9bdc09f2c0 legacy=0x7f9bdc13cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:12.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bda7fc640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T18:23:12.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bda7fc640 1 --2- 172.21.15.121:0/3602330424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9bc4003ac0 0x7f9bc4003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:12.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bda7fc640 10 monclient(hunting): picked mon.a con 0x7f9bc4003ac0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:23:12.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bda7fc640 1 --2- 172.21.15.121:0/3602330424 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9bc4004800 0x7f9bc4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:12.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bda7fc640 10 monclient(hunting): picked mon.c con 0x7f9bc4004800 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:23:12.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bda7fc640 1 --2- 172.21.15.121:0/3602330424 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9bc40053c0 0x7f9bc400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:12.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bda7fc640 10 monclient(hunting): picked mon.b con 0x7f9bc40053c0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:23:12.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bda7fc640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:12.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bda7fc640 1 -- 172.21.15.121:0/3602330424 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9bc400c270 con 0x7f9bc4003ac0 2024-09-17T18:23:12.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bda7fc640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:12.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bda7fc640 1 -- 172.21.15.121:0/3602330424 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9bc400c420 con 0x7f9bc40053c0 2024-09-17T18:23:12.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bda7fc640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:12.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bda7fc640 1 -- 172.21.15.121:0/3602330424 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9bc400c620 con 0x7f9bc4004800 2024-09-17T18:23:12.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bda7fc640 10 monclient(hunting): _renew_subs 2024-09-17T18:23:12.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bda7fc640 10 monclient(hunting): _finish_auth 0 2024-09-17T18:23:12.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9be206e640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-17T18:23:12.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bdb7fe640 1 --2- 172.21.15.121:0/3602330424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9bc4003ac0 0x7f9bc4003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:12.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bdb7fe640 10 monclient(hunting): get_auth_request con 0x7f9bc4003ac0 auth_method 0 2024-09-17T18:23:12.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bdb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:12.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bdb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:12.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bdb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:23:12.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bdaffd640 1 --2- 172.21.15.121:0/3602330424 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9bc4004800 0x7f9bc4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:12.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bdaffd640 10 monclient(hunting): get_auth_request con 0x7f9bc4004800 auth_method 0 2024-09-17T18:23:12.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bdaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:12.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bdaffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:12.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bdaffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:23:12.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bdb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:12.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bdb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:12.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bdbfff640 1 --2- 172.21.15.121:0/3602330424 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9bc40053c0 0x7f9bc400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:12.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bdb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:23:12.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bdbfff640 10 monclient(hunting): get_auth_request con 0x7f9bc40053c0 auth_method 0 2024-09-17T18:23:12.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bdbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:12.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bdbfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:12.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bdbfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:23:12.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bdaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:12.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bdaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:12.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bdaffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:23:12.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bdbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:12.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bdbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:12.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bdbfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:23:12.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bdb7fe640 10 monclient(hunting): handle_auth_done global_id 44145 payload 293 2024-09-17T18:23:12.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bdb7fe640 1 -- 172.21.15.121:0/3602330424 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9bc4004800 msgr2=0x7f9bc4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:23:12.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bdb7fe640 1 --2- 172.21.15.121:0/3602330424 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9bc4004800 0x7f9bc4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:12.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bdb7fe640 1 -- 172.21.15.121:0/3602330424 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9bc40053c0 msgr2=0x7f9bc400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:23:12.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bdb7fe640 1 --2- 172.21.15.121:0/3602330424 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9bc40053c0 0x7f9bc400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:12.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bdb7fe640 10 monclient: _finish_hunting 0 2024-09-17T18:23:12.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bdb7fe640 1 monclient: found mon.a 2024-09-17T18:23:12.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bdb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:23:12.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bdb7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:23:12.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bdbfff640 1 -- 172.21.15.121:0/3602330424 reap_dead start 2024-09-17T18:23:12.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bdb7fe640 1 -- 172.21.15.121:0/3602330424 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bc4003520 con 0x7f9bc4003ac0 2024-09-17T18:23:12.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bdb7fe640 10 monclient: _finish_auth 0 2024-09-17T18:23:12.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bdb7fe640 10 monclient: _check_auth_tickets 2024-09-17T18:23:12.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.407+0000 7f9bdb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:23:12.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.410+0000 7f9bdb7fe640 1 -- 172.21.15.121:0/3602330424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9bc4003ac0 msgr2=0x7f9bc4003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:12.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.410+0000 7f9bdb7fe640 1 --2- 172.21.15.121:0/3602330424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9bc4003ac0 0x7f9bc4003ee0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc009ac0 tx=0x7f9bc4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91fc769d63af4351 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:12.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.410+0000 7f9bda7fc640 1 -- 172.21.15.121:0/3602330424 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc036070 con 0x7f9bc4003ac0 2024-09-17T18:23:12.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.410+0000 7f9bda7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:12.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.410+0000 7f9bda7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:23:12.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.410+0000 7f9bda7fc640 10 monclient: dump: 2024-09-17T18:23:12.429 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:12.429 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:12.429 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.429 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.430 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:12.430 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:12.430 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:12.430 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:12.430 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:12.430 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:12.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.410+0000 7f9bda7fc640 1 -- 172.21.15.121:0/3602330424 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bcc02f510 con 0x7f9bc4003ac0 2024-09-17T18:23:12.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.410+0000 7f9be206e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:23:12.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.410+0000 7f9bda7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:23:12.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.410+0000 7f9be206e640 10 monclient: get_monmap_and_config success 2024-09-17T18:23:12.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.410+0000 7f9bda7fc640 1 -- 172.21.15.121:0/3602330424 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc009690 con 0x7f9bc4003ac0 2024-09-17T18:23:12.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.410+0000 7f9bda7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:12.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.410+0000 7f9be206e640 4 set_mon_vals no callback set 2024-09-17T18:23:12.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.410+0000 7f9bda7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:23:12.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.410+0000 7f9bda7fc640 10 monclient: dump: 2024-09-17T18:23:12.431 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:12.431 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:12.431 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.431 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.431 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:12.431 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:12.431 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:12.431 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:12.431 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:12.431 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:12.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.411+0000 7f9be206e640 10 monclient: shutdown 2024-09-17T18:23:12.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.411+0000 7f9be206e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:23:12.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.411+0000 7f9be206e640 1 -- 172.21.15.121:0/3602330424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9bc4003ac0 msgr2=0x7f9bc4003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:12.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.411+0000 7f9be206e640 1 --2- 172.21.15.121:0/3602330424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9bc4003ac0 0x7f9bc4003ee0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc009ac0 tx=0x7f9bc4005830 comp rx=0 tx=0).stop 2024-09-17T18:23:12.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.411+0000 7f9be206e640 1 -- 172.21.15.121:0/3602330424 shutdown_connections 2024-09-17T18:23:12.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.411+0000 7f9be206e640 1 --2- 172.21.15.121:0/3602330424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9bc4003ac0 0x7f9bc4003ee0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:12.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.411+0000 7f9be206e640 1 -- 172.21.15.121:0/3602330424 >> 172.21.15.121:0/3602330424 conn(0x7f9bdc0a7830 msgr2=0x7f9bdc0a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:23:12.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.411+0000 7f9be206e640 1 -- 172.21.15.121:0/3602330424 shutdown_connections 2024-09-17T18:23:12.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.411+0000 7f9be206e640 1 -- 172.21.15.121:0/3602330424 wait complete. 2024-09-17T18:23:12.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.411+0000 7f9be206e640 10 monclient: build_initial_monmap 2024-09-17T18:23:12.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.411+0000 7f9be206e640 10 monclient: monmap: 2024-09-17T18:23:12.432 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:23:12.432 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:23:12.432 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:23:12.433 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:23:12.433 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:23:12.433 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:12.433 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:23:12.433 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:23:12.433 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:23:12.433 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:12.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.412+0000 7f9be206e640 1 librados: starting msgr at 2024-09-17T18:23:12.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.412+0000 7f9be206e640 1 librados: starting objecter 2024-09-17T18:23:12.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.412+0000 7f9be206e640 1 Processor -- start 2024-09-17T18:23:12.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.412+0000 7f9be206e640 1 -- start start 2024-09-17T18:23:12.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.412+0000 7f9be206e640 1 librados: setting wanted keys 2024-09-17T18:23:12.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.412+0000 7f9be206e640 1 librados: calling monclient init 2024-09-17T18:23:12.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.412+0000 7f9be206e640 10 monclient: init 2024-09-17T18:23:12.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.412+0000 7f9be206e640 10 monclient: _reopen_session rank -1 2024-09-17T18:23:12.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.412+0000 7f9be206e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T18:23:12.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.412+0000 7f9be206e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9bc4003ac0 0x7f9bdc1cdac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:12.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.412+0000 7f9be206e640 10 monclient(hunting): picked mon.noname-a con 0x7f9bc4003ac0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:23:12.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.412+0000 7f9be206e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9bdc1ce000 0x7f9bdc135850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:12.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.412+0000 7f9be206e640 10 monclient(hunting): picked mon.noname-c con 0x7f9bdc1ce000 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:23:12.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.412+0000 7f9be206e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9bdc1ce440 0x7f9bdc135d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:12.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.412+0000 7f9be206e640 10 monclient(hunting): picked mon.noname-b con 0x7f9bdc1ce440 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:23:12.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.412+0000 7f9be206e640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:12.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.412+0000 7f9be206e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9bdc09de60 con 0x7f9bc4003ac0 2024-09-17T18:23:12.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.413+0000 7f9be206e640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:12.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.413+0000 7f9be206e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9bdc09dce0 con 0x7f9bdc1ce440 2024-09-17T18:23:12.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.413+0000 7f9be206e640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:12.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.413+0000 7f9be206e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9bdc0747e0 con 0x7f9bdc1ce000 2024-09-17T18:23:12.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.413+0000 7f9be206e640 10 monclient(hunting): _renew_subs 2024-09-17T18:23:12.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.413+0000 7f9be206e640 10 monclient(hunting): authenticate will time out at 996.129031s 2024-09-17T18:23:12.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.413+0000 7f9bdb7fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9bc4003ac0 0x7f9bdc1cdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:12.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.413+0000 7f9bdb7fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9bc4003ac0 0x7f9bdc1cdac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60282/0 (socket says 172.21.15.121:60282) 2024-09-17T18:23:12.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.413+0000 7f9bdb7fe640 1 -- 172.21.15.121:0/2937397104 learned_addr learned my addr 172.21.15.121:0/2937397104 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:23:12.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.413+0000 7f9bdb7fe640 10 monclient(hunting): get_auth_request con 0x7f9bc4003ac0 auth_method 0 2024-09-17T18:23:12.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.413+0000 7f9bdb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:12.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.413+0000 7f9bdb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:12.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.413+0000 7f9bdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:12.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.413+0000 7f9bdaffd640 1 --2- 172.21.15.121:0/2937397104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9bdc1ce000 0x7f9bdc135850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:12.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.413+0000 7f9bdaffd640 10 monclient(hunting): get_auth_request con 0x7f9bdc1ce000 auth_method 0 2024-09-17T18:23:12.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.413+0000 7f9bdaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:12.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.413+0000 7f9bdaffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:12.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.413+0000 7f9bdaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:12.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.413+0000 7f9bdb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:12.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.413+0000 7f9bdb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:12.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.413+0000 7f9bdb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:12.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.413+0000 7f9bdaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:12.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.413+0000 7f9bdaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:12.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.413+0000 7f9bdaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:12.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.413+0000 7f9bdbfff640 1 --2- 172.21.15.121:0/2937397104 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9bdc1ce440 0x7f9bdc135d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:12.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.413+0000 7f9bdaffd640 10 monclient(hunting): handle_auth_done global_id 34463 payload 824 2024-09-17T18:23:12.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.413+0000 7f9bdaffd640 1 -- 172.21.15.121:0/2937397104 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9bdc1ce440 msgr2=0x7f9bdc135d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:12.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.413+0000 7f9bdaffd640 1 --2- 172.21.15.121:0/2937397104 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9bdc1ce440 0x7f9bdc135d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:12.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.413+0000 7f9bdaffd640 1 -- 172.21.15.121:0/2937397104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9bc4003ac0 msgr2=0x7f9bdc1cdac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:12.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.413+0000 7f9bdaffd640 1 --2- 172.21.15.121:0/2937397104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9bc4003ac0 0x7f9bdc1cdac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:12.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.413+0000 7f9bdaffd640 10 monclient: _finish_hunting 0 2024-09-17T18:23:12.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.413+0000 7f9bdaffd640 1 monclient: found mon.noname-c 2024-09-17T18:23:12.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.413+0000 7f9bdaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:23:12.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.413+0000 7f9bdaffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:23:12.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.413+0000 7f9bdaffd640 1 -- 172.21.15.121:0/2937397104 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bdc125970 con 0x7f9bdc1ce000 2024-09-17T18:23:12.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.413+0000 7f9bdbfff640 10 monclient: get_auth_request con 0x7f9bdc1ce440 auth_method 0 2024-09-17T18:23:12.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.413+0000 7f9bdb7fe640 1 --2- 172.21.15.121:0/2937397104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9bc4003ac0 0x7f9bdc1cdac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:23:12.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.413+0000 7f9bdbfff640 1 --2- 172.21.15.121:0/2937397104 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9bdc1ce440 0x7f9bdc135d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T18:23:12.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.415+0000 7f9bdaffd640 1 -- 172.21.15.121:0/2937397104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9bdc1ce000 msgr2=0x7f9bdc135850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:12.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.415+0000 7f9bdaffd640 1 --2- 172.21.15.121:0/2937397104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9bdc1ce000 0x7f9bdc135850 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0006dd0 tx=0x7f9bd00019c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:12.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.415+0000 7f9bc3fff640 10 client.?.objecter ms_handle_connect 0x7f9bdc1ce000 2024-09-17T18:23:12.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.415+0000 7f9bc3fff640 10 client.?.objecter resend_mon_ops 2024-09-17T18:23:12.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.415+0000 7f9bc3fff640 1 -- 172.21.15.121:0/2937397104 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd0017020 con 0x7f9bdc1ce000 2024-09-17T18:23:12.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.415+0000 7f9bc3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:12.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.415+0000 7f9bc3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:23:12.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.415+0000 7f9bc3fff640 10 monclient: dump: 2024-09-17T18:23:12.438 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:12.438 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:12.438 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.438 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.438 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:12.438 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:12.438 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:12.438 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:12.438 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:12.438 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:12.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.415+0000 7f9bc3fff640 10 monclient: _finish_auth 0 2024-09-17T18:23:12.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.415+0000 7f9bc3fff640 10 monclient: _check_auth_tickets 2024-09-17T18:23:12.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.415+0000 7f9bc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:23:12.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.415+0000 7f9bc3fff640 1 -- 172.21.15.121:0/2937397104 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bd0013810 con 0x7f9bdc1ce000 2024-09-17T18:23:12.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.415+0000 7f9be206e640 5 monclient: authenticate success, global_id 34463 2024-09-17T18:23:12.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.415+0000 7f9be206e640 10 monclient: _renew_subs 2024-09-17T18:23:12.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.415+0000 7f9be206e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:23:12.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.415+0000 7f9be206e640 1 -- 172.21.15.121:0/2937397104 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bdc125c50 con 0x7f9bdc1ce000 2024-09-17T18:23:12.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.415+0000 7f9bc3fff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:23:12.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.415+0000 7f9bc3fff640 1 -- 172.21.15.121:0/2937397104 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd0013bc0 con 0x7f9bdc1ce000 2024-09-17T18:23:12.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.415+0000 7f9bc3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:12.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.415+0000 7f9bc3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:23:12.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.415+0000 7f9bc3fff640 10 monclient: dump: 2024-09-17T18:23:12.439 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:12.440 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:12.440 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.440 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.440 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:12.440 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:12.440 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:12.440 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:12.440 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:12.440 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:12.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.415+0000 7f9be206e640 10 client.34463.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:23:12.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.415+0000 7f9be206e640 10 monclient: _renew_subs 2024-09-17T18:23:12.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.415+0000 7f9be206e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:23:12.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.415+0000 7f9bd9ffb640 4 set_mon_vals no callback set 2024-09-17T18:23:12.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.415+0000 7f9be206e640 1 -- 172.21.15.121:0/2937397104 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bdc0a6190 con 0x7f9bdc1ce000 2024-09-17T18:23:12.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.415+0000 7f9be206e640 1 librados: init done 2024-09-17T18:23:12.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.416+0000 7f9be206e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:23:12.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.416+0000 7f9be206e640 1 --2- 172.21.15.121:0/2937397104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9bdc136a90 0x7f9bdc02d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:12.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.416+0000 7f9be206e640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:12.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.416+0000 7f9be206e640 1 -- 172.21.15.121:0/2937397104 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9bdc09e930 con 0x7f9bdc136a90 2024-09-17T18:23:12.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.416+0000 7f9bdb7fe640 1 --2- 172.21.15.121:0/2937397104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9bdc136a90 0x7f9bdc02d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:12.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.416+0000 7f9bdb7fe640 10 monclient: get_auth_request con 0x7f9bdc136a90 auth_method 0 2024-09-17T18:23:12.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.416+0000 7f9bdb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:12.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.416+0000 7f9bdb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:12.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.416+0000 7f9bdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:12.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.416+0000 7f9bc3fff640 1 -- 172.21.15.121:0/2937397104 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f9bd001a020 con 0x7f9bdc1ce000 2024-09-17T18:23:12.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.416+0000 7f9bdb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:12.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.416+0000 7f9bdb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:12.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.416+0000 7f9bdb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:12.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.416+0000 7f9bdb7fe640 10 monclient(hunting): handle_auth_done global_id 34466 payload 824 2024-09-17T18:23:12.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.416+0000 7f9bdb7fe640 1 -- 172.21.15.121:0/2937397104 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9bdc072fa0 con 0x7f9bdc136a90 2024-09-17T18:23:12.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.416+0000 7f9bc3fff640 1 --2- 172.21.15.121:0/2937397104 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9bc8040170 0x7f9bc8042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:12.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.417+0000 7f9bc3fff640 1 -- 172.21.15.121:0/2937397104 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(178..178 src has 1..178) ==== 5795+0+0 (secure 0 0 0) 0x7f9bd00537a0 con 0x7f9bdc1ce000 2024-09-17T18:23:12.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.417+0000 7f9bc3fff640 10 client.34463.objecter ms_dispatch 0x7f9bdc000b90 osd_map(178..178 src has 1..178) 2024-09-17T18:23:12.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.417+0000 7f9bc3fff640 3 client.34463.objecter handle_osd_map got epochs [178,178] > 0 2024-09-17T18:23:12.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.417+0000 7f9bc3fff640 3 client.34463.objecter handle_osd_map decoding full epoch 178 2024-09-17T18:23:12.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.417+0000 7f9bc3fff640 20 client.34463.objecter dump_active .. 0 homeless 2024-09-17T18:23:12.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.417+0000 7f9bdbfff640 1 --2- 172.21.15.121:0/2937397104 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9bc8040170 0x7f9bc8042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:12.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.417+0000 7f9bdbfff640 10 monclient: get_auth_request con 0x7f9bc8040170 auth_method 0 2024-09-17T18:23:12.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.419+0000 7f9bc3fff640 10 client.34463.objecter ms_handle_connect 0x7f9bdc136a90 2024-09-17T18:23:12.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.419+0000 7f9bc3fff640 10 client.34463.objecter resend_mon_ops 2024-09-17T18:23:12.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.419+0000 7f9bdb7fe640 1 -- 172.21.15.121:0/2937397104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9bdc136a90 msgr2=0x7f9bdc02d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:12.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.419+0000 7f9bdb7fe640 1 --2- 172.21.15.121:0/2937397104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9bdc136a90 0x7f9bdc02d440 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc000c00 tx=0x7f9bcc004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:12.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.419+0000 7f9bc3fff640 1 -- 172.21.15.121:0/2937397104 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc041030 con 0x7f9bdc136a90 2024-09-17T18:23:12.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.419+0000 7f9bc3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:12.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.419+0000 7f9bc3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:23:12.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.419+0000 7f9bc3fff640 10 monclient: dump: 2024-09-17T18:23:12.444 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:12.444 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:12.444 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.444 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.444 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:12.444 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:12.444 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:12.444 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:12.444 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:12.444 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:12.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.419+0000 7f9bc3fff640 1 -- 172.21.15.121:0/2937397104 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9bcc0043f0 con 0x7f9bdc136a90 2024-09-17T18:23:12.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.419+0000 7f9bc3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:23:12.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.419+0000 7f9bc3fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:23:12.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.419+0000 7f9bc3fff640 1 -- 172.21.15.121:0/2937397104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9bdc136a90 msgr2=0x7f9bdc02d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:12.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.419+0000 7f9bc3fff640 1 --2- 172.21.15.121:0/2937397104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9bdc136a90 0x7f9bdc02d440 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc000c00 tx=0x7f9bcc004730 comp rx=0 tx=0).stop 2024-09-17T18:23:12.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.419+0000 7f9bc3fff640 1 -- 172.21.15.121:0/2937397104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9bdc136a90 msgr2=0x7f9bdc02d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:23:12.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.419+0000 7f9bc3fff640 1 --2- 172.21.15.121:0/2937397104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9bdc136a90 0x7f9bdc02d440 secure :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc000c00 tx=0x7f9bcc004730 comp rx=0 tx=0).stop 2024-09-17T18:23:12.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.419+0000 7f9bc3fff640 10 client.34463.objecter ms_handle_connect 0x7f9bc8040170 2024-09-17T18:23:12.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.419+0000 7f9bdbfff640 1 --2- 172.21.15.121:0/2937397104 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9bc8040170 0x7f9bc8042630 secure :-1 s=READY pgs=3936 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc06a7d0 tx=0x7f9bd4006040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:12.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.425+0000 7f9be206e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:23:12.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.425+0000 7f9be206e640 1 --2- 172.21.15.121:0/2937397104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9bdc02d980 0x7f9bdc02dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:12.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.425+0000 7f9be206e640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:12.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.425+0000 7f9be206e640 1 -- 172.21.15.121:0/2937397104 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9bdc06cd40 con 0x7f9bdc02d980 2024-09-17T18:23:12.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.425+0000 7f9bdb7fe640 1 --2- 172.21.15.121:0/2937397104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9bdc02d980 0x7f9bdc02dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:12.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.425+0000 7f9bdb7fe640 10 monclient: get_auth_request con 0x7f9bdc02d980 auth_method 0 2024-09-17T18:23:12.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.425+0000 7f9bdb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:12.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.425+0000 7f9bdb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:12.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.425+0000 7f9bdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:12.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.425+0000 7f9bdb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:12.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.425+0000 7f9bdb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:12.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.425+0000 7f9bdb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:12.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.425+0000 7f9bdb7fe640 10 monclient(hunting): handle_auth_done global_id 34469 payload 824 2024-09-17T18:23:12.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.426+0000 7f9bdb7fe640 1 -- 172.21.15.121:0/2937397104 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9bdc125690 con 0x7f9bdc02d980 2024-09-17T18:23:12.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.428+0000 7f9bdb7fe640 1 -- 172.21.15.121:0/2937397104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9bdc02d980 msgr2=0x7f9bdc02dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:12.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.428+0000 7f9bdb7fe640 1 --2- 172.21.15.121:0/2937397104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9bdc02d980 0x7f9bdc02dd90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc09eab0 tx=0x7f9bcc02f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:12.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.428+0000 7f9bc3fff640 10 client.34463.objecter ms_handle_connect 0x7f9bdc02d980 2024-09-17T18:23:12.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.428+0000 7f9bc3fff640 10 client.34463.objecter resend_mon_ops 2024-09-17T18:23:12.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.428+0000 7f9bc3fff640 1 -- 172.21.15.121:0/2937397104 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc041040 con 0x7f9bdc02d980 2024-09-17T18:23:12.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.428+0000 7f9bc3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:12.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.428+0000 7f9bc3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:23:12.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.428+0000 7f9bc3fff640 10 monclient: dump: 2024-09-17T18:23:12.448 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:12.448 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:12.448 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.448 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.449 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:12.449 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:12.449 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:12.449 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:12.449 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:12.449 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:12.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.428+0000 7f9bc3fff640 1 -- 172.21.15.121:0/2937397104 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7f9bcc03d670 con 0x7f9bdc02d980 2024-09-17T18:23:12.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.428+0000 7f9bc3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:23:12.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.428+0000 7f9bc3fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:23:12.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.428+0000 7f9bc3fff640 1 -- 172.21.15.121:0/2937397104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9bdc02d980 msgr2=0x7f9bdc02dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:12.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.428+0000 7f9bc3fff640 1 --2- 172.21.15.121:0/2937397104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9bdc02d980 0x7f9bdc02dd90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc09eab0 tx=0x7f9bcc02f000 comp rx=0 tx=0).stop 2024-09-17T18:23:12.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.428+0000 7f9bc3fff640 1 -- 172.21.15.121:0/2937397104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9bdc02d980 msgr2=0x7f9bdc02dd90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:23:12.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.428+0000 7f9bc3fff640 1 --2- 172.21.15.121:0/2937397104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9bdc02d980 0x7f9bdc02dd90 secure :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc09eab0 tx=0x7f9bcc02f000 comp rx=0 tx=0).stop 2024-09-17T18:23:12.450 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:23:12.450 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:23:12.450 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:23:12.450 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:23:12.450 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 20, 2024-09-17T18:23:12.450 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 137705, 2024-09-17T18:23:12.450 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:23:12.450 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:23:12.450 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:23:12.450 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:23:12.451 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:23:12.451 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 2, 2024-09-17T18:23:12.451 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:23:12.451 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:23:12.451 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:23:12.451 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:23:12.451 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:23:12.451 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:23:12.451 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:23:12.451 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:23:12.451 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:23:12.451 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:23:12.451 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:23:12.451 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:23:12.452 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:23:12.452 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:23:12.452 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:23:12.452 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:23:12.452 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:23:12.452 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:23:12.452 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:23:12.452 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:23:12.452 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:23:12.452 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:23:12.452 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:23:12.452 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:23:12.452 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:23:12.453 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:23:12.453 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:23:12.453 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:23:12.453 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:12.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:12.453 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:23:12.453 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:23:12.453 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:23:12.453 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:23:12.453 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:23:12.453 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:23:12.453 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:23:12.453 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:23:12.454 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:23:12.454 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:23:12.454 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:23:12.454 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:23:12.454 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:23:12.454 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:23:12.454 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:23:12.454 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:23:12.454 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:23:12.454 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:23:12.454 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:23:12.454 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:23:12.454 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:23:12.455 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:23:12.455 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:23:12.455 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:23:12.455 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:23:12.455 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:23:12.455 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:23:12.455 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:23:12.455 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:23:12.455 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:23:12.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:12.455 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:23:12.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:12.455 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:23:12.455 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:23:12.456 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:23:12.456 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:23:12.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:12.456 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:23:12.456 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:23:12.456 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:12.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:12.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:12.456 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:23:12.456 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:23:12.456 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:12.456 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:12.456 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:12.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:12.457 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:23:12.457 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:23:12.457 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:23:12.457 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:23:12.457 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:23:12.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:12.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:12.457 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:23:12.457 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:23:12.457 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:23:12.457 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:23:12.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:12.458 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:23:12.458 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:23:12.458 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:12.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:12.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:12.458 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:23:12.458 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:23:12.458 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:12.458 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:12.458 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:12.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:12.458 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:23:12.458 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:23:12.458 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:23:12.458 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:23:12.459 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:23:12.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:12.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:12.459 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:23:12.459 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:23:12.459 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:23:12.459 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:23:12.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:12.459 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:23:12.459 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:23:12.459 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:12.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:12.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:12.460 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:23:12.460 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:23:12.460 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:12.460 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:12.460 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:12.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:12.460 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:23:12.460 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:23:12.460 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:23:12.460 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:23:12.460 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:23:12.460 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:12.460 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:12.461 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:12.461 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:23:12.461 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:23:12.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:12.461 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:23:12.461 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:23:12.461 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:23:12.461 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:12.461 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:23:12.461 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:23:12.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:12.461 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:23:12.461 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:23:12.461 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 3 2024-09-17T18:23:12.462 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:12.462 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:23:12.462 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:23:12.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:12.462 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:23:12.462 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:23:12.462 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-17T18:23:12.462 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:12.462 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:12.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:12.462 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:23:12.462 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:23:12.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.428+0000 7f9be206e640 10 librados: watch_flush enter 2024-09-17T18:23:12.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.428+0000 7f9be206e640 10 librados: watch_flush exit 2024-09-17T18:23:12.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.428+0000 7f9be206e640 20 client.34463.objecter shutdown clearing up homeless session... 2024-09-17T18:23:12.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.428+0000 7f9be206e640 10 client.34463.objecter successfully canceled tick 2024-09-17T18:23:12.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.428+0000 7f9be206e640 1 -- 172.21.15.121:0/2937397104 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9bc8040170 msgr2=0x7f9bc8042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:12.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.428+0000 7f9be206e640 1 --2- 172.21.15.121:0/2937397104 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9bc8040170 0x7f9bc8042630 secure :-1 s=READY pgs=3936 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc06a7d0 tx=0x7f9bd4006040 comp rx=0 tx=0).stop 2024-09-17T18:23:12.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.428+0000 7f9be206e640 10 monclient: shutdown 2024-09-17T18:23:12.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.428+0000 7f9be206e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:23:12.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.428+0000 7f9be206e640 1 -- 172.21.15.121:0/2937397104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9bdc1ce000 msgr2=0x7f9bdc135850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:12.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.428+0000 7f9bdbfff640 1 -- 172.21.15.121:0/2937397104 reap_dead start 2024-09-17T18:23:12.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.428+0000 7f9be206e640 1 --2- 172.21.15.121:0/2937397104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9bdc1ce000 0x7f9bdc135850 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0006dd0 tx=0x7f9bd00019c0 comp rx=0 tx=0).stop 2024-09-17T18:23:12.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.428+0000 7f9be206e640 1 -- 172.21.15.121:0/2937397104 shutdown_connections 2024-09-17T18:23:12.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.428+0000 7f9be206e640 1 --2- 172.21.15.121:0/2937397104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9bdc1ce000 0x7f9bdc135850 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:12.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.428+0000 7f9be206e640 1 -- 172.21.15.121:0/2937397104 >> 172.21.15.121:0/2937397104 conn(0x7f9bdc0a7830 msgr2=0x7f9bdc134c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:23:12.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.429+0000 7f9be206e640 1 -- 172.21.15.121:0/2937397104 shutdown_connections 2024-09-17T18:23:12.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.429+0000 7f9be206e640 1 -- 172.21.15.121:0/2937397104 wait complete. 2024-09-17T18:23:12.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.429+0000 7f9be206e640 1 librados: shutdown 2024-09-17T18:23:12.466 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:23:12.466 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T18:23:12.466 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T18:23:12.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.563+0000 7f2bee552640 10 monclient: get_monmap_and_config 2024-09-17T18:23:12.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.563+0000 7f2bee552640 10 monclient: build_initial_monmap 2024-09-17T18:23:12.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.563+0000 7f2bee552640 10 monclient: monmap: 2024-09-17T18:23:12.565 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:23:12.565 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:12.565 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:23:12.563812+0000 2024-09-17T18:23:12.565 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:23:12.563812+0000 2024-09-17T18:23:12.565 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:23:12.565 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:12.565 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:23:12.565 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:23:12.565 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:23:12.566 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:23:12.566 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:23:12.566 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:12.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.563+0000 7f2bee552640 1 Processor -- start 2024-09-17T18:23:12.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.564+0000 7f2bee552640 1 -- start start 2024-09-17T18:23:12.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.564+0000 7f2bee552640 10 monclient: init 2024-09-17T18:23:12.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.564+0000 7f2bee552640 10 monclient: _reopen_session rank -1 2024-09-17T18:23:12.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.564+0000 7f2bee552640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-17T18:23:12.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.564+0000 7f2bee552640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2be80700f0 addr v1:172.21.15.121:6789/0 2024-09-17T18:23:12.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.564+0000 7f2bee552640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2be809e810 0x7f2be813d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:12.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.564+0000 7f2bee552640 10 monclient(hunting): picked mon.noname-a con 0x7f2be809e810 addr v2:172.21.15.121:3300/0 2024-09-17T18:23:12.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.564+0000 7f2bee552640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2be809ec60 addr v1:172.21.15.173:6789/0 2024-09-17T18:23:12.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.564+0000 7f2bee552640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2be8135560 con 0x7f2be80700f0 2024-09-17T18:23:12.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.564+0000 7f2bee552640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2be809ca60 con 0x7f2be809ec60 2024-09-17T18:23:12.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.564+0000 7f2bee552640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:12.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.564+0000 7f2bee552640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f2be80704d0 con 0x7f2be809e810 2024-09-17T18:23:12.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.564+0000 7f2bee552640 10 monclient(hunting): _renew_subs 2024-09-17T18:23:12.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.564+0000 7f2bee552640 10 monclient(hunting): authenticate will time out at 996.280492s 2024-09-17T18:23:12.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.564+0000 7f2be77fe640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2be809e810 0x7f2be813d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:12.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.564+0000 7f2be77fe640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2be809e810 0x7f2be813d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60296/0 (socket says 172.21.15.121:60296) 2024-09-17T18:23:12.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.564+0000 7f2be77fe640 1 -- 172.21.15.121:0/3122350360 learned_addr learned my addr 172.21.15.121:0/3122350360 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:23:12.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.564+0000 7f2be77fe640 10 monclient(hunting): get_auth_request con 0x7f2be809e810 auth_method 0 2024-09-17T18:23:12.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.564+0000 7f2be77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:12.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.564+0000 7f2be77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:12.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.564+0000 7f2be77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:12.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.564+0000 7f2be7fff640 1 --1- 172.21.15.121:0/3122350360 >> v1:172.21.15.121:6789/0 conn(0x7f2be80700f0 0x7f2be8134e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:50856/0 (socket says 172.21.15.121:50856) 2024-09-17T18:23:12.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.564+0000 7f2be77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:12.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.564+0000 7f2be77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:12.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.564+0000 7f2be77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:12.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.566+0000 7f2be7fff640 1 -- 172.21.15.121:0/3122350360 >> v1:172.21.15.121:6789/0 conn(0x7f2be80700f0 legacy=0x7f2be8134e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:12.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.566+0000 7f2becaef640 1 -- 172.21.15.121:0/3122350360 >> v1:172.21.15.173:6789/0 conn(0x7f2be809ec60 legacy=0x7f2be813fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:12.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.566+0000 7f2be77fe640 10 monclient(hunting): handle_auth_done global_id 44151 payload 274 2024-09-17T18:23:12.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.566+0000 7f2be77fe640 1 -- 172.21.15.121:0/3122350360 >> v1:172.21.15.173:6789/0 conn(0x7f2be809ec60 legacy=0x7f2be813fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:12.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.567+0000 7f2be77fe640 1 -- 172.21.15.121:0/3122350360 >> v1:172.21.15.121:6789/0 conn(0x7f2be80700f0 legacy=0x7f2be8134e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:12.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.567+0000 7f2be77fe640 10 monclient: _finish_hunting 0 2024-09-17T18:23:12.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.567+0000 7f2be77fe640 1 monclient: found mon.noname-a 2024-09-17T18:23:12.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.567+0000 7f2be77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:23:12.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.567+0000 7f2be77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:23:12.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.567+0000 7f2be77fe640 1 -- 172.21.15.121:0/3122350360 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2be809cd20 con 0x7f2be809e810 2024-09-17T18:23:12.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.569+0000 7f2be77fe640 1 -- 172.21.15.121:0/3122350360 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2be809e810 msgr2=0x7f2be813d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:12.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.569+0000 7f2be77fe640 1 --2- 172.21.15.121:0/3122350360 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2be809e810 0x7f2be813d6e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0009970 tx=0x7f2bd002a2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1de1440be8f02fc5 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:12.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.569+0000 7f2be6ffd640 1 -- 172.21.15.121:0/3122350360 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd0037070 con 0x7f2be809e810 2024-09-17T18:23:12.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.570+0000 7f2be6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:12.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.570+0000 7f2be6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:23:12.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.570+0000 7f2be6ffd640 10 monclient: dump: 2024-09-17T18:23:12.573 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:12.573 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:12.573 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.573 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.574 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:12.574 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:12.574 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:12.574 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:12.574 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:12.574 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:12.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.570+0000 7f2be6ffd640 10 monclient: _finish_auth 0 2024-09-17T18:23:12.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.570+0000 7f2be6ffd640 10 monclient: _check_auth_tickets 2024-09-17T18:23:12.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.570+0000 7f2be6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:23:12.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.570+0000 7f2be6ffd640 1 -- 172.21.15.121:0/3122350360 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bd0032640 con 0x7f2be809e810 2024-09-17T18:23:12.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.570+0000 7f2bee552640 5 monclient: authenticate success, global_id 44151 2024-09-17T18:23:12.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.570+0000 7f2be6ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:23:12.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.570+0000 7f2be6ffd640 1 -- 172.21.15.121:0/3122350360 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd0032a10 con 0x7f2be809e810 2024-09-17T18:23:12.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.570+0000 7f2bee552640 10 monclient: get_monmap_and_config success 2024-09-17T18:23:12.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.570+0000 7f2bee552640 4 set_mon_vals no callback set 2024-09-17T18:23:12.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.570+0000 7f2be6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:12.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.570+0000 7f2be6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:23:12.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.570+0000 7f2be6ffd640 10 monclient: dump: 2024-09-17T18:23:12.575 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:12.575 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:12.575 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.575 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.575 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:12.575 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:12.575 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:12.575 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:12.575 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:12.575 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:12.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.570+0000 7f2bee552640 10 monclient: shutdown 2024-09-17T18:23:12.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.570+0000 7f2bee552640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:23:12.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.570+0000 7f2bee552640 1 -- 172.21.15.121:0/3122350360 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2be809e810 msgr2=0x7f2be813d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:12.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.570+0000 7f2bee552640 1 --2- 172.21.15.121:0/3122350360 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2be809e810 0x7f2be813d6e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0009970 tx=0x7f2bd002a2e0 comp rx=0 tx=0).stop 2024-09-17T18:23:12.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.570+0000 7f2bee552640 1 -- 172.21.15.121:0/3122350360 shutdown_connections 2024-09-17T18:23:12.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.570+0000 7f2bee552640 1 --2- 172.21.15.121:0/3122350360 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2be809e810 0x7f2be813d6e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:12.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.570+0000 7f2bee552640 1 -- 172.21.15.121:0/3122350360 >> 172.21.15.121:0/3122350360 conn(0x7f2be80ab830 msgr2=0x7f2be80a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:23:12.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.570+0000 7f2bee552640 1 -- 172.21.15.121:0/3122350360 shutdown_connections 2024-09-17T18:23:12.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.570+0000 7f2bee552640 1 -- 172.21.15.121:0/3122350360 wait complete. 2024-09-17T18:23:12.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.571+0000 7f2bee552640 10 monclient: build_initial_monmap 2024-09-17T18:23:12.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.571+0000 7f2bee552640 10 monclient: monmap: 2024-09-17T18:23:12.576 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:23:12.576 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:23:12.577 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:23:12.577 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:23:12.577 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:23:12.577 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:12.577 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:23:12.577 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:23:12.577 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:23:12.577 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:12.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.571+0000 7f2bee552640 1 librados: starting msgr at 2024-09-17T18:23:12.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.571+0000 7f2bee552640 1 librados: starting objecter 2024-09-17T18:23:12.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.571+0000 7f2bee552640 1 Processor -- start 2024-09-17T18:23:12.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.571+0000 7f2bee552640 1 -- start start 2024-09-17T18:23:12.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.571+0000 7f2bee552640 1 librados: setting wanted keys 2024-09-17T18:23:12.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.571+0000 7f2bee552640 1 librados: calling monclient init 2024-09-17T18:23:12.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.571+0000 7f2bee552640 10 monclient: init 2024-09-17T18:23:12.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2bee552640 10 monclient: _reopen_session rank -1 2024-09-17T18:23:12.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2bee552640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:23:12.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2bee552640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2be80700f0 0x7f2be8132010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:12.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2bee552640 10 monclient(hunting): picked mon.noname-a con 0x7f2be80700f0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:23:12.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2bee552640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2be809e810 0x7f2be8132550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:12.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2bee552640 10 monclient(hunting): picked mon.noname-b con 0x7f2be809e810 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:23:12.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2bee552640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2be809ec60 0x7f2be809fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:12.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2bee552640 10 monclient(hunting): picked mon.noname-c con 0x7f2be809ec60 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:23:12.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2bee552640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:12.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2bee552640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2be8141350 con 0x7f2be80700f0 2024-09-17T18:23:12.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2bee552640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:12.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2bee552640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2be809d0c0 con 0x7f2be809e810 2024-09-17T18:23:12.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2bee552640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:12.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2bee552640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2be81414d0 con 0x7f2be809ec60 2024-09-17T18:23:12.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2bee552640 10 monclient(hunting): _renew_subs 2024-09-17T18:23:12.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2bee552640 10 monclient(hunting): authenticate will time out at 996.288189s 2024-09-17T18:23:12.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2be7fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2be80700f0 0x7f2be8132010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:12.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2be7fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2be80700f0 0x7f2be8132010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60308/0 (socket says 172.21.15.121:60308) 2024-09-17T18:23:12.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2be7fff640 1 -- 172.21.15.121:0/2614614537 learned_addr learned my addr 172.21.15.121:0/2614614537 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:23:12.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2be7fff640 10 monclient(hunting): get_auth_request con 0x7f2be80700f0 auth_method 0 2024-09-17T18:23:12.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2be7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:12.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2be7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:12.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2be7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:12.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2becaef640 1 --2- 172.21.15.121:0/2614614537 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2be809ec60 0x7f2be809fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:12.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2becaef640 10 monclient(hunting): get_auth_request con 0x7f2be809ec60 auth_method 0 2024-09-17T18:23:12.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2becaef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:12.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2becaef640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:12.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2becaef640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:12.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2be7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:12.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2be7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:12.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2be7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:12.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2becaef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:12.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2becaef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:12.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2becaef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:12.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2be77fe640 1 --2- 172.21.15.121:0/2614614537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2be809e810 0x7f2be8132550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:12.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2be77fe640 10 monclient(hunting): get_auth_request con 0x7f2be809e810 auth_method 0 2024-09-17T18:23:12.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2be77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:12.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2be77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:12.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2be77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:12.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2be7fff640 10 monclient(hunting): handle_auth_done global_id 44154 payload 824 2024-09-17T18:23:12.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2be7fff640 1 -- 172.21.15.121:0/2614614537 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2be809ec60 msgr2=0x7f2be809fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:12.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2be7fff640 1 --2- 172.21.15.121:0/2614614537 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2be809ec60 0x7f2be809fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:12.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2be7fff640 1 -- 172.21.15.121:0/2614614537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2be809e810 msgr2=0x7f2be8132550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:12.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2be7fff640 1 --2- 172.21.15.121:0/2614614537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2be809e810 0x7f2be8132550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:12.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2be7fff640 10 monclient: _finish_hunting 0 2024-09-17T18:23:12.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2be7fff640 1 monclient: found mon.noname-a 2024-09-17T18:23:12.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2be7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:23:12.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2be7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:23:12.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2be7fff640 1 -- 172.21.15.121:0/2614614537 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2be813abd0 con 0x7f2be80700f0 2024-09-17T18:23:12.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.572+0000 7f2be77fe640 1 --2- 172.21.15.121:0/2614614537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2be809e810 0x7f2be8132550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T18:23:12.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.574+0000 7f2be7fff640 1 -- 172.21.15.121:0/2614614537 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2be80700f0 msgr2=0x7f2be8132010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:12.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.574+0000 7f2be7fff640 1 --2- 172.21.15.121:0/2614614537 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2be80700f0 0x7f2be8132010 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2bd800c500 tx=0x7f2bd800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:12.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.574+0000 7f2be4ff9640 10 client.?.objecter ms_handle_connect 0x7f2be80700f0 2024-09-17T18:23:12.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.574+0000 7f2be4ff9640 10 client.?.objecter resend_mon_ops 2024-09-17T18:23:12.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.574+0000 7f2be4ff9640 1 -- 172.21.15.121:0/2614614537 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd80047e0 con 0x7f2be80700f0 2024-09-17T18:23:12.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.574+0000 7f2be4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:12.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.574+0000 7f2be4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:23:12.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.574+0000 7f2be4ff9640 10 monclient: dump: 2024-09-17T18:23:12.588 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:12.588 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:12.588 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.588 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.588 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:12.588 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:12.588 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:12.588 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:12.588 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:12.588 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:12.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.574+0000 7f2be4ff9640 10 monclient: _finish_auth 0 2024-09-17T18:23:12.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.574+0000 7f2be4ff9640 10 monclient: _check_auth_tickets 2024-09-17T18:23:12.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.574+0000 7f2be4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:23:12.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.574+0000 7f2bee552640 5 monclient: authenticate success, global_id 44154 2024-09-17T18:23:12.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.574+0000 7f2be4ff9640 1 -- 172.21.15.121:0/2614614537 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bd8004980 con 0x7f2be80700f0 2024-09-17T18:23:12.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.574+0000 7f2bee552640 10 monclient: _renew_subs 2024-09-17T18:23:12.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.574+0000 7f2bee552640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:23:12.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.574+0000 7f2bee552640 1 -- 172.21.15.121:0/2614614537 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2be80a1920 con 0x7f2be80700f0 2024-09-17T18:23:12.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.574+0000 7f2be4ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T18:23:12.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.574+0000 7f2be4ff9640 1 -- 172.21.15.121:0/2614614537 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd8004d50 con 0x7f2be80700f0 2024-09-17T18:23:12.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.575+0000 7f2be4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:12.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.575+0000 7f2be4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:23:12.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.575+0000 7f2be4ff9640 10 monclient: dump: 2024-09-17T18:23:12.589 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:12.589 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:12.589 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.590 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.590 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:12.590 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:12.590 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:12.590 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:12.590 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:12.590 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:12.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.575+0000 7f2bee552640 10 client.44154.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:23:12.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.575+0000 7f2bee552640 10 monclient: _renew_subs 2024-09-17T18:23:12.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.574+0000 7f2be67fc640 4 set_mon_vals no callback set 2024-09-17T18:23:12.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.575+0000 7f2bee552640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:23:12.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.575+0000 7f2bee552640 1 -- 172.21.15.121:0/2614614537 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2be813aeb0 con 0x7f2be80700f0 2024-09-17T18:23:12.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.575+0000 7f2bee552640 1 librados: init done 2024-09-17T18:23:12.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.575+0000 7f2bee552640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:23:12.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.575+0000 7f2bee552640 1 --2- 172.21.15.121:0/2614614537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2be80a03a0 0x7f2be80a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:12.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.575+0000 7f2bee552640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:12.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.575+0000 7f2bee552640 1 -- 172.21.15.121:0/2614614537 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2be813a610 con 0x7f2be80a03a0 2024-09-17T18:23:12.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.576+0000 7f2be4ff9640 1 -- 172.21.15.121:0/2614614537 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f2bd801a020 con 0x7f2be80700f0 2024-09-17T18:23:12.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.576+0000 7f2be4ff9640 1 --2- 172.21.15.121:0/2614614537 >> v2:172.21.15.173:6812/2423786037 conn(0x7f2bc8040170 0x7f2bc8042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:12.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.576+0000 7f2be77fe640 1 --2- 172.21.15.121:0/2614614537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2be80a03a0 0x7f2be80a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:12.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.576+0000 7f2be77fe640 10 monclient: get_auth_request con 0x7f2be80a03a0 auth_method 0 2024-09-17T18:23:12.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.576+0000 7f2be77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:12.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.576+0000 7f2be77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:12.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.576+0000 7f2be77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:12.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.576+0000 7f2be4ff9640 1 -- 172.21.15.121:0/2614614537 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(178..178 src has 1..178) ==== 5795+0+0 (secure 0 0 0) 0x7f2bd8053030 con 0x7f2be80700f0 2024-09-17T18:23:12.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.576+0000 7f2be4ff9640 10 client.44154.objecter ms_dispatch 0x7f2be8000b90 osd_map(178..178 src has 1..178) 2024-09-17T18:23:12.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.576+0000 7f2be4ff9640 3 client.44154.objecter handle_osd_map got epochs [178,178] > 0 2024-09-17T18:23:12.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.576+0000 7f2be4ff9640 3 client.44154.objecter handle_osd_map decoding full epoch 178 2024-09-17T18:23:12.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.576+0000 7f2be77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:12.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.576+0000 7f2be77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:12.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.576+0000 7f2be77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:12.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.576+0000 7f2be4ff9640 20 client.44154.objecter dump_active .. 0 homeless 2024-09-17T18:23:12.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.576+0000 7f2becaef640 1 --2- 172.21.15.121:0/2614614537 >> v2:172.21.15.173:6812/2423786037 conn(0x7f2bc8040170 0x7f2bc8042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:12.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.576+0000 7f2becaef640 10 monclient: get_auth_request con 0x7f2bc8040170 auth_method 0 2024-09-17T18:23:12.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.576+0000 7f2be77fe640 10 monclient(hunting): handle_auth_done global_id 24520 payload 824 2024-09-17T18:23:12.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.576+0000 7f2be77fe640 1 -- 172.21.15.121:0/2614614537 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2be8134e50 con 0x7f2be80a03a0 2024-09-17T18:23:12.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.578+0000 7f2be4ff9640 10 client.44154.objecter ms_handle_connect 0x7f2be80a03a0 2024-09-17T18:23:12.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.578+0000 7f2be4ff9640 10 client.44154.objecter resend_mon_ops 2024-09-17T18:23:12.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.578+0000 7f2be77fe640 1 -- 172.21.15.121:0/2614614537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2be80a03a0 msgr2=0x7f2be80a0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:12.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.578+0000 7f2be77fe640 1 --2- 172.21.15.121:0/2614614537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2be80a03a0 0x7f2be80a0780 secure :-1 s=READY pgs=7280 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0034000 tx=0x7f2bd00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:12.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.578+0000 7f2be4ff9640 10 client.44154.objecter ms_handle_connect 0x7f2bc8040170 2024-09-17T18:23:12.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.578+0000 7f2becaef640 1 --2- 172.21.15.121:0/2614614537 >> v2:172.21.15.173:6812/2423786037 conn(0x7f2bc8040170 0x7f2bc8042630 secure :-1 s=READY pgs=3937 cs=0 l=1 rev1=1 crypto rx=0x7f2be8139540 tx=0x7f2be0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:12.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.579+0000 7f2be4ff9640 1 -- 172.21.15.121:0/2614614537 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd0042070 con 0x7f2be80a03a0 2024-09-17T18:23:12.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.579+0000 7f2be4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:12.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.579+0000 7f2be4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:23:12.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.579+0000 7f2be4ff9640 10 monclient: dump: 2024-09-17T18:23:12.593 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:12.593 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:12.593 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.593 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.593 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:12.593 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:12.593 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:12.593 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:12.593 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:12.594 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:12.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.579+0000 7f2be4ff9640 1 -- 172.21.15.121:0/2614614537 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2bd003e660 con 0x7f2be80a03a0 2024-09-17T18:23:12.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.579+0000 7f2be4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:23:12.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.579+0000 7f2be4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:23:12.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.579+0000 7f2be4ff9640 1 -- 172.21.15.121:0/2614614537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2be80a03a0 msgr2=0x7f2be80a0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:12.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.579+0000 7f2be4ff9640 1 --2- 172.21.15.121:0/2614614537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2be80a03a0 0x7f2be80a0780 secure :-1 s=READY pgs=7280 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0034000 tx=0x7f2bd00046c0 comp rx=0 tx=0).stop 2024-09-17T18:23:12.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.579+0000 7f2be4ff9640 1 -- 172.21.15.121:0/2614614537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2be80a03a0 msgr2=0x7f2be80a0780 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:23:12.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.579+0000 7f2be4ff9640 1 --2- 172.21.15.121:0/2614614537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2be80a03a0 0x7f2be80a0780 secure :-1 s=CLOSED pgs=7280 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0034000 tx=0x7f2bd00046c0 comp rx=0 tx=0).stop 2024-09-17T18:23:12.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.585+0000 7f2bee552640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:23:12.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.585+0000 7f2bee552640 1 --2- 172.21.15.121:0/2614614537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2be80a0cc0 0x7f2be8135f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:12.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.585+0000 7f2bee552640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:12.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.585+0000 7f2bee552640 1 -- 172.21.15.121:0/2614614537 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2be8140db0 con 0x7f2be80a0cc0 2024-09-17T18:23:12.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.585+0000 7f2be77fe640 1 --2- 172.21.15.121:0/2614614537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2be80a0cc0 0x7f2be8135f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:12.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.585+0000 7f2be77fe640 10 monclient: get_auth_request con 0x7f2be80a0cc0 auth_method 0 2024-09-17T18:23:12.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.585+0000 7f2be77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:12.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.585+0000 7f2be77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:12.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.585+0000 7f2be77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:12.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.585+0000 7f2be77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:12.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.585+0000 7f2be77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:12.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.585+0000 7f2be77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:12.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.585+0000 7f2be77fe640 10 monclient(hunting): handle_auth_done global_id 24523 payload 824 2024-09-17T18:23:12.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.585+0000 7f2be77fe640 1 -- 172.21.15.121:0/2614614537 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2be81396b0 con 0x7f2be80a0cc0 2024-09-17T18:23:12.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.588+0000 7f2be77fe640 1 -- 172.21.15.121:0/2614614537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2be80a0cc0 msgr2=0x7f2be8135f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:12.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.588+0000 7f2be4ff9640 10 client.44154.objecter ms_handle_connect 0x7f2be80a0cc0 2024-09-17T18:23:12.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.588+0000 7f2be77fe640 1 --2- 172.21.15.121:0/2614614537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2be80a0cc0 0x7f2be8135f30 secure :-1 s=READY pgs=7281 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0009970 tx=0x7f2bd003b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:12.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.588+0000 7f2be4ff9640 10 client.44154.objecter resend_mon_ops 2024-09-17T18:23:12.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.588+0000 7f2be4ff9640 1 -- 172.21.15.121:0/2614614537 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd0037040 con 0x7f2be80a0cc0 2024-09-17T18:23:12.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.588+0000 7f2be4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:12.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.588+0000 7f2be4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:23:12.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.588+0000 7f2be4ff9640 10 monclient: dump: 2024-09-17T18:23:12.597 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:12.597 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:12.597 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.598 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.598 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:12.598 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:12.598 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:12.598 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:12.598 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:12.598 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:12.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.588+0000 7f2be4ff9640 1 -- 172.21.15.121:0/2614614537 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f2bd002f4b0 con 0x7f2be80a0cc0 2024-09-17T18:23:12.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.588+0000 7f2be4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:23:12.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.588+0000 7f2be4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:23:12.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.588+0000 7f2be4ff9640 1 -- 172.21.15.121:0/2614614537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2be80a0cc0 msgr2=0x7f2be8135f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:12.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.588+0000 7f2be4ff9640 1 --2- 172.21.15.121:0/2614614537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2be80a0cc0 0x7f2be8135f30 secure :-1 s=READY pgs=7281 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0009970 tx=0x7f2bd003b000 comp rx=0 tx=0).stop 2024-09-17T18:23:12.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.588+0000 7f2be4ff9640 1 -- 172.21.15.121:0/2614614537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2be80a0cc0 msgr2=0x7f2be8135f30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:23:12.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.588+0000 7f2be4ff9640 1 --2- 172.21.15.121:0/2614614537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2be80a0cc0 0x7f2be8135f30 secure :-1 s=CLOSED pgs=7281 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0009970 tx=0x7f2bd003b000 comp rx=0 tx=0).stop 2024-09-17T18:23:12.599 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:23:12.599 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:23:12.599 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:23:12.599 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:23:12.599 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 20, 2024-09-17T18:23:12.599 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 265534, 2024-09-17T18:23:12.600 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:23:12.600 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:23:12.600 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:23:12.600 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:23:12.600 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:23:12.600 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 2, 2024-09-17T18:23:12.600 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:23:12.600 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:23:12.600 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:23:12.600 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:23:12.600 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:23:12.600 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:23:12.600 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:23:12.601 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:23:12.601 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:23:12.601 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:23:12.601 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:23:12.601 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:23:12.601 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:23:12.601 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:23:12.601 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:23:12.601 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:23:12.601 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:23:12.601 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:23:12.601 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:23:12.601 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:23:12.601 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:23:12.602 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:23:12.602 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:23:12.602 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:23:12.602 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:23:12.602 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:23:12.602 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:23:12.602 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:23:12.602 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:12.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:12.602 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:23:12.602 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:23:12.602 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:23:12.602 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:23:12.602 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:23:12.603 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:23:12.603 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:23:12.603 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:23:12.603 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:23:12.603 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:23:12.603 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:23:12.603 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:23:12.603 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:23:12.603 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:23:12.603 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:23:12.603 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:23:12.603 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:23:12.603 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:23:12.603 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:23:12.604 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:23:12.604 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:23:12.604 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:23:12.604 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:23:12.604 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:23:12.604 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:23:12.604 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:23:12.604 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:23:12.604 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:23:12.604 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:23:12.604 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:23:12.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:12.604 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:23:12.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:12.605 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:23:12.605 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:23:12.605 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:23:12.605 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:23:12.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:12.605 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:23:12.605 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:23:12.605 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:12.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:12.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:12.605 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:23:12.605 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:23:12.605 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:12.605 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:12.606 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:12.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:12.606 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:23:12.606 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:23:12.606 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:23:12.606 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:23:12.606 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:23:12.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:12.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:12.606 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:23:12.606 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:23:12.606 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:23:12.606 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:23:12.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:12.607 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:23:12.607 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:23:12.607 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:12.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:12.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:12.607 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:23:12.607 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:23:12.607 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:12.607 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:12.607 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:12.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:12.607 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:23:12.607 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:23:12.608 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:23:12.608 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:23:12.608 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:23:12.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:12.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:12.608 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:23:12.608 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:23:12.608 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:23:12.608 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:23:12.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:12.608 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:23:12.608 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:23:12.608 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:12.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:12.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:12.609 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:23:12.609 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:23:12.609 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:12.609 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:12.609 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:12.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:12.609 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:23:12.609 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:23:12.609 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:23:12.609 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:23:12.609 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:23:12.609 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:12.609 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:12.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:12.610 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:23:12.610 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:23:12.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:12.610 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:23:12.610 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:23:12.610 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:23:12.610 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:12.610 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:23:12.610 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:23:12.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:12.610 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:23:12.610 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:23:12.611 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 3 2024-09-17T18:23:12.611 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:12.611 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:23:12.611 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:23:12.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:12.611 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:23:12.611 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:23:12.611 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 8 2024-09-17T18:23:12.611 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:12.611 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:23:12.611 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": [ 2024-09-17T18:23:12.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:12.611 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:23:12.611 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:23:12.612 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:23:12.612 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:12.612 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:12.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:12.612 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:23:12.612 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:23:12.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.589+0000 7f2bee552640 10 librados: watch_flush enter 2024-09-17T18:23:12.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.589+0000 7f2bee552640 10 librados: watch_flush exit 2024-09-17T18:23:12.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.589+0000 7f2bee552640 20 client.44154.objecter shutdown clearing up homeless session... 2024-09-17T18:23:12.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.589+0000 7f2bee552640 10 client.44154.objecter successfully canceled tick 2024-09-17T18:23:12.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.589+0000 7f2bee552640 1 -- 172.21.15.121:0/2614614537 >> v2:172.21.15.173:6812/2423786037 conn(0x7f2bc8040170 msgr2=0x7f2bc8042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:12.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.589+0000 7f2bee552640 1 --2- 172.21.15.121:0/2614614537 >> v2:172.21.15.173:6812/2423786037 conn(0x7f2bc8040170 0x7f2bc8042630 secure :-1 s=READY pgs=3937 cs=0 l=1 rev1=1 crypto rx=0x7f2be8139540 tx=0x7f2be0009040 comp rx=0 tx=0).stop 2024-09-17T18:23:12.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.589+0000 7f2bee552640 10 monclient: shutdown 2024-09-17T18:23:12.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.589+0000 7f2bee552640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:23:12.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.589+0000 7f2bee552640 1 -- 172.21.15.121:0/2614614537 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2be80700f0 msgr2=0x7f2be8132010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:12.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.589+0000 7f2bee552640 1 --2- 172.21.15.121:0/2614614537 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2be80700f0 0x7f2be8132010 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2bd800c500 tx=0x7f2bd800c9c0 comp rx=0 tx=0).stop 2024-09-17T18:23:12.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.589+0000 7f2becaef640 1 -- 172.21.15.121:0/2614614537 reap_dead start 2024-09-17T18:23:12.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.589+0000 7f2becaef640 1 -- 172.21.15.121:0/2614614537 reap_dead start 2024-09-17T18:23:12.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.589+0000 7f2bee552640 1 -- 172.21.15.121:0/2614614537 shutdown_connections 2024-09-17T18:23:12.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.589+0000 7f2bee552640 1 -- 172.21.15.121:0/2614614537 >> 172.21.15.121:0/2614614537 conn(0x7f2be80ab830 msgr2=0x7f2be8132ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:23:12.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.589+0000 7f2bee552640 1 -- 172.21.15.121:0/2614614537 shutdown_connections 2024-09-17T18:23:12.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.589+0000 7f2bee552640 1 -- 172.21.15.121:0/2614614537 wait complete. 2024-09-17T18:23:12.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.589+0000 7f2bee552640 1 librados: shutdown 2024-09-17T18:23:12.615 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:23:12.616 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-17T18:23:12.616 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-17T18:23:12.616 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-17T18:23:12.616 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c store 2024-09-17T18:23:12.616 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c sync_force --yes-i-really-mean-it 2024-09-17T18:23:12.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.713+0000 7f6d1ea31640 10 monclient: get_monmap_and_config 2024-09-17T18:23:12.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.713+0000 7f6d1ea31640 10 monclient: build_initial_monmap 2024-09-17T18:23:12.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.713+0000 7f6d1ea31640 10 monclient: monmap: 2024-09-17T18:23:12.715 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:23:12.715 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:12.715 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:23:12.713767+0000 2024-09-17T18:23:12.715 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:23:12.713767+0000 2024-09-17T18:23:12.715 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:23:12.715 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:12.715 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:23:12.715 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:23:12.715 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:23:12.715 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:23:12.716 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:23:12.716 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:12.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.713+0000 7f6d1ea31640 1 Processor -- start 2024-09-17T18:23:12.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.714+0000 7f6d1ea31640 1 -- start start 2024-09-17T18:23:12.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.714+0000 7f6d1ea31640 10 monclient: init 2024-09-17T18:23:12.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.714+0000 7f6d1ea31640 10 monclient: _reopen_session rank -1 2024-09-17T18:23:12.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.714+0000 7f6d1ea31640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-17T18:23:12.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.714+0000 7f6d1ea31640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6d180a5b90 addr v1:172.21.15.121:6789/0 2024-09-17T18:23:12.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.714+0000 7f6d1ea31640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f6d180a71c0 0x7f6d1813dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:12.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.714+0000 7f6d1ea31640 10 monclient(hunting): picked mon.noname-c con 0x7f6d180a71c0 addr v2:172.21.15.173:3300/0 2024-09-17T18:23:12.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.714+0000 7f6d1ea31640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d180a6680 0x7f6d181401c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:12.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.714+0000 7f6d1ea31640 10 monclient(hunting): picked mon.noname-b con 0x7f6d180a6680 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:23:12.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.714+0000 7f6d1ea31640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d1809cb90 con 0x7f6d180a5b90 2024-09-17T18:23:12.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.714+0000 7f6d1ea31640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:12.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.714+0000 7f6d1ea31640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f6d1809d4c0 con 0x7f6d180a71c0 2024-09-17T18:23:12.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.714+0000 7f6d1ea31640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:12.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.714+0000 7f6d1ea31640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6d180a6c50 con 0x7f6d180a6680 2024-09-17T18:23:12.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.714+0000 7f6d1ea31640 10 monclient(hunting): _renew_subs 2024-09-17T18:23:12.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.714+0000 7f6d1ea31640 10 monclient(hunting): authenticate will time out at 996.430613s 2024-09-17T18:23:12.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.714+0000 7f6d17fff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f6d180a5b90 0x7f6d180a5f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:50872/0 (socket says 172.21.15.121:50872) 2024-09-17T18:23:12.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.714+0000 7f6d17fff640 1 -- 172.21.15.121:0/2679302038 learned_addr learned my addr 172.21.15.121:0/2679302038 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:23:12.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.714+0000 7f6d1cfce640 1 --2- 172.21.15.121:0/2679302038 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d180a6680 0x7f6d181401c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:12.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.714+0000 7f6d1cfce640 10 monclient(hunting): get_auth_request con 0x7f6d180a6680 auth_method 0 2024-09-17T18:23:12.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.714+0000 7f6d1cfce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:12.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.714+0000 7f6d1cfce640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:12.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.714+0000 7f6d1cfce640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:12.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.714+0000 7f6d1cfce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:12.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.714+0000 7f6d1cfce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:12.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.714+0000 7f6d1cfce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:12.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.714+0000 7f6d1cfce640 10 monclient(hunting): handle_auth_done global_id 34475 payload 274 2024-09-17T18:23:12.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.714+0000 7f6d1cfce640 1 -- 172.21.15.121:0/2679302038 >> v2:172.21.15.173:3300/0 conn(0x7f6d180a71c0 msgr2=0x7f6d1813dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:23:12.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.714+0000 7f6d1cfce640 1 --2- 172.21.15.121:0/2679302038 >> v2:172.21.15.173:3300/0 conn(0x7f6d180a71c0 0x7f6d1813dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:12.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.714+0000 7f6d1cfce640 1 -- 172.21.15.121:0/2679302038 >> v1:172.21.15.121:6789/0 conn(0x7f6d180a5b90 legacy=0x7f6d180a5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:23:12.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.714+0000 7f6d1cfce640 10 monclient: _finish_hunting 0 2024-09-17T18:23:12.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.714+0000 7f6d1cfce640 1 monclient: found mon.noname-b 2024-09-17T18:23:12.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.714+0000 7f6d1cfce640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:23:12.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.714+0000 7f6d1cfce640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:23:12.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.714+0000 7f6d1cfce640 1 -- 172.21.15.121:0/2679302038 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d18070550 con 0x7f6d180a6680 2024-09-17T18:23:12.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.717+0000 7f6d1cfce640 1 -- 172.21.15.121:0/2679302038 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d180a6680 msgr2=0x7f6d181401c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:12.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.717+0000 7f6d1cfce640 1 --2- 172.21.15.121:0/2679302038 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d180a6680 0x7f6d181401c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6d10009d90 tx=0x7f6d1002a490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54a83e62908fcd68 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:12.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.717+0000 7f6d16ffd640 1 -- 172.21.15.121:0/2679302038 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d10007a30 con 0x7f6d180a6680 2024-09-17T18:23:12.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.718+0000 7f6d16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:12.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.718+0000 7f6d16ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:23:12.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.718+0000 7f6d16ffd640 10 monclient: dump: 2024-09-17T18:23:12.732 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:12.732 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:12.732 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.732 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.732 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:12.732 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:12.732 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:12.732 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:12.732 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:12.732 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:12.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.718+0000 7f6d16ffd640 10 monclient: _finish_auth 0 2024-09-17T18:23:12.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.718+0000 7f6d16ffd640 10 monclient: _check_auth_tickets 2024-09-17T18:23:12.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.718+0000 7f6d16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:23:12.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.718+0000 7f6d16ffd640 1 -- 172.21.15.121:0/2679302038 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d10007bd0 con 0x7f6d180a6680 2024-09-17T18:23:12.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.718+0000 7f6d1ea31640 5 monclient: authenticate success, global_id 34475 2024-09-17T18:23:12.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.718+0000 7f6d1ea31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:23:12.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.718+0000 7f6d16ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:23:12.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.718+0000 7f6d16ffd640 1 -- 172.21.15.121:0/2679302038 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d10031400 con 0x7f6d180a6680 2024-09-17T18:23:12.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.718+0000 7f6d1ea31640 10 monclient: get_monmap_and_config success 2024-09-17T18:23:12.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.718+0000 7f6d1ea31640 4 set_mon_vals no callback set 2024-09-17T18:23:12.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.718+0000 7f6d16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:12.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.718+0000 7f6d16ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:23:12.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.718+0000 7f6d16ffd640 10 monclient: dump: 2024-09-17T18:23:12.734 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:12.734 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:12.734 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.734 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.734 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:12.734 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:12.734 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:12.734 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:12.734 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:12.735 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:12.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.718+0000 7f6d1ea31640 10 monclient: shutdown 2024-09-17T18:23:12.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.718+0000 7f6d1ea31640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:23:12.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.718+0000 7f6d1ea31640 1 -- 172.21.15.121:0/2679302038 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d180a6680 msgr2=0x7f6d181401c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:12.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.718+0000 7f6d1ea31640 1 --2- 172.21.15.121:0/2679302038 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d180a6680 0x7f6d181401c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6d10009d90 tx=0x7f6d1002a490 comp rx=0 tx=0).stop 2024-09-17T18:23:12.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.718+0000 7f6d1ea31640 1 -- 172.21.15.121:0/2679302038 shutdown_connections 2024-09-17T18:23:12.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.718+0000 7f6d1ea31640 1 --2- 172.21.15.121:0/2679302038 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d180a6680 0x7f6d181401c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:12.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.718+0000 7f6d1ea31640 1 --2- 172.21.15.121:0/2679302038 >> v2:172.21.15.173:3300/0 conn(0x7f6d180a71c0 0x7f6d1813dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:12.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.718+0000 7f6d1ea31640 1 -- 172.21.15.121:0/2679302038 >> 172.21.15.121:0/2679302038 conn(0x7f6d180a7c90 msgr2=0x7f6d180a80a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:23:12.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.718+0000 7f6d1ea31640 1 -- 172.21.15.121:0/2679302038 shutdown_connections 2024-09-17T18:23:12.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.718+0000 7f6d1ea31640 1 -- 172.21.15.121:0/2679302038 wait complete. 2024-09-17T18:23:12.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.719+0000 7f6d1ea31640 10 monclient: build_initial_monmap 2024-09-17T18:23:12.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.719+0000 7f6d1ea31640 10 monclient: monmap: 2024-09-17T18:23:12.735 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:23:12.736 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:23:12.736 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:23:12.736 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:23:12.736 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:23:12.736 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:12.736 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:23:12.736 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:23:12.736 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:23:12.736 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:12.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.719+0000 7f6d1ea31640 1 librados: starting msgr at 2024-09-17T18:23:12.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.719+0000 7f6d1ea31640 1 librados: starting objecter 2024-09-17T18:23:12.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.719+0000 7f6d1ea31640 1 Processor -- start 2024-09-17T18:23:12.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.719+0000 7f6d1ea31640 1 -- start start 2024-09-17T18:23:12.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.719+0000 7f6d1ea31640 1 librados: setting wanted keys 2024-09-17T18:23:12.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.719+0000 7f6d1ea31640 1 librados: calling monclient init 2024-09-17T18:23:12.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.719+0000 7f6d1ea31640 10 monclient: init 2024-09-17T18:23:12.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d1ea31640 10 monclient: _reopen_session rank -1 2024-09-17T18:23:12.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d1ea31640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T18:23:12.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d1ea31640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6d180a5b90 0x7f6d181cd7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:12.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d1ea31640 10 monclient(hunting): picked mon.noname-b con 0x7f6d180a5b90 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:23:12.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d1ea31640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d180a6680 0x7f6d18138c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:12.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d1ea31640 10 monclient(hunting): picked mon.noname-c con 0x7f6d180a6680 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:23:12.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d1ea31640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d180a71c0 0x7f6d1813de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:12.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d1ea31640 10 monclient(hunting): picked mon.noname-a con 0x7f6d180a71c0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:23:12.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d1ea31640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:12.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d1ea31640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6d180a8ba0 con 0x7f6d180a71c0 2024-09-17T18:23:12.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d1ea31640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:12.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d1ea31640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6d180a8a20 con 0x7f6d180a5b90 2024-09-17T18:23:12.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d1ea31640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:12.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d1ea31640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6d18140700 con 0x7f6d180a6680 2024-09-17T18:23:12.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d1ea31640 10 monclient(hunting): _renew_subs 2024-09-17T18:23:12.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d1ea31640 10 monclient(hunting): authenticate will time out at 996.436282s 2024-09-17T18:23:12.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d1cfce640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d180a71c0 0x7f6d1813de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:12.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d177fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d180a6680 0x7f6d18138c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:12.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d1cfce640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d180a71c0 0x7f6d1813de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60314/0 (socket says 172.21.15.121:60314) 2024-09-17T18:23:12.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d1cfce640 1 -- 172.21.15.121:0/2511672215 learned_addr learned my addr 172.21.15.121:0/2511672215 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:23:12.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d1cfce640 10 monclient(hunting): get_auth_request con 0x7f6d180a71c0 auth_method 0 2024-09-17T18:23:12.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d177fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d180a6680 0x7f6d18138c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:52876/0 (socket says 172.21.15.121:52876) 2024-09-17T18:23:12.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d1cfce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:12.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d1cfce640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:12.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d1cfce640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:12.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d177fe640 10 monclient(hunting): get_auth_request con 0x7f6d180a6680 auth_method 0 2024-09-17T18:23:12.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:12.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d177fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:12.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:12.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d1cfce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:12.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d1cfce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:12.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d1cfce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:12.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:12.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:12.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:12.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d17fff640 1 --2- 172.21.15.121:0/2511672215 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6d180a5b90 0x7f6d181cd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:12.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d17fff640 10 monclient(hunting): get_auth_request con 0x7f6d180a5b90 auth_method 0 2024-09-17T18:23:12.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:12.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d17fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:12.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d17fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:12.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d177fe640 10 monclient(hunting): handle_auth_done global_id 34478 payload 824 2024-09-17T18:23:12.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d177fe640 1 -- 172.21.15.121:0/2511672215 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6d180a5b90 msgr2=0x7f6d181cd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:12.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d177fe640 1 --2- 172.21.15.121:0/2511672215 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6d180a5b90 0x7f6d181cd7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:12.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d177fe640 1 -- 172.21.15.121:0/2511672215 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d180a71c0 msgr2=0x7f6d1813de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:12.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d177fe640 1 --2- 172.21.15.121:0/2511672215 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d180a71c0 0x7f6d1813de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:12.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d177fe640 10 monclient: _finish_hunting 0 2024-09-17T18:23:12.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d177fe640 1 monclient: found mon.noname-c 2024-09-17T18:23:12.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:23:12.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d177fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:23:12.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d177fe640 1 -- 172.21.15.121:0/2511672215 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d181235f0 con 0x7f6d180a6680 2024-09-17T18:23:12.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d1cfce640 1 --2- 172.21.15.121:0/2511672215 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d180a71c0 0x7f6d1813de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:23:12.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.720+0000 7f6d17fff640 1 --2- 172.21.15.121:0/2511672215 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6d180a5b90 0x7f6d181cd7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T18:23:12.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.722+0000 7f6d177fe640 1 -- 172.21.15.121:0/2511672215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d180a6680 msgr2=0x7f6d18138c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:12.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.722+0000 7f6d177fe640 1 --2- 172.21.15.121:0/2511672215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d180a6680 0x7f6d18138c40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c00b3f0 tx=0x7f6d0c00b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:12.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.722+0000 7f6d14ff9640 10 client.?.objecter ms_handle_connect 0x7f6d180a6680 2024-09-17T18:23:12.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.722+0000 7f6d14ff9640 10 client.?.objecter resend_mon_ops 2024-09-17T18:23:12.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.722+0000 7f6d14ff9640 1 -- 172.21.15.121:0/2511672215 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d0c015020 con 0x7f6d180a6680 2024-09-17T18:23:12.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.723+0000 7f6d14ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:12.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.723+0000 7f6d14ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:23:12.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.723+0000 7f6d14ff9640 10 monclient: dump: 2024-09-17T18:23:12.741 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:12.741 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:12.741 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.741 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.741 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:12.741 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:12.742 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:12.742 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:12.742 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:12.742 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:12.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.723+0000 7f6d14ff9640 10 monclient: _finish_auth 0 2024-09-17T18:23:12.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.723+0000 7f6d14ff9640 10 monclient: _check_auth_tickets 2024-09-17T18:23:12.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.723+0000 7f6d14ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:23:12.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.723+0000 7f6d14ff9640 1 -- 172.21.15.121:0/2511672215 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d0c011a40 con 0x7f6d180a6680 2024-09-17T18:23:12.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.723+0000 7f6d1ea31640 5 monclient: authenticate success, global_id 34478 2024-09-17T18:23:12.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.723+0000 7f6d14ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T18:23:12.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.723+0000 7f6d14ff9640 1 -- 172.21.15.121:0/2511672215 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d0c011e30 con 0x7f6d180a6680 2024-09-17T18:23:12.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.723+0000 7f6d1ea31640 10 monclient: _renew_subs 2024-09-17T18:23:12.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.723+0000 7f6d1ea31640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:23:12.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.723+0000 7f6d1ea31640 1 -- 172.21.15.121:0/2511672215 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d1813f910 con 0x7f6d180a6680 2024-09-17T18:23:12.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.723+0000 7f6d14ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:12.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.723+0000 7f6d14ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:23:12.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.723+0000 7f6d14ff9640 10 monclient: dump: 2024-09-17T18:23:12.743 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:12.743 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:12.743 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.743 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.743 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:12.743 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:12.743 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:12.743 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:12.743 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:12.743 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:12.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.723+0000 7f6d167fc640 4 set_mon_vals no callback set 2024-09-17T18:23:12.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.723+0000 7f6d1ea31640 10 client.34478.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:23:12.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.723+0000 7f6d1ea31640 10 monclient: _renew_subs 2024-09-17T18:23:12.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.723+0000 7f6d1ea31640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:23:12.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.723+0000 7f6d1ea31640 1 -- 172.21.15.121:0/2511672215 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d181221a0 con 0x7f6d180a6680 2024-09-17T18:23:12.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.723+0000 7f6d1ea31640 1 librados: init done 2024-09-17T18:23:12.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.723+0000 7f6d1ea31640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:23:12.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.723+0000 7f6d1ea31640 1 --2- 172.21.15.121:0/2511672215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d181230c0 0x7f6d1813e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:12.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.723+0000 7f6d1ea31640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:12.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.723+0000 7f6d1ea31640 1 -- 172.21.15.121:0/2511672215 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6d18121df0 con 0x7f6d181230c0 2024-09-17T18:23:12.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.723+0000 7f6d14ff9640 1 -- 172.21.15.121:0/2511672215 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f6d0c018020 con 0x7f6d180a6680 2024-09-17T18:23:12.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.723+0000 7f6d17fff640 1 --2- 172.21.15.121:0/2511672215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d181230c0 0x7f6d1813e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:12.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.723+0000 7f6d17fff640 10 monclient: get_auth_request con 0x7f6d181230c0 auth_method 0 2024-09-17T18:23:12.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.723+0000 7f6d17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:12.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.723+0000 7f6d17fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:12.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.723+0000 7f6d17fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:12.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.724+0000 7f6d17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:12.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.724+0000 7f6d17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:12.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.724+0000 7f6d17fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:12.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.724+0000 7f6d14ff9640 1 --2- 172.21.15.121:0/2511672215 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6d04040170 0x7f6d04042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:12.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.724+0000 7f6d17fff640 10 monclient(hunting): handle_auth_done global_id 34481 payload 824 2024-09-17T18:23:12.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.724+0000 7f6d17fff640 1 -- 172.21.15.121:0/2511672215 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6d180a2380 con 0x7f6d181230c0 2024-09-17T18:23:12.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.724+0000 7f6d14ff9640 1 -- 172.21.15.121:0/2511672215 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(178..178 src has 1..178) ==== 5795+0+0 (secure 0 0 0) 0x7f6d0c053810 con 0x7f6d180a6680 2024-09-17T18:23:12.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.724+0000 7f6d14ff9640 10 client.34478.objecter ms_dispatch 0x7f6d18000b90 osd_map(178..178 src has 1..178) 2024-09-17T18:23:12.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.724+0000 7f6d14ff9640 3 client.34478.objecter handle_osd_map got epochs [178,178] > 0 2024-09-17T18:23:12.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.724+0000 7f6d14ff9640 3 client.34478.objecter handle_osd_map decoding full epoch 178 2024-09-17T18:23:12.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.724+0000 7f6d14ff9640 20 client.34478.objecter dump_active .. 0 homeless 2024-09-17T18:23:12.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.724+0000 7f6d1cfce640 1 --2- 172.21.15.121:0/2511672215 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6d04040170 0x7f6d04042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:12.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.724+0000 7f6d1cfce640 10 monclient: get_auth_request con 0x7f6d04040170 auth_method 0 2024-09-17T18:23:12.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.726+0000 7f6d14ff9640 10 client.34478.objecter ms_handle_connect 0x7f6d181230c0 2024-09-17T18:23:12.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.726+0000 7f6d17fff640 1 -- 172.21.15.121:0/2511672215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d181230c0 msgr2=0x7f6d1813e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:12.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.726+0000 7f6d14ff9640 10 client.34478.objecter resend_mon_ops 2024-09-17T18:23:12.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.726+0000 7f6d17fff640 1 --2- 172.21.15.121:0/2511672215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d181230c0 0x7f6d1813e390 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6d0800e520 tx=0x7f6d0800e9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:12.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.726+0000 7f6d14ff9640 1 -- 172.21.15.121:0/2511672215 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d08005cf0 con 0x7f6d181230c0 2024-09-17T18:23:12.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.726+0000 7f6d14ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:12.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.726+0000 7f6d14ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:23:12.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.726+0000 7f6d14ff9640 10 monclient: dump: 2024-09-17T18:23:12.746 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:12.746 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:12.746 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.746 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.746 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:12.747 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:12.747 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:12.747 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:12.747 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:12.747 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:12.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.726+0000 7f6d14ff9640 1 -- 172.21.15.121:0/2511672215 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6d08018070 con 0x7f6d181230c0 2024-09-17T18:23:12.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.726+0000 7f6d14ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:23:12.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.726+0000 7f6d14ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:23:12.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.726+0000 7f6d14ff9640 1 -- 172.21.15.121:0/2511672215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d181230c0 msgr2=0x7f6d1813e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:12.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.726+0000 7f6d14ff9640 1 --2- 172.21.15.121:0/2511672215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d181230c0 0x7f6d1813e390 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6d0800e520 tx=0x7f6d0800e9e0 comp rx=0 tx=0).stop 2024-09-17T18:23:12.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.726+0000 7f6d14ff9640 1 -- 172.21.15.121:0/2511672215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d181230c0 msgr2=0x7f6d1813e390 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:23:12.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.726+0000 7f6d14ff9640 1 --2- 172.21.15.121:0/2511672215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d181230c0 0x7f6d1813e390 secure :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6d0800e520 tx=0x7f6d0800e9e0 comp rx=0 tx=0).stop 2024-09-17T18:23:12.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.726+0000 7f6d14ff9640 10 client.34478.objecter ms_handle_connect 0x7f6d04040170 2024-09-17T18:23:12.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.726+0000 7f6d1cfce640 1 --2- 172.21.15.121:0/2511672215 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6d04040170 0x7f6d04042630 secure :-1 s=READY pgs=3938 cs=0 l=1 rev1=1 crypto rx=0x7f6d10009bd0 tx=0x7f6d1002f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:12.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.733+0000 7f6d1ea31640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-17T18:23:12.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.733+0000 7f6d1ea31640 1 --2- 172.21.15.121:0/2511672215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d18139180 0x7f6d1813e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:12.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.733+0000 7f6d1ea31640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:12.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.733+0000 7f6d1ea31640 1 -- 172.21.15.121:0/2511672215 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6d180a5f70 con 0x7f6d18139180 2024-09-17T18:23:12.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.733+0000 7f6d17fff640 1 --2- 172.21.15.121:0/2511672215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d18139180 0x7f6d1813e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:12.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.733+0000 7f6d17fff640 10 monclient: get_auth_request con 0x7f6d18139180 auth_method 0 2024-09-17T18:23:12.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.733+0000 7f6d17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:12.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.733+0000 7f6d17fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:12.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.733+0000 7f6d17fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:12.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.733+0000 7f6d17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:12.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.733+0000 7f6d17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:12.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.733+0000 7f6d17fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:12.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.734+0000 7f6d17fff640 10 monclient(hunting): handle_auth_done global_id 34484 payload 824 2024-09-17T18:23:12.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.734+0000 7f6d17fff640 1 -- 172.21.15.121:0/2511672215 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7f6d181229a0 con 0x7f6d18139180 2024-09-17T18:23:12.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.736+0000 7f6d14ff9640 10 client.34478.objecter ms_handle_connect 0x7f6d18139180 2024-09-17T18:23:12.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.736+0000 7f6d17fff640 1 -- 172.21.15.121:0/2511672215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d18139180 msgr2=0x7f6d1813e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:12.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.736+0000 7f6d14ff9640 10 client.34478.objecter resend_mon_ops 2024-09-17T18:23:12.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.736+0000 7f6d17fff640 1 --2- 172.21.15.121:0/2511672215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d18139180 0x7f6d1813e8d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6d08017000 tx=0x7f6d08004940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:12.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.736+0000 7f6d14ff9640 1 -- 172.21.15.121:0/2511672215 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d080072e0 con 0x7f6d18139180 2024-09-17T18:23:12.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.736+0000 7f6d14ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:12.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.736+0000 7f6d14ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:23:12.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.736+0000 7f6d14ff9640 10 monclient: dump: 2024-09-17T18:23:12.751 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:12.751 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:12.751 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.751 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.751 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:12.751 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:12.751 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:12.751 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:12.752 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:12.752 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:12.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.736+0000 7f6d14ff9640 1 -- 172.21.15.121:0/2511672215 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (secure 0 0 0) 0x7f6d080072e0 con 0x7f6d18139180 2024-09-17T18:23:12.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.736+0000 7f6d14ff9640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-17T18:23:12.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.736+0000 7f6d14ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:23:12.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.736+0000 7f6d14ff9640 1 -- 172.21.15.121:0/2511672215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d18139180 msgr2=0x7f6d1813e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:12.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.736+0000 7f6d14ff9640 1 --2- 172.21.15.121:0/2511672215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d18139180 0x7f6d1813e8d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6d08017000 tx=0x7f6d08004940 comp rx=0 tx=0).stop 2024-09-17T18:23:12.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.736+0000 7f6d14ff9640 1 -- 172.21.15.121:0/2511672215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d18139180 msgr2=0x7f6d1813e8d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:23:12.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.736+0000 7f6d14ff9640 1 --2- 172.21.15.121:0/2511672215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d18139180 0x7f6d1813e8d0 secure :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6d08017000 tx=0x7f6d08004940 comp rx=0 tx=0).stop 2024-09-17T18:23:12.752 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:23:12.753 INFO:teuthology.orchestra.run.smithi121.stdout: "ret": 0, 2024-09-17T18:23:12.753 INFO:teuthology.orchestra.run.smithi121.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-09-17T18:23:12.753 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:23:12.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.737+0000 7f6d1ea31640 10 librados: watch_flush enter 2024-09-17T18:23:12.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.737+0000 7f6d1ea31640 10 librados: watch_flush exit 2024-09-17T18:23:12.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.737+0000 7f6d1ea31640 20 client.34478.objecter shutdown clearing up homeless session... 2024-09-17T18:23:12.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.737+0000 7f6d1ea31640 10 client.34478.objecter successfully canceled tick 2024-09-17T18:23:12.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.737+0000 7f6d1ea31640 1 -- 172.21.15.121:0/2511672215 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6d04040170 msgr2=0x7f6d04042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:12.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.737+0000 7f6d1ea31640 1 --2- 172.21.15.121:0/2511672215 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6d04040170 0x7f6d04042630 secure :-1 s=READY pgs=3938 cs=0 l=1 rev1=1 crypto rx=0x7f6d10009bd0 tx=0x7f6d1002f000 comp rx=0 tx=0).stop 2024-09-17T18:23:12.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.737+0000 7f6d1ea31640 10 monclient: shutdown 2024-09-17T18:23:12.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.737+0000 7f6d1ea31640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:23:12.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.737+0000 7f6d1ea31640 1 -- 172.21.15.121:0/2511672215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d180a6680 msgr2=0x7f6d18138c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:12.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.737+0000 7f6d1ea31640 1 --2- 172.21.15.121:0/2511672215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d180a6680 0x7f6d18138c40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c00b3f0 tx=0x7f6d0c00b8b0 comp rx=0 tx=0).stop 2024-09-17T18:23:12.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.737+0000 7f6d1cfce640 1 -- 172.21.15.121:0/2511672215 reap_dead start 2024-09-17T18:23:12.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.737+0000 7f6d1ea31640 1 -- 172.21.15.121:0/2511672215 shutdown_connections 2024-09-17T18:23:12.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.737+0000 7f6d1ea31640 1 --2- 172.21.15.121:0/2511672215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d180a6680 0x7f6d18138c40 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:12.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.737+0000 7f6d1ea31640 1 -- 172.21.15.121:0/2511672215 >> 172.21.15.121:0/2511672215 conn(0x7f6d180a7c90 msgr2=0x7f6d180a12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:23:12.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.737+0000 7f6d1ea31640 1 -- 172.21.15.121:0/2511672215 shutdown_connections 2024-09-17T18:23:12.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.737+0000 7f6d1ea31640 1 -- 172.21.15.121:0/2511672215 wait complete. 2024-09-17T18:23:12.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.737+0000 7f6d1ea31640 1 librados: shutdown 2024-09-17T18:23:12.755 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-17T18:23:12.756 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-17T18:23:12.756 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T18:23:12.767 INFO:tasks.ceph.mon.c:Stopped 2024-09-17T18:23:12.767 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-17T18:23:12.767 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:23:12.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.901+0000 7f6d36818640 10 monclient: get_monmap_and_config 2024-09-17T18:23:12.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.901+0000 7f6d36818640 10 monclient: build_initial_monmap 2024-09-17T18:23:12.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.901+0000 7f6d36818640 10 monclient: monmap: 2024-09-17T18:23:12.903 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:23:12.903 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:12.903 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:23:12.902073+0000 2024-09-17T18:23:12.903 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:23:12.902073+0000 2024-09-17T18:23:12.903 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:23:12.904 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:12.904 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:23:12.904 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:23:12.904 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:23:12.904 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:23:12.904 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:23:12.904 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:12.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.902+0000 7f6d36818640 1 Processor -- start 2024-09-17T18:23:12.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.902+0000 7f6d36818640 1 -- start start 2024-09-17T18:23:12.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.902+0000 7f6d36818640 10 monclient: init 2024-09-17T18:23:12.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.902+0000 7f6d36818640 10 monclient: _reopen_session rank -1 2024-09-17T18:23:12.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.902+0000 7f6d36818640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-17T18:23:12.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.902+0000 7f6d36818640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6d300a0a70 addr v1:172.21.15.173:6789/0 2024-09-17T18:23:12.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.902+0000 7f6d36818640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6d301348f0 0x7f6d30139c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:12.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.902+0000 7f6d36818640 10 monclient(hunting): picked mon.noname-a con 0x7f6d301348f0 addr v2:172.21.15.121:3300/0 2024-09-17T18:23:12.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.902+0000 7f6d36818640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6d3013a150 addr v1:172.21.15.121:6789/0 2024-09-17T18:23:12.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.902+0000 7f6d36818640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d3009f5b0 con 0x7f6d3013a150 2024-09-17T18:23:12.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.902+0000 7f6d36818640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d300a1560 con 0x7f6d300a0a70 2024-09-17T18:23:12.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.902+0000 7f6d36818640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:12.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.902+0000 7f6d36818640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f6d3009ef80 con 0x7f6d301348f0 2024-09-17T18:23:12.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.902+0000 7f6d36818640 10 monclient(hunting): _renew_subs 2024-09-17T18:23:12.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.902+0000 7f6d36818640 10 monclient(hunting): authenticate will time out at 996.618850s 2024-09-17T18:23:12.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.902+0000 7f6d2f7fe640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6d301348f0 0x7f6d30139c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:12.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.902+0000 7f6d34db5640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f6d3013a150 0x7f6d3013c540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:50880/0 (socket says 172.21.15.121:50880) 2024-09-17T18:23:12.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.902+0000 7f6d34db5640 1 -- 172.21.15.121:0/2363055053 learned_addr learned my addr 172.21.15.121:0/2363055053 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:23:12.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.902+0000 7f6d2f7fe640 10 monclient(hunting): get_auth_request con 0x7f6d301348f0 auth_method 0 2024-09-17T18:23:12.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.902+0000 7f6d2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:12.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.902+0000 7f6d2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:12.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.902+0000 7f6d2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:12.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.903+0000 7f6d2f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:12.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.903+0000 7f6d2f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:12.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.903+0000 7f6d2f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:12.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.903+0000 7f6d2f7fe640 10 monclient(hunting): handle_auth_done global_id 44160 payload 274 2024-09-17T18:23:12.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.903+0000 7f6d2f7fe640 1 -- 172.21.15.121:0/2363055053 >> v1:172.21.15.173:6789/0 conn(0x7f6d300a0a70 legacy=0x7f6d300a0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:23:12.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.903+0000 7f6d2f7fe640 1 -- 172.21.15.121:0/2363055053 >> v1:172.21.15.121:6789/0 conn(0x7f6d3013a150 legacy=0x7f6d3013c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:23:12.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.903+0000 7f6d2f7fe640 10 monclient: _finish_hunting 0 2024-09-17T18:23:12.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.903+0000 7f6d2f7fe640 1 monclient: found mon.noname-a 2024-09-17T18:23:12.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.903+0000 7f6d2f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:23:12.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.903+0000 7f6d2f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:23:12.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.903+0000 7f6d2f7fe640 1 -- 172.21.15.121:0/2363055053 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d3009eca0 con 0x7f6d301348f0 2024-09-17T18:23:12.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.906+0000 7f6d2f7fe640 1 -- 172.21.15.121:0/2363055053 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d301348f0 msgr2=0x7f6d30139c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:12.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.906+0000 7f6d2f7fe640 1 --2- 172.21.15.121:0/2363055053 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d301348f0 0x7f6d30139c10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6d240097d0 tx=0x7f6d2402b110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b805279654c23ef4 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:12.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.906+0000 7f6d2effd640 1 -- 172.21.15.121:0/2363055053 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d24038070 con 0x7f6d301348f0 2024-09-17T18:23:12.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.906+0000 7f6d2effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:12.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.906+0000 7f6d2effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:23:12.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.906+0000 7f6d2effd640 10 monclient: dump: 2024-09-17T18:23:12.912 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:12.912 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:12.912 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.913 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.913 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:12.913 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:12.913 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:12.913 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:12.913 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:12.913 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:12.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.906+0000 7f6d2effd640 10 monclient: _finish_auth 0 2024-09-17T18:23:12.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.906+0000 7f6d2effd640 10 monclient: _check_auth_tickets 2024-09-17T18:23:12.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.906+0000 7f6d2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:23:12.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.906+0000 7f6d2effd640 1 -- 172.21.15.121:0/2363055053 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d24033640 con 0x7f6d301348f0 2024-09-17T18:23:12.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.906+0000 7f6d36818640 5 monclient: authenticate success, global_id 44160 2024-09-17T18:23:12.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.906+0000 7f6d36818640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:23:12.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.906+0000 7f6d2effd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:23:12.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.906+0000 7f6d36818640 10 monclient: get_monmap_and_config success 2024-09-17T18:23:12.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.906+0000 7f6d2effd640 1 -- 172.21.15.121:0/2363055053 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d24033a10 con 0x7f6d301348f0 2024-09-17T18:23:12.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.906+0000 7f6d2effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:12.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.906+0000 7f6d36818640 4 set_mon_vals no callback set 2024-09-17T18:23:12.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.906+0000 7f6d2effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:23:12.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.906+0000 7f6d2effd640 10 monclient: dump: 2024-09-17T18:23:12.914 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:12.914 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:12.914 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.914 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.914 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:12.914 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:12.914 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:12.914 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:12.914 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:12.915 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:12.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.906+0000 7f6d36818640 10 monclient: shutdown 2024-09-17T18:23:12.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.906+0000 7f6d36818640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:23:12.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.906+0000 7f6d36818640 1 -- 172.21.15.121:0/2363055053 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d301348f0 msgr2=0x7f6d30139c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:12.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.906+0000 7f6d36818640 1 --2- 172.21.15.121:0/2363055053 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d301348f0 0x7f6d30139c10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6d240097d0 tx=0x7f6d2402b110 comp rx=0 tx=0).stop 2024-09-17T18:23:12.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.906+0000 7f6d36818640 1 -- 172.21.15.121:0/2363055053 shutdown_connections 2024-09-17T18:23:12.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.906+0000 7f6d36818640 1 --2- 172.21.15.121:0/2363055053 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d301348f0 0x7f6d30139c10 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:12.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.906+0000 7f6d36818640 1 -- 172.21.15.121:0/2363055053 >> 172.21.15.121:0/2363055053 conn(0x7f6d300a3b50 msgr2=0x7f6d300a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:23:12.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.906+0000 7f6d36818640 1 -- 172.21.15.121:0/2363055053 shutdown_connections 2024-09-17T18:23:12.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.906+0000 7f6d36818640 1 -- 172.21.15.121:0/2363055053 wait complete. 2024-09-17T18:23:12.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.907+0000 7f6d36818640 10 monclient: build_initial_monmap 2024-09-17T18:23:12.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.907+0000 7f6d36818640 10 monclient: monmap: 2024-09-17T18:23:12.915 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:23:12.915 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:23:12.916 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:23:12.916 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:23:12.916 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:23:12.916 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:12.916 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:23:12.916 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:23:12.916 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:23:12.916 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:12.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.907+0000 7f6d36818640 1 librados: starting msgr at 2024-09-17T18:23:12.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.907+0000 7f6d36818640 1 librados: starting objecter 2024-09-17T18:23:12.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.907+0000 7f6d36818640 1 Processor -- start 2024-09-17T18:23:12.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.907+0000 7f6d36818640 1 -- start start 2024-09-17T18:23:12.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.907+0000 7f6d36818640 1 librados: setting wanted keys 2024-09-17T18:23:12.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.907+0000 7f6d36818640 1 librados: calling monclient init 2024-09-17T18:23:12.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.907+0000 7f6d36818640 10 monclient: init 2024-09-17T18:23:12.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d36818640 10 monclient: _reopen_session rank -1 2024-09-17T18:23:12.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d36818640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T18:23:12.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d36818640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d300a0a70 0x7f6d300734a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:12.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d36818640 10 monclient(hunting): picked mon.noname-c con 0x7f6d300a0a70 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:23:12.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d36818640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d301348f0 0x7f6d30133ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:12.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d36818640 10 monclient(hunting): picked mon.noname-a con 0x7f6d301348f0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:23:12.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d36818640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6d3013a150 0x7f6d30134010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:12.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d36818640 10 monclient(hunting): picked mon.noname-b con 0x7f6d3013a150 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:23:12.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d36818640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:12.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d36818640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6d3009de60 con 0x7f6d301348f0 2024-09-17T18:23:12.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d36818640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:12.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d36818640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6d3009dce0 con 0x7f6d3013a150 2024-09-17T18:23:12.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d36818640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:12.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d2ffff640 1 -- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d300a0a70 msgr2=0x7f6d300734a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:23:12.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d36818640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6d300a1a50 con 0x7f6d300a0a70 2024-09-17T18:23:12.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d36818640 10 monclient(hunting): _renew_subs 2024-09-17T18:23:12.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d36818640 10 monclient(hunting): authenticate will time out at 996.624453s 2024-09-17T18:23:12.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d2ffff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d300a0a70 0x7f6d300734a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:23:12.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d2f7fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d301348f0 0x7f6d30133ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:12.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d2f7fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d301348f0 0x7f6d30133ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60342/0 (socket says 172.21.15.121:60342) 2024-09-17T18:23:12.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d2f7fe640 1 -- 172.21.15.121:0/1282007289 learned_addr learned my addr 172.21.15.121:0/1282007289 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:23:12.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d2f7fe640 10 monclient(hunting): get_auth_request con 0x7f6d301348f0 auth_method 0 2024-09-17T18:23:12.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:12.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:12.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:12.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d2f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:12.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d2f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:12.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d2f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:12.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d34db5640 1 --2- 172.21.15.121:0/1282007289 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6d3013a150 0x7f6d30134010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:12.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d34db5640 10 monclient(hunting): get_auth_request con 0x7f6d3013a150 auth_method 0 2024-09-17T18:23:12.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d34db5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:12.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d34db5640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:12.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d34db5640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:12.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d2f7fe640 10 monclient(hunting): handle_auth_done global_id 44163 payload 824 2024-09-17T18:23:12.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d2f7fe640 1 -- 172.21.15.121:0/1282007289 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d300a0a70 msgr2=0x7f6d300734a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:23:12.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d2f7fe640 1 --2- 172.21.15.121:0/1282007289 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d300a0a70 0x7f6d300734a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:12.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d2f7fe640 1 -- 172.21.15.121:0/1282007289 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6d3013a150 msgr2=0x7f6d30134010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:12.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d2f7fe640 1 --2- 172.21.15.121:0/1282007289 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6d3013a150 0x7f6d30134010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:12.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d2f7fe640 10 monclient: _finish_hunting 0 2024-09-17T18:23:12.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d2f7fe640 1 monclient: found mon.noname-a 2024-09-17T18:23:12.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d2f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:23:12.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d2f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:23:12.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.908+0000 7f6d2f7fe640 1 -- 172.21.15.121:0/1282007289 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d3009e240 con 0x7f6d301348f0 2024-09-17T18:23:12.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.911+0000 7f6d2f7fe640 1 -- 172.21.15.121:0/1282007289 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d301348f0 msgr2=0x7f6d30133ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:12.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.911+0000 7f6d2f7fe640 1 --2- 172.21.15.121:0/1282007289 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d301348f0 0x7f6d30133ad0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6d240097d0 tx=0x7f6d240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:12.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.911+0000 7f6d2cff9640 10 client.?.objecter ms_handle_connect 0x7f6d301348f0 2024-09-17T18:23:12.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.911+0000 7f6d2cff9640 10 client.?.objecter resend_mon_ops 2024-09-17T18:23:12.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.911+0000 7f6d2cff9640 1 -- 172.21.15.121:0/1282007289 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d24042020 con 0x7f6d301348f0 2024-09-17T18:23:12.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.911+0000 7f6d2cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:12.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.911+0000 7f6d2cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:23:12.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.911+0000 7f6d2cff9640 10 monclient: dump: 2024-09-17T18:23:12.923 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:12.923 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:12.923 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.923 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.923 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:12.923 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:12.923 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:12.923 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:12.924 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:12.924 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:12.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.911+0000 7f6d2cff9640 10 monclient: _finish_auth 0 2024-09-17T18:23:12.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.911+0000 7f6d2cff9640 10 monclient: _check_auth_tickets 2024-09-17T18:23:12.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.911+0000 7f6d2cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:23:12.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.911+0000 7f6d2cff9640 1 -- 172.21.15.121:0/1282007289 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d24031880 con 0x7f6d301348f0 2024-09-17T18:23:12.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.911+0000 7f6d36818640 5 monclient: authenticate success, global_id 44163 2024-09-17T18:23:12.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.911+0000 7f6d36818640 10 monclient: _renew_subs 2024-09-17T18:23:12.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.911+0000 7f6d36818640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:23:12.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.911+0000 7f6d36818640 1 -- 172.21.15.121:0/1282007289 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d30121c20 con 0x7f6d301348f0 2024-09-17T18:23:12.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.911+0000 7f6d2cff9640 10 monclient: handle_config config(0 keys) 2024-09-17T18:23:12.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.911+0000 7f6d2cff9640 1 -- 172.21.15.121:0/1282007289 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d24038040 con 0x7f6d301348f0 2024-09-17T18:23:12.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.911+0000 7f6d2cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:12.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.911+0000 7f6d2cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:23:12.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.911+0000 7f6d2cff9640 10 monclient: dump: 2024-09-17T18:23:12.925 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:12.925 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:12.925 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.925 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:12.925 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:12.925 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:12.925 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:12.925 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:12.925 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:12.925 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:12.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.911+0000 7f6d2e7fc640 4 set_mon_vals no callback set 2024-09-17T18:23:12.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.911+0000 7f6d36818640 10 client.44163.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:23:12.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.911+0000 7f6d36818640 10 monclient: _renew_subs 2024-09-17T18:23:12.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.911+0000 7f6d36818640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:23:12.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.911+0000 7f6d36818640 1 -- 172.21.15.121:0/1282007289 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d30120cc0 con 0x7f6d301348f0 2024-09-17T18:23:12.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.911+0000 7f6d36818640 1 librados: init done 2024-09-17T18:23:12.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.912+0000 7f6d36818640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:23:12.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.912+0000 7f6d36818640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:23:12.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.912+0000 7f6d36818640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:23:12.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.912+0000 7f6d2cff9640 1 -- 172.21.15.121:0/1282007289 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f6d24045020 con 0x7f6d301348f0 2024-09-17T18:23:12.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.912+0000 7f6d36818640 1 -- 172.21.15.121:0/1282007289 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d300a0e50 con 0x7f6d301348f0 2024-09-17T18:23:12.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.912+0000 7f6d2cff9640 1 --2- 172.21.15.121:0/1282007289 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6d1c040120 0x7f6d1c0425e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:12.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.912+0000 7f6d2cff9640 1 -- 172.21.15.121:0/1282007289 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(178..178 src has 1..178) ==== 5795+0+0 (secure 0 0 0) 0x7f6d24074090 con 0x7f6d301348f0 2024-09-17T18:23:12.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.912+0000 7f6d2cff9640 10 client.44163.objecter ms_dispatch 0x7f6d30000b90 osd_map(178..178 src has 1..178) 2024-09-17T18:23:12.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.912+0000 7f6d2cff9640 3 client.44163.objecter handle_osd_map got epochs [178,178] > 0 2024-09-17T18:23:12.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.912+0000 7f6d2cff9640 3 client.44163.objecter handle_osd_map decoding full epoch 178 2024-09-17T18:23:12.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.912+0000 7f6d2cff9640 20 client.44163.objecter dump_active .. 0 homeless 2024-09-17T18:23:12.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.912+0000 7f6d2ffff640 1 --2- 172.21.15.121:0/1282007289 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6d1c040120 0x7f6d1c0425e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:12.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.912+0000 7f6d2ffff640 10 monclient: get_auth_request con 0x7f6d1c040120 auth_method 0 2024-09-17T18:23:12.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.915+0000 7f6d2cff9640 10 client.44163.objecter ms_handle_connect 0x7f6d1c040120 2024-09-17T18:23:12.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.915+0000 7f6d2ffff640 1 --2- 172.21.15.121:0/1282007289 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6d1c040120 0x7f6d1c0425e0 secure :-1 s=READY pgs=3939 cs=0 l=1 rev1=1 crypto rx=0x7f6d20003770 tx=0x7f6d20007a10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:12.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.916+0000 7f6d2cff9640 1 -- 172.21.15.121:0/1282007289 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6d2404c030 con 0x7f6d301348f0 2024-09-17T18:23:12.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.916+0000 7f6d2cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:23:12.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:12.916+0000 7f6d2cff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:23:13.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:13.157+0000 7f6d36818640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:23:13.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:13.157+0000 7f6d36818640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:23:13.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:13.157+0000 7f6d36818640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:23:13.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:13.157+0000 7f6d36818640 1 -- 172.21.15.121:0/1282007289 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6d3006ce60 con 0x7f6d301348f0 2024-09-17T18:23:13.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:13.158+0000 7f6d2cff9640 1 -- 172.21.15.121:0/1282007289 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f6d300a1560 con 0x7f6d301348f0 2024-09-17T18:23:13.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:13.158+0000 7f6d2cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:23:13.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:13.158+0000 7f6d2cff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:23:13.160 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:23:13.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:13.160+0000 7f6d36818640 10 librados: watch_flush enter 2024-09-17T18:23:13.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:13.160+0000 7f6d36818640 10 librados: watch_flush exit 2024-09-17T18:23:13.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:13.160+0000 7f6d36818640 20 client.44163.objecter shutdown clearing up homeless session... 2024-09-17T18:23:13.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:13.160+0000 7f6d36818640 10 client.44163.objecter successfully canceled tick 2024-09-17T18:23:13.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:13.160+0000 7f6d36818640 1 -- 172.21.15.121:0/1282007289 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6d1c040120 msgr2=0x7f6d1c0425e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:13.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:13.160+0000 7f6d36818640 1 --2- 172.21.15.121:0/1282007289 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6d1c040120 0x7f6d1c0425e0 secure :-1 s=READY pgs=3939 cs=0 l=1 rev1=1 crypto rx=0x7f6d20003770 tx=0x7f6d20007a10 comp rx=0 tx=0).stop 2024-09-17T18:23:13.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:13.160+0000 7f6d36818640 10 monclient: shutdown 2024-09-17T18:23:13.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:13.160+0000 7f6d36818640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:23:13.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:13.160+0000 7f6d36818640 1 -- 172.21.15.121:0/1282007289 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d301348f0 msgr2=0x7f6d30133ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:13.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:13.160+0000 7f6d36818640 1 --2- 172.21.15.121:0/1282007289 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d301348f0 0x7f6d30133ad0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6d240097d0 tx=0x7f6d240046c0 comp rx=0 tx=0).stop 2024-09-17T18:23:13.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:13.160+0000 7f6d36818640 1 -- 172.21.15.121:0/1282007289 shutdown_connections 2024-09-17T18:23:13.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:13.160+0000 7f6d36818640 1 --2- 172.21.15.121:0/1282007289 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6d1c040120 0x7f6d1c0425e0 unknown :-1 s=CLOSED pgs=3939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:13.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:13.160+0000 7f6d36818640 1 --2- 172.21.15.121:0/1282007289 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6d3013a150 0x7f6d30134010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:13.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:13.160+0000 7f6d36818640 1 --2- 172.21.15.121:0/1282007289 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d301348f0 0x7f6d30133ad0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:13.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:13.160+0000 7f6d36818640 1 --2- 172.21.15.121:0/1282007289 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d300a0a70 0x7f6d300734a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:13.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:13.160+0000 7f6d36818640 1 -- 172.21.15.121:0/1282007289 >> 172.21.15.121:0/1282007289 conn(0x7f6d300a3b50 msgr2=0x7f6d300a1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:23:13.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:13.160+0000 7f6d36818640 1 -- 172.21.15.121:0/1282007289 shutdown_connections 2024-09-17T18:23:13.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:13.160+0000 7f6d36818640 1 -- 172.21.15.121:0/1282007289 wait complete. 2024-09-17T18:23:13.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:13.160+0000 7f6d36818640 1 librados: shutdown 2024-09-17T18:23:16.174 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:23:16.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.280+0000 7fd36487f640 10 monclient: get_monmap_and_config 2024-09-17T18:23:16.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.280+0000 7fd36487f640 10 monclient: build_initial_monmap 2024-09-17T18:23:16.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.280+0000 7fd36487f640 10 monclient: monmap: 2024-09-17T18:23:16.282 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:23:16.282 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:16.282 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:23:16.280682+0000 2024-09-17T18:23:16.282 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:23:16.280682+0000 2024-09-17T18:23:16.282 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:23:16.282 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:16.282 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:23:16.282 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:23:16.282 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:23:16.282 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:23:16.283 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:23:16.283 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:16.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.280+0000 7fd36487f640 1 Processor -- start 2024-09-17T18:23:16.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.280+0000 7fd36487f640 1 -- start start 2024-09-17T18:23:16.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.280+0000 7fd36487f640 10 monclient: init 2024-09-17T18:23:16.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd36487f640 10 monclient: _reopen_session rank -1 2024-09-17T18:23:16.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd36487f640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-17T18:23:16.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd36487f640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd3600a8620 0x7fd3600a5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:16.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd36487f640 10 monclient(hunting): picked mon.noname-a con 0x7fd3600a8620 addr v2:172.21.15.121:3300/0 2024-09-17T18:23:16.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd36487f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3600a6350 0x7fd3600a6730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:16.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd36487f640 10 monclient(hunting): picked mon.noname-b con 0x7fd3600a6350 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:23:16.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd36487f640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fd36013cbf0 0x7fd36013efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:16.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd36487f640 10 monclient(hunting): picked mon.noname-c con 0x7fd36013cbf0 addr v2:172.21.15.173:3300/0 2024-09-17T18:23:16.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd36487f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:16.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd36487f640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fd3600a8a00 con 0x7fd3600a8620 2024-09-17T18:23:16.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd36487f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:16.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd36487f640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fd36009cf50 con 0x7fd36013cbf0 2024-09-17T18:23:16.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd36487f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:16.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd36487f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd36009f0e0 con 0x7fd3600a6350 2024-09-17T18:23:16.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd36487f640 10 monclient(hunting): _renew_subs 2024-09-17T18:23:16.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd36487f640 10 monclient(hunting): authenticate will time out at 999.997507s 2024-09-17T18:23:16.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd35dd9b640 1 -- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3600a6350 msgr2=0x7fd3600a6730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:23:16.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd35dd9b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3600a6350 0x7fd3600a6730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:23:16.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd35e59c640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd3600a8620 0x7fd3600a5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:16.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd35e59c640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd3600a8620 0x7fd3600a5e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60354/0 (socket says 172.21.15.121:60354) 2024-09-17T18:23:16.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd35e59c640 1 -- 172.21.15.121:0/2601519051 learned_addr learned my addr 172.21.15.121:0/2601519051 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:23:16.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd35e59c640 10 monclient(hunting): get_auth_request con 0x7fd3600a8620 auth_method 0 2024-09-17T18:23:16.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd35e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:16.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd35e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:16.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd35e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:16.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd35e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:16.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd35e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:16.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd35e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:16.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd35ed9d640 1 --2- 172.21.15.121:0/2601519051 >> v2:172.21.15.173:3300/0 conn(0x7fd36013cbf0 0x7fd36013efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:16.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd35ed9d640 10 monclient(hunting): get_auth_request con 0x7fd36013cbf0 auth_method 0 2024-09-17T18:23:16.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd35ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:16.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd35ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:16.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd35ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:16.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd35e59c640 10 monclient(hunting): handle_auth_done global_id 44166 payload 274 2024-09-17T18:23:16.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd35e59c640 1 -- 172.21.15.121:0/2601519051 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3600a6350 msgr2=0x7fd3600a6730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:23:16.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd35e59c640 1 --2- 172.21.15.121:0/2601519051 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3600a6350 0x7fd3600a6730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:16.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd35e59c640 1 -- 172.21.15.121:0/2601519051 >> v2:172.21.15.173:3300/0 conn(0x7fd36013cbf0 msgr2=0x7fd36013efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:23:16.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd35e59c640 1 --2- 172.21.15.121:0/2601519051 >> v2:172.21.15.173:3300/0 conn(0x7fd36013cbf0 0x7fd36013efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:16.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd35e59c640 10 monclient: _finish_hunting 0 2024-09-17T18:23:16.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd35e59c640 1 monclient: found mon.noname-a 2024-09-17T18:23:16.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd35e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:23:16.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd35e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:23:16.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.281+0000 7fd35e59c640 1 -- 172.21.15.121:0/2601519051 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd360070900 con 0x7fd3600a8620 2024-09-17T18:23:16.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.284+0000 7fd35e59c640 1 -- 172.21.15.121:0/2601519051 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3600a8620 msgr2=0x7fd3600a5e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:16.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.284+0000 7fd35e59c640 1 --2- 172.21.15.121:0/2601519051 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3600a8620 0x7fd3600a5e10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd36009e3c0 tx=0x7fd35002a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=745f445fe388a35d server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:16.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.284+0000 7fd35d59a640 1 -- 172.21.15.121:0/2601519051 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd350037070 con 0x7fd3600a8620 2024-09-17T18:23:16.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.284+0000 7fd35d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:16.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.284+0000 7fd35d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:23:16.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.284+0000 7fd35d59a640 10 monclient: dump: 2024-09-17T18:23:16.295 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:16.295 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:16.295 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:16.295 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:16.295 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:16.295 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:16.296 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:16.296 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:16.296 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:16.296 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:16.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.284+0000 7fd35d59a640 10 monclient: _finish_auth 0 2024-09-17T18:23:16.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.284+0000 7fd35d59a640 10 monclient: _check_auth_tickets 2024-09-17T18:23:16.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.284+0000 7fd35d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:23:16.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.284+0000 7fd35d59a640 1 -- 172.21.15.121:0/2601519051 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd350032640 con 0x7fd3600a8620 2024-09-17T18:23:16.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.284+0000 7fd35d59a640 10 monclient: handle_config config(0 keys) 2024-09-17T18:23:16.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.284+0000 7fd35d59a640 1 -- 172.21.15.121:0/2601519051 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd350032a10 con 0x7fd3600a8620 2024-09-17T18:23:16.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.284+0000 7fd36487f640 5 monclient: authenticate success, global_id 44166 2024-09-17T18:23:16.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.284+0000 7fd36487f640 10 monclient: get_monmap_and_config success 2024-09-17T18:23:16.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.284+0000 7fd36487f640 4 set_mon_vals no callback set 2024-09-17T18:23:16.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.284+0000 7fd35d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:16.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.284+0000 7fd35d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:23:16.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.284+0000 7fd35d59a640 10 monclient: dump: 2024-09-17T18:23:16.297 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:16.297 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:16.297 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:16.297 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:16.297 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:16.297 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:16.297 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:16.297 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:16.297 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:16.297 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:16.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.285+0000 7fd36487f640 10 monclient: shutdown 2024-09-17T18:23:16.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.285+0000 7fd36487f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:23:16.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.285+0000 7fd36487f640 1 -- 172.21.15.121:0/2601519051 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3600a8620 msgr2=0x7fd3600a5e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:16.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.285+0000 7fd36487f640 1 --2- 172.21.15.121:0/2601519051 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3600a8620 0x7fd3600a5e10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd36009e3c0 tx=0x7fd35002a0d0 comp rx=0 tx=0).stop 2024-09-17T18:23:16.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.285+0000 7fd36487f640 1 -- 172.21.15.121:0/2601519051 shutdown_connections 2024-09-17T18:23:16.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.285+0000 7fd36487f640 1 --2- 172.21.15.121:0/2601519051 >> v2:172.21.15.173:3300/0 conn(0x7fd36013cbf0 0x7fd36013efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:16.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.285+0000 7fd36487f640 1 --2- 172.21.15.121:0/2601519051 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3600a6350 0x7fd3600a6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:16.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.285+0000 7fd36487f640 1 --2- 172.21.15.121:0/2601519051 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3600a8620 0x7fd3600a5e10 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:16.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.285+0000 7fd36487f640 1 -- 172.21.15.121:0/2601519051 >> 172.21.15.121:0/2601519051 conn(0x7fd3600abcb0 msgr2=0x7fd3600a7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:23:16.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.285+0000 7fd36487f640 1 -- 172.21.15.121:0/2601519051 shutdown_connections 2024-09-17T18:23:16.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.285+0000 7fd36487f640 1 -- 172.21.15.121:0/2601519051 wait complete. 2024-09-17T18:23:16.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.286+0000 7fd36487f640 10 monclient: build_initial_monmap 2024-09-17T18:23:16.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.286+0000 7fd36487f640 10 monclient: monmap: 2024-09-17T18:23:16.298 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:23:16.299 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:23:16.299 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:23:16.299 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:23:16.299 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:23:16.299 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:16.299 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:23:16.299 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:23:16.299 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:23:16.299 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:16.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.286+0000 7fd36487f640 1 librados: starting msgr at 2024-09-17T18:23:16.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.286+0000 7fd36487f640 1 librados: starting objecter 2024-09-17T18:23:16.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.286+0000 7fd36487f640 1 Processor -- start 2024-09-17T18:23:16.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.286+0000 7fd36487f640 1 -- start start 2024-09-17T18:23:16.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.286+0000 7fd36487f640 1 librados: setting wanted keys 2024-09-17T18:23:16.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.286+0000 7fd36487f640 1 librados: calling monclient init 2024-09-17T18:23:16.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.286+0000 7fd36487f640 10 monclient: init 2024-09-17T18:23:16.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd36487f640 10 monclient: _reopen_session rank -1 2024-09-17T18:23:16.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd36487f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T18:23:16.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd36487f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3600a6350 0x7fd3601cd760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:16.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd36487f640 10 monclient(hunting): picked mon.noname-b con 0x7fd3600a6350 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:23:16.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd36487f640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3600a8620 0x7fd36013bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:16.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd36487f640 10 monclient(hunting): picked mon.noname-a con 0x7fd3600a8620 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:23:16.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd36487f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd36013cbf0 0x7fd360123c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:16.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd36487f640 10 monclient(hunting): picked mon.noname-c con 0x7fd36013cbf0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:23:16.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd36487f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:16.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd36487f640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd3600a7490 con 0x7fd3600a8620 2024-09-17T18:23:16.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd36487f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:16.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd36487f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd36009d930 con 0x7fd3600a6350 2024-09-17T18:23:16.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd36487f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:16.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd36487f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd3600a7610 con 0x7fd36013cbf0 2024-09-17T18:23:16.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd36487f640 10 monclient(hunting): _renew_subs 2024-09-17T18:23:16.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd36487f640 10 monclient(hunting): authenticate will time out at 1000.003187s 2024-09-17T18:23:16.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd35ed9d640 1 -- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd36013cbf0 msgr2=0x7fd360123c60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:23:16.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd35ed9d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd36013cbf0 0x7fd360123c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:23:16.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd35dd9b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3600a8620 0x7fd36013bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:16.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd35dd9b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3600a8620 0x7fd36013bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60358/0 (socket says 172.21.15.121:60358) 2024-09-17T18:23:16.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd35dd9b640 1 -- 172.21.15.121:0/2992072324 learned_addr learned my addr 172.21.15.121:0/2992072324 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:23:16.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd35dd9b640 10 monclient(hunting): get_auth_request con 0x7fd3600a8620 auth_method 0 2024-09-17T18:23:16.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd35dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:16.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd35dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:16.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd35dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:16.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd35dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:16.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd35dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:16.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd35dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:16.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd35e59c640 1 --2- 172.21.15.121:0/2992072324 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3600a6350 0x7fd3601cd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:16.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd35e59c640 10 monclient(hunting): get_auth_request con 0x7fd3600a6350 auth_method 0 2024-09-17T18:23:16.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd35e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:16.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd35e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:16.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd35e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:16.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd35dd9b640 10 monclient(hunting): handle_auth_done global_id 44169 payload 824 2024-09-17T18:23:16.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd35dd9b640 1 -- 172.21.15.121:0/2992072324 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd36013cbf0 msgr2=0x7fd360123c60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:23:16.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd35dd9b640 1 --2- 172.21.15.121:0/2992072324 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd36013cbf0 0x7fd360123c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:16.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd35dd9b640 1 -- 172.21.15.121:0/2992072324 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3600a6350 msgr2=0x7fd3601cd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:16.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd35dd9b640 1 --2- 172.21.15.121:0/2992072324 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3600a6350 0x7fd3601cd760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:16.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd35dd9b640 10 monclient: _finish_hunting 0 2024-09-17T18:23:16.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd35dd9b640 1 monclient: found mon.noname-a 2024-09-17T18:23:16.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd35dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:23:16.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd35dd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:23:16.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.287+0000 7fd35dd9b640 1 -- 172.21.15.121:0/2992072324 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd36006fc10 con 0x7fd3600a8620 2024-09-17T18:23:16.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.289+0000 7fd35dd9b640 1 -- 172.21.15.121:0/2992072324 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3600a8620 msgr2=0x7fd36013bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:16.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.289+0000 7fd35dd9b640 1 --2- 172.21.15.121:0/2992072324 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3600a8620 0x7fd36013bf80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd35400b580 tx=0x7fd35400ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:16.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.289+0000 7fd34effd640 10 client.?.objecter ms_handle_connect 0x7fd3600a8620 2024-09-17T18:23:16.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.289+0000 7fd34effd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:23:16.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.289+0000 7fd34effd640 1 -- 172.21.15.121:0/2992072324 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3540050f0 con 0x7fd3600a8620 2024-09-17T18:23:16.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.289+0000 7fd34effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:16.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.289+0000 7fd34effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:23:16.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.289+0000 7fd34effd640 10 monclient: dump: 2024-09-17T18:23:16.304 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:16.304 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:16.304 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:16.304 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:16.304 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:16.304 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:16.304 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:16.304 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:16.304 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:16.304 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:16.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.289+0000 7fd34effd640 10 monclient: _finish_auth 0 2024-09-17T18:23:16.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.289+0000 7fd34effd640 10 monclient: _check_auth_tickets 2024-09-17T18:23:16.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.289+0000 7fd34effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:23:16.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.289+0000 7fd34effd640 1 -- 172.21.15.121:0/2992072324 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd354015070 con 0x7fd3600a8620 2024-09-17T18:23:16.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.289+0000 7fd34effd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:23:16.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.289+0000 7fd34effd640 1 -- 172.21.15.121:0/2992072324 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3540054d0 con 0x7fd3600a8620 2024-09-17T18:23:16.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.289+0000 7fd36487f640 5 monclient: authenticate success, global_id 44169 2024-09-17T18:23:16.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.289+0000 7fd36487f640 10 monclient: _renew_subs 2024-09-17T18:23:16.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.289+0000 7fd36487f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:23:16.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.289+0000 7fd36487f640 1 -- 172.21.15.121:0/2992072324 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3600a2600 con 0x7fd3600a8620 2024-09-17T18:23:16.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.289+0000 7fd34effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:16.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.289+0000 7fd34effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:23:16.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.289+0000 7fd34effd640 10 monclient: dump: 2024-09-17T18:23:16.305 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:16.305 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:16.305 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:16.305 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:16.305 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:16.305 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:16.305 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:16.305 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:16.306 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:16.306 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:16.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.289+0000 7fd35cd99640 4 set_mon_vals no callback set 2024-09-17T18:23:16.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.290+0000 7fd36487f640 10 client.44169.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:23:16.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.290+0000 7fd36487f640 10 monclient: _renew_subs 2024-09-17T18:23:16.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.290+0000 7fd36487f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:23:16.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.290+0000 7fd36487f640 1 -- 172.21.15.121:0/2992072324 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd360127920 con 0x7fd3600a8620 2024-09-17T18:23:16.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.290+0000 7fd36487f640 1 librados: init done 2024-09-17T18:23:16.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.290+0000 7fd34effd640 1 -- 172.21.15.121:0/2992072324 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fd354005670 con 0x7fd3600a8620 2024-09-17T18:23:16.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.290+0000 7fd36487f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:23:16.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.290+0000 7fd36487f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:23:16.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.290+0000 7fd36487f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:23:16.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.290+0000 7fd36487f640 1 -- 172.21.15.121:0/2992072324 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd36009e430 con 0x7fd3600a8620 2024-09-17T18:23:16.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.291+0000 7fd34effd640 1 --2- 172.21.15.121:0/2992072324 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd340040120 0x7fd3400425e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:16.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.291+0000 7fd34effd640 1 -- 172.21.15.121:0/2992072324 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(178..178 src has 1..178) ==== 5795+0+0 (secure 0 0 0) 0x7fd354053720 con 0x7fd3600a8620 2024-09-17T18:23:16.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.291+0000 7fd34effd640 10 client.44169.objecter ms_dispatch 0x7fd360000b90 osd_map(178..178 src has 1..178) 2024-09-17T18:23:16.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.291+0000 7fd34effd640 3 client.44169.objecter handle_osd_map got epochs [178,178] > 0 2024-09-17T18:23:16.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.291+0000 7fd34effd640 3 client.44169.objecter handle_osd_map decoding full epoch 178 2024-09-17T18:23:16.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.291+0000 7fd34effd640 20 client.44169.objecter dump_active .. 0 homeless 2024-09-17T18:23:16.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.291+0000 7fd35e59c640 1 --2- 172.21.15.121:0/2992072324 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd340040120 0x7fd3400425e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:16.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.291+0000 7fd35e59c640 10 monclient: get_auth_request con 0x7fd340040120 auth_method 0 2024-09-17T18:23:16.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.293+0000 7fd35e59c640 1 --2- 172.21.15.121:0/2992072324 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd340040120 0x7fd3400425e0 secure :-1 s=READY pgs=3940 cs=0 l=1 rev1=1 crypto rx=0x7fd360077c30 tx=0x7fd35002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:16.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.293+0000 7fd34effd640 10 client.44169.objecter ms_handle_connect 0x7fd340040120 2024-09-17T18:23:16.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.295+0000 7fd34effd640 1 -- 172.21.15.121:0/2992072324 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd36009e430 con 0x7fd3600a8620 2024-09-17T18:23:16.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.295+0000 7fd34effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:23:16.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.295+0000 7fd34effd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:23:16.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.537+0000 7fd36487f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:23:16.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.537+0000 7fd36487f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:23:16.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.537+0000 7fd36487f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:23:16.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.537+0000 7fd36487f640 1 -- 172.21.15.121:0/2992072324 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd36006cd40 con 0x7fd3600a8620 2024-09-17T18:23:16.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.538+0000 7fd34effd640 1 -- 172.21.15.121:0/2992072324 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fd3600a2600 con 0x7fd3600a8620 2024-09-17T18:23:16.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.538+0000 7fd34effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:23:16.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.538+0000 7fd34effd640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:23:16.540 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:23:16.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.540+0000 7fd36487f640 10 librados: watch_flush enter 2024-09-17T18:23:16.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.540+0000 7fd36487f640 10 librados: watch_flush exit 2024-09-17T18:23:16.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.540+0000 7fd36487f640 20 client.44169.objecter shutdown clearing up homeless session... 2024-09-17T18:23:16.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.540+0000 7fd36487f640 10 client.44169.objecter successfully canceled tick 2024-09-17T18:23:16.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.540+0000 7fd36487f640 1 -- 172.21.15.121:0/2992072324 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd340040120 msgr2=0x7fd3400425e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:16.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.540+0000 7fd36487f640 1 --2- 172.21.15.121:0/2992072324 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd340040120 0x7fd3400425e0 secure :-1 s=READY pgs=3940 cs=0 l=1 rev1=1 crypto rx=0x7fd360077c30 tx=0x7fd35002e000 comp rx=0 tx=0).stop 2024-09-17T18:23:16.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.540+0000 7fd36487f640 10 monclient: shutdown 2024-09-17T18:23:16.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.540+0000 7fd36487f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:23:16.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.540+0000 7fd36487f640 1 -- 172.21.15.121:0/2992072324 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3600a8620 msgr2=0x7fd36013bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:16.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.540+0000 7fd36487f640 1 --2- 172.21.15.121:0/2992072324 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3600a8620 0x7fd36013bf80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd35400b580 tx=0x7fd35400ba40 comp rx=0 tx=0).stop 2024-09-17T18:23:16.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.540+0000 7fd36487f640 1 -- 172.21.15.121:0/2992072324 shutdown_connections 2024-09-17T18:23:16.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.540+0000 7fd36487f640 1 --2- 172.21.15.121:0/2992072324 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd340040120 0x7fd3400425e0 unknown :-1 s=CLOSED pgs=3940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:16.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.540+0000 7fd36487f640 1 --2- 172.21.15.121:0/2992072324 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd36013cbf0 0x7fd360123c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:16.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.540+0000 7fd36487f640 1 --2- 172.21.15.121:0/2992072324 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3600a8620 0x7fd36013bf80 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:16.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.540+0000 7fd36487f640 1 --2- 172.21.15.121:0/2992072324 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3600a6350 0x7fd3601cd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:16.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.540+0000 7fd36487f640 1 -- 172.21.15.121:0/2992072324 >> 172.21.15.121:0/2992072324 conn(0x7fd3600abcb0 msgr2=0x7fd36013ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:23:16.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.540+0000 7fd36487f640 1 -- 172.21.15.121:0/2992072324 shutdown_connections 2024-09-17T18:23:16.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.540+0000 7fd36487f640 1 -- 172.21.15.121:0/2992072324 wait complete. 2024-09-17T18:23:16.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:16.540+0000 7fd36487f640 1 librados: shutdown 2024-09-17T18:23:19.554 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:23:19.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.660+0000 7f14252ed640 10 monclient: get_monmap_and_config 2024-09-17T18:23:19.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.660+0000 7f14252ed640 10 monclient: build_initial_monmap 2024-09-17T18:23:19.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.660+0000 7f14252ed640 10 monclient: monmap: 2024-09-17T18:23:19.662 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:23:19.662 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:19.662 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:23:19.661031+0000 2024-09-17T18:23:19.662 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:23:19.661031+0000 2024-09-17T18:23:19.662 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:23:19.662 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:19.663 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:23:19.663 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:23:19.663 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:23:19.663 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:23:19.663 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:23:19.663 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:19.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.661+0000 7f14252ed640 1 Processor -- start 2024-09-17T18:23:19.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.661+0000 7f14252ed640 1 -- start start 2024-09-17T18:23:19.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.661+0000 7f14252ed640 10 monclient: init 2024-09-17T18:23:19.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.661+0000 7f14252ed640 10 monclient: _reopen_session rank -1 2024-09-17T18:23:19.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.661+0000 7f14252ed640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-17T18:23:19.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.661+0000 7f14252ed640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f1420070200 0x7f14200a5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:19.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.661+0000 7f14252ed640 10 monclient(hunting): picked mon.noname-c con 0x7f1420070200 addr v2:172.21.15.173:3300/0 2024-09-17T18:23:19.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.661+0000 7f14252ed640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f14200a61e0 addr v1:172.21.15.121:6789/0 2024-09-17T18:23:19.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.661+0000 7f14252ed640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f14200a65f0 0x7f14201401b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:19.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.661+0000 7f14252ed640 10 monclient(hunting): picked mon.noname-a con 0x7f14200a65f0 addr v2:172.21.15.121:3300/0 2024-09-17T18:23:19.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.661+0000 7f14252ed640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14200a5740 con 0x7f14200a61e0 2024-09-17T18:23:19.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.661+0000 7f14252ed640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:19.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.661+0000 7f14252ed640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f14200705e0 con 0x7f14200a65f0 2024-09-17T18:23:19.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.661+0000 7f14252ed640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:19.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.661+0000 7f14252ed640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f14200a2910 con 0x7f1420070200 2024-09-17T18:23:19.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.661+0000 7f14252ed640 10 monclient(hunting): _renew_subs 2024-09-17T18:23:19.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.661+0000 7f14252ed640 10 monclient(hunting): authenticate will time out at 1003.377777s 2024-09-17T18:23:19.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.661+0000 7f141f7fe640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f14200a65f0 0x7f14201401b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:19.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.661+0000 7f1417fff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f14200a61e0 0x7f142013ca80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:39662/0 (socket says 172.21.15.121:39662) 2024-09-17T18:23:19.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.661+0000 7f1417fff640 1 -- 172.21.15.121:0/755987410 learned_addr learned my addr 172.21.15.121:0/755987410 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:23:19.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.661+0000 7f141f7fe640 10 monclient(hunting): get_auth_request con 0x7f14200a65f0 auth_method 0 2024-09-17T18:23:19.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.661+0000 7f141f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:19.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.661+0000 7f141f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:19.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.661+0000 7f141f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:19.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.662+0000 7f141f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:19.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.662+0000 7f141f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:19.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.662+0000 7f141f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:19.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.662+0000 7f141effd640 1 --2- 172.21.15.121:0/755987410 >> v2:172.21.15.173:3300/0 conn(0x7f1420070200 0x7f14200a5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:19.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.662+0000 7f141effd640 10 monclient(hunting): get_auth_request con 0x7f1420070200 auth_method 0 2024-09-17T18:23:19.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.662+0000 7f141effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:19.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.662+0000 7f141effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:19.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.662+0000 7f141effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:19.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.662+0000 7f141f7fe640 10 monclient(hunting): handle_auth_done global_id 44172 payload 274 2024-09-17T18:23:19.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.662+0000 7f141f7fe640 1 -- 172.21.15.121:0/755987410 >> v2:172.21.15.173:3300/0 conn(0x7f1420070200 msgr2=0x7f14200a5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:23:19.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.662+0000 7f141f7fe640 1 --2- 172.21.15.121:0/755987410 >> v2:172.21.15.173:3300/0 conn(0x7f1420070200 0x7f14200a5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:19.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.662+0000 7f141f7fe640 1 -- 172.21.15.121:0/755987410 >> v1:172.21.15.121:6789/0 conn(0x7f14200a61e0 legacy=0x7f142013ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:23:19.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.662+0000 7f141f7fe640 10 monclient: _finish_hunting 0 2024-09-17T18:23:19.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.662+0000 7f141f7fe640 1 monclient: found mon.noname-a 2024-09-17T18:23:19.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.662+0000 7f141f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:23:19.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.662+0000 7f141f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:23:19.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.662+0000 7f141f7fe640 1 -- 172.21.15.121:0/755987410 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14200a6dd0 con 0x7f14200a65f0 2024-09-17T18:23:19.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.664+0000 7f141f7fe640 1 -- 172.21.15.121:0/755987410 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14200a65f0 msgr2=0x7f14201401b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:19.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.664+0000 7f141f7fe640 1 --2- 172.21.15.121:0/755987410 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14200a65f0 0x7f14201401b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1420070900 tx=0x7f141802a4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1092cea76a2fc592 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:19.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.665+0000 7f141e7fc640 1 -- 172.21.15.121:0/755987410 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1418007950 con 0x7f14200a65f0 2024-09-17T18:23:19.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.665+0000 7f141e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:19.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.665+0000 7f141e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:23:19.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.665+0000 7f141e7fc640 10 monclient: dump: 2024-09-17T18:23:19.677 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:19.677 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:19.677 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:19.677 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:19.677 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:19.678 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:19.678 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:19.678 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:19.678 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:19.678 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:19.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.665+0000 7f141e7fc640 10 monclient: _finish_auth 0 2024-09-17T18:23:19.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.665+0000 7f141e7fc640 10 monclient: _check_auth_tickets 2024-09-17T18:23:19.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.665+0000 7f141e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:23:19.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.665+0000 7f141e7fc640 1 -- 172.21.15.121:0/755987410 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1418007af0 con 0x7f14200a65f0 2024-09-17T18:23:19.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.665+0000 7f14252ed640 5 monclient: authenticate success, global_id 44172 2024-09-17T18:23:19.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.665+0000 7f14252ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:23:19.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.665+0000 7f141e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:23:19.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.665+0000 7f14252ed640 10 monclient: get_monmap_and_config success 2024-09-17T18:23:19.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.665+0000 7f141e7fc640 1 -- 172.21.15.121:0/755987410 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1418031400 con 0x7f14200a65f0 2024-09-17T18:23:19.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.665+0000 7f141e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:19.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.665+0000 7f14252ed640 4 set_mon_vals no callback set 2024-09-17T18:23:19.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.665+0000 7f141e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:23:19.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.665+0000 7f141e7fc640 10 monclient: dump: 2024-09-17T18:23:19.679 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:19.679 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:19.679 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:19.679 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:19.679 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:19.680 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:19.680 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:19.680 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:19.680 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:19.680 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:19.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.665+0000 7f14252ed640 10 monclient: shutdown 2024-09-17T18:23:19.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.665+0000 7f14252ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:23:19.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.665+0000 7f14252ed640 1 -- 172.21.15.121:0/755987410 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14200a65f0 msgr2=0x7f14201401b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:19.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.665+0000 7f14252ed640 1 --2- 172.21.15.121:0/755987410 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14200a65f0 0x7f14201401b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1420070900 tx=0x7f141802a4e0 comp rx=0 tx=0).stop 2024-09-17T18:23:19.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.665+0000 7f14252ed640 1 -- 172.21.15.121:0/755987410 shutdown_connections 2024-09-17T18:23:19.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.665+0000 7f14252ed640 1 --2- 172.21.15.121:0/755987410 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14200a65f0 0x7f14201401b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:19.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.665+0000 7f14252ed640 1 --2- 172.21.15.121:0/755987410 >> v2:172.21.15.173:3300/0 conn(0x7f1420070200 0x7f14200a5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:19.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.665+0000 7f14252ed640 1 -- 172.21.15.121:0/755987410 >> 172.21.15.121:0/755987410 conn(0x7f14200aba80 msgr2=0x7f14200a7da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:23:19.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.665+0000 7f14252ed640 1 -- 172.21.15.121:0/755987410 shutdown_connections 2024-09-17T18:23:19.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.665+0000 7f14252ed640 1 -- 172.21.15.121:0/755987410 wait complete. 2024-09-17T18:23:19.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.666+0000 7f14252ed640 10 monclient: build_initial_monmap 2024-09-17T18:23:19.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.666+0000 7f14252ed640 10 monclient: monmap: 2024-09-17T18:23:19.681 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:23:19.681 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:23:19.681 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:23:19.681 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:23:19.681 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:23:19.682 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:19.682 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:23:19.682 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:23:19.682 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:23:19.682 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:19.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.666+0000 7f14252ed640 1 librados: starting msgr at 2024-09-17T18:23:19.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.666+0000 7f14252ed640 1 librados: starting objecter 2024-09-17T18:23:19.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.666+0000 7f14252ed640 1 Processor -- start 2024-09-17T18:23:19.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.666+0000 7f14252ed640 1 -- start start 2024-09-17T18:23:19.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.666+0000 7f14252ed640 1 librados: setting wanted keys 2024-09-17T18:23:19.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.666+0000 7f14252ed640 1 librados: calling monclient init 2024-09-17T18:23:19.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.666+0000 7f14252ed640 10 monclient: init 2024-09-17T18:23:19.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f14252ed640 10 monclient: _reopen_session rank -1 2024-09-17T18:23:19.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f14252ed640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T18:23:19.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f14252ed640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1420070200 0x7f14201ada10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:19.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f14252ed640 10 monclient(hunting): picked mon.noname-c con 0x7f1420070200 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:23:19.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f14252ed640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14200a61e0 0x7f14201cddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:19.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f14252ed640 10 monclient(hunting): picked mon.noname-b con 0x7f14200a61e0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:23:19.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f14252ed640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14200a65f0 0x7f14201c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:19.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f14252ed640 10 monclient(hunting): picked mon.noname-a con 0x7f14200a65f0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:23:19.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f14252ed640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:19.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f14252ed640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f14200a6a40 con 0x7f14200a65f0 2024-09-17T18:23:19.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f14252ed640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:19.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f14252ed640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f14200a31b0 con 0x7f14200a61e0 2024-09-17T18:23:19.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f14252ed640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:19.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f14252ed640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f14200a8540 con 0x7f1420070200 2024-09-17T18:23:19.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f14252ed640 10 monclient(hunting): _renew_subs 2024-09-17T18:23:19.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f141effd640 1 -- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1420070200 msgr2=0x7f14201ada10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:23:19.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f14252ed640 10 monclient(hunting): authenticate will time out at 1003.383300s 2024-09-17T18:23:19.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f141effd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1420070200 0x7f14201ada10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:23:19.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f141f7fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14200a65f0 0x7f14201c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:19.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f141f7fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14200a65f0 0x7f14201c77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35018/0 (socket says 172.21.15.121:35018) 2024-09-17T18:23:19.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f141f7fe640 1 -- 172.21.15.121:0/1997082857 learned_addr learned my addr 172.21.15.121:0/1997082857 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:23:19.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f141f7fe640 10 monclient(hunting): get_auth_request con 0x7f14200a65f0 auth_method 0 2024-09-17T18:23:19.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f141f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:19.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f141f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:19.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f141f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:19.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f141f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:19.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f141f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:19.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f141f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:19.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f1417fff640 1 --2- 172.21.15.121:0/1997082857 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14200a61e0 0x7f14201cddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:19.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f1417fff640 10 monclient(hunting): get_auth_request con 0x7f14200a61e0 auth_method 0 2024-09-17T18:23:19.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f1417fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:19.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f1417fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:19.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f1417fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:19.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f141f7fe640 10 monclient(hunting): handle_auth_done global_id 44175 payload 824 2024-09-17T18:23:19.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f141f7fe640 1 -- 172.21.15.121:0/1997082857 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1420070200 msgr2=0x7f14201ada10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:23:19.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f141f7fe640 1 --2- 172.21.15.121:0/1997082857 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1420070200 0x7f14201ada10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:19.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f141f7fe640 1 -- 172.21.15.121:0/1997082857 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14200a61e0 msgr2=0x7f14201cddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:19.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f141f7fe640 1 --2- 172.21.15.121:0/1997082857 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14200a61e0 0x7f14201cddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:19.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f141f7fe640 10 monclient: _finish_hunting 0 2024-09-17T18:23:19.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f141f7fe640 1 monclient: found mon.noname-a 2024-09-17T18:23:19.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f141f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:23:19.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f141f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:23:19.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f141f7fe640 1 -- 172.21.15.121:0/1997082857 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14201ae3a0 con 0x7f14200a65f0 2024-09-17T18:23:19.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.667+0000 7f1417fff640 1 --2- 172.21.15.121:0/1997082857 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14200a61e0 0x7f14201cddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T18:23:19.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.669+0000 7f141f7fe640 1 -- 172.21.15.121:0/1997082857 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14200a65f0 msgr2=0x7f14201c77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:19.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.669+0000 7f141f7fe640 1 --2- 172.21.15.121:0/1997082857 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14200a65f0 0x7f14201c77b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1418009d90 tx=0x7f1418004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:19.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.669+0000 7f14177fe640 10 client.?.objecter ms_handle_connect 0x7f14200a65f0 2024-09-17T18:23:19.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.669+0000 7f14177fe640 10 client.?.objecter resend_mon_ops 2024-09-17T18:23:19.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.669+0000 7f14177fe640 1 -- 172.21.15.121:0/1997082857 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1418040070 con 0x7f14200a65f0 2024-09-17T18:23:19.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.669+0000 7f14177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:19.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.670+0000 7f14177fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:23:19.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.670+0000 7f14177fe640 10 monclient: dump: 2024-09-17T18:23:19.687 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:19.688 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:19.688 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:19.688 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:19.688 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:19.688 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:19.688 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:19.688 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:19.688 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:19.688 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:19.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.670+0000 7f14177fe640 10 monclient: _finish_auth 0 2024-09-17T18:23:19.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.670+0000 7f14177fe640 10 monclient: _check_auth_tickets 2024-09-17T18:23:19.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.670+0000 7f14177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:23:19.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.670+0000 7f14177fe640 1 -- 172.21.15.121:0/1997082857 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f141803c620 con 0x7f14200a65f0 2024-09-17T18:23:19.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.670+0000 7f14252ed640 5 monclient: authenticate success, global_id 44175 2024-09-17T18:23:19.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.670+0000 7f14252ed640 10 monclient: _renew_subs 2024-09-17T18:23:19.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.670+0000 7f14252ed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:23:19.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.670+0000 7f14252ed640 1 -- 172.21.15.121:0/1997082857 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14200a2490 con 0x7f14200a65f0 2024-09-17T18:23:19.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.670+0000 7f14177fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:23:19.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.670+0000 7f14177fe640 1 -- 172.21.15.121:0/1997082857 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14180314c0 con 0x7f14200a65f0 2024-09-17T18:23:19.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.670+0000 7f14177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:19.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.670+0000 7f14177fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:23:19.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.670+0000 7f14177fe640 10 monclient: dump: 2024-09-17T18:23:19.690 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:19.690 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:19.690 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:19.690 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:19.690 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:19.690 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:19.691 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:19.691 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:19.691 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:19.691 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:19.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.670+0000 7f14252ed640 10 client.44175.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:23:19.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.670+0000 7f141dffb640 4 set_mon_vals no callback set 2024-09-17T18:23:19.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.670+0000 7f14252ed640 10 monclient: _renew_subs 2024-09-17T18:23:19.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.670+0000 7f14252ed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:23:19.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.670+0000 7f14252ed640 1 -- 172.21.15.121:0/1997082857 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14201af210 con 0x7f14200a65f0 2024-09-17T18:23:19.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.670+0000 7f14252ed640 1 librados: init done 2024-09-17T18:23:19.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.671+0000 7f14177fe640 1 -- 172.21.15.121:0/1997082857 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f1418039040 con 0x7f14200a65f0 2024-09-17T18:23:19.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.671+0000 7f14252ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:23:19.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.671+0000 7f14252ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:23:19.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.671+0000 7f14252ed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:23:19.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.671+0000 7f14252ed640 1 -- 172.21.15.121:0/1997082857 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14200a5ca0 con 0x7f14200a65f0 2024-09-17T18:23:19.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.671+0000 7f14177fe640 1 --2- 172.21.15.121:0/1997082857 >> v2:172.21.15.173:6812/2423786037 conn(0x7f1400040170 0x7f1400042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:19.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.671+0000 7f14177fe640 1 -- 172.21.15.121:0/1997082857 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(178..178 src has 1..178) ==== 5795+0+0 (secure 0 0 0) 0x7f1418074ea0 con 0x7f14200a65f0 2024-09-17T18:23:19.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.671+0000 7f14177fe640 10 client.44175.objecter ms_dispatch 0x7f1420000b90 osd_map(178..178 src has 1..178) 2024-09-17T18:23:19.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.671+0000 7f14177fe640 3 client.44175.objecter handle_osd_map got epochs [178,178] > 0 2024-09-17T18:23:19.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.671+0000 7f14177fe640 3 client.44175.objecter handle_osd_map decoding full epoch 178 2024-09-17T18:23:19.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.671+0000 7f14177fe640 20 client.44175.objecter dump_active .. 0 homeless 2024-09-17T18:23:19.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.671+0000 7f141effd640 1 --2- 172.21.15.121:0/1997082857 >> v2:172.21.15.173:6812/2423786037 conn(0x7f1400040170 0x7f1400042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:19.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.671+0000 7f141effd640 10 monclient: get_auth_request con 0x7f1400040170 auth_method 0 2024-09-17T18:23:19.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.674+0000 7f14177fe640 10 client.44175.objecter ms_handle_connect 0x7f1400040170 2024-09-17T18:23:19.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.674+0000 7f141effd640 1 --2- 172.21.15.121:0/1997082857 >> v2:172.21.15.173:6812/2423786037 conn(0x7f1400040170 0x7f1400042630 secure :-1 s=READY pgs=3941 cs=0 l=1 rev1=1 crypto rx=0x7f1410006fd0 tx=0x7f1410008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:19.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.675+0000 7f14177fe640 1 -- 172.21.15.121:0/1997082857 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f14180734a0 con 0x7f14200a65f0 2024-09-17T18:23:19.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.675+0000 7f14177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:23:19.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.675+0000 7f14177fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:23:19.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.919+0000 7f14252ed640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:23:19.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.919+0000 7f14252ed640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:23:19.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.919+0000 7f14252ed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:23:19.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.919+0000 7f14252ed640 1 -- 172.21.15.121:0/1997082857 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f142006cd40 con 0x7f14200a65f0 2024-09-17T18:23:19.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.920+0000 7f14177fe640 1 -- 172.21.15.121:0/1997082857 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f141803bcd0 con 0x7f14200a65f0 2024-09-17T18:23:19.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.920+0000 7f14177fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:23:19.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.920+0000 7f14177fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:23:19.922 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:23:19.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.921+0000 7f14252ed640 10 librados: watch_flush enter 2024-09-17T18:23:19.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.922+0000 7f14252ed640 10 librados: watch_flush exit 2024-09-17T18:23:19.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.922+0000 7f14252ed640 20 client.44175.objecter shutdown clearing up homeless session... 2024-09-17T18:23:19.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.922+0000 7f14252ed640 10 client.44175.objecter successfully canceled tick 2024-09-17T18:23:19.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.922+0000 7f14252ed640 1 -- 172.21.15.121:0/1997082857 >> v2:172.21.15.173:6812/2423786037 conn(0x7f1400040170 msgr2=0x7f1400042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:19.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.922+0000 7f14252ed640 1 --2- 172.21.15.121:0/1997082857 >> v2:172.21.15.173:6812/2423786037 conn(0x7f1400040170 0x7f1400042630 secure :-1 s=READY pgs=3941 cs=0 l=1 rev1=1 crypto rx=0x7f1410006fd0 tx=0x7f1410008040 comp rx=0 tx=0).stop 2024-09-17T18:23:19.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.922+0000 7f14252ed640 10 monclient: shutdown 2024-09-17T18:23:19.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.922+0000 7f14252ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:23:19.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.922+0000 7f14252ed640 1 -- 172.21.15.121:0/1997082857 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14200a65f0 msgr2=0x7f14201c77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:19.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.922+0000 7f14252ed640 1 --2- 172.21.15.121:0/1997082857 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14200a65f0 0x7f14201c77b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1418009d90 tx=0x7f1418004a40 comp rx=0 tx=0).stop 2024-09-17T18:23:19.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.922+0000 7f14252ed640 1 -- 172.21.15.121:0/1997082857 shutdown_connections 2024-09-17T18:23:19.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.922+0000 7f14252ed640 1 --2- 172.21.15.121:0/1997082857 >> v2:172.21.15.173:6812/2423786037 conn(0x7f1400040170 0x7f1400042630 unknown :-1 s=CLOSED pgs=3941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:19.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.922+0000 7f14252ed640 1 --2- 172.21.15.121:0/1997082857 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14200a61e0 0x7f14201cddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:19.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.922+0000 7f14252ed640 1 --2- 172.21.15.121:0/1997082857 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14200a65f0 0x7f14201c77b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:19.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.922+0000 7f14252ed640 1 --2- 172.21.15.121:0/1997082857 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1420070200 0x7f14201ada10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:19.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.922+0000 7f14252ed640 1 -- 172.21.15.121:0/1997082857 >> 172.21.15.121:0/1997082857 conn(0x7f14200aba80 msgr2=0x7f142006f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:23:19.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.922+0000 7f14252ed640 1 -- 172.21.15.121:0/1997082857 shutdown_connections 2024-09-17T18:23:19.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.922+0000 7f14252ed640 1 -- 172.21.15.121:0/1997082857 wait complete. 2024-09-17T18:23:19.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:19.922+0000 7f14252ed640 1 librados: shutdown 2024-09-17T18:23:22.935 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:23:23.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.040+0000 7fe427510640 10 monclient: get_monmap_and_config 2024-09-17T18:23:23.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.040+0000 7fe427510640 10 monclient: build_initial_monmap 2024-09-17T18:23:23.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.040+0000 7fe427510640 10 monclient: monmap: 2024-09-17T18:23:23.042 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:23:23.042 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:23.042 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:23:23.041339+0000 2024-09-17T18:23:23.043 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:23:23.041339+0000 2024-09-17T18:23:23.043 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:23:23.043 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:23.043 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:23:23.043 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:23:23.043 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:23:23.043 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:23:23.043 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:23:23.043 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:23.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.041+0000 7fe427510640 1 Processor -- start 2024-09-17T18:23:23.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.041+0000 7fe427510640 1 -- start start 2024-09-17T18:23:23.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.041+0000 7fe427510640 10 monclient: init 2024-09-17T18:23:23.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.042+0000 7fe427510640 10 monclient: _reopen_session rank -1 2024-09-17T18:23:23.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.042+0000 7fe427510640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-17T18:23:23.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.042+0000 7fe427510640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fe4201386b0 0x7fe4200707d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:23.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.042+0000 7fe427510640 10 monclient(hunting): picked mon.noname-c con 0x7fe4201386b0 addr v2:172.21.15.173:3300/0 2024-09-17T18:23:23.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.042+0000 7fe427510640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe42006ec00 addr v1:172.21.15.121:6789/0 2024-09-17T18:23:23.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.042+0000 7fe427510640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4201391a0 0x7fe42013fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:23.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.042+0000 7fe427510640 10 monclient(hunting): picked mon.noname-b con 0x7fe4201391a0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:23:23.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.042+0000 7fe427510640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4200a5570 con 0x7fe42006ec00 2024-09-17T18:23:23.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.042+0000 7fe427510640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:23.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.042+0000 7fe427510640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fe42006f830 con 0x7fe4201386b0 2024-09-17T18:23:23.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.042+0000 7fe427510640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:23.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.042+0000 7fe427510640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe42006fb10 con 0x7fe4201391a0 2024-09-17T18:23:23.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.042+0000 7fe427510640 10 monclient(hunting): _renew_subs 2024-09-17T18:23:23.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.042+0000 7fe427510640 10 monclient(hunting): authenticate will time out at 1006.758175s 2024-09-17T18:23:23.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.042+0000 7fe425aad640 1 -- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4201391a0 msgr2=0x7fe42013fbf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:23:23.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.042+0000 7fe425aad640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4201391a0 0x7fe42013fbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:23:23.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.042+0000 7fe424aab640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fe42006ec00 0x7fe420138a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:39666/0 (socket says 172.21.15.121:39666) 2024-09-17T18:23:23.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.042+0000 7fe424aab640 1 -- 172.21.15.121:0/1637198397 learned_addr learned my addr 172.21.15.121:0/1637198397 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:23:23.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.042+0000 7fe4252ac640 1 --2- 172.21.15.121:0/1637198397 >> v2:172.21.15.173:3300/0 conn(0x7fe4201386b0 0x7fe4200707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:23.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.042+0000 7fe4252ac640 10 monclient(hunting): get_auth_request con 0x7fe4201386b0 auth_method 0 2024-09-17T18:23:23.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.042+0000 7fe4252ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:23.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.042+0000 7fe4252ac640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:23.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.042+0000 7fe4252ac640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:23.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.042+0000 7fe4252ac640 1 -- 172.21.15.121:0/1637198397 >> v2:172.21.15.173:3300/0 conn(0x7fe4201386b0 msgr2=0x7fe4200707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:23:23.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.042+0000 7fe4252ac640 1 -- 172.21.15.121:0/1637198397 >> v2:172.21.15.173:3300/0 conn(0x7fe4201386b0 msgr2=0x7fe4200707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:23:23.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.042+0000 7fe4252ac640 1 --2- 172.21.15.121:0/1637198397 >> v2:172.21.15.173:3300/0 conn(0x7fe4201386b0 0x7fe4200707d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:23:23.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.042+0000 7fe4252ac640 1 --2- 172.21.15.121:0/1637198397 >> v2:172.21.15.173:3300/0 conn(0x7fe4201386b0 0x7fe4200707d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:23:23.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.044+0000 7fe424aab640 1 -- 172.21.15.121:0/1637198397 >> v1:172.21.15.121:6789/0 conn(0x7fe42006ec00 legacy=0x7fe420138a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:23.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.243+0000 7fe425aad640 1 -- 172.21.15.121:0/1637198397 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4201391a0 msgr2=0x7fe42013fbf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:23:23.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.243+0000 7fe425aad640 1 --2- 172.21.15.121:0/1637198397 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4201391a0 0x7fe42013fbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T18:23:23.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.244+0000 7fe4252ac640 1 --2- 172.21.15.121:0/1637198397 >> v2:172.21.15.173:3300/0 conn(0x7fe4201386b0 0x7fe4200707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:23.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.244+0000 7fe4252ac640 10 monclient(hunting): get_auth_request con 0x7fe4201386b0 auth_method 0 2024-09-17T18:23:23.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.244+0000 7fe4252ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:23.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.244+0000 7fe4252ac640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:23.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.244+0000 7fe4252ac640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:23:23.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.244+0000 7fe4252ac640 1 -- 172.21.15.121:0/1637198397 >> v2:172.21.15.173:3300/0 conn(0x7fe4201386b0 msgr2=0x7fe4200707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:23:23.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.244+0000 7fe4252ac640 1 -- 172.21.15.121:0/1637198397 >> v2:172.21.15.173:3300/0 conn(0x7fe4201386b0 msgr2=0x7fe4200707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:23:23.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.244+0000 7fe4252ac640 1 --2- 172.21.15.121:0/1637198397 >> v2:172.21.15.173:3300/0 conn(0x7fe4201386b0 0x7fe4200707d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:23:23.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.244+0000 7fe4252ac640 1 --2- 172.21.15.121:0/1637198397 >> v2:172.21.15.173:3300/0 conn(0x7fe4201386b0 0x7fe4200707d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T18:23:23.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.644+0000 7fe425aad640 1 -- 172.21.15.121:0/1637198397 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4201391a0 msgr2=0x7fe42013fbf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:23:23.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.644+0000 7fe425aad640 1 --2- 172.21.15.121:0/1637198397 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4201391a0 0x7fe42013fbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T18:23:23.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.646+0000 7fe4252ac640 1 --2- 172.21.15.121:0/1637198397 >> v2:172.21.15.173:3300/0 conn(0x7fe4201386b0 0x7fe4200707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:23.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.646+0000 7fe4252ac640 10 monclient(hunting): get_auth_request con 0x7fe4201386b0 auth_method 0 2024-09-17T18:23:23.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.646+0000 7fe4252ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:23.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.646+0000 7fe4252ac640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:23.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.646+0000 7fe4252ac640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:23:23.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.646+0000 7fe4252ac640 1 -- 172.21.15.121:0/1637198397 >> v2:172.21.15.173:3300/0 conn(0x7fe4201386b0 msgr2=0x7fe4200707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:23:23.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.646+0000 7fe4252ac640 1 -- 172.21.15.121:0/1637198397 >> v2:172.21.15.173:3300/0 conn(0x7fe4201386b0 msgr2=0x7fe4200707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:23:23.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.646+0000 7fe4252ac640 1 --2- 172.21.15.121:0/1637198397 >> v2:172.21.15.173:3300/0 conn(0x7fe4201386b0 0x7fe4200707d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:23:23.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:23.646+0000 7fe4252ac640 1 --2- 172.21.15.121:0/1637198397 >> v2:172.21.15.173:3300/0 conn(0x7fe4201386b0 0x7fe4200707d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T18:23:24.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:24.445+0000 7fe425aad640 1 -- 172.21.15.121:0/1637198397 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4201391a0 msgr2=0x7fe42013fbf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:23:24.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:24.445+0000 7fe425aad640 1 --2- 172.21.15.121:0/1637198397 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4201391a0 0x7fe42013fbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T18:23:24.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:24.448+0000 7fe4252ac640 1 --2- 172.21.15.121:0/1637198397 >> v2:172.21.15.173:3300/0 conn(0x7fe4201386b0 0x7fe4200707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:24.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:24.448+0000 7fe4252ac640 10 monclient(hunting): get_auth_request con 0x7fe4201386b0 auth_method 0 2024-09-17T18:23:24.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:24.448+0000 7fe4252ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:24.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:24.448+0000 7fe4252ac640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:24.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:24.448+0000 7fe4252ac640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:23:24.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:24.448+0000 7fe4252ac640 1 -- 172.21.15.121:0/1637198397 >> v2:172.21.15.173:3300/0 conn(0x7fe4201386b0 msgr2=0x7fe4200707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:23:24.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:24.448+0000 7fe4252ac640 1 -- 172.21.15.121:0/1637198397 >> v2:172.21.15.173:3300/0 conn(0x7fe4201386b0 msgr2=0x7fe4200707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:23:24.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:24.448+0000 7fe4252ac640 1 --2- 172.21.15.121:0/1637198397 >> v2:172.21.15.173:3300/0 conn(0x7fe4201386b0 0x7fe4200707d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:23:24.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:24.448+0000 7fe4252ac640 1 --2- 172.21.15.121:0/1637198397 >> v2:172.21.15.173:3300/0 conn(0x7fe4201386b0 0x7fe4200707d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T18:23:26.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.042+0000 7fe416ffd640 10 monclient(hunting): tick 2024-09-17T18:23:26.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.042+0000 7fe416ffd640 10 monclient(hunting): _check_auth_tickets 2024-09-17T18:23:26.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.042+0000 7fe416ffd640 1 monclient(hunting): continuing hunt 2024-09-17T18:23:26.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.042+0000 7fe416ffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T18:23:26.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.042+0000 7fe416ffd640 1 -- 172.21.15.121:0/1637198397 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4201391a0 msgr2=0x7fe42013fbf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:23:26.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.042+0000 7fe416ffd640 1 --2- 172.21.15.121:0/1637198397 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4201391a0 0x7fe42013fbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:26.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.042+0000 7fe416ffd640 1 -- 172.21.15.121:0/1637198397 >> v2:172.21.15.173:3300/0 conn(0x7fe4201386b0 msgr2=0x7fe4200707d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:23:26.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.042+0000 7fe416ffd640 1 --2- 172.21.15.121:0/1637198397 >> v2:172.21.15.173:3300/0 conn(0x7fe4201386b0 0x7fe4200707d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:26.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.042+0000 7fe416ffd640 1 -- 172.21.15.121:0/1637198397 >> v1:172.21.15.121:6789/0 conn(0x7fe42006ec00 legacy=0x7fe420138a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:26.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.042+0000 7fe416ffd640 10 monclient: _add_conns ranks=[3,1] 2024-09-17T18:23:26.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.042+0000 7fe416ffd640 1 --2- 172.21.15.121:0/1637198397 >> v2:172.21.15.121:3300/0 conn(0x7fe404003790 0x7fe404005bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:26.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.042+0000 7fe416ffd640 10 monclient(hunting): picked mon.noname-a con 0x7fe404003790 addr v2:172.21.15.121:3300/0 2024-09-17T18:23:26.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.042+0000 7fe416ffd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe4040062e0 addr v1:172.21.15.173:6789/0 2024-09-17T18:23:26.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.042+0000 7fe416ffd640 1 -- 172.21.15.121:0/1637198397 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe420070d10 con 0x7fe4040062e0 2024-09-17T18:23:26.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.042+0000 7fe416ffd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:26.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.042+0000 7fe416ffd640 1 -- 172.21.15.121:0/1637198397 --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fe42006f830 con 0x7fe404003790 2024-09-17T18:23:26.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.042+0000 7fe416ffd640 10 monclient(hunting): _renew_subs 2024-09-17T18:23:26.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.042+0000 7fe4252ac640 1 --2- 172.21.15.121:0/1637198397 >> v2:172.21.15.121:3300/0 conn(0x7fe404003790 0x7fe404005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:26.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.042+0000 7fe4252ac640 10 monclient(hunting): get_auth_request con 0x7fe404003790 auth_method 0 2024-09-17T18:23:26.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.042+0000 7fe4252ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:26.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.042+0000 7fe4252ac640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:26.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.042+0000 7fe4252ac640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:26.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.042+0000 7fe4252ac640 1 -- 172.21.15.121:0/1637198397 >> v2:172.21.15.121:3300/0 conn(0x7fe404003790 msgr2=0x7fe404005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:23:26.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.042+0000 7fe4252ac640 1 -- 172.21.15.121:0/1637198397 >> v2:172.21.15.121:3300/0 conn(0x7fe404003790 msgr2=0x7fe404005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:23:26.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.042+0000 7fe4252ac640 1 --2- 172.21.15.121:0/1637198397 >> v2:172.21.15.121:3300/0 conn(0x7fe404003790 0x7fe404005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:23:26.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.042+0000 7fe4252ac640 1 --2- 172.21.15.121:0/1637198397 >> v2:172.21.15.121:3300/0 conn(0x7fe404003790 0x7fe404005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:23:26.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.045+0000 7fe424aab640 1 -- 172.21.15.121:0/1637198397 >> v1:172.21.15.173:6789/0 conn(0x7fe4040062e0 legacy=0x7fe404008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:26.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.243+0000 7fe4252ac640 1 --2- 172.21.15.121:0/1637198397 >> v2:172.21.15.121:3300/0 conn(0x7fe404003790 0x7fe404005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:26.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.243+0000 7fe4252ac640 10 monclient(hunting): get_auth_request con 0x7fe404003790 auth_method 0 2024-09-17T18:23:26.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.243+0000 7fe4252ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:26.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.243+0000 7fe4252ac640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:26.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.243+0000 7fe4252ac640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:23:26.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.243+0000 7fe4252ac640 1 -- 172.21.15.121:0/1637198397 >> v2:172.21.15.121:3300/0 conn(0x7fe404003790 msgr2=0x7fe404005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:23:26.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.243+0000 7fe4252ac640 1 -- 172.21.15.121:0/1637198397 >> v2:172.21.15.121:3300/0 conn(0x7fe404003790 msgr2=0x7fe404005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:23:26.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.243+0000 7fe4252ac640 1 --2- 172.21.15.121:0/1637198397 >> v2:172.21.15.121:3300/0 conn(0x7fe404003790 0x7fe404005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:23:26.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.243+0000 7fe4252ac640 1 --2- 172.21.15.121:0/1637198397 >> v2:172.21.15.121:3300/0 conn(0x7fe404003790 0x7fe404005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T18:23:26.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.645+0000 7fe4252ac640 1 --2- 172.21.15.121:0/1637198397 >> v2:172.21.15.121:3300/0 conn(0x7fe404003790 0x7fe404005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:26.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.645+0000 7fe4252ac640 10 monclient(hunting): get_auth_request con 0x7fe404003790 auth_method 0 2024-09-17T18:23:26.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.645+0000 7fe4252ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:26.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.645+0000 7fe4252ac640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:26.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.645+0000 7fe4252ac640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:23:26.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.645+0000 7fe4252ac640 1 -- 172.21.15.121:0/1637198397 >> v2:172.21.15.121:3300/0 conn(0x7fe404003790 msgr2=0x7fe404005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:23:26.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.645+0000 7fe4252ac640 1 -- 172.21.15.121:0/1637198397 >> v2:172.21.15.121:3300/0 conn(0x7fe404003790 msgr2=0x7fe404005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:23:26.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.645+0000 7fe4252ac640 1 --2- 172.21.15.121:0/1637198397 >> v2:172.21.15.121:3300/0 conn(0x7fe404003790 0x7fe404005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:23:26.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:26.645+0000 7fe4252ac640 1 --2- 172.21.15.121:0/1637198397 >> v2:172.21.15.121:3300/0 conn(0x7fe404003790 0x7fe404005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T18:23:27.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.127+0000 7fe417fff640 1 -- 172.21.15.121:0/1637198397 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 34865834 0 0) 0x7fe420070d10 con 0x7fe4040062e0 2024-09-17T18:23:27.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.127+0000 7fe417fff640 10 monclient(hunting): _negotiate 2024-09-17T18:23:27.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.127+0000 7fe417fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:27.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.127+0000 7fe417fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:27.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.127+0000 7fe417fff640 10 monclient(hunting): my global_id is 24544 2024-09-17T18:23:27.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.127+0000 7fe417fff640 1 -- 172.21.15.121:0/1637198397 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe40c003520 con 0x7fe4040062e0 2024-09-17T18:23:27.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe417fff640 1 -- 172.21.15.121:0/1637198397 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1260972044 0 0) 0x7fe40c003520 con 0x7fe4040062e0 2024-09-17T18:23:27.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe417fff640 1 -- 172.21.15.121:0/1637198397 >> v2:172.21.15.121:3300/0 conn(0x7fe404003790 msgr2=0x7fe404005bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:23:27.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe417fff640 1 --2- 172.21.15.121:0/1637198397 >> v2:172.21.15.121:3300/0 conn(0x7fe404003790 0x7fe404005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:27.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe417fff640 10 monclient: _finish_hunting 0 2024-09-17T18:23:27.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe417fff640 1 monclient: found mon.noname-c-legacy 2024-09-17T18:23:27.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe417fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:23:27.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe417fff640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.173:6789/0 2024-09-17T18:23:27.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe417fff640 1 -- 172.21.15.121:0/1637198397 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe40400b090 con 0x7fe4040062e0 2024-09-17T18:23:27.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe417fff640 10 monclient: _finish_auth 0 2024-09-17T18:23:27.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe417fff640 10 monclient: _check_auth_tickets 2024-09-17T18:23:27.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe417fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:23:27.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe417fff640 1 -- 172.21.15.121:0/1637198397 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4002019445 0 0) 0x7fe410001a20 con 0x7fe4040062e0 2024-09-17T18:23:27.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe417fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:27.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe417fff640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-17T18:23:27.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe417fff640 10 monclient: dump: 2024-09-17T18:23:27.134 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:27.134 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:27.134 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:27.134 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:27.134 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:27.134 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:27.134 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:27.134 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:27.134 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:27.134 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:27.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe417fff640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] but i'm connected to v1:172.21.15.173:6789/0, reconnecting 2024-09-17T18:23:27.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe417fff640 10 monclient: _reopen_session rank -1 2024-09-17T18:23:27.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe417fff640 1 -- 172.21.15.121:0/1637198397 >> v1:172.21.15.173:6789/0 conn(0x7fe4040062e0 legacy=0x7fe404008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:27.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe417fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T18:23:27.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe425aad640 1 -- 172.21.15.121:0/1637198397 reap_dead start 2024-09-17T18:23:27.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe417fff640 1 --2- 172.21.15.121:0/1637198397 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe40c0039b0 0x7fe40c003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:27.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe417fff640 10 monclient(hunting): picked mon.a con 0x7fe40c0039b0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:23:27.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe417fff640 1 --2- 172.21.15.121:0/1637198397 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe40c004800 0x7fe40c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:27.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe417fff640 10 monclient(hunting): picked mon.c con 0x7fe40c004800 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:23:27.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe417fff640 1 --2- 172.21.15.121:0/1637198397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe40c0052e0 0x7fe40c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:27.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe417fff640 10 monclient(hunting): picked mon.b con 0x7fe40c0052e0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:23:27.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe417fff640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:27.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe417fff640 1 -- 172.21.15.121:0/1637198397 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe42006f830 con 0x7fe40c0039b0 2024-09-17T18:23:27.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe417fff640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:27.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe417fff640 1 -- 172.21.15.121:0/1637198397 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe40c00c190 con 0x7fe40c0052e0 2024-09-17T18:23:27.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe417fff640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:27.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe417fff640 1 -- 172.21.15.121:0/1637198397 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe40c00c390 con 0x7fe40c004800 2024-09-17T18:23:27.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe417fff640 10 monclient(hunting): _renew_subs 2024-09-17T18:23:27.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe417fff640 10 monclient(hunting): _finish_auth 0 2024-09-17T18:23:27.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe424aab640 1 -- 172.21.15.121:0/1637198397 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe40c004800 msgr2=0x7fe40c004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:23:27.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe424aab640 1 --2- 172.21.15.121:0/1637198397 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe40c004800 0x7fe40c004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:23:27.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe425aad640 1 --2- 172.21.15.121:0/1637198397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe40c0052e0 0x7fe40c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:27.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe425aad640 10 monclient(hunting): get_auth_request con 0x7fe40c0052e0 auth_method 0 2024-09-17T18:23:27.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe425aad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:27.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe425aad640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:27.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe425aad640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:23:27.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe425aad640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:27.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe425aad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:27.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe425aad640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:23:27.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe4252ac640 1 --2- 172.21.15.121:0/1637198397 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe40c0039b0 0x7fe40c003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:27.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe4252ac640 10 monclient(hunting): get_auth_request con 0x7fe40c0039b0 auth_method 0 2024-09-17T18:23:27.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe4252ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:27.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe4252ac640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:27.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe4252ac640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:23:27.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe425aad640 10 monclient(hunting): handle_auth_done global_id 24544 payload 293 2024-09-17T18:23:27.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.128+0000 7fe425aad640 1 -- 172.21.15.121:0/1637198397 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe40c004800 msgr2=0x7fe40c004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:23:27.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.129+0000 7fe425aad640 1 --2- 172.21.15.121:0/1637198397 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe40c004800 0x7fe40c004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:27.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.129+0000 7fe425aad640 1 -- 172.21.15.121:0/1637198397 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe40c0039b0 msgr2=0x7fe40c003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:23:27.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.129+0000 7fe425aad640 1 --2- 172.21.15.121:0/1637198397 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe40c0039b0 0x7fe40c003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:27.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.129+0000 7fe425aad640 10 monclient: _finish_hunting 0 2024-09-17T18:23:27.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.129+0000 7fe425aad640 1 monclient: found mon.b 2024-09-17T18:23:27.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.129+0000 7fe425aad640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:23:27.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.129+0000 7fe425aad640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:23:27.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.129+0000 7fe425aad640 1 -- 172.21.15.121:0/1637198397 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe40c003520 con 0x7fe40c0052e0 2024-09-17T18:23:27.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.129+0000 7fe425aad640 10 monclient: _finish_auth 0 2024-09-17T18:23:27.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.129+0000 7fe425aad640 10 monclient: _check_auth_tickets 2024-09-17T18:23:27.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.129+0000 7fe425aad640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:23:27.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.129+0000 7fe427510640 5 monclient: authenticate success, global_id 24544 2024-09-17T18:23:27.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.129+0000 7fe427510640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:23:27.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.132+0000 7fe425aad640 1 -- 172.21.15.121:0/1637198397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe40c0052e0 msgr2=0x7fe40c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:27.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.132+0000 7fe425aad640 1 --2- 172.21.15.121:0/1637198397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe40c0052e0 0x7fe40c00b9c0 secure :-1 s=READY pgs=7293 cs=0 l=1 rev1=1 crypto rx=0x7fe4200705a0 tx=0x7fe40c005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5e56aa4d9e5b868 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:27.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.132+0000 7fe417fff640 1 -- 172.21.15.121:0/1637198397 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe418007a50 con 0x7fe40c0052e0 2024-09-17T18:23:27.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.132+0000 7fe417fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:27.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.132+0000 7fe417fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:23:27.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.132+0000 7fe417fff640 10 monclient: dump: 2024-09-17T18:23:27.145 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:27.145 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:27.145 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:27.145 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:27.145 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:27.145 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:27.145 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:27.145 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:27.145 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:27.145 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:27.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.132+0000 7fe417fff640 1 -- 172.21.15.121:0/1637198397 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe418035070 con 0x7fe40c0052e0 2024-09-17T18:23:27.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.132+0000 7fe427510640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:23:27.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.132+0000 7fe417fff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:23:27.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.132+0000 7fe427510640 10 monclient: get_monmap_and_config success 2024-09-17T18:23:27.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.132+0000 7fe417fff640 1 -- 172.21.15.121:0/1637198397 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41802c040 con 0x7fe40c0052e0 2024-09-17T18:23:27.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.132+0000 7fe417fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:27.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.132+0000 7fe427510640 4 set_mon_vals no callback set 2024-09-17T18:23:27.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.132+0000 7fe417fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:23:27.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.132+0000 7fe417fff640 10 monclient: dump: 2024-09-17T18:23:27.146 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:27.146 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:27.146 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:27.146 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:27.146 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:27.147 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:27.147 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:27.147 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:27.147 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:27.147 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:27.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.133+0000 7fe427510640 10 monclient: shutdown 2024-09-17T18:23:27.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.133+0000 7fe427510640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:23:27.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.133+0000 7fe427510640 1 -- 172.21.15.121:0/1637198397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe40c0052e0 msgr2=0x7fe40c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:27.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.133+0000 7fe427510640 1 --2- 172.21.15.121:0/1637198397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe40c0052e0 0x7fe40c00b9c0 secure :-1 s=READY pgs=7293 cs=0 l=1 rev1=1 crypto rx=0x7fe4200705a0 tx=0x7fe40c005830 comp rx=0 tx=0).stop 2024-09-17T18:23:27.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.133+0000 7fe427510640 1 -- 172.21.15.121:0/1637198397 shutdown_connections 2024-09-17T18:23:27.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.133+0000 7fe427510640 1 --2- 172.21.15.121:0/1637198397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe40c0052e0 0x7fe40c00b9c0 unknown :-1 s=CLOSED pgs=7293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:27.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.133+0000 7fe427510640 1 --2- 172.21.15.121:0/1637198397 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe40c004800 0x7fe40c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:27.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.133+0000 7fe427510640 1 --2- 172.21.15.121:0/1637198397 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe40c0039b0 0x7fe40c003db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:27.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.133+0000 7fe427510640 1 -- 172.21.15.121:0/1637198397 >> 172.21.15.121:0/1637198397 conn(0x7fe4200a91e0 msgr2=0x7fe4200a95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:23:27.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.133+0000 7fe427510640 1 -- 172.21.15.121:0/1637198397 shutdown_connections 2024-09-17T18:23:27.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.133+0000 7fe427510640 1 -- 172.21.15.121:0/1637198397 wait complete. 2024-09-17T18:23:27.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.134+0000 7fe427510640 10 monclient: build_initial_monmap 2024-09-17T18:23:27.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.134+0000 7fe427510640 10 monclient: monmap: 2024-09-17T18:23:27.148 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:23:27.148 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:23:27.148 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:23:27.148 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:23:27.148 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:23:27.148 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:27.148 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:23:27.148 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:23:27.149 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:23:27.149 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:27.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.134+0000 7fe427510640 1 librados: starting msgr at 2024-09-17T18:23:27.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.134+0000 7fe427510640 1 librados: starting objecter 2024-09-17T18:23:27.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.134+0000 7fe427510640 1 Processor -- start 2024-09-17T18:23:27.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.134+0000 7fe427510640 1 -- start start 2024-09-17T18:23:27.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.134+0000 7fe427510640 1 librados: setting wanted keys 2024-09-17T18:23:27.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.134+0000 7fe427510640 1 librados: calling monclient init 2024-09-17T18:23:27.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.134+0000 7fe427510640 10 monclient: init 2024-09-17T18:23:27.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.135+0000 7fe427510640 10 monclient: _reopen_session rank -1 2024-09-17T18:23:27.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.135+0000 7fe427510640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T18:23:27.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.135+0000 7fe427510640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe40c0039b0 0x7fe420135920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:27.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.135+0000 7fe427510640 10 monclient(hunting): picked mon.noname-b con 0x7fe40c0039b0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:23:27.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.135+0000 7fe427510640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe40c004800 0x7fe42009ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:27.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.135+0000 7fe427510640 10 monclient(hunting): picked mon.noname-c con 0x7fe40c004800 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:23:27.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.135+0000 7fe427510640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe40c0052e0 0x7fe42009ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:27.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.135+0000 7fe427510640 10 monclient(hunting): picked mon.noname-a con 0x7fe40c0052e0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:23:27.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.135+0000 7fe427510640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:27.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.135+0000 7fe427510640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe420070b60 con 0x7fe40c0052e0 2024-09-17T18:23:27.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.135+0000 7fe427510640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:27.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.135+0000 7fe427510640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe4200709e0 con 0x7fe40c0039b0 2024-09-17T18:23:27.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.135+0000 7fe427510640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:27.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.135+0000 7fe427510640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe420138a90 con 0x7fe40c004800 2024-09-17T18:23:27.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.135+0000 7fe427510640 10 monclient(hunting): _renew_subs 2024-09-17T18:23:27.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.135+0000 7fe427510640 10 monclient(hunting): authenticate will time out at 1010.851561s 2024-09-17T18:23:27.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.135+0000 7fe424aab640 1 -- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe40c004800 msgr2=0x7fe42009ea30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:23:27.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.135+0000 7fe424aab640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe40c004800 0x7fe42009ea30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:23:27.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.135+0000 7fe4252ac640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe40c0039b0 0x7fe420135920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:27.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.135+0000 7fe4252ac640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe40c0039b0 0x7fe420135920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.121:53550/0 (socket says 172.21.15.121:53550) 2024-09-17T18:23:27.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.135+0000 7fe425aad640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe40c0052e0 0x7fe42009ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:27.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.135+0000 7fe4252ac640 1 -- 172.21.15.121:0/1088283724 learned_addr learned my addr 172.21.15.121:0/1088283724 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:23:27.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.135+0000 7fe4252ac640 10 monclient(hunting): get_auth_request con 0x7fe40c0039b0 auth_method 0 2024-09-17T18:23:27.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.135+0000 7fe4252ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:27.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.135+0000 7fe4252ac640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:27.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.135+0000 7fe4252ac640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:27.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.135+0000 7fe425aad640 10 monclient(hunting): get_auth_request con 0x7fe40c0052e0 auth_method 0 2024-09-17T18:23:27.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.135+0000 7fe425aad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:27.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.135+0000 7fe425aad640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:27.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.135+0000 7fe425aad640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:27.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.135+0000 7fe4252ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:27.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.135+0000 7fe4252ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:27.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.135+0000 7fe4252ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:27.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.135+0000 7fe425aad640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:27.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.135+0000 7fe425aad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:27.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.135+0000 7fe425aad640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:27.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.136+0000 7fe4252ac640 10 monclient(hunting): handle_auth_done global_id 24547 payload 824 2024-09-17T18:23:27.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.136+0000 7fe4252ac640 1 -- 172.21.15.121:0/1088283724 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe40c004800 msgr2=0x7fe42009ea30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:23:27.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.136+0000 7fe4252ac640 1 --2- 172.21.15.121:0/1088283724 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe40c004800 0x7fe42009ea30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:27.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.136+0000 7fe4252ac640 1 -- 172.21.15.121:0/1088283724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe40c0052e0 msgr2=0x7fe42009ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:27.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.136+0000 7fe4252ac640 1 --2- 172.21.15.121:0/1088283724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe40c0052e0 0x7fe42009ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:27.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.136+0000 7fe4252ac640 10 monclient: _finish_hunting 0 2024-09-17T18:23:27.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.136+0000 7fe4252ac640 1 monclient: found mon.noname-b 2024-09-17T18:23:27.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.136+0000 7fe4252ac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:23:27.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.136+0000 7fe4252ac640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.173:3300/0 2024-09-17T18:23:27.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.136+0000 7fe4252ac640 1 -- 172.21.15.121:0/1088283724 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe420135220 con 0x7fe40c0039b0 2024-09-17T18:23:27.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.138+0000 7fe4252ac640 1 -- 172.21.15.121:0/1088283724 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe40c0039b0 msgr2=0x7fe420135920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:27.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.138+0000 7fe4252ac640 1 --2- 172.21.15.121:0/1088283724 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe40c0039b0 0x7fe420135920 secure :-1 s=READY pgs=7294 cs=0 l=1 rev1=1 crypto rx=0x7fe41c017a60 tx=0x7fe41c017f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:27.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.138+0000 7fe415ffb640 10 client.?.objecter ms_handle_connect 0x7fe40c0039b0 2024-09-17T18:23:27.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.138+0000 7fe415ffb640 10 client.?.objecter resend_mon_ops 2024-09-17T18:23:27.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.138+0000 7fe415ffb640 1 -- 172.21.15.121:0/1088283724 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41c016b30 con 0x7fe40c0039b0 2024-09-17T18:23:27.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.138+0000 7fe415ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:27.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.138+0000 7fe415ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:23:27.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.138+0000 7fe415ffb640 10 monclient: dump: 2024-09-17T18:23:27.153 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:27.153 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:27.153 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:27.153 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:27.154 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:27.154 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:27.154 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:27.154 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:27.154 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:27.154 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:27.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.138+0000 7fe415ffb640 10 monclient: _finish_auth 0 2024-09-17T18:23:27.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.138+0000 7fe415ffb640 10 monclient: _check_auth_tickets 2024-09-17T18:23:27.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.138+0000 7fe415ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:23:27.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.138+0000 7fe427510640 5 monclient: authenticate success, global_id 24547 2024-09-17T18:23:27.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.138+0000 7fe427510640 10 monclient: _renew_subs 2024-09-17T18:23:27.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.138+0000 7fe427510640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:23:27.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.138+0000 7fe427510640 1 -- 172.21.15.121:0/1088283724 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe420135500 con 0x7fe40c0039b0 2024-09-17T18:23:27.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.138+0000 7fe415ffb640 1 -- 172.21.15.121:0/1088283724 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe41c016cd0 con 0x7fe40c0039b0 2024-09-17T18:23:27.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.138+0000 7fe415ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:23:27.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.138+0000 7fe415ffb640 1 -- 172.21.15.121:0/1088283724 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41c018450 con 0x7fe40c0039b0 2024-09-17T18:23:27.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.138+0000 7fe415ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:27.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.138+0000 7fe415ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:23:27.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.138+0000 7fe415ffb640 10 monclient: dump: 2024-09-17T18:23:27.155 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:27.155 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:27.155 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:27.155 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:27.155 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:27.155 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:27.155 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:27.155 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:27.155 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:27.155 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:27.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.139+0000 7fe427510640 10 client.24547.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:23:27.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.138+0000 7fe4177fe640 4 set_mon_vals no callback set 2024-09-17T18:23:27.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.139+0000 7fe427510640 10 monclient: _renew_subs 2024-09-17T18:23:27.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.139+0000 7fe427510640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:23:27.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.139+0000 7fe427510640 1 -- 172.21.15.121:0/1088283724 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4200a8640 con 0x7fe40c0039b0 2024-09-17T18:23:27.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.139+0000 7fe427510640 1 librados: init done 2024-09-17T18:23:27.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.140+0000 7fe427510640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:23:27.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.140+0000 7fe427510640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:23:27.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.140+0000 7fe427510640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:23:27.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.140+0000 7fe427510640 1 -- 172.21.15.121:0/1088283724 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe420138e70 con 0x7fe40c0039b0 2024-09-17T18:23:27.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.140+0000 7fe415ffb640 1 -- 172.21.15.121:0/1088283724 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fe41c0185f0 con 0x7fe40c0039b0 2024-09-17T18:23:27.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.140+0000 7fe415ffb640 1 --2- 172.21.15.121:0/1088283724 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe404040a90 0x7fe404042f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:27.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.140+0000 7fe415ffb640 1 -- 172.21.15.121:0/1088283724 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(178..178 src has 1..178) ==== 5795+0+0 (secure 0 0 0) 0x7fe41c055140 con 0x7fe40c0039b0 2024-09-17T18:23:27.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.140+0000 7fe415ffb640 10 client.24547.objecter ms_dispatch 0x7fe420000b90 osd_map(178..178 src has 1..178) 2024-09-17T18:23:27.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.140+0000 7fe415ffb640 3 client.24547.objecter handle_osd_map got epochs [178,178] > 0 2024-09-17T18:23:27.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.140+0000 7fe415ffb640 3 client.24547.objecter handle_osd_map decoding full epoch 178 2024-09-17T18:23:27.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.140+0000 7fe415ffb640 20 client.24547.objecter dump_active .. 0 homeless 2024-09-17T18:23:27.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.140+0000 7fe424aab640 1 --2- 172.21.15.121:0/1088283724 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe404040a90 0x7fe404042f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:27.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.140+0000 7fe424aab640 10 monclient: get_auth_request con 0x7fe404040a90 auth_method 0 2024-09-17T18:23:27.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.143+0000 7fe424aab640 1 --2- 172.21.15.121:0/1088283724 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe404040a90 0x7fe404042f50 secure :-1 s=READY pgs=3942 cs=0 l=1 rev1=1 crypto rx=0x7fe410001090 tx=0x7fe4100018f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:27.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.143+0000 7fe415ffb640 10 client.24547.objecter ms_handle_connect 0x7fe404040a90 2024-09-17T18:23:27.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.145+0000 7fe415ffb640 1 -- 172.21.15.121:0/1088283724 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe41c021370 con 0x7fe40c0039b0 2024-09-17T18:23:27.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.145+0000 7fe415ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:23:27.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.145+0000 7fe415ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:23:27.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.391+0000 7fe427510640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:23:27.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.391+0000 7fe427510640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:23:27.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.391+0000 7fe427510640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:23:27.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.391+0000 7fe427510640 1 -- 172.21.15.121:0/1088283724 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe42006cd40 con 0x7fe40c0039b0 2024-09-17T18:23:27.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.392+0000 7fe415ffb640 1 -- 172.21.15.121:0/1088283724 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7fe41c022e10 con 0x7fe40c0039b0 2024-09-17T18:23:27.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.392+0000 7fe415ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:23:27.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.392+0000 7fe415ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:23:27.394 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":24,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:23:27.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.394+0000 7fe427510640 10 librados: watch_flush enter 2024-09-17T18:23:27.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.394+0000 7fe427510640 10 librados: watch_flush exit 2024-09-17T18:23:27.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.394+0000 7fe427510640 20 client.24547.objecter shutdown clearing up homeless session... 2024-09-17T18:23:27.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.394+0000 7fe427510640 10 client.24547.objecter successfully canceled tick 2024-09-17T18:23:27.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.394+0000 7fe427510640 1 -- 172.21.15.121:0/1088283724 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe404040a90 msgr2=0x7fe404042f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:27.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.394+0000 7fe427510640 1 --2- 172.21.15.121:0/1088283724 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe404040a90 0x7fe404042f50 secure :-1 s=READY pgs=3942 cs=0 l=1 rev1=1 crypto rx=0x7fe410001090 tx=0x7fe4100018f0 comp rx=0 tx=0).stop 2024-09-17T18:23:27.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.394+0000 7fe427510640 10 monclient: shutdown 2024-09-17T18:23:27.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.394+0000 7fe427510640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:23:27.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.394+0000 7fe427510640 1 -- 172.21.15.121:0/1088283724 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe40c0039b0 msgr2=0x7fe420135920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:27.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.394+0000 7fe427510640 1 --2- 172.21.15.121:0/1088283724 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe40c0039b0 0x7fe420135920 secure :-1 s=READY pgs=7294 cs=0 l=1 rev1=1 crypto rx=0x7fe41c017a60 tx=0x7fe41c017f20 comp rx=0 tx=0).stop 2024-09-17T18:23:27.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.394+0000 7fe427510640 1 -- 172.21.15.121:0/1088283724 shutdown_connections 2024-09-17T18:23:27.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.394+0000 7fe427510640 1 --2- 172.21.15.121:0/1088283724 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe404040a90 0x7fe404042f50 unknown :-1 s=CLOSED pgs=3942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:27.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.394+0000 7fe427510640 1 --2- 172.21.15.121:0/1088283724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe40c0052e0 0x7fe42009ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:27.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.394+0000 7fe427510640 1 --2- 172.21.15.121:0/1088283724 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe40c004800 0x7fe42009ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:27.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.394+0000 7fe427510640 1 --2- 172.21.15.121:0/1088283724 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe40c0039b0 0x7fe420135920 unknown :-1 s=CLOSED pgs=7294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:27.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.394+0000 7fe427510640 1 -- 172.21.15.121:0/1088283724 >> 172.21.15.121:0/1088283724 conn(0x7fe4200a91e0 msgr2=0x7fe4200742a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:23:27.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.394+0000 7fe427510640 1 -- 172.21.15.121:0/1088283724 shutdown_connections 2024-09-17T18:23:27.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.394+0000 7fe427510640 1 -- 172.21.15.121:0/1088283724 wait complete. 2024-09-17T18:23:27.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.394+0000 7fe427510640 1 librados: shutdown 2024-09-17T18:23:27.407 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-17T18:23:27.407 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-17T18:23:27.407 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T18:23:27.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.502+0000 7f28036b8640 10 monclient: get_monmap_and_config 2024-09-17T18:23:27.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.502+0000 7f28036b8640 10 monclient: build_initial_monmap 2024-09-17T18:23:27.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.502+0000 7f28036b8640 10 monclient: monmap: 2024-09-17T18:23:27.504 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:23:27.505 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:27.505 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:23:27.503347+0000 2024-09-17T18:23:27.505 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:23:27.503347+0000 2024-09-17T18:23:27.505 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:23:27.505 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:27.505 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:23:27.505 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:23:27.505 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:23:27.505 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:23:27.505 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:23:27.505 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:27.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.503+0000 7f28036b8640 1 Processor -- start 2024-09-17T18:23:27.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.503+0000 7f28036b8640 1 -- start start 2024-09-17T18:23:27.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.503+0000 7f28036b8640 10 monclient: init 2024-09-17T18:23:27.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.503+0000 7f28036b8640 10 monclient: _reopen_session rank -1 2024-09-17T18:23:27.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.503+0000 7f28036b8640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-17T18:23:27.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.503+0000 7f28036b8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f27fc09da70 addr v1:172.21.15.173:6789/0 2024-09-17T18:23:27.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.503+0000 7f28036b8640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f27fc1368d0 0x7f27fc13b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:27.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.503+0000 7f28036b8640 10 monclient(hunting): picked mon.noname-c con 0x7f27fc1368d0 addr v2:172.21.15.173:3300/0 2024-09-17T18:23:27.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.503+0000 7f28036b8640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f27fc136d00 0x7f27fc13da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:27.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.503+0000 7f28036b8640 10 monclient(hunting): picked mon.noname-b con 0x7f27fc136d00 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:23:27.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.503+0000 7f28036b8640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27fc0a2d40 con 0x7f27fc09da70 2024-09-17T18:23:27.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.504+0000 7f28036b8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:27.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.504+0000 7f28036b8640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f27fc09de50 con 0x7f27fc1368d0 2024-09-17T18:23:27.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.504+0000 7f28036b8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:27.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.504+0000 7f28036b8640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f27fc0a0e20 con 0x7f27fc136d00 2024-09-17T18:23:27.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.504+0000 7f28036b8640 10 monclient(hunting): _renew_subs 2024-09-17T18:23:27.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.504+0000 7f28036b8640 10 monclient(hunting): authenticate will time out at 1011.220024s 2024-09-17T18:23:27.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.504+0000 7f2801c55640 1 -- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f27fc136d00 msgr2=0x7f27fc13da90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:23:27.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.504+0000 7f2801c55640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f27fc136d00 0x7f27fc13da90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:23:27.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.504+0000 7f2800c53640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f27fc1368d0 0x7f27fc13b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:27.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.504+0000 7f2800c53640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f27fc1368d0 0x7f27fc13b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.121:53552/0 (socket says 172.21.15.121:53552) 2024-09-17T18:23:27.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.504+0000 7f2800c53640 1 -- 172.21.15.121:0/4133639871 learned_addr learned my addr 172.21.15.121:0/4133639871 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:23:27.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.504+0000 7f2800c53640 10 monclient(hunting): get_auth_request con 0x7f27fc1368d0 auth_method 0 2024-09-17T18:23:27.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.504+0000 7f2800c53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:27.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.504+0000 7f2800c53640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:27.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.504+0000 7f2800c53640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:27.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.504+0000 7f2800c53640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:27.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.504+0000 7f2800c53640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:27.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.504+0000 7f2800c53640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:27.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.504+0000 7f2800c53640 10 monclient(hunting): handle_auth_done global_id 24550 payload 274 2024-09-17T18:23:27.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.504+0000 7f2800c53640 1 -- 172.21.15.121:0/4133639871 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f27fc136d00 msgr2=0x7f27fc13da90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:23:27.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.504+0000 7f2800c53640 1 --2- 172.21.15.121:0/4133639871 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f27fc136d00 0x7f27fc13da90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:27.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.504+0000 7f2800c53640 1 -- 172.21.15.121:0/4133639871 >> v1:172.21.15.173:6789/0 conn(0x7f27fc09da70 legacy=0x7f27fc1361c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:23:27.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.504+0000 7f2800c53640 10 monclient: _finish_hunting 0 2024-09-17T18:23:27.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.504+0000 7f2800c53640 1 monclient: found mon.noname-c 2024-09-17T18:23:27.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.504+0000 7f2800c53640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:23:27.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.504+0000 7f2800c53640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.173:3300/0 2024-09-17T18:23:27.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.504+0000 7f2800c53640 1 -- 172.21.15.121:0/4133639871 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27fc0a3020 con 0x7f27fc1368d0 2024-09-17T18:23:27.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.507+0000 7f2800c53640 1 -- 172.21.15.121:0/4133639871 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f27fc1368d0 msgr2=0x7f27fc13b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:27.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.507+0000 7f2800c53640 1 --2- 172.21.15.121:0/4133639871 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f27fc1368d0 0x7f27fc13b540 secure :-1 s=READY pgs=7296 cs=0 l=1 rev1=1 crypto rx=0x7f27ec0098a0 tx=0x7f27ec02a0e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=beba903719bce760 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:27.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.507+0000 7f27f3fff640 1 -- 172.21.15.121:0/4133639871 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27ec037070 con 0x7f27fc1368d0 2024-09-17T18:23:27.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.507+0000 7f27f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:27.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.507+0000 7f27f3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:23:27.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.507+0000 7f27f3fff640 10 monclient: dump: 2024-09-17T18:23:27.517 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:27.517 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:27.517 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:27.517 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:27.517 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:27.517 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:27.517 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:27.517 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:27.517 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:27.517 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:27.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.507+0000 7f27f3fff640 10 monclient: _finish_auth 0 2024-09-17T18:23:27.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.507+0000 7f27f3fff640 10 monclient: _check_auth_tickets 2024-09-17T18:23:27.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.507+0000 7f27f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:23:27.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.507+0000 7f27f3fff640 1 -- 172.21.15.121:0/4133639871 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27ec032640 con 0x7f27fc1368d0 2024-09-17T18:23:27.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.507+0000 7f27f3fff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:23:27.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.507+0000 7f27f3fff640 1 -- 172.21.15.121:0/4133639871 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27ec032a10 con 0x7f27fc1368d0 2024-09-17T18:23:27.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.507+0000 7f27f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:27.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.507+0000 7f27f3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:23:27.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.507+0000 7f27f3fff640 10 monclient: dump: 2024-09-17T18:23:27.518 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:27.518 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:27.518 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:27.518 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:27.518 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:27.518 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:27.518 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:27.519 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:27.519 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:27.519 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:27.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.507+0000 7f28036b8640 5 monclient: authenticate success, global_id 24550 2024-09-17T18:23:27.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.507+0000 7f28036b8640 10 monclient: get_monmap_and_config success 2024-09-17T18:23:27.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.507+0000 7f28036b8640 4 set_mon_vals no callback set 2024-09-17T18:23:27.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.508+0000 7f28036b8640 10 monclient: shutdown 2024-09-17T18:23:27.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.508+0000 7f28036b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:23:27.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.508+0000 7f28036b8640 1 -- 172.21.15.121:0/4133639871 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f27fc1368d0 msgr2=0x7f27fc13b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:27.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.508+0000 7f28036b8640 1 --2- 172.21.15.121:0/4133639871 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f27fc1368d0 0x7f27fc13b540 secure :-1 s=READY pgs=7296 cs=0 l=1 rev1=1 crypto rx=0x7f27ec0098a0 tx=0x7f27ec02a0e0 comp rx=0 tx=0).stop 2024-09-17T18:23:27.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.508+0000 7f28036b8640 1 -- 172.21.15.121:0/4133639871 shutdown_connections 2024-09-17T18:23:27.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.508+0000 7f28036b8640 1 --2- 172.21.15.121:0/4133639871 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f27fc136d00 0x7f27fc13da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:27.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.508+0000 7f28036b8640 1 --2- 172.21.15.121:0/4133639871 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f27fc1368d0 0x7f27fc13b540 unknown :-1 s=CLOSED pgs=7296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:27.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.508+0000 7f28036b8640 1 -- 172.21.15.121:0/4133639871 >> 172.21.15.121:0/4133639871 conn(0x7f27fc0a7830 msgr2=0x7f27fc0a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:23:27.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.508+0000 7f28036b8640 1 -- 172.21.15.121:0/4133639871 shutdown_connections 2024-09-17T18:23:27.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.508+0000 7f28036b8640 1 -- 172.21.15.121:0/4133639871 wait complete. 2024-09-17T18:23:27.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.508+0000 7f28036b8640 10 monclient: build_initial_monmap 2024-09-17T18:23:27.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.508+0000 7f28036b8640 10 monclient: monmap: 2024-09-17T18:23:27.520 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:23:27.520 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:23:27.520 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:23:27.520 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:23:27.520 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:23:27.520 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:27.520 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:23:27.520 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:23:27.520 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:23:27.520 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:27.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.509+0000 7f28036b8640 1 librados: starting msgr at 2024-09-17T18:23:27.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.509+0000 7f28036b8640 1 librados: starting objecter 2024-09-17T18:23:27.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.509+0000 7f28036b8640 1 Processor -- start 2024-09-17T18:23:27.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.509+0000 7f28036b8640 1 -- start start 2024-09-17T18:23:27.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.509+0000 7f28036b8640 1 librados: setting wanted keys 2024-09-17T18:23:27.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.509+0000 7f28036b8640 1 librados: calling monclient init 2024-09-17T18:23:27.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.509+0000 7f28036b8640 10 monclient: init 2024-09-17T18:23:27.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.509+0000 7f28036b8640 10 monclient: _reopen_session rank -1 2024-09-17T18:23:27.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.509+0000 7f28036b8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T18:23:27.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.509+0000 7f28036b8640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f27fc09da70 0x7f27fc13d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:27.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.509+0000 7f28036b8640 10 monclient(hunting): picked mon.noname-b con 0x7f27fc09da70 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:23:27.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.509+0000 7f28036b8640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f27fc1368d0 0x7f27fc13d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:27.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.509+0000 7f28036b8640 10 monclient(hunting): picked mon.noname-c con 0x7f27fc1368d0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:23:27.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.509+0000 7f28036b8640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27fc136d00 0x7f27fc137170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:27.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.509+0000 7f28036b8640 10 monclient(hunting): picked mon.noname-a con 0x7f27fc136d00 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:23:27.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.509+0000 7f28036b8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:27.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.509+0000 7f28036b8640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f27fc13dfd0 con 0x7f27fc136d00 2024-09-17T18:23:27.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.509+0000 7f28036b8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:27.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.509+0000 7f28036b8640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f27fc0a3930 con 0x7f27fc09da70 2024-09-17T18:23:27.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.509+0000 7f28036b8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:27.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.509+0000 7f28036b8640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f27fc13e150 con 0x7f27fc1368d0 2024-09-17T18:23:27.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.509+0000 7f28036b8640 10 monclient(hunting): _renew_subs 2024-09-17T18:23:27.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.509+0000 7f28036b8640 10 monclient(hunting): authenticate will time out at 1011.225732s 2024-09-17T18:23:27.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.509+0000 7f2800c53640 1 -- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f27fc1368d0 msgr2=0x7f27fc13d6b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:23:27.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.509+0000 7f2800c53640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f27fc1368d0 0x7f27fc13d6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:23:27.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.509+0000 7f2801c55640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27fc136d00 0x7f27fc137170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:27.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.509+0000 7f2801c55640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27fc136d00 0x7f27fc137170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35058/0 (socket says 172.21.15.121:35058) 2024-09-17T18:23:27.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.509+0000 7f2801c55640 1 -- 172.21.15.121:0/2643790216 learned_addr learned my addr 172.21.15.121:0/2643790216 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:23:27.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.509+0000 7f2801c55640 10 monclient(hunting): get_auth_request con 0x7f27fc136d00 auth_method 0 2024-09-17T18:23:27.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.509+0000 7f2801c55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:27.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.509+0000 7f2801c55640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:27.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.509+0000 7f2801c55640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:27.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.509+0000 7f2801c55640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:27.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.509+0000 7f2801c55640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:27.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.509+0000 7f2801c55640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:27.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.510+0000 7f2801454640 1 --2- 172.21.15.121:0/2643790216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f27fc09da70 0x7f27fc13d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:27.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.510+0000 7f2801454640 10 monclient(hunting): get_auth_request con 0x7f27fc09da70 auth_method 0 2024-09-17T18:23:27.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.510+0000 7f2801454640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:27.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.510+0000 7f2801454640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:27.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.510+0000 7f2801454640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:27.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.510+0000 7f2801c55640 10 monclient(hunting): handle_auth_done global_id 44181 payload 824 2024-09-17T18:23:27.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.510+0000 7f2801c55640 1 -- 172.21.15.121:0/2643790216 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f27fc1368d0 msgr2=0x7f27fc13d6b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:23:27.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.510+0000 7f2801c55640 1 --2- 172.21.15.121:0/2643790216 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f27fc1368d0 0x7f27fc13d6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:27.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.510+0000 7f2801c55640 1 -- 172.21.15.121:0/2643790216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f27fc09da70 msgr2=0x7f27fc13d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:27.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.510+0000 7f2801c55640 1 --2- 172.21.15.121:0/2643790216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f27fc09da70 0x7f27fc13d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:27.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.510+0000 7f2801c55640 10 monclient: _finish_hunting 0 2024-09-17T18:23:27.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.510+0000 7f2801c55640 1 monclient: found mon.noname-a 2024-09-17T18:23:27.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.510+0000 7f2801c55640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:23:27.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.510+0000 7f2801c55640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:23:27.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.510+0000 7f2801c55640 1 -- 172.21.15.121:0/2643790216 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27fc09e240 con 0x7f27fc136d00 2024-09-17T18:23:27.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.510+0000 7f2801454640 1 --2- 172.21.15.121:0/2643790216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f27fc09da70 0x7f27fc13d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T18:23:27.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.512+0000 7f2801c55640 1 -- 172.21.15.121:0/2643790216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27fc136d00 msgr2=0x7f27fc137170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:27.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.512+0000 7f2801c55640 1 --2- 172.21.15.121:0/2643790216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27fc136d00 0x7f27fc137170 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f27f400be60 tx=0x7f27f400bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:27.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.512+0000 7f27f1ffb640 10 client.?.objecter ms_handle_connect 0x7f27fc136d00 2024-09-17T18:23:27.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.512+0000 7f27f1ffb640 10 client.?.objecter resend_mon_ops 2024-09-17T18:23:27.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.512+0000 7f27f1ffb640 1 -- 172.21.15.121:0/2643790216 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f4016630 con 0x7f27fc136d00 2024-09-17T18:23:27.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.512+0000 7f27f1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:27.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.512+0000 7f27f1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:23:27.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.512+0000 7f27f1ffb640 10 monclient: dump: 2024-09-17T18:23:27.524 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:27.524 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:27.525 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:27.525 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:27.525 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:27.525 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:27.525 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:27.525 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:27.525 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:27.525 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:27.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.512+0000 7f27f1ffb640 10 monclient: _finish_auth 0 2024-09-17T18:23:27.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.512+0000 7f27f1ffb640 10 monclient: _check_auth_tickets 2024-09-17T18:23:27.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.512+0000 7f27f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:23:27.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.512+0000 7f27f1ffb640 1 -- 172.21.15.121:0/2643790216 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27f4010030 con 0x7f27fc136d00 2024-09-17T18:23:27.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.512+0000 7f28036b8640 5 monclient: authenticate success, global_id 44181 2024-09-17T18:23:27.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.512+0000 7f27f1ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:23:27.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.512+0000 7f28036b8640 10 monclient: _renew_subs 2024-09-17T18:23:27.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.512+0000 7f27f1ffb640 1 -- 172.21.15.121:0/2643790216 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f4016a10 con 0x7f27fc136d00 2024-09-17T18:23:27.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.512+0000 7f28036b8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:23:27.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.512+0000 7f28036b8640 1 -- 172.21.15.121:0/2643790216 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27fc13ac30 con 0x7f27fc136d00 2024-09-17T18:23:27.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.512+0000 7f27f1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:27.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.512+0000 7f27f1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:23:27.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.512+0000 7f27f1ffb640 10 monclient: dump: 2024-09-17T18:23:27.526 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:27.526 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:27.526 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:27.526 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:27.526 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:27.526 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:27.526 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:27.527 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:27.527 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:27.527 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:27.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.512+0000 7f27f37fe640 4 set_mon_vals no callback set 2024-09-17T18:23:27.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.512+0000 7f28036b8640 10 client.44181.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:23:27.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.512+0000 7f28036b8640 10 monclient: _renew_subs 2024-09-17T18:23:27.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.512+0000 7f28036b8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:23:27.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.512+0000 7f28036b8640 1 -- 172.21.15.121:0/2643790216 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27fc13a950 con 0x7f27fc136d00 2024-09-17T18:23:27.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.512+0000 7f28036b8640 1 librados: init done 2024-09-17T18:23:27.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.513+0000 7f27f1ffb640 1 -- 172.21.15.121:0/2643790216 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f27f40153e0 con 0x7f27fc136d00 2024-09-17T18:23:27.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.513+0000 7f28036b8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:23:27.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.513+0000 7f28036b8640 1 --2- 172.21.15.121:0/2643790216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27fc1cb6f0 0x7f27fc1376b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:27.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.513+0000 7f28036b8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:27.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.513+0000 7f28036b8640 1 -- 172.21.15.121:0/2643790216 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f27fc0a6a00 con 0x7f27fc1cb6f0 2024-09-17T18:23:27.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.513+0000 7f2801454640 1 --2- 172.21.15.121:0/2643790216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27fc1cb6f0 0x7f27fc1376b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:27.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.513+0000 7f2801454640 10 monclient: get_auth_request con 0x7f27fc1cb6f0 auth_method 0 2024-09-17T18:23:27.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.513+0000 7f2801454640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:27.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.513+0000 7f2801454640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:27.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.513+0000 7f2801454640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:27.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.513+0000 7f27f1ffb640 1 --2- 172.21.15.121:0/2643790216 >> v2:172.21.15.173:6812/2423786037 conn(0x7f27e0040150 0x7f27e0042610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:27.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.513+0000 7f2801454640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:27.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.513+0000 7f2801454640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:27.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.513+0000 7f2801454640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:27.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.513+0000 7f27f1ffb640 1 -- 172.21.15.121:0/2643790216 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(178..178 src has 1..178) ==== 5795+0+0 (secure 0 0 0) 0x7f27f4054450 con 0x7f27fc136d00 2024-09-17T18:23:27.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.513+0000 7f27f1ffb640 10 client.44181.objecter ms_dispatch 0x7f27fc000b90 osd_map(178..178 src has 1..178) 2024-09-17T18:23:27.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.513+0000 7f27f1ffb640 3 client.44181.objecter handle_osd_map got epochs [178,178] > 0 2024-09-17T18:23:27.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.513+0000 7f27f1ffb640 3 client.44181.objecter handle_osd_map decoding full epoch 178 2024-09-17T18:23:27.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.514+0000 7f27f1ffb640 20 client.44181.objecter dump_active .. 0 homeless 2024-09-17T18:23:27.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.514+0000 7f2801454640 10 monclient(hunting): handle_auth_done global_id 44184 payload 824 2024-09-17T18:23:27.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.514+0000 7f2801454640 1 -- 172.21.15.121:0/2643790216 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f27fc1361c0 con 0x7f27fc1cb6f0 2024-09-17T18:23:27.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.514+0000 7f2800c53640 1 --2- 172.21.15.121:0/2643790216 >> v2:172.21.15.173:6812/2423786037 conn(0x7f27e0040150 0x7f27e0042610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:27.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.514+0000 7f2800c53640 10 monclient: get_auth_request con 0x7f27e0040150 auth_method 0 2024-09-17T18:23:27.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.516+0000 7f2801454640 1 -- 172.21.15.121:0/2643790216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27fc1cb6f0 msgr2=0x7f27fc1376b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:27.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.516+0000 7f27f1ffb640 10 client.44181.objecter ms_handle_connect 0x7f27fc1cb6f0 2024-09-17T18:23:27.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.516+0000 7f27f1ffb640 10 client.44181.objecter resend_mon_ops 2024-09-17T18:23:27.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.516+0000 7f2801454640 1 --2- 172.21.15.121:0/2643790216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27fc1cb6f0 0x7f27fc1376b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f27f800e4f0 tx=0x7f27f800e9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:27.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.516+0000 7f27f1ffb640 1 -- 172.21.15.121:0/2643790216 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f8013800 con 0x7f27fc1cb6f0 2024-09-17T18:23:27.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.516+0000 7f27f1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:27.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.516+0000 7f27f1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:23:27.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.516+0000 7f27f1ffb640 10 monclient: dump: 2024-09-17T18:23:27.533 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:27.533 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:27.533 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:27.533 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:27.533 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:27.533 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:27.533 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:27.533 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:27.533 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:27.534 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:27.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.516+0000 7f27f1ffb640 1 -- 172.21.15.121:0/2643790216 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f27f8013800 con 0x7f27fc1cb6f0 2024-09-17T18:23:27.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.516+0000 7f27f1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:23:27.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.516+0000 7f27f1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:23:27.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.516+0000 7f27f1ffb640 1 -- 172.21.15.121:0/2643790216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27fc1cb6f0 msgr2=0x7f27fc1376b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:27.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.516+0000 7f27f1ffb640 1 --2- 172.21.15.121:0/2643790216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27fc1cb6f0 0x7f27fc1376b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f27f800e4f0 tx=0x7f27f800e9b0 comp rx=0 tx=0).stop 2024-09-17T18:23:27.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.516+0000 7f27f1ffb640 1 -- 172.21.15.121:0/2643790216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27fc1cb6f0 msgr2=0x7f27fc1376b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:23:27.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.516+0000 7f27f1ffb640 1 --2- 172.21.15.121:0/2643790216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27fc1cb6f0 0x7f27fc1376b0 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f27f800e4f0 tx=0x7f27f800e9b0 comp rx=0 tx=0).stop 2024-09-17T18:23:27.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.516+0000 7f27f1ffb640 10 client.44181.objecter ms_handle_connect 0x7f27e0040150 2024-09-17T18:23:27.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.516+0000 7f2800c53640 1 --2- 172.21.15.121:0/2643790216 >> v2:172.21.15.173:6812/2423786037 conn(0x7f27e0040150 0x7f27e0042610 secure :-1 s=READY pgs=3943 cs=0 l=1 rev1=1 crypto rx=0x7f27ec034000 tx=0x7f27ec02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:27.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.522+0000 7f28036b8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:23:27.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.522+0000 7f28036b8640 1 --2- 172.21.15.121:0/2643790216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27fc137bf0 0x7f27fc06f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:27.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.522+0000 7f28036b8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:27.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.522+0000 7f28036b8640 1 -- 172.21.15.121:0/2643790216 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f27fc138000 con 0x7f27fc137bf0 2024-09-17T18:23:27.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.522+0000 7f2801454640 1 --2- 172.21.15.121:0/2643790216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27fc137bf0 0x7f27fc06f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:27.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.522+0000 7f2801454640 10 monclient: get_auth_request con 0x7f27fc137bf0 auth_method 0 2024-09-17T18:23:27.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.522+0000 7f2801454640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:27.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.522+0000 7f2801454640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:27.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.522+0000 7f2801454640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:27.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.522+0000 7f2801454640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:27.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.522+0000 7f2801454640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:27.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.522+0000 7f2801454640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:27.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.523+0000 7f2801454640 10 monclient(hunting): handle_auth_done global_id 44187 payload 824 2024-09-17T18:23:27.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.523+0000 7f2801454640 1 -- 172.21.15.121:0/2643790216 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f27fc13a3f0 con 0x7f27fc137bf0 2024-09-17T18:23:27.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.525+0000 7f2801454640 1 -- 172.21.15.121:0/2643790216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27fc137bf0 msgr2=0x7f27fc06f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:27.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.525+0000 7f2801454640 1 --2- 172.21.15.121:0/2643790216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27fc137bf0 0x7f27fc06f2c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f27f8018000 tx=0x7f27f8003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:27.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.525+0000 7f27f1ffb640 10 client.44181.objecter ms_handle_connect 0x7f27fc137bf0 2024-09-17T18:23:27.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.525+0000 7f27f1ffb640 10 client.44181.objecter resend_mon_ops 2024-09-17T18:23:27.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.525+0000 7f27f1ffb640 1 -- 172.21.15.121:0/2643790216 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f8006be0 con 0x7f27fc137bf0 2024-09-17T18:23:27.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.525+0000 7f27f1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:27.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.525+0000 7f27f1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:23:27.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.525+0000 7f27f1ffb640 10 monclient: dump: 2024-09-17T18:23:27.536 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:27.536 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:27.536 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:27.536 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:27.536 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:27.536 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:27.536 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:27.536 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:27.536 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:27.537 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:27.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.525+0000 7f27f1ffb640 1 -- 172.21.15.121:0/2643790216 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4710 (secure 0 0 0) 0x7f27f8006be0 con 0x7f27fc137bf0 2024-09-17T18:23:27.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.525+0000 7f27f1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:23:27.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.525+0000 7f27f1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:23:27.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.525+0000 7f27f1ffb640 1 -- 172.21.15.121:0/2643790216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27fc137bf0 msgr2=0x7f27fc06f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:27.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.525+0000 7f27f1ffb640 1 --2- 172.21.15.121:0/2643790216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27fc137bf0 0x7f27fc06f2c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f27f8018000 tx=0x7f27f8003760 comp rx=0 tx=0).stop 2024-09-17T18:23:27.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.525+0000 7f27f1ffb640 1 -- 172.21.15.121:0/2643790216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27fc137bf0 msgr2=0x7f27fc06f2c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:23:27.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.525+0000 7f27f1ffb640 1 --2- 172.21.15.121:0/2643790216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27fc137bf0 0x7f27fc06f2c0 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f27f8018000 tx=0x7f27f8003760 comp rx=0 tx=0).stop 2024-09-17T18:23:27.537 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:23:27.537 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:23:27.537 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:23:27.538 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "leader", 2024-09-17T18:23:27.538 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 24, 2024-09-17T18:23:27.538 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 33189, 2024-09-17T18:23:27.538 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:23:27.538 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:23:27.538 INFO:teuthology.orchestra.run.smithi121.stdout: 1 2024-09-17T18:23:27.538 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:23:27.538 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 0, 2024-09-17T18:23:27.538 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:23:27.538 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:23:27.538 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:23:27.538 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:23:27.538 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:23:27.539 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:23:27.539 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:23:27.539 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:23:27.539 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:23:27.539 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:23:27.539 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:23:27.539 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:23:27.539 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:23:27.539 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:23:27.539 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:23:27.539 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:23:27.539 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:23:27.539 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:23:27.539 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:23:27.540 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:23:27.540 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:23:27.540 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:23:27.540 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:23:27.540 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:23:27.540 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:23:27.540 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:23:27.540 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:23:27.540 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:23:27.540 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:27.540 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:27.540 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:23:27.540 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:23:27.540 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:23:27.541 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:23:27.541 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:23:27.541 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:23:27.541 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:23:27.541 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:23:27.541 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:23:27.541 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:23:27.541 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:23:27.541 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:23:27.541 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:23:27.541 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:23:27.541 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:23:27.541 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:23:27.542 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:23:27.542 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:23:27.542 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:23:27.542 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:23:27.542 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:23:27.542 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:23:27.542 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:23:27.542 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:23:27.542 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:23:27.542 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:23:27.542 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:23:27.542 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:23:27.542 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:23:27.542 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:23:27.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:27.543 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:23:27.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:27.543 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:23:27.543 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:23:27.543 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:23:27.543 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:23:27.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:27.543 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:23:27.543 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:23:27.543 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:27.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:27.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:27.544 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:23:27.544 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:23:27.544 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:27.544 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:27.544 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:27.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:27.544 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:23:27.544 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:23:27.544 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:23:27.544 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:23:27.544 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:23:27.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:27.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:27.545 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:23:27.545 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:23:27.545 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:23:27.545 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:23:27.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:27.545 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:23:27.545 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:23:27.545 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:27.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:27.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:27.545 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:23:27.545 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:23:27.545 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:27.546 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:27.546 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:27.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:27.546 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:23:27.546 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:23:27.546 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:23:27.546 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:23:27.546 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:23:27.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:27.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:27.546 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:23:27.546 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:23:27.546 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:23:27.546 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:23:27.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:27.547 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:23:27.547 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:23:27.547 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:27.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:27.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:27.547 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:23:27.547 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:23:27.547 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:27.547 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:27.547 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:27.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:27.547 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:23:27.547 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:23:27.547 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:23:27.547 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:23:27.548 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:23:27.548 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:27.548 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:27.548 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:27.548 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:23:27.548 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:23:27.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:27.548 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:23:27.548 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:23:27.548 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:23:27.548 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:27.548 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:23:27.548 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:23:27.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:27.549 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:23:27.549 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:23:27.549 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:23:27.549 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:27.549 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:23:27.549 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:23:27.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:27.549 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:23:27.549 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:23:27.549 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-17T18:23:27.549 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:27.549 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:27.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:27.550 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:23:27.550 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:23:27.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.525+0000 7f28036b8640 10 librados: watch_flush enter 2024-09-17T18:23:27.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.525+0000 7f28036b8640 10 librados: watch_flush exit 2024-09-17T18:23:27.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.525+0000 7f28036b8640 20 client.44181.objecter shutdown clearing up homeless session... 2024-09-17T18:23:27.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.525+0000 7f28036b8640 10 client.44181.objecter successfully canceled tick 2024-09-17T18:23:27.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.526+0000 7f28036b8640 1 -- 172.21.15.121:0/2643790216 >> v2:172.21.15.173:6812/2423786037 conn(0x7f27e0040150 msgr2=0x7f27e0042610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:27.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.526+0000 7f28036b8640 1 --2- 172.21.15.121:0/2643790216 >> v2:172.21.15.173:6812/2423786037 conn(0x7f27e0040150 0x7f27e0042610 secure :-1 s=READY pgs=3943 cs=0 l=1 rev1=1 crypto rx=0x7f27ec034000 tx=0x7f27ec02e000 comp rx=0 tx=0).stop 2024-09-17T18:23:27.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.526+0000 7f28036b8640 10 monclient: shutdown 2024-09-17T18:23:27.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.526+0000 7f28036b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:23:27.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.526+0000 7f28036b8640 1 -- 172.21.15.121:0/2643790216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27fc136d00 msgr2=0x7f27fc137170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:27.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.526+0000 7f28036b8640 1 --2- 172.21.15.121:0/2643790216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27fc136d00 0x7f27fc137170 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f27f400be60 tx=0x7f27f400bf50 comp rx=0 tx=0).stop 2024-09-17T18:23:27.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.526+0000 7f2801c55640 1 -- 172.21.15.121:0/2643790216 reap_dead start 2024-09-17T18:23:27.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.526+0000 7f2801c55640 1 -- 172.21.15.121:0/2643790216 reap_dead start 2024-09-17T18:23:27.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.526+0000 7f28036b8640 1 -- 172.21.15.121:0/2643790216 shutdown_connections 2024-09-17T18:23:27.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.526+0000 7f28036b8640 1 -- 172.21.15.121:0/2643790216 >> 172.21.15.121:0/2643790216 conn(0x7f27fc0a7830 msgr2=0x7f27fc133db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:23:27.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.526+0000 7f28036b8640 1 -- 172.21.15.121:0/2643790216 shutdown_connections 2024-09-17T18:23:27.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.526+0000 7f28036b8640 1 -- 172.21.15.121:0/2643790216 wait complete. 2024-09-17T18:23:27.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.526+0000 7f28036b8640 1 librados: shutdown 2024-09-17T18:23:27.553 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T18:23:27.553 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-17T18:23:27.553 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T18:23:27.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.648+0000 7f3762944640 10 monclient: get_monmap_and_config 2024-09-17T18:23:27.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.648+0000 7f3762944640 10 monclient: build_initial_monmap 2024-09-17T18:23:27.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.648+0000 7f3762944640 10 monclient: monmap: 2024-09-17T18:23:27.650 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:23:27.650 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:27.650 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:23:27.649319+0000 2024-09-17T18:23:27.651 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:23:27.649319+0000 2024-09-17T18:23:27.651 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:23:27.651 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:27.651 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:23:27.651 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:23:27.651 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:23:27.651 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:23:27.651 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:23:27.651 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:27.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.649+0000 7f3762944640 1 Processor -- start 2024-09-17T18:23:27.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.649+0000 7f3762944640 1 -- start start 2024-09-17T18:23:27.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.649+0000 7f3762944640 10 monclient: init 2024-09-17T18:23:27.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.649+0000 7f3762944640 10 monclient: _reopen_session rank -1 2024-09-17T18:23:27.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.649+0000 7f3762944640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-17T18:23:27.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.649+0000 7f3762944640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f375c09e550 0x7f375c09e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:27.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.649+0000 7f3762944640 10 monclient(hunting): picked mon.noname-b con 0x7f375c09e550 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:23:27.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.649+0000 7f3762944640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f375c09ee70 addr v1:172.21.15.121:6789/0 2024-09-17T18:23:27.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.649+0000 7f3762944640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f375c09f290 0x7f375c13cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:27.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.649+0000 7f3762944640 10 monclient(hunting): picked mon.noname-a con 0x7f375c09f290 addr v2:172.21.15.121:3300/0 2024-09-17T18:23:27.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.649+0000 7f3762944640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f375c0a4460 con 0x7f375c09ee70 2024-09-17T18:23:27.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.649+0000 7f3762944640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:27.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.649+0000 7f3762944640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f375c0a4950 con 0x7f375c09f290 2024-09-17T18:23:27.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.649+0000 7f3762944640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:27.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.649+0000 7f3762944640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f375c09f840 con 0x7f375c09e550 2024-09-17T18:23:27.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.649+0000 7f3762944640 10 monclient(hunting): _renew_subs 2024-09-17T18:23:27.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.649+0000 7f3762944640 10 monclient(hunting): authenticate will time out at 1011.365964s 2024-09-17T18:23:27.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.649+0000 7f375bfff640 1 -- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f375c09e550 msgr2=0x7f375c09e930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:23:27.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.650+0000 7f375bfff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f375c09e550 0x7f375c09e930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:23:27.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.650+0000 7f3760ee1640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f375c09f290 0x7f375c13cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:27.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.650+0000 7f375b7fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f375c09ee70 0x7f375c139840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:39678/0 (socket says 172.21.15.121:39678) 2024-09-17T18:23:27.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.650+0000 7f375b7fe640 1 -- 172.21.15.121:0/1979156528 learned_addr learned my addr 172.21.15.121:0/1979156528 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:23:27.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.650+0000 7f3760ee1640 10 monclient(hunting): get_auth_request con 0x7f375c09f290 auth_method 0 2024-09-17T18:23:27.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.650+0000 7f3760ee1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:27.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.650+0000 7f3760ee1640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:27.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.650+0000 7f3760ee1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:27.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.650+0000 7f3760ee1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:27.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.650+0000 7f3760ee1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:27.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.650+0000 7f3760ee1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:27.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.650+0000 7f3760ee1640 10 monclient(hunting): handle_auth_done global_id 44190 payload 274 2024-09-17T18:23:27.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.650+0000 7f3760ee1640 1 -- 172.21.15.121:0/1979156528 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f375c09e550 msgr2=0x7f375c09e930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:23:27.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.650+0000 7f3760ee1640 1 --2- 172.21.15.121:0/1979156528 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f375c09e550 0x7f375c09e930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:27.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.650+0000 7f3760ee1640 1 -- 172.21.15.121:0/1979156528 >> v1:172.21.15.121:6789/0 conn(0x7f375c09ee70 legacy=0x7f375c139840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:23:27.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.650+0000 7f3760ee1640 10 monclient: _finish_hunting 0 2024-09-17T18:23:27.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.650+0000 7f3760ee1640 1 monclient: found mon.noname-a 2024-09-17T18:23:27.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.650+0000 7f3760ee1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:23:27.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.650+0000 7f3760ee1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:23:27.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.650+0000 7f3760ee1640 1 -- 172.21.15.121:0/1979156528 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f375c0a07e0 con 0x7f375c09f290 2024-09-17T18:23:27.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.653+0000 7f3760ee1640 1 -- 172.21.15.121:0/1979156528 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f375c09f290 msgr2=0x7f375c13cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:27.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.653+0000 7f3760ee1640 1 --2- 172.21.15.121:0/1979156528 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f375c09f290 0x7f375c13cf70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f375c09fbb0 tx=0x7f375402a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=705e3725b175dcd8 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:27.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.653+0000 7f375affd640 1 -- 172.21.15.121:0/1979156528 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3754007a30 con 0x7f375c09f290 2024-09-17T18:23:27.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.653+0000 7f375affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:27.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.653+0000 7f375affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:23:27.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.653+0000 7f375affd640 10 monclient: dump: 2024-09-17T18:23:27.662 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:27.662 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:27.663 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:27.663 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:27.663 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:27.663 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:27.663 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:27.663 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:27.663 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:27.663 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:27.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.653+0000 7f375affd640 10 monclient: _finish_auth 0 2024-09-17T18:23:27.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.653+0000 7f375affd640 10 monclient: _check_auth_tickets 2024-09-17T18:23:27.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.653+0000 7f375affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:23:27.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.653+0000 7f3762944640 5 monclient: authenticate success, global_id 44190 2024-09-17T18:23:27.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.653+0000 7f375affd640 1 -- 172.21.15.121:0/1979156528 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3754007bd0 con 0x7f375c09f290 2024-09-17T18:23:27.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.653+0000 7f3762944640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:23:27.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.653+0000 7f375affd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:23:27.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.653+0000 7f3762944640 10 monclient: get_monmap_and_config success 2024-09-17T18:23:27.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.653+0000 7f375affd640 1 -- 172.21.15.121:0/1979156528 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3754031400 con 0x7f375c09f290 2024-09-17T18:23:27.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.653+0000 7f375affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:27.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.653+0000 7f3762944640 4 set_mon_vals no callback set 2024-09-17T18:23:27.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.653+0000 7f375affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:23:27.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.653+0000 7f375affd640 10 monclient: dump: 2024-09-17T18:23:27.664 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:27.664 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:27.664 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:27.664 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:27.664 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:27.664 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:27.665 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:27.665 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:27.665 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:27.665 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:27.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.653+0000 7f3762944640 10 monclient: shutdown 2024-09-17T18:23:27.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.653+0000 7f3762944640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:23:27.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.653+0000 7f3762944640 1 -- 172.21.15.121:0/1979156528 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f375c09f290 msgr2=0x7f375c13cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:27.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.653+0000 7f3762944640 1 --2- 172.21.15.121:0/1979156528 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f375c09f290 0x7f375c13cf70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f375c09fbb0 tx=0x7f375402a480 comp rx=0 tx=0).stop 2024-09-17T18:23:27.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.653+0000 7f3762944640 1 -- 172.21.15.121:0/1979156528 shutdown_connections 2024-09-17T18:23:27.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.653+0000 7f3762944640 1 --2- 172.21.15.121:0/1979156528 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f375c09f290 0x7f375c13cf70 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:27.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.653+0000 7f3762944640 1 --2- 172.21.15.121:0/1979156528 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f375c09e550 0x7f375c09e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:27.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.653+0000 7f3762944640 1 -- 172.21.15.121:0/1979156528 >> 172.21.15.121:0/1979156528 conn(0x7f375c0a7830 msgr2=0x7f375c0a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:23:27.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.653+0000 7f3762944640 1 -- 172.21.15.121:0/1979156528 shutdown_connections 2024-09-17T18:23:27.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.653+0000 7f3762944640 1 -- 172.21.15.121:0/1979156528 wait complete. 2024-09-17T18:23:27.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.654+0000 7f3762944640 10 monclient: build_initial_monmap 2024-09-17T18:23:27.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.654+0000 7f3762944640 10 monclient: monmap: 2024-09-17T18:23:27.666 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:23:27.666 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:23:27.666 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:23:27.666 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:23:27.666 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:23:27.666 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:27.666 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:23:27.666 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:23:27.666 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:23:27.666 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:27.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.654+0000 7f3762944640 1 librados: starting msgr at 2024-09-17T18:23:27.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.654+0000 7f3762944640 1 librados: starting objecter 2024-09-17T18:23:27.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.654+0000 7f3762944640 1 Processor -- start 2024-09-17T18:23:27.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.654+0000 7f3762944640 1 -- start start 2024-09-17T18:23:27.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.654+0000 7f3762944640 1 librados: setting wanted keys 2024-09-17T18:23:27.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.654+0000 7f3762944640 1 librados: calling monclient init 2024-09-17T18:23:27.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.654+0000 7f3762944640 10 monclient: init 2024-09-17T18:23:27.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.655+0000 7f3762944640 10 monclient: _reopen_session rank -1 2024-09-17T18:23:27.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.655+0000 7f3762944640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:23:27.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.655+0000 7f3762944640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f375c09e550 0x7f375c13ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:27.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.655+0000 7f3762944640 10 monclient(hunting): picked mon.noname-a con 0x7f375c09e550 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:23:27.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.655+0000 7f3762944640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f375c09ee70 0x7f375c132860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:27.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.655+0000 7f3762944640 10 monclient(hunting): picked mon.noname-b con 0x7f375c09ee70 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:23:27.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.655+0000 7f3762944640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f375c09f290 0x7f375c13b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:27.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.655+0000 7f3762944640 10 monclient(hunting): picked mon.noname-c con 0x7f375c09f290 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:23:27.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.655+0000 7f3762944640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:27.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.655+0000 7f3762944640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f375c074960 con 0x7f375c09e550 2024-09-17T18:23:27.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.655+0000 7f3762944640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:27.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.655+0000 7f3762944640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f375c0747e0 con 0x7f375c09ee70 2024-09-17T18:23:27.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.655+0000 7f3762944640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:27.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.655+0000 7f3762944640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f375c13d4b0 con 0x7f375c09f290 2024-09-17T18:23:27.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.655+0000 7f3762944640 10 monclient(hunting): _renew_subs 2024-09-17T18:23:27.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.655+0000 7f3762944640 10 monclient(hunting): authenticate will time out at 1011.371221s 2024-09-17T18:23:27.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.655+0000 7f3760ee1640 1 -- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f375c09f290 msgr2=0x7f375c13b220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:23:27.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.655+0000 7f3760ee1640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f375c09f290 0x7f375c13b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:23:27.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.655+0000 7f375bfff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f375c09e550 0x7f375c13ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:27.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.655+0000 7f375bfff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f375c09e550 0x7f375c13ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35084/0 (socket says 172.21.15.121:35084) 2024-09-17T18:23:27.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.655+0000 7f375bfff640 1 -- 172.21.15.121:0/1459780945 learned_addr learned my addr 172.21.15.121:0/1459780945 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:23:27.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.655+0000 7f375bfff640 10 monclient(hunting): get_auth_request con 0x7f375c09e550 auth_method 0 2024-09-17T18:23:27.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.655+0000 7f375bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:27.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.655+0000 7f375bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:27.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.655+0000 7f375bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:27.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.655+0000 7f375bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:27.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.655+0000 7f375bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:27.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.655+0000 7f375bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:27.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.655+0000 7f375bfff640 10 monclient(hunting): handle_auth_done global_id 44193 payload 824 2024-09-17T18:23:27.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.655+0000 7f375bfff640 1 -- 172.21.15.121:0/1459780945 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f375c09f290 msgr2=0x7f375c13b220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:23:27.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.655+0000 7f375bfff640 1 --2- 172.21.15.121:0/1459780945 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f375c09f290 0x7f375c13b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:27.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.655+0000 7f375bfff640 1 -- 172.21.15.121:0/1459780945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f375c09ee70 msgr2=0x7f375c132860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:27.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.655+0000 7f375bfff640 1 --2- 172.21.15.121:0/1459780945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f375c09ee70 0x7f375c132860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:27.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.655+0000 7f375bfff640 10 monclient: _finish_hunting 0 2024-09-17T18:23:27.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.655+0000 7f375bfff640 1 monclient: found mon.noname-a 2024-09-17T18:23:27.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.655+0000 7f375bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:23:27.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.655+0000 7f375bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:23:27.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.655+0000 7f375bfff640 1 -- 172.21.15.121:0/1459780945 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f375c1338b0 con 0x7f375c09e550 2024-09-17T18:23:27.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.657+0000 7f375bfff640 1 -- 172.21.15.121:0/1459780945 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f375c09e550 msgr2=0x7f375c13ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:27.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.657+0000 7f375bfff640 1 --2- 172.21.15.121:0/1459780945 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f375c09e550 0x7f375c13ea20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f374c00b580 tx=0x7f374c00ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:27.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.657+0000 7f3758ff9640 10 client.?.objecter ms_handle_connect 0x7f375c09e550 2024-09-17T18:23:27.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.657+0000 7f3758ff9640 10 client.?.objecter resend_mon_ops 2024-09-17T18:23:27.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.657+0000 7f3758ff9640 1 -- 172.21.15.121:0/1459780945 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f374c0050f0 con 0x7f375c09e550 2024-09-17T18:23:27.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.657+0000 7f3758ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:27.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.657+0000 7f3758ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:23:27.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.657+0000 7f3758ff9640 10 monclient: dump: 2024-09-17T18:23:27.670 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:27.670 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:27.671 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:27.671 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:27.671 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:27.671 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:27.671 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:27.671 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:27.671 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:27.671 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:27.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.657+0000 7f3758ff9640 10 monclient: _finish_auth 0 2024-09-17T18:23:27.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.657+0000 7f3758ff9640 10 monclient: _check_auth_tickets 2024-09-17T18:23:27.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.657+0000 7f3758ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:23:27.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.657+0000 7f3758ff9640 1 -- 172.21.15.121:0/1459780945 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f374c015070 con 0x7f375c09e550 2024-09-17T18:23:27.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.657+0000 7f3762944640 5 monclient: authenticate success, global_id 44193 2024-09-17T18:23:27.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.657+0000 7f3762944640 10 monclient: _renew_subs 2024-09-17T18:23:27.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.657+0000 7f3762944640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:23:27.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.657+0000 7f3762944640 1 -- 172.21.15.121:0/1459780945 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f375c1c5610 con 0x7f375c09e550 2024-09-17T18:23:27.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.657+0000 7f3758ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T18:23:27.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.657+0000 7f3758ff9640 1 -- 172.21.15.121:0/1459780945 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f374c0054d0 con 0x7f375c09e550 2024-09-17T18:23:27.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.657+0000 7f3758ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:27.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.657+0000 7f3758ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:23:27.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.657+0000 7f3758ff9640 10 monclient: dump: 2024-09-17T18:23:27.673 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:27.673 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:27.673 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:27.673 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:27.673 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:27.673 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:27.673 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:27.674 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:27.674 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:27.674 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:27.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.657+0000 7f3762944640 10 client.44193.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:23:27.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.657+0000 7f3762944640 10 monclient: _renew_subs 2024-09-17T18:23:27.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.657+0000 7f3762944640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:23:27.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.657+0000 7f3762944640 1 -- 172.21.15.121:0/1459780945 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f375c13f980 con 0x7f375c09e550 2024-09-17T18:23:27.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.657+0000 7f3762944640 1 librados: init done 2024-09-17T18:23:27.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.657+0000 7f375a7fc640 4 set_mon_vals no callback set 2024-09-17T18:23:27.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.658+0000 7f3762944640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:23:27.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.658+0000 7f3762944640 1 --2- 172.21.15.121:0/1459780945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f375c132da0 0x7f375c13b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:27.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.658+0000 7f3762944640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:27.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.658+0000 7f3762944640 1 -- 172.21.15.121:0/1459780945 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f375c13fd80 con 0x7f375c132da0 2024-09-17T18:23:27.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.658+0000 7f3758ff9640 1 -- 172.21.15.121:0/1459780945 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f374c005670 con 0x7f375c09e550 2024-09-17T18:23:27.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.658+0000 7f375b7fe640 1 --2- 172.21.15.121:0/1459780945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f375c132da0 0x7f375c13b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:27.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.659+0000 7f375b7fe640 10 monclient: get_auth_request con 0x7f375c132da0 auth_method 0 2024-09-17T18:23:27.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.659+0000 7f375b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:27.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.659+0000 7f375b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:27.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.659+0000 7f375b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:27.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.659+0000 7f3758ff9640 1 --2- 172.21.15.121:0/1459780945 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3748040150 0x7f3748042610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:27.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.659+0000 7f375b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:27.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.659+0000 7f375b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:27.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.659+0000 7f375b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:27.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.659+0000 7f3758ff9640 1 -- 172.21.15.121:0/1459780945 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(178..178 src has 1..178) ==== 5795+0+0 (secure 0 0 0) 0x7f374c053810 con 0x7f375c09e550 2024-09-17T18:23:27.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.659+0000 7f3758ff9640 10 client.44193.objecter ms_dispatch 0x7f375c000b90 osd_map(178..178 src has 1..178) 2024-09-17T18:23:27.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.659+0000 7f3758ff9640 3 client.44193.objecter handle_osd_map got epochs [178,178] > 0 2024-09-17T18:23:27.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.659+0000 7f3758ff9640 3 client.44193.objecter handle_osd_map decoding full epoch 178 2024-09-17T18:23:27.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.659+0000 7f3758ff9640 20 client.44193.objecter dump_active .. 0 homeless 2024-09-17T18:23:27.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.659+0000 7f3760ee1640 1 --2- 172.21.15.121:0/1459780945 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3748040150 0x7f3748042610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:27.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.659+0000 7f3760ee1640 10 monclient: get_auth_request con 0x7f3748040150 auth_method 0 2024-09-17T18:23:27.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.659+0000 7f375b7fe640 10 monclient(hunting): handle_auth_done global_id 24556 payload 824 2024-09-17T18:23:27.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.659+0000 7f375b7fe640 1 -- 172.21.15.121:0/1459780945 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f375c140060 con 0x7f375c132da0 2024-09-17T18:23:27.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.661+0000 7f375b7fe640 1 -- 172.21.15.121:0/1459780945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f375c132da0 msgr2=0x7f375c13b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:27.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.661+0000 7f3758ff9640 10 client.44193.objecter ms_handle_connect 0x7f375c132da0 2024-09-17T18:23:27.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.661+0000 7f3758ff9640 10 client.44193.objecter resend_mon_ops 2024-09-17T18:23:27.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.661+0000 7f375b7fe640 1 --2- 172.21.15.121:0/1459780945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f375c132da0 0x7f375c13b760 secure :-1 s=READY pgs=7297 cs=0 l=1 rev1=1 crypto rx=0x7f375c0a5540 tx=0x7f375000c7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:27.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.661+0000 7f3758ff9640 1 -- 172.21.15.121:0/1459780945 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3750017070 con 0x7f375c132da0 2024-09-17T18:23:27.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.661+0000 7f3758ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:27.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.661+0000 7f3758ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:23:27.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.661+0000 7f3758ff9640 10 monclient: dump: 2024-09-17T18:23:27.679 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:27.679 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:27.679 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:27.680 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:27.680 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:27.680 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:27.680 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:27.680 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:27.680 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:27.680 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:27.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.661+0000 7f3758ff9640 10 client.44193.objecter ms_handle_connect 0x7f3748040150 2024-09-17T18:23:27.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.661+0000 7f3760ee1640 1 --2- 172.21.15.121:0/1459780945 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3748040150 0x7f3748042610 secure :-1 s=READY pgs=3944 cs=0 l=1 rev1=1 crypto rx=0x7f3754034000 tx=0x7f375402f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:27.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.662+0000 7f3758ff9640 1 -- 172.21.15.121:0/1459780945 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f37500070d0 con 0x7f375c132da0 2024-09-17T18:23:27.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.662+0000 7f3758ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:23:27.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.662+0000 7f3758ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:23:27.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.662+0000 7f3758ff9640 1 -- 172.21.15.121:0/1459780945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f375c132da0 msgr2=0x7f375c13b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:27.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.662+0000 7f3758ff9640 1 --2- 172.21.15.121:0/1459780945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f375c132da0 0x7f375c13b760 secure :-1 s=READY pgs=7297 cs=0 l=1 rev1=1 crypto rx=0x7f375c0a5540 tx=0x7f375000c7a0 comp rx=0 tx=0).stop 2024-09-17T18:23:27.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.662+0000 7f3758ff9640 1 -- 172.21.15.121:0/1459780945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f375c132da0 msgr2=0x7f375c13b760 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:23:27.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.662+0000 7f3758ff9640 1 --2- 172.21.15.121:0/1459780945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f375c132da0 0x7f375c13b760 secure :-1 s=CLOSED pgs=7297 cs=0 l=1 rev1=1 crypto rx=0x7f375c0a5540 tx=0x7f375000c7a0 comp rx=0 tx=0).stop 2024-09-17T18:23:27.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.668+0000 7f3762944640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:23:27.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.668+0000 7f3762944640 1 --2- 172.21.15.121:0/1459780945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f375c13bca0 0x7f375c06f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:23:27.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.668+0000 7f3762944640 10 monclient(hunting): start opening mon connection 2024-09-17T18:23:27.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.668+0000 7f3762944640 1 -- 172.21.15.121:0/1459780945 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f375c13c0b0 con 0x7f375c13bca0 2024-09-17T18:23:27.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.668+0000 7f375b7fe640 1 --2- 172.21.15.121:0/1459780945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f375c13bca0 0x7f375c06f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:23:27.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.668+0000 7f375b7fe640 10 monclient: get_auth_request con 0x7f375c13bca0 auth_method 0 2024-09-17T18:23:27.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.668+0000 7f375b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:23:27.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.668+0000 7f375b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:23:27.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.668+0000 7f375b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:23:27.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.668+0000 7f375b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:23:27.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.668+0000 7f375b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:23:27.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.668+0000 7f375b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:23:27.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.668+0000 7f375b7fe640 10 monclient(hunting): handle_auth_done global_id 24559 payload 824 2024-09-17T18:23:27.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.668+0000 7f375b7fe640 1 -- 172.21.15.121:0/1459780945 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f375c09e930 con 0x7f375c13bca0 2024-09-17T18:23:27.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.671+0000 7f375b7fe640 1 -- 172.21.15.121:0/1459780945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f375c13bca0 msgr2=0x7f375c06f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:23:27.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.671+0000 7f3758ff9640 10 client.44193.objecter ms_handle_connect 0x7f375c13bca0 2024-09-17T18:23:27.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.671+0000 7f3758ff9640 10 client.44193.objecter resend_mon_ops 2024-09-17T18:23:27.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.671+0000 7f375b7fe640 1 --2- 172.21.15.121:0/1459780945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f375c13bca0 0x7f375c06f2c0 secure :-1 s=READY pgs=7298 cs=0 l=1 rev1=1 crypto rx=0x7f375c0a5540 tx=0x7f3750004940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:23:27.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.671+0000 7f3758ff9640 1 -- 172.21.15.121:0/1459780945 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3750017040 con 0x7f375c13bca0 2024-09-17T18:23:27.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.671+0000 7f3758ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:23:27.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.671+0000 7f3758ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:23:27.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.671+0000 7f3758ff9640 10 monclient: dump: 2024-09-17T18:23:27.683 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:23:27.683 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:23:27.683 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:27.683 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:23:27.683 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:23:27.683 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:23:27.683 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:23:27.683 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:23:27.683 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:23:27.683 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:23:27.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.671+0000 7f3758ff9640 1 -- 172.21.15.121:0/1459780945 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f3750019ab0 con 0x7f375c13bca0 2024-09-17T18:23:27.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.671+0000 7f3758ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:23:27.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.671+0000 7f3758ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:23:27.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.671+0000 7f3758ff9640 1 -- 172.21.15.121:0/1459780945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f375c13bca0 msgr2=0x7f375c06f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:27.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.671+0000 7f3758ff9640 1 --2- 172.21.15.121:0/1459780945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f375c13bca0 0x7f375c06f2c0 secure :-1 s=READY pgs=7298 cs=0 l=1 rev1=1 crypto rx=0x7f375c0a5540 tx=0x7f3750004940 comp rx=0 tx=0).stop 2024-09-17T18:23:27.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.671+0000 7f3758ff9640 1 -- 172.21.15.121:0/1459780945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f375c13bca0 msgr2=0x7f375c06f2c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:23:27.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.671+0000 7f3758ff9640 1 --2- 172.21.15.121:0/1459780945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f375c13bca0 0x7f375c06f2c0 unknown :-1 s=CLOSED pgs=7298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:23:27.684 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:23:27.684 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:23:27.685 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:23:27.685 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:23:27.685 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 24, 2024-09-17T18:23:27.685 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 280616, 2024-09-17T18:23:27.685 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:23:27.685 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:23:27.685 INFO:teuthology.orchestra.run.smithi121.stdout: 1 2024-09-17T18:23:27.685 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:23:27.685 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 0, 2024-09-17T18:23:27.685 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:23:27.686 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:23:27.686 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:23:27.686 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:23:27.686 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:23:27.686 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:23:27.686 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:23:27.687 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:23:27.687 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:23:27.687 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:23:27.687 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:23:27.687 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:23:27.687 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:23:27.688 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:23:27.688 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:23:27.688 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:23:27.688 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:23:27.688 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:23:27.688 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:23:27.688 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:23:27.689 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:23:27.689 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:23:27.689 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:23:27.689 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:23:27.689 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:23:27.689 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:23:27.690 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:23:27.690 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:23:27.690 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:27.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:27.690 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:23:27.690 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:23:27.690 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:23:27.690 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:23:27.690 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:23:27.690 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:23:27.691 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:23:27.691 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:23:27.691 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:23:27.691 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:23:27.691 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:23:27.691 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:23:27.691 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:23:27.691 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:23:27.691 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:23:27.691 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:23:27.692 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:23:27.692 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:23:27.692 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:23:27.692 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:23:27.692 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:23:27.692 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:23:27.692 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:23:27.692 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:23:27.692 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:23:27.692 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:23:27.692 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:23:27.692 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:23:27.692 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:23:27.692 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:23:27.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:27.693 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:23:27.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:27.693 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:23:27.693 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:23:27.693 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:23:27.693 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:23:27.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:27.693 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:23:27.693 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:23:27.693 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:27.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:27.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:27.694 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:23:27.694 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:23:27.694 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:27.694 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:27.694 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:27.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:27.694 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:23:27.694 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:23:27.694 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:23:27.694 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:23:27.694 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:23:27.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:27.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:27.695 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:23:27.695 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:23:27.695 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:23:27.695 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:23:27.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:27.695 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:23:27.695 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:23:27.695 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:27.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:27.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:27.695 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:23:27.696 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:23:27.696 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:27.696 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:27.696 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:27.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:27.696 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:23:27.697 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:23:27.697 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:23:27.697 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:23:27.697 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:23:27.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:27.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:27.698 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:23:27.698 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:23:27.698 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:23:27.698 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:23:27.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:27.698 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:23:27.698 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:23:27.699 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:27.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:27.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:27.699 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:23:27.699 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:23:27.699 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:23:27.699 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:27.700 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:27.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:27.700 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:23:27.700 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:23:27.700 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:23:27.700 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:23:27.701 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:23:27.701 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:27.701 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:27.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:27.701 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:23:27.701 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:23:27.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:27.702 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:23:27.702 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:23:27.702 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:23:27.702 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:27.702 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:23:27.702 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:23:27.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:27.703 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:23:27.703 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:23:27.703 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 5 2024-09-17T18:23:27.703 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:27.703 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:23:27.703 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:23:27.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:27.703 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:23:27.703 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:23:27.703 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 8 2024-09-17T18:23:27.703 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:27.704 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:23:27.704 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": [ 2024-09-17T18:23:27.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:23:27.704 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:23:27.704 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:23:27.704 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:23:27.704 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:23:27.704 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:23:27.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:23:27.704 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:23:27.704 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:23:27.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.671+0000 7f3762944640 10 librados: watch_flush enter 2024-09-17T18:23:27.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.672+0000 7f3762944640 10 librados: watch_flush exit 2024-09-17T18:23:27.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.672+0000 7f3762944640 20 client.44193.objecter shutdown clearing up homeless session... 2024-09-17T18:23:27.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.672+0000 7f3762944640 10 client.44193.objecter successfully canceled tick 2024-09-17T18:23:27.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.672+0000 7f3762944640 1 -- 172.21.15.121:0/1459780945 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3748040150 msgr2=0x7f3748042610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:27.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.672+0000 7f3762944640 1 --2- 172.21.15.121:0/1459780945 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3748040150 0x7f3748042610 secure :-1 s=READY pgs=3944 cs=0 l=1 rev1=1 crypto rx=0x7f3754034000 tx=0x7f375402f000 comp rx=0 tx=0).stop 2024-09-17T18:23:27.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.672+0000 7f3762944640 10 monclient: shutdown 2024-09-17T18:23:27.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.672+0000 7f3762944640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:23:27.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.672+0000 7f3762944640 1 -- 172.21.15.121:0/1459780945 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f375c09e550 msgr2=0x7f375c13ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:23:27.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.672+0000 7f3762944640 1 --2- 172.21.15.121:0/1459780945 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f375c09e550 0x7f375c13ea20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f374c00b580 tx=0x7f374c00ba40 comp rx=0 tx=0).stop 2024-09-17T18:23:27.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.672+0000 7f3760ee1640 1 -- 172.21.15.121:0/1459780945 reap_dead start 2024-09-17T18:23:27.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.672+0000 7f3760ee1640 1 -- 172.21.15.121:0/1459780945 reap_dead start 2024-09-17T18:23:27.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.672+0000 7f3762944640 1 -- 172.21.15.121:0/1459780945 shutdown_connections 2024-09-17T18:23:27.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.672+0000 7f3762944640 1 -- 172.21.15.121:0/1459780945 >> 172.21.15.121:0/1459780945 conn(0x7f375c0a7830 msgr2=0x7f375c13db60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:23:27.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.672+0000 7f3762944640 1 -- 172.21.15.121:0/1459780945 shutdown_connections 2024-09-17T18:23:27.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.672+0000 7f3762944640 1 -- 172.21.15.121:0/1459780945 wait complete. 2024-09-17T18:23:27.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:23:27.672+0000 7f3762944640 1 librados: shutdown 2024-09-17T18:23:27.708 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T18:23:27.708 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-17T18:23:55.388 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-09-17T18:23:55.389 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-09-17T18:23:55.389 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-09-17T18:23:55.389 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: seed 35399 2024-09-17T18:23:55.389 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-09-17T18:23:55.389 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (218 ms) 2024-09-17T18:23:55.389 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-09-17T18:23:55.389 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (10686 ms) 2024-09-17T18:23:55.390 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-09-17T18:23:55.390 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (63 ms) 2024-09-17T18:23:55.390 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (10967 ms total) 2024-09-17T18:23:55.390 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: 2024-09-17T18:23:55.390 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-09-17T18:23:55.390 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-09-17T18:23:55.390 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (6992 ms) 2024-09-17T18:23:55.390 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-09-17T18:23:55.390 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (8023 ms) 2024-09-17T18:23:55.390 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-09-17T18:23:55.390 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (9989 ms) 2024-09-17T18:23:55.390 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-09-17T18:23:55.390 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: my_snaps [3] 2024-09-17T18:23:55.390 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: my_snaps [4,3] 2024-09-17T18:23:55.391 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: my_snaps [5,4,3] 2024-09-17T18:23:55.391 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: my_snaps [6,5,4,3] 2024-09-17T18:23:55.391 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: promoting some heads 2024-09-17T18:23:55.391 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: promoting from clones for snap 6 2024-09-17T18:23:55.391 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: promoting from clones for snap 5 2024-09-17T18:23:55.391 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: promoting from clones for snap 4 2024-09-17T18:23:55.391 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: promoting from clones for snap 3 2024-09-17T18:23:55.391 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: waiting for scrubs... 2024-09-17T18:23:55.391 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: done waiting 2024-09-17T18:23:55.391 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (47037 ms) 2024-09-17T18:23:55.391 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-09-17T18:23:55.391 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (10005 ms) 2024-09-17T18:23:55.391 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-09-17T18:23:55.391 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (8442 ms) 2024-09-17T18:23:55.391 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-09-17T18:23:55.392 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (7563 ms) 2024-09-17T18:23:55.392 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-09-17T18:23:55.392 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (8003 ms) 2024-09-17T18:23:55.392 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-09-17T18:23:55.392 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (10025 ms) 2024-09-17T18:23:55.392 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-09-17T18:23:55.392 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (9318 ms) 2024-09-17T18:23:55.392 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-09-17T18:23:55.392 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (9659 ms) 2024-09-17T18:23:55.392 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-09-17T18:23:55.392 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (13007 ms) 2024-09-17T18:23:55.392 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-09-17T18:23:55.392 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (8004 ms) 2024-09-17T18:23:55.393 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-09-17T18:23:55.393 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (19658 ms) 2024-09-17T18:23:55.393 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-09-17T18:23:55.393 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (27575 ms) 2024-09-17T18:23:55.393 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-09-17T18:23:55.393 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (8009 ms) 2024-09-17T18:23:55.393 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-09-17T18:23:55.393 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (8025 ms) 2024-09-17T18:23:59.368 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSe api_aio: [ OK ] LibRadosAioEC.SimpleWrite (6994 ms) 2024-09-17T18:23:59.368 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-09-17T18:23:59.368 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (7008 ms) 2024-09-17T18:23:59.368 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-09-17T18:23:59.368 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (6996 ms) 2024-09-17T18:23:59.368 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-09-17T18:23:59.368 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (7622 ms) 2024-09-17T18:23:59.368 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-09-17T18:23:59.368 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (6421 ms) 2024-09-17T18:23:59.368 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-09-17T18:23:59.369 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (7310 ms) 2024-09-17T18:23:59.369 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-09-17T18:23:59.369 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (6652 ms) 2024-09-17T18:23:59.369 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-09-17T18:23:59.369 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (7015 ms) 2024-09-17T18:23:59.369 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-09-17T18:23:59.369 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (6995 ms) 2024-09-17T18:23:59.369 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-09-17T18:23:59.369 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7001 ms) 2024-09-17T18:23:59.369 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-09-17T18:23:59.369 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (21609 ms) 2024-09-17T18:23:59.369 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-09-17T18:23:59.369 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (21699 ms) 2024-09-17T18:23:59.370 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-09-17T18:23:59.370 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (7165 ms) 2024-09-17T18:23:59.370 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-09-17T18:23:59.370 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (6765 ms) 2024-09-17T18:23:59.370 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-09-17T18:23:59.370 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7017 ms) 2024-09-17T18:23:59.370 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-09-17T18:23:59.370 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (6991 ms) 2024-09-17T18:23:59.370 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (141260 ms total) 2024-09-17T18:23:59.370 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: 2024-09-17T18:23:59.370 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [----------] Global test environment tear-down 2024-09-17T18:23:59.370 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (238738 ms total) 2024-09-17T18:23:59.370 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio: [ PASSED ] 42 tests. 2024-09-17T18:24:10.675 INFO:tasks.workunit.client.0.smithi173.stdout:tRead 2024-09-17T18:24:10.675 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: hmm, no HitSet yet 2024-09-17T18:24:10.675 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: ok, hit_set contains 267:602f83fe:::foo:head 2024-09-17T18:24:10.675 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (8991 ms) 2024-09-17T18:24:10.675 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-09-17T18:24:10.675 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: pg_num = 32 2024-09-17T18:24:10.675 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: pg 0 ls 1726597451,0 2024-09-17T18:24:10.675 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: pg 1 ls 1726597451,0 2024-09-17T18:24:10.675 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: pg 2 ls 1726597451,0 2024-09-17T18:24:10.676 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: pg 3 ls 1726597451,0 2024-09-17T18:24:10.676 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: pg 4 ls 1726597451,0 2024-09-17T18:24:10.676 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: pg 5 ls 1726597451,0 2024-09-17T18:24:10.676 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: pg 6 ls 1726597451,0 2024-09-17T18:24:10.676 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: pg 7 ls 1726597451,0 2024-09-17T18:24:10.676 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: pg 8 ls 1726597451,0 2024-09-17T18:24:10.676 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: pg 9 ls 1726597451,0 2024-09-17T18:24:10.676 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: pg 10 ls 1726597451,0 2024-09-17T18:24:10.676 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: pg 11 ls 1726597451,0 2024-09-17T18:24:10.676 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: pg 12 ls 1726597451,0 2024-09-17T18:24:10.676 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: pg 13 ls 1726597451,0 2024-09-17T18:24:10.676 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: pg 14 ls 1726597451,0 2024-09-17T18:24:10.676 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: pg 15 ls 1726597451,0 2024-09-17T18:24:10.676 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: pg 16 ls 1726597451,0 2024-09-17T18:24:10.676 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: pg 17 ls 1726597451,0 2024-09-17T18:24:10.677 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: pg 18 ls 1726597451,0 2024-09-17T18:24:10.677 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: pg 19 ls 1726597451,0 2024-09-17T18:24:10.677 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: pg 20 ls 1726597451,0 2024-09-17T18:24:10.677 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: pg 21 ls 1726597451,0 2024-09-17T18:24:10.677 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: pg 22 ls 1726597451,0 2024-09-17T18:24:10.677 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: pg 23 ls 1726597451,0 2024-09-17T18:24:10.677 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: pg 24 ls 1726597451,0 2024-09-17T18:24:10.677 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: pg 25 ls 1726597451,0 2024-09-17T18:24:10.677 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: pg 26 ls 1726597451,0 2024-09-17T18:24:10.677 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: pg 27 ls 1726597451,0 2024-09-17T18:24:10.677 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: pg 28 ls 1726597451,0 2024-09-17T18:24:10.677 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: pg 29 ls 1726597451,0 2024-09-17T18:24:10.677 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: pg 30 ls 1726597451,0 2024-09-17T18:24:10.677 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: pg 31 ls 1726597451,0 2024-09-17T18:24:10.677 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: pg_num = 32 2024-09-17T18:24:10.678 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:6cac518f:::0:head 2024-09-17T18:24:10.678 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:02547ec2:::1:head 2024-09-17T18:24:10.678 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:f905c69b:::2:head 2024-09-17T18:24:10.678 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:cfc208b3:::3:head 2024-09-17T18:24:10.678 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:d83876eb:::4:head 2024-09-17T18:24:10.678 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:b29083e3:::5:head 2024-09-17T18:24:10.678 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:c4fdafeb:::6:head 2024-09-17T18:24:10.678 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:5c6b0b28:::7:head 2024-09-17T18:24:10.678 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:bd63b0f1:::8:head 2024-09-17T18:24:10.678 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:e960b815:::9:head 2024-09-17T18:24:10.678 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:52ea6a34:::10:head 2024-09-17T18:24:10.678 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:89d3ae78:::11:head 2024-09-17T18:24:10.678 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:de5d7c5f:::12:head 2024-09-17T18:24:10.678 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:566253c9:::13:head 2024-09-17T18:24:10.679 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:62a1935d:::14:head 2024-09-17T18:24:10.679 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:863748b0:::15:head 2024-09-17T18:24:10.679 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:3958e169:::16:head 2024-09-17T18:24:10.679 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:4d4dabf9:::17:head 2024-09-17T18:24:10.679 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:8391935d:::18:head 2024-09-17T18:24:10.679 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:28883081:::19:head 2024-09-17T18:24:10.679 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:69259c59:::20:head 2024-09-17T18:24:10.679 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:4bdb80b7:::21:head 2024-09-17T18:24:10.679 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:a11c5d71:::22:head 2024-09-17T18:24:10.679 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:271af37b:::23:head 2024-09-17T18:24:10.679 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:95b121be:::24:head 2024-09-17T18:24:10.679 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:58d1031b:::25:head 2024-09-17T18:24:10.679 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:0a050783:::26:head 2024-09-17T18:24:10.679 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:c709704c:::27:head 2024-09-17T18:24:10.679 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:cbe56eaf:::28:head 2024-09-17T18:24:10.680 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:86b4b162:::29:head 2024-09-17T18:24:10.680 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:70d89383:::30:head 2024-09-17T18:24:10.680 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:dd450c7c:::31:head 2024-09-17T18:24:10.680 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:6d5729b1:::32:head 2024-09-17T18:24:10.680 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:c388f3fb:::33:head 2024-09-17T18:24:10.680 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:56cfea31:::34:head 2024-09-17T18:24:10.680 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:9dbc1bf7:::35:head 2024-09-17T18:24:10.680 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:40b74ccd:::36:head 2024-09-17T18:24:10.680 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:4d5aaf42:::37:head 2024-09-17T18:24:10.680 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:920f362c:::38:head 2024-09-17T18:24:10.680 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:6cc53222:::39:head 2024-09-17T18:24:10.680 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:9cad833f:::40:head 2024-09-17T18:24:10.681 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:1ea84d41:::41:head 2024-09-17T18:24:10.681 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:c4480ef6:::42:head 2024-09-17T18:24:10.681 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:a694361e:::43:head 2024-09-17T18:24:10.681 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:d1bd33e9:::44:head 2024-09-17T18:24:10.681 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:ddc2cd5d:::45:head 2024-09-17T18:24:10.681 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:2b782207:::46:head 2024-09-17T18:24:10.681 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:7b187fca:::47:head 2024-09-17T18:24:10.681 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:90ecdf6f:::48:head 2024-09-17T18:24:10.681 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:a5ed95fe:::49:head 2024-09-17T18:24:10.681 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:ea0eaa55:::50:head 2024-09-17T18:24:10.681 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:f33ef17b:::51:head 2024-09-17T18:24:10.681 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:a0d1b2f6:::52:head 2024-09-17T18:24:10.681 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:60c5229e:::53:head 2024-09-17T18:24:10.681 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:edcbc575:::54:head 2024-09-17T18:24:10.681 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:102cf253:::55:head 2024-09-17T18:24:10.682 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:efb7fb0b:::56:head 2024-09-17T18:24:10.682 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:50d0a326:::57:head 2024-09-17T18:24:10.682 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:d4dc5daf:::58:head 2024-09-17T18:24:10.682 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:3a130462:::59:head 2024-09-17T18:24:10.682 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:ec87ed71:::60:head 2024-09-17T18:24:10.682 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:d5bc9454:::61:head 2024-09-17T18:24:10.682 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:3ddfe313:::62:head 2024-09-17T18:24:10.682 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:7c2816b9:::63:head 2024-09-17T18:24:10.682 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:47e00e4d:::64:head 2024-09-17T18:24:10.682 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:c6410c18:::65:head 2024-09-17T18:24:10.682 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:b48ed237:::66:head 2024-09-17T18:24:10.682 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:cd63ad31:::67:head 2024-09-17T18:24:10.682 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:b179e92b:::68:head 2024-09-17T18:24:10.682 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:0d9f741a:::69:head 2024-09-17T18:24:10.682 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:6d3352ae:::70:head 2024-09-17T18:24:10.683 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:c6d5c19e:::71:head 2024-09-17T18:24:10.683 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:bc4729c3:::72:head 2024-09-17T18:24:10.683 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:77e930b9:::73:head 2024-09-17T18:24:10.683 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:0abeecfd:::74:head 2024-09-17T18:24:10.683 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:b7c37e15:::75:head 2024-09-17T18:24:10.683 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:b6378398:::76:head 2024-09-17T18:24:10.683 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:02bd68de:::77:head 2024-09-17T18:24:10.683 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:cc795d2d:::78:head 2024-09-17T18:24:10.683 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:630d4fea:::79:head 2024-09-17T18:24:10.683 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:e0d29ef5:::80:head 2024-09-17T18:24:10.683 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:fd6f13d2:::81:head 2024-09-17T18:24:10.683 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:606461d5:::82:head 2024-09-17T18:24:10.683 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:eadbdc43:::83:head 2024-09-17T18:24:10.683 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:8761d0bb:::84:head 2024-09-17T18:24:10.683 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:9ef0186f:::85:head 2024-09-17T18:24:10.683 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:e0d41294:::86:head 2024-09-17T18:24:10.684 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:961de695:::87:head 2024-09-17T18:24:10.684 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:1423148f:::88:head 2024-09-17T18:24:10.684 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:633a8fa2:::89:head 2024-09-17T18:24:10.684 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:a8653809:::90:head 2024-09-17T18:24:10.684 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:3dac8b33:::91:head 2024-09-17T18:24:10.684 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:35aad435:::92:head 2024-09-17T18:24:10.684 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:f6dcc343:::93:head 2024-09-17T18:24:10.684 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:dbbdad87:::94:head 2024-09-17T18:24:10.684 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:1cb48ce0:::95:head 2024-09-17T18:24:10.684 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:03cd461c:::96:head 2024-09-17T18:24:10.684 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:17a4ea99:::97:head 2024-09-17T18:24:10.685 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:9993c9a7:::98:head 2024-09-17T18:24:10.685 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:6394211c:::99:head 2024-09-17T18:24:10.685 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:94c7ae57:::100:head 2024-09-17T18:24:10.685 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:6fdee5bb:::101:head 2024-09-17T18:24:10.685 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:9a477fd1:::102:head 2024-09-17T18:24:10.685 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:eb850916:::103:head 2024-09-17T18:24:10.685 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:affc56b9:::104:head 2024-09-17T18:24:10.685 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:b42dc814:::105:head 2024-09-17T18:24:10.685 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:f319f8f0:::106:head 2024-09-17T18:24:10.685 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:9a40b9de:::107:head 2024-09-17T18:24:10.685 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:8b524f28:::108:head 2024-09-17T18:24:10.685 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:e3de589f:::109:head 2024-09-17T18:24:10.685 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:90f90a5b:::110:head 2024-09-17T18:24:10.686 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:a7b4f1d7:::111:head 2024-09-17T18:24:10.686 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:af51766e:::112:head 2024-09-17T18:24:10.686 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:b6f90bd1:::113:head 2024-09-17T18:24:10.686 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:e0261208:::114:head 2024-09-17T18:24:10.686 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:c9569ef7:::115:head 2024-09-17T18:24:10.686 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:61bebe50:::116:head 2024-09-17T18:24:10.686 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:fe93412b:::117:head 2024-09-17T18:24:10.686 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:d3d38bee:::118:head 2024-09-17T18:24:10.686 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:3100ba0c:::119:head 2024-09-17T18:24:10.686 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:d0560ada:::120:head 2024-09-17T18:24:10.686 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:f0ea8b35:::121:head 2024-09-17T18:24:10.686 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:766f231a:::122:head 2024-09-17T18:24:10.686 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:a07a2582:::123:head 2024-09-17T18:24:10.686 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:bd7c6b3a:::124:head 2024-09-17T18:24:10.686 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:fb2ddaff:::125:head 2024-09-17T18:24:10.687 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:4408e1fe:::126:head 2024-09-17T18:24:10.687 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:ee1df7a7:::127:head 2024-09-17T18:24:10.687 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:c3002909:::128:head 2024-09-17T18:24:10.687 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:4f48ffa9:::129:head 2024-09-17T18:24:10.687 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:edf38733:::130:head 2024-09-17T18:24:10.687 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:c08425c0:::131:head 2024-09-17T18:24:10.687 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:5f902d98:::132:head 2024-09-17T18:24:10.687 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:41ea2c93:::133:head 2024-09-17T18:24:10.687 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:813cee13:::134:head 2024-09-17T18:24:10.687 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:0131818d:::135:head 2024-09-17T18:24:10.687 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:26ba5a85:::136:head 2024-09-17T18:24:10.687 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:381b8a5a:::137:head 2024-09-17T18:24:10.687 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:28797e47:::138:head 2024-09-17T18:24:10.687 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:bfca7f22:::139:head 2024-09-17T18:24:10.687 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:36807075:::140:head 2024-09-17T18:24:10.688 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:80b03975:::141:head 2024-09-17T18:24:10.688 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:5c15709b:::142:head 2024-09-17T18:24:10.688 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:f39ea15e:::143:head 2024-09-17T18:24:10.688 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:ea992956:::144:head 2024-09-17T18:24:10.688 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:48887b1c:::145:head 2024-09-17T18:24:10.688 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:9f24a9dd:::146:head 2024-09-17T18:24:10.688 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:987f100b:::147:head 2024-09-17T18:24:10.688 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:d2dd3581:::148:head 2024-09-17T18:24:10.688 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:7fed1808:::149:head 2024-09-17T18:24:10.688 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:c80b70e9:::150:head 2024-09-17T18:24:10.688 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:85ed90f9:::151:head 2024-09-17T18:24:10.688 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:36428b24:::152:head 2024-09-17T18:24:10.688 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:d044c34a:::153:head 2024-09-17T18:24:10.688 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:7c18bf58:::154:head 2024-09-17T18:24:10.689 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:d1c21232:::155:head 2024-09-17T18:24:10.689 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:a7a3c575:::156:head 2024-09-17T18:24:10.689 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:87da0633:::157:head 2024-09-17T18:24:10.689 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:d5ac3822:::158:head 2024-09-17T18:24:10.689 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:3f20522d:::159:head 2024-09-17T18:24:10.689 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:6ca26563:::160:head 2024-09-17T18:24:10.689 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:532ce135:::161:head 2024-09-17T18:24:10.689 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:c78863e6:::162:head 2024-09-17T18:24:49.953 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:7462ddf6:::o.RoundTripAppendPP (2992 ms) 2024-09-17T18:24:49.953 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-09-17T18:24:49.954 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3004 ms) 2024-09-17T18:24:49.954 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-09-17T18:24:49.954 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3008 ms) 2024-09-17T18:24:49.954 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-09-17T18:24:49.954 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3615 ms) 2024-09-17T18:24:49.954 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-09-17T18:24:49.954 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: setting pool EIO 2024-09-17T18:24:49.954 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: max_success 99, min_failed 100 2024-09-17T18:24:49.954 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (3391 ms) 2024-09-17T18:24:49.954 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-09-17T18:24:49.954 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (3026 ms) 2024-09-17T18:24:49.954 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (132515 ms total) 2024-09-17T18:24:49.955 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: 2024-09-17T18:24:49.955 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-09-17T18:24:49.955 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-09-17T18:24:49.955 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3008 ms) 2024-09-17T18:24:49.955 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-09-17T18:24:49.955 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9322 ms) 2024-09-17T18:24:49.955 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-09-17T18:24:49.955 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (14639 ms) 2024-09-17T18:24:49.955 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-09-17T18:24:49.955 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3010 ms) 2024-09-17T18:24:49.955 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (29979 ms total) 2024-09-17T18:24:49.955 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: 2024-09-17T18:24:49.956 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-09-17T18:24:49.956 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-09-17T18:24:49.956 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3008 ms) 2024-09-17T18:24:49.956 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3008 ms total) 2024-09-17T18:24:49.956 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: 2024-09-17T18:24:49.956 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-09-17T18:24:49.956 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-09-17T18:24:49.956 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (43017 ms) 2024-09-17T18:24:49.956 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-09-17T18:24:49.956 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (7437 ms) 2024-09-17T18:24:49.956 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-09-17T18:24:49.956 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (6790 ms) 2024-09-17T18:24:49.957 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-09-17T18:24:49.957 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (6995 ms) 2024-09-17T18:24:49.957 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-09-17T18:24:49.957 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3005 ms) 2024-09-17T18:24:49.957 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-09-17T18:24:49.957 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (6994 ms) 2024-09-17T18:24:49.957 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-09-17T18:24:49.957 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (7030 ms) 2024-09-17T18:24:49.957 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-09-17T18:24:49.957 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (6975 ms) 2024-09-17T18:24:49.957 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-09-17T18:24:49.958 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (6993 ms) 2024-09-17T18:24:49.958 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-09-17T18:24:49.958 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (7041 ms) 2024-09-17T18:24:49.958 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-09-17T18:24:49.958 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7166 ms) 2024-09-17T18:24:49.958 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-09-17T18:24:49.959 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (6805 ms) 2024-09-17T18:24:49.959 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-09-17T18:24:49.959 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (7575 ms) 2024-09-17T18:24:57.709 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-17T18:24:57.709 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-17T18:24:57.710 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-17T18:24:57.713 INFO:tasks.ceph.mon.c:Started 2024-09-17T18:24:57.713 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T18:24:57.713 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:24:57.814 INFO:tasks.ceph.mon.c.smithi121.stderr:2024-09-17T18:24:57.812+0000 7fd071411c80 -1 mon.c@-1(???).log v0 log_external_backlog rewinding external_log_to from 189 -> 0 (sync_force? mon rebuild?) 2024-09-17T18:24:57.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.818+0000 7f75f4485640 10 monclient: get_monmap_and_config 2024-09-17T18:24:57.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.818+0000 7f75f4485640 10 monclient: build_initial_monmap 2024-09-17T18:24:57.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.818+0000 7f75f4485640 10 monclient: monmap: 2024-09-17T18:24:57.821 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:24:57.821 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:24:57.821 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:24:57.820426+0000 2024-09-17T18:24:57.821 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:24:57.820426+0000 2024-09-17T18:24:57.822 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:24:57.822 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:24:57.822 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:24:57.822 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:24:57.822 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:24:57.822 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:24:57.822 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:24:57.822 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:24:57.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.819+0000 7f75f4485640 1 Processor -- start 2024-09-17T18:24:57.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.819+0000 7f75f4485640 1 -- start start 2024-09-17T18:24:57.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.819+0000 7f75f4485640 10 monclient: init 2024-09-17T18:24:57.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.820+0000 7f75f4485640 10 monclient: _reopen_session rank -1 2024-09-17T18:24:57.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.820+0000 7f75f4485640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-17T18:24:57.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.820+0000 7f75f4485640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f75ec0a5c00 0x7f75ec09d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:24:57.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.820+0000 7f75f4485640 10 monclient(hunting): picked mon.noname-c con 0x7f75ec0a5c00 addr v2:172.21.15.173:3300/0 2024-09-17T18:24:57.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.820+0000 7f75f4485640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f75ec0a5fe0 addr v1:172.21.15.121:6789/0 2024-09-17T18:24:57.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.820+0000 7f75f4485640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f75ec13dc00 0x7f75ec13fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:24:57.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.820+0000 7f75f4485640 10 monclient(hunting): picked mon.noname-a con 0x7f75ec13dc00 addr v2:172.21.15.121:3300/0 2024-09-17T18:24:57.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.820+0000 7f75f4485640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75ec0a8a80 con 0x7f75ec0a5fe0 2024-09-17T18:24:57.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.820+0000 7f75f4485640 10 monclient(hunting): start opening mon connection 2024-09-17T18:24:57.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.820+0000 7f75f4485640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f75ec0a7110 con 0x7f75ec13dc00 2024-09-17T18:24:57.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.820+0000 7f75f4485640 10 monclient(hunting): start opening mon connection 2024-09-17T18:24:57.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.820+0000 7f75f4485640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f75ec09de00 con 0x7f75ec0a5c00 2024-09-17T18:24:57.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.820+0000 7f75f4485640 10 monclient(hunting): _renew_subs 2024-09-17T18:24:57.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.820+0000 7f75f4485640 10 monclient(hunting): authenticate will time out at 1101.537156s 2024-09-17T18:24:57.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.820+0000 7f75f2a22640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f75ec13dc00 0x7f75ec13fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:24:57.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.820+0000 7f75f1a20640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f75ec0a5fe0 0x7f75ec0a63f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:54694/0 (socket says 172.21.15.121:54694) 2024-09-17T18:24:57.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.820+0000 7f75f1a20640 1 -- 172.21.15.121:0/3406807003 learned_addr learned my addr 172.21.15.121:0/3406807003 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:24:57.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.820+0000 7f75f2a22640 10 monclient(hunting): get_auth_request con 0x7f75ec13dc00 auth_method 0 2024-09-17T18:24:57.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.820+0000 7f75f2a22640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:24:57.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.820+0000 7f75f2a22640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:24:57.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.820+0000 7f75f2a22640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:24:57.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.820+0000 7f75f2a22640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:24:57.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.820+0000 7f75f2a22640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:24:57.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.820+0000 7f75f2a22640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:24:57.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.820+0000 7f75f2a22640 10 monclient(hunting): handle_auth_done global_id 44292 payload 274 2024-09-17T18:24:57.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.820+0000 7f75f2221640 1 --2- 172.21.15.121:0/3406807003 >> v2:172.21.15.173:3300/0 conn(0x7f75ec0a5c00 0x7f75ec09d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:24:57.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.820+0000 7f75f2a22640 1 -- 172.21.15.121:0/3406807003 >> v2:172.21.15.173:3300/0 conn(0x7f75ec0a5c00 msgr2=0x7f75ec09d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:24:57.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.820+0000 7f75f2a22640 1 --2- 172.21.15.121:0/3406807003 >> v2:172.21.15.173:3300/0 conn(0x7f75ec0a5c00 0x7f75ec09d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:24:57.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.820+0000 7f75f2a22640 1 -- 172.21.15.121:0/3406807003 >> v1:172.21.15.121:6789/0 conn(0x7f75ec0a5fe0 legacy=0x7f75ec0a63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:24:57.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.820+0000 7f75f2a22640 10 monclient: _finish_hunting 0 2024-09-17T18:24:57.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.820+0000 7f75f2a22640 1 monclient: found mon.noname-a 2024-09-17T18:24:57.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.820+0000 7f75f2a22640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:24:57.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.820+0000 7f75f2a22640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:24:57.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.820+0000 7f75f2a22640 1 -- 172.21.15.121:0/3406807003 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75ec0a74d0 con 0x7f75ec13dc00 2024-09-17T18:24:57.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.820+0000 7f75f2221640 10 monclient: get_auth_request con 0x7f75ec0a5c00 auth_method 0 2024-09-17T18:24:57.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.820+0000 7f75f2221640 1 --2- 172.21.15.121:0/3406807003 >> v2:172.21.15.173:3300/0 conn(0x7f75ec0a5c00 0x7f75ec09d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T18:24:57.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.823+0000 7f75f2a22640 1 -- 172.21.15.121:0/3406807003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75ec13dc00 msgr2=0x7f75ec13fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:24:57.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.823+0000 7f75f2a22640 1 --2- 172.21.15.121:0/3406807003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75ec13dc00 0x7f75ec13fff0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f75ec06a3f0 tx=0x7f75e402a4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12d67102add14732 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:24:57.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.823+0000 7f75f121f640 1 -- 172.21.15.121:0/3406807003 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e4007a30 con 0x7f75ec13dc00 2024-09-17T18:24:57.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.823+0000 7f75f121f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:24:57.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.823+0000 7f75f121f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:24:57.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.823+0000 7f75f121f640 10 monclient: dump: 2024-09-17T18:24:57.834 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:24:57.834 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:24:57.834 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:24:57.834 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:24:57.834 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:24:57.835 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:24:57.835 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:24:57.835 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:24:57.835 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:24:57.835 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:24:57.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.823+0000 7f75f121f640 10 monclient: _finish_auth 0 2024-09-17T18:24:57.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.823+0000 7f75f121f640 10 monclient: _check_auth_tickets 2024-09-17T18:24:57.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.823+0000 7f75f121f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:24:57.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.823+0000 7f75f121f640 1 -- 172.21.15.121:0/3406807003 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75e4007bd0 con 0x7f75ec13dc00 2024-09-17T18:24:57.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.823+0000 7f75f121f640 10 monclient: handle_config config(0 keys) 2024-09-17T18:24:57.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.823+0000 7f75f121f640 1 -- 172.21.15.121:0/3406807003 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e4031400 con 0x7f75ec13dc00 2024-09-17T18:24:57.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.823+0000 7f75f121f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:24:57.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.823+0000 7f75f121f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:24:57.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.823+0000 7f75f121f640 10 monclient: dump: 2024-09-17T18:24:57.835 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:24:57.836 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:24:57.836 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:24:57.836 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:24:57.836 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:24:57.836 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:24:57.836 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:24:57.836 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:24:57.836 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:24:57.836 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:24:57.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.823+0000 7f75f4485640 5 monclient: authenticate success, global_id 44292 2024-09-17T18:24:57.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.823+0000 7f75f4485640 10 monclient: get_monmap_and_config success 2024-09-17T18:24:57.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.823+0000 7f75f4485640 4 set_mon_vals no callback set 2024-09-17T18:24:57.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.824+0000 7f75f4485640 10 monclient: shutdown 2024-09-17T18:24:57.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.824+0000 7f75f4485640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:24:57.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.824+0000 7f75f4485640 1 -- 172.21.15.121:0/3406807003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75ec13dc00 msgr2=0x7f75ec13fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:24:57.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.824+0000 7f75f4485640 1 --2- 172.21.15.121:0/3406807003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75ec13dc00 0x7f75ec13fff0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f75ec06a3f0 tx=0x7f75e402a4b0 comp rx=0 tx=0).stop 2024-09-17T18:24:57.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.824+0000 7f75f4485640 1 -- 172.21.15.121:0/3406807003 shutdown_connections 2024-09-17T18:24:57.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.824+0000 7f75f4485640 1 --2- 172.21.15.121:0/3406807003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75ec13dc00 0x7f75ec13fff0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:24:57.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.824+0000 7f75f4485640 1 --2- 172.21.15.121:0/3406807003 >> v2:172.21.15.173:3300/0 conn(0x7f75ec0a5c00 0x7f75ec09d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:24:57.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.824+0000 7f75f4485640 1 -- 172.21.15.121:0/3406807003 >> 172.21.15.121:0/3406807003 conn(0x7f75ec0ab9b0 msgr2=0x7f75ec0a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:24:57.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.824+0000 7f75f4485640 1 -- 172.21.15.121:0/3406807003 shutdown_connections 2024-09-17T18:24:57.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.824+0000 7f75f4485640 1 -- 172.21.15.121:0/3406807003 wait complete. 2024-09-17T18:24:57.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.824+0000 7f75f4485640 10 monclient: build_initial_monmap 2024-09-17T18:24:57.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.824+0000 7f75f4485640 10 monclient: monmap: 2024-09-17T18:24:57.837 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:24:57.837 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:24:57.837 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:24:57.837 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:24:57.838 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:24:57.838 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:24:57.838 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:24:57.838 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:24:57.838 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:24:57.838 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:24:57.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.824+0000 7f75f4485640 1 librados: starting msgr at 2024-09-17T18:24:57.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.824+0000 7f75f4485640 1 librados: starting objecter 2024-09-17T18:24:57.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f4485640 1 Processor -- start 2024-09-17T18:24:57.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f4485640 1 -- start start 2024-09-17T18:24:57.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f4485640 1 librados: setting wanted keys 2024-09-17T18:24:57.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f4485640 1 librados: calling monclient init 2024-09-17T18:24:57.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f4485640 10 monclient: init 2024-09-17T18:24:57.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f4485640 10 monclient: _reopen_session rank -1 2024-09-17T18:24:57.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f4485640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T18:24:57.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f4485640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f75ec0a5c00 0x7f75ec1cd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:24:57.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f4485640 10 monclient(hunting): picked mon.noname-b con 0x7f75ec0a5c00 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:24:57.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f4485640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75ec0a5fe0 0x7f75ec138a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:24:57.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f4485640 10 monclient(hunting): picked mon.noname-c con 0x7f75ec0a5fe0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:24:57.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f4485640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75ec13dc00 0x7f75ec13ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:24:57.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f4485640 10 monclient(hunting): picked mon.noname-a con 0x7f75ec13dc00 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:24:57.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f4485640 10 monclient(hunting): start opening mon connection 2024-09-17T18:24:57.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f4485640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f75ec0703d0 con 0x7f75ec13dc00 2024-09-17T18:24:57.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f4485640 10 monclient(hunting): start opening mon connection 2024-09-17T18:24:57.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f4485640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f75ec09ce70 con 0x7f75ec0a5c00 2024-09-17T18:24:57.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f4485640 10 monclient(hunting): start opening mon connection 2024-09-17T18:24:57.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f4485640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f75ec070550 con 0x7f75ec0a5fe0 2024-09-17T18:24:57.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f4485640 10 monclient(hunting): _renew_subs 2024-09-17T18:24:57.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f4485640 10 monclient(hunting): authenticate will time out at 1101.542544s 2024-09-17T18:24:57.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f2a22640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75ec13dc00 0x7f75ec13ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:24:57.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f1a20640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75ec0a5fe0 0x7f75ec138a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:24:57.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f2a22640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75ec13dc00 0x7f75ec13ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45218/0 (socket says 172.21.15.121:45218) 2024-09-17T18:24:57.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f2a22640 1 -- 172.21.15.121:0/2342575382 learned_addr learned my addr 172.21.15.121:0/2342575382 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:24:57.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f1a20640 10 monclient(hunting): get_auth_request con 0x7f75ec0a5fe0 auth_method 0 2024-09-17T18:24:57.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f1a20640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:24:57.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f1a20640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:24:57.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f1a20640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:24:57.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f2a22640 10 monclient(hunting): get_auth_request con 0x7f75ec13dc00 auth_method 0 2024-09-17T18:24:57.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f2a22640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:24:57.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f2a22640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:24:57.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f2a22640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:24:57.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f1a20640 1 -- 172.21.15.121:0/2342575382 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75ec0a5fe0 msgr2=0x7f75ec138a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T18:24:57.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f1a20640 1 -- 172.21.15.121:0/2342575382 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75ec0a5fe0 msgr2=0x7f75ec138a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:24:57.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f1a20640 1 --2- 172.21.15.121:0/2342575382 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75ec0a5fe0 0x7f75ec138a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:24:57.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f2a22640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:24:57.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f2a22640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:24:57.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f2a22640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:24:57.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f2221640 1 --2- 172.21.15.121:0/2342575382 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f75ec0a5c00 0x7f75ec1cd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:24:57.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f1a20640 1 --2- 172.21.15.121:0/2342575382 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75ec0a5fe0 0x7f75ec138a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:24:57.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f2221640 10 monclient(hunting): get_auth_request con 0x7f75ec0a5c00 auth_method 0 2024-09-17T18:24:57.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f2221640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:24:57.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f2221640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:24:57.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f2221640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:24:57.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f2221640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:24:57.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f2221640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:24:57.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f2221640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:24:57.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f2a22640 10 monclient(hunting): handle_auth_done global_id 44295 payload 824 2024-09-17T18:24:57.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f2a22640 1 -- 172.21.15.121:0/2342575382 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75ec0a5fe0 msgr2=0x7f75ec138a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:24:57.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f2a22640 1 --2- 172.21.15.121:0/2342575382 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75ec0a5fe0 0x7f75ec138a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:24:57.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f2a22640 1 -- 172.21.15.121:0/2342575382 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f75ec0a5c00 msgr2=0x7f75ec1cd4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:24:57.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.825+0000 7f75f2a22640 1 --2- 172.21.15.121:0/2342575382 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f75ec0a5c00 0x7f75ec1cd4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:24:57.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.826+0000 7f75f2a22640 10 monclient: _finish_hunting 0 2024-09-17T18:24:57.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.826+0000 7f75f2a22640 1 monclient: found mon.noname-a 2024-09-17T18:24:57.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.826+0000 7f75f2a22640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:24:57.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.826+0000 7f75f2a22640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:24:57.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.826+0000 7f75f2a22640 1 -- 172.21.15.121:0/2342575382 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75ec123140 con 0x7f75ec13dc00 2024-09-17T18:24:57.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.828+0000 7f75f2a22640 1 -- 172.21.15.121:0/2342575382 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75ec13dc00 msgr2=0x7f75ec13ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:24:57.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.828+0000 7f75f2a22640 1 --2- 172.21.15.121:0/2342575382 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75ec13dc00 0x7f75ec13ba50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f75e402aaa0 tx=0x7f75e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:24:57.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.828+0000 7f75e2ffd640 10 client.?.objecter ms_handle_connect 0x7f75ec13dc00 2024-09-17T18:24:57.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.828+0000 7f75e2ffd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:24:57.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.828+0000 7f75e2ffd640 1 -- 172.21.15.121:0/2342575382 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e4040070 con 0x7f75ec13dc00 2024-09-17T18:24:57.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.828+0000 7f75e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:24:57.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.828+0000 7f75e2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:24:57.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.828+0000 7f75e2ffd640 10 monclient: dump: 2024-09-17T18:24:57.843 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:24:57.843 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:24:57.843 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:24:57.843 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:24:57.843 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:24:57.843 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:24:57.843 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:24:57.843 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:24:57.843 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:24:57.843 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:24:57.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.828+0000 7f75e2ffd640 10 monclient: _finish_auth 0 2024-09-17T18:24:57.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.828+0000 7f75e2ffd640 10 monclient: _check_auth_tickets 2024-09-17T18:24:57.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.828+0000 7f75e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:24:57.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.828+0000 7f75e2ffd640 1 -- 172.21.15.121:0/2342575382 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75e4045660 con 0x7f75ec13dc00 2024-09-17T18:24:57.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.828+0000 7f75e2ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:24:57.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.828+0000 7f75e2ffd640 1 -- 172.21.15.121:0/2342575382 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e40314c0 con 0x7f75ec13dc00 2024-09-17T18:24:57.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.828+0000 7f75f4485640 5 monclient: authenticate success, global_id 44295 2024-09-17T18:24:57.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.828+0000 7f75f4485640 10 monclient: _renew_subs 2024-09-17T18:24:57.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.828+0000 7f75f4485640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:24:57.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.828+0000 7f75f4485640 1 -- 172.21.15.121:0/2342575382 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75ec123420 con 0x7f75ec13dc00 2024-09-17T18:24:57.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.828+0000 7f75e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:24:57.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.828+0000 7f75f0a1e640 4 set_mon_vals no callback set 2024-09-17T18:24:57.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.828+0000 7f75e2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:24:57.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.828+0000 7f75e2ffd640 10 monclient: dump: 2024-09-17T18:24:57.845 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:24:57.845 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:24:57.845 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:24:57.845 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:24:57.845 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:24:57.845 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:24:57.845 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:24:57.845 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:24:57.845 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:24:57.845 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:24:57.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.828+0000 7f75f4485640 10 client.44295.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:24:57.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.828+0000 7f75f4485640 10 monclient: _renew_subs 2024-09-17T18:24:57.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.828+0000 7f75f4485640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:24:57.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.828+0000 7f75f4485640 1 -- 172.21.15.121:0/2342575382 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75ec0a1f90 con 0x7f75ec13dc00 2024-09-17T18:24:57.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.828+0000 7f75f4485640 1 librados: init done 2024-09-17T18:24:57.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.829+0000 7f75e2ffd640 1 -- 172.21.15.121:0/2342575382 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f75e4039040 con 0x7f75ec13dc00 2024-09-17T18:24:57.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.829+0000 7f75f4485640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:24:57.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.829+0000 7f75f4485640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:24:57.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.829+0000 7f75f4485640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:24:57.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.829+0000 7f75f4485640 1 -- 172.21.15.121:0/2342575382 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75ec140050 con 0x7f75ec13dc00 2024-09-17T18:24:57.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.829+0000 7f75e2ffd640 1 --2- 172.21.15.121:0/2342575382 >> v2:172.21.15.173:6812/2423786037 conn(0x7f75d0040170 0x7f75d0042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:24:57.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.829+0000 7f75e2ffd640 1 -- 172.21.15.121:0/2342575382 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(269..269 src has 1..269) ==== 5784+0+0 (secure 0 0 0) 0x7f75e4075340 con 0x7f75ec13dc00 2024-09-17T18:24:57.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.829+0000 7f75e2ffd640 10 client.44295.objecter ms_dispatch 0x7f75ec000b90 osd_map(269..269 src has 1..269) 2024-09-17T18:24:57.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.829+0000 7f75e2ffd640 3 client.44295.objecter handle_osd_map got epochs [269,269] > 0 2024-09-17T18:24:57.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.829+0000 7f75e2ffd640 3 client.44295.objecter handle_osd_map decoding full epoch 269 2024-09-17T18:24:57.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.829+0000 7f75e2ffd640 20 client.44295.objecter dump_active .. 0 homeless 2024-09-17T18:24:57.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.829+0000 7f75f2221640 1 --2- 172.21.15.121:0/2342575382 >> v2:172.21.15.173:6812/2423786037 conn(0x7f75d0040170 0x7f75d0042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:24:57.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.829+0000 7f75f2221640 10 monclient: get_auth_request con 0x7f75d0040170 auth_method 0 2024-09-17T18:24:57.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.832+0000 7f75f2221640 1 --2- 172.21.15.121:0/2342575382 >> v2:172.21.15.173:6812/2423786037 conn(0x7f75d0040170 0x7f75d0042630 secure :-1 s=READY pgs=3963 cs=0 l=1 rev1=1 crypto rx=0x7f75e800ab60 tx=0x7f75e8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:24:57.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.832+0000 7f75e2ffd640 10 client.44295.objecter ms_handle_connect 0x7f75d0040170 2024-09-17T18:24:57.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.834+0000 7f75e2ffd640 1 -- 172.21.15.121:0/2342575382 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f75ec140050 con 0x7f75ec13dc00 2024-09-17T18:24:57.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.834+0000 7f75e2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:24:57.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:57.834+0000 7f75e2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:24:58.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:58.077+0000 7f75f4485640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:24:58.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:58.077+0000 7f75f4485640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:24:58.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:58.077+0000 7f75f4485640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:24:58.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:58.077+0000 7f75f4485640 1 -- 172.21.15.121:0/2342575382 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f75ec13d420 con 0x7f75ec13dc00 2024-09-17T18:24:58.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:58.078+0000 7f75e2ffd640 1 -- 172.21.15.121:0/2342575382 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f75e4073950 con 0x7f75ec13dc00 2024-09-17T18:24:58.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:58.078+0000 7f75e2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:24:58.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:58.078+0000 7f75e2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:24:58.081 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":24,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":91,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:24:58.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:58.079+0000 7f75f4485640 10 librados: watch_flush enter 2024-09-17T18:24:58.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:58.079+0000 7f75f4485640 10 librados: watch_flush exit 2024-09-17T18:24:58.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:58.079+0000 7f75f4485640 20 client.44295.objecter shutdown clearing up homeless session... 2024-09-17T18:24:58.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:58.079+0000 7f75f4485640 10 client.44295.objecter successfully canceled tick 2024-09-17T18:24:58.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:58.080+0000 7f75f4485640 1 -- 172.21.15.121:0/2342575382 >> v2:172.21.15.173:6812/2423786037 conn(0x7f75d0040170 msgr2=0x7f75d0042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:24:58.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:58.080+0000 7f75f4485640 1 --2- 172.21.15.121:0/2342575382 >> v2:172.21.15.173:6812/2423786037 conn(0x7f75d0040170 0x7f75d0042630 secure :-1 s=READY pgs=3963 cs=0 l=1 rev1=1 crypto rx=0x7f75e800ab60 tx=0x7f75e8008040 comp rx=0 tx=0).stop 2024-09-17T18:24:58.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:58.080+0000 7f75f4485640 10 monclient: shutdown 2024-09-17T18:24:58.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:58.080+0000 7f75f4485640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:24:58.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:58.080+0000 7f75f4485640 1 -- 172.21.15.121:0/2342575382 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75ec13dc00 msgr2=0x7f75ec13ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:24:58.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:58.080+0000 7f75f4485640 1 --2- 172.21.15.121:0/2342575382 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75ec13dc00 0x7f75ec13ba50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f75e402aaa0 tx=0x7f75e4004a40 comp rx=0 tx=0).stop 2024-09-17T18:24:58.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:58.080+0000 7f75f4485640 1 -- 172.21.15.121:0/2342575382 shutdown_connections 2024-09-17T18:24:58.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:58.080+0000 7f75f4485640 1 --2- 172.21.15.121:0/2342575382 >> v2:172.21.15.173:6812/2423786037 conn(0x7f75d0040170 0x7f75d0042630 unknown :-1 s=CLOSED pgs=3963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:24:58.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:58.080+0000 7f75f4485640 1 --2- 172.21.15.121:0/2342575382 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75ec13dc00 0x7f75ec13ba50 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:24:58.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:58.080+0000 7f75f4485640 1 --2- 172.21.15.121:0/2342575382 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75ec0a5fe0 0x7f75ec138a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:24:58.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:58.080+0000 7f75f4485640 1 --2- 172.21.15.121:0/2342575382 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f75ec0a5c00 0x7f75ec1cd4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:24:58.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:58.080+0000 7f75f4485640 1 -- 172.21.15.121:0/2342575382 >> 172.21.15.121:0/2342575382 conn(0x7f75ec0ab9b0 msgr2=0x7f75ec0a0e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:24:58.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:58.080+0000 7f75f4485640 1 -- 172.21.15.121:0/2342575382 shutdown_connections 2024-09-17T18:24:58.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:58.080+0000 7f75f4485640 1 -- 172.21.15.121:0/2342575382 wait complete. 2024-09-17T18:24:58.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:24:58.080+0000 7f75f4485640 1 librados: shutdown 2024-09-17T18:25:01.093 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:25:01.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.198+0000 7f60cad56640 10 monclient: get_monmap_and_config 2024-09-17T18:25:01.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.198+0000 7f60cad56640 10 monclient: build_initial_monmap 2024-09-17T18:25:01.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.198+0000 7f60cad56640 10 monclient: monmap: 2024-09-17T18:25:01.201 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:25:01.202 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:01.202 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:25:01.200419+0000 2024-09-17T18:25:01.202 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:25:01.200419+0000 2024-09-17T18:25:01.202 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:25:01.202 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:01.202 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:25:01.202 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:25:01.202 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:25:01.202 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:25:01.202 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:25:01.202 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:01.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.199+0000 7f60cad56640 1 Processor -- start 2024-09-17T18:25:01.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.199+0000 7f60cad56640 1 -- start start 2024-09-17T18:25:01.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.199+0000 7f60cad56640 10 monclient: init 2024-09-17T18:25:01.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60cad56640 10 monclient: _reopen_session rank -1 2024-09-17T18:25:01.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60cad56640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-17T18:25:01.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60cad56640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f60c4136800 addr v1:172.21.15.173:6789/0 2024-09-17T18:25:01.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60cad56640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60c4136170 0x7f60c4136be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:01.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60cad56640 10 monclient(hunting): picked mon.noname-b con 0x7f60c4136170 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:25:01.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60cad56640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f60c4133360 0x7f60c413e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:01.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60cad56640 10 monclient(hunting): picked mon.noname-a con 0x7f60c4133360 addr v2:172.21.15.121:3300/0 2024-09-17T18:25:01.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60cad56640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60c40a0c20 con 0x7f60c4136800 2024-09-17T18:25:01.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60cad56640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:01.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60cad56640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f60c4137120 con 0x7f60c4133360 2024-09-17T18:25:01.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60cad56640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:01.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60cad56640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f60c40a2cb0 con 0x7f60c4136170 2024-09-17T18:25:01.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60cad56640 10 monclient(hunting): _renew_subs 2024-09-17T18:25:01.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60cad56640 10 monclient(hunting): authenticate will time out at 1104.917172s 2024-09-17T18:25:01.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60c3fff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60c4136170 0x7f60c4136be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:01.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60c92f3640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f60c4133360 0x7f60c413e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:01.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60c3fff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60c4136170 0x7f60c4136be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:33754/0 (socket says 172.21.15.121:33754) 2024-09-17T18:25:01.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60c92f3640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f60c4133360 0x7f60c413e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:49530/0 (socket says 172.21.15.121:49530) 2024-09-17T18:25:01.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60c3fff640 1 -- 172.21.15.121:0/921937421 learned_addr learned my addr 172.21.15.121:0/921937421 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:25:01.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60c3fff640 10 monclient(hunting): get_auth_request con 0x7f60c4136170 auth_method 0 2024-09-17T18:25:01.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:01.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:01.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:01.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60c92f3640 10 monclient(hunting): get_auth_request con 0x7f60c4133360 auth_method 0 2024-09-17T18:25:01.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60c92f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:01.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60c92f3640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:01.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60c92f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:01.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60c3fff640 1 -- 172.21.15.121:0/921937421 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60c4136170 msgr2=0x7f60c4136be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T18:25:01.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60c3fff640 1 -- 172.21.15.121:0/921937421 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60c4136170 msgr2=0x7f60c4136be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:25:01.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60c3fff640 1 --2- 172.21.15.121:0/921937421 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60c4136170 0x7f60c4136be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:25:01.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60c92f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:01.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60c92f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:01.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60c92f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:01.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60c3fff640 1 --2- 172.21.15.121:0/921937421 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60c4136170 0x7f60c4136be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:25:01.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60c92f3640 10 monclient(hunting): handle_auth_done global_id 44304 payload 274 2024-09-17T18:25:01.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60c92f3640 1 -- 172.21.15.121:0/921937421 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60c4136170 msgr2=0x7f60c4136be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:25:01.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60c92f3640 1 --2- 172.21.15.121:0/921937421 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60c4136170 0x7f60c4136be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:01.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60c92f3640 1 -- 172.21.15.121:0/921937421 >> v1:172.21.15.173:6789/0 conn(0x7f60c4136800 legacy=0x7f60c4132c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:25:01.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60c92f3640 10 monclient: _finish_hunting 0 2024-09-17T18:25:01.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60c92f3640 1 monclient: found mon.noname-a 2024-09-17T18:25:01.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60c92f3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:25:01.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60c92f3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:25:01.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.200+0000 7f60c92f3640 1 -- 172.21.15.121:0/921937421 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60c40a2f90 con 0x7f60c4133360 2024-09-17T18:25:01.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.203+0000 7f60c92f3640 1 -- 172.21.15.121:0/921937421 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60c4133360 msgr2=0x7f60c413e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:01.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.203+0000 7f60c92f3640 1 --2- 172.21.15.121:0/921937421 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60c4133360 0x7f60c413e510 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f60c4076980 tx=0x7f60bc02a460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bce166f9f0f418e server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:01.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.203+0000 7f60c37fe640 1 -- 172.21.15.121:0/921937421 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60bc007950 con 0x7f60c4133360 2024-09-17T18:25:01.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.203+0000 7f60c37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:01.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.203+0000 7f60c37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:25:01.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.203+0000 7f60c37fe640 10 monclient: dump: 2024-09-17T18:25:01.215 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:01.215 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:01.215 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:01.215 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:01.215 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:01.215 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:01.215 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:01.215 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:01.215 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:01.215 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:01.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.203+0000 7f60c37fe640 10 monclient: _finish_auth 0 2024-09-17T18:25:01.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.203+0000 7f60c37fe640 10 monclient: _check_auth_tickets 2024-09-17T18:25:01.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.203+0000 7f60c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:25:01.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.203+0000 7f60c37fe640 1 -- 172.21.15.121:0/921937421 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60bc007af0 con 0x7f60c4133360 2024-09-17T18:25:01.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.203+0000 7f60cad56640 5 monclient: authenticate success, global_id 44304 2024-09-17T18:25:01.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.203+0000 7f60cad56640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:25:01.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.203+0000 7f60c37fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:25:01.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.203+0000 7f60c37fe640 1 -- 172.21.15.121:0/921937421 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60bc031400 con 0x7f60c4133360 2024-09-17T18:25:01.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.203+0000 7f60c37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:01.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.203+0000 7f60c37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:25:01.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.203+0000 7f60c37fe640 10 monclient: dump: 2024-09-17T18:25:01.216 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:01.216 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:01.216 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:01.217 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:01.217 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:01.217 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:01.217 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:01.217 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:01.217 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:01.217 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:01.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.203+0000 7f60cad56640 10 monclient: get_monmap_and_config success 2024-09-17T18:25:01.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.203+0000 7f60cad56640 4 set_mon_vals no callback set 2024-09-17T18:25:01.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.204+0000 7f60cad56640 10 monclient: shutdown 2024-09-17T18:25:01.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.204+0000 7f60cad56640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:25:01.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.204+0000 7f60cad56640 1 -- 172.21.15.121:0/921937421 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60c4133360 msgr2=0x7f60c413e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:01.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.204+0000 7f60cad56640 1 --2- 172.21.15.121:0/921937421 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60c4133360 0x7f60c413e510 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f60c4076980 tx=0x7f60bc02a460 comp rx=0 tx=0).stop 2024-09-17T18:25:01.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.204+0000 7f60cad56640 1 -- 172.21.15.121:0/921937421 shutdown_connections 2024-09-17T18:25:01.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.204+0000 7f60cad56640 1 --2- 172.21.15.121:0/921937421 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60c4133360 0x7f60c413e510 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:01.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.204+0000 7f60cad56640 1 --2- 172.21.15.121:0/921937421 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60c4136170 0x7f60c4136be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:01.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.204+0000 7f60cad56640 1 -- 172.21.15.121:0/921937421 >> 172.21.15.121:0/921937421 conn(0x7f60c40a3b50 msgr2=0x7f60c40a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:25:01.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.204+0000 7f60cad56640 1 -- 172.21.15.121:0/921937421 shutdown_connections 2024-09-17T18:25:01.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.204+0000 7f60cad56640 1 -- 172.21.15.121:0/921937421 wait complete. 2024-09-17T18:25:01.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.204+0000 7f60cad56640 10 monclient: build_initial_monmap 2024-09-17T18:25:01.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.204+0000 7f60cad56640 10 monclient: monmap: 2024-09-17T18:25:01.218 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:25:01.218 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:25:01.218 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:25:01.218 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:25:01.218 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:25:01.218 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:01.218 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:25:01.218 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:25:01.218 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:25:01.219 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:01.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.204+0000 7f60cad56640 1 librados: starting msgr at 2024-09-17T18:25:01.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.204+0000 7f60cad56640 1 librados: starting objecter 2024-09-17T18:25:01.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.205+0000 7f60cad56640 1 Processor -- start 2024-09-17T18:25:01.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.205+0000 7f60cad56640 1 -- start start 2024-09-17T18:25:01.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.205+0000 7f60cad56640 1 librados: setting wanted keys 2024-09-17T18:25:01.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.205+0000 7f60cad56640 1 librados: calling monclient init 2024-09-17T18:25:01.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.205+0000 7f60cad56640 10 monclient: init 2024-09-17T18:25:01.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.205+0000 7f60cad56640 10 monclient: _reopen_session rank -1 2024-09-17T18:25:01.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.205+0000 7f60cad56640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:25:01.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.205+0000 7f60cad56640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60c4133360 0x7f60c41cb3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:01.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.205+0000 7f60cad56640 10 monclient(hunting): picked mon.noname-a con 0x7f60c4133360 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:25:01.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.205+0000 7f60cad56640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60c4136170 0x7f60c412faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:01.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.205+0000 7f60cad56640 10 monclient(hunting): picked mon.noname-b con 0x7f60c4136170 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:25:01.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.205+0000 7f60cad56640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60c4136800 0x7f60c411d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:01.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.205+0000 7f60cad56640 10 monclient(hunting): picked mon.noname-c con 0x7f60c4136800 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:25:01.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.205+0000 7f60cad56640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:01.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.205+0000 7f60cad56640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f60c41365b0 con 0x7f60c4133360 2024-09-17T18:25:01.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.205+0000 7f60cad56640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:01.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.205+0000 7f60cad56640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f60c40a3900 con 0x7f60c4136170 2024-09-17T18:25:01.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.205+0000 7f60cad56640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:01.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.205+0000 7f60cad56640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f60c409dce0 con 0x7f60c4136800 2024-09-17T18:25:01.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.205+0000 7f60cad56640 10 monclient(hunting): _renew_subs 2024-09-17T18:25:01.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.205+0000 7f60cad56640 10 monclient(hunting): authenticate will time out at 1104.922824s 2024-09-17T18:25:01.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.205+0000 7f60c8af2640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60c4133360 0x7f60c41cb3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:01.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.205+0000 7f60c92f3640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60c4136800 0x7f60c411d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:01.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.205+0000 7f60c8af2640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60c4133360 0x7f60c41cb3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:49544/0 (socket says 172.21.15.121:49544) 2024-09-17T18:25:01.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.205+0000 7f60c92f3640 10 monclient(hunting): get_auth_request con 0x7f60c4136800 auth_method 0 2024-09-17T18:25:01.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.205+0000 7f60c8af2640 1 -- 172.21.15.121:0/1035752346 learned_addr learned my addr 172.21.15.121:0/1035752346 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:25:01.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.205+0000 7f60c92f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:01.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.205+0000 7f60c92f3640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:01.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.205+0000 7f60c92f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:01.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.205+0000 7f60c8af2640 10 monclient(hunting): get_auth_request con 0x7f60c4133360 auth_method 0 2024-09-17T18:25:01.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.205+0000 7f60c8af2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:01.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.205+0000 7f60c8af2640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:01.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.205+0000 7f60c8af2640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:01.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.206+0000 7f60c92f3640 1 -- 172.21.15.121:0/1035752346 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60c4136800 msgr2=0x7f60c411d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-17T18:25:01.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.206+0000 7f60c92f3640 1 -- 172.21.15.121:0/1035752346 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60c4136800 msgr2=0x7f60c411d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:25:01.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.206+0000 7f60c92f3640 1 --2- 172.21.15.121:0/1035752346 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60c4136800 0x7f60c411d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:25:01.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.206+0000 7f60c8af2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:01.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.206+0000 7f60c8af2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:01.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.206+0000 7f60c8af2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:01.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.206+0000 7f60c92f3640 1 --2- 172.21.15.121:0/1035752346 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60c4136800 0x7f60c411d6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:25:01.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.206+0000 7f60c3fff640 1 --2- 172.21.15.121:0/1035752346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60c4136170 0x7f60c412faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:01.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.206+0000 7f60c8af2640 10 monclient(hunting): handle_auth_done global_id 44307 payload 824 2024-09-17T18:25:01.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.206+0000 7f60c8af2640 1 -- 172.21.15.121:0/1035752346 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60c4136800 msgr2=0x7f60c411d6a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:25:01.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.206+0000 7f60c8af2640 1 --2- 172.21.15.121:0/1035752346 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60c4136800 0x7f60c411d6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:01.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.206+0000 7f60c8af2640 1 -- 172.21.15.121:0/1035752346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60c4136170 msgr2=0x7f60c412faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:01.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.206+0000 7f60c8af2640 1 --2- 172.21.15.121:0/1035752346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60c4136170 0x7f60c412faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:01.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.206+0000 7f60c8af2640 10 monclient: _finish_hunting 0 2024-09-17T18:25:01.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.206+0000 7f60c8af2640 1 monclient: found mon.noname-a 2024-09-17T18:25:01.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.206+0000 7f60c8af2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:25:01.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.206+0000 7f60c8af2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:25:01.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.206+0000 7f60c8af2640 1 -- 172.21.15.121:0/1035752346 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60c41330e0 con 0x7f60c4133360 2024-09-17T18:25:01.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.206+0000 7f60c3fff640 10 monclient: get_auth_request con 0x7f60c4136170 auth_method 0 2024-09-17T18:25:01.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.206+0000 7f60c3fff640 1 --2- 172.21.15.121:0/1035752346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60c4136170 0x7f60c412faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T18:25:01.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.208+0000 7f60c8af2640 1 -- 172.21.15.121:0/1035752346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60c4133360 msgr2=0x7f60c41cb3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:01.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.208+0000 7f60c8af2640 1 --2- 172.21.15.121:0/1035752346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60c4133360 0x7f60c41cb3f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f60b400c430 tx=0x7f60b400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:01.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.208+0000 7f60c17fa640 10 client.?.objecter ms_handle_connect 0x7f60c4133360 2024-09-17T18:25:01.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.208+0000 7f60c17fa640 10 client.?.objecter resend_mon_ops 2024-09-17T18:25:01.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.208+0000 7f60c17fa640 1 -- 172.21.15.121:0/1035752346 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b40119a0 con 0x7f60c4133360 2024-09-17T18:25:01.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.208+0000 7f60c17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:01.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.208+0000 7f60c17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:25:01.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.208+0000 7f60c17fa640 10 monclient: dump: 2024-09-17T18:25:01.223 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:01.223 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:01.223 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:01.223 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:01.223 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:01.223 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:01.223 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:01.223 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:01.223 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:01.224 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:01.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.208+0000 7f60c17fa640 10 monclient: _finish_auth 0 2024-09-17T18:25:01.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.208+0000 7f60c17fa640 10 monclient: _check_auth_tickets 2024-09-17T18:25:01.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.208+0000 7f60c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:25:01.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.208+0000 7f60c17fa640 1 -- 172.21.15.121:0/1035752346 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60b4017070 con 0x7f60c4133360 2024-09-17T18:25:01.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.208+0000 7f60cad56640 5 monclient: authenticate success, global_id 44307 2024-09-17T18:25:01.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.208+0000 7f60c17fa640 10 monclient: handle_config config(0 keys) 2024-09-17T18:25:01.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.208+0000 7f60c17fa640 1 -- 172.21.15.121:0/1035752346 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b4011d80 con 0x7f60c4133360 2024-09-17T18:25:01.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.208+0000 7f60cad56640 10 monclient: _renew_subs 2024-09-17T18:25:01.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.208+0000 7f60cad56640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:01.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.208+0000 7f60cad56640 1 -- 172.21.15.121:0/1035752346 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60c4121270 con 0x7f60c4133360 2024-09-17T18:25:01.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.208+0000 7f60c17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:01.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.208+0000 7f60c17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:25:01.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.208+0000 7f60c17fa640 10 monclient: dump: 2024-09-17T18:25:01.224 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:01.225 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:01.225 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:01.225 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:01.225 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:01.225 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:01.225 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:01.225 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:01.225 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:01.225 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:01.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.208+0000 7f60c2ffd640 4 set_mon_vals no callback set 2024-09-17T18:25:01.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.208+0000 7f60cad56640 10 client.44307.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:25:01.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.208+0000 7f60cad56640 10 monclient: _renew_subs 2024-09-17T18:25:01.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.208+0000 7f60cad56640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:01.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.208+0000 7f60cad56640 1 -- 172.21.15.121:0/1035752346 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60c411fa70 con 0x7f60c4133360 2024-09-17T18:25:01.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.208+0000 7f60cad56640 1 librados: init done 2024-09-17T18:25:01.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.209+0000 7f60c17fa640 1 -- 172.21.15.121:0/1035752346 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f60b401c460 con 0x7f60c4133360 2024-09-17T18:25:01.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.209+0000 7f60cad56640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:01.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.209+0000 7f60cad56640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:01.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.209+0000 7f60cad56640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:01.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.209+0000 7f60cad56640 1 -- 172.21.15.121:0/1035752346 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60c4133860 con 0x7f60c4133360 2024-09-17T18:25:01.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.209+0000 7f60c17fa640 1 --2- 172.21.15.121:0/1035752346 >> v2:172.21.15.173:6812/2423786037 conn(0x7f60b0040170 0x7f60b0042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:01.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.209+0000 7f60c17fa640 1 -- 172.21.15.121:0/1035752346 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 5679+0+0 (secure 0 0 0) 0x7f60b40536f0 con 0x7f60c4133360 2024-09-17T18:25:01.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.209+0000 7f60c17fa640 10 client.44307.objecter ms_dispatch 0x7f60c4000b90 osd_map(272..272 src has 1..272) 2024-09-17T18:25:01.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.209+0000 7f60c17fa640 3 client.44307.objecter handle_osd_map got epochs [272,272] > 0 2024-09-17T18:25:01.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.209+0000 7f60c17fa640 3 client.44307.objecter handle_osd_map decoding full epoch 272 2024-09-17T18:25:01.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.210+0000 7f60c17fa640 20 client.44307.objecter dump_active .. 0 homeless 2024-09-17T18:25:01.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.210+0000 7f60c3fff640 1 --2- 172.21.15.121:0/1035752346 >> v2:172.21.15.173:6812/2423786037 conn(0x7f60b0040170 0x7f60b0042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:01.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.210+0000 7f60c3fff640 10 monclient: get_auth_request con 0x7f60b0040170 auth_method 0 2024-09-17T18:25:01.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.212+0000 7f60c17fa640 10 client.44307.objecter ms_handle_connect 0x7f60b0040170 2024-09-17T18:25:01.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.212+0000 7f60c3fff640 1 --2- 172.21.15.121:0/1035752346 >> v2:172.21.15.173:6812/2423786037 conn(0x7f60b0040170 0x7f60b0042630 secure :-1 s=READY pgs=3965 cs=0 l=1 rev1=1 crypto rx=0x7f60b8009100 tx=0x7f60b8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:01.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.214+0000 7f60c17fa640 1 -- 172.21.15.121:0/1035752346 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f60b4051d70 con 0x7f60c4133360 2024-09-17T18:25:01.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.214+0000 7f60c17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:01.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.214+0000 7f60c17fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:25:01.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.460+0000 7f60cad56640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:25:01.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.460+0000 7f60cad56640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:25:01.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.460+0000 7f60cad56640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:01.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.460+0000 7f60cad56640 1 -- 172.21.15.121:0/1035752346 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f60c409e1c0 con 0x7f60c4133360 2024-09-17T18:25:01.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.461+0000 7f60c17fa640 1 -- 172.21.15.121:0/1035752346 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f60b401c930 con 0x7f60c4133360 2024-09-17T18:25:01.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.461+0000 7f60c17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:25:01.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.461+0000 7f60c17fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:25:01.464 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":24,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":94,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:25:01.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.463+0000 7f60cad56640 10 librados: watch_flush enter 2024-09-17T18:25:01.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.463+0000 7f60cad56640 10 librados: watch_flush exit 2024-09-17T18:25:01.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.463+0000 7f60cad56640 20 client.44307.objecter shutdown clearing up homeless session... 2024-09-17T18:25:01.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.463+0000 7f60cad56640 10 client.44307.objecter successfully canceled tick 2024-09-17T18:25:01.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.463+0000 7f60cad56640 1 -- 172.21.15.121:0/1035752346 >> v2:172.21.15.173:6812/2423786037 conn(0x7f60b0040170 msgr2=0x7f60b0042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:01.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.463+0000 7f60cad56640 1 --2- 172.21.15.121:0/1035752346 >> v2:172.21.15.173:6812/2423786037 conn(0x7f60b0040170 0x7f60b0042630 secure :-1 s=READY pgs=3965 cs=0 l=1 rev1=1 crypto rx=0x7f60b8009100 tx=0x7f60b8008040 comp rx=0 tx=0).stop 2024-09-17T18:25:01.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.463+0000 7f60cad56640 10 monclient: shutdown 2024-09-17T18:25:01.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.463+0000 7f60cad56640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:25:01.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.463+0000 7f60cad56640 1 -- 172.21.15.121:0/1035752346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60c4133360 msgr2=0x7f60c41cb3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:01.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.463+0000 7f60cad56640 1 --2- 172.21.15.121:0/1035752346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60c4133360 0x7f60c41cb3f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f60b400c430 tx=0x7f60b400c8f0 comp rx=0 tx=0).stop 2024-09-17T18:25:01.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.464+0000 7f60cad56640 1 -- 172.21.15.121:0/1035752346 shutdown_connections 2024-09-17T18:25:01.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.464+0000 7f60cad56640 1 --2- 172.21.15.121:0/1035752346 >> v2:172.21.15.173:6812/2423786037 conn(0x7f60b0040170 0x7f60b0042630 unknown :-1 s=CLOSED pgs=3965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:01.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.464+0000 7f60cad56640 1 --2- 172.21.15.121:0/1035752346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60c4136170 0x7f60c412faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:01.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.464+0000 7f60cad56640 1 --2- 172.21.15.121:0/1035752346 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60c4136800 0x7f60c411d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:01.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.464+0000 7f60cad56640 1 --2- 172.21.15.121:0/1035752346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60c4133360 0x7f60c41cb3f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:01.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.464+0000 7f60cad56640 1 -- 172.21.15.121:0/1035752346 >> 172.21.15.121:0/1035752346 conn(0x7f60c40a3b50 msgr2=0x7f60c4134750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:25:01.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.464+0000 7f60cad56640 1 -- 172.21.15.121:0/1035752346 shutdown_connections 2024-09-17T18:25:01.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.464+0000 7f60cad56640 1 -- 172.21.15.121:0/1035752346 wait complete. 2024-09-17T18:25:01.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:01.464+0000 7f60cad56640 1 librados: shutdown 2024-09-17T18:25:04.481 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:25:04.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.585+0000 7f43010ef640 10 monclient: get_monmap_and_config 2024-09-17T18:25:04.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.585+0000 7f43010ef640 10 monclient: build_initial_monmap 2024-09-17T18:25:04.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.585+0000 7f43010ef640 10 monclient: monmap: 2024-09-17T18:25:04.589 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:25:04.589 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:04.589 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:25:04.587462+0000 2024-09-17T18:25:04.589 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:25:04.587462+0000 2024-09-17T18:25:04.589 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:25:04.589 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:04.589 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:25:04.589 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:25:04.589 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:25:04.589 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:25:04.589 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:25:04.589 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:04.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.586+0000 7f43010ef640 1 Processor -- start 2024-09-17T18:25:04.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.586+0000 7f43010ef640 1 -- start start 2024-09-17T18:25:04.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.586+0000 7f43010ef640 10 monclient: init 2024-09-17T18:25:04.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.586+0000 7f43010ef640 10 monclient: _reopen_session rank -1 2024-09-17T18:25:04.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.586+0000 7f43010ef640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-17T18:25:04.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.586+0000 7f43010ef640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f42fc0a3a00 0x7f42fc0a3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:04.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.586+0000 7f43010ef640 10 monclient(hunting): picked mon.noname-a con 0x7f42fc0a3a00 addr v2:172.21.15.121:3300/0 2024-09-17T18:25:04.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.586+0000 7f43010ef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f42fc0a4320 addr v1:172.21.15.173:6789/0 2024-09-17T18:25:04.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.586+0000 7f43010ef640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f42fc138f60 0x7f42fc140b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:04.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.586+0000 7f43010ef640 10 monclient(hunting): picked mon.noname-c con 0x7f42fc138f60 addr v2:172.21.15.173:3300/0 2024-09-17T18:25:04.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.586+0000 7f43010ef640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42fc074b80 con 0x7f42fc0a4320 2024-09-17T18:25:04.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.586+0000 7f43010ef640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:04.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.586+0000 7f43010ef640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f42fc09ce70 con 0x7f42fc0a3a00 2024-09-17T18:25:04.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.586+0000 7f43010ef640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:04.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.586+0000 7f43010ef640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f42fc09cb90 con 0x7f42fc138f60 2024-09-17T18:25:04.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.586+0000 7f43010ef640 10 monclient(hunting): _renew_subs 2024-09-17T18:25:04.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.587+0000 7f43010ef640 10 monclient(hunting): authenticate will time out at 1108.304008s 2024-09-17T18:25:04.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.587+0000 7f42fad9d640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f42fc0a3a00 0x7f42fc0a3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:04.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.587+0000 7f42fad9d640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f42fc0a3a00 0x7f42fc0a3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:49554/0 (socket says 172.21.15.121:49554) 2024-09-17T18:25:04.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.587+0000 7f42fad9d640 1 -- 172.21.15.121:0/2553339214 learned_addr learned my addr 172.21.15.121:0/2553339214 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:25:04.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.587+0000 7f42fad9d640 10 monclient(hunting): get_auth_request con 0x7f42fc0a3a00 auth_method 0 2024-09-17T18:25:04.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.587+0000 7f42fad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:04.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.587+0000 7f42fad9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:04.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.587+0000 7f42fad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:04.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.587+0000 7f42fad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:04.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.587+0000 7f42fad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:04.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.587+0000 7f42fad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:04.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.587+0000 7f42fad9d640 10 monclient(hunting): handle_auth_done global_id 44310 payload 274 2024-09-17T18:25:04.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.587+0000 7f42fad9d640 1 -- 172.21.15.121:0/2553339214 >> v2:172.21.15.173:3300/0 conn(0x7f42fc138f60 msgr2=0x7f42fc140b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:25:04.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.587+0000 7f42fad9d640 1 --2- 172.21.15.121:0/2553339214 >> v2:172.21.15.173:3300/0 conn(0x7f42fc138f60 0x7f42fc140b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:04.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.587+0000 7f42fad9d640 1 -- 172.21.15.121:0/2553339214 >> v1:172.21.15.173:6789/0 conn(0x7f42fc0a4320 legacy=0x7f42fc138850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:25:04.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.587+0000 7f42fad9d640 10 monclient: _finish_hunting 0 2024-09-17T18:25:04.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.587+0000 7f42fad9d640 1 monclient: found mon.noname-a 2024-09-17T18:25:04.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.587+0000 7f42fad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:25:04.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.587+0000 7f42fad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:25:04.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.587+0000 7f42fad9d640 1 -- 172.21.15.121:0/2553339214 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42fc0a4730 con 0x7f42fc0a3a00 2024-09-17T18:25:04.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.590+0000 7f42fad9d640 1 -- 172.21.15.121:0/2553339214 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42fc0a3a00 msgr2=0x7f42fc0a3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:04.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.590+0000 7f42fad9d640 1 --2- 172.21.15.121:0/2553339214 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42fc0a3a00 0x7f42fc0a3de0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f42fc06a4b0 tx=0x7f42ec02a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ed1409e7c8e2a94 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:04.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.590+0000 7f42f9d9b640 1 -- 172.21.15.121:0/2553339214 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42ec037070 con 0x7f42fc0a3a00 2024-09-17T18:25:04.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.590+0000 7f42f9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:04.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.590+0000 7f42f9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:25:04.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.590+0000 7f42f9d9b640 10 monclient: dump: 2024-09-17T18:25:04.601 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:04.601 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:04.601 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:04.601 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:04.601 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:04.601 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:04.601 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:04.601 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:04.602 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:04.602 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:04.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.590+0000 7f42f9d9b640 10 monclient: _finish_auth 0 2024-09-17T18:25:04.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.590+0000 7f42f9d9b640 10 monclient: _check_auth_tickets 2024-09-17T18:25:04.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.590+0000 7f42f9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:25:04.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.590+0000 7f42f9d9b640 1 -- 172.21.15.121:0/2553339214 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42ec032640 con 0x7f42fc0a3a00 2024-09-17T18:25:04.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.590+0000 7f43010ef640 5 monclient: authenticate success, global_id 44310 2024-09-17T18:25:04.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.590+0000 7f43010ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:25:04.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.590+0000 7f42f9d9b640 10 monclient: handle_config config(0 keys) 2024-09-17T18:25:04.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.590+0000 7f43010ef640 10 monclient: get_monmap_and_config success 2024-09-17T18:25:04.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.590+0000 7f42f9d9b640 1 -- 172.21.15.121:0/2553339214 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42ec032a10 con 0x7f42fc0a3a00 2024-09-17T18:25:04.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.590+0000 7f42f9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:04.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.590+0000 7f43010ef640 4 set_mon_vals no callback set 2024-09-17T18:25:04.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.590+0000 7f42f9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:25:04.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.590+0000 7f42f9d9b640 10 monclient: dump: 2024-09-17T18:25:04.603 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:04.603 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:04.603 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:04.603 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:04.603 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:04.603 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:04.603 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:04.603 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:04.603 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:04.603 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:04.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.590+0000 7f43010ef640 10 monclient: shutdown 2024-09-17T18:25:04.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.590+0000 7f43010ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:25:04.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.590+0000 7f43010ef640 1 -- 172.21.15.121:0/2553339214 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42fc0a3a00 msgr2=0x7f42fc0a3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:04.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.590+0000 7f43010ef640 1 --2- 172.21.15.121:0/2553339214 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42fc0a3a00 0x7f42fc0a3de0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f42fc06a4b0 tx=0x7f42ec02a0b0 comp rx=0 tx=0).stop 2024-09-17T18:25:04.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.590+0000 7f43010ef640 1 -- 172.21.15.121:0/2553339214 shutdown_connections 2024-09-17T18:25:04.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.590+0000 7f43010ef640 1 --2- 172.21.15.121:0/2553339214 >> v2:172.21.15.173:3300/0 conn(0x7f42fc138f60 0x7f42fc140b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:04.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.590+0000 7f43010ef640 1 --2- 172.21.15.121:0/2553339214 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42fc0a3a00 0x7f42fc0a3de0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:04.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.590+0000 7f43010ef640 1 -- 172.21.15.121:0/2553339214 >> 172.21.15.121:0/2553339214 conn(0x7f42fc0a8620 msgr2=0x7f42fc0a4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:25:04.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.590+0000 7f43010ef640 1 -- 172.21.15.121:0/2553339214 shutdown_connections 2024-09-17T18:25:04.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.590+0000 7f43010ef640 1 -- 172.21.15.121:0/2553339214 wait complete. 2024-09-17T18:25:04.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.591+0000 7f43010ef640 10 monclient: build_initial_monmap 2024-09-17T18:25:04.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.591+0000 7f43010ef640 10 monclient: monmap: 2024-09-17T18:25:04.604 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:25:04.604 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:25:04.604 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:25:04.605 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:25:04.605 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:25:04.605 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:04.605 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:25:04.605 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:25:04.605 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:25:04.605 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:04.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.591+0000 7f43010ef640 1 librados: starting msgr at 2024-09-17T18:25:04.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.591+0000 7f43010ef640 1 librados: starting objecter 2024-09-17T18:25:04.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.592+0000 7f43010ef640 1 Processor -- start 2024-09-17T18:25:04.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.592+0000 7f43010ef640 1 -- start start 2024-09-17T18:25:04.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.592+0000 7f43010ef640 1 librados: setting wanted keys 2024-09-17T18:25:04.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.592+0000 7f43010ef640 1 librados: calling monclient init 2024-09-17T18:25:04.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.592+0000 7f43010ef640 10 monclient: init 2024-09-17T18:25:04.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.592+0000 7f43010ef640 10 monclient: _reopen_session rank -1 2024-09-17T18:25:04.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.592+0000 7f43010ef640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T18:25:04.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.592+0000 7f43010ef640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f42fc0a3a00 0x7f42fc1cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:04.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.592+0000 7f43010ef640 10 monclient(hunting): picked mon.noname-c con 0x7f42fc0a3a00 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:25:04.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.592+0000 7f43010ef640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42fc0a4320 0x7f42fc1ce090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:04.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.592+0000 7f43010ef640 10 monclient(hunting): picked mon.noname-a con 0x7f42fc0a4320 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:25:04.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.592+0000 7f43010ef640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f42fc138f60 0x7f42fc125300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:04.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.592+0000 7f43010ef640 10 monclient(hunting): picked mon.noname-b con 0x7f42fc138f60 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:25:04.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.592+0000 7f43010ef640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:04.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.592+0000 7f43010ef640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f42fc076e70 con 0x7f42fc0a4320 2024-09-17T18:25:04.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.592+0000 7f43010ef640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:04.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.592+0000 7f43010ef640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f42fc076cf0 con 0x7f42fc138f60 2024-09-17T18:25:04.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.592+0000 7f43010ef640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:04.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.592+0000 7f43010ef640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f42fc0770b0 con 0x7f42fc0a3a00 2024-09-17T18:25:04.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.592+0000 7f43010ef640 10 monclient(hunting): _renew_subs 2024-09-17T18:25:04.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.592+0000 7f43010ef640 10 monclient(hunting): authenticate will time out at 1108.309693s 2024-09-17T18:25:04.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.592+0000 7f42fad9d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f42fc0a3a00 0x7f42fc1cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:04.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.592+0000 7f42fad9d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f42fc0a3a00 0x7f42fc1cdb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:33770/0 (socket says 172.21.15.121:33770) 2024-09-17T18:25:04.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.592+0000 7f42fad9d640 1 -- 172.21.15.121:0/1957709825 learned_addr learned my addr 172.21.15.121:0/1957709825 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:25:04.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.592+0000 7f42fa59c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42fc0a4320 0x7f42fc1ce090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:04.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.592+0000 7f42fad9d640 10 monclient(hunting): get_auth_request con 0x7f42fc0a3a00 auth_method 0 2024-09-17T18:25:04.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.592+0000 7f42fad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:04.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.592+0000 7f42fad9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:04.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.592+0000 7f42fad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:04.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.592+0000 7f42fa59c640 10 monclient(hunting): get_auth_request con 0x7f42fc0a4320 auth_method 0 2024-09-17T18:25:04.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.592+0000 7f42fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:04.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.592+0000 7f42fa59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:04.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.592+0000 7f42fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:04.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.592+0000 7f42fa59c640 1 -- 172.21.15.121:0/1957709825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42fc0a4320 msgr2=0x7f42fc1ce090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T18:25:04.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.592+0000 7f42fa59c640 1 -- 172.21.15.121:0/1957709825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42fc0a4320 msgr2=0x7f42fc1ce090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:25:04.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.592+0000 7f42fa59c640 1 --2- 172.21.15.121:0/1957709825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42fc0a4320 0x7f42fc1ce090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:25:04.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.592+0000 7f42fa59c640 1 --2- 172.21.15.121:0/1957709825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42fc0a4320 0x7f42fc1ce090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:25:04.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.592+0000 7f42fad9d640 1 -- 172.21.15.121:0/1957709825 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f42fc0a3a00 msgr2=0x7f42fc1cdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T18:25:04.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.592+0000 7f42fad9d640 1 -- 172.21.15.121:0/1957709825 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f42fc0a3a00 msgr2=0x7f42fc1cdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:25:04.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.592+0000 7f42fad9d640 1 --2- 172.21.15.121:0/1957709825 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f42fc0a3a00 0x7f42fc1cdb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:25:04.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.592+0000 7f42fad9d640 1 --2- 172.21.15.121:0/1957709825 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f42fc0a3a00 0x7f42fc1cdb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:25:04.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.593+0000 7f42fb59e640 1 --2- 172.21.15.121:0/1957709825 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f42fc138f60 0x7f42fc125300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:04.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.593+0000 7f42fb59e640 10 monclient(hunting): get_auth_request con 0x7f42fc138f60 auth_method 0 2024-09-17T18:25:04.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.593+0000 7f42fb59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:04.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.593+0000 7f42fb59e640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:04.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.593+0000 7f42fb59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:04.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.593+0000 7f42fb59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:04.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.593+0000 7f42fb59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:04.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.593+0000 7f42fb59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:04.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.593+0000 7f42fb59e640 10 monclient(hunting): handle_auth_done global_id 24664 payload 824 2024-09-17T18:25:04.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.593+0000 7f42fb59e640 1 -- 172.21.15.121:0/1957709825 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f42fc0a3a00 msgr2=0x7f42fc1cdb50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:25:04.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.593+0000 7f42fb59e640 1 --2- 172.21.15.121:0/1957709825 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f42fc0a3a00 0x7f42fc1cdb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:04.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.593+0000 7f42fb59e640 1 -- 172.21.15.121:0/1957709825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42fc0a4320 msgr2=0x7f42fc1ce090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:25:04.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.593+0000 7f42fb59e640 1 --2- 172.21.15.121:0/1957709825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42fc0a4320 0x7f42fc1ce090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:04.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.593+0000 7f42fb59e640 10 monclient: _finish_hunting 0 2024-09-17T18:25:04.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.593+0000 7f42fb59e640 1 monclient: found mon.noname-b 2024-09-17T18:25:04.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.593+0000 7f42fb59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:25:04.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.593+0000 7f42fb59e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.173:3300/0 2024-09-17T18:25:04.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.593+0000 7f42fb59e640 1 -- 172.21.15.121:0/1957709825 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42fc129090 con 0x7f42fc138f60 2024-09-17T18:25:04.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.595+0000 7f42fb59e640 1 -- 172.21.15.121:0/1957709825 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f42fc138f60 msgr2=0x7f42fc125300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:04.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.595+0000 7f42fb59e640 1 --2- 172.21.15.121:0/1957709825 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f42fc138f60 0x7f42fc125300 secure :-1 s=READY pgs=7365 cs=0 l=1 rev1=1 crypto rx=0x7f42f400be20 tx=0x7f42f400bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:04.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.595+0000 7f42e37fe640 10 client.?.objecter ms_handle_connect 0x7f42fc138f60 2024-09-17T18:25:04.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.595+0000 7f42e37fe640 10 client.?.objecter resend_mon_ops 2024-09-17T18:25:04.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.596+0000 7f42e37fe640 1 -- 172.21.15.121:0/1957709825 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f4019070 con 0x7f42fc138f60 2024-09-17T18:25:04.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.596+0000 7f42e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:04.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.596+0000 7f42e37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:25:04.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.596+0000 7f42e37fe640 10 monclient: dump: 2024-09-17T18:25:04.610 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:04.610 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:04.610 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:04.610 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:04.610 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:04.610 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:04.611 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:04.611 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:04.611 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:04.611 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:04.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.596+0000 7f42e37fe640 10 monclient: _finish_auth 0 2024-09-17T18:25:04.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.596+0000 7f42e37fe640 10 monclient: _check_auth_tickets 2024-09-17T18:25:04.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.596+0000 7f42e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:25:04.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.596+0000 7f42e37fe640 1 -- 172.21.15.121:0/1957709825 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42f4014660 con 0x7f42fc138f60 2024-09-17T18:25:04.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.596+0000 7f43010ef640 5 monclient: authenticate success, global_id 24664 2024-09-17T18:25:04.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.596+0000 7f42e37fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:25:04.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.596+0000 7f42e37fe640 1 -- 172.21.15.121:0/1957709825 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f4014a50 con 0x7f42fc138f60 2024-09-17T18:25:04.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.596+0000 7f42e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:04.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.596+0000 7f42e37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:25:04.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.596+0000 7f42e37fe640 10 monclient: dump: 2024-09-17T18:25:04.611 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:04.612 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:04.612 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:04.612 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:04.612 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:04.612 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:04.612 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:04.612 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:04.612 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:04.612 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:04.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.596+0000 7f43010ef640 10 monclient: _renew_subs 2024-09-17T18:25:04.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.596+0000 7f43010ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:25:04.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.596+0000 7f43010ef640 1 -- 172.21.15.121:0/1957709825 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42fc126d30 con 0x7f42fc138f60 2024-09-17T18:25:04.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.596+0000 7f42f959a640 4 set_mon_vals no callback set 2024-09-17T18:25:04.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.596+0000 7f43010ef640 10 client.24664.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:25:04.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.596+0000 7f43010ef640 10 monclient: _renew_subs 2024-09-17T18:25:04.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.596+0000 7f43010ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:25:04.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.596+0000 7f43010ef640 1 -- 172.21.15.121:0/1957709825 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42fc127890 con 0x7f42fc138f60 2024-09-17T18:25:04.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.596+0000 7f43010ef640 1 librados: init done 2024-09-17T18:25:04.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.597+0000 7f42e37fe640 1 -- 172.21.15.121:0/1957709825 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f42f401c070 con 0x7f42fc138f60 2024-09-17T18:25:04.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.597+0000 7f43010ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:04.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.597+0000 7f43010ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:04.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.597+0000 7f43010ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:25:04.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.597+0000 7f42e37fe640 1 --2- 172.21.15.121:0/1957709825 >> v2:172.21.15.173:6812/2423786037 conn(0x7f42e80401c0 0x7f42e8042680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:04.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.597+0000 7f43010ef640 1 -- 172.21.15.121:0/1957709825 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42fc1393b0 con 0x7f42fc138f60 2024-09-17T18:25:04.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.597+0000 7f42e37fe640 1 -- 172.21.15.121:0/1957709825 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 6205+0+0 (secure 0 0 0) 0x7f42f4053e10 con 0x7f42fc138f60 2024-09-17T18:25:04.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.597+0000 7f42e37fe640 10 client.24664.objecter ms_dispatch 0x7f42fc000b90 osd_map(275..275 src has 1..275) 2024-09-17T18:25:04.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.597+0000 7f42e37fe640 3 client.24664.objecter handle_osd_map got epochs [275,275] > 0 2024-09-17T18:25:04.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.597+0000 7f42e37fe640 3 client.24664.objecter handle_osd_map decoding full epoch 275 2024-09-17T18:25:04.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.597+0000 7f42e37fe640 20 client.24664.objecter dump_active .. 0 homeless 2024-09-17T18:25:04.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.597+0000 7f42fad9d640 1 --2- 172.21.15.121:0/1957709825 >> v2:172.21.15.173:6812/2423786037 conn(0x7f42e80401c0 0x7f42e8042680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:04.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.597+0000 7f42fad9d640 10 monclient: get_auth_request con 0x7f42e80401c0 auth_method 0 2024-09-17T18:25:04.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.597+0000 7f42fad9d640 1 -- 172.21.15.121:0/1957709825 >> v2:172.21.15.173:6812/2423786037 conn(0x7f42e80401c0 msgr2=0x7f42e8042680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T18:25:04.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.597+0000 7f42fad9d640 1 -- 172.21.15.121:0/1957709825 >> v2:172.21.15.173:6812/2423786037 conn(0x7f42e80401c0 msgr2=0x7f42e8042680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:25:04.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.597+0000 7f42fad9d640 1 --2- 172.21.15.121:0/1957709825 >> v2:172.21.15.173:6812/2423786037 conn(0x7f42e80401c0 0x7f42e8042680 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f42ec009630 tx=0x7f42ec02e000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:25:04.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.597+0000 7f42fad9d640 1 --2- 172.21.15.121:0/1957709825 >> v2:172.21.15.173:6812/2423786037 conn(0x7f42e80401c0 0x7f42e8042680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:25:04.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.602+0000 7f42e37fe640 1 -- 172.21.15.121:0/1957709825 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f42f400eb80 con 0x7f42fc138f60 2024-09-17T18:25:04.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.602+0000 7f42e37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:04.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.602+0000 7f42e37fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:25:04.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.798+0000 7f42fad9d640 1 --2- 172.21.15.121:0/1957709825 >> v2:172.21.15.173:6812/2423786037 conn(0x7f42e80401c0 0x7f42e8042680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:04.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.798+0000 7f42fad9d640 10 monclient: get_auth_request con 0x7f42e80401c0 auth_method 0 2024-09-17T18:25:04.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.801+0000 7f42fad9d640 1 --2- 172.21.15.121:0/1957709825 >> v2:172.21.15.173:6812/2423786037 conn(0x7f42e80401c0 0x7f42e8042680 secure :-1 s=READY pgs=3967 cs=0 l=1 rev1=1 crypto rx=0x7f42ec03d000 tx=0x7f42ec008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:04.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.801+0000 7f42e37fe640 10 client.24664.objecter ms_handle_connect 0x7f42e80401c0 2024-09-17T18:25:04.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.853+0000 7f43010ef640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:25:04.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.853+0000 7f43010ef640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:25:04.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.853+0000 7f43010ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.173:3300/0 2024-09-17T18:25:04.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.853+0000 7f43010ef640 1 -- 172.21.15.121:0/1957709825 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f42fc076a60 con 0x7f42fc138f60 2024-09-17T18:25:04.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.854+0000 7f42e37fe640 1 -- 172.21.15.121:0/1957709825 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f42f400edc0 con 0x7f42fc138f60 2024-09-17T18:25:04.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.854+0000 7f42e37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:25:04.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.854+0000 7f42e37fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:25:04.857 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":24,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":98,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:25:04.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.856+0000 7f43010ef640 10 librados: watch_flush enter 2024-09-17T18:25:04.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.856+0000 7f43010ef640 10 librados: watch_flush exit 2024-09-17T18:25:04.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.856+0000 7f43010ef640 20 client.24664.objecter shutdown clearing up homeless session... 2024-09-17T18:25:04.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.856+0000 7f43010ef640 10 client.24664.objecter successfully canceled tick 2024-09-17T18:25:04.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.856+0000 7f43010ef640 1 -- 172.21.15.121:0/1957709825 >> v2:172.21.15.173:6812/2423786037 conn(0x7f42e80401c0 msgr2=0x7f42e8042680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:04.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.856+0000 7f43010ef640 1 --2- 172.21.15.121:0/1957709825 >> v2:172.21.15.173:6812/2423786037 conn(0x7f42e80401c0 0x7f42e8042680 secure :-1 s=READY pgs=3967 cs=0 l=1 rev1=1 crypto rx=0x7f42ec03d000 tx=0x7f42ec008030 comp rx=0 tx=0).stop 2024-09-17T18:25:04.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.856+0000 7f43010ef640 10 monclient: shutdown 2024-09-17T18:25:04.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.856+0000 7f43010ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:25:04.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.856+0000 7f43010ef640 1 -- 172.21.15.121:0/1957709825 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f42fc138f60 msgr2=0x7f42fc125300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:04.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.856+0000 7f43010ef640 1 --2- 172.21.15.121:0/1957709825 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f42fc138f60 0x7f42fc125300 secure :-1 s=READY pgs=7365 cs=0 l=1 rev1=1 crypto rx=0x7f42f400be20 tx=0x7f42f400bf10 comp rx=0 tx=0).stop 2024-09-17T18:25:04.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.856+0000 7f43010ef640 1 -- 172.21.15.121:0/1957709825 shutdown_connections 2024-09-17T18:25:04.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.856+0000 7f43010ef640 1 --2- 172.21.15.121:0/1957709825 >> v2:172.21.15.173:6812/2423786037 conn(0x7f42e80401c0 0x7f42e8042680 unknown :-1 s=CLOSED pgs=3967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:04.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.856+0000 7f43010ef640 1 --2- 172.21.15.121:0/1957709825 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f42fc138f60 0x7f42fc125300 unknown :-1 s=CLOSED pgs=7365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:04.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.856+0000 7f43010ef640 1 --2- 172.21.15.121:0/1957709825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42fc0a4320 0x7f42fc1ce090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:04.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.856+0000 7f43010ef640 1 --2- 172.21.15.121:0/1957709825 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f42fc0a3a00 0x7f42fc1cdb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:04.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.856+0000 7f43010ef640 1 -- 172.21.15.121:0/1957709825 >> 172.21.15.121:0/1957709825 conn(0x7f42fc0a8620 msgr2=0x7f42fc0a09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:25:04.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.856+0000 7f43010ef640 1 -- 172.21.15.121:0/1957709825 shutdown_connections 2024-09-17T18:25:04.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.856+0000 7f43010ef640 1 -- 172.21.15.121:0/1957709825 wait complete. 2024-09-17T18:25:04.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:04.856+0000 7f43010ef640 1 librados: shutdown 2024-09-17T18:25:06.169 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2163:head 2024-09-17T18:25:06.169 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:5d165639:::164:head 2024-09-17T18:25:06.169 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:f43765fc:::165:head 2024-09-17T18:25:06.169 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:b4c720e9:::166:head 2024-09-17T18:25:06.169 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:e694b040:::167:head 2024-09-17T18:25:06.169 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:afa38db2:::168:head 2024-09-17T18:25:06.169 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:77ba9f53:::169:head 2024-09-17T18:25:06.170 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:87495034:::170:head 2024-09-17T18:25:06.170 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:7c96bf0e:::171:head 2024-09-17T18:25:06.170 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:dbe346cc:::172:head 2024-09-17T18:25:06.170 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:e943ec24:::173:head 2024-09-17T18:25:06.170 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:f97a9c0c:::174:head 2024-09-17T18:25:06.170 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:6f26e74d:::175:head 2024-09-17T18:25:06.170 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:4f95e106:::176:head 2024-09-17T18:25:06.170 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:0e6f2f8f:::177:head 2024-09-17T18:25:06.170 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:05db05f1:::178:head 2024-09-17T18:25:06.170 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:38a78d66:::179:head 2024-09-17T18:25:06.170 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:d095610b:::180:head 2024-09-17T18:25:06.170 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:a1a9d709:::181:head 2024-09-17T18:25:06.170 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:1e5d39db:::182:head 2024-09-17T18:25:06.170 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:f7df4fb9:::183:head 2024-09-17T18:25:06.171 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:03a7f161:::184:head 2024-09-17T18:25:06.171 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:ba70721e:::185:head 2024-09-17T18:25:06.171 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:28e5662d:::186:head 2024-09-17T18:25:06.171 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:973d52de:::187:head 2024-09-17T18:25:06.171 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:4303eb1c:::188:head 2024-09-17T18:25:06.171 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:b990b48e:::189:head 2024-09-17T18:25:06.171 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:29b8165b:::190:head 2024-09-17T18:25:06.171 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:3547f197:::191:head 2024-09-17T18:25:06.171 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:7e260936:::192:head 2024-09-17T18:25:06.171 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:1abec7b1:::193:head 2024-09-17T18:25:06.171 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:10fdda93:::194:head 2024-09-17T18:25:06.171 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:15817eea:::195:head 2024-09-17T18:25:06.171 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:770bab57:::196:head 2024-09-17T18:25:06.171 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:ed9e13e7:::197:head 2024-09-17T18:25:06.171 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:71471a8f:::198:head 2024-09-17T18:25:06.171 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: checking for 269:10fb1d02:::199:head 2024-09-17T18:25:06.172 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (7984 ms) 2024-09-17T18:25:06.172 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-09-17T18:25:06.172 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: got ls 1726597461,0 2024-09-17T18:25:06.172 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: first is 1726597461 2024-09-17T18:25:06.172 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: got ls 1726597461,0 2024-09-17T18:25:06.172 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: got ls 1726597461,0 2024-09-17T18:25:06.172 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: got ls 1726597461,1726597463,1726597464,0 2024-09-17T18:25:06.172 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: got ls 1726597461,1726597463,1726597464,0 2024-09-17T18:25:06.172 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: got ls 1726597461,1726597463,1726597464,0 2024-09-17T18:25:06.172 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: got ls 1726597461,1726597463,1726597464,1726597466,1726597467,0 2024-09-17T18:25:06.172 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: got ls 1726597461,1726597463,1726597464,1726597466,1726597467,0 2024-09-17T18:25:06.172 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: got ls 1726597461,1726597463,1726597464,1726597466,1726597467,0 2024-09-17T18:25:06.172 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: got ls 1726597461,1726597463,1726597464,1726597466,1726597467,1726597469,1726597470,0 2024-09-17T18:25:06.172 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: got ls 1726597461,1726597463,1726597464,1726597466,1726597467,1726597469,1726597470,0 2024-09-17T18:25:06.173 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: got ls 1726597461,1726597463,1726597464,1726597466,1726597467,1726597469,1726597470,0 2024-09-17T18:25:06.173 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: got ls 1726597464,1726597466,1726597467,1726597469,1726597470,1726597472,1726597473,0 2024-09-17T18:25:06.173 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: first now 1726597464, trimmed 2024-09-17T18:25:06.173 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (21201 ms) 2024-09-17T18:25:06.173 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-09-17T18:25:06.173 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: foo0 2024-09-17T18:25:06.173 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-09-17T18:25:06.173 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (13833 ms) 2024-09-17T18:25:06.173 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-09-17T18:25:07.871 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:25:07.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.976+0000 7fdd545f5640 10 monclient: get_monmap_and_config 2024-09-17T18:25:07.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.976+0000 7fdd545f5640 10 monclient: build_initial_monmap 2024-09-17T18:25:07.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.976+0000 7fdd545f5640 10 monclient: monmap: 2024-09-17T18:25:07.979 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:25:07.979 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:07.979 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:25:07.977777+0000 2024-09-17T18:25:07.979 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:25:07.977777+0000 2024-09-17T18:25:07.979 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:25:07.979 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:07.979 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:25:07.979 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:25:07.980 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:25:07.980 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:25:07.980 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:25:07.980 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:07.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.976+0000 7fdd545f5640 1 Processor -- start 2024-09-17T18:25:07.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.976+0000 7fdd545f5640 1 -- start start 2024-09-17T18:25:07.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.976+0000 7fdd545f5640 10 monclient: init 2024-09-17T18:25:07.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.977+0000 7fdd545f5640 10 monclient: _reopen_session rank -1 2024-09-17T18:25:07.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.977+0000 7fdd545f5640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-17T18:25:07.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.977+0000 7fdd545f5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd4c0a5c00 0x7fdd4c09d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:07.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.977+0000 7fdd545f5640 10 monclient(hunting): picked mon.noname-b con 0x7fdd4c0a5c00 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:25:07.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.977+0000 7fdd545f5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdd4c0a5fe0 addr v1:172.21.15.173:6789/0 2024-09-17T18:25:07.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.977+0000 7fdd545f5640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fdd4c13dc00 0x7fdd4c13fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:07.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.977+0000 7fdd545f5640 10 monclient(hunting): picked mon.noname-c con 0x7fdd4c13dc00 addr v2:172.21.15.173:3300/0 2024-09-17T18:25:07.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.977+0000 7fdd545f5640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd4c0a8a80 con 0x7fdd4c0a5fe0 2024-09-17T18:25:07.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.977+0000 7fdd545f5640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:07.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.977+0000 7fdd545f5640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fdd4c0a7110 con 0x7fdd4c13dc00 2024-09-17T18:25:07.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.977+0000 7fdd545f5640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:07.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.977+0000 7fdd545f5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fdd4c09de00 con 0x7fdd4c0a5c00 2024-09-17T18:25:07.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.977+0000 7fdd545f5640 10 monclient(hunting): _renew_subs 2024-09-17T18:25:07.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.977+0000 7fdd545f5640 10 monclient(hunting): authenticate will time out at 1111.694529s 2024-09-17T18:25:07.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.977+0000 7fdd52391640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd4c0a5c00 0x7fdd4c09d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:07.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.977+0000 7fdd52391640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd4c0a5c00 0x7fdd4c09d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:33780/0 (socket says 172.21.15.121:33780) 2024-09-17T18:25:07.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.977+0000 7fdd52391640 1 -- 172.21.15.121:0/3270721528 learned_addr learned my addr 172.21.15.121:0/3270721528 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:25:07.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.977+0000 7fdd52391640 10 monclient(hunting): get_auth_request con 0x7fdd4c0a5c00 auth_method 0 2024-09-17T18:25:07.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.977+0000 7fdd52391640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:07.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.977+0000 7fdd52391640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:07.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.977+0000 7fdd52391640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:07.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.977+0000 7fdd52391640 1 -- 172.21.15.121:0/3270721528 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd4c0a5c00 msgr2=0x7fdd4c09d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:25:07.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.977+0000 7fdd52391640 1 -- 172.21.15.121:0/3270721528 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd4c0a5c00 msgr2=0x7fdd4c09d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:25:07.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.977+0000 7fdd52391640 1 --2- 172.21.15.121:0/3270721528 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd4c0a5c00 0x7fdd4c09d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:25:07.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.977+0000 7fdd52391640 1 --2- 172.21.15.121:0/3270721528 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd4c0a5c00 0x7fdd4c09d8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:25:07.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.977+0000 7fdd52b92640 1 --2- 172.21.15.121:0/3270721528 >> v2:172.21.15.173:3300/0 conn(0x7fdd4c13dc00 0x7fdd4c13fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:07.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.977+0000 7fdd52b92640 10 monclient(hunting): get_auth_request con 0x7fdd4c13dc00 auth_method 0 2024-09-17T18:25:07.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.977+0000 7fdd52b92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:07.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.977+0000 7fdd52b92640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:07.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.977+0000 7fdd52b92640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:07.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.977+0000 7fdd52b92640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:07.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.977+0000 7fdd52b92640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:07.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.977+0000 7fdd52b92640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:07.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.978+0000 7fdd52b92640 10 monclient(hunting): handle_auth_done global_id 24670 payload 274 2024-09-17T18:25:07.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.978+0000 7fdd52b92640 1 -- 172.21.15.121:0/3270721528 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd4c0a5c00 msgr2=0x7fdd4c09d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:25:07.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.978+0000 7fdd52b92640 1 --2- 172.21.15.121:0/3270721528 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd4c0a5c00 0x7fdd4c09d8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:07.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.978+0000 7fdd52b92640 1 -- 172.21.15.121:0/3270721528 >> v1:172.21.15.173:6789/0 conn(0x7fdd4c0a5fe0 legacy=0x7fdd4c0a63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:25:07.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.978+0000 7fdd52b92640 10 monclient: _finish_hunting 0 2024-09-17T18:25:07.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.978+0000 7fdd52b92640 1 monclient: found mon.noname-c 2024-09-17T18:25:07.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.978+0000 7fdd52b92640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:25:07.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.978+0000 7fdd52b92640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.173:3300/0 2024-09-17T18:25:07.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.978+0000 7fdd52b92640 1 -- 172.21.15.121:0/3270721528 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd4c0a74d0 con 0x7fdd4c13dc00 2024-09-17T18:25:07.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.981+0000 7fdd52b92640 1 -- 172.21.15.121:0/3270721528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd4c13dc00 msgr2=0x7fdd4c13fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:07.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.981+0000 7fdd52b92640 1 --2- 172.21.15.121:0/3270721528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd4c13dc00 0x7fdd4c13fff0 secure :-1 s=READY pgs=7369 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c09dfd0 tx=0x7fdd3c02a480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5645997c0daa30ef server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:07.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.981+0000 7fdd5138f640 1 -- 172.21.15.121:0/3270721528 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c007950 con 0x7fdd4c13dc00 2024-09-17T18:25:07.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.981+0000 7fdd5138f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:07.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.981+0000 7fdd5138f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:25:07.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.981+0000 7fdd5138f640 10 monclient: dump: 2024-09-17T18:25:07.987 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:07.988 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:07.988 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:07.988 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:07.988 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:07.988 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:07.988 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:07.988 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:07.988 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:07.988 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:07.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.981+0000 7fdd5138f640 10 monclient: _finish_auth 0 2024-09-17T18:25:07.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.981+0000 7fdd5138f640 10 monclient: _check_auth_tickets 2024-09-17T18:25:07.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.981+0000 7fdd5138f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:25:07.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.981+0000 7fdd5138f640 1 -- 172.21.15.121:0/3270721528 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd3c007af0 con 0x7fdd4c13dc00 2024-09-17T18:25:07.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.981+0000 7fdd545f5640 5 monclient: authenticate success, global_id 24670 2024-09-17T18:25:07.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.981+0000 7fdd545f5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:25:07.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.981+0000 7fdd5138f640 10 monclient: handle_config config(0 keys) 2024-09-17T18:25:07.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.981+0000 7fdd545f5640 10 monclient: get_monmap_and_config success 2024-09-17T18:25:07.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.981+0000 7fdd5138f640 1 -- 172.21.15.121:0/3270721528 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c031400 con 0x7fdd4c13dc00 2024-09-17T18:25:07.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.981+0000 7fdd545f5640 4 set_mon_vals no callback set 2024-09-17T18:25:07.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.981+0000 7fdd5138f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:07.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.981+0000 7fdd5138f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:25:07.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.981+0000 7fdd5138f640 10 monclient: dump: 2024-09-17T18:25:07.989 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:07.989 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:07.989 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:07.989 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:07.989 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:07.989 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:07.989 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:07.990 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:07.990 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:07.990 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:07.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.981+0000 7fdd545f5640 10 monclient: shutdown 2024-09-17T18:25:07.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.981+0000 7fdd545f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:25:07.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.981+0000 7fdd545f5640 1 -- 172.21.15.121:0/3270721528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd4c13dc00 msgr2=0x7fdd4c13fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:07.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.981+0000 7fdd545f5640 1 --2- 172.21.15.121:0/3270721528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd4c13dc00 0x7fdd4c13fff0 secure :-1 s=READY pgs=7369 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c09dfd0 tx=0x7fdd3c02a480 comp rx=0 tx=0).stop 2024-09-17T18:25:07.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.981+0000 7fdd545f5640 1 -- 172.21.15.121:0/3270721528 shutdown_connections 2024-09-17T18:25:07.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.981+0000 7fdd545f5640 1 --2- 172.21.15.121:0/3270721528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd4c13dc00 0x7fdd4c13fff0 unknown :-1 s=CLOSED pgs=7369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:07.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.981+0000 7fdd545f5640 1 --2- 172.21.15.121:0/3270721528 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd4c0a5c00 0x7fdd4c09d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:07.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.981+0000 7fdd545f5640 1 -- 172.21.15.121:0/3270721528 >> 172.21.15.121:0/3270721528 conn(0x7fdd4c0ab9b0 msgr2=0x7fdd4c0a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:25:07.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.981+0000 7fdd545f5640 1 -- 172.21.15.121:0/3270721528 shutdown_connections 2024-09-17T18:25:07.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.981+0000 7fdd545f5640 1 -- 172.21.15.121:0/3270721528 wait complete. 2024-09-17T18:25:07.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.982+0000 7fdd545f5640 10 monclient: build_initial_monmap 2024-09-17T18:25:07.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.982+0000 7fdd545f5640 10 monclient: monmap: 2024-09-17T18:25:07.996 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:25:07.996 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:25:07.996 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:25:07.996 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:25:07.996 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:25:07.997 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:07.997 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:25:07.997 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:25:07.997 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:25:07.997 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:07.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.982+0000 7fdd545f5640 1 librados: starting msgr at 2024-09-17T18:25:07.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.982+0000 7fdd545f5640 1 librados: starting objecter 2024-09-17T18:25:07.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.982+0000 7fdd545f5640 1 Processor -- start 2024-09-17T18:25:07.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.982+0000 7fdd545f5640 1 -- start start 2024-09-17T18:25:07.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.982+0000 7fdd545f5640 1 librados: setting wanted keys 2024-09-17T18:25:07.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.982+0000 7fdd545f5640 1 librados: calling monclient init 2024-09-17T18:25:07.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.982+0000 7fdd545f5640 10 monclient: init 2024-09-17T18:25:07.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd545f5640 10 monclient: _reopen_session rank -1 2024-09-17T18:25:07.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd545f5640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T18:25:07.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd545f5640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd4c0a5c00 0x7fdd4c138250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:07.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd545f5640 10 monclient(hunting): picked mon.noname-b con 0x7fdd4c0a5c00 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:25:07.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd545f5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd4c0a5fe0 0x7fdd4c138790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:07.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd545f5640 10 monclient(hunting): picked mon.noname-c con 0x7fdd4c0a5fe0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:25:07.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd545f5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd4c13dc00 0x7fdd4c13b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:07.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd545f5640 10 monclient(hunting): picked mon.noname-a con 0x7fdd4c13dc00 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:25:07.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd545f5640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:07.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd545f5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fdd4c070550 con 0x7fdd4c13dc00 2024-09-17T18:25:07.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd545f5640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:07.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd545f5640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdd4c0703d0 con 0x7fdd4c0a5c00 2024-09-17T18:25:07.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd545f5640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:07.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd545f5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fdd4c140530 con 0x7fdd4c0a5fe0 2024-09-17T18:25:07.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd545f5640 10 monclient(hunting): _renew_subs 2024-09-17T18:25:07.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd545f5640 10 monclient(hunting): authenticate will time out at 1111.700490s 2024-09-17T18:25:07.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd51b90640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd4c0a5fe0 0x7fdd4c138790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:07.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd51b90640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd4c0a5fe0 0x7fdd4c138790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:33784/0 (socket says 172.21.15.121:33784) 2024-09-17T18:25:07.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd52b92640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd4c13dc00 0x7fdd4c13b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:07.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd51b90640 1 -- 172.21.15.121:0/4144275825 learned_addr learned my addr 172.21.15.121:0/4144275825 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:25:07.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd51b90640 10 monclient(hunting): get_auth_request con 0x7fdd4c0a5fe0 auth_method 0 2024-09-17T18:25:07.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd51b90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:07.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd51b90640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:07.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd51b90640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:07.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd52b92640 10 monclient(hunting): get_auth_request con 0x7fdd4c13dc00 auth_method 0 2024-09-17T18:25:07.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd52b92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:07.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd52b92640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:07.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd52b92640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:08.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd51b90640 1 -- 172.21.15.121:0/4144275825 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd4c0a5fe0 msgr2=0x7fdd4c138790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T18:25:08.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd51b90640 1 -- 172.21.15.121:0/4144275825 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd4c0a5fe0 msgr2=0x7fdd4c138790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:25:08.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd51b90640 1 --2- 172.21.15.121:0/4144275825 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd4c0a5fe0 0x7fdd4c138790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:25:08.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd52b92640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:08.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd52b92640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:08.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd51b90640 1 --2- 172.21.15.121:0/4144275825 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd4c0a5fe0 0x7fdd4c138790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:25:08.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd52b92640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:08.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd52391640 1 --2- 172.21.15.121:0/4144275825 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd4c0a5c00 0x7fdd4c138250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:08.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd52391640 10 monclient(hunting): get_auth_request con 0x7fdd4c0a5c00 auth_method 0 2024-09-17T18:25:08.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd52391640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:08.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd52391640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:08.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd52391640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:08.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd52391640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:08.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd52391640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:08.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd52391640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:08.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd52b92640 10 monclient(hunting): handle_auth_done global_id 44319 payload 824 2024-09-17T18:25:08.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd52b92640 1 -- 172.21.15.121:0/4144275825 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd4c0a5fe0 msgr2=0x7fdd4c138790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:25:08.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd52b92640 1 --2- 172.21.15.121:0/4144275825 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd4c0a5fe0 0x7fdd4c138790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:08.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd52b92640 1 -- 172.21.15.121:0/4144275825 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd4c0a5c00 msgr2=0x7fdd4c138250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:08.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd52b92640 1 --2- 172.21.15.121:0/4144275825 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd4c0a5c00 0x7fdd4c138250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:08.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd52b92640 10 monclient: _finish_hunting 0 2024-09-17T18:25:08.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd52b92640 1 monclient: found mon.noname-a 2024-09-17T18:25:08.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd52b92640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:25:08.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd52b92640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:25:08.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.983+0000 7fdd52b92640 1 -- 172.21.15.121:0/4144275825 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd4c122c90 con 0x7fdd4c13dc00 2024-09-17T18:25:08.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.986+0000 7fdd52b92640 1 -- 172.21.15.121:0/4144275825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd4c13dc00 msgr2=0x7fdd4c13b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:08.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.986+0000 7fdd52b92640 1 --2- 172.21.15.121:0/4144275825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd4c13dc00 0x7fdd4c13b8b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c031000 tx=0x7fdd3c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:08.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.986+0000 7fdd4affd640 10 client.?.objecter ms_handle_connect 0x7fdd4c13dc00 2024-09-17T18:25:08.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.986+0000 7fdd4affd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:25:08.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.986+0000 7fdd4affd640 1 -- 172.21.15.121:0/4144275825 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c040070 con 0x7fdd4c13dc00 2024-09-17T18:25:08.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.986+0000 7fdd4affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:08.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.986+0000 7fdd4affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:25:08.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.986+0000 7fdd4affd640 10 monclient: dump: 2024-09-17T18:25:08.002 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:08.002 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:08.002 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:08.002 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:08.002 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:08.002 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:08.003 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:08.003 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:08.003 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:08.003 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:08.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.986+0000 7fdd4affd640 10 monclient: _finish_auth 0 2024-09-17T18:25:08.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.986+0000 7fdd4affd640 10 monclient: _check_auth_tickets 2024-09-17T18:25:08.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.986+0000 7fdd4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:25:08.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.986+0000 7fdd545f5640 5 monclient: authenticate success, global_id 44319 2024-09-17T18:25:08.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.986+0000 7fdd545f5640 10 monclient: _renew_subs 2024-09-17T18:25:08.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.986+0000 7fdd545f5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:08.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.986+0000 7fdd4affd640 1 -- 172.21.15.121:0/4144275825 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd3c045660 con 0x7fdd4c13dc00 2024-09-17T18:25:08.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.986+0000 7fdd545f5640 1 -- 172.21.15.121:0/4144275825 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd4c122f70 con 0x7fdd4c13dc00 2024-09-17T18:25:08.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.986+0000 7fdd4affd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:25:08.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.986+0000 7fdd4affd640 1 -- 172.21.15.121:0/4144275825 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c0314c0 con 0x7fdd4c13dc00 2024-09-17T18:25:08.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.986+0000 7fdd4affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:08.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.986+0000 7fdd4affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:25:08.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.986+0000 7fdd4affd640 10 monclient: dump: 2024-09-17T18:25:08.004 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:08.004 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:08.004 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:08.004 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:08.004 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:08.004 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:08.004 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:08.004 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:08.004 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:08.004 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:08.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.986+0000 7fdd545f5640 10 client.44319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:25:08.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.986+0000 7fdd545f5640 10 monclient: _renew_subs 2024-09-17T18:25:08.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.986+0000 7fdd50b8e640 4 set_mon_vals no callback set 2024-09-17T18:25:08.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.986+0000 7fdd545f5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:08.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.986+0000 7fdd545f5640 1 -- 172.21.15.121:0/4144275825 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd4c1229b0 con 0x7fdd4c13dc00 2024-09-17T18:25:08.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.986+0000 7fdd545f5640 1 librados: init done 2024-09-17T18:25:08.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.987+0000 7fdd545f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:08.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.987+0000 7fdd545f5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:08.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.987+0000 7fdd545f5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:08.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.987+0000 7fdd545f5640 1 -- 172.21.15.121:0/4144275825 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd4c09d8c0 con 0x7fdd4c13dc00 2024-09-17T18:25:08.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.987+0000 7fdd4affd640 1 -- 172.21.15.121:0/4144275825 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fdd3c039040 con 0x7fdd4c13dc00 2024-09-17T18:25:08.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.987+0000 7fdd4affd640 1 --2- 172.21.15.121:0/4144275825 >> v2:172.21.15.173:6812/2423786037 conn(0x7fdd38040170 0x7fdd38042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:08.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.987+0000 7fdd4affd640 1 -- 172.21.15.121:0/4144275825 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(278..278 src has 1..278) ==== 5329+0+0 (secure 0 0 0) 0x7fdd3c074fd0 con 0x7fdd4c13dc00 2024-09-17T18:25:08.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.987+0000 7fdd4affd640 10 client.44319.objecter ms_dispatch 0x7fdd4c000b90 osd_map(278..278 src has 1..278) 2024-09-17T18:25:08.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.987+0000 7fdd4affd640 3 client.44319.objecter handle_osd_map got epochs [278,278] > 0 2024-09-17T18:25:08.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.987+0000 7fdd4affd640 3 client.44319.objecter handle_osd_map decoding full epoch 278 2024-09-17T18:25:08.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.987+0000 7fdd4affd640 20 client.44319.objecter dump_active .. 0 homeless 2024-09-17T18:25:08.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.988+0000 7fdd52391640 1 --2- 172.21.15.121:0/4144275825 >> v2:172.21.15.173:6812/2423786037 conn(0x7fdd38040170 0x7fdd38042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:08.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.988+0000 7fdd52391640 10 monclient: get_auth_request con 0x7fdd38040170 auth_method 0 2024-09-17T18:25:08.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.990+0000 7fdd4affd640 10 client.44319.objecter ms_handle_connect 0x7fdd38040170 2024-09-17T18:25:08.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.990+0000 7fdd52391640 1 --2- 172.21.15.121:0/4144275825 >> v2:172.21.15.173:6812/2423786037 conn(0x7fdd38040170 0x7fdd38042630 secure :-1 s=READY pgs=3969 cs=0 l=1 rev1=1 crypto rx=0x7fdd44009100 tx=0x7fdd44008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:08.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.992+0000 7fdd4affd640 1 -- 172.21.15.121:0/4144275825 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fdd3c03c420 con 0x7fdd4c13dc00 2024-09-17T18:25:08.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.992+0000 7fdd4affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:08.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:07.992+0000 7fdd4affd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:25:08.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:08.255+0000 7fdd545f5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:25:08.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:08.255+0000 7fdd545f5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:25:08.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:08.255+0000 7fdd545f5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:08.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:08.255+0000 7fdd545f5640 1 -- 172.21.15.121:0/4144275825 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fdd4c06cd40 con 0x7fdd4c13dc00 2024-09-17T18:25:08.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:08.255+0000 7fdd4affd640 1 -- 172.21.15.121:0/4144275825 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1564 (secure 0 0 0) 0x7fdd3c045a50 con 0x7fdd4c13dc00 2024-09-17T18:25:08.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:08.255+0000 7fdd4affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:25:08.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:08.255+0000 7fdd4affd640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:25:08.259 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":24,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":101,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:25:08.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:08.258+0000 7fdd545f5640 10 librados: watch_flush enter 2024-09-17T18:25:08.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:08.258+0000 7fdd545f5640 10 librados: watch_flush exit 2024-09-17T18:25:08.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:08.258+0000 7fdd545f5640 20 client.44319.objecter shutdown clearing up homeless session... 2024-09-17T18:25:08.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:08.258+0000 7fdd545f5640 10 client.44319.objecter successfully canceled tick 2024-09-17T18:25:08.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:08.258+0000 7fdd545f5640 1 -- 172.21.15.121:0/4144275825 >> v2:172.21.15.173:6812/2423786037 conn(0x7fdd38040170 msgr2=0x7fdd38042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:08.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:08.258+0000 7fdd545f5640 1 --2- 172.21.15.121:0/4144275825 >> v2:172.21.15.173:6812/2423786037 conn(0x7fdd38040170 0x7fdd38042630 secure :-1 s=READY pgs=3969 cs=0 l=1 rev1=1 crypto rx=0x7fdd44009100 tx=0x7fdd44008040 comp rx=0 tx=0).stop 2024-09-17T18:25:08.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:08.258+0000 7fdd545f5640 10 monclient: shutdown 2024-09-17T18:25:08.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:08.258+0000 7fdd545f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:25:08.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:08.258+0000 7fdd545f5640 1 -- 172.21.15.121:0/4144275825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd4c13dc00 msgr2=0x7fdd4c13b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:08.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:08.258+0000 7fdd545f5640 1 --2- 172.21.15.121:0/4144275825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd4c13dc00 0x7fdd4c13b8b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c031000 tx=0x7fdd3c004a40 comp rx=0 tx=0).stop 2024-09-17T18:25:08.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:08.258+0000 7fdd545f5640 1 -- 172.21.15.121:0/4144275825 shutdown_connections 2024-09-17T18:25:08.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:08.258+0000 7fdd545f5640 1 --2- 172.21.15.121:0/4144275825 >> v2:172.21.15.173:6812/2423786037 conn(0x7fdd38040170 0x7fdd38042630 unknown :-1 s=CLOSED pgs=3969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:08.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:08.258+0000 7fdd545f5640 1 --2- 172.21.15.121:0/4144275825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd4c13dc00 0x7fdd4c13b8b0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:08.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:08.258+0000 7fdd545f5640 1 --2- 172.21.15.121:0/4144275825 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd4c0a5fe0 0x7fdd4c138790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:08.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:08.258+0000 7fdd545f5640 1 --2- 172.21.15.121:0/4144275825 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd4c0a5c00 0x7fdd4c138250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:08.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:08.258+0000 7fdd545f5640 1 -- 172.21.15.121:0/4144275825 >> 172.21.15.121:0/4144275825 conn(0x7fdd4c0ab9b0 msgr2=0x7fdd4c0a9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:25:08.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:08.258+0000 7fdd545f5640 1 -- 172.21.15.121:0/4144275825 shutdown_connections 2024-09-17T18:25:08.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:08.258+0000 7fdd545f5640 1 -- 172.21.15.121:0/4144275825 wait complete. 2024-09-17T18:25:08.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:08.258+0000 7fdd545f5640 1 librados: shutdown 2024-09-17T18:25:11.274 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:25:11.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.380+0000 7f90c8e47640 10 monclient: get_monmap_and_config 2024-09-17T18:25:11.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.380+0000 7f90c8e47640 10 monclient: build_initial_monmap 2024-09-17T18:25:11.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.380+0000 7f90c8e47640 10 monclient: monmap: 2024-09-17T18:25:11.383 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:25:11.383 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:11.383 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:25:11.381697+0000 2024-09-17T18:25:11.383 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:25:11.381697+0000 2024-09-17T18:25:11.383 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:25:11.383 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:11.384 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:25:11.384 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:25:11.384 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:25:11.384 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:25:11.384 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:25:11.384 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:11.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.380+0000 7f90c8e47640 1 Processor -- start 2024-09-17T18:25:11.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.380+0000 7f90c8e47640 1 -- start start 2024-09-17T18:25:11.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.380+0000 7f90c8e47640 10 monclient: init 2024-09-17T18:25:11.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.381+0000 7f90c8e47640 10 monclient: _reopen_session rank -1 2024-09-17T18:25:11.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.381+0000 7f90c8e47640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-17T18:25:11.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.381+0000 7f90c8e47640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f90c40a5c00 addr v1:172.21.15.121:6789/0 2024-09-17T18:25:11.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.381+0000 7f90c8e47640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f90c40a5fe0 0x7f90c40a63f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:11.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.381+0000 7f90c8e47640 10 monclient(hunting): picked mon.noname-a con 0x7f90c40a5fe0 addr v2:172.21.15.121:3300/0 2024-09-17T18:25:11.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.381+0000 7f90c8e47640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f90c413dc00 0x7f90c413fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:11.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.381+0000 7f90c8e47640 10 monclient(hunting): picked mon.noname-c con 0x7f90c413dc00 addr v2:172.21.15.173:3300/0 2024-09-17T18:25:11.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.381+0000 7f90c8e47640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90c40a8a80 con 0x7f90c40a5c00 2024-09-17T18:25:11.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.381+0000 7f90c8e47640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:11.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.381+0000 7f90c8e47640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f90c40a74d0 con 0x7f90c40a5fe0 2024-09-17T18:25:11.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.381+0000 7f90c8e47640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:11.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.381+0000 7f90c8e47640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f90c409cb90 con 0x7f90c413dc00 2024-09-17T18:25:11.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.381+0000 7f90c8e47640 10 monclient(hunting): _renew_subs 2024-09-17T18:25:11.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.381+0000 7f90c8e47640 10 monclient(hunting): authenticate will time out at 1115.098477s 2024-09-17T18:25:11.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.381+0000 7f90c1d9b640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f90c40a5fe0 0x7f90c40a63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:11.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.381+0000 7f90c259c640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f90c40a5c00 0x7f90c409d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:38340/0 (socket says 172.21.15.121:38340) 2024-09-17T18:25:11.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.381+0000 7f90c259c640 1 -- 172.21.15.121:0/1855061568 learned_addr learned my addr 172.21.15.121:0/1855061568 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:25:11.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.381+0000 7f90c1d9b640 10 monclient(hunting): get_auth_request con 0x7f90c40a5fe0 auth_method 0 2024-09-17T18:25:11.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.381+0000 7f90c1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:11.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.381+0000 7f90c1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:11.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.381+0000 7f90c1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:11.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.381+0000 7f90c1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:11.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.381+0000 7f90c1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:11.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.381+0000 7f90c1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:11.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.381+0000 7f90c2d9d640 1 --2- 172.21.15.121:0/1855061568 >> v2:172.21.15.173:3300/0 conn(0x7f90c413dc00 0x7f90c413fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:11.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.381+0000 7f90c2d9d640 10 monclient(hunting): get_auth_request con 0x7f90c413dc00 auth_method 0 2024-09-17T18:25:11.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.381+0000 7f90c2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:11.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.381+0000 7f90c2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:11.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.381+0000 7f90c2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:11.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.381+0000 7f90c1d9b640 10 monclient(hunting): handle_auth_done global_id 44322 payload 274 2024-09-17T18:25:11.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.381+0000 7f90c1d9b640 1 -- 172.21.15.121:0/1855061568 >> v2:172.21.15.173:3300/0 conn(0x7f90c413dc00 msgr2=0x7f90c413fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:25:11.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.381+0000 7f90c1d9b640 1 --2- 172.21.15.121:0/1855061568 >> v2:172.21.15.173:3300/0 conn(0x7f90c413dc00 0x7f90c413fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:11.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.381+0000 7f90c1d9b640 1 -- 172.21.15.121:0/1855061568 >> v1:172.21.15.121:6789/0 conn(0x7f90c40a5c00 legacy=0x7f90c409d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:25:11.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.381+0000 7f90c1d9b640 10 monclient: _finish_hunting 0 2024-09-17T18:25:11.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.381+0000 7f90c1d9b640 1 monclient: found mon.noname-a 2024-09-17T18:25:11.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.381+0000 7f90c1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:25:11.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.381+0000 7f90c1d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:25:11.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.381+0000 7f90c1d9b640 1 -- 172.21.15.121:0/1855061568 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90c409ce70 con 0x7f90c40a5fe0 2024-09-17T18:25:11.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.384+0000 7f90c1d9b640 1 -- 172.21.15.121:0/1855061568 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90c40a5fe0 msgr2=0x7f90c40a63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:11.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.384+0000 7f90c1d9b640 1 --2- 172.21.15.121:0/1855061568 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90c40a5fe0 0x7f90c40a63f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f90c409d210 tx=0x7f90b802a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49a1166ab70c180 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:11.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.384+0000 7f90c159a640 1 -- 172.21.15.121:0/1855061568 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b8037070 con 0x7f90c40a5fe0 2024-09-17T18:25:11.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.384+0000 7f90c159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:11.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.385+0000 7f90c159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:25:11.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.385+0000 7f90c159a640 10 monclient: dump: 2024-09-17T18:25:11.397 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:11.397 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:11.397 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:11.397 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:11.397 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:11.398 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:11.398 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:11.398 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:11.398 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:11.398 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:11.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.385+0000 7f90c159a640 10 monclient: _finish_auth 0 2024-09-17T18:25:11.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.385+0000 7f90c159a640 10 monclient: _check_auth_tickets 2024-09-17T18:25:11.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.385+0000 7f90c159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:25:11.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.385+0000 7f90c159a640 1 -- 172.21.15.121:0/1855061568 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90b8032440 con 0x7f90c40a5fe0 2024-09-17T18:25:11.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.385+0000 7f90c8e47640 5 monclient: authenticate success, global_id 44322 2024-09-17T18:25:11.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.385+0000 7f90c8e47640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:25:11.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.385+0000 7f90c159a640 10 monclient: handle_config config(0 keys) 2024-09-17T18:25:11.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.385+0000 7f90c8e47640 10 monclient: get_monmap_and_config success 2024-09-17T18:25:11.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.385+0000 7f90c159a640 1 -- 172.21.15.121:0/1855061568 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b8032810 con 0x7f90c40a5fe0 2024-09-17T18:25:11.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.385+0000 7f90c8e47640 4 set_mon_vals no callback set 2024-09-17T18:25:11.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.385+0000 7f90c159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:11.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.385+0000 7f90c159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:25:11.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.385+0000 7f90c159a640 10 monclient: dump: 2024-09-17T18:25:11.399 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:11.399 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:11.399 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:11.399 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:11.399 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:11.399 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:11.399 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:11.399 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:11.400 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:11.400 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:11.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.385+0000 7f90c8e47640 10 monclient: shutdown 2024-09-17T18:25:11.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.385+0000 7f90c8e47640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:25:11.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.385+0000 7f90c8e47640 1 -- 172.21.15.121:0/1855061568 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90c40a5fe0 msgr2=0x7f90c40a63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:11.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.385+0000 7f90c8e47640 1 --2- 172.21.15.121:0/1855061568 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90c40a5fe0 0x7f90c40a63f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f90c409d210 tx=0x7f90b802a000 comp rx=0 tx=0).stop 2024-09-17T18:25:11.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.385+0000 7f90c8e47640 1 -- 172.21.15.121:0/1855061568 shutdown_connections 2024-09-17T18:25:11.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.385+0000 7f90c8e47640 1 --2- 172.21.15.121:0/1855061568 >> v2:172.21.15.173:3300/0 conn(0x7f90c413dc00 0x7f90c413fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:11.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.385+0000 7f90c8e47640 1 --2- 172.21.15.121:0/1855061568 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90c40a5fe0 0x7f90c40a63f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:11.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.385+0000 7f90c8e47640 1 -- 172.21.15.121:0/1855061568 >> 172.21.15.121:0/1855061568 conn(0x7f90c40ab9b0 msgr2=0x7f90c40a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:25:11.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.385+0000 7f90c8e47640 1 -- 172.21.15.121:0/1855061568 shutdown_connections 2024-09-17T18:25:11.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.385+0000 7f90c8e47640 1 -- 172.21.15.121:0/1855061568 wait complete. 2024-09-17T18:25:11.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.386+0000 7f90c8e47640 10 monclient: build_initial_monmap 2024-09-17T18:25:11.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.386+0000 7f90c8e47640 10 monclient: monmap: 2024-09-17T18:25:11.401 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:25:11.401 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:25:11.401 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:25:11.401 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:25:11.401 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:25:11.401 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:11.401 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:25:11.402 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:25:11.402 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:25:11.402 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:11.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.386+0000 7f90c8e47640 1 librados: starting msgr at 2024-09-17T18:25:11.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.386+0000 7f90c8e47640 1 librados: starting objecter 2024-09-17T18:25:11.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.386+0000 7f90c8e47640 1 Processor -- start 2024-09-17T18:25:11.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.386+0000 7f90c8e47640 1 -- start start 2024-09-17T18:25:11.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.386+0000 7f90c8e47640 1 librados: setting wanted keys 2024-09-17T18:25:11.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.386+0000 7f90c8e47640 1 librados: calling monclient init 2024-09-17T18:25:11.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.386+0000 7f90c8e47640 10 monclient: init 2024-09-17T18:25:11.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c8e47640 10 monclient: _reopen_session rank -1 2024-09-17T18:25:11.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c8e47640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T18:25:11.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c8e47640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f90c40a5c00 0x7f90c41cd740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:11.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c8e47640 10 monclient(hunting): picked mon.noname-b con 0x7f90c40a5c00 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:25:11.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c8e47640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f90c40a5fe0 0x7f90c41cde70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:11.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c8e47640 10 monclient(hunting): picked mon.noname-c con 0x7f90c40a5fe0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:25:11.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c8e47640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90c413dc00 0x7f90c413bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:11.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c8e47640 10 monclient(hunting): picked mon.noname-a con 0x7f90c413dc00 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:25:11.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c8e47640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:11.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c8e47640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f90c4070550 con 0x7f90c413dc00 2024-09-17T18:25:11.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c8e47640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:11.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c8e47640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f90c40703d0 con 0x7f90c40a5c00 2024-09-17T18:25:11.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c8e47640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:11.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c8e47640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f90c4140530 con 0x7f90c40a5fe0 2024-09-17T18:25:11.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c8e47640 10 monclient(hunting): _renew_subs 2024-09-17T18:25:11.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c8e47640 10 monclient(hunting): authenticate will time out at 1115.104342s 2024-09-17T18:25:11.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c2d9d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90c413dc00 0x7f90c413bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:11.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c1d9b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f90c40a5fe0 0x7f90c41cde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:11.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c2d9d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90c413dc00 0x7f90c413bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44474/0 (socket says 172.21.15.121:44474) 2024-09-17T18:25:11.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c2d9d640 1 -- 172.21.15.121:0/2725270351 learned_addr learned my addr 172.21.15.121:0/2725270351 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:25:11.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c1d9b640 10 monclient(hunting): get_auth_request con 0x7f90c40a5fe0 auth_method 0 2024-09-17T18:25:11.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:11.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:11.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:11.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c2d9d640 10 monclient(hunting): get_auth_request con 0x7f90c413dc00 auth_method 0 2024-09-17T18:25:11.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:11.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:11.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:11.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c259c640 1 --2- 172.21.15.121:0/2725270351 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f90c40a5c00 0x7f90c41cd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:11.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c259c640 10 monclient(hunting): get_auth_request con 0x7f90c40a5c00 auth_method 0 2024-09-17T18:25:11.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:11.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:11.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:11.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c1d9b640 1 -- 172.21.15.121:0/2725270351 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f90c40a5fe0 msgr2=0x7f90c41cde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T18:25:11.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:11.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:11.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c1d9b640 1 -- 172.21.15.121:0/2725270351 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f90c40a5fe0 msgr2=0x7f90c41cde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:25:11.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c1d9b640 1 --2- 172.21.15.121:0/2725270351 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f90c40a5fe0 0x7f90c41cde70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:25:11.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:11.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c1d9b640 1 --2- 172.21.15.121:0/2725270351 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f90c40a5fe0 0x7f90c41cde70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:25:11.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:11.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:11.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:11.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c2d9d640 10 monclient(hunting): handle_auth_done global_id 44325 payload 824 2024-09-17T18:25:11.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c2d9d640 1 -- 172.21.15.121:0/2725270351 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f90c40a5fe0 msgr2=0x7f90c41cde70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:25:11.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c2d9d640 1 --2- 172.21.15.121:0/2725270351 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f90c40a5fe0 0x7f90c41cde70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:11.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c2d9d640 1 -- 172.21.15.121:0/2725270351 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f90c40a5c00 msgr2=0x7f90c41cd740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:11.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c2d9d640 1 --2- 172.21.15.121:0/2725270351 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f90c40a5c00 0x7f90c41cd740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:11.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c2d9d640 10 monclient: _finish_hunting 0 2024-09-17T18:25:11.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c2d9d640 1 monclient: found mon.noname-a 2024-09-17T18:25:11.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:25:11.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c2d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:25:11.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.387+0000 7f90c2d9d640 1 -- 172.21.15.121:0/2725270351 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90c40a23f0 con 0x7f90c413dc00 2024-09-17T18:25:11.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.390+0000 7f90c2d9d640 1 -- 172.21.15.121:0/2725270351 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90c413dc00 msgr2=0x7f90c413bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:11.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.390+0000 7f90c2d9d640 1 --2- 172.21.15.121:0/2725270351 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90c413dc00 0x7f90c413bb20 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f90bc007e30 tx=0x7f90bc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:11.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.390+0000 7f90b2ffd640 10 client.?.objecter ms_handle_connect 0x7f90c413dc00 2024-09-17T18:25:11.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.390+0000 7f90b2ffd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:25:11.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.390+0000 7f90b2ffd640 1 -- 172.21.15.121:0/2725270351 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90bc013790 con 0x7f90c413dc00 2024-09-17T18:25:11.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.390+0000 7f90b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:11.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.390+0000 7f90b2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:25:11.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.390+0000 7f90b2ffd640 10 monclient: dump: 2024-09-17T18:25:11.407 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:11.407 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:11.407 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:11.407 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:11.407 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:11.407 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:11.407 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:11.407 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:11.407 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:11.408 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:11.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.390+0000 7f90b2ffd640 10 monclient: _finish_auth 0 2024-09-17T18:25:11.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.390+0000 7f90b2ffd640 10 monclient: _check_auth_tickets 2024-09-17T18:25:11.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.390+0000 7f90b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:25:11.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.390+0000 7f90b2ffd640 1 -- 172.21.15.121:0/2725270351 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90bc019070 con 0x7f90c413dc00 2024-09-17T18:25:11.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.390+0000 7f90c8e47640 5 monclient: authenticate success, global_id 44325 2024-09-17T18:25:11.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.390+0000 7f90c8e47640 10 monclient: _renew_subs 2024-09-17T18:25:11.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.390+0000 7f90c8e47640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:11.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.390+0000 7f90c8e47640 1 -- 172.21.15.121:0/2725270351 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90c4123580 con 0x7f90c413dc00 2024-09-17T18:25:11.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.390+0000 7f90b2ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:25:11.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.390+0000 7f90b2ffd640 1 -- 172.21.15.121:0/2725270351 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90bc013b70 con 0x7f90c413dc00 2024-09-17T18:25:11.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.390+0000 7f90b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:11.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.390+0000 7f90b2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:25:11.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.390+0000 7f90b2ffd640 10 monclient: dump: 2024-09-17T18:25:11.410 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:11.410 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:11.410 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:11.410 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:11.410 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:11.410 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:11.410 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:11.411 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:11.412 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:11.412 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:11.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.390+0000 7f90c0d99640 4 set_mon_vals no callback set 2024-09-17T18:25:11.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.390+0000 7f90c8e47640 10 client.44325.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:25:11.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.390+0000 7f90c8e47640 10 monclient: _renew_subs 2024-09-17T18:25:11.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.390+0000 7f90c8e47640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:11.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.390+0000 7f90c8e47640 1 -- 172.21.15.121:0/2725270351 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90c41232a0 con 0x7f90c413dc00 2024-09-17T18:25:11.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.390+0000 7f90c8e47640 1 librados: init done 2024-09-17T18:25:11.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.391+0000 7f90b2ffd640 1 -- 172.21.15.121:0/2725270351 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f90bc013d10 con 0x7f90c413dc00 2024-09-17T18:25:11.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.391+0000 7f90c8e47640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:11.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.391+0000 7f90c8e47640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:11.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.391+0000 7f90c8e47640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:11.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.391+0000 7f90c8e47640 1 -- 172.21.15.121:0/2725270351 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90c40a63f0 con 0x7f90c413dc00 2024-09-17T18:25:11.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.391+0000 7f90b2ffd640 1 --2- 172.21.15.121:0/2725270351 >> v2:172.21.15.173:6812/2423786037 conn(0x7f90a40401c0 0x7f90a4042680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:11.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.392+0000 7f90b2ffd640 1 -- 172.21.15.121:0/2725270351 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(283..283 src has 1..283) ==== 5784+0+0 (secure 0 0 0) 0x7f90bc054100 con 0x7f90c413dc00 2024-09-17T18:25:11.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.392+0000 7f90b2ffd640 10 client.44325.objecter ms_dispatch 0x7f90c4000b90 osd_map(283..283 src has 1..283) 2024-09-17T18:25:11.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.392+0000 7f90b2ffd640 3 client.44325.objecter handle_osd_map got epochs [283,283] > 0 2024-09-17T18:25:11.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.392+0000 7f90b2ffd640 3 client.44325.objecter handle_osd_map decoding full epoch 283 2024-09-17T18:25:11.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.392+0000 7f90b2ffd640 20 client.44325.objecter dump_active .. 0 homeless 2024-09-17T18:25:11.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.392+0000 7f90c259c640 1 --2- 172.21.15.121:0/2725270351 >> v2:172.21.15.173:6812/2423786037 conn(0x7f90a40401c0 0x7f90a4042680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:11.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.393+0000 7f90c259c640 10 monclient: get_auth_request con 0x7f90a40401c0 auth_method 0 2024-09-17T18:25:11.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.395+0000 7f90b2ffd640 10 client.44325.objecter ms_handle_connect 0x7f90a40401c0 2024-09-17T18:25:11.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.395+0000 7f90c259c640 1 --2- 172.21.15.121:0/2725270351 >> v2:172.21.15.173:6812/2423786037 conn(0x7f90a40401c0 0x7f90a4042680 secure :-1 s=READY pgs=3971 cs=0 l=1 rev1=1 crypto rx=0x7f90b40092a0 tx=0x7f90b4009230 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:11.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.398+0000 7f90b2ffd640 1 -- 172.21.15.121:0/2725270351 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f90c40a23f0 con 0x7f90c413dc00 2024-09-17T18:25:11.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.398+0000 7f90b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:11.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.398+0000 7f90b2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:25:11.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.645+0000 7f90c8e47640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:25:11.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.645+0000 7f90c8e47640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:25:11.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.645+0000 7f90c8e47640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:11.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.645+0000 7f90c8e47640 1 -- 172.21.15.121:0/2725270351 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f90c413dfe0 con 0x7f90c413dc00 2024-09-17T18:25:11.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.645+0000 7f90b2ffd640 1 -- 172.21.15.121:0/2725270351 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f90bc052710 con 0x7f90c413dc00 2024-09-17T18:25:11.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.645+0000 7f90b2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:25:11.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.645+0000 7f90b2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:25:11.648 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:25:11.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.647+0000 7f90c8e47640 10 librados: watch_flush enter 2024-09-17T18:25:11.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.647+0000 7f90c8e47640 10 librados: watch_flush exit 2024-09-17T18:25:11.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.647+0000 7f90c8e47640 20 client.44325.objecter shutdown clearing up homeless session... 2024-09-17T18:25:11.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.647+0000 7f90c8e47640 10 client.44325.objecter successfully canceled tick 2024-09-17T18:25:11.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.647+0000 7f90c8e47640 1 -- 172.21.15.121:0/2725270351 >> v2:172.21.15.173:6812/2423786037 conn(0x7f90a40401c0 msgr2=0x7f90a4042680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:11.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.647+0000 7f90c8e47640 1 --2- 172.21.15.121:0/2725270351 >> v2:172.21.15.173:6812/2423786037 conn(0x7f90a40401c0 0x7f90a4042680 secure :-1 s=READY pgs=3971 cs=0 l=1 rev1=1 crypto rx=0x7f90b40092a0 tx=0x7f90b4009230 comp rx=0 tx=0).stop 2024-09-17T18:25:11.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.647+0000 7f90c8e47640 10 monclient: shutdown 2024-09-17T18:25:11.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.647+0000 7f90c8e47640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:25:11.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.647+0000 7f90c8e47640 1 -- 172.21.15.121:0/2725270351 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90c413dc00 msgr2=0x7f90c413bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:11.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.647+0000 7f90c8e47640 1 --2- 172.21.15.121:0/2725270351 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90c413dc00 0x7f90c413bb20 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f90bc007e30 tx=0x7f90bc00e4b0 comp rx=0 tx=0).stop 2024-09-17T18:25:11.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.647+0000 7f90c8e47640 1 -- 172.21.15.121:0/2725270351 shutdown_connections 2024-09-17T18:25:11.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.647+0000 7f90c8e47640 1 --2- 172.21.15.121:0/2725270351 >> v2:172.21.15.173:6812/2423786037 conn(0x7f90a40401c0 0x7f90a4042680 unknown :-1 s=CLOSED pgs=3971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:11.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.647+0000 7f90c8e47640 1 --2- 172.21.15.121:0/2725270351 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90c413dc00 0x7f90c413bb20 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:11.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.647+0000 7f90c8e47640 1 --2- 172.21.15.121:0/2725270351 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f90c40a5fe0 0x7f90c41cde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:11.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.647+0000 7f90c8e47640 1 --2- 172.21.15.121:0/2725270351 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f90c40a5c00 0x7f90c41cd740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:11.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.647+0000 7f90c8e47640 1 -- 172.21.15.121:0/2725270351 >> 172.21.15.121:0/2725270351 conn(0x7f90c40ab9b0 msgr2=0x7f90c40a0e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:25:11.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.647+0000 7f90c8e47640 1 -- 172.21.15.121:0/2725270351 shutdown_connections 2024-09-17T18:25:11.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.647+0000 7f90c8e47640 1 -- 172.21.15.121:0/2725270351 wait complete. 2024-09-17T18:25:11.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.647+0000 7f90c8e47640 1 librados: shutdown 2024-09-17T18:25:11.660 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T18:25:11.660 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T18:25:11.660 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T18:25:11.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.755+0000 7f5a74d5d640 10 monclient: get_monmap_and_config 2024-09-17T18:25:11.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.755+0000 7f5a74d5d640 10 monclient: build_initial_monmap 2024-09-17T18:25:11.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.755+0000 7f5a74d5d640 10 monclient: monmap: 2024-09-17T18:25:11.758 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:25:11.758 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:11.758 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:25:11.757418+0000 2024-09-17T18:25:11.759 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:25:11.757418+0000 2024-09-17T18:25:11.759 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:25:11.759 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:11.759 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:25:11.759 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:25:11.759 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:25:11.759 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:25:11.759 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:25:11.759 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:11.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.756+0000 7f5a74d5d640 1 Processor -- start 2024-09-17T18:25:11.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.756+0000 7f5a74d5d640 1 -- start start 2024-09-17T18:25:11.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.756+0000 7f5a74d5d640 10 monclient: init 2024-09-17T18:25:11.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.756+0000 7f5a74d5d640 10 monclient: _reopen_session rank -1 2024-09-17T18:25:11.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.756+0000 7f5a74d5d640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-17T18:25:11.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.756+0000 7f5a74d5d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a700a3a00 0x7f5a7009da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:11.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.757+0000 7f5a74d5d640 10 monclient(hunting): picked mon.noname-b con 0x7f5a700a3a00 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:25:11.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.757+0000 7f5a74d5d640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f5a700a3de0 0x7f5a700a41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:11.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.757+0000 7f5a74d5d640 10 monclient(hunting): picked mon.noname-c con 0x7f5a700a3de0 addr v2:172.21.15.173:3300/0 2024-09-17T18:25:11.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.757+0000 7f5a74d5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5a701387e0 addr v1:172.21.15.121:6789/0 2024-09-17T18:25:11.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.757+0000 7f5a74d5d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a7009cd80 con 0x7f5a701387e0 2024-09-17T18:25:11.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.757+0000 7f5a74d5d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:11.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.757+0000 7f5a74d5d640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f5a7009d060 con 0x7f5a700a3de0 2024-09-17T18:25:11.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.757+0000 7f5a74d5d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:11.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.757+0000 7f5a74d5d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5a7009d340 con 0x7f5a700a3a00 2024-09-17T18:25:11.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.757+0000 7f5a74d5d640 10 monclient(hunting): _renew_subs 2024-09-17T18:25:11.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.757+0000 7f5a74d5d640 10 monclient(hunting): authenticate will time out at 1115.474071s 2024-09-17T18:25:11.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.757+0000 7f5a6e59c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a700a3a00 0x7f5a7009da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:11.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.757+0000 7f5a6ed9d640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f5a701387e0 0x7f5a70138c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:38350/0 (socket says 172.21.15.121:38350) 2024-09-17T18:25:11.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.757+0000 7f5a6ed9d640 1 -- 172.21.15.121:0/1433844749 learned_addr learned my addr 172.21.15.121:0/1433844749 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:25:11.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.757+0000 7f5a6e59c640 10 monclient(hunting): get_auth_request con 0x7f5a700a3a00 auth_method 0 2024-09-17T18:25:11.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.757+0000 7f5a6e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:11.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.757+0000 7f5a6e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:11.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.757+0000 7f5a6e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:11.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.757+0000 7f5a6e59c640 1 -- 172.21.15.121:0/1433844749 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a700a3a00 msgr2=0x7f5a7009da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:25:11.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.757+0000 7f5a6e59c640 1 -- 172.21.15.121:0/1433844749 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a700a3a00 msgr2=0x7f5a7009da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:25:11.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.757+0000 7f5a6e59c640 1 --2- 172.21.15.121:0/1433844749 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a700a3a00 0x7f5a7009da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:25:11.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.757+0000 7f5a6e59c640 1 --2- 172.21.15.121:0/1433844749 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a700a3a00 0x7f5a7009da40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:25:11.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.757+0000 7f5a67fff640 1 --2- 172.21.15.121:0/1433844749 >> v2:172.21.15.173:3300/0 conn(0x7f5a700a3de0 0x7f5a700a41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:11.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.757+0000 7f5a67fff640 10 monclient(hunting): get_auth_request con 0x7f5a700a3de0 auth_method 0 2024-09-17T18:25:11.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.757+0000 7f5a67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:11.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.757+0000 7f5a67fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:11.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.757+0000 7f5a67fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:11.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.757+0000 7f5a67fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:11.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.757+0000 7f5a67fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:11.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.757+0000 7f5a67fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:11.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.757+0000 7f5a67fff640 10 monclient(hunting): handle_auth_done global_id 24682 payload 274 2024-09-17T18:25:11.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.757+0000 7f5a67fff640 1 -- 172.21.15.121:0/1433844749 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a700a3a00 msgr2=0x7f5a7009da40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:25:11.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.757+0000 7f5a67fff640 1 --2- 172.21.15.121:0/1433844749 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a700a3a00 0x7f5a7009da40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:11.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.757+0000 7f5a67fff640 1 -- 172.21.15.121:0/1433844749 >> v1:172.21.15.121:6789/0 conn(0x7f5a701387e0 legacy=0x7f5a70138c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:25:11.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.757+0000 7f5a67fff640 10 monclient: _finish_hunting 0 2024-09-17T18:25:11.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.757+0000 7f5a67fff640 1 monclient: found mon.noname-c 2024-09-17T18:25:11.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.757+0000 7f5a67fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:25:11.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.757+0000 7f5a67fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.173:3300/0 2024-09-17T18:25:11.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.757+0000 7f5a67fff640 1 -- 172.21.15.121:0/1433844749 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a700a4700 con 0x7f5a700a3de0 2024-09-17T18:25:11.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.760+0000 7f5a67fff640 1 -- 172.21.15.121:0/1433844749 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a700a3de0 msgr2=0x7f5a700a41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:11.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.760+0000 7f5a67fff640 1 --2- 172.21.15.121:0/1433844749 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a700a3de0 0x7f5a700a41c0 secure :-1 s=READY pgs=7372 cs=0 l=1 rev1=1 crypto rx=0x7f5a580097a0 tx=0x7f5a58029fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f40ede66281e47f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:11.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.760+0000 7f5a6dd9b640 1 -- 172.21.15.121:0/1433844749 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a58037070 con 0x7f5a700a3de0 2024-09-17T18:25:11.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.760+0000 7f5a6dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:11.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.760+0000 7f5a6dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:25:11.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.760+0000 7f5a6dd9b640 10 monclient: dump: 2024-09-17T18:25:11.771 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:11.771 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:11.771 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:11.771 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:11.771 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:11.771 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:11.771 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:11.771 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:11.771 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:11.772 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:11.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.760+0000 7f5a6dd9b640 10 monclient: _finish_auth 0 2024-09-17T18:25:11.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.760+0000 7f5a6dd9b640 10 monclient: _check_auth_tickets 2024-09-17T18:25:11.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.760+0000 7f5a6dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:25:11.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.760+0000 7f5a74d5d640 5 monclient: authenticate success, global_id 24682 2024-09-17T18:25:11.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.760+0000 7f5a6dd9b640 1 -- 172.21.15.121:0/1433844749 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a58032440 con 0x7f5a700a3de0 2024-09-17T18:25:11.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.760+0000 7f5a74d5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:25:11.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.760+0000 7f5a6dd9b640 10 monclient: handle_config config(0 keys) 2024-09-17T18:25:11.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.760+0000 7f5a74d5d640 10 monclient: get_monmap_and_config success 2024-09-17T18:25:11.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.760+0000 7f5a6dd9b640 1 -- 172.21.15.121:0/1433844749 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a58032810 con 0x7f5a700a3de0 2024-09-17T18:25:11.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.760+0000 7f5a74d5d640 4 set_mon_vals no callback set 2024-09-17T18:25:11.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.760+0000 7f5a6dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:11.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.760+0000 7f5a6dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:25:11.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.760+0000 7f5a6dd9b640 10 monclient: dump: 2024-09-17T18:25:11.772 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:11.773 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:11.773 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:11.773 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:11.773 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:11.773 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:11.773 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:11.773 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:11.773 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:11.773 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:11.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.761+0000 7f5a74d5d640 10 monclient: shutdown 2024-09-17T18:25:11.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.761+0000 7f5a74d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:25:11.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.761+0000 7f5a74d5d640 1 -- 172.21.15.121:0/1433844749 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a700a3de0 msgr2=0x7f5a700a41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:11.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.761+0000 7f5a74d5d640 1 --2- 172.21.15.121:0/1433844749 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a700a3de0 0x7f5a700a41c0 secure :-1 s=READY pgs=7372 cs=0 l=1 rev1=1 crypto rx=0x7f5a580097a0 tx=0x7f5a58029fd0 comp rx=0 tx=0).stop 2024-09-17T18:25:11.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.761+0000 7f5a74d5d640 1 -- 172.21.15.121:0/1433844749 shutdown_connections 2024-09-17T18:25:11.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.761+0000 7f5a74d5d640 1 --2- 172.21.15.121:0/1433844749 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a700a3de0 0x7f5a700a41c0 unknown :-1 s=CLOSED pgs=7372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:11.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.761+0000 7f5a74d5d640 1 --2- 172.21.15.121:0/1433844749 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a700a3a00 0x7f5a7009da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:11.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.761+0000 7f5a74d5d640 1 -- 172.21.15.121:0/1433844749 >> 172.21.15.121:0/1433844749 conn(0x7f5a700a8680 msgr2=0x7f5a700a4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:25:11.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.761+0000 7f5a74d5d640 1 -- 172.21.15.121:0/1433844749 shutdown_connections 2024-09-17T18:25:11.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.761+0000 7f5a74d5d640 1 -- 172.21.15.121:0/1433844749 wait complete. 2024-09-17T18:25:11.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.762+0000 7f5a74d5d640 10 monclient: build_initial_monmap 2024-09-17T18:25:11.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.762+0000 7f5a74d5d640 10 monclient: monmap: 2024-09-17T18:25:11.774 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:25:11.774 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:25:11.774 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:25:11.774 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:25:11.774 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:25:11.774 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:11.774 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:25:11.774 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:25:11.775 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:25:11.775 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:11.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.762+0000 7f5a74d5d640 1 librados: starting msgr at 2024-09-17T18:25:11.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.762+0000 7f5a74d5d640 1 librados: starting objecter 2024-09-17T18:25:11.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.762+0000 7f5a74d5d640 1 Processor -- start 2024-09-17T18:25:11.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.762+0000 7f5a74d5d640 1 -- start start 2024-09-17T18:25:11.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.762+0000 7f5a74d5d640 1 librados: setting wanted keys 2024-09-17T18:25:11.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.762+0000 7f5a74d5d640 1 librados: calling monclient init 2024-09-17T18:25:11.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.762+0000 7f5a74d5d640 10 monclient: init 2024-09-17T18:25:11.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.762+0000 7f5a74d5d640 10 monclient: _reopen_session rank -1 2024-09-17T18:25:11.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.762+0000 7f5a74d5d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:25:11.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.762+0000 7f5a74d5d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a700a3a00 0x7f5a70137930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:11.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.762+0000 7f5a74d5d640 10 monclient(hunting): picked mon.noname-a con 0x7f5a700a3a00 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:25:11.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.762+0000 7f5a74d5d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a700a3de0 0x7f5a70137e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:11.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.762+0000 7f5a74d5d640 10 monclient(hunting): picked mon.noname-b con 0x7f5a700a3de0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:25:11.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.762+0000 7f5a74d5d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a701387e0 0x7f5a70125580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:11.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.762+0000 7f5a74d5d640 10 monclient(hunting): picked mon.noname-c con 0x7f5a701387e0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:25:11.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.762+0000 7f5a74d5d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:11.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.762+0000 7f5a74d5d640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5a70076f70 con 0x7f5a700a3a00 2024-09-17T18:25:11.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.762+0000 7f5a74d5d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:11.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.762+0000 7f5a74d5d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5a70076df0 con 0x7f5a700a3de0 2024-09-17T18:25:11.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.762+0000 7f5a74d5d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:11.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.762+0000 7f5a74d5d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5a70139340 con 0x7f5a701387e0 2024-09-17T18:25:11.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.763+0000 7f5a74d5d640 10 monclient(hunting): _renew_subs 2024-09-17T18:25:11.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.763+0000 7f5a74d5d640 10 monclient(hunting): authenticate will time out at 1115.480011s 2024-09-17T18:25:11.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.763+0000 7f5a6e59c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a700a3a00 0x7f5a70137930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:11.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.763+0000 7f5a6ed9d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a701387e0 0x7f5a70125580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:11.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.763+0000 7f5a6e59c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a700a3a00 0x7f5a70137930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44488/0 (socket says 172.21.15.121:44488) 2024-09-17T18:25:11.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.763+0000 7f5a6e59c640 1 -- 172.21.15.121:0/2077318994 learned_addr learned my addr 172.21.15.121:0/2077318994 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:25:11.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.763+0000 7f5a6e59c640 10 monclient(hunting): get_auth_request con 0x7f5a700a3a00 auth_method 0 2024-09-17T18:25:11.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.763+0000 7f5a6e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:11.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.763+0000 7f5a6e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:11.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.763+0000 7f5a6e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:11.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.763+0000 7f5a6ed9d640 10 monclient(hunting): get_auth_request con 0x7f5a701387e0 auth_method 0 2024-09-17T18:25:11.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.763+0000 7f5a6ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:11.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.763+0000 7f5a6ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:11.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.763+0000 7f5a6ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:11.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.763+0000 7f5a6e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:11.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.763+0000 7f5a6e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:11.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.763+0000 7f5a6ed9d640 1 -- 172.21.15.121:0/2077318994 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a701387e0 msgr2=0x7f5a70125580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-17T18:25:11.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.763+0000 7f5a6ed9d640 1 -- 172.21.15.121:0/2077318994 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a701387e0 msgr2=0x7f5a70125580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:25:11.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.763+0000 7f5a6e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:11.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.763+0000 7f5a6ed9d640 1 --2- 172.21.15.121:0/2077318994 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a701387e0 0x7f5a70125580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:25:11.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.763+0000 7f5a6ed9d640 1 --2- 172.21.15.121:0/2077318994 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a701387e0 0x7f5a70125580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:25:11.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.763+0000 7f5a67fff640 1 --2- 172.21.15.121:0/2077318994 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a700a3de0 0x7f5a70137e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:11.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.763+0000 7f5a67fff640 10 monclient(hunting): get_auth_request con 0x7f5a700a3de0 auth_method 0 2024-09-17T18:25:11.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.763+0000 7f5a67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:11.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.763+0000 7f5a67fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:11.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.763+0000 7f5a67fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:11.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.763+0000 7f5a6e59c640 10 monclient(hunting): handle_auth_done global_id 44328 payload 824 2024-09-17T18:25:11.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.763+0000 7f5a6e59c640 1 -- 172.21.15.121:0/2077318994 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a701387e0 msgr2=0x7f5a70125580 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:25:11.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.763+0000 7f5a6e59c640 1 --2- 172.21.15.121:0/2077318994 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a701387e0 0x7f5a70125580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:11.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.763+0000 7f5a6e59c640 1 -- 172.21.15.121:0/2077318994 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a700a3de0 msgr2=0x7f5a70137e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:11.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.763+0000 7f5a6e59c640 1 --2- 172.21.15.121:0/2077318994 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a700a3de0 0x7f5a70137e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:11.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.763+0000 7f5a6e59c640 10 monclient: _finish_hunting 0 2024-09-17T18:25:11.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.763+0000 7f5a6e59c640 1 monclient: found mon.noname-a 2024-09-17T18:25:11.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.763+0000 7f5a6e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:25:11.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.763+0000 7f5a6e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:25:11.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.763+0000 7f5a6e59c640 1 -- 172.21.15.121:0/2077318994 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a70126ff0 con 0x7f5a700a3a00 2024-09-17T18:25:11.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.763+0000 7f5a67fff640 1 --2- 172.21.15.121:0/2077318994 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a700a3de0 0x7f5a70137e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T18:25:11.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.765+0000 7f5a6e59c640 1 -- 172.21.15.121:0/2077318994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a700a3a00 msgr2=0x7f5a70137930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:11.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.765+0000 7f5a6e59c640 1 --2- 172.21.15.121:0/2077318994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a700a3a00 0x7f5a70137930 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5a6000d7d0 tx=0x7f5a6000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:11.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.765+0000 7f5a66ffd640 10 client.?.objecter ms_handle_connect 0x7f5a700a3a00 2024-09-17T18:25:11.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.765+0000 7f5a66ffd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:25:11.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.765+0000 7f5a66ffd640 1 -- 172.21.15.121:0/2077318994 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6000b8d0 con 0x7f5a700a3a00 2024-09-17T18:25:11.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.765+0000 7f5a66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:11.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.765+0000 7f5a66ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:25:11.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.765+0000 7f5a66ffd640 10 monclient: dump: 2024-09-17T18:25:11.779 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:11.779 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:11.779 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:11.779 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:11.779 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:11.779 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:11.779 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:11.780 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:11.780 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:11.780 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:11.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.765+0000 7f5a66ffd640 10 monclient: _finish_auth 0 2024-09-17T18:25:11.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.765+0000 7f5a66ffd640 10 monclient: _check_auth_tickets 2024-09-17T18:25:11.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.765+0000 7f5a66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:25:11.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.765+0000 7f5a74d5d640 5 monclient: authenticate success, global_id 44328 2024-09-17T18:25:11.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.765+0000 7f5a66ffd640 1 -- 172.21.15.121:0/2077318994 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a60019070 con 0x7f5a700a3a00 2024-09-17T18:25:11.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.765+0000 7f5a74d5d640 10 monclient: _renew_subs 2024-09-17T18:25:11.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.765+0000 7f5a74d5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:11.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.765+0000 7f5a74d5d640 1 -- 172.21.15.121:0/2077318994 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a7009f1e0 con 0x7f5a700a3a00 2024-09-17T18:25:11.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.765+0000 7f5a66ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:25:11.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.765+0000 7f5a66ffd640 1 -- 172.21.15.121:0/2077318994 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6000bcd0 con 0x7f5a700a3a00 2024-09-17T18:25:11.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.765+0000 7f5a66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:11.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.765+0000 7f5a66ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:25:11.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.765+0000 7f5a66ffd640 10 monclient: dump: 2024-09-17T18:25:11.781 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:11.781 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:11.781 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:11.781 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:11.781 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:11.781 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:11.781 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:11.781 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:11.781 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:11.781 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:11.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.765+0000 7f5a74d5d640 10 client.44328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:25:11.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.765+0000 7f5a6d59a640 4 set_mon_vals no callback set 2024-09-17T18:25:11.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.765+0000 7f5a74d5d640 10 monclient: _renew_subs 2024-09-17T18:25:11.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.765+0000 7f5a74d5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:11.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.765+0000 7f5a74d5d640 1 -- 172.21.15.121:0/2077318994 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a70129350 con 0x7f5a700a3a00 2024-09-17T18:25:11.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.765+0000 7f5a74d5d640 1 librados: init done 2024-09-17T18:25:11.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.766+0000 7f5a74d5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:11.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.766+0000 7f5a74d5d640 1 --2- 172.21.15.121:0/2077318994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a701cd980 0x7f5a7002d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:11.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.766+0000 7f5a74d5d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:11.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.766+0000 7f5a74d5d640 1 -- 172.21.15.121:0/2077318994 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5a700a2190 con 0x7f5a701cd980 2024-09-17T18:25:11.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.766+0000 7f5a67fff640 1 --2- 172.21.15.121:0/2077318994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a701cd980 0x7f5a7002d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:11.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.766+0000 7f5a66ffd640 1 -- 172.21.15.121:0/2077318994 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f5a6001e460 con 0x7f5a700a3a00 2024-09-17T18:25:11.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.766+0000 7f5a67fff640 10 monclient: get_auth_request con 0x7f5a701cd980 auth_method 0 2024-09-17T18:25:11.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.766+0000 7f5a67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:11.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.766+0000 7f5a67fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:11.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.766+0000 7f5a67fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:11.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.766+0000 7f5a67fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:11.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.766+0000 7f5a67fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:11.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.766+0000 7f5a67fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:11.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.766+0000 7f5a67fff640 10 monclient(hunting): handle_auth_done global_id 44331 payload 824 2024-09-17T18:25:11.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.767+0000 7f5a66ffd640 1 --2- 172.21.15.121:0/2077318994 >> v2:172.21.15.173:6812/2423786037 conn(0x7f5a50040170 0x7f5a50042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:11.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.767+0000 7f5a67fff640 1 -- 172.21.15.121:0/2077318994 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5a7009da40 con 0x7f5a701cd980 2024-09-17T18:25:11.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.767+0000 7f5a66ffd640 1 -- 172.21.15.121:0/2077318994 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(283..283 src has 1..283) ==== 5784+0+0 (secure 0 0 0) 0x7f5a60053db0 con 0x7f5a700a3a00 2024-09-17T18:25:11.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.767+0000 7f5a66ffd640 10 client.44328.objecter ms_dispatch 0x7f5a70000b90 osd_map(283..283 src has 1..283) 2024-09-17T18:25:11.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.767+0000 7f5a66ffd640 3 client.44328.objecter handle_osd_map got epochs [283,283] > 0 2024-09-17T18:25:11.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.767+0000 7f5a66ffd640 3 client.44328.objecter handle_osd_map decoding full epoch 283 2024-09-17T18:25:11.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.767+0000 7f5a66ffd640 20 client.44328.objecter dump_active .. 0 homeless 2024-09-17T18:25:11.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.767+0000 7f5a6ed9d640 1 --2- 172.21.15.121:0/2077318994 >> v2:172.21.15.173:6812/2423786037 conn(0x7f5a50040170 0x7f5a50042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:11.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.767+0000 7f5a6ed9d640 10 monclient: get_auth_request con 0x7f5a50040170 auth_method 0 2024-09-17T18:25:11.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.769+0000 7f5a67fff640 1 -- 172.21.15.121:0/2077318994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a701cd980 msgr2=0x7f5a7002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:11.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.769+0000 7f5a66ffd640 10 client.44328.objecter ms_handle_connect 0x7f5a701cd980 2024-09-17T18:25:11.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.769+0000 7f5a66ffd640 10 client.44328.objecter resend_mon_ops 2024-09-17T18:25:11.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.769+0000 7f5a67fff640 1 --2- 172.21.15.121:0/2077318994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a701cd980 0x7f5a7002d440 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5a700a28f0 tx=0x7f5a580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:11.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.769+0000 7f5a66ffd640 1 -- 172.21.15.121:0/2077318994 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a58042070 con 0x7f5a701cd980 2024-09-17T18:25:11.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.769+0000 7f5a66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:11.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.769+0000 7f5a66ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:25:11.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.769+0000 7f5a66ffd640 10 monclient: dump: 2024-09-17T18:25:11.786 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:11.786 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:11.786 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:11.786 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:11.786 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:11.786 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:11.786 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:11.786 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:11.786 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:11.786 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:11.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.769+0000 7f5a66ffd640 1 -- 172.21.15.121:0/2077318994 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5a5803e870 con 0x7f5a701cd980 2024-09-17T18:25:11.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.769+0000 7f5a66ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:11.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.769+0000 7f5a66ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:25:11.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.769+0000 7f5a66ffd640 1 -- 172.21.15.121:0/2077318994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a701cd980 msgr2=0x7f5a7002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:11.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.769+0000 7f5a66ffd640 1 --2- 172.21.15.121:0/2077318994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a701cd980 0x7f5a7002d440 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5a700a28f0 tx=0x7f5a580046c0 comp rx=0 tx=0).stop 2024-09-17T18:25:11.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.769+0000 7f5a66ffd640 1 -- 172.21.15.121:0/2077318994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a701cd980 msgr2=0x7f5a7002d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:25:11.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.769+0000 7f5a66ffd640 1 --2- 172.21.15.121:0/2077318994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a701cd980 0x7f5a7002d440 secure :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5a700a28f0 tx=0x7f5a580046c0 comp rx=0 tx=0).stop 2024-09-17T18:25:11.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.769+0000 7f5a66ffd640 10 client.44328.objecter ms_handle_connect 0x7f5a50040170 2024-09-17T18:25:11.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.769+0000 7f5a6ed9d640 1 --2- 172.21.15.121:0/2077318994 >> v2:172.21.15.173:6812/2423786037 conn(0x7f5a50040170 0x7f5a50042630 secure :-1 s=READY pgs=3972 cs=0 l=1 rev1=1 crypto rx=0x7f5a70129820 tx=0x7f5a68009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:11.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.775+0000 7f5a74d5d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:25:11.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.775+0000 7f5a74d5d640 1 --2- 172.21.15.121:0/2077318994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a7002d980 0x7f5a7002dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:11.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.775+0000 7f5a74d5d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:11.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.775+0000 7f5a74d5d640 1 -- 172.21.15.121:0/2077318994 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5a70127b50 con 0x7f5a7002d980 2024-09-17T18:25:11.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.775+0000 7f5a67fff640 1 --2- 172.21.15.121:0/2077318994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a7002d980 0x7f5a7002dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:11.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.775+0000 7f5a67fff640 10 monclient: get_auth_request con 0x7f5a7002d980 auth_method 0 2024-09-17T18:25:11.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.775+0000 7f5a67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:11.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.775+0000 7f5a67fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:11.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.775+0000 7f5a67fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:11.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.776+0000 7f5a67fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:11.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.776+0000 7f5a67fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:11.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.776+0000 7f5a67fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:11.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.776+0000 7f5a67fff640 10 monclient(hunting): handle_auth_done global_id 44334 payload 824 2024-09-17T18:25:11.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.776+0000 7f5a67fff640 1 -- 172.21.15.121:0/2077318994 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5a701383b0 con 0x7f5a7002d980 2024-09-17T18:25:11.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.778+0000 7f5a67fff640 1 -- 172.21.15.121:0/2077318994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a7002d980 msgr2=0x7f5a7002dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:11.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.778+0000 7f5a66ffd640 10 client.44328.objecter ms_handle_connect 0x7f5a7002d980 2024-09-17T18:25:11.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.778+0000 7f5a67fff640 1 --2- 172.21.15.121:0/2077318994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a7002d980 0x7f5a7002dd60 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5a580097a0 tx=0x7f5a58008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:11.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.778+0000 7f5a66ffd640 10 client.44328.objecter resend_mon_ops 2024-09-17T18:25:11.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.778+0000 7f5a66ffd640 1 -- 172.21.15.121:0/2077318994 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a58037040 con 0x7f5a7002d980 2024-09-17T18:25:11.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.778+0000 7f5a66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:11.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.778+0000 7f5a66ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:25:11.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.778+0000 7f5a66ffd640 10 monclient: dump: 2024-09-17T18:25:11.789 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:11.789 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:11.789 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:11.789 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:11.789 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:11.789 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:11.789 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:11.789 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:11.789 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:11.789 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:11.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.778+0000 7f5a66ffd640 1 -- 172.21.15.121:0/2077318994 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7f5a58003820 con 0x7f5a7002d980 2024-09-17T18:25:11.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.778+0000 7f5a66ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:25:11.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.778+0000 7f5a66ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:25:11.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.778+0000 7f5a66ffd640 1 -- 172.21.15.121:0/2077318994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a7002d980 msgr2=0x7f5a7002dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:11.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.778+0000 7f5a66ffd640 1 --2- 172.21.15.121:0/2077318994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a7002d980 0x7f5a7002dd60 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5a580097a0 tx=0x7f5a58008030 comp rx=0 tx=0).stop 2024-09-17T18:25:11.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.778+0000 7f5a66ffd640 1 -- 172.21.15.121:0/2077318994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a7002d980 msgr2=0x7f5a7002dd60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:25:11.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.778+0000 7f5a66ffd640 1 --2- 172.21.15.121:0/2077318994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a7002d980 0x7f5a7002dd60 secure :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5a580097a0 tx=0x7f5a58008030 comp rx=0 tx=0).stop 2024-09-17T18:25:11.790 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:25:11.790 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:25:11.790 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:25:11.790 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "leader", 2024-09-17T18:25:11.790 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 26, 2024-09-17T18:25:11.791 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 137443, 2024-09-17T18:25:11.791 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:25:11.791 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:25:11.791 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:25:11.791 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:25:11.791 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:25:11.791 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 3, 2024-09-17T18:25:11.791 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:25:11.791 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:25:11.791 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:25:11.791 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:25:11.791 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:25:11.791 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:25:11.791 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:25:11.791 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:25:11.792 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:25:11.792 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:25:11.792 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:25:11.792 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:25:11.792 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:25:11.792 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:25:11.792 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:25:11.792 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:25:11.792 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:25:11.792 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:25:11.792 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:25:11.792 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:25:11.792 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:25:11.792 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:25:11.793 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:25:11.793 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:25:11.793 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:25:11.793 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:25:11.793 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:25:11.793 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:25:11.793 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:11.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:11.793 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:25:11.793 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:25:11.793 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:25:11.793 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:25:11.793 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:25:11.794 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:25:11.794 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:25:11.794 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:25:11.794 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:25:11.794 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:25:11.794 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:25:11.794 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:25:11.794 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:25:11.794 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:25:11.794 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:25:11.794 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:25:11.794 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:25:11.794 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:25:11.794 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:25:11.794 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:25:11.794 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:25:11.795 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:25:11.795 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:25:11.795 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:25:11.795 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:25:11.795 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:25:11.795 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:25:11.795 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:25:11.795 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:25:11.795 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:25:11.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:11.795 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:25:11.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:11.795 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:25:11.795 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:25:11.795 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:25:11.796 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:25:11.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:11.796 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:25:11.796 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:25:11.796 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:11.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:11.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:11.796 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:25:11.796 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:25:11.796 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:11.796 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:11.796 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:11.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:11.796 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:25:11.796 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:25:11.797 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:25:11.797 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:25:11.797 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:25:11.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:11.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:11.797 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:25:11.797 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:25:11.797 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:25:11.797 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:25:11.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:11.797 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:25:11.797 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:25:11.797 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:11.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:11.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:11.798 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:25:11.798 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:25:11.798 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:11.798 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:11.798 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:11.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:11.798 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:25:11.798 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:25:11.798 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:25:11.798 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:25:11.798 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:25:11.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:11.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:11.798 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:25:11.798 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:25:11.798 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:25:11.799 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:25:11.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:11.799 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:25:11.799 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:25:11.799 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:11.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:11.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:11.799 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:25:11.799 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:25:11.799 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:11.799 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:11.799 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:11.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:11.799 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:25:11.799 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:25:11.800 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:25:11.800 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:25:11.800 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:25:11.800 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:11.800 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:11.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:11.800 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:25:11.800 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:25:11.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:11.800 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:25:11.800 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:25:11.800 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:25:11.800 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:11.800 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:25:11.800 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:25:11.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:11.801 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:25:11.801 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:25:11.801 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-17T18:25:11.801 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:11.801 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:25:11.801 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:25:11.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:11.801 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:25:11.801 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:25:11.801 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-17T18:25:11.801 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:11.801 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:11.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:11.801 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:25:11.801 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:25:11.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.778+0000 7f5a74d5d640 10 librados: watch_flush enter 2024-09-17T18:25:11.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.778+0000 7f5a74d5d640 10 librados: watch_flush exit 2024-09-17T18:25:11.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.779+0000 7f5a74d5d640 20 client.44328.objecter shutdown clearing up homeless session... 2024-09-17T18:25:11.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.779+0000 7f5a74d5d640 10 client.44328.objecter successfully canceled tick 2024-09-17T18:25:11.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.779+0000 7f5a74d5d640 1 -- 172.21.15.121:0/2077318994 >> v2:172.21.15.173:6812/2423786037 conn(0x7f5a50040170 msgr2=0x7f5a50042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:11.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.779+0000 7f5a74d5d640 1 --2- 172.21.15.121:0/2077318994 >> v2:172.21.15.173:6812/2423786037 conn(0x7f5a50040170 0x7f5a50042630 secure :-1 s=READY pgs=3972 cs=0 l=1 rev1=1 crypto rx=0x7f5a70129820 tx=0x7f5a68009040 comp rx=0 tx=0).stop 2024-09-17T18:25:11.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.779+0000 7f5a74d5d640 10 monclient: shutdown 2024-09-17T18:25:11.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.779+0000 7f5a74d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:25:11.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.779+0000 7f5a74d5d640 1 -- 172.21.15.121:0/2077318994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a700a3a00 msgr2=0x7f5a70137930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:11.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.779+0000 7f5a74d5d640 1 --2- 172.21.15.121:0/2077318994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a700a3a00 0x7f5a70137930 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5a6000d7d0 tx=0x7f5a6000dc90 comp rx=0 tx=0).stop 2024-09-17T18:25:11.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.779+0000 7f5a6ed9d640 1 -- 172.21.15.121:0/2077318994 reap_dead start 2024-09-17T18:25:11.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.779+0000 7f5a6ed9d640 1 -- 172.21.15.121:0/2077318994 reap_dead start 2024-09-17T18:25:11.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.779+0000 7f5a74d5d640 1 -- 172.21.15.121:0/2077318994 shutdown_connections 2024-09-17T18:25:11.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.779+0000 7f5a74d5d640 1 -- 172.21.15.121:0/2077318994 >> 172.21.15.121:0/2077318994 conn(0x7f5a700a8680 msgr2=0x7f5a700a6710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:25:11.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.779+0000 7f5a74d5d640 1 -- 172.21.15.121:0/2077318994 shutdown_connections 2024-09-17T18:25:11.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.779+0000 7f5a74d5d640 1 -- 172.21.15.121:0/2077318994 wait complete. 2024-09-17T18:25:11.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.779+0000 7f5a74d5d640 1 librados: shutdown 2024-09-17T18:25:11.804 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:25:11.804 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T18:25:11.804 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T18:25:11.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.899+0000 7fd7b369b640 10 monclient: get_monmap_and_config 2024-09-17T18:25:11.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.899+0000 7fd7b369b640 10 monclient: build_initial_monmap 2024-09-17T18:25:11.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.899+0000 7fd7b369b640 10 monclient: monmap: 2024-09-17T18:25:11.902 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:25:11.902 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:11.902 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:25:11.900730+0000 2024-09-17T18:25:11.902 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:25:11.900730+0000 2024-09-17T18:25:11.902 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:25:11.902 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:11.902 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:25:11.902 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:25:11.902 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:25:11.902 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:25:11.902 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:25:11.902 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:11.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.899+0000 7fd7b369b640 1 Processor -- start 2024-09-17T18:25:11.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.899+0000 7fd7b369b640 1 -- start start 2024-09-17T18:25:11.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.899+0000 7fd7b369b640 10 monclient: init 2024-09-17T18:25:11.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b369b640 10 monclient: _reopen_session rank -1 2024-09-17T18:25:11.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b369b640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-17T18:25:11.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b369b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0700f0 0x7fd7ac134e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:11.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b369b640 10 monclient(hunting): picked mon.noname-b con 0x7fd7ac0700f0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:25:11.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b369b640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd7ac09e810 0x7fd7ac138360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:11.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b369b640 10 monclient(hunting): picked mon.noname-a con 0x7fd7ac09e810 addr v2:172.21.15.121:3300/0 2024-09-17T18:25:11.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b369b640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fd7ac09ec30 0x7fd7ac1388a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:11.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b369b640 10 monclient(hunting): picked mon.noname-c con 0x7fd7ac09ec30 addr v2:172.21.15.173:3300/0 2024-09-17T18:25:11.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b369b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:11.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b369b640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fd7ac0704d0 con 0x7fd7ac09e810 2024-09-17T18:25:11.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b369b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:11.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b369b640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fd7ac09ca60 con 0x7fd7ac09ec30 2024-09-17T18:25:11.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b369b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:11.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b369b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd7ac09cd20 con 0x7fd7ac0700f0 2024-09-17T18:25:11.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b369b640 10 monclient(hunting): _renew_subs 2024-09-17T18:25:11.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b369b640 10 monclient(hunting): authenticate will time out at 1115.617464s 2024-09-17T18:25:11.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b1437640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0700f0 0x7fd7ac134e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:11.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b0c36640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd7ac09e810 0x7fd7ac138360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:11.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b1437640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0700f0 0x7fd7ac134e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:53438/0 (socket says 172.21.15.121:53438) 2024-09-17T18:25:11.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b1437640 1 -- 172.21.15.121:0/2528047063 learned_addr learned my addr 172.21.15.121:0/2528047063 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:25:11.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b1437640 10 monclient(hunting): get_auth_request con 0x7fd7ac0700f0 auth_method 0 2024-09-17T18:25:11.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b1437640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:11.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b1437640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:11.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b1437640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:11.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b0c36640 10 monclient(hunting): get_auth_request con 0x7fd7ac09e810 auth_method 0 2024-09-17T18:25:11.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b0c36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:11.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b0c36640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:11.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b0c36640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:11.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b1437640 1 -- 172.21.15.121:0/2528047063 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0700f0 msgr2=0x7fd7ac134e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:25:11.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b0c36640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:11.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b1437640 1 -- 172.21.15.121:0/2528047063 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0700f0 msgr2=0x7fd7ac134e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:25:11.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b0c36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:11.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b1437640 1 --2- 172.21.15.121:0/2528047063 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0700f0 0x7fd7ac134e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:25:11.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b0c36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:11.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b1437640 1 --2- 172.21.15.121:0/2528047063 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0700f0 0x7fd7ac134e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:25:11.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b0c36640 10 monclient(hunting): handle_auth_done global_id 44337 payload 274 2024-09-17T18:25:11.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b0c36640 1 -- 172.21.15.121:0/2528047063 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0700f0 msgr2=0x7fd7ac134e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:25:11.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b0c36640 1 --2- 172.21.15.121:0/2528047063 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0700f0 0x7fd7ac134e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:11.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b0c36640 1 -- 172.21.15.121:0/2528047063 >> v2:172.21.15.173:3300/0 conn(0x7fd7ac09ec30 msgr2=0x7fd7ac1388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:25:11.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b0c36640 1 --2- 172.21.15.121:0/2528047063 >> v2:172.21.15.173:3300/0 conn(0x7fd7ac09ec30 0x7fd7ac1388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:11.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b0c36640 10 monclient: _finish_hunting 0 2024-09-17T18:25:11.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b0c36640 1 monclient: found mon.noname-a 2024-09-17T18:25:11.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b0c36640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:25:11.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b0c36640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:25:11.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.900+0000 7fd7b0c36640 1 -- 172.21.15.121:0/2528047063 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7ac09d000 con 0x7fd7ac09e810 2024-09-17T18:25:11.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.903+0000 7fd7b0c36640 1 -- 172.21.15.121:0/2528047063 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7ac09e810 msgr2=0x7fd7ac138360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:11.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.903+0000 7fd7b0c36640 1 --2- 172.21.15.121:0/2528047063 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7ac09e810 0x7fd7ac138360 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac0a5870 tx=0x7fd79c02a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9aca2a80325c835 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:11.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.904+0000 7fd7a3fff640 1 -- 172.21.15.121:0/2528047063 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd79c037070 con 0x7fd7ac09e810 2024-09-17T18:25:11.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.904+0000 7fd7a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:11.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.904+0000 7fd7a3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:25:11.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.904+0000 7fd7a3fff640 10 monclient: dump: 2024-09-17T18:25:11.914 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:11.914 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:11.915 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:11.915 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:11.915 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:11.915 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:11.915 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:11.915 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:11.915 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:11.915 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:11.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.904+0000 7fd7a3fff640 10 monclient: _finish_auth 0 2024-09-17T18:25:11.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.904+0000 7fd7a3fff640 10 monclient: _check_auth_tickets 2024-09-17T18:25:11.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.904+0000 7fd7a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:25:11.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.904+0000 7fd7a3fff640 1 -- 172.21.15.121:0/2528047063 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd79c032440 con 0x7fd7ac09e810 2024-09-17T18:25:11.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.904+0000 7fd7b369b640 5 monclient: authenticate success, global_id 44337 2024-09-17T18:25:11.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.904+0000 7fd7b369b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:25:11.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.904+0000 7fd7a3fff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:25:11.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.904+0000 7fd7a3fff640 1 -- 172.21.15.121:0/2528047063 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd79c0327f0 con 0x7fd7ac09e810 2024-09-17T18:25:11.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.904+0000 7fd7a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:11.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.904+0000 7fd7a3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:25:11.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.904+0000 7fd7a3fff640 10 monclient: dump: 2024-09-17T18:25:11.916 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:11.916 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:11.916 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:11.916 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:11.916 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:11.916 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:11.916 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:11.916 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:11.916 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:11.916 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:11.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.904+0000 7fd7b369b640 10 monclient: get_monmap_and_config success 2024-09-17T18:25:11.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.904+0000 7fd7b369b640 4 set_mon_vals no callback set 2024-09-17T18:25:11.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.904+0000 7fd7b369b640 10 monclient: shutdown 2024-09-17T18:25:11.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.904+0000 7fd7b369b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:25:11.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.904+0000 7fd7b369b640 1 -- 172.21.15.121:0/2528047063 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7ac09e810 msgr2=0x7fd7ac138360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:11.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.904+0000 7fd7b369b640 1 --2- 172.21.15.121:0/2528047063 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7ac09e810 0x7fd7ac138360 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac0a5870 tx=0x7fd79c02a080 comp rx=0 tx=0).stop 2024-09-17T18:25:11.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.904+0000 7fd7b369b640 1 -- 172.21.15.121:0/2528047063 shutdown_connections 2024-09-17T18:25:11.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.904+0000 7fd7b369b640 1 --2- 172.21.15.121:0/2528047063 >> v2:172.21.15.173:3300/0 conn(0x7fd7ac09ec30 0x7fd7ac1388a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:11.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.904+0000 7fd7b369b640 1 --2- 172.21.15.121:0/2528047063 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7ac09e810 0x7fd7ac138360 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:11.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.904+0000 7fd7b369b640 1 --2- 172.21.15.121:0/2528047063 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0700f0 0x7fd7ac134e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:11.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.904+0000 7fd7b369b640 1 -- 172.21.15.121:0/2528047063 >> 172.21.15.121:0/2528047063 conn(0x7fd7ac0ab830 msgr2=0x7fd7ac0a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:25:11.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.904+0000 7fd7b369b640 1 -- 172.21.15.121:0/2528047063 shutdown_connections 2024-09-17T18:25:11.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.904+0000 7fd7b369b640 1 -- 172.21.15.121:0/2528047063 wait complete. 2024-09-17T18:25:11.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.905+0000 7fd7b369b640 10 monclient: build_initial_monmap 2024-09-17T18:25:11.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.905+0000 7fd7b369b640 10 monclient: monmap: 2024-09-17T18:25:11.918 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:25:11.918 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:25:11.918 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:25:11.918 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:25:11.918 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:25:11.918 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:11.918 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:25:11.918 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:25:11.918 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:25:11.918 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:11.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.905+0000 7fd7b369b640 1 librados: starting msgr at 2024-09-17T18:25:11.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.905+0000 7fd7b369b640 1 librados: starting objecter 2024-09-17T18:25:11.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.905+0000 7fd7b369b640 1 Processor -- start 2024-09-17T18:25:11.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.905+0000 7fd7b369b640 1 -- start start 2024-09-17T18:25:11.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.905+0000 7fd7b369b640 1 librados: setting wanted keys 2024-09-17T18:25:11.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.905+0000 7fd7b369b640 1 librados: calling monclient init 2024-09-17T18:25:11.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.905+0000 7fd7b369b640 10 monclient: init 2024-09-17T18:25:11.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b369b640 10 monclient: _reopen_session rank -1 2024-09-17T18:25:11.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b369b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T18:25:11.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b369b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0700f0 0x7fd7ac1cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:11.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b369b640 10 monclient(hunting): picked mon.noname-c con 0x7fd7ac0700f0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:25:11.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b369b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7ac09e810 0x7fd7ac131b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:11.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b369b640 10 monclient(hunting): picked mon.noname-b con 0x7fd7ac09e810 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:25:11.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b369b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7ac09ec30 0x7fd7ac132050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:11.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b369b640 10 monclient(hunting): picked mon.noname-a con 0x7fd7ac09ec30 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:25:11.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b369b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:11.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b369b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd7ac135510 con 0x7fd7ac09ec30 2024-09-17T18:25:11.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b369b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:11.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b369b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd7ac135390 con 0x7fd7ac09e810 2024-09-17T18:25:11.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b369b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:11.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b369b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd7ac138de0 con 0x7fd7ac0700f0 2024-09-17T18:25:11.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b369b640 10 monclient(hunting): _renew_subs 2024-09-17T18:25:11.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b369b640 10 monclient(hunting): authenticate will time out at 1115.623185s 2024-09-17T18:25:11.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b1437640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0700f0 0x7fd7ac1cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:11.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b1437640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0700f0 0x7fd7ac1cd6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:53442/0 (socket says 172.21.15.121:53442) 2024-09-17T18:25:11.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b1437640 1 -- 172.21.15.121:0/538296681 learned_addr learned my addr 172.21.15.121:0/538296681 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:25:11.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b1437640 10 monclient(hunting): get_auth_request con 0x7fd7ac0700f0 auth_method 0 2024-09-17T18:25:11.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b1437640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:11.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b1437640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:11.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b1437640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:11.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b1c38640 1 --2- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7ac09ec30 0x7fd7ac132050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:11.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b1c38640 10 monclient(hunting): get_auth_request con 0x7fd7ac09ec30 auth_method 0 2024-09-17T18:25:11.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b1c38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:11.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b1c38640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:11.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b1c38640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:11.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b1437640 1 -- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0700f0 msgr2=0x7fd7ac1cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T18:25:11.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b1437640 1 -- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0700f0 msgr2=0x7fd7ac1cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:25:11.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b1437640 1 --2- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0700f0 0x7fd7ac1cd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:25:11.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b1437640 1 --2- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0700f0 0x7fd7ac1cd6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:25:11.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b1c38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:11.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b1c38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:11.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b1c38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:11.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b1c38640 10 monclient(hunting): handle_auth_done global_id 44340 payload 824 2024-09-17T18:25:11.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b0c36640 1 --2- 172.21.15.121:0/538296681 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7ac09e810 0x7fd7ac131b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:11.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b1c38640 1 -- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0700f0 msgr2=0x7fd7ac1cd6e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:25:11.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b1c38640 1 --2- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0700f0 0x7fd7ac1cd6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:11.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b1c38640 1 -- 172.21.15.121:0/538296681 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7ac09e810 msgr2=0x7fd7ac131b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:11.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b1c38640 1 --2- 172.21.15.121:0/538296681 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7ac09e810 0x7fd7ac131b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:11.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b1c38640 10 monclient: _finish_hunting 0 2024-09-17T18:25:11.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b1c38640 1 monclient: found mon.noname-a 2024-09-17T18:25:11.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b1c38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:25:11.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b1c38640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:25:11.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.906+0000 7fd7b1c38640 1 -- 172.21.15.121:0/538296681 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7ac0a5100 con 0x7fd7ac09ec30 2024-09-17T18:25:11.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.908+0000 7fd7b1c38640 1 -- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7ac09ec30 msgr2=0x7fd7ac132050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:11.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.908+0000 7fd7b1c38640 1 --2- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7ac09ec30 0x7fd7ac132050 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd7a800be20 tx=0x7fd7a800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:11.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.908+0000 7fd7a1ffb640 10 client.?.objecter ms_handle_connect 0x7fd7ac09ec30 2024-09-17T18:25:11.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.908+0000 7fd7a1ffb640 10 client.?.objecter resend_mon_ops 2024-09-17T18:25:11.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.908+0000 7fd7a1ffb640 1 -- 172.21.15.121:0/538296681 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a8019070 con 0x7fd7ac09ec30 2024-09-17T18:25:11.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.908+0000 7fd7a1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:11.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.908+0000 7fd7a1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:25:11.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.908+0000 7fd7a1ffb640 10 monclient: dump: 2024-09-17T18:25:11.923 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:11.923 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:11.923 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:11.923 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:11.923 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:11.923 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:11.923 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:11.923 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:11.923 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:11.923 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:11.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.908+0000 7fd7a1ffb640 10 monclient: _finish_auth 0 2024-09-17T18:25:11.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.908+0000 7fd7a1ffb640 10 monclient: _check_auth_tickets 2024-09-17T18:25:11.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.908+0000 7fd7a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:25:11.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.908+0000 7fd7a1ffb640 1 -- 172.21.15.121:0/538296681 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7a8014660 con 0x7fd7ac09ec30 2024-09-17T18:25:11.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.908+0000 7fd7b369b640 5 monclient: authenticate success, global_id 44340 2024-09-17T18:25:11.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.908+0000 7fd7a1ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:25:11.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.908+0000 7fd7a1ffb640 1 -- 172.21.15.121:0/538296681 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a8014a50 con 0x7fd7ac09ec30 2024-09-17T18:25:11.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.908+0000 7fd7a1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:11.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.908+0000 7fd7a1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:25:11.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.908+0000 7fd7a1ffb640 10 monclient: dump: 2024-09-17T18:25:11.924 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:11.924 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:11.924 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:11.924 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:11.924 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:11.924 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:11.924 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:11.925 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:11.925 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:11.925 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:11.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.908+0000 7fd7b369b640 10 monclient: _renew_subs 2024-09-17T18:25:11.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.908+0000 7fd7a37fe640 4 set_mon_vals no callback set 2024-09-17T18:25:11.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.908+0000 7fd7b369b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:11.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.908+0000 7fd7b369b640 1 -- 172.21.15.121:0/538296681 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7ac132590 con 0x7fd7ac09ec30 2024-09-17T18:25:11.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.909+0000 7fd7b369b640 10 client.44340.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:25:11.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.909+0000 7fd7b369b640 10 monclient: _renew_subs 2024-09-17T18:25:11.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.909+0000 7fd7b369b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:11.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.909+0000 7fd7b369b640 1 -- 172.21.15.121:0/538296681 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7ac121d10 con 0x7fd7ac09ec30 2024-09-17T18:25:11.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.909+0000 7fd7b369b640 1 librados: init done 2024-09-17T18:25:11.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.910+0000 7fd7b369b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:11.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.910+0000 7fd7b369b640 1 --2- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0a3730 0x7fd7ac0a3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:11.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.910+0000 7fd7b369b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:11.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.910+0000 7fd7b369b640 1 -- 172.21.15.121:0/538296681 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd7ac0a23f0 con 0x7fd7ac0a3730 2024-09-17T18:25:11.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.910+0000 7fd7a1ffb640 1 -- 172.21.15.121:0/538296681 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fd7a801c070 con 0x7fd7ac09ec30 2024-09-17T18:25:11.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.910+0000 7fd7b1437640 1 --2- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0a3730 0x7fd7ac0a3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:11.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.910+0000 7fd7b1437640 10 monclient: get_auth_request con 0x7fd7ac0a3730 auth_method 0 2024-09-17T18:25:11.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.910+0000 7fd7b1437640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:11.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.910+0000 7fd7b1437640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:11.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.910+0000 7fd7b1437640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:11.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.910+0000 7fd7b1437640 1 -- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0a3730 msgr2=0x7fd7ac0a3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T18:25:11.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.910+0000 7fd7b1437640 1 -- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0a3730 msgr2=0x7fd7ac0a3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:25:11.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.910+0000 7fd7b1437640 1 --2- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0a3730 0x7fd7ac0a3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:25:11.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.910+0000 7fd7b1437640 1 --2- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0a3730 0x7fd7ac0a3b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:25:11.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.910+0000 7fd7a1ffb640 1 --2- 172.21.15.121:0/538296681 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd790040170 0x7fd790042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:11.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.910+0000 7fd7a1ffb640 1 -- 172.21.15.121:0/538296681 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(283..283 src has 1..283) ==== 5784+0+0 (secure 0 0 0) 0x7fd7a8053ca0 con 0x7fd7ac09ec30 2024-09-17T18:25:11.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.910+0000 7fd7a1ffb640 10 client.44340.objecter ms_dispatch 0x7fd7ac000b90 osd_map(283..283 src has 1..283) 2024-09-17T18:25:11.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.910+0000 7fd7a1ffb640 3 client.44340.objecter handle_osd_map got epochs [283,283] > 0 2024-09-17T18:25:11.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.910+0000 7fd7a1ffb640 3 client.44340.objecter handle_osd_map decoding full epoch 283 2024-09-17T18:25:11.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.910+0000 7fd7a1ffb640 20 client.44340.objecter dump_active .. 0 homeless 2024-09-17T18:25:11.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.910+0000 7fd7b0c36640 1 --2- 172.21.15.121:0/538296681 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd790040170 0x7fd790042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:11.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.910+0000 7fd7b0c36640 10 monclient: get_auth_request con 0x7fd790040170 auth_method 0 2024-09-17T18:25:11.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.913+0000 7fd7a1ffb640 10 client.44340.objecter ms_handle_connect 0x7fd790040170 2024-09-17T18:25:11.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:11.913+0000 7fd7b0c36640 1 --2- 172.21.15.121:0/538296681 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd790040170 0x7fd790042630 secure :-1 s=READY pgs=3973 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac070720 tx=0x7fd79c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:12.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:12.111+0000 7fd7b1437640 1 --2- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0a3730 0x7fd7ac0a3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:12.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:12.111+0000 7fd7b1437640 10 monclient: get_auth_request con 0x7fd7ac0a3730 auth_method 0 2024-09-17T18:25:12.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:12.111+0000 7fd7b1437640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:12.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:12.111+0000 7fd7b1437640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:12.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:12.111+0000 7fd7b1437640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:25:12.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:12.111+0000 7fd7b1437640 1 -- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0a3730 msgr2=0x7fd7ac0a3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T18:25:12.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:12.111+0000 7fd7b1437640 1 -- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0a3730 msgr2=0x7fd7ac0a3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:25:12.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:12.111+0000 7fd7b1437640 1 --2- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0a3730 0x7fd7ac0a3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:25:12.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:12.111+0000 7fd7b1437640 1 --2- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0a3730 0x7fd7ac0a3b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T18:25:12.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:12.513+0000 7fd7b1437640 1 --2- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0a3730 0x7fd7ac0a3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:12.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:12.513+0000 7fd7b1437640 10 monclient: get_auth_request con 0x7fd7ac0a3730 auth_method 0 2024-09-17T18:25:12.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:12.513+0000 7fd7b1437640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:12.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:12.513+0000 7fd7b1437640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:12.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:12.513+0000 7fd7b1437640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:25:12.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:12.513+0000 7fd7b1437640 1 -- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0a3730 msgr2=0x7fd7ac0a3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T18:25:12.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:12.513+0000 7fd7b1437640 1 -- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0a3730 msgr2=0x7fd7ac0a3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:25:12.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:12.513+0000 7fd7b1437640 1 --2- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0a3730 0x7fd7ac0a3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:25:12.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:12.513+0000 7fd7b1437640 1 --2- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0a3730 0x7fd7ac0a3b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T18:25:13.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.314+0000 7fd7b1437640 1 --2- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0a3730 0x7fd7ac0a3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:13.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.314+0000 7fd7b1437640 10 monclient: get_auth_request con 0x7fd7ac0a3730 auth_method 0 2024-09-17T18:25:13.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.314+0000 7fd7b1437640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:13.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.314+0000 7fd7b1437640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:13.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.314+0000 7fd7b1437640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:25:13.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.314+0000 7fd7b1437640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:13.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.314+0000 7fd7b1437640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:13.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.314+0000 7fd7b1437640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:13.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.314+0000 7fd7b1437640 10 monclient(hunting): handle_auth_done global_id 54101 payload 824 2024-09-17T18:25:13.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.314+0000 7fd7b1437640 1 -- 172.21.15.121:0/538296681 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd7ac134660 con 0x7fd7ac0a3730 2024-09-17T18:25:13.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.316+0000 7fd7b1437640 1 -- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0a3730 msgr2=0x7fd7ac0a3b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:13.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.316+0000 7fd7b1437640 1 --2- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0a3730 0x7fd7ac0a3b10 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd7a4016910 tx=0x7fd7a4016dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:13.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.316+0000 7fd7a1ffb640 10 client.44340.objecter ms_handle_connect 0x7fd7ac0a3730 2024-09-17T18:25:13.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.316+0000 7fd7a1ffb640 10 client.44340.objecter resend_mon_ops 2024-09-17T18:25:13.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.316+0000 7fd7a1ffb640 1 -- 172.21.15.121:0/538296681 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a400f040 con 0x7fd7ac0a3730 2024-09-17T18:25:13.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.316+0000 7fd7a1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:13.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.316+0000 7fd7a1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:25:13.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.316+0000 7fd7a1ffb640 10 monclient: dump: 2024-09-17T18:25:13.319 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:13.320 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:13.320 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:13.320 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:13.320 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:13.320 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:13.320 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:13.320 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:13.320 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:13.320 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:13.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.317+0000 7fd7a1ffb640 1 -- 172.21.15.121:0/538296681 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd7a401e070 con 0x7fd7ac0a3730 2024-09-17T18:25:13.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.317+0000 7fd7a1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:13.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.317+0000 7fd7a1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:25:13.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.317+0000 7fd7a1ffb640 1 -- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0a3730 msgr2=0x7fd7ac0a3b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:13.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.317+0000 7fd7a1ffb640 1 --2- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0a3730 0x7fd7ac0a3b10 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd7a4016910 tx=0x7fd7a4016dd0 comp rx=0 tx=0).stop 2024-09-17T18:25:13.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.317+0000 7fd7a1ffb640 1 -- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0a3730 msgr2=0x7fd7ac0a3b10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:25:13.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.317+0000 7fd7a1ffb640 1 --2- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0a3730 0x7fd7ac0a3b10 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd7a4016910 tx=0x7fd7a4016dd0 comp rx=0 tx=0).stop 2024-09-17T18:25:13.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.324+0000 7fd7b369b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:25:13.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.324+0000 7fd7b369b640 1 --2- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0a4050 0x7fd7ac09f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:13.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.324+0000 7fd7b369b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:13.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.324+0000 7fd7b369b640 1 -- 172.21.15.121:0/538296681 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd7ac1ce3a0 con 0x7fd7ac0a4050 2024-09-17T18:25:13.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.324+0000 7fd7b1437640 1 --2- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0a4050 0x7fd7ac09f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:13.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.324+0000 7fd7b1437640 10 monclient: get_auth_request con 0x7fd7ac0a4050 auth_method 0 2024-09-17T18:25:13.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.324+0000 7fd7b1437640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:13.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.324+0000 7fd7b1437640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:13.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.324+0000 7fd7b1437640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:13.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.324+0000 7fd7b1437640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:13.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.324+0000 7fd7b1437640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:13.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.324+0000 7fd7b1437640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:13.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.324+0000 7fd7b1437640 10 monclient(hunting): handle_auth_done global_id 54104 payload 824 2024-09-17T18:25:13.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.324+0000 7fd7b1437640 1 -- 172.21.15.121:0/538296681 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd7ac0a96a0 con 0x7fd7ac0a4050 2024-09-17T18:25:13.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.326+0000 7fd7b1437640 1 -- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0a4050 msgr2=0x7fd7ac09f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:13.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.326+0000 7fd7b1437640 1 --2- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0a4050 0x7fd7ac09f100 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd7a4016910 tx=0x7fd7a4004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:13.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.326+0000 7fd7a1ffb640 10 client.44340.objecter ms_handle_connect 0x7fd7ac0a4050 2024-09-17T18:25:13.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.326+0000 7fd7a1ffb640 10 client.44340.objecter resend_mon_ops 2024-09-17T18:25:13.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.327+0000 7fd7a1ffb640 1 -- 172.21.15.121:0/538296681 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a4003b10 con 0x7fd7ac0a4050 2024-09-17T18:25:13.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.327+0000 7fd7a1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:13.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.327+0000 7fd7a1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:25:13.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.327+0000 7fd7a1ffb640 10 monclient: dump: 2024-09-17T18:25:13.330 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:13.330 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:13.331 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:13.331 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:13.331 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:13.331 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:13.331 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:13.331 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:13.331 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:13.331 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:13.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.327+0000 7fd7a1ffb640 1 -- 172.21.15.121:0/538296681 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7fd7a4003b10 con 0x7fd7ac0a4050 2024-09-17T18:25:13.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.327+0000 7fd7a1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:25:13.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.327+0000 7fd7a1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:25:13.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.327+0000 7fd7a1ffb640 1 -- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0a4050 msgr2=0x7fd7ac09f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:13.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.327+0000 7fd7a1ffb640 1 --2- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0a4050 0x7fd7ac09f100 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd7a4016910 tx=0x7fd7a4004730 comp rx=0 tx=0).stop 2024-09-17T18:25:13.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.327+0000 7fd7a1ffb640 1 -- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0a4050 msgr2=0x7fd7ac09f100 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:25:13.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.327+0000 7fd7a1ffb640 1 --2- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7ac0a4050 0x7fd7ac09f100 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd7a4016910 tx=0x7fd7a4004730 comp rx=0 tx=0).stop 2024-09-17T18:25:13.332 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:25:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:25:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:25:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:25:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 26, 2024-09-17T18:25:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 15525, 2024-09-17T18:25:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:25:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:25:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:25:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:25:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:25:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 4, 2024-09-17T18:25:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:25:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:25:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:25:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:25:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:25:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:25:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:25:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:25:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:25:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:25:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:25:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:25:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:25:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:25:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:25:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:25:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:25:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:25:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:25:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:25:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:25:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:25:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:25:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:25:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:25:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:25:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:25:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:25:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:25:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:25:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:25:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:25:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:25:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:25:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:25:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:25:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:25:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:25:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:25:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:25:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:25:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:25:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:25:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:25:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:25:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:25:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:25:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:25:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:25:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:25:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:25:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:25:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:25:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:25:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:25:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:25:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:25:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:25:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:25:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:25:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:25:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:25:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:25:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:25:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:25:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:25:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:25:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:25:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:25:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:25:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:25:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:25:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:25:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:25:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:25:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:25:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:25:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:25:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:25:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:25:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:25:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:25:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:25:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:25:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:25:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:25:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:25:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:25:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:25:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:25:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:25:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:25:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:25:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:25:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:25:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:25:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:25:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:25:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:25:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:25:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:25:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:25:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:25:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:25:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:25:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:25:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:25:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:25:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:25:13.344 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:25:13.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.327+0000 7fd7b369b640 10 librados: watch_flush enter 2024-09-17T18:25:13.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.327+0000 7fd7b369b640 10 librados: watch_flush exit 2024-09-17T18:25:13.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.327+0000 7fd7b369b640 20 client.44340.objecter shutdown clearing up homeless session... 2024-09-17T18:25:13.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.327+0000 7fd7b369b640 10 client.44340.objecter successfully canceled tick 2024-09-17T18:25:13.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.327+0000 7fd7b369b640 1 -- 172.21.15.121:0/538296681 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd790040170 msgr2=0x7fd790042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:13.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.327+0000 7fd7b369b640 1 --2- 172.21.15.121:0/538296681 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd790040170 0x7fd790042630 secure :-1 s=READY pgs=3973 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac070720 tx=0x7fd79c02e000 comp rx=0 tx=0).stop 2024-09-17T18:25:13.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.328+0000 7fd7b369b640 10 monclient: shutdown 2024-09-17T18:25:13.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.328+0000 7fd7b1c38640 1 -- 172.21.15.121:0/538296681 reap_dead start 2024-09-17T18:25:13.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.328+0000 7fd7b369b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:25:13.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.328+0000 7fd7b369b640 1 -- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7ac09ec30 msgr2=0x7fd7ac132050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:13.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.328+0000 7fd7b369b640 1 --2- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7ac09ec30 0x7fd7ac132050 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd7a800be20 tx=0x7fd7a800bf10 comp rx=0 tx=0).stop 2024-09-17T18:25:13.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.328+0000 7fd7b369b640 1 -- 172.21.15.121:0/538296681 shutdown_connections 2024-09-17T18:25:13.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.328+0000 7fd7b369b640 1 --2- 172.21.15.121:0/538296681 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7ac09ec30 0x7fd7ac132050 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:13.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.328+0000 7fd7b369b640 1 -- 172.21.15.121:0/538296681 >> 172.21.15.121:0/538296681 conn(0x7fd7ac0ab830 msgr2=0x7fd7ac133280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:25:13.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.328+0000 7fd7b369b640 1 -- 172.21.15.121:0/538296681 shutdown_connections 2024-09-17T18:25:13.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.328+0000 7fd7b369b640 1 -- 172.21.15.121:0/538296681 wait complete. 2024-09-17T18:25:13.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.328+0000 7fd7b369b640 1 librados: shutdown 2024-09-17T18:25:13.347 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:25:13.347 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T18:25:13.347 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T18:25:13.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.456+0000 7fb6f449b640 10 monclient: get_monmap_and_config 2024-09-17T18:25:13.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.456+0000 7fb6f449b640 10 monclient: build_initial_monmap 2024-09-17T18:25:13.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.456+0000 7fb6f449b640 10 monclient: monmap: 2024-09-17T18:25:13.459 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:25:13.459 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:13.459 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:25:13.457679+0000 2024-09-17T18:25:13.459 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:25:13.457679+0000 2024-09-17T18:25:13.459 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:25:13.459 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:13.459 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:25:13.459 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:25:13.459 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:25:13.459 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:25:13.460 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:25:13.460 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:13.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.456+0000 7fb6f449b640 1 Processor -- start 2024-09-17T18:25:13.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.456+0000 7fb6f449b640 1 -- start start 2024-09-17T18:25:13.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.456+0000 7fb6f449b640 10 monclient: init 2024-09-17T18:25:13.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.457+0000 7fb6f449b640 10 monclient: _reopen_session rank -1 2024-09-17T18:25:13.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.457+0000 7fb6f449b640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-17T18:25:13.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.457+0000 7fb6f449b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb6ec0700f0 addr v1:172.21.15.173:6789/0 2024-09-17T18:25:13.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.457+0000 7fb6f449b640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb6ec09e810 0x7fb6ec13d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:13.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.457+0000 7fb6f449b640 10 monclient(hunting): picked mon.noname-a con 0x7fb6ec09e810 addr v2:172.21.15.121:3300/0 2024-09-17T18:25:13.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.457+0000 7fb6f449b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb6ec09ec60 0x7fb6ec13fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:13.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.457+0000 7fb6f449b640 10 monclient(hunting): picked mon.noname-b con 0x7fb6ec09ec60 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:25:13.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.457+0000 7fb6f449b640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6ec135560 con 0x7fb6ec0700f0 2024-09-17T18:25:13.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.457+0000 7fb6f449b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:13.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.457+0000 7fb6f449b640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fb6ec0704d0 con 0x7fb6ec09e810 2024-09-17T18:25:13.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.457+0000 7fb6f449b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:13.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.457+0000 7fb6f449b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb6ec09ca60 con 0x7fb6ec09ec60 2024-09-17T18:25:13.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.457+0000 7fb6f449b640 10 monclient(hunting): _renew_subs 2024-09-17T18:25:13.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.457+0000 7fb6f449b640 10 monclient(hunting): authenticate will time out at 1117.174435s 2024-09-17T18:25:13.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.457+0000 7fb6f1a36640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb6ec09e810 0x7fb6ec13d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:13.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.457+0000 7fb6f2a38640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb6ec09ec60 0x7fb6ec13fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:13.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.457+0000 7fb6f1a36640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb6ec09e810 0x7fb6ec13d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44534/0 (socket says 172.21.15.121:44534) 2024-09-17T18:25:13.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.457+0000 7fb6f1a36640 1 -- 172.21.15.121:0/4234309593 learned_addr learned my addr 172.21.15.121:0/4234309593 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:25:13.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.457+0000 7fb6f2a38640 10 monclient(hunting): get_auth_request con 0x7fb6ec09ec60 auth_method 0 2024-09-17T18:25:13.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.457+0000 7fb6f2a38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:13.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.457+0000 7fb6f2a38640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:13.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.457+0000 7fb6f2a38640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:13.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.457+0000 7fb6f1a36640 10 monclient(hunting): get_auth_request con 0x7fb6ec09e810 auth_method 0 2024-09-17T18:25:13.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.457+0000 7fb6f1a36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:13.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.457+0000 7fb6f1a36640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:13.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.457+0000 7fb6f1a36640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:13.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.457+0000 7fb6f2a38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:13.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.457+0000 7fb6f2a38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:13.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.457+0000 7fb6f2a38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:13.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.457+0000 7fb6f1a36640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:13.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.457+0000 7fb6f1a36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:13.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.457+0000 7fb6f1a36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:13.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.457+0000 7fb6f2a38640 10 monclient(hunting): handle_auth_done global_id 54110 payload 274 2024-09-17T18:25:13.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.457+0000 7fb6f2a38640 1 -- 172.21.15.121:0/4234309593 >> v2:172.21.15.121:3300/0 conn(0x7fb6ec09e810 msgr2=0x7fb6ec13d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:25:13.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.457+0000 7fb6f2a38640 1 --2- 172.21.15.121:0/4234309593 >> v2:172.21.15.121:3300/0 conn(0x7fb6ec09e810 0x7fb6ec13d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:13.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.457+0000 7fb6f2a38640 1 -- 172.21.15.121:0/4234309593 >> v1:172.21.15.173:6789/0 conn(0x7fb6ec0700f0 legacy=0x7fb6ec134e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:25:13.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.457+0000 7fb6f2a38640 10 monclient: _finish_hunting 0 2024-09-17T18:25:13.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.457+0000 7fb6f2a38640 1 monclient: found mon.noname-b 2024-09-17T18:25:13.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.457+0000 7fb6f2a38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:25:13.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.457+0000 7fb6f2a38640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:25:13.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.457+0000 7fb6f2a38640 1 -- 172.21.15.121:0/4234309593 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6ec09cd20 con 0x7fb6ec09ec60 2024-09-17T18:25:13.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.457+0000 7fb6f1a36640 1 --2- 172.21.15.121:0/4234309593 >> v2:172.21.15.121:3300/0 conn(0x7fb6ec09e810 0x7fb6ec13d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:25:13.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.460+0000 7fb6f2a38640 1 -- 172.21.15.121:0/4234309593 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb6ec09ec60 msgr2=0x7fb6ec13fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:13.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.460+0000 7fb6f2a38640 1 --2- 172.21.15.121:0/4234309593 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb6ec09ec60 0x7fb6ec13fc30 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc009d90 tx=0x7fb6dc02a490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da7b7689555c10d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:13.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.460+0000 7fb6f1235640 1 -- 172.21.15.121:0/4234309593 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6dc007950 con 0x7fb6ec09ec60 2024-09-17T18:25:13.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.460+0000 7fb6f1235640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:13.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.460+0000 7fb6f1235640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:25:13.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.460+0000 7fb6f1235640 10 monclient: dump: 2024-09-17T18:25:13.473 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:13.473 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:13.473 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:13.473 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:13.473 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:13.473 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:13.473 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:13.473 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:13.473 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:13.474 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:13.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.460+0000 7fb6f1235640 10 monclient: _finish_auth 0 2024-09-17T18:25:13.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.460+0000 7fb6f1235640 10 monclient: _check_auth_tickets 2024-09-17T18:25:13.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.460+0000 7fb6f1235640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:25:13.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.460+0000 7fb6f1235640 1 -- 172.21.15.121:0/4234309593 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6dc007af0 con 0x7fb6ec09ec60 2024-09-17T18:25:13.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.460+0000 7fb6f449b640 5 monclient: authenticate success, global_id 54110 2024-09-17T18:25:13.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.460+0000 7fb6f449b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:25:13.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.460+0000 7fb6f1235640 10 monclient: handle_config config(0 keys) 2024-09-17T18:25:13.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.460+0000 7fb6f449b640 10 monclient: get_monmap_and_config success 2024-09-17T18:25:13.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.460+0000 7fb6f1235640 1 -- 172.21.15.121:0/4234309593 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6dc031400 con 0x7fb6ec09ec60 2024-09-17T18:25:13.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.460+0000 7fb6f449b640 4 set_mon_vals no callback set 2024-09-17T18:25:13.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.460+0000 7fb6f1235640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:13.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.460+0000 7fb6f1235640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:25:13.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.460+0000 7fb6f1235640 10 monclient: dump: 2024-09-17T18:25:13.475 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:13.475 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:13.475 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:13.475 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:13.475 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:13.475 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:13.475 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:13.475 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:13.475 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:13.475 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:13.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.461+0000 7fb6f449b640 10 monclient: shutdown 2024-09-17T18:25:13.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.461+0000 7fb6f449b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:25:13.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.461+0000 7fb6f449b640 1 -- 172.21.15.121:0/4234309593 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb6ec09ec60 msgr2=0x7fb6ec13fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:13.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.461+0000 7fb6f449b640 1 --2- 172.21.15.121:0/4234309593 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb6ec09ec60 0x7fb6ec13fc30 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc009d90 tx=0x7fb6dc02a490 comp rx=0 tx=0).stop 2024-09-17T18:25:13.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.461+0000 7fb6f449b640 1 -- 172.21.15.121:0/4234309593 shutdown_connections 2024-09-17T18:25:13.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.461+0000 7fb6f449b640 1 --2- 172.21.15.121:0/4234309593 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb6ec09ec60 0x7fb6ec13fc30 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:13.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.461+0000 7fb6f449b640 1 --2- 172.21.15.121:0/4234309593 >> v2:172.21.15.121:3300/0 conn(0x7fb6ec09e810 0x7fb6ec13d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:13.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.461+0000 7fb6f449b640 1 -- 172.21.15.121:0/4234309593 >> 172.21.15.121:0/4234309593 conn(0x7fb6ec0ab830 msgr2=0x7fb6ec0a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:25:13.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.461+0000 7fb6f449b640 1 -- 172.21.15.121:0/4234309593 shutdown_connections 2024-09-17T18:25:13.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.461+0000 7fb6f449b640 1 -- 172.21.15.121:0/4234309593 wait complete. 2024-09-17T18:25:13.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.461+0000 7fb6f449b640 10 monclient: build_initial_monmap 2024-09-17T18:25:13.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.461+0000 7fb6f449b640 10 monclient: monmap: 2024-09-17T18:25:13.476 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:25:13.476 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:25:13.476 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:25:13.476 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:25:13.476 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:25:13.476 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:13.477 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:25:13.477 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:25:13.477 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:25:13.477 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:13.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.462+0000 7fb6f449b640 1 librados: starting msgr at 2024-09-17T18:25:13.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.462+0000 7fb6f449b640 1 librados: starting objecter 2024-09-17T18:25:13.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.462+0000 7fb6f449b640 1 Processor -- start 2024-09-17T18:25:13.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.462+0000 7fb6f449b640 1 -- start start 2024-09-17T18:25:13.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.462+0000 7fb6f449b640 1 librados: setting wanted keys 2024-09-17T18:25:13.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.462+0000 7fb6f449b640 1 librados: calling monclient init 2024-09-17T18:25:13.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.462+0000 7fb6f449b640 10 monclient: init 2024-09-17T18:25:13.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.462+0000 7fb6f449b640 10 monclient: _reopen_session rank -1 2024-09-17T18:25:13.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.462+0000 7fb6f449b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T18:25:13.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.462+0000 7fb6f449b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb6ec0700f0 0x7fb6ec131bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:13.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.462+0000 7fb6f449b640 10 monclient(hunting): picked mon.noname-c con 0x7fb6ec0700f0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:25:13.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.462+0000 7fb6f449b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6ec09e810 0x7fb6ec132100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:13.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.462+0000 7fb6f449b640 10 monclient(hunting): picked mon.noname-a con 0x7fb6ec09e810 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:25:13.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.462+0000 7fb6f449b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6ec09ec60 0x7fb6ec136d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:13.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.462+0000 7fb6f449b640 10 monclient(hunting): picked mon.noname-b con 0x7fb6ec09ec60 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:25:13.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.462+0000 7fb6f449b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:13.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.462+0000 7fb6f449b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb6ec140170 con 0x7fb6ec09e810 2024-09-17T18:25:13.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.462+0000 7fb6f449b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:13.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.462+0000 7fb6f449b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb6ec1402f0 con 0x7fb6ec09ec60 2024-09-17T18:25:13.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.462+0000 7fb6f449b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:13.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.462+0000 7fb6f449b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb6ec140470 con 0x7fb6ec0700f0 2024-09-17T18:25:13.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.462+0000 7fb6f449b640 10 monclient(hunting): _renew_subs 2024-09-17T18:25:13.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.462+0000 7fb6f449b640 10 monclient(hunting): authenticate will time out at 1117.179827s 2024-09-17T18:25:13.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.462+0000 7fb6f2237640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb6ec0700f0 0x7fb6ec131bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:13.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.462+0000 7fb6f1a36640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6ec09e810 0x7fb6ec132100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:13.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.462+0000 7fb6f2237640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb6ec0700f0 0x7fb6ec131bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:53486/0 (socket says 172.21.15.121:53486) 2024-09-17T18:25:13.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.462+0000 7fb6f1a36640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6ec09e810 0x7fb6ec132100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44536/0 (socket says 172.21.15.121:44536) 2024-09-17T18:25:13.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.462+0000 7fb6f2237640 1 -- 172.21.15.121:0/2630712881 learned_addr learned my addr 172.21.15.121:0/2630712881 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:25:13.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.462+0000 7fb6f2237640 10 monclient(hunting): get_auth_request con 0x7fb6ec0700f0 auth_method 0 2024-09-17T18:25:13.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.462+0000 7fb6f2237640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:13.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.462+0000 7fb6f2237640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:13.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.462+0000 7fb6f2237640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:13.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.462+0000 7fb6f1a36640 10 monclient(hunting): get_auth_request con 0x7fb6ec09e810 auth_method 0 2024-09-17T18:25:13.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.462+0000 7fb6f1a36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:13.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.462+0000 7fb6f1a36640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:13.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.462+0000 7fb6f1a36640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:13.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.462+0000 7fb6f2237640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:13.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.462+0000 7fb6f2237640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:13.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.463+0000 7fb6f2237640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:13.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.463+0000 7fb6f1a36640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:13.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.463+0000 7fb6f1a36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:13.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.463+0000 7fb6f1a36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:13.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.463+0000 7fb6f2a38640 1 --2- 172.21.15.121:0/2630712881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6ec09ec60 0x7fb6ec136d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:13.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.463+0000 7fb6f2a38640 10 monclient(hunting): get_auth_request con 0x7fb6ec09ec60 auth_method 0 2024-09-17T18:25:13.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.463+0000 7fb6f2a38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:13.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.463+0000 7fb6f2a38640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:13.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.463+0000 7fb6f2a38640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:13.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.463+0000 7fb6f2237640 10 monclient(hunting): handle_auth_done global_id 54113 payload 824 2024-09-17T18:25:13.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.463+0000 7fb6f2237640 1 -- 172.21.15.121:0/2630712881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6ec09ec60 msgr2=0x7fb6ec136d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:13.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.463+0000 7fb6f2237640 1 --2- 172.21.15.121:0/2630712881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6ec09ec60 0x7fb6ec136d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:13.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.463+0000 7fb6f2237640 1 -- 172.21.15.121:0/2630712881 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6ec09e810 msgr2=0x7fb6ec132100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:13.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.463+0000 7fb6f2237640 1 --2- 172.21.15.121:0/2630712881 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6ec09e810 0x7fb6ec132100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:13.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.463+0000 7fb6f2237640 10 monclient: _finish_hunting 0 2024-09-17T18:25:13.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.463+0000 7fb6f2237640 1 monclient: found mon.noname-c 2024-09-17T18:25:13.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.463+0000 7fb6f2237640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:25:13.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.463+0000 7fb6f2237640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:25:13.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.463+0000 7fb6f2237640 1 -- 172.21.15.121:0/2630712881 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6ec13adb0 con 0x7fb6ec0700f0 2024-09-17T18:25:13.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.463+0000 7fb6f1a36640 1 --2- 172.21.15.121:0/2630712881 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6ec09e810 0x7fb6ec132100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:25:13.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.465+0000 7fb6f2237640 1 -- 172.21.15.121:0/2630712881 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb6ec0700f0 msgr2=0x7fb6ec131bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:13.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.465+0000 7fb6f2237640 1 --2- 172.21.15.121:0/2630712881 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb6ec0700f0 0x7fb6ec131bc0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb6e400c460 tx=0x7fb6e400c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:13.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.465+0000 7fb6eaffd640 10 client.?.objecter ms_handle_connect 0x7fb6ec0700f0 2024-09-17T18:25:13.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.465+0000 7fb6eaffd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:25:13.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.465+0000 7fb6eaffd640 1 -- 172.21.15.121:0/2630712881 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6e4016020 con 0x7fb6ec0700f0 2024-09-17T18:25:13.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.465+0000 7fb6eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:13.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.465+0000 7fb6eaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:25:13.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.465+0000 7fb6eaffd640 10 monclient: dump: 2024-09-17T18:25:13.482 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:13.482 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:13.482 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:13.482 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:13.482 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:13.482 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:13.482 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:13.482 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:13.482 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:13.482 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:13.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.465+0000 7fb6eaffd640 10 monclient: _finish_auth 0 2024-09-17T18:25:13.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.465+0000 7fb6eaffd640 10 monclient: _check_auth_tickets 2024-09-17T18:25:13.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.465+0000 7fb6eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:25:13.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.465+0000 7fb6eaffd640 1 -- 172.21.15.121:0/2630712881 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6e4012a80 con 0x7fb6ec0700f0 2024-09-17T18:25:13.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.465+0000 7fb6f449b640 5 monclient: authenticate success, global_id 54113 2024-09-17T18:25:13.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.465+0000 7fb6f449b640 10 monclient: _renew_subs 2024-09-17T18:25:13.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.465+0000 7fb6f449b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:25:13.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.465+0000 7fb6f449b640 1 -- 172.21.15.121:0/2630712881 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6ec0a23f0 con 0x7fb6ec0700f0 2024-09-17T18:25:13.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.465+0000 7fb6eaffd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:25:13.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.465+0000 7fb6eaffd640 1 -- 172.21.15.121:0/2630712881 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6e4011380 con 0x7fb6ec0700f0 2024-09-17T18:25:13.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.465+0000 7fb6eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:13.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.465+0000 7fb6eaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:25:13.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.465+0000 7fb6eaffd640 10 monclient: dump: 2024-09-17T18:25:13.483 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:13.483 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:13.483 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:13.483 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:13.484 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:13.484 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:13.484 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:13.484 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:13.484 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:13.484 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:13.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.465+0000 7fb6f449b640 10 client.54113.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:25:13.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.465+0000 7fb6f449b640 10 monclient: _renew_subs 2024-09-17T18:25:13.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.465+0000 7fb6f449b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:25:13.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.465+0000 7fb6f0a34640 4 set_mon_vals no callback set 2024-09-17T18:25:13.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.465+0000 7fb6f449b640 1 -- 172.21.15.121:0/2630712881 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6ec132640 con 0x7fb6ec0700f0 2024-09-17T18:25:13.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.465+0000 7fb6f449b640 1 librados: init done 2024-09-17T18:25:13.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.466+0000 7fb6eaffd640 1 -- 172.21.15.121:0/2630712881 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fb6e4019020 con 0x7fb6ec0700f0 2024-09-17T18:25:13.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.466+0000 7fb6f449b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:13.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.466+0000 7fb6f449b640 1 --2- 172.21.15.121:0/2630712881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6ec09f100 0x7fb6ec09f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:13.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.466+0000 7fb6f449b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:13.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.466+0000 7fb6f449b640 1 -- 172.21.15.121:0/2630712881 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb6ec09d0b0 con 0x7fb6ec09f100 2024-09-17T18:25:13.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.466+0000 7fb6eaffd640 1 --2- 172.21.15.121:0/2630712881 >> v2:172.21.15.173:6812/2423786037 conn(0x7fb6d8040170 0x7fb6d8042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:13.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.466+0000 7fb6eaffd640 1 -- 172.21.15.121:0/2630712881 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(285..285 src has 1..285) ==== 5376+0+0 (secure 0 0 0) 0x7fb6e4053170 con 0x7fb6ec0700f0 2024-09-17T18:25:13.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.466+0000 7fb6eaffd640 10 client.54113.objecter ms_dispatch 0x7fb6ec000b90 osd_map(285..285 src has 1..285) 2024-09-17T18:25:13.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.466+0000 7fb6eaffd640 3 client.54113.objecter handle_osd_map got epochs [285,285] > 0 2024-09-17T18:25:13.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.466+0000 7fb6eaffd640 3 client.54113.objecter handle_osd_map decoding full epoch 285 2024-09-17T18:25:13.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.466+0000 7fb6f1a36640 1 --2- 172.21.15.121:0/2630712881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6ec09f100 0x7fb6ec09f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:13.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.466+0000 7fb6f1a36640 10 monclient: get_auth_request con 0x7fb6ec09f100 auth_method 0 2024-09-17T18:25:13.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.466+0000 7fb6f1a36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:13.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.466+0000 7fb6f1a36640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:13.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.466+0000 7fb6eaffd640 20 client.54113.objecter dump_active .. 0 homeless 2024-09-17T18:25:13.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.466+0000 7fb6f1a36640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:13.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.466+0000 7fb6f2a38640 1 --2- 172.21.15.121:0/2630712881 >> v2:172.21.15.173:6812/2423786037 conn(0x7fb6d8040170 0x7fb6d8042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:13.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.466+0000 7fb6f2a38640 10 monclient: get_auth_request con 0x7fb6d8040170 auth_method 0 2024-09-17T18:25:13.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.466+0000 7fb6f1a36640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:13.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.466+0000 7fb6f1a36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:13.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.466+0000 7fb6f1a36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:13.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.467+0000 7fb6f1a36640 10 monclient(hunting): handle_auth_done global_id 24697 payload 824 2024-09-17T18:25:13.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.467+0000 7fb6f1a36640 1 -- 172.21.15.121:0/2630712881 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb6ec134e50 con 0x7fb6ec09f100 2024-09-17T18:25:13.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.469+0000 7fb6eaffd640 10 client.54113.objecter ms_handle_connect 0x7fb6d8040170 2024-09-17T18:25:13.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.469+0000 7fb6f2a38640 1 --2- 172.21.15.121:0/2630712881 >> v2:172.21.15.173:6812/2423786037 conn(0x7fb6d8040170 0x7fb6d8042630 secure :-1 s=READY pgs=3975 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc009d90 tx=0x7fb6dc02f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:13.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.469+0000 7fb6eaffd640 10 client.54113.objecter ms_handle_connect 0x7fb6ec09f100 2024-09-17T18:25:13.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.469+0000 7fb6eaffd640 10 client.54113.objecter resend_mon_ops 2024-09-17T18:25:13.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.469+0000 7fb6f1a36640 1 -- 172.21.15.121:0/2630712881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6ec09f100 msgr2=0x7fb6ec09f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:13.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.469+0000 7fb6f1a36640 1 --2- 172.21.15.121:0/2630712881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6ec09f100 0x7fb6ec09f4e0 secure :-1 s=READY pgs=7377 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec139ed0 tx=0x7fb6e000f610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:13.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.469+0000 7fb6eaffd640 1 -- 172.21.15.121:0/2630712881 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6e0014650 con 0x7fb6ec09f100 2024-09-17T18:25:13.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.469+0000 7fb6eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:13.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.469+0000 7fb6eaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:25:13.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.469+0000 7fb6eaffd640 10 monclient: dump: 2024-09-17T18:25:13.487 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:13.487 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:13.487 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:13.487 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:13.487 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:13.487 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:13.487 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:13.487 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:13.487 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:13.487 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:13.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.469+0000 7fb6eaffd640 1 -- 172.21.15.121:0/2630712881 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb6e0014650 con 0x7fb6ec09f100 2024-09-17T18:25:13.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.469+0000 7fb6eaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:13.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.469+0000 7fb6eaffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:25:13.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.469+0000 7fb6eaffd640 1 -- 172.21.15.121:0/2630712881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6ec09f100 msgr2=0x7fb6ec09f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:13.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.469+0000 7fb6eaffd640 1 --2- 172.21.15.121:0/2630712881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6ec09f100 0x7fb6ec09f4e0 secure :-1 s=READY pgs=7377 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec139ed0 tx=0x7fb6e000f610 comp rx=0 tx=0).stop 2024-09-17T18:25:13.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.469+0000 7fb6eaffd640 1 -- 172.21.15.121:0/2630712881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6ec09f100 msgr2=0x7fb6ec09f4e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:25:13.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.469+0000 7fb6eaffd640 1 --2- 172.21.15.121:0/2630712881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6ec09f100 0x7fb6ec09f4e0 secure :-1 s=CLOSED pgs=7377 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec139ed0 tx=0x7fb6e000f610 comp rx=0 tx=0).stop 2024-09-17T18:25:13.489 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.475+0000 7fb6f449b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:25:13.489 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.475+0000 7fb6f449b640 1 --2- 172.21.15.121:0/2630712881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6ec09fa20 0x7fb6ec137240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:13.489 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.475+0000 7fb6f449b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:13.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.475+0000 7fb6f449b640 1 -- 172.21.15.121:0/2630712881 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb6ec06a660 con 0x7fb6ec09fa20 2024-09-17T18:25:13.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.476+0000 7fb6f1a36640 1 --2- 172.21.15.121:0/2630712881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6ec09fa20 0x7fb6ec137240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:13.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.476+0000 7fb6f1a36640 10 monclient: get_auth_request con 0x7fb6ec09fa20 auth_method 0 2024-09-17T18:25:13.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.476+0000 7fb6f1a36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:13.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.476+0000 7fb6f1a36640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:13.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.476+0000 7fb6f1a36640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:13.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.476+0000 7fb6f1a36640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:13.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.476+0000 7fb6f1a36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:13.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.476+0000 7fb6f1a36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:13.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.476+0000 7fb6f1a36640 10 monclient(hunting): handle_auth_done global_id 24700 payload 824 2024-09-17T18:25:13.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.476+0000 7fb6f1a36640 1 -- 172.21.15.121:0/2630712881 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb6ec1395b0 con 0x7fb6ec09fa20 2024-09-17T18:25:13.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.478+0000 7fb6f1a36640 1 -- 172.21.15.121:0/2630712881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6ec09fa20 msgr2=0x7fb6ec137240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:13.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.478+0000 7fb6eaffd640 10 client.54113.objecter ms_handle_connect 0x7fb6ec09fa20 2024-09-17T18:25:13.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.478+0000 7fb6f1a36640 1 --2- 172.21.15.121:0/2630712881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6ec09fa20 0x7fb6ec137240 secure :-1 s=READY pgs=7378 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec139ed0 tx=0x7fb6e00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:13.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.478+0000 7fb6eaffd640 10 client.54113.objecter resend_mon_ops 2024-09-17T18:25:13.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.478+0000 7fb6eaffd640 1 -- 172.21.15.121:0/2630712881 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6e0022b90 con 0x7fb6ec09fa20 2024-09-17T18:25:13.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.478+0000 7fb6eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:13.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.478+0000 7fb6eaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:25:13.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.478+0000 7fb6eaffd640 10 monclient: dump: 2024-09-17T18:25:13.491 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:13.491 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:13.491 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:13.491 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:13.491 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:13.491 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:13.491 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:13.491 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:13.491 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:13.491 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:13.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.479+0000 7fb6eaffd640 1 -- 172.21.15.121:0/2630712881 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7fb6e001d400 con 0x7fb6ec09fa20 2024-09-17T18:25:13.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.479+0000 7fb6eaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:25:13.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.479+0000 7fb6eaffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:25:13.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.479+0000 7fb6eaffd640 1 -- 172.21.15.121:0/2630712881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6ec09fa20 msgr2=0x7fb6ec137240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:13.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.479+0000 7fb6eaffd640 1 --2- 172.21.15.121:0/2630712881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6ec09fa20 0x7fb6ec137240 secure :-1 s=READY pgs=7378 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec139ed0 tx=0x7fb6e00046c0 comp rx=0 tx=0).stop 2024-09-17T18:25:13.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.479+0000 7fb6eaffd640 1 -- 172.21.15.121:0/2630712881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6ec09fa20 msgr2=0x7fb6ec137240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:25:13.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.479+0000 7fb6eaffd640 1 --2- 172.21.15.121:0/2630712881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6ec09fa20 0x7fb6ec137240 unknown :-1 s=CLOSED pgs=7378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:13.492 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:25:13.492 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:25:13.492 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:25:13.492 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:25:13.493 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 26, 2024-09-17T18:25:13.493 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 386425, 2024-09-17T18:25:13.493 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:25:13.493 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:25:13.493 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:25:13.493 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:25:13.493 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:25:13.493 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 5, 2024-09-17T18:25:13.493 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:25:13.493 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:25:13.493 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:25:13.493 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:25:13.493 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:25:13.493 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:25:13.494 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:25:13.494 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:25:13.494 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:25:13.494 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:25:13.494 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:25:13.494 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:25:13.494 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:25:13.494 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:25:13.494 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:25:13.494 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:25:13.494 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:25:13.494 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:25:13.494 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:25:13.494 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:25:13.494 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:25:13.495 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:25:13.495 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:25:13.495 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:25:13.495 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:25:13.495 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:25:13.495 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:25:13.495 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:25:13.495 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:13.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:13.495 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:25:13.495 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:25:13.495 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:25:13.495 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:25:13.495 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:25:13.495 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:25:13.496 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:25:13.496 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:25:13.496 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:25:13.496 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:25:13.496 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:25:13.496 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:25:13.496 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:25:13.496 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:25:13.496 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:25:13.496 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:25:13.496 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:25:13.496 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:25:13.496 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:25:13.496 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:25:13.496 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:25:13.497 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:25:13.497 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:25:13.497 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:25:13.497 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:25:13.497 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:25:13.497 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:25:13.497 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:25:13.497 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:25:13.497 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:25:13.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:13.497 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:25:13.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:13.497 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:25:13.497 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:25:13.498 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:25:13.498 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:25:13.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:13.498 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:25:13.498 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:25:13.498 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:13.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:13.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:13.498 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:25:13.498 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:25:13.498 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:13.498 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:13.498 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:13.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:13.498 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:25:13.499 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:25:13.499 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:25:13.499 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:25:13.499 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:25:13.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:13.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:13.499 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:25:13.499 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:25:13.499 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:25:13.499 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:25:13.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:13.499 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:25:13.499 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:25:13.500 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:13.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:13.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:13.500 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:25:13.500 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:25:13.500 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:13.500 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:13.500 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:13.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:13.500 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:25:13.500 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:25:13.500 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:25:13.500 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:25:13.500 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:25:13.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:13.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:13.501 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:25:13.501 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:25:13.501 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:25:13.501 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:25:13.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:13.501 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:25:13.501 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:25:13.501 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:13.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:13.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:13.501 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:25:13.501 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:25:13.501 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:13.501 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:13.502 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:13.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:13.502 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:25:13.502 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:25:13.502 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:25:13.502 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:25:13.502 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:25:13.502 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:13.502 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:13.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:13.502 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:25:13.502 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:25:13.502 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:13.502 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:25:13.503 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:25:13.503 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:25:13.503 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:13.503 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:25:13.503 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:25:13.503 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:13.503 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:25:13.503 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:25:13.503 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 4 2024-09-17T18:25:13.503 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:13.503 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:25:13.503 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:25:13.503 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:13.503 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:25:13.504 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:25:13.504 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 7 2024-09-17T18:25:13.504 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:13.504 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:25:13.504 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": [ 2024-09-17T18:25:13.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:13.504 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:25:13.504 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:25:13.504 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:25:13.504 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:13.504 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:13.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:13.504 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:25:13.505 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:25:13.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.479+0000 7fb6f449b640 10 librados: watch_flush enter 2024-09-17T18:25:13.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.479+0000 7fb6f449b640 10 librados: watch_flush exit 2024-09-17T18:25:13.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.479+0000 7fb6f449b640 20 client.54113.objecter shutdown clearing up homeless session... 2024-09-17T18:25:13.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.479+0000 7fb6f449b640 10 client.54113.objecter successfully canceled tick 2024-09-17T18:25:13.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.479+0000 7fb6f449b640 1 -- 172.21.15.121:0/2630712881 >> v2:172.21.15.173:6812/2423786037 conn(0x7fb6d8040170 msgr2=0x7fb6d8042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:13.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.479+0000 7fb6f449b640 1 --2- 172.21.15.121:0/2630712881 >> v2:172.21.15.173:6812/2423786037 conn(0x7fb6d8040170 0x7fb6d8042630 secure :-1 s=READY pgs=3975 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc009d90 tx=0x7fb6dc02f000 comp rx=0 tx=0).stop 2024-09-17T18:25:13.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.479+0000 7fb6f449b640 10 monclient: shutdown 2024-09-17T18:25:13.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.479+0000 7fb6f449b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:25:13.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.479+0000 7fb6f449b640 1 -- 172.21.15.121:0/2630712881 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb6ec0700f0 msgr2=0x7fb6ec131bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:13.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.479+0000 7fb6f449b640 1 --2- 172.21.15.121:0/2630712881 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb6ec0700f0 0x7fb6ec131bc0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb6e400c460 tx=0x7fb6e400c920 comp rx=0 tx=0).stop 2024-09-17T18:25:13.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.479+0000 7fb6f2a38640 1 -- 172.21.15.121:0/2630712881 reap_dead start 2024-09-17T18:25:13.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.479+0000 7fb6f449b640 1 -- 172.21.15.121:0/2630712881 shutdown_connections 2024-09-17T18:25:13.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.479+0000 7fb6f449b640 1 --2- 172.21.15.121:0/2630712881 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb6ec0700f0 0x7fb6ec131bc0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:13.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.479+0000 7fb6f449b640 1 -- 172.21.15.121:0/2630712881 >> 172.21.15.121:0/2630712881 conn(0x7fb6ec0ab830 msgr2=0x7fb6ec0a9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:25:13.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.479+0000 7fb6f449b640 1 -- 172.21.15.121:0/2630712881 shutdown_connections 2024-09-17T18:25:13.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.479+0000 7fb6f449b640 1 -- 172.21.15.121:0/2630712881 wait complete. 2024-09-17T18:25:13.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.480+0000 7fb6f449b640 1 librados: shutdown 2024-09-17T18:25:13.508 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:25:13.508 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-17T18:25:13.508 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-17T18:25:13.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.601+0000 7f0dad416640 10 monclient: get_monmap_and_config 2024-09-17T18:25:13.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.601+0000 7f0dad416640 10 monclient: build_initial_monmap 2024-09-17T18:25:13.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.601+0000 7f0dad416640 10 monclient: monmap: 2024-09-17T18:25:13.604 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:25:13.605 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:13.605 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:25:13.603446+0000 2024-09-17T18:25:13.605 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:25:13.603446+0000 2024-09-17T18:25:13.605 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:25:13.605 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:13.605 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:25:13.605 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:25:13.605 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:25:13.605 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:25:13.605 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:25:13.605 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:13.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.602+0000 7f0dad416640 1 Processor -- start 2024-09-17T18:25:13.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.602+0000 7f0dad416640 1 -- start start 2024-09-17T18:25:13.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.602+0000 7f0dad416640 10 monclient: init 2024-09-17T18:25:13.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.602+0000 7f0dad416640 10 monclient: _reopen_session rank -1 2024-09-17T18:25:13.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.602+0000 7f0dad416640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-17T18:25:13.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.602+0000 7f0dad416640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0da80a8620 0x7f0da80a5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:13.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.602+0000 7f0dad416640 10 monclient(hunting): picked mon.noname-a con 0x7f0da80a8620 addr v2:172.21.15.121:3300/0 2024-09-17T18:25:13.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.602+0000 7f0dad416640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0da80a6350 addr v1:172.21.15.173:6789/0 2024-09-17T18:25:13.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.602+0000 7f0dad416640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0da80a6730 0x7f0da8140320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:13.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.602+0000 7f0dad416640 10 monclient(hunting): picked mon.noname-b con 0x7f0da80a6730 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:25:13.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.602+0000 7f0dad416640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0da809f0e0 con 0x7f0da80a6350 2024-09-17T18:25:13.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.602+0000 7f0dad416640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:13.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.602+0000 7f0dad416640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f0da809cf50 con 0x7f0da80a8620 2024-09-17T18:25:13.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.602+0000 7f0dad416640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:13.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.602+0000 7f0dad416640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0da8070900 con 0x7f0da80a6730 2024-09-17T18:25:13.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.602+0000 7f0dad416640 10 monclient(hunting): _renew_subs 2024-09-17T18:25:13.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.602+0000 7f0dad416640 10 monclient(hunting): authenticate will time out at 1117.319985s 2024-09-17T18:25:13.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.603+0000 7f0da6ffd640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0da80a8620 0x7f0da80a5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:13.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.603+0000 7f0da77fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0da80a6730 0x7f0da8140320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:13.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.603+0000 7f0da6ffd640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0da80a8620 0x7f0da80a5e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44546/0 (socket says 172.21.15.121:44546) 2024-09-17T18:25:13.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.603+0000 7f0da6ffd640 1 -- 172.21.15.121:0/2026581079 learned_addr learned my addr 172.21.15.121:0/2026581079 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:25:13.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.603+0000 7f0da77fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0da80a6730 0x7f0da8140320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:53488/0 (socket says 172.21.15.121:53488) 2024-09-17T18:25:13.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.603+0000 7f0da6ffd640 10 monclient(hunting): get_auth_request con 0x7f0da80a8620 auth_method 0 2024-09-17T18:25:13.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.603+0000 7f0da6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:13.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.603+0000 7f0da6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:13.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.603+0000 7f0da6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:13.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.603+0000 7f0da77fe640 10 monclient(hunting): get_auth_request con 0x7f0da80a6730 auth_method 0 2024-09-17T18:25:13.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.603+0000 7f0da77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:13.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.603+0000 7f0da77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:13.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.603+0000 7f0da77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:13.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.603+0000 7f0da6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:13.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.603+0000 7f0da6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:13.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.603+0000 7f0da6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:13.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.603+0000 7f0da77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:13.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.603+0000 7f0da77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:13.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.603+0000 7f0da77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:13.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.603+0000 7f0da6ffd640 10 monclient(hunting): handle_auth_done global_id 44352 payload 274 2024-09-17T18:25:13.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.603+0000 7f0da6ffd640 1 -- 172.21.15.121:0/2026581079 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0da80a6730 msgr2=0x7f0da8140320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:25:13.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.603+0000 7f0da6ffd640 1 --2- 172.21.15.121:0/2026581079 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0da80a6730 0x7f0da8140320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:13.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.603+0000 7f0da6ffd640 1 -- 172.21.15.121:0/2026581079 >> v1:172.21.15.173:6789/0 conn(0x7f0da80a6350 legacy=0x7f0da813cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:25:13.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.603+0000 7f0da6ffd640 10 monclient: _finish_hunting 0 2024-09-17T18:25:13.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.603+0000 7f0da6ffd640 1 monclient: found mon.noname-a 2024-09-17T18:25:13.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.603+0000 7f0da6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:25:13.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.603+0000 7f0da6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:25:13.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.603+0000 7f0da6ffd640 1 -- 172.21.15.121:0/2026581079 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0da809cc70 con 0x7f0da80a8620 2024-09-17T18:25:13.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.606+0000 7f0da6ffd640 1 -- 172.21.15.121:0/2026581079 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0da80a8620 msgr2=0x7f0da80a5e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:13.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.606+0000 7f0da6ffd640 1 --2- 172.21.15.121:0/2026581079 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0da80a8620 0x7f0da80a5e10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0da8077c30 tx=0x7f0d9802a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=194efc9bbc1934b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:13.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.606+0000 7f0da67fc640 1 -- 172.21.15.121:0/2026581079 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d98037070 con 0x7f0da80a8620 2024-09-17T18:25:13.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.606+0000 7f0da67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:13.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.606+0000 7f0da67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:25:13.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.606+0000 7f0da67fc640 10 monclient: dump: 2024-09-17T18:25:13.617 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:13.617 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:13.617 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:13.617 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:13.617 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:13.617 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:13.617 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:13.617 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:13.617 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:13.617 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:13.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.606+0000 7f0da67fc640 10 monclient: _finish_auth 0 2024-09-17T18:25:13.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.606+0000 7f0da67fc640 10 monclient: _check_auth_tickets 2024-09-17T18:25:13.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.606+0000 7f0da67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:25:13.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.606+0000 7f0dad416640 5 monclient: authenticate success, global_id 44352 2024-09-17T18:25:13.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.606+0000 7f0dad416640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:25:13.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.606+0000 7f0da67fc640 1 -- 172.21.15.121:0/2026581079 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d98032640 con 0x7f0da80a8620 2024-09-17T18:25:13.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.606+0000 7f0da67fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:25:13.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.606+0000 7f0dad416640 10 monclient: get_monmap_and_config success 2024-09-17T18:25:13.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.606+0000 7f0dad416640 4 set_mon_vals no callback set 2024-09-17T18:25:13.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.606+0000 7f0da67fc640 1 -- 172.21.15.121:0/2026581079 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d9802ae60 con 0x7f0da80a8620 2024-09-17T18:25:13.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.606+0000 7f0da67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:13.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.606+0000 7f0da67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:25:13.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.606+0000 7f0da67fc640 10 monclient: dump: 2024-09-17T18:25:13.618 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:13.618 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:13.618 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:13.619 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:13.619 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:13.619 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:13.619 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:13.619 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:13.619 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:13.619 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:13.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.606+0000 7f0dad416640 10 monclient: shutdown 2024-09-17T18:25:13.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.606+0000 7f0dad416640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:25:13.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.606+0000 7f0dad416640 1 -- 172.21.15.121:0/2026581079 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0da80a8620 msgr2=0x7f0da80a5e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:13.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.606+0000 7f0dad416640 1 --2- 172.21.15.121:0/2026581079 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0da80a8620 0x7f0da80a5e10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0da8077c30 tx=0x7f0d9802a100 comp rx=0 tx=0).stop 2024-09-17T18:25:13.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.606+0000 7f0dad416640 1 -- 172.21.15.121:0/2026581079 shutdown_connections 2024-09-17T18:25:13.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.606+0000 7f0dad416640 1 --2- 172.21.15.121:0/2026581079 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0da80a6730 0x7f0da8140320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:13.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.606+0000 7f0dad416640 1 --2- 172.21.15.121:0/2026581079 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0da80a8620 0x7f0da80a5e10 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:13.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.606+0000 7f0dad416640 1 -- 172.21.15.121:0/2026581079 >> 172.21.15.121:0/2026581079 conn(0x7f0da80abcb0 msgr2=0x7f0da80a7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:25:13.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.606+0000 7f0dad416640 1 -- 172.21.15.121:0/2026581079 shutdown_connections 2024-09-17T18:25:13.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.606+0000 7f0dad416640 1 -- 172.21.15.121:0/2026581079 wait complete. 2024-09-17T18:25:13.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.607+0000 7f0dad416640 10 monclient: build_initial_monmap 2024-09-17T18:25:13.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.607+0000 7f0dad416640 10 monclient: monmap: 2024-09-17T18:25:13.620 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:25:13.620 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:25:13.620 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:25:13.620 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:25:13.620 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:25:13.621 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:13.621 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:25:13.621 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:25:13.621 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:25:13.621 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:13.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.607+0000 7f0dad416640 1 librados: starting msgr at 2024-09-17T18:25:13.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.607+0000 7f0dad416640 1 librados: starting objecter 2024-09-17T18:25:13.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.607+0000 7f0dad416640 1 Processor -- start 2024-09-17T18:25:13.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.607+0000 7f0dad416640 1 -- start start 2024-09-17T18:25:13.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.607+0000 7f0dad416640 1 librados: setting wanted keys 2024-09-17T18:25:13.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.607+0000 7f0dad416640 1 librados: calling monclient init 2024-09-17T18:25:13.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.607+0000 7f0dad416640 10 monclient: init 2024-09-17T18:25:13.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0dad416640 10 monclient: _reopen_session rank -1 2024-09-17T18:25:13.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0dad416640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T18:25:13.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0dad416640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0da80a6350 0x7f0da81adaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:13.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0dad416640 10 monclient(hunting): picked mon.noname-c con 0x7f0da80a6350 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:25:13.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0dad416640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0da80a6730 0x7f0da81cde60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:13.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0dad416640 10 monclient(hunting): picked mon.noname-b con 0x7f0da80a6730 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:25:13.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0dad416640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0da80a8620 0x7f0da81c7840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:13.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0dad416640 10 monclient(hunting): picked mon.noname-a con 0x7f0da80a8620 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:25:13.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0dad416640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:13.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0dad416640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0da80a6b80 con 0x7f0da80a8620 2024-09-17T18:25:13.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0dad416640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:13.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0dad416640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0da80706f0 con 0x7f0da80a6730 2024-09-17T18:25:13.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0dad416640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:13.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0dad416640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0da80a7490 con 0x7f0da80a6350 2024-09-17T18:25:13.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0dad416640 10 monclient(hunting): _renew_subs 2024-09-17T18:25:13.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0dad416640 10 monclient(hunting): authenticate will time out at 1117.325376s 2024-09-17T18:25:13.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0da6ffd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0da80a6350 0x7f0da81adaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:13.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0da6ffd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0da80a6350 0x7f0da81adaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:53500/0 (socket says 172.21.15.121:53500) 2024-09-17T18:25:13.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0da77fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0da80a8620 0x7f0da81c7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:13.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0da6ffd640 1 -- 172.21.15.121:0/1280114712 learned_addr learned my addr 172.21.15.121:0/1280114712 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:25:13.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0da6ffd640 10 monclient(hunting): get_auth_request con 0x7f0da80a6350 auth_method 0 2024-09-17T18:25:13.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0da6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:13.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0da6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:13.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0da6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:13.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0da77fe640 10 monclient(hunting): get_auth_request con 0x7f0da80a8620 auth_method 0 2024-09-17T18:25:13.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0da77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:13.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0da77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:13.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0da77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:13.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0da6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:13.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0da6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:13.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0da6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:13.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0d9e7fc640 1 --2- 172.21.15.121:0/1280114712 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0da80a6730 0x7f0da81cde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:13.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0da77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:13.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0da77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:13.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0da77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:13.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0d9e7fc640 10 monclient(hunting): get_auth_request con 0x7f0da80a6730 auth_method 0 2024-09-17T18:25:13.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0d9e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:13.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0d9e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:13.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0d9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:13.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0d9e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:13.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0d9e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:13.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0d9e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:13.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0da77fe640 10 monclient(hunting): handle_auth_done global_id 44355 payload 824 2024-09-17T18:25:13.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0da77fe640 1 -- 172.21.15.121:0/1280114712 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0da80a6350 msgr2=0x7f0da81adaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:13.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0da77fe640 1 --2- 172.21.15.121:0/1280114712 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0da80a6350 0x7f0da81adaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:13.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0da77fe640 1 -- 172.21.15.121:0/1280114712 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0da80a6730 msgr2=0x7f0da81cde60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:13.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0da77fe640 1 --2- 172.21.15.121:0/1280114712 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0da80a6730 0x7f0da81cde60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:13.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0da77fe640 10 monclient: _finish_hunting 0 2024-09-17T18:25:13.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0da77fe640 1 monclient: found mon.noname-a 2024-09-17T18:25:13.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0da77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:25:13.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0da77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:25:13.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0da77fe640 1 -- 172.21.15.121:0/1280114712 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0da81ae430 con 0x7f0da80a8620 2024-09-17T18:25:13.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.608+0000 7f0da6ffd640 1 --2- 172.21.15.121:0/1280114712 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0da80a6350 0x7f0da81adaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:25:13.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.610+0000 7f0da77fe640 1 -- 172.21.15.121:0/1280114712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0da80a8620 msgr2=0x7f0da81c7840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:13.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.610+0000 7f0da77fe640 1 --2- 172.21.15.121:0/1280114712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0da80a8620 0x7f0da81c7840 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0da000efc0 tx=0x7f0da000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:13.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.610+0000 7f0d9ffff640 10 client.?.objecter ms_handle_connect 0x7f0da80a8620 2024-09-17T18:25:13.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.610+0000 7f0d9ffff640 10 client.?.objecter resend_mon_ops 2024-09-17T18:25:13.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.611+0000 7f0d9ffff640 1 -- 172.21.15.121:0/1280114712 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da0018020 con 0x7f0da80a8620 2024-09-17T18:25:13.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.611+0000 7f0d9ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:13.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.611+0000 7f0d9ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:25:13.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.611+0000 7f0d9ffff640 10 monclient: dump: 2024-09-17T18:25:13.626 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:13.626 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:13.626 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:13.626 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:13.626 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:13.626 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:13.626 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:13.626 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:13.626 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:13.626 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:13.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.611+0000 7f0d9ffff640 10 monclient: _finish_auth 0 2024-09-17T18:25:13.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.611+0000 7f0d9ffff640 10 monclient: _check_auth_tickets 2024-09-17T18:25:13.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.611+0000 7f0d9ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:25:13.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.611+0000 7f0dad416640 5 monclient: authenticate success, global_id 44355 2024-09-17T18:25:13.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.611+0000 7f0d9ffff640 1 -- 172.21.15.121:0/1280114712 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0da00146a0 con 0x7f0da80a8620 2024-09-17T18:25:13.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.611+0000 7f0dad416640 10 monclient: _renew_subs 2024-09-17T18:25:13.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.611+0000 7f0dad416640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:13.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.611+0000 7f0dad416640 1 -- 172.21.15.121:0/1280114712 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0da80a2600 con 0x7f0da80a8620 2024-09-17T18:25:13.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.611+0000 7f0d9ffff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:25:13.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.611+0000 7f0d9ffff640 1 -- 172.21.15.121:0/1280114712 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da0014a90 con 0x7f0da80a8620 2024-09-17T18:25:13.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.611+0000 7f0d9ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:13.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.611+0000 7f0d9ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:25:13.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.611+0000 7f0d9ffff640 10 monclient: dump: 2024-09-17T18:25:13.627 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:13.627 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:13.627 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:13.628 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:13.628 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:13.628 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:13.628 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:13.628 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:13.628 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:13.628 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:13.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.611+0000 7f0dad416640 10 client.44355.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:25:13.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.611+0000 7f0dad416640 10 monclient: _renew_subs 2024-09-17T18:25:13.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.611+0000 7f0da5ffb640 4 set_mon_vals no callback set 2024-09-17T18:25:13.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.611+0000 7f0dad416640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:13.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.611+0000 7f0dad416640 1 -- 172.21.15.121:0/1280114712 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0da81af2a0 con 0x7f0da80a8620 2024-09-17T18:25:13.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.611+0000 7f0dad416640 1 librados: init done 2024-09-17T18:25:13.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.612+0000 7f0dad416640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:13.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.612+0000 7f0dad416640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:13.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.612+0000 7f0dad416640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:13.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.612+0000 7f0d9ffff640 1 -- 172.21.15.121:0/1280114712 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f0da0005d10 con 0x7f0da80a8620 2024-09-17T18:25:13.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.612+0000 7f0dad416640 1 -- 172.21.15.121:0/1280114712 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0da806feb0 con 0x7f0da80a8620 2024-09-17T18:25:13.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.612+0000 7f0d9ffff640 1 --2- 172.21.15.121:0/1280114712 >> v2:172.21.15.173:6812/2423786037 conn(0x7f0d880401a0 0x7f0d88042660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:13.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.612+0000 7f0d9ffff640 1 -- 172.21.15.121:0/1280114712 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(285..285 src has 1..285) ==== 5376+0+0 (secure 0 0 0) 0x7f0da00545c0 con 0x7f0da80a8620 2024-09-17T18:25:13.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.612+0000 7f0d9ffff640 10 client.44355.objecter ms_dispatch 0x7f0da8000b90 osd_map(285..285 src has 1..285) 2024-09-17T18:25:13.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.612+0000 7f0d9ffff640 3 client.44355.objecter handle_osd_map got epochs [285,285] > 0 2024-09-17T18:25:13.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.612+0000 7f0d9ffff640 3 client.44355.objecter handle_osd_map decoding full epoch 285 2024-09-17T18:25:13.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.612+0000 7f0d9ffff640 20 client.44355.objecter dump_active .. 0 homeless 2024-09-17T18:25:13.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.612+0000 7f0da6ffd640 1 --2- 172.21.15.121:0/1280114712 >> v2:172.21.15.173:6812/2423786037 conn(0x7f0d880401a0 0x7f0d88042660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:13.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.612+0000 7f0da6ffd640 10 monclient: get_auth_request con 0x7f0d880401a0 auth_method 0 2024-09-17T18:25:13.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.615+0000 7f0d9ffff640 10 client.44355.objecter ms_handle_connect 0x7f0d880401a0 2024-09-17T18:25:13.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.615+0000 7f0da6ffd640 1 --2- 172.21.15.121:0/1280114712 >> v2:172.21.15.173:6812/2423786037 conn(0x7f0d880401a0 0x7f0d88042660 secure :-1 s=READY pgs=3976 cs=0 l=1 rev1=1 crypto rx=0x7f0d9802a140 tx=0x7f0d9802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:13.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.616+0000 7f0d9ffff640 1 -- 172.21.15.121:0/1280114712 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0da806feb0 con 0x7f0da80a8620 2024-09-17T18:25:13.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.616+0000 7f0d9ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:13.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.616+0000 7f0d9ffff640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:25:13.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.857+0000 7f0dad416640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-17T18:25:13.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.857+0000 7f0dad416640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-17T18:25:13.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.857+0000 7f0dad416640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:13.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.857+0000 7f0dad416640 1 -- 172.21.15.121:0/1280114712 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f0da80a5e10 con 0x7f0da80a8620 2024-09-17T18:25:13.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.858+0000 7f0d9ffff640 1 -- 172.21.15.121:0/1280114712 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f0da0022030 con 0x7f0da80a8620 2024-09-17T18:25:13.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.858+0000 7f0d9ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-17T18:25:13.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.858+0000 7f0d9ffff640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:25:13.862 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.859+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.862 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.859+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=1,health=12,kv=14,logm=61} crc {auth=646690931,config=1228700967,health=534850548,kv=2790417503,logm=375355179}) 2024-09-17T18:25:13.862 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.859+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=11,config=2,health=12,kv=14,logm=61} crc {auth=2053333124,config=2586702516,health=534850548,kv=2790417503,logm=1060306737}) 2024-09-17T18:25:13.862 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.859+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.862 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.859+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=1,health=12,kv=14,logm=61} crc {auth=646690931,config=1228700967,health=534850548,kv=2790417503,logm=375355179}) 2024-09-17T18:25:13.862 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.859+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=10,config=2,health=11,kv=14,logm=63} crc {auth=2080260829,config=2586702516,health=1490792222,kv=2790417503,logm=2351214233}) 2024-09-17T18:25:13.863 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.860+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.863 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.860+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1783658627}) 2024-09-17T18:25:13.863 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.860+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2261429931}) 2024-09-17T18:25:13.863 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.860+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.863 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.860+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1783658627}) 2024-09-17T18:25:13.863 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.860+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2533595266}) 2024-09-17T18:25:13.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.860+0000 7f0dad416640 10 librados: watch_flush enter 2024-09-17T18:25:13.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.861+0000 7f0dad416640 10 librados: watch_flush exit 2024-09-17T18:25:13.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.861+0000 7f0dad416640 20 client.44355.objecter shutdown clearing up homeless session... 2024-09-17T18:25:13.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.861+0000 7f0dad416640 10 client.44355.objecter successfully canceled tick 2024-09-17T18:25:13.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.861+0000 7f0dad416640 1 -- 172.21.15.121:0/1280114712 >> v2:172.21.15.173:6812/2423786037 conn(0x7f0d880401a0 msgr2=0x7f0d88042660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:13.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.861+0000 7f0dad416640 1 --2- 172.21.15.121:0/1280114712 >> v2:172.21.15.173:6812/2423786037 conn(0x7f0d880401a0 0x7f0d88042660 secure :-1 s=READY pgs=3976 cs=0 l=1 rev1=1 crypto rx=0x7f0d9802a140 tx=0x7f0d9802e000 comp rx=0 tx=0).stop 2024-09-17T18:25:13.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.861+0000 7f0dad416640 10 monclient: shutdown 2024-09-17T18:25:13.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.861+0000 7f0dad416640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:25:13.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.861+0000 7f0dad416640 1 -- 172.21.15.121:0/1280114712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0da80a8620 msgr2=0x7f0da81c7840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:13.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.861+0000 7f0dad416640 1 --2- 172.21.15.121:0/1280114712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0da80a8620 0x7f0da81c7840 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0da000efc0 tx=0x7f0da000c490 comp rx=0 tx=0).stop 2024-09-17T18:25:13.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.861+0000 7f0dad416640 1 -- 172.21.15.121:0/1280114712 shutdown_connections 2024-09-17T18:25:13.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.861+0000 7f0dad416640 1 --2- 172.21.15.121:0/1280114712 >> v2:172.21.15.173:6812/2423786037 conn(0x7f0d880401a0 0x7f0d88042660 unknown :-1 s=CLOSED pgs=3976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:13.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.861+0000 7f0dad416640 1 --2- 172.21.15.121:0/1280114712 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0da80a6730 0x7f0da81cde60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:13.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.861+0000 7f0dad416640 1 --2- 172.21.15.121:0/1280114712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0da80a8620 0x7f0da81c7840 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:13.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.861+0000 7f0dad416640 1 --2- 172.21.15.121:0/1280114712 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0da80a6350 0x7f0da81adaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:13.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.861+0000 7f0dad416640 1 -- 172.21.15.121:0/1280114712 >> 172.21.15.121:0/1280114712 conn(0x7f0da80abcb0 msgr2=0x7f0da80aa1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:25:13.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.861+0000 7f0dad416640 1 -- 172.21.15.121:0/1280114712 shutdown_connections 2024-09-17T18:25:13.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.861+0000 7f0dad416640 1 -- 172.21.15.121:0/1280114712 wait complete. 2024-09-17T18:25:13.866 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.861+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.866 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.861+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=27304692}) 2024-09-17T18:25:13.866 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.861+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2053941678}) 2024-09-17T18:25:13.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:13.861+0000 7f0dad416640 1 librados: shutdown 2024-09-17T18:25:13.869 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.861+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.869 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.861+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=27304692}) 2024-09-17T18:25:13.869 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.861+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=236856444}) 2024-09-17T18:25:13.869 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.862+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.869 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.862+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1183276339}) 2024-09-17T18:25:13.869 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.862+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=860945794}) 2024-09-17T18:25:13.869 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.862+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.869 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.862+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1183276339}) 2024-09-17T18:25:13.869 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.862+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1534036516}) 2024-09-17T18:25:13.869 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.863+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.869 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.863+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=810235659}) 2024-09-17T18:25:13.869 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.863+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=742015546}) 2024-09-17T18:25:13.869 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.863+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.870 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.863+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=810235659}) 2024-09-17T18:25:13.870 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.863+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1134376475}) 2024-09-17T18:25:13.870 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.864+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.870 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.864+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1730833382}) 2024-09-17T18:25:13.870 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.864+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1673184927}) 2024-09-17T18:25:13.870 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.864+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.870 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.864+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1730833382}) 2024-09-17T18:25:13.870 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.864+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=663936864}) 2024-09-17T18:25:13.870 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.865+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.870 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.865+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4236035506}) 2024-09-17T18:25:13.870 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.865+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3864398424}) 2024-09-17T18:25:13.870 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.865+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.870 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.865+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4236035506}) 2024-09-17T18:25:13.870 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.865+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2892632139}) 2024-09-17T18:25:13.870 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.866+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.871 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.866+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=855218305}) 2024-09-17T18:25:13.871 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.866+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=393458595}) 2024-09-17T18:25:13.871 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.866+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.871 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.866+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=855218305}) 2024-09-17T18:25:13.871 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.866+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2227914423}) 2024-09-17T18:25:13.872 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.867+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.872 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.867+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2016548691}) 2024-09-17T18:25:13.873 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.867+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1838948059}) 2024-09-17T18:25:13.873 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.867+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.873 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.867+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2016548691}) 2024-09-17T18:25:13.873 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.867+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3661860846}) 2024-09-17T18:25:13.873 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.868+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.873 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.868+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4007898306}) 2024-09-17T18:25:13.873 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.868+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3110094458}) 2024-09-17T18:25:13.873 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.868+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.873 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.868+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4007898306}) 2024-09-17T18:25:13.873 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.868+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1469184952}) 2024-09-17T18:25:13.873 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.869+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.873 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.869+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1774172833}) 2024-09-17T18:25:13.874 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.869+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4179561920}) 2024-09-17T18:25:13.874 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.869+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.874 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.869+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1774172833}) 2024-09-17T18:25:13.874 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.869+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=975535532}) 2024-09-17T18:25:13.874 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.870+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.874 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.870+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2674340798}) 2024-09-17T18:25:13.874 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.870+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1689331579}) 2024-09-17T18:25:13.874 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.870+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.874 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.870+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2674340798}) 2024-09-17T18:25:13.874 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.870+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1191390297}) 2024-09-17T18:25:13.875 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.871+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.875 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.871+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1763629225}) 2024-09-17T18:25:13.875 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.871+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2575436542}) 2024-09-17T18:25:13.875 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.871+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.875 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.871+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1763629225}) 2024-09-17T18:25:13.875 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.871+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4172140954}) 2024-09-17T18:25:13.876 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-17T18:25:13.877 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.872+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.878 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.872+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1195304385}) 2024-09-17T18:25:13.878 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.872+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1262420357}) 2024-09-17T18:25:13.878 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.872+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.878 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.872+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1195304385}) 2024-09-17T18:25:13.878 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.872+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1830462847}) 2024-09-17T18:25:13.878 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.873+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.878 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.873+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3640662469}) 2024-09-17T18:25:13.878 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.873+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3059742421}) 2024-09-17T18:25:13.878 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.873+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.878 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.873+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3640662469}) 2024-09-17T18:25:13.878 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.873+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3115238706}) 2024-09-17T18:25:13.878 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.874+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.878 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.874+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3880348510}) 2024-09-17T18:25:13.879 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.874+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2613154483}) 2024-09-17T18:25:13.879 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.874+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.879 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.874+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3880348510}) 2024-09-17T18:25:13.879 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.874+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1436363799}) 2024-09-17T18:25:13.879 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.874+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.879 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.874+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=985052081}) 2024-09-17T18:25:13.879 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.875+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=7847801}) 2024-09-17T18:25:13.879 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.875+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.879 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.875+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=985052081}) 2024-09-17T18:25:13.879 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.875+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2151875052}) 2024-09-17T18:25:13.880 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.875+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.881 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.875+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1712921382}) 2024-09-17T18:25:13.881 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.875+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=522064229}) 2024-09-17T18:25:13.881 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.875+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.881 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.875+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1712921382}) 2024-09-17T18:25:13.881 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.875+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2690013769}) 2024-09-17T18:25:13.881 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.876+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.881 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.876+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=49378488}) 2024-09-17T18:25:13.881 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.876+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2869457513}) 2024-09-17T18:25:13.881 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.876+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.881 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.876+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=49378488}) 2024-09-17T18:25:13.881 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.876+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3207040462}) 2024-09-17T18:25:13.882 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.877+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.882 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.877+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2625598584}) 2024-09-17T18:25:13.882 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.877+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3985670781}) 2024-09-17T18:25:13.882 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.877+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.882 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.877+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2625598584}) 2024-09-17T18:25:13.882 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.877+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3400417738}) 2024-09-17T18:25:13.884 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.878+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.884 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.878+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1500973441}) 2024-09-17T18:25:13.884 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.878+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=784670304}) 2024-09-17T18:25:13.884 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.878+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.884 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.878+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1500973441}) 2024-09-17T18:25:13.884 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.878+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=687821825}) 2024-09-17T18:25:13.884 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.879+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.884 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.879+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=31977562}) 2024-09-17T18:25:13.884 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.879+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=737202045}) 2024-09-17T18:25:13.884 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.879+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.884 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.879+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=31977562}) 2024-09-17T18:25:13.884 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.879+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1676729820}) 2024-09-17T18:25:13.885 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.880+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.885 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.880+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2075332608}) 2024-09-17T18:25:13.885 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.880+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=603310328}) 2024-09-17T18:25:13.885 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.880+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.885 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.880+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2075332608}) 2024-09-17T18:25:13.885 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.880+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2563583223}) 2024-09-17T18:25:13.885 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.881+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.885 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.881+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=773127770}) 2024-09-17T18:25:13.885 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.881+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1293043070}) 2024-09-17T18:25:13.885 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.881+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.885 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.881+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=773127770}) 2024-09-17T18:25:13.885 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.881+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3028215704}) 2024-09-17T18:25:13.887 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.882+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.887 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.882+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2234765085}) 2024-09-17T18:25:13.887 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.882+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1099636548}) 2024-09-17T18:25:13.887 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.882+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.887 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.882+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2234765085}) 2024-09-17T18:25:13.887 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.882+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2983614147}) 2024-09-17T18:25:13.887 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.883+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.887 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.883+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1514028932}) 2024-09-17T18:25:13.888 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.883+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2031492490}) 2024-09-17T18:25:13.888 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.883+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.888 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.883+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1514028932}) 2024-09-17T18:25:13.888 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.883+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1333608618}) 2024-09-17T18:25:13.888 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.884+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.888 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.884+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1689311353}) 2024-09-17T18:25:13.888 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.884+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3033209709}) 2024-09-17T18:25:13.888 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.884+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.888 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.884+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1689311353}) 2024-09-17T18:25:13.888 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.884+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3434697842}) 2024-09-17T18:25:13.889 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.885+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.890 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.885+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2598157054}) 2024-09-17T18:25:13.890 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.885+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2631292904}) 2024-09-17T18:25:13.890 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.885+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.890 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.885+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2598157054}) 2024-09-17T18:25:13.890 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.885+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2021803278}) 2024-09-17T18:25:13.890 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.885+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.890 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.885+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3771405299}) 2024-09-17T18:25:13.890 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.885+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1627306258}) 2024-09-17T18:25:13.890 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.885+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.890 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.886+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3771405299}) 2024-09-17T18:25:13.890 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.886+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2578867943}) 2024-09-17T18:25:13.890 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.886+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.890 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.886+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4223874690}) 2024-09-17T18:25:13.891 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.886+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3750965207}) 2024-09-17T18:25:13.891 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.886+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.891 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.886+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4223874690}) 2024-09-17T18:25:13.891 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.886+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4157465345}) 2024-09-17T18:25:13.892 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.887+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.892 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.887+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3516487037}) 2024-09-17T18:25:13.892 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.887+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2423037366}) 2024-09-17T18:25:13.892 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.887+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.892 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.887+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3516487037}) 2024-09-17T18:25:13.892 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.887+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3062824398}) 2024-09-17T18:25:13.892 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.889+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.892 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.889+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=70,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=8} crc {logm=2679039690,mdsmap=761013554,mgr=3361498112,mgr_command_descs=1957073259,mgr_metadata=402068389,mgrstat=3912554003,mon_config_key=564713193}) 2024-09-17T18:25:13.892 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.889+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=71,mdsmap=3,mgr=6,mgr_command_descs=1,mgrstat=8,mon_config_key=11} crc {logm=3550225287,mdsmap=761013554,mgr=1740323263,mgr_command_descs=1957073259,mgrstat=2003478351,mon_config_key=1659789312}) 2024-09-17T18:25:13.892 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.889+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.893 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.889+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=70,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=8} crc {logm=2679039690,mdsmap=761013554,mgr=3361498112,mgr_command_descs=1957073259,mgr_metadata=402068389,mgrstat=3912554003,mon_config_key=564713193}) 2024-09-17T18:25:13.893 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.889+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=69,mdsmap=3,mgr=7,mgr_command_descs=1,mgrstat=10,mon_config_key=10} crc {logm=3565915972,mdsmap=761013554,mgr=834655845,mgr_command_descs=1957073259,mgrstat=2609695811,mon_config_key=3924080977}) 2024-09-17T18:25:13.893 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.890+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.893 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.890+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=9,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=22,osdmap=56} crc {mon_config_key=3417260645,monmap=2277444362,nvmeofgw=598337686,osd_metadata=2978216528,osd_pg_creating=3863582991,osd_snap=2959930728,osdmap=2828479580}) 2024-09-17T18:25:13.893 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.890+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=9,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23,osdmap=55} crc {mon_config_key=3417260645,monmap=2277444362,nvmeofgw=598337686,osd_metadata=121115626,osd_pg_creating=3863582992,osd_snap=24258210,osdmap=3209444272}) 2024-09-17T18:25:13.894 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.890+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.894 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.890+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=9,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=22,osdmap=56} crc {mon_config_key=3417260645,monmap=2277444362,nvmeofgw=598337686,osd_metadata=2978216528,osd_pg_creating=3863582991,osd_snap=2959930728,osdmap=2828479580}) 2024-09-17T18:25:13.894 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.890+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=9,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=21,osdmap=58} crc {mon_config_key=3417260645,monmap=2277444362,nvmeofgw=1596038039,osd_metadata=121115626,osd_pg_creating=3863582991,osd_snap=3397806526,osdmap=4106847791}) 2024-09-17T18:25:13.894 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.891+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.894 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.891+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1141918204}) 2024-09-17T18:25:13.894 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.891+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=723328625}) 2024-09-17T18:25:13.894 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.891+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:13.895 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.891+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1141918204}) 2024-09-17T18:25:13.895 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:13.891+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2457319786}) 2024-09-17T18:25:14.015 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:14.012+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:14.015 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:14.012+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=199581991}) 2024-09-17T18:25:14.015 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:14.012+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2885579430}) 2024-09-17T18:25:14.016 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:14.012+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:14.016 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:14.012+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=199581991}) 2024-09-17T18:25:14.016 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:14.012+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=21982489}) 2024-09-17T18:25:14.017 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:14.014+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:14.017 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:14.014+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1220925350}) 2024-09-17T18:25:14.017 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:14.014+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=839328396}) 2024-09-17T18:25:14.018 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:14.015+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:14.018 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:14.015+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1220925350}) 2024-09-17T18:25:14.018 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:14.015+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=680178707}) 2024-09-17T18:25:14.029 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:14.026+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:14.029 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:14.026+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=62} crc {osdmap=3671264090}) 2024-09-17T18:25:14.029 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:14.026+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=63} crc {osdmap=834698566}) 2024-09-17T18:25:14.029 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:14.026+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:25:14.029 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:14.026+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=62} crc {osdmap=3671264090}) 2024-09-17T18:25:14.029 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:25:14.026+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=62} crc {osdmap=3889355990}) 2024-09-17T18:25:14.877 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T18:25:14.877 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:25:14.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.975+0000 7fab62031640 10 monclient: get_monmap_and_config 2024-09-17T18:25:14.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.975+0000 7fab62031640 10 monclient: build_initial_monmap 2024-09-17T18:25:14.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.975+0000 7fab62031640 10 monclient: monmap: 2024-09-17T18:25:14.979 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:25:14.979 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:14.979 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:25:14.977513+0000 2024-09-17T18:25:14.979 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:25:14.977513+0000 2024-09-17T18:25:14.979 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:25:14.979 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:14.979 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:25:14.979 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:25:14.979 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:25:14.979 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:25:14.979 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:25:14.979 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:14.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.976+0000 7fab62031640 1 Processor -- start 2024-09-17T18:25:14.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.976+0000 7fab62031640 1 -- start start 2024-09-17T18:25:14.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.976+0000 7fab62031640 10 monclient: init 2024-09-17T18:25:14.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.977+0000 7fab62031640 10 monclient: _reopen_session rank -1 2024-09-17T18:25:14.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.977+0000 7fab62031640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-17T18:25:14.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.977+0000 7fab62031640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fab5c136800 0x7fab5c136170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:14.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.977+0000 7fab62031640 10 monclient(hunting): picked mon.noname-c con 0x7fab5c136800 addr v2:172.21.15.173:3300/0 2024-09-17T18:25:14.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.977+0000 7fab62031640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fab5c136be0 addr v1:172.21.15.173:6789/0 2024-09-17T18:25:14.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.977+0000 7fab62031640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fab5c133360 addr v1:172.21.15.121:6789/0 2024-09-17T18:25:14.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.977+0000 7fab62031640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab5c0a0c20 con 0x7fab5c133360 2024-09-17T18:25:14.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.977+0000 7fab62031640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab5c0a2cb0 con 0x7fab5c136be0 2024-09-17T18:25:14.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.977+0000 7fab62031640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:14.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.977+0000 7fab62031640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fab5c0a2f90 con 0x7fab5c136800 2024-09-17T18:25:14.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.977+0000 7fab62031640 10 monclient(hunting): _renew_subs 2024-09-17T18:25:14.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.977+0000 7fab62031640 10 monclient(hunting): authenticate will time out at 1118.694218s 2024-09-17T18:25:14.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.977+0000 7fab5bfff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fab5c133360 0x7fab5c13e510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:38364/0 (socket says 172.21.15.121:38364) 2024-09-17T18:25:14.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.977+0000 7fab5bfff640 1 -- 172.21.15.121:0/2593288618 learned_addr learned my addr 172.21.15.121:0/2593288618 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:25:14.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.977+0000 7fab5b7fe640 1 --2- 172.21.15.121:0/2593288618 >> v2:172.21.15.173:3300/0 conn(0x7fab5c136800 0x7fab5c136170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:14.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.977+0000 7fab5b7fe640 10 monclient(hunting): get_auth_request con 0x7fab5c136800 auth_method 0 2024-09-17T18:25:14.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.977+0000 7fab5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:14.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.977+0000 7fab5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:14.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.977+0000 7fab5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:14.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.977+0000 7fab5b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:14.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.977+0000 7fab5b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:14.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.977+0000 7fab5b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:14.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.977+0000 7fab5b7fe640 10 monclient(hunting): handle_auth_done global_id 24706 payload 274 2024-09-17T18:25:14.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.977+0000 7fab5b7fe640 1 -- 172.21.15.121:0/2593288618 >> v1:172.21.15.173:6789/0 conn(0x7fab5c136be0 legacy=0x7fab5c132c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:25:14.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.977+0000 7fab5b7fe640 1 -- 172.21.15.121:0/2593288618 >> v1:172.21.15.121:6789/0 conn(0x7fab5c133360 legacy=0x7fab5c13e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:25:14.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.977+0000 7fab5b7fe640 10 monclient: _finish_hunting 0 2024-09-17T18:25:14.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.977+0000 7fab5b7fe640 1 monclient: found mon.noname-c 2024-09-17T18:25:14.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.977+0000 7fab5b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:25:14.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.977+0000 7fab5b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.173:3300/0 2024-09-17T18:25:14.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.977+0000 7fab5b7fe640 1 -- 172.21.15.121:0/2593288618 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab5c0a3270 con 0x7fab5c136800 2024-09-17T18:25:14.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.980+0000 7fab5b7fe640 1 -- 172.21.15.121:0/2593288618 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab5c136800 msgr2=0x7fab5c136170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:14.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.980+0000 7fab5b7fe640 1 --2- 172.21.15.121:0/2593288618 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab5c136800 0x7fab5c136170 secure :-1 s=READY pgs=7381 cs=0 l=1 rev1=1 crypto rx=0x7fab4c009d90 tx=0x7fab4c02a490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d233f6a55a3f6472 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:14.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.981+0000 7fab5a7fc640 1 -- 172.21.15.121:0/2593288618 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab4c007a30 con 0x7fab5c136800 2024-09-17T18:25:14.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.981+0000 7fab5a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:14.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.981+0000 7fab5a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:25:14.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.981+0000 7fab5a7fc640 10 monclient: dump: 2024-09-17T18:25:14.991 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:14.991 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:14.991 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:14.991 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:14.991 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:14.991 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:14.991 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:14.991 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:14.991 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:14.991 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:14.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.981+0000 7fab5a7fc640 10 monclient: _finish_auth 0 2024-09-17T18:25:14.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.981+0000 7fab5a7fc640 10 monclient: _check_auth_tickets 2024-09-17T18:25:14.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.981+0000 7fab5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:25:14.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.981+0000 7fab5a7fc640 1 -- 172.21.15.121:0/2593288618 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab4c007bd0 con 0x7fab5c136800 2024-09-17T18:25:14.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.981+0000 7fab5a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:25:14.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.981+0000 7fab5a7fc640 1 -- 172.21.15.121:0/2593288618 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab4c031400 con 0x7fab5c136800 2024-09-17T18:25:14.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.981+0000 7fab62031640 5 monclient: authenticate success, global_id 24706 2024-09-17T18:25:14.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.981+0000 7fab62031640 10 monclient: get_monmap_and_config success 2024-09-17T18:25:14.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.981+0000 7fab62031640 4 set_mon_vals no callback set 2024-09-17T18:25:14.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.981+0000 7fab5a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:14.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.981+0000 7fab5a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:25:14.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.981+0000 7fab5a7fc640 10 monclient: dump: 2024-09-17T18:25:14.992 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:14.992 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:14.992 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:14.992 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:14.992 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:14.993 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:14.993 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:14.993 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:14.993 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:14.993 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:14.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.981+0000 7fab62031640 10 monclient: shutdown 2024-09-17T18:25:14.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.981+0000 7fab62031640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:25:14.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.981+0000 7fab62031640 1 -- 172.21.15.121:0/2593288618 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab5c136800 msgr2=0x7fab5c136170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:14.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.981+0000 7fab62031640 1 --2- 172.21.15.121:0/2593288618 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab5c136800 0x7fab5c136170 secure :-1 s=READY pgs=7381 cs=0 l=1 rev1=1 crypto rx=0x7fab4c009d90 tx=0x7fab4c02a490 comp rx=0 tx=0).stop 2024-09-17T18:25:14.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.981+0000 7fab62031640 1 -- 172.21.15.121:0/2593288618 shutdown_connections 2024-09-17T18:25:14.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.981+0000 7fab62031640 1 --2- 172.21.15.121:0/2593288618 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab5c136800 0x7fab5c136170 unknown :-1 s=CLOSED pgs=7381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:14.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.981+0000 7fab62031640 1 -- 172.21.15.121:0/2593288618 >> 172.21.15.121:0/2593288618 conn(0x7fab5c0a3b50 msgr2=0x7fab5c0a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:25:14.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.981+0000 7fab62031640 1 -- 172.21.15.121:0/2593288618 shutdown_connections 2024-09-17T18:25:14.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.981+0000 7fab62031640 1 -- 172.21.15.121:0/2593288618 wait complete. 2024-09-17T18:25:14.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.982+0000 7fab62031640 10 monclient: build_initial_monmap 2024-09-17T18:25:14.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.982+0000 7fab62031640 10 monclient: monmap: 2024-09-17T18:25:14.994 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:25:14.994 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:25:14.994 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:25:14.994 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:25:14.994 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:25:14.994 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:14.994 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:25:14.994 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:25:14.994 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:25:14.994 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:14.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.982+0000 7fab62031640 1 librados: starting msgr at 2024-09-17T18:25:14.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.982+0000 7fab62031640 1 librados: starting objecter 2024-09-17T18:25:14.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.982+0000 7fab62031640 1 Processor -- start 2024-09-17T18:25:14.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.982+0000 7fab62031640 1 -- start start 2024-09-17T18:25:14.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.982+0000 7fab62031640 1 librados: setting wanted keys 2024-09-17T18:25:14.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.982+0000 7fab62031640 1 librados: calling monclient init 2024-09-17T18:25:14.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.982+0000 7fab62031640 10 monclient: init 2024-09-17T18:25:14.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab62031640 10 monclient: _reopen_session rank -1 2024-09-17T18:25:14.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab62031640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T18:25:14.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab62031640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab5c133360 0x7fab5c1301c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:14.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab62031640 10 monclient(hunting): picked mon.noname-a con 0x7fab5c133360 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:25:14.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab62031640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab5c136800 0x7fab5c073820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:14.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab62031640 10 monclient(hunting): picked mon.noname-c con 0x7fab5c136800 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:25:14.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab62031640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab5c136be0 0x7fab5c09e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:14.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab62031640 10 monclient(hunting): picked mon.noname-b con 0x7fab5c136be0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:25:14.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab62031640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:14.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab62031640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fab5c09dce0 con 0x7fab5c133360 2024-09-17T18:25:14.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab62031640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:14.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab62031640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fab5c137050 con 0x7fab5c136be0 2024-09-17T18:25:14.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab62031640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:14.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab62031640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fab5c09de60 con 0x7fab5c136800 2024-09-17T18:25:14.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab62031640 10 monclient(hunting): _renew_subs 2024-09-17T18:25:14.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab62031640 10 monclient(hunting): authenticate will time out at 1118.700223s 2024-09-17T18:25:14.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab5b7fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab5c133360 0x7fab5c1301c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:14.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab5affd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab5c136800 0x7fab5c073820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:14.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab5b7fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab5c133360 0x7fab5c1301c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44566/0 (socket says 172.21.15.121:44566) 2024-09-17T18:25:14.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab5b7fe640 1 -- 172.21.15.121:0/1559462161 learned_addr learned my addr 172.21.15.121:0/1559462161 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:25:14.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab5b7fe640 10 monclient(hunting): get_auth_request con 0x7fab5c133360 auth_method 0 2024-09-17T18:25:14.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:14.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:14.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:14.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab5affd640 10 monclient(hunting): get_auth_request con 0x7fab5c136800 auth_method 0 2024-09-17T18:25:14.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:14.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab5affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:14.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab5affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:14.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab5affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:14.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab5affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:14.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab5affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:14.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab5b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:14.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab5b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:14.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab5bfff640 1 --2- 172.21.15.121:0/1559462161 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab5c136be0 0x7fab5c09e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:14.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab5b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:14.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab5bfff640 10 monclient(hunting): get_auth_request con 0x7fab5c136be0 auth_method 0 2024-09-17T18:25:14.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:14.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:14.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:14.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:14.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:14.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:14.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab5affd640 10 monclient(hunting): handle_auth_done global_id 54122 payload 824 2024-09-17T18:25:14.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab5affd640 1 -- 172.21.15.121:0/1559462161 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab5c136be0 msgr2=0x7fab5c09e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:14.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab5affd640 1 --2- 172.21.15.121:0/1559462161 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab5c136be0 0x7fab5c09e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:14.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab5affd640 1 -- 172.21.15.121:0/1559462161 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab5c133360 msgr2=0x7fab5c1301c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:14.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab5affd640 1 --2- 172.21.15.121:0/1559462161 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab5c133360 0x7fab5c1301c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:14.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab5affd640 10 monclient: _finish_hunting 0 2024-09-17T18:25:14.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab5affd640 1 monclient: found mon.noname-c 2024-09-17T18:25:14.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab5affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:25:14.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab5affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:25:14.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab5affd640 1 -- 172.21.15.121:0/1559462161 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab5c071f30 con 0x7fab5c136800 2024-09-17T18:25:14.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.983+0000 7fab5b7fe640 1 --2- 172.21.15.121:0/1559462161 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab5c133360 0x7fab5c1301c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:25:14.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.985+0000 7fab5affd640 1 -- 172.21.15.121:0/1559462161 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab5c136800 msgr2=0x7fab5c073820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:14.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.985+0000 7fab5affd640 1 --2- 172.21.15.121:0/1559462161 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab5c136800 0x7fab5c073820 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fab5000c410 tx=0x7fab5000c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:14.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.985+0000 7fab43fff640 10 client.?.objecter ms_handle_connect 0x7fab5c136800 2024-09-17T18:25:14.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.985+0000 7fab43fff640 10 client.?.objecter resend_mon_ops 2024-09-17T18:25:14.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.985+0000 7fab43fff640 1 -- 172.21.15.121:0/1559462161 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab50017070 con 0x7fab5c136800 2024-09-17T18:25:14.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.985+0000 7fab43fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:14.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.986+0000 7fab43fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:25:14.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.986+0000 7fab43fff640 10 monclient: dump: 2024-09-17T18:25:14.999 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:14.999 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:14.999 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:14.999 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:14.999 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:14.999 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:15.000 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:15.000 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:15.000 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:15.000 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:15.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.986+0000 7fab43fff640 10 monclient: _finish_auth 0 2024-09-17T18:25:15.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.986+0000 7fab43fff640 10 monclient: _check_auth_tickets 2024-09-17T18:25:15.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.986+0000 7fab43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:25:15.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.986+0000 7fab62031640 5 monclient: authenticate success, global_id 54122 2024-09-17T18:25:15.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.986+0000 7fab43fff640 1 -- 172.21.15.121:0/1559462161 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab50005230 con 0x7fab5c136800 2024-09-17T18:25:15.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.986+0000 7fab62031640 10 monclient: _renew_subs 2024-09-17T18:25:15.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.986+0000 7fab62031640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:25:15.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.986+0000 7fab62031640 1 -- 172.21.15.121:0/1559462161 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab5c09e240 con 0x7fab5c136800 2024-09-17T18:25:15.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.986+0000 7fab43fff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:25:15.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.986+0000 7fab43fff640 1 -- 172.21.15.121:0/1559462161 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab500117a0 con 0x7fab5c136800 2024-09-17T18:25:15.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.986+0000 7fab43fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:15.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.986+0000 7fab43fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:25:15.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.986+0000 7fab43fff640 10 monclient: dump: 2024-09-17T18:25:15.001 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:15.001 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:15.001 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.001 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.001 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:15.001 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:15.001 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:15.001 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:15.001 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:15.001 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:15.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.986+0000 7fab62031640 10 client.54122.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:25:15.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.986+0000 7fab62031640 10 monclient: _renew_subs 2024-09-17T18:25:15.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.986+0000 7fab62031640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:25:15.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.986+0000 7fab62031640 1 -- 172.21.15.121:0/1559462161 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fab5c071c50 con 0x7fab5c136800 2024-09-17T18:25:15.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.986+0000 7fab62031640 1 librados: init done 2024-09-17T18:25:15.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.986+0000 7fab59ffb640 4 set_mon_vals no callback set 2024-09-17T18:25:15.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.986+0000 7fab43fff640 1 -- 172.21.15.121:0/1559462161 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fab5001a070 con 0x7fab5c136800 2024-09-17T18:25:15.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.987+0000 7fab62031640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:15.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.987+0000 7fab62031640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:15.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.987+0000 7fab62031640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:25:15.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.987+0000 7fab62031640 1 -- 172.21.15.121:0/1559462161 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab5c073f30 con 0x7fab5c136800 2024-09-17T18:25:15.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.987+0000 7fab43fff640 1 --2- 172.21.15.121:0/1559462161 >> v2:172.21.15.173:6812/2423786037 conn(0x7fab48040170 0x7fab48042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:15.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.987+0000 7fab43fff640 1 -- 172.21.15.121:0/1559462161 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(286..286 src has 1..286) ==== 5705+0+0 (secure 0 0 0) 0x7fab50052c90 con 0x7fab5c136800 2024-09-17T18:25:15.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.987+0000 7fab43fff640 10 client.54122.objecter ms_dispatch 0x7fab5c000b90 osd_map(286..286 src has 1..286) 2024-09-17T18:25:15.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.987+0000 7fab43fff640 3 client.54122.objecter handle_osd_map got epochs [286,286] > 0 2024-09-17T18:25:15.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.987+0000 7fab43fff640 3 client.54122.objecter handle_osd_map decoding full epoch 286 2024-09-17T18:25:15.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.987+0000 7fab43fff640 20 client.54122.objecter dump_active .. 0 homeless 2024-09-17T18:25:15.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.987+0000 7fab5b7fe640 1 --2- 172.21.15.121:0/1559462161 >> v2:172.21.15.173:6812/2423786037 conn(0x7fab48040170 0x7fab48042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:15.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.987+0000 7fab5b7fe640 10 monclient: get_auth_request con 0x7fab48040170 auth_method 0 2024-09-17T18:25:15.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.990+0000 7fab43fff640 10 client.54122.objecter ms_handle_connect 0x7fab48040170 2024-09-17T18:25:15.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.990+0000 7fab5b7fe640 1 --2- 172.21.15.121:0/1559462161 >> v2:172.21.15.173:6812/2423786037 conn(0x7fab48040170 0x7fab48042630 secure :-1 s=READY pgs=3977 cs=0 l=1 rev1=1 crypto rx=0x7fab4c009bd0 tx=0x7fab4c02f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:15.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.992+0000 7fab43fff640 1 -- 172.21.15.121:0/1559462161 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fab5c073f30 con 0x7fab5c136800 2024-09-17T18:25:15.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.992+0000 7fab43fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:15.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:14.992+0000 7fab43fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:25:15.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.231+0000 7fab62031640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:25:15.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.231+0000 7fab62031640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:25:15.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.231+0000 7fab62031640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:25:15.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.231+0000 7fab62031640 1 -- 172.21.15.121:0/1559462161 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fab5c1337b0 con 0x7fab5c136800 2024-09-17T18:25:15.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.231+0000 7fab43fff640 1 -- 172.21.15.121:0/1559462161 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fab50059080 con 0x7fab5c136800 2024-09-17T18:25:15.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.231+0000 7fab43fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:25:15.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.231+0000 7fab43fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:25:15.235 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:25:15.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.233+0000 7fab62031640 10 librados: watch_flush enter 2024-09-17T18:25:15.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.233+0000 7fab62031640 10 librados: watch_flush exit 2024-09-17T18:25:15.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.233+0000 7fab62031640 20 client.54122.objecter shutdown clearing up homeless session... 2024-09-17T18:25:15.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.234+0000 7fab62031640 10 client.54122.objecter successfully canceled tick 2024-09-17T18:25:15.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.234+0000 7fab62031640 1 -- 172.21.15.121:0/1559462161 >> v2:172.21.15.173:6812/2423786037 conn(0x7fab48040170 msgr2=0x7fab48042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:15.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.234+0000 7fab62031640 1 --2- 172.21.15.121:0/1559462161 >> v2:172.21.15.173:6812/2423786037 conn(0x7fab48040170 0x7fab48042630 secure :-1 s=READY pgs=3977 cs=0 l=1 rev1=1 crypto rx=0x7fab4c009bd0 tx=0x7fab4c02f000 comp rx=0 tx=0).stop 2024-09-17T18:25:15.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.234+0000 7fab62031640 10 monclient: shutdown 2024-09-17T18:25:15.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.234+0000 7fab62031640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:25:15.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.234+0000 7fab62031640 1 -- 172.21.15.121:0/1559462161 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab5c136800 msgr2=0x7fab5c073820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:15.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.234+0000 7fab62031640 1 --2- 172.21.15.121:0/1559462161 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab5c136800 0x7fab5c073820 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fab5000c410 tx=0x7fab5000c8d0 comp rx=0 tx=0).stop 2024-09-17T18:25:15.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.234+0000 7fab62031640 1 -- 172.21.15.121:0/1559462161 shutdown_connections 2024-09-17T18:25:15.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.234+0000 7fab62031640 1 --2- 172.21.15.121:0/1559462161 >> v2:172.21.15.173:6812/2423786037 conn(0x7fab48040170 0x7fab48042630 unknown :-1 s=CLOSED pgs=3977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:15.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.234+0000 7fab62031640 1 --2- 172.21.15.121:0/1559462161 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab5c136be0 0x7fab5c09e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:15.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.234+0000 7fab62031640 1 --2- 172.21.15.121:0/1559462161 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab5c136800 0x7fab5c073820 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:15.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.234+0000 7fab62031640 1 --2- 172.21.15.121:0/1559462161 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab5c133360 0x7fab5c1301c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:15.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.234+0000 7fab62031640 1 -- 172.21.15.121:0/1559462161 >> 172.21.15.121:0/1559462161 conn(0x7fab5c0a3b50 msgr2=0x7fab5c135780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:25:15.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.234+0000 7fab62031640 1 -- 172.21.15.121:0/1559462161 shutdown_connections 2024-09-17T18:25:15.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.234+0000 7fab62031640 1 -- 172.21.15.121:0/1559462161 wait complete. 2024-09-17T18:25:15.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.234+0000 7fab62031640 1 librados: shutdown 2024-09-17T18:25:15.247 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T18:25:15.247 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-17T18:25:15.247 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T18:25:15.247 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T18:25:15.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.342+0000 7f0814b0e640 10 monclient: get_monmap_and_config 2024-09-17T18:25:15.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.342+0000 7f0814b0e640 10 monclient: build_initial_monmap 2024-09-17T18:25:15.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.342+0000 7f0814b0e640 10 monclient: monmap: 2024-09-17T18:25:15.346 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:25:15.346 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:15.346 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:25:15.344479+0000 2024-09-17T18:25:15.346 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:25:15.344479+0000 2024-09-17T18:25:15.346 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:25:15.346 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:15.346 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:25:15.346 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:25:15.347 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:25:15.347 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:25:15.347 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:25:15.347 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:15.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.343+0000 7f0814b0e640 1 Processor -- start 2024-09-17T18:25:15.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.343+0000 7f0814b0e640 1 -- start start 2024-09-17T18:25:15.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.343+0000 7f0814b0e640 10 monclient: init 2024-09-17T18:25:15.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f0814b0e640 10 monclient: _reopen_session rank -1 2024-09-17T18:25:15.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f0814b0e640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-17T18:25:15.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f0814b0e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f081009da70 0x7f08101361c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:15.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f0814b0e640 10 monclient(hunting): picked mon.noname-b con 0x7f081009da70 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:25:15.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f0814b0e640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f0810136700 0x7f0810136b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:15.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f0814b0e640 10 monclient(hunting): picked mon.noname-c con 0x7f0810136700 addr v2:172.21.15.173:3300/0 2024-09-17T18:25:15.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f0814b0e640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0810132cc0 0x7f0810133150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:15.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f0814b0e640 10 monclient(hunting): picked mon.noname-a con 0x7f0810132cc0 addr v2:172.21.15.121:3300/0 2024-09-17T18:25:15.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f0814b0e640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:15.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f0814b0e640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f081009de50 con 0x7f0810132cc0 2024-09-17T18:25:15.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f0814b0e640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:15.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f0814b0e640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f08100a0e20 con 0x7f0810136700 2024-09-17T18:25:15.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f0814b0e640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:15.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f0814b0e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f08100a2d40 con 0x7f081009da70 2024-09-17T18:25:15.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f0814b0e640 10 monclient(hunting): _renew_subs 2024-09-17T18:25:15.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f0814b0e640 10 monclient(hunting): authenticate will time out at 1119.061138s 2024-09-17T18:25:15.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f080e59c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f081009da70 0x7f08101361c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:15.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f080ed9d640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0810132cc0 0x7f0810133150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:15.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f080ed9d640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0810132cc0 0x7f0810133150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44574/0 (socket says 172.21.15.121:44574) 2024-09-17T18:25:15.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f080ed9d640 1 -- 172.21.15.121:0/2345660599 learned_addr learned my addr 172.21.15.121:0/2345660599 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:25:15.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f080ed9d640 10 monclient(hunting): get_auth_request con 0x7f0810132cc0 auth_method 0 2024-09-17T18:25:15.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f080ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:15.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f080ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:15.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f080ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:15.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f080e59c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f081009da70 0x7f08101361c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:53512/0 (socket says 172.21.15.121:53512) 2024-09-17T18:25:15.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f080e59c640 10 monclient(hunting): get_auth_request con 0x7f081009da70 auth_method 0 2024-09-17T18:25:15.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f080e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:15.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f080e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:15.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f080e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:15.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f080ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:15.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f080ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:15.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f080ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:15.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f080e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:15.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f080e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:15.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f080e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:15.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f080ed9d640 10 monclient(hunting): handle_auth_done global_id 44361 payload 274 2024-09-17T18:25:15.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f080ed9d640 1 -- 172.21.15.121:0/2345660599 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f081009da70 msgr2=0x7f08101361c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:25:15.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f080ed9d640 1 --2- 172.21.15.121:0/2345660599 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f081009da70 0x7f08101361c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:15.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f080ed9d640 1 -- 172.21.15.121:0/2345660599 >> v2:172.21.15.173:3300/0 conn(0x7f0810136700 msgr2=0x7f0810136b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:25:15.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f080dd9b640 1 --2- 172.21.15.121:0/2345660599 >> v2:172.21.15.173:3300/0 conn(0x7f0810136700 0x7f0810136b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:15.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f080ed9d640 1 --2- 172.21.15.121:0/2345660599 >> v2:172.21.15.173:3300/0 conn(0x7f0810136700 0x7f0810136b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:15.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f080ed9d640 10 monclient: _finish_hunting 0 2024-09-17T18:25:15.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f080ed9d640 1 monclient: found mon.noname-a 2024-09-17T18:25:15.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f080ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:25:15.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f080ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:25:15.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f080ed9d640 1 -- 172.21.15.121:0/2345660599 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08100a3020 con 0x7f0810132cc0 2024-09-17T18:25:15.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.344+0000 7f080e59c640 1 --2- 172.21.15.121:0/2345660599 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f081009da70 0x7f08101361c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:25:15.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.347+0000 7f080ed9d640 1 -- 172.21.15.121:0/2345660599 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0810132cc0 msgr2=0x7f0810133150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:15.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.347+0000 7f080ed9d640 1 --2- 172.21.15.121:0/2345660599 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0810132cc0 0x7f0810133150 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f081009e1a0 tx=0x7f080802a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53f1f7e515cdb74 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:15.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.349+0000 7f080d59a640 1 -- 172.21.15.121:0/2345660599 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0808037070 con 0x7f0810132cc0 2024-09-17T18:25:15.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.349+0000 7f080d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:15.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.349+0000 7f080d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:25:15.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.349+0000 7f080d59a640 10 monclient: dump: 2024-09-17T18:25:15.356 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:15.356 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:15.356 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.356 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.356 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:15.356 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:15.356 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:15.357 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:15.357 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:15.357 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:15.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.349+0000 7f080d59a640 10 monclient: _finish_auth 0 2024-09-17T18:25:15.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.349+0000 7f080d59a640 10 monclient: _check_auth_tickets 2024-09-17T18:25:15.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.349+0000 7f080d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:25:15.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.349+0000 7f080d59a640 1 -- 172.21.15.121:0/2345660599 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0808032640 con 0x7f0810132cc0 2024-09-17T18:25:15.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.349+0000 7f0814b0e640 5 monclient: authenticate success, global_id 44361 2024-09-17T18:25:15.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.349+0000 7f0814b0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:25:15.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.349+0000 7f080d59a640 10 monclient: handle_config config(0 keys) 2024-09-17T18:25:15.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.349+0000 7f0814b0e640 10 monclient: get_monmap_and_config success 2024-09-17T18:25:15.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.349+0000 7f080d59a640 1 -- 172.21.15.121:0/2345660599 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0808032a10 con 0x7f0810132cc0 2024-09-17T18:25:15.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.349+0000 7f0814b0e640 4 set_mon_vals no callback set 2024-09-17T18:25:15.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.349+0000 7f080d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:15.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.349+0000 7f080d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:25:15.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.349+0000 7f080d59a640 10 monclient: dump: 2024-09-17T18:25:15.358 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:15.358 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:15.358 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.358 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.358 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:15.358 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:15.358 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:15.358 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:15.358 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:15.358 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:15.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.350+0000 7f0814b0e640 10 monclient: shutdown 2024-09-17T18:25:15.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.350+0000 7f0814b0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:25:15.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.350+0000 7f0814b0e640 1 -- 172.21.15.121:0/2345660599 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0810132cc0 msgr2=0x7f0810133150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:15.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.350+0000 7f0814b0e640 1 --2- 172.21.15.121:0/2345660599 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0810132cc0 0x7f0810133150 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f081009e1a0 tx=0x7f080802a100 comp rx=0 tx=0).stop 2024-09-17T18:25:15.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.350+0000 7f0814b0e640 1 -- 172.21.15.121:0/2345660599 shutdown_connections 2024-09-17T18:25:15.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.350+0000 7f0814b0e640 1 --2- 172.21.15.121:0/2345660599 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0810132cc0 0x7f0810133150 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:15.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.350+0000 7f0814b0e640 1 --2- 172.21.15.121:0/2345660599 >> v2:172.21.15.173:3300/0 conn(0x7f0810136700 0x7f0810136b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:15.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.350+0000 7f0814b0e640 1 --2- 172.21.15.121:0/2345660599 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f081009da70 0x7f08101361c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:15.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.350+0000 7f0814b0e640 1 -- 172.21.15.121:0/2345660599 >> 172.21.15.121:0/2345660599 conn(0x7f08100a7830 msgr2=0x7f08100a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:25:15.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.350+0000 7f0814b0e640 1 -- 172.21.15.121:0/2345660599 shutdown_connections 2024-09-17T18:25:15.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.350+0000 7f0814b0e640 1 -- 172.21.15.121:0/2345660599 wait complete. 2024-09-17T18:25:15.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.350+0000 7f0814b0e640 10 monclient: build_initial_monmap 2024-09-17T18:25:15.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.350+0000 7f0814b0e640 10 monclient: monmap: 2024-09-17T18:25:15.359 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:25:15.359 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:25:15.359 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:25:15.360 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:25:15.360 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:25:15.360 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:15.360 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:25:15.360 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:25:15.360 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:25:15.360 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:15.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.351+0000 7f0814b0e640 1 librados: starting msgr at 2024-09-17T18:25:15.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.351+0000 7f0814b0e640 1 librados: starting objecter 2024-09-17T18:25:15.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.351+0000 7f0814b0e640 1 Processor -- start 2024-09-17T18:25:15.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.351+0000 7f0814b0e640 1 -- start start 2024-09-17T18:25:15.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.351+0000 7f0814b0e640 1 librados: setting wanted keys 2024-09-17T18:25:15.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.351+0000 7f0814b0e640 1 librados: calling monclient init 2024-09-17T18:25:15.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.351+0000 7f0814b0e640 10 monclient: init 2024-09-17T18:25:15.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.351+0000 7f0814b0e640 10 monclient: _reopen_session rank -1 2024-09-17T18:25:15.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.351+0000 7f0814b0e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T18:25:15.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.351+0000 7f0814b0e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f081009da70 0x7f08101cd410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:15.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.351+0000 7f0814b0e640 10 monclient(hunting): picked mon.noname-b con 0x7f081009da70 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:25:15.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.351+0000 7f0814b0e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0810132cc0 0x7f081012f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:15.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.351+0000 7f0814b0e640 10 monclient(hunting): picked mon.noname-a con 0x7f0810132cc0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:25:15.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.351+0000 7f0814b0e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0810136700 0x7f081009e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:15.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.351+0000 7f0814b0e640 10 monclient(hunting): picked mon.noname-c con 0x7f0810136700 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:25:15.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.351+0000 7f0814b0e640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:15.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.351+0000 7f0814b0e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0810133690 con 0x7f0810132cc0 2024-09-17T18:25:15.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.351+0000 7f0814b0e640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:15.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.351+0000 7f0814b0e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f08100a3930 con 0x7f081009da70 2024-09-17T18:25:15.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.351+0000 7f0814b0e640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:15.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.351+0000 7f0814b0e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0810133810 con 0x7f0810136700 2024-09-17T18:25:15.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.351+0000 7f0814b0e640 10 monclient(hunting): _renew_subs 2024-09-17T18:25:15.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.351+0000 7f0814b0e640 10 monclient(hunting): authenticate will time out at 1119.068741s 2024-09-17T18:25:15.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.351+0000 7f080dd9b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0810132cc0 0x7f081012f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:15.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.351+0000 7f080ed9d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0810136700 0x7f081009e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:15.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.351+0000 7f080dd9b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0810132cc0 0x7f081012f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44590/0 (socket says 172.21.15.121:44590) 2024-09-17T18:25:15.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.351+0000 7f080dd9b640 1 -- 172.21.15.121:0/860654903 learned_addr learned my addr 172.21.15.121:0/860654903 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:25:15.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.351+0000 7f080dd9b640 10 monclient(hunting): get_auth_request con 0x7f0810132cc0 auth_method 0 2024-09-17T18:25:15.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.351+0000 7f080dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:15.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.351+0000 7f080dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:15.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.351+0000 7f080dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:15.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.351+0000 7f080ed9d640 10 monclient(hunting): get_auth_request con 0x7f0810136700 auth_method 0 2024-09-17T18:25:15.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.351+0000 7f080ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:15.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.351+0000 7f080ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:15.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.351+0000 7f080ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:15.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.351+0000 7f080ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:15.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.351+0000 7f080ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:15.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.351+0000 7f080ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:15.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.351+0000 7f080dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:15.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.351+0000 7f080dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:15.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.351+0000 7f080dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:15.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.352+0000 7f080e59c640 1 --2- 172.21.15.121:0/860654903 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f081009da70 0x7f08101cd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:15.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.352+0000 7f080e59c640 10 monclient(hunting): get_auth_request con 0x7f081009da70 auth_method 0 2024-09-17T18:25:15.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.352+0000 7f080e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:15.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.352+0000 7f080e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:15.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.352+0000 7f080e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:15.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.352+0000 7f080ed9d640 10 monclient(hunting): handle_auth_done global_id 54128 payload 824 2024-09-17T18:25:15.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.352+0000 7f080ed9d640 1 -- 172.21.15.121:0/860654903 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f081009da70 msgr2=0x7f08101cd410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:15.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.352+0000 7f080ed9d640 1 --2- 172.21.15.121:0/860654903 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f081009da70 0x7f08101cd410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:15.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.352+0000 7f080ed9d640 1 -- 172.21.15.121:0/860654903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0810132cc0 msgr2=0x7f081012f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:15.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.352+0000 7f080ed9d640 1 --2- 172.21.15.121:0/860654903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0810132cc0 0x7f081012f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:15.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.352+0000 7f080ed9d640 10 monclient: _finish_hunting 0 2024-09-17T18:25:15.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.352+0000 7f080ed9d640 1 monclient: found mon.noname-c 2024-09-17T18:25:15.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.352+0000 7f080ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:25:15.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.352+0000 7f080ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:25:15.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.352+0000 7f080ed9d640 1 -- 172.21.15.121:0/860654903 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08101ce330 con 0x7f0810136700 2024-09-17T18:25:15.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.352+0000 7f080dd9b640 1 --2- 172.21.15.121:0/860654903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0810132cc0 0x7f081012f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:25:15.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.354+0000 7f080ed9d640 1 -- 172.21.15.121:0/860654903 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0810136700 msgr2=0x7f081009e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:15.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.354+0000 7f080ed9d640 1 --2- 172.21.15.121:0/860654903 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0810136700 0x7f081009e550 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0808034000 tx=0x7f08080046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:15.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.354+0000 7f07feffd640 10 client.?.objecter ms_handle_connect 0x7f0810136700 2024-09-17T18:25:15.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.354+0000 7f07feffd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:25:15.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.354+0000 7f07feffd640 1 -- 172.21.15.121:0/860654903 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0808037040 con 0x7f0810136700 2024-09-17T18:25:15.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.354+0000 7f07feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:15.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.354+0000 7f07feffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:25:15.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.354+0000 7f07feffd640 10 monclient: dump: 2024-09-17T18:25:15.368 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:15.369 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:15.369 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.369 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.369 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:15.369 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:15.369 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:15.369 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:15.369 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:15.369 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:15.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.354+0000 7f07feffd640 10 monclient: _finish_auth 0 2024-09-17T18:25:15.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.354+0000 7f07feffd640 10 monclient: _check_auth_tickets 2024-09-17T18:25:15.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.354+0000 7f07feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:25:15.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.354+0000 7f07feffd640 1 -- 172.21.15.121:0/860654903 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0808042070 con 0x7f0810136700 2024-09-17T18:25:15.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.354+0000 7f0814b0e640 5 monclient: authenticate success, global_id 54128 2024-09-17T18:25:15.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.354+0000 7f07feffd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:25:15.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.354+0000 7f0814b0e640 10 monclient: _renew_subs 2024-09-17T18:25:15.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.354+0000 7f07feffd640 1 -- 172.21.15.121:0/860654903 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f080803ba80 con 0x7f0810136700 2024-09-17T18:25:15.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.354+0000 7f0814b0e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:25:15.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.354+0000 7f0814b0e640 1 -- 172.21.15.121:0/860654903 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f081011ef20 con 0x7f0810136700 2024-09-17T18:25:15.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.354+0000 7f07feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:15.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.354+0000 7f07feffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:25:15.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.354+0000 7f07feffd640 10 monclient: dump: 2024-09-17T18:25:15.370 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:15.370 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:15.370 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.370 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.370 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:15.370 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:15.370 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:15.370 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:15.370 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:15.371 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:15.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.354+0000 7f080cd99640 4 set_mon_vals no callback set 2024-09-17T18:25:15.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.354+0000 7f0814b0e640 10 client.54128.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:25:15.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.354+0000 7f0814b0e640 10 monclient: _renew_subs 2024-09-17T18:25:15.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.354+0000 7f0814b0e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:25:15.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.354+0000 7f0814b0e640 1 -- 172.21.15.121:0/860654903 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f081011f200 con 0x7f0810136700 2024-09-17T18:25:15.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.354+0000 7f0814b0e640 1 librados: init done 2024-09-17T18:25:15.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.355+0000 7f0814b0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:15.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.355+0000 7f0814b0e640 1 --2- 172.21.15.121:0/860654903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f081012fd60 0x7f081002d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:15.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.355+0000 7f0814b0e640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:15.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.355+0000 7f0814b0e640 1 -- 172.21.15.121:0/860654903 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0810133150 con 0x7f081012fd60 2024-09-17T18:25:15.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.355+0000 7f07feffd640 1 -- 172.21.15.121:0/860654903 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f08080324c0 con 0x7f0810136700 2024-09-17T18:25:15.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.355+0000 7f080e59c640 1 --2- 172.21.15.121:0/860654903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f081012fd60 0x7f081002d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:15.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.355+0000 7f080e59c640 10 monclient: get_auth_request con 0x7f081012fd60 auth_method 0 2024-09-17T18:25:15.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.355+0000 7f080e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:15.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.355+0000 7f080e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:15.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.355+0000 7f080e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:15.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.355+0000 7f080e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:15.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.355+0000 7f080e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:15.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.355+0000 7f080e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:15.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.355+0000 7f080e59c640 10 monclient(hunting): handle_auth_done global_id 44367 payload 824 2024-09-17T18:25:15.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.355+0000 7f07feffd640 1 --2- 172.21.15.121:0/860654903 >> v2:172.21.15.173:6812/2423786037 conn(0x7f07f0040170 0x7f07f0042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:15.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.355+0000 7f080e59c640 1 -- 172.21.15.121:0/860654903 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f08101bd580 con 0x7f081012fd60 2024-09-17T18:25:15.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.355+0000 7f07feffd640 1 -- 172.21.15.121:0/860654903 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(286..286 src has 1..286) ==== 5705+0+0 (secure 0 0 0) 0x7f0808073e60 con 0x7f0810136700 2024-09-17T18:25:15.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.355+0000 7f07feffd640 10 client.54128.objecter ms_dispatch 0x7f0810000b90 osd_map(286..286 src has 1..286) 2024-09-17T18:25:15.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.355+0000 7f07feffd640 3 client.54128.objecter handle_osd_map got epochs [286,286] > 0 2024-09-17T18:25:15.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.355+0000 7f07feffd640 3 client.54128.objecter handle_osd_map decoding full epoch 286 2024-09-17T18:25:15.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.355+0000 7f07feffd640 20 client.54128.objecter dump_active .. 0 homeless 2024-09-17T18:25:15.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.356+0000 7f080dd9b640 1 --2- 172.21.15.121:0/860654903 >> v2:172.21.15.173:6812/2423786037 conn(0x7f07f0040170 0x7f07f0042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:15.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.356+0000 7f080dd9b640 10 monclient: get_auth_request con 0x7f07f0040170 auth_method 0 2024-09-17T18:25:15.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.357+0000 7f080e59c640 1 -- 172.21.15.121:0/860654903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f081012fd60 msgr2=0x7f081002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:15.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.357+0000 7f07feffd640 10 client.54128.objecter ms_handle_connect 0x7f081012fd60 2024-09-17T18:25:15.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.357+0000 7f07feffd640 10 client.54128.objecter resend_mon_ops 2024-09-17T18:25:15.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.357+0000 7f080e59c640 1 --2- 172.21.15.121:0/860654903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f081012fd60 0x7f081002d440 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0800007f00 tx=0x7f08000115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:15.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.358+0000 7f07feffd640 1 -- 172.21.15.121:0/860654903 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f080001a020 con 0x7f081012fd60 2024-09-17T18:25:15.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.358+0000 7f07feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:15.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.358+0000 7f07feffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:25:15.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.358+0000 7f07feffd640 10 monclient: dump: 2024-09-17T18:25:15.373 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:15.373 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:15.373 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.373 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.373 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:15.373 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:15.373 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:15.374 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:15.374 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:15.374 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:15.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.358+0000 7f07feffd640 1 -- 172.21.15.121:0/860654903 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0800016890 con 0x7f081012fd60 2024-09-17T18:25:15.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.358+0000 7f07feffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:15.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.358+0000 7f07feffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:25:15.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.358+0000 7f07feffd640 1 -- 172.21.15.121:0/860654903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f081012fd60 msgr2=0x7f081002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:15.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.358+0000 7f07feffd640 1 --2- 172.21.15.121:0/860654903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f081012fd60 0x7f081002d440 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0800007f00 tx=0x7f08000115a0 comp rx=0 tx=0).stop 2024-09-17T18:25:15.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.358+0000 7f07feffd640 1 -- 172.21.15.121:0/860654903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f081012fd60 msgr2=0x7f081002d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:25:15.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.358+0000 7f07feffd640 1 --2- 172.21.15.121:0/860654903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f081012fd60 0x7f081002d440 secure :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0800007f00 tx=0x7f08000115a0 comp rx=0 tx=0).stop 2024-09-17T18:25:15.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.358+0000 7f07feffd640 10 client.54128.objecter ms_handle_connect 0x7f07f0040170 2024-09-17T18:25:15.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.358+0000 7f080dd9b640 1 --2- 172.21.15.121:0/860654903 >> v2:172.21.15.173:6812/2423786037 conn(0x7f07f0040170 0x7f07f0042630 secure :-1 s=READY pgs=3978 cs=0 l=1 rev1=1 crypto rx=0x7f08100a05d0 tx=0x7f0804008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:15.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.364+0000 7f0814b0e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:25:15.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.364+0000 7f0814b0e640 1 --2- 172.21.15.121:0/860654903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f081002d980 0x7f081002dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:15.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.364+0000 7f0814b0e640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:15.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.364+0000 7f0814b0e640 1 -- 172.21.15.121:0/860654903 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f081011eb60 con 0x7f081002d980 2024-09-17T18:25:15.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.364+0000 7f080e59c640 1 --2- 172.21.15.121:0/860654903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f081002d980 0x7f081002dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:15.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.364+0000 7f080e59c640 10 monclient: get_auth_request con 0x7f081002d980 auth_method 0 2024-09-17T18:25:15.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.364+0000 7f080e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:15.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.364+0000 7f080e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:15.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.364+0000 7f080e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:15.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.364+0000 7f080e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:15.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.364+0000 7f080e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:15.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.364+0000 7f080e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:15.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.364+0000 7f080e59c640 10 monclient(hunting): handle_auth_done global_id 44370 payload 824 2024-09-17T18:25:15.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.364+0000 7f080e59c640 1 -- 172.21.15.121:0/860654903 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0810130220 con 0x7f081002d980 2024-09-17T18:25:15.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.367+0000 7f07feffd640 10 client.54128.objecter ms_handle_connect 0x7f081002d980 2024-09-17T18:25:15.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.367+0000 7f07feffd640 10 client.54128.objecter resend_mon_ops 2024-09-17T18:25:15.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.367+0000 7f080e59c640 1 -- 172.21.15.121:0/860654903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f081002d980 msgr2=0x7f081002dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:15.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.367+0000 7f080e59c640 1 --2- 172.21.15.121:0/860654903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f081002d980 0x7f081002dd90 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0800007f00 tx=0x7f0800004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:15.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.367+0000 7f07feffd640 1 -- 172.21.15.121:0/860654903 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f080001a030 con 0x7f081002d980 2024-09-17T18:25:15.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.367+0000 7f07feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:15.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.367+0000 7f07feffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:25:15.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.367+0000 7f07feffd640 10 monclient: dump: 2024-09-17T18:25:15.377 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:15.377 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:15.377 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.377 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.377 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:15.377 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:15.377 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:15.377 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:15.377 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:15.377 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:15.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.367+0000 7f07feffd640 1 -- 172.21.15.121:0/860654903 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7f0800021220 con 0x7f081002d980 2024-09-17T18:25:15.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.367+0000 7f07feffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:25:15.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.367+0000 7f07feffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:25:15.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.367+0000 7f07feffd640 1 -- 172.21.15.121:0/860654903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f081002d980 msgr2=0x7f081002dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:15.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.367+0000 7f07feffd640 1 --2- 172.21.15.121:0/860654903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f081002d980 0x7f081002dd90 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0800007f00 tx=0x7f0800004a40 comp rx=0 tx=0).stop 2024-09-17T18:25:15.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.367+0000 7f07feffd640 1 -- 172.21.15.121:0/860654903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f081002d980 msgr2=0x7f081002dd90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:25:15.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.367+0000 7f07feffd640 1 --2- 172.21.15.121:0/860654903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f081002d980 0x7f081002dd90 secure :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0800007f00 tx=0x7f0800004a40 comp rx=0 tx=0).stop 2024-09-17T18:25:15.378 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:25:15.378 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:25:15.378 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:25:15.378 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "leader", 2024-09-17T18:25:15.378 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 26, 2024-09-17T18:25:15.378 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 141032, 2024-09-17T18:25:15.378 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:25:15.378 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:25:15.378 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:25:15.379 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:25:15.379 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:25:15.379 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 7, 2024-09-17T18:25:15.379 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:25:15.379 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:25:15.379 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:25:15.379 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:25:15.379 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:25:15.379 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:25:15.379 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:25:15.379 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:25:15.379 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:25:15.379 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:25:15.379 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:25:15.380 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:25:15.380 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:25:15.380 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:25:15.380 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:25:15.380 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:25:15.380 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:25:15.380 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:25:15.380 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:25:15.380 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:25:15.380 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:25:15.380 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:25:15.380 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:25:15.380 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:25:15.380 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:25:15.380 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:25:15.381 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:25:15.381 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:25:15.381 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:15.381 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:15.381 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:25:15.381 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:25:15.381 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:25:15.381 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:25:15.381 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:25:15.381 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:25:15.381 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:25:15.381 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:25:15.381 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:25:15.381 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:25:15.381 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:25:15.382 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:25:15.382 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:25:15.382 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:25:15.382 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:25:15.382 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:25:15.382 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:25:15.382 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:25:15.382 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:25:15.382 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:25:15.382 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:25:15.382 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:25:15.382 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:25:15.382 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:25:15.382 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:25:15.382 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:25:15.383 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:25:15.383 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:25:15.383 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:25:15.383 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:25:15.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:15.383 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:25:15.383 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:15.383 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:25:15.383 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:25:15.383 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:25:15.383 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:25:15.383 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:15.383 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:25:15.383 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:25:15.383 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:15.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:15.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:15.384 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:25:15.384 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:25:15.384 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:15.384 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:15.384 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:15.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:15.384 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:25:15.384 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:25:15.384 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:25:15.384 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:25:15.384 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:25:15.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:15.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:15.384 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:25:15.385 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:25:15.385 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:25:15.385 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:25:15.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:15.385 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:25:15.385 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:25:15.385 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:15.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:15.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:15.385 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:25:15.385 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:25:15.385 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:15.385 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:15.385 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:15.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:15.386 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:25:15.386 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:25:15.386 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:25:15.386 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:25:15.386 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:25:15.386 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:15.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:15.386 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:25:15.386 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:25:15.386 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:25:15.386 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:25:15.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:15.386 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:25:15.386 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:25:15.386 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:15.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:15.387 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:15.387 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:25:15.387 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:25:15.387 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:15.387 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:15.387 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:15.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:15.387 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:25:15.387 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:25:15.387 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:25:15.387 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:25:15.387 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:25:15.387 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:15.387 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:15.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:15.388 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:25:15.388 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:25:15.388 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:15.388 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:25:15.388 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:25:15.388 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:25:15.388 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:15.388 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:25:15.388 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:25:15.388 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:15.388 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:25:15.388 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:25:15.388 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-17T18:25:15.388 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:15.388 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:25:15.389 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:25:15.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:15.389 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:25:15.389 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:25:15.389 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:25:15.389 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:15.389 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:15.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:15.389 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:25:15.389 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:25:15.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.367+0000 7f0814b0e640 10 librados: watch_flush enter 2024-09-17T18:25:15.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.367+0000 7f0814b0e640 10 librados: watch_flush exit 2024-09-17T18:25:15.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.367+0000 7f0814b0e640 20 client.54128.objecter shutdown clearing up homeless session... 2024-09-17T18:25:15.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.367+0000 7f0814b0e640 10 client.54128.objecter successfully canceled tick 2024-09-17T18:25:15.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.367+0000 7f0814b0e640 1 -- 172.21.15.121:0/860654903 >> v2:172.21.15.173:6812/2423786037 conn(0x7f07f0040170 msgr2=0x7f07f0042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:15.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.367+0000 7f0814b0e640 1 --2- 172.21.15.121:0/860654903 >> v2:172.21.15.173:6812/2423786037 conn(0x7f07f0040170 0x7f07f0042630 secure :-1 s=READY pgs=3978 cs=0 l=1 rev1=1 crypto rx=0x7f08100a05d0 tx=0x7f0804008040 comp rx=0 tx=0).stop 2024-09-17T18:25:15.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.367+0000 7f0814b0e640 10 monclient: shutdown 2024-09-17T18:25:15.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.367+0000 7f0814b0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:25:15.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.367+0000 7f0814b0e640 1 -- 172.21.15.121:0/860654903 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0810136700 msgr2=0x7f081009e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:15.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.367+0000 7f0814b0e640 1 --2- 172.21.15.121:0/860654903 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0810136700 0x7f081009e550 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0808034000 tx=0x7f08080046c0 comp rx=0 tx=0).stop 2024-09-17T18:25:15.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.367+0000 7f080ed9d640 1 -- 172.21.15.121:0/860654903 reap_dead start 2024-09-17T18:25:15.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.367+0000 7f0814b0e640 1 -- 172.21.15.121:0/860654903 shutdown_connections 2024-09-17T18:25:15.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.367+0000 7f0814b0e640 1 --2- 172.21.15.121:0/860654903 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0810136700 0x7f081009e550 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:15.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.367+0000 7f0814b0e640 1 -- 172.21.15.121:0/860654903 >> 172.21.15.121:0/860654903 conn(0x7f08100a7830 msgr2=0x7f08101324e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:25:15.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.368+0000 7f0814b0e640 1 -- 172.21.15.121:0/860654903 shutdown_connections 2024-09-17T18:25:15.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.368+0000 7f0814b0e640 1 -- 172.21.15.121:0/860654903 wait complete. 2024-09-17T18:25:15.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.368+0000 7f0814b0e640 1 librados: shutdown 2024-09-17T18:25:15.392 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:25:15.392 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T18:25:15.392 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T18:25:15.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.488+0000 7f16558d6640 10 monclient: get_monmap_and_config 2024-09-17T18:25:15.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.488+0000 7f16558d6640 10 monclient: build_initial_monmap 2024-09-17T18:25:15.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.488+0000 7f16558d6640 10 monclient: monmap: 2024-09-17T18:25:15.491 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:25:15.491 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:15.491 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:25:15.489963+0000 2024-09-17T18:25:15.491 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:25:15.489963+0000 2024-09-17T18:25:15.491 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:25:15.491 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:15.491 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:25:15.492 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:25:15.492 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:25:15.492 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:25:15.492 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:25:15.492 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:15.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.489+0000 7f16558d6640 1 Processor -- start 2024-09-17T18:25:15.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.489+0000 7f16558d6640 1 -- start start 2024-09-17T18:25:15.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.489+0000 7f16558d6640 10 monclient: init 2024-09-17T18:25:15.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.489+0000 7f16558d6640 10 monclient: _reopen_session rank -1 2024-09-17T18:25:15.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.489+0000 7f16558d6640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-17T18:25:15.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.489+0000 7f16558d6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f16500700f0 addr v1:172.21.15.121:6789/0 2024-09-17T18:25:15.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.489+0000 7f16558d6640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f165009e810 0x7f165013d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:15.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.489+0000 7f16558d6640 10 monclient(hunting): picked mon.noname-b con 0x7f165009e810 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:25:15.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.489+0000 7f16558d6640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f165009ec60 0x7f165013fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:15.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.489+0000 7f16558d6640 10 monclient(hunting): picked mon.noname-c con 0x7f165009ec60 addr v2:172.21.15.173:3300/0 2024-09-17T18:25:15.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.489+0000 7f16558d6640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1650135560 con 0x7f16500700f0 2024-09-17T18:25:15.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.489+0000 7f16558d6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:15.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.489+0000 7f16558d6640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f16500704d0 con 0x7f165009ec60 2024-09-17T18:25:15.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.489+0000 7f16558d6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:15.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.489+0000 7f16558d6640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f165009ca60 con 0x7f165009e810 2024-09-17T18:25:15.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.489+0000 7f16558d6640 10 monclient(hunting): _renew_subs 2024-09-17T18:25:15.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.489+0000 7f16558d6640 10 monclient(hunting): authenticate will time out at 1119.206655s 2024-09-17T18:25:15.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.489+0000 7f1647fff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f165009e810 0x7f165013d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:15.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.489+0000 7f164effd640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f16500700f0 0x7f1650134e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:38376/0 (socket says 172.21.15.121:38376) 2024-09-17T18:25:15.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.489+0000 7f164effd640 1 -- 172.21.15.121:0/1235247982 learned_addr learned my addr 172.21.15.121:0/1235247982 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:25:15.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.489+0000 7f1647fff640 10 monclient(hunting): get_auth_request con 0x7f165009e810 auth_method 0 2024-09-17T18:25:15.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.489+0000 7f1647fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:15.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.489+0000 7f1647fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:15.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.489+0000 7f1647fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:15.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.489+0000 7f1647fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:15.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.489+0000 7f1647fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:15.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.489+0000 7f1647fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:15.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.489+0000 7f1647fff640 10 monclient(hunting): handle_auth_done global_id 54131 payload 274 2024-09-17T18:25:15.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.489+0000 7f1647fff640 1 -- 172.21.15.121:0/1235247982 >> v2:172.21.15.173:3300/0 conn(0x7f165009ec60 msgr2=0x7f165013fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:25:15.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.489+0000 7f1647fff640 1 --2- 172.21.15.121:0/1235247982 >> v2:172.21.15.173:3300/0 conn(0x7f165009ec60 0x7f165013fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:15.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.490+0000 7f1647fff640 1 -- 172.21.15.121:0/1235247982 >> v1:172.21.15.121:6789/0 conn(0x7f16500700f0 legacy=0x7f1650134e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:25:15.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.490+0000 7f1647fff640 10 monclient: _finish_hunting 0 2024-09-17T18:25:15.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.490+0000 7f1647fff640 1 monclient: found mon.noname-b 2024-09-17T18:25:15.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.490+0000 7f1647fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:25:15.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.490+0000 7f1647fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:25:15.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.490+0000 7f1647fff640 1 -- 172.21.15.121:0/1235247982 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f165009cd20 con 0x7f165009e810 2024-09-17T18:25:15.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.492+0000 7f1647fff640 1 -- 172.21.15.121:0/1235247982 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f165009e810 msgr2=0x7f165013d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:15.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.492+0000 7f1647fff640 1 --2- 172.21.15.121:0/1235247982 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f165009e810 0x7f165013d6e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f16380097a0 tx=0x7f1638029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff5008a4c192c6b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:15.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.492+0000 7f164e7fc640 1 -- 172.21.15.121:0/1235247982 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1638037070 con 0x7f165009e810 2024-09-17T18:25:15.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.493+0000 7f164e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:15.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.493+0000 7f164e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:25:15.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.493+0000 7f164e7fc640 10 monclient: dump: 2024-09-17T18:25:15.499 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:15.499 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:15.500 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.500 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.500 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:15.500 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:15.500 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:15.500 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:15.500 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:15.500 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:15.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.493+0000 7f164e7fc640 10 monclient: _finish_auth 0 2024-09-17T18:25:15.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.493+0000 7f164e7fc640 10 monclient: _check_auth_tickets 2024-09-17T18:25:15.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.493+0000 7f164e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:25:15.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.493+0000 7f164e7fc640 1 -- 172.21.15.121:0/1235247982 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1638032640 con 0x7f165009e810 2024-09-17T18:25:15.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.493+0000 7f16558d6640 5 monclient: authenticate success, global_id 54131 2024-09-17T18:25:15.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.493+0000 7f16558d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:25:15.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.493+0000 7f164e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:25:15.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.493+0000 7f16558d6640 10 monclient: get_monmap_and_config success 2024-09-17T18:25:15.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.493+0000 7f164e7fc640 1 -- 172.21.15.121:0/1235247982 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1638032a10 con 0x7f165009e810 2024-09-17T18:25:15.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.493+0000 7f164e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:15.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.493+0000 7f16558d6640 4 set_mon_vals no callback set 2024-09-17T18:25:15.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.493+0000 7f164e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:25:15.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.493+0000 7f164e7fc640 10 monclient: dump: 2024-09-17T18:25:15.501 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:15.501 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:15.501 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.501 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.501 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:15.501 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:15.501 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:15.501 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:15.501 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:15.502 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:15.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.493+0000 7f16558d6640 10 monclient: shutdown 2024-09-17T18:25:15.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.493+0000 7f16558d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:25:15.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.493+0000 7f16558d6640 1 -- 172.21.15.121:0/1235247982 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f165009e810 msgr2=0x7f165013d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:15.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.493+0000 7f16558d6640 1 --2- 172.21.15.121:0/1235247982 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f165009e810 0x7f165013d6e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f16380097a0 tx=0x7f1638029fd0 comp rx=0 tx=0).stop 2024-09-17T18:25:15.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.493+0000 7f16558d6640 1 -- 172.21.15.121:0/1235247982 shutdown_connections 2024-09-17T18:25:15.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.493+0000 7f16558d6640 1 --2- 172.21.15.121:0/1235247982 >> v2:172.21.15.173:3300/0 conn(0x7f165009ec60 0x7f165013fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:15.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.493+0000 7f16558d6640 1 --2- 172.21.15.121:0/1235247982 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f165009e810 0x7f165013d6e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:15.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.493+0000 7f16558d6640 1 -- 172.21.15.121:0/1235247982 >> 172.21.15.121:0/1235247982 conn(0x7f16500ab830 msgr2=0x7f16500a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:25:15.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.493+0000 7f16558d6640 1 -- 172.21.15.121:0/1235247982 shutdown_connections 2024-09-17T18:25:15.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.493+0000 7f16558d6640 1 -- 172.21.15.121:0/1235247982 wait complete. 2024-09-17T18:25:15.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.494+0000 7f16558d6640 10 monclient: build_initial_monmap 2024-09-17T18:25:15.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.494+0000 7f16558d6640 10 monclient: monmap: 2024-09-17T18:25:15.502 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:25:15.502 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:25:15.503 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:25:15.503 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:25:15.503 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:25:15.503 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:15.503 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:25:15.503 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:25:15.503 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:25:15.503 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:15.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.494+0000 7f16558d6640 1 librados: starting msgr at 2024-09-17T18:25:15.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.494+0000 7f16558d6640 1 librados: starting objecter 2024-09-17T18:25:15.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.494+0000 7f16558d6640 1 Processor -- start 2024-09-17T18:25:15.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.494+0000 7f16558d6640 1 -- start start 2024-09-17T18:25:15.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.494+0000 7f16558d6640 1 librados: setting wanted keys 2024-09-17T18:25:15.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.494+0000 7f16558d6640 1 librados: calling monclient init 2024-09-17T18:25:15.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.494+0000 7f16558d6640 10 monclient: init 2024-09-17T18:25:15.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f16558d6640 10 monclient: _reopen_session rank -1 2024-09-17T18:25:15.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f16558d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T18:25:15.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f16558d6640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16500700f0 0x7f165013f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:15.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f16558d6640 10 monclient(hunting): picked mon.noname-a con 0x7f16500700f0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:25:15.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f16558d6640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f165009e810 0x7f165013f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:15.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f16558d6640 10 monclient(hunting): picked mon.noname-c con 0x7f165009e810 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:25:15.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f16558d6640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f165009ec60 0x7f16500a01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:15.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f16558d6640 10 monclient(hunting): picked mon.noname-b con 0x7f165009ec60 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:25:15.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f16558d6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:15.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f16558d6640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f16501402f0 con 0x7f16500700f0 2024-09-17T18:25:15.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f16558d6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:15.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f16558d6640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1650140170 con 0x7f165009ec60 2024-09-17T18:25:15.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f16558d6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:15.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f16558d6640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1650140470 con 0x7f165009e810 2024-09-17T18:25:15.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f16558d6640 10 monclient(hunting): _renew_subs 2024-09-17T18:25:15.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f16558d6640 10 monclient(hunting): authenticate will time out at 1119.212140s 2024-09-17T18:25:15.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f164effd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16500700f0 0x7f165013f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:15.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f1647fff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f165009e810 0x7f165013f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:15.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f164effd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16500700f0 0x7f165013f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44610/0 (socket says 172.21.15.121:44610) 2024-09-17T18:25:15.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f164effd640 1 -- 172.21.15.121:0/4120299481 learned_addr learned my addr 172.21.15.121:0/4120299481 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:25:15.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f164effd640 10 monclient(hunting): get_auth_request con 0x7f16500700f0 auth_method 0 2024-09-17T18:25:15.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f164effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:15.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f164effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:15.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f164effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:15.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f1647fff640 10 monclient(hunting): get_auth_request con 0x7f165009e810 auth_method 0 2024-09-17T18:25:15.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f1647fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:15.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f1647fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:15.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f1647fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:15.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f164effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:15.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f164effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:15.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f164effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:15.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f1647fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:15.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f1647fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:15.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f1647fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:15.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f164effd640 10 monclient(hunting): handle_auth_done global_id 44373 payload 824 2024-09-17T18:25:15.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f164f7fe640 1 --2- 172.21.15.121:0/4120299481 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f165009ec60 0x7f16500a01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:15.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f164effd640 1 -- 172.21.15.121:0/4120299481 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f165009e810 msgr2=0x7f165013f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:15.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f164effd640 1 --2- 172.21.15.121:0/4120299481 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f165009e810 0x7f165013f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:15.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f164effd640 1 -- 172.21.15.121:0/4120299481 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f165009ec60 msgr2=0x7f16500a01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:15.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f164effd640 1 --2- 172.21.15.121:0/4120299481 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f165009ec60 0x7f16500a01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:15.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f164effd640 10 monclient: _finish_hunting 0 2024-09-17T18:25:15.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f164effd640 1 monclient: found mon.noname-a 2024-09-17T18:25:15.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f164effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:25:15.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f164effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:25:15.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f164effd640 1 -- 172.21.15.121:0/4120299481 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1650137bf0 con 0x7f16500700f0 2024-09-17T18:25:15.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f164f7fe640 10 monclient: get_auth_request con 0x7f165009ec60 auth_method 0 2024-09-17T18:25:15.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f164f7fe640 1 --2- 172.21.15.121:0/4120299481 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f165009ec60 0x7f16500a01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T18:25:15.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.495+0000 7f1647fff640 1 --2- 172.21.15.121:0/4120299481 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f165009e810 0x7f165013f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:25:15.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.497+0000 7f164effd640 1 -- 172.21.15.121:0/4120299481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16500700f0 msgr2=0x7f165013f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:15.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.497+0000 7f164effd640 1 --2- 172.21.15.121:0/4120299481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16500700f0 0x7f165013f180 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f164000c460 tx=0x7f164000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:15.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.497+0000 7f164cff9640 10 client.?.objecter ms_handle_connect 0x7f16500700f0 2024-09-17T18:25:15.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.497+0000 7f164cff9640 10 client.?.objecter resend_mon_ops 2024-09-17T18:25:15.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.497+0000 7f164cff9640 1 -- 172.21.15.121:0/4120299481 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1640007070 con 0x7f16500700f0 2024-09-17T18:25:15.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.497+0000 7f164cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:15.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.497+0000 7f164cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:25:15.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.497+0000 7f164cff9640 10 monclient: dump: 2024-09-17T18:25:15.510 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:15.510 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:15.510 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.510 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.510 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:15.510 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:15.510 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:15.511 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:15.511 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:15.511 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:15.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.497+0000 7f164cff9640 10 monclient: _finish_auth 0 2024-09-17T18:25:15.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.497+0000 7f164cff9640 10 monclient: _check_auth_tickets 2024-09-17T18:25:15.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.497+0000 7f164cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:25:15.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.497+0000 7f16558d6640 5 monclient: authenticate success, global_id 44373 2024-09-17T18:25:15.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.497+0000 7f164cff9640 1 -- 172.21.15.121:0/4120299481 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1640007230 con 0x7f16500700f0 2024-09-17T18:25:15.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.497+0000 7f164cff9640 10 monclient: handle_config config(0 keys) 2024-09-17T18:25:15.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.497+0000 7f164cff9640 1 -- 172.21.15.121:0/4120299481 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1640007620 con 0x7f16500700f0 2024-09-17T18:25:15.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.497+0000 7f16558d6640 10 monclient: _renew_subs 2024-09-17T18:25:15.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.497+0000 7f16558d6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:15.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.497+0000 7f16558d6640 1 -- 172.21.15.121:0/4120299481 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16501348f0 con 0x7f16500700f0 2024-09-17T18:25:15.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.497+0000 7f164cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:15.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.497+0000 7f164cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:25:15.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.497+0000 7f164cff9640 10 monclient: dump: 2024-09-17T18:25:15.512 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:15.512 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:15.512 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.512 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.512 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:15.512 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:15.512 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:15.512 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:15.512 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:15.512 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:15.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.497+0000 7f164dffb640 4 set_mon_vals no callback set 2024-09-17T18:25:15.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.497+0000 7f16558d6640 10 client.44373.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:25:15.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.497+0000 7f16558d6640 10 monclient: _renew_subs 2024-09-17T18:25:15.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.497+0000 7f16558d6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:15.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.497+0000 7f16558d6640 1 -- 172.21.15.121:0/4120299481 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f165013fc00 con 0x7f16500700f0 2024-09-17T18:25:15.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.497+0000 7f16558d6640 1 librados: init done 2024-09-17T18:25:15.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.498+0000 7f16558d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:15.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.498+0000 7f16558d6640 1 --2- 172.21.15.121:0/4120299481 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16501cd7f0 0x7f16500a06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:15.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.498+0000 7f16558d6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:15.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.498+0000 7f16558d6640 1 -- 172.21.15.121:0/4120299481 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1650137590 con 0x7f16501cd7f0 2024-09-17T18:25:15.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.498+0000 7f1647fff640 1 --2- 172.21.15.121:0/4120299481 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16501cd7f0 0x7f16500a06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:15.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.498+0000 7f1647fff640 10 monclient: get_auth_request con 0x7f16501cd7f0 auth_method 0 2024-09-17T18:25:15.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.498+0000 7f1647fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:15.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.498+0000 7f1647fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:15.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.498+0000 7f1647fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:15.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.498+0000 7f164cff9640 1 -- 172.21.15.121:0/4120299481 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f164001b400 con 0x7f16500700f0 2024-09-17T18:25:15.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.498+0000 7f1647fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:15.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.498+0000 7f1647fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:15.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.499+0000 7f1647fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:15.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.499+0000 7f1647fff640 10 monclient(hunting): handle_auth_done global_id 54137 payload 824 2024-09-17T18:25:15.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.499+0000 7f1647fff640 1 -- 172.21.15.121:0/4120299481 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f16501373b0 con 0x7f16501cd7f0 2024-09-17T18:25:15.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.499+0000 7f164cff9640 1 --2- 172.21.15.121:0/4120299481 >> v2:172.21.15.173:6812/2423786037 conn(0x7f1630040170 0x7f1630042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:15.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.499+0000 7f164cff9640 1 -- 172.21.15.121:0/4120299481 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(286..286 src has 1..286) ==== 5705+0+0 (secure 0 0 0) 0x7f16400534f0 con 0x7f16500700f0 2024-09-17T18:25:15.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.499+0000 7f164cff9640 10 client.44373.objecter ms_dispatch 0x7f1650000b90 osd_map(286..286 src has 1..286) 2024-09-17T18:25:15.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.499+0000 7f164cff9640 3 client.44373.objecter handle_osd_map got epochs [286,286] > 0 2024-09-17T18:25:15.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.499+0000 7f164cff9640 3 client.44373.objecter handle_osd_map decoding full epoch 286 2024-09-17T18:25:15.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.499+0000 7f164cff9640 20 client.44373.objecter dump_active .. 0 homeless 2024-09-17T18:25:15.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.499+0000 7f164f7fe640 1 --2- 172.21.15.121:0/4120299481 >> v2:172.21.15.173:6812/2423786037 conn(0x7f1630040170 0x7f1630042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:15.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.499+0000 7f164f7fe640 10 monclient: get_auth_request con 0x7f1630040170 auth_method 0 2024-09-17T18:25:15.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.501+0000 7f164cff9640 10 client.44373.objecter ms_handle_connect 0x7f16501cd7f0 2024-09-17T18:25:15.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.501+0000 7f1647fff640 1 -- 172.21.15.121:0/4120299481 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16501cd7f0 msgr2=0x7f16500a06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:15.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.501+0000 7f164cff9640 10 client.44373.objecter resend_mon_ops 2024-09-17T18:25:15.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.501+0000 7f1647fff640 1 --2- 172.21.15.121:0/4120299481 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16501cd7f0 0x7f16500a06f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1650077b10 tx=0x7f16380046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:15.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.501+0000 7f164cff9640 1 -- 172.21.15.121:0/4120299481 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1638037040 con 0x7f16501cd7f0 2024-09-17T18:25:15.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.501+0000 7f164cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:15.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.501+0000 7f164cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:25:15.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.501+0000 7f164cff9640 10 monclient: dump: 2024-09-17T18:25:15.515 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:15.515 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:15.515 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.515 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.515 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:15.515 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:15.515 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:15.515 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:15.515 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:15.515 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:15.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.501+0000 7f164cff9640 1 -- 172.21.15.121:0/4120299481 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f163803cb10 con 0x7f16501cd7f0 2024-09-17T18:25:15.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.501+0000 7f164cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:15.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.501+0000 7f164cff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:25:15.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.501+0000 7f164cff9640 1 -- 172.21.15.121:0/4120299481 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16501cd7f0 msgr2=0x7f16500a06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:15.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.501+0000 7f164cff9640 1 --2- 172.21.15.121:0/4120299481 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16501cd7f0 0x7f16500a06f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1650077b10 tx=0x7f16380046c0 comp rx=0 tx=0).stop 2024-09-17T18:25:15.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.501+0000 7f164cff9640 1 -- 172.21.15.121:0/4120299481 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16501cd7f0 msgr2=0x7f16500a06f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:25:15.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.501+0000 7f164cff9640 1 --2- 172.21.15.121:0/4120299481 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16501cd7f0 0x7f16500a06f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:15.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.501+0000 7f164cff9640 10 client.44373.objecter ms_handle_connect 0x7f1630040170 2024-09-17T18:25:15.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.501+0000 7f164f7fe640 1 --2- 172.21.15.121:0/4120299481 >> v2:172.21.15.173:6812/2423786037 conn(0x7f1630040170 0x7f1630042630 secure :-1 s=READY pgs=3979 cs=0 l=1 rev1=1 crypto rx=0x7f1650136180 tx=0x7f1648007360 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:15.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.507+0000 7f16558d6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:25:15.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.507+0000 7f16558d6640 1 --2- 172.21.15.121:0/4120299481 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16500a0c30 0x7f16500a25d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:15.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.507+0000 7f16558d6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:15.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.507+0000 7f16558d6640 1 -- 172.21.15.121:0/4120299481 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f16500a1010 con 0x7f16500a0c30 2024-09-17T18:25:15.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.508+0000 7f1647fff640 1 --2- 172.21.15.121:0/4120299481 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16500a0c30 0x7f16500a25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:15.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.508+0000 7f1647fff640 10 monclient: get_auth_request con 0x7f16500a0c30 auth_method 0 2024-09-17T18:25:15.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.508+0000 7f1647fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:15.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.508+0000 7f1647fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:15.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.508+0000 7f1647fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:15.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.508+0000 7f1647fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:15.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.508+0000 7f1647fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:15.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.508+0000 7f1647fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:15.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.508+0000 7f1647fff640 10 monclient(hunting): handle_auth_done global_id 54140 payload 824 2024-09-17T18:25:15.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.508+0000 7f1647fff640 1 -- 172.21.15.121:0/4120299481 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1650134e50 con 0x7f16500a0c30 2024-09-17T18:25:15.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.510+0000 7f1647fff640 1 -- 172.21.15.121:0/4120299481 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16500a0c30 msgr2=0x7f16500a25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:15.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.510+0000 7f164cff9640 10 client.44373.objecter ms_handle_connect 0x7f16500a0c30 2024-09-17T18:25:15.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.510+0000 7f164cff9640 10 client.44373.objecter resend_mon_ops 2024-09-17T18:25:15.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.510+0000 7f1647fff640 1 --2- 172.21.15.121:0/4120299481 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16500a0c30 0x7f16500a25d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f163802a010 tx=0x7f1638008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:15.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.510+0000 7f164cff9640 1 -- 172.21.15.121:0/4120299481 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1638037030 con 0x7f16500a0c30 2024-09-17T18:25:15.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.510+0000 7f164cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:15.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.510+0000 7f164cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:25:15.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.510+0000 7f164cff9640 10 monclient: dump: 2024-09-17T18:25:15.519 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:15.519 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:15.519 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.519 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.519 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:15.519 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:15.520 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:15.520 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:15.520 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:15.520 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:15.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.510+0000 7f164cff9640 1 -- 172.21.15.121:0/4120299481 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7f1638055420 con 0x7f16500a0c30 2024-09-17T18:25:15.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.510+0000 7f164cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:25:15.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.510+0000 7f164cff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:25:15.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.510+0000 7f164cff9640 1 -- 172.21.15.121:0/4120299481 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16500a0c30 msgr2=0x7f16500a25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:15.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.510+0000 7f164cff9640 1 --2- 172.21.15.121:0/4120299481 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16500a0c30 0x7f16500a25d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f163802a010 tx=0x7f1638008030 comp rx=0 tx=0).stop 2024-09-17T18:25:15.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.510+0000 7f164cff9640 1 -- 172.21.15.121:0/4120299481 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16500a0c30 msgr2=0x7f16500a25d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:25:15.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.510+0000 7f164cff9640 1 --2- 172.21.15.121:0/4120299481 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16500a0c30 0x7f16500a25d0 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f163802a010 tx=0x7f1638008030 comp rx=0 tx=0).stop 2024-09-17T18:25:15.520 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:25:15.521 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:25:15.521 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:25:15.521 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:25:15.521 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 26, 2024-09-17T18:25:15.521 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 17708, 2024-09-17T18:25:15.521 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:25:15.521 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:25:15.521 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:25:15.521 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:25:15.521 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:25:15.521 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 7, 2024-09-17T18:25:15.521 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:25:15.522 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:25:15.522 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:25:15.522 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:25:15.522 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:25:15.522 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:25:15.522 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:25:15.522 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:25:15.522 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:25:15.522 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:25:15.522 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:25:15.522 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:25:15.522 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:25:15.522 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:25:15.522 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:25:15.522 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:25:15.522 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:25:15.523 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:25:15.523 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:25:15.523 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:25:15.523 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:25:15.523 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:25:15.523 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:25:15.523 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:25:15.523 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:25:15.523 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:25:15.523 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:25:15.523 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:25:15.523 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:15.523 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:15.523 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:25:15.523 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:25:15.524 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:25:15.524 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:25:15.524 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:25:15.524 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:25:15.524 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:25:15.524 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:25:15.524 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:25:15.524 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:25:15.524 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:25:15.524 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:25:15.524 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:25:15.524 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:25:15.524 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:25:15.524 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:25:15.524 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:25:15.524 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:25:15.525 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:25:15.525 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:25:15.525 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:25:15.525 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:25:15.525 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:25:15.525 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:25:15.525 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:25:15.525 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:25:15.525 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:25:15.526 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:25:15.526 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:25:15.526 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:25:15.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:15.526 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:25:15.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:15.526 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:25:15.526 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:25:15.526 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:25:15.526 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:25:15.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:15.526 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:25:15.526 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:25:15.526 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:15.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:15.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:15.527 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:25:15.527 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:25:15.527 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:15.527 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:15.527 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:15.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:15.527 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:25:15.527 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:25:15.527 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:25:15.527 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:25:15.527 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:25:15.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:15.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:15.527 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:25:15.528 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:25:15.528 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:25:15.528 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:25:15.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:15.528 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:25:15.528 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:25:15.528 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:15.528 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:15.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:15.528 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:25:15.528 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:25:15.528 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:15.528 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:15.528 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:15.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:15.529 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:25:15.529 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:25:15.529 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:25:15.529 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:25:15.529 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:25:15.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:15.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:15.529 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:25:15.529 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:25:15.529 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:25:15.529 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:25:15.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:15.529 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:25:15.529 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:25:15.530 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:15.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:15.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:15.530 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:25:15.530 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:25:15.530 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:15.530 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:15.530 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:15.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:15.530 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:25:15.530 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:25:15.530 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:25:15.530 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:25:15.530 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:25:15.530 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:15.530 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:15.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:15.531 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:25:15.531 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:25:15.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:15.531 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:25:15.531 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:25:15.531 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:25:15.531 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:15.531 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:25:15.531 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:25:15.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:15.531 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:25:15.531 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:25:15.531 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:25:15.531 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:15.532 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:15.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:15.532 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:25:15.532 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:25:15.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.511+0000 7f16558d6640 10 librados: watch_flush enter 2024-09-17T18:25:15.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.511+0000 7f16558d6640 10 librados: watch_flush exit 2024-09-17T18:25:15.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.511+0000 7f16558d6640 20 client.44373.objecter shutdown clearing up homeless session... 2024-09-17T18:25:15.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.511+0000 7f16558d6640 10 client.44373.objecter successfully canceled tick 2024-09-17T18:25:15.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.511+0000 7f16558d6640 1 -- 172.21.15.121:0/4120299481 >> v2:172.21.15.173:6812/2423786037 conn(0x7f1630040170 msgr2=0x7f1630042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:15.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.511+0000 7f16558d6640 1 --2- 172.21.15.121:0/4120299481 >> v2:172.21.15.173:6812/2423786037 conn(0x7f1630040170 0x7f1630042630 secure :-1 s=READY pgs=3979 cs=0 l=1 rev1=1 crypto rx=0x7f1650136180 tx=0x7f1648007360 comp rx=0 tx=0).stop 2024-09-17T18:25:15.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.511+0000 7f16558d6640 10 monclient: shutdown 2024-09-17T18:25:15.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.511+0000 7f16558d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:25:15.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.511+0000 7f16558d6640 1 -- 172.21.15.121:0/4120299481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16500700f0 msgr2=0x7f165013f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:15.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.511+0000 7f16558d6640 1 --2- 172.21.15.121:0/4120299481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16500700f0 0x7f165013f180 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f164000c460 tx=0x7f164000c920 comp rx=0 tx=0).stop 2024-09-17T18:25:15.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.511+0000 7f164f7fe640 1 -- 172.21.15.121:0/4120299481 reap_dead start 2024-09-17T18:25:15.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.511+0000 7f16558d6640 1 -- 172.21.15.121:0/4120299481 shutdown_connections 2024-09-17T18:25:15.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.511+0000 7f16558d6640 1 --2- 172.21.15.121:0/4120299481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16500700f0 0x7f165013f180 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:15.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.511+0000 7f16558d6640 1 -- 172.21.15.121:0/4120299481 >> 172.21.15.121:0/4120299481 conn(0x7f16500ab830 msgr2=0x7f1650131b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:25:15.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.511+0000 7f16558d6640 1 -- 172.21.15.121:0/4120299481 shutdown_connections 2024-09-17T18:25:15.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.511+0000 7f16558d6640 1 -- 172.21.15.121:0/4120299481 wait complete. 2024-09-17T18:25:15.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.511+0000 7f16558d6640 1 librados: shutdown 2024-09-17T18:25:15.535 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:25:15.535 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T18:25:15.536 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T18:25:15.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.634+0000 7ff044da3640 10 monclient: get_monmap_and_config 2024-09-17T18:25:15.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.634+0000 7ff044da3640 10 monclient: build_initial_monmap 2024-09-17T18:25:15.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.634+0000 7ff044da3640 10 monclient: monmap: 2024-09-17T18:25:15.637 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:25:15.637 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:15.637 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:25:15.636368+0000 2024-09-17T18:25:15.638 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:25:15.636368+0000 2024-09-17T18:25:15.638 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:25:15.638 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:15.638 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:25:15.638 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:25:15.638 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:25:15.638 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:25:15.638 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:25:15.638 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:15.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.635+0000 7ff044da3640 1 Processor -- start 2024-09-17T18:25:15.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.635+0000 7ff044da3640 1 -- start start 2024-09-17T18:25:15.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.635+0000 7ff044da3640 10 monclient: init 2024-09-17T18:25:15.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.635+0000 7ff044da3640 10 monclient: _reopen_session rank -1 2024-09-17T18:25:15.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.635+0000 7ff044da3640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-17T18:25:15.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.635+0000 7ff044da3640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff0400a0a00 0x7ff0400a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:15.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.636+0000 7ff044da3640 10 monclient(hunting): picked mon.noname-a con 0x7ff0400a0a00 addr v2:172.21.15.121:3300/0 2024-09-17T18:25:15.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.636+0000 7ff044da3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff0400a1320 addr v1:172.21.15.121:6789/0 2024-09-17T18:25:15.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.636+0000 7ff044da3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff040135090 addr v1:172.21.15.173:6789/0 2024-09-17T18:25:15.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.636+0000 7ff044da3640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0400a3820 con 0x7ff0400a1320 2024-09-17T18:25:15.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.636+0000 7ff044da3640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0400a2ea0 con 0x7ff040135090 2024-09-17T18:25:15.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.636+0000 7ff044da3640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:15.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.636+0000 7ff044da3640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7ff04009e800 con 0x7ff0400a0a00 2024-09-17T18:25:15.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.636+0000 7ff044da3640 10 monclient(hunting): _renew_subs 2024-09-17T18:25:15.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.636+0000 7ff044da3640 10 monclient(hunting): authenticate will time out at 1119.353083s 2024-09-17T18:25:15.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.636+0000 7ff03e59c640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff0400a0a00 0x7ff0400a0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:15.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.636+0000 7ff03dd9b640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7ff0400a1320 0x7ff040134980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:38390/0 (socket says 172.21.15.121:38390) 2024-09-17T18:25:15.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.636+0000 7ff03e59c640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff0400a0a00 0x7ff0400a0de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44614/0 (socket says 172.21.15.121:44614) 2024-09-17T18:25:15.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.636+0000 7ff03dd9b640 1 -- 172.21.15.121:0/1304223956 learned_addr learned my addr 172.21.15.121:0/1304223956 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:25:15.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.636+0000 7ff03e59c640 10 monclient(hunting): get_auth_request con 0x7ff0400a0a00 auth_method 0 2024-09-17T18:25:15.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.636+0000 7ff03e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:15.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.636+0000 7ff03e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:15.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.636+0000 7ff03e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:15.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.636+0000 7ff03e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:15.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.636+0000 7ff03e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:15.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.636+0000 7ff03e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:15.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.636+0000 7ff03e59c640 10 monclient(hunting): handle_auth_done global_id 44376 payload 274 2024-09-17T18:25:15.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.636+0000 7ff03e59c640 1 -- 172.21.15.121:0/1304223956 >> v1:172.21.15.173:6789/0 conn(0x7ff040135090 legacy=0x7ff04013ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:25:15.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.636+0000 7ff03e59c640 1 -- 172.21.15.121:0/1304223956 >> v1:172.21.15.121:6789/0 conn(0x7ff0400a1320 legacy=0x7ff040134980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:25:15.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.636+0000 7ff03e59c640 10 monclient: _finish_hunting 0 2024-09-17T18:25:15.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.636+0000 7ff03e59c640 1 monclient: found mon.noname-a 2024-09-17T18:25:15.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.636+0000 7ff03e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:25:15.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.636+0000 7ff03e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:25:15.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.636+0000 7ff03e59c640 1 -- 172.21.15.121:0/1304223956 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0400a2c30 con 0x7ff0400a0a00 2024-09-17T18:25:15.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.639+0000 7ff03e59c640 1 -- 172.21.15.121:0/1304223956 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0400a0a00 msgr2=0x7ff0400a0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:15.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.639+0000 7ff03e59c640 1 --2- 172.21.15.121:0/1304223956 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0400a0a00 0x7ff0400a0de0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff03000aea0 tx=0x7ff03002b5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2008f2fb56ed51b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:15.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.639+0000 7ff03d59a640 1 -- 172.21.15.121:0/1304223956 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff030037070 con 0x7ff0400a0a00 2024-09-17T18:25:15.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.639+0000 7ff03d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:15.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.639+0000 7ff03d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:25:15.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.639+0000 7ff03d59a640 10 monclient: dump: 2024-09-17T18:25:15.650 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:15.650 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:15.650 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.650 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.650 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:15.650 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:15.650 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:15.650 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:15.650 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:15.650 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:15.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.639+0000 7ff03d59a640 10 monclient: _finish_auth 0 2024-09-17T18:25:15.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.639+0000 7ff03d59a640 10 monclient: _check_auth_tickets 2024-09-17T18:25:15.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.639+0000 7ff03d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:25:15.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.639+0000 7ff03d59a640 1 -- 172.21.15.121:0/1304223956 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff030008850 con 0x7ff0400a0a00 2024-09-17T18:25:15.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.639+0000 7ff044da3640 5 monclient: authenticate success, global_id 44376 2024-09-17T18:25:15.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.639+0000 7ff044da3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:25:15.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.639+0000 7ff03d59a640 10 monclient: handle_config config(0 keys) 2024-09-17T18:25:15.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.639+0000 7ff044da3640 10 monclient: get_monmap_and_config success 2024-09-17T18:25:15.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.639+0000 7ff044da3640 4 set_mon_vals no callback set 2024-09-17T18:25:15.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.639+0000 7ff03d59a640 1 -- 172.21.15.121:0/1304223956 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff030008c20 con 0x7ff0400a0a00 2024-09-17T18:25:15.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.639+0000 7ff03d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:15.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.639+0000 7ff03d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:25:15.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.639+0000 7ff03d59a640 10 monclient: dump: 2024-09-17T18:25:15.651 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:15.652 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:15.652 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.652 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.652 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:15.652 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:15.652 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:15.652 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:15.652 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:15.652 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:15.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.639+0000 7ff044da3640 10 monclient: shutdown 2024-09-17T18:25:15.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.639+0000 7ff044da3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:25:15.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.639+0000 7ff044da3640 1 -- 172.21.15.121:0/1304223956 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0400a0a00 msgr2=0x7ff0400a0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:15.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.639+0000 7ff044da3640 1 --2- 172.21.15.121:0/1304223956 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0400a0a00 0x7ff0400a0de0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff03000aea0 tx=0x7ff03002b5e0 comp rx=0 tx=0).stop 2024-09-17T18:25:15.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.639+0000 7ff044da3640 1 -- 172.21.15.121:0/1304223956 shutdown_connections 2024-09-17T18:25:15.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.639+0000 7ff044da3640 1 --2- 172.21.15.121:0/1304223956 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0400a0a00 0x7ff0400a0de0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:15.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.639+0000 7ff044da3640 1 -- 172.21.15.121:0/1304223956 >> 172.21.15.121:0/1304223956 conn(0x7ff0400a7830 msgr2=0x7ff0400a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:25:15.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.640+0000 7ff044da3640 1 -- 172.21.15.121:0/1304223956 shutdown_connections 2024-09-17T18:25:15.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.640+0000 7ff044da3640 1 -- 172.21.15.121:0/1304223956 wait complete. 2024-09-17T18:25:15.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.640+0000 7ff044da3640 10 monclient: build_initial_monmap 2024-09-17T18:25:15.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.640+0000 7ff044da3640 10 monclient: monmap: 2024-09-17T18:25:15.653 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:25:15.653 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:25:15.653 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:25:15.653 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:25:15.653 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:25:15.653 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:15.653 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:25:15.654 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:25:15.654 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:25:15.654 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:15.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.640+0000 7ff044da3640 1 librados: starting msgr at 2024-09-17T18:25:15.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.640+0000 7ff044da3640 1 librados: starting objecter 2024-09-17T18:25:15.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff044da3640 1 Processor -- start 2024-09-17T18:25:15.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff044da3640 1 -- start start 2024-09-17T18:25:15.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff044da3640 1 librados: setting wanted keys 2024-09-17T18:25:15.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff044da3640 1 librados: calling monclient init 2024-09-17T18:25:15.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff044da3640 10 monclient: init 2024-09-17T18:25:15.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff044da3640 10 monclient: _reopen_session rank -1 2024-09-17T18:25:15.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff044da3640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T18:25:15.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff044da3640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff0400a0a00 0x7ff0401cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:15.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff044da3640 10 monclient(hunting): picked mon.noname-b con 0x7ff0400a0a00 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:25:15.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff044da3640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0400a1320 0x7ff0401ce060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:15.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff044da3640 10 monclient(hunting): picked mon.noname-c con 0x7ff0400a1320 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:25:15.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff044da3640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff040135090 0x7ff04006f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:15.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff044da3640 10 monclient(hunting): picked mon.noname-a con 0x7ff040135090 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:25:15.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff044da3640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:15.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff044da3640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff0400a3120 con 0x7ff040135090 2024-09-17T18:25:15.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff044da3640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:15.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff044da3640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff04009ecf0 con 0x7ff0400a0a00 2024-09-17T18:25:15.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff044da3640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:15.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff044da3640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff0400a32a0 con 0x7ff0400a1320 2024-09-17T18:25:15.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff044da3640 10 monclient(hunting): _renew_subs 2024-09-17T18:25:15.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff044da3640 10 monclient(hunting): authenticate will time out at 1119.358549s 2024-09-17T18:25:15.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff03dd9b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0400a1320 0x7ff0401ce060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:15.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff03ed9d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff040135090 0x7ff04006f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:15.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff03ed9d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff040135090 0x7ff04006f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44622/0 (socket says 172.21.15.121:44622) 2024-09-17T18:25:15.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff03dd9b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0400a1320 0x7ff0401ce060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:53566/0 (socket says 172.21.15.121:53566) 2024-09-17T18:25:15.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff03ed9d640 1 -- 172.21.15.121:0/3719990398 learned_addr learned my addr 172.21.15.121:0/3719990398 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:25:15.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff03ed9d640 10 monclient(hunting): get_auth_request con 0x7ff040135090 auth_method 0 2024-09-17T18:25:15.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff03ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:15.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff03ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:15.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff03ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:15.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff03dd9b640 10 monclient(hunting): get_auth_request con 0x7ff0400a1320 auth_method 0 2024-09-17T18:25:15.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff03dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:15.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff03dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:15.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff03dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:15.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff03ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:15.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff03ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:15.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff03e59c640 1 --2- 172.21.15.121:0/3719990398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff0400a0a00 0x7ff0401cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:15.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff03ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:15.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff03dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:15.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff03dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:15.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff03dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:15.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff03e59c640 10 monclient(hunting): get_auth_request con 0x7ff0400a0a00 auth_method 0 2024-09-17T18:25:15.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff03e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:15.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff03e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:15.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff03e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:15.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff03e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:15.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff03e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:15.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff03e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:15.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff03dd9b640 10 monclient(hunting): handle_auth_done global_id 54143 payload 824 2024-09-17T18:25:15.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff03dd9b640 1 -- 172.21.15.121:0/3719990398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff0400a0a00 msgr2=0x7ff0401cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:15.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff03dd9b640 1 --2- 172.21.15.121:0/3719990398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff0400a0a00 0x7ff0401cdb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:15.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff03dd9b640 1 -- 172.21.15.121:0/3719990398 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff040135090 msgr2=0x7ff04006f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:15.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff03dd9b640 1 --2- 172.21.15.121:0/3719990398 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff040135090 0x7ff04006f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:15.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff03dd9b640 10 monclient: _finish_hunting 0 2024-09-17T18:25:15.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff03dd9b640 1 monclient: found mon.noname-c 2024-09-17T18:25:15.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff03dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:25:15.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff03dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:25:15.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff03dd9b640 1 -- 172.21.15.121:0/3719990398 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff040133f20 con 0x7ff0400a1320 2024-09-17T18:25:15.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.641+0000 7ff03ed9d640 1 --2- 172.21.15.121:0/3719990398 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff040135090 0x7ff04006f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:25:15.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.644+0000 7ff03dd9b640 1 -- 172.21.15.121:0/3719990398 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0400a1320 msgr2=0x7ff0401ce060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:15.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.644+0000 7ff03dd9b640 1 --2- 172.21.15.121:0/3719990398 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0400a1320 0x7ff0401ce060 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff03400c3e0 tx=0x7ff03400c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:15.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.644+0000 7ff02effd640 10 client.?.objecter ms_handle_connect 0x7ff0400a1320 2024-09-17T18:25:15.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.644+0000 7ff02effd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:25:15.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.644+0000 7ff02effd640 1 -- 172.21.15.121:0/3719990398 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff034017070 con 0x7ff0400a1320 2024-09-17T18:25:15.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.644+0000 7ff02effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:15.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.644+0000 7ff02effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:25:15.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.644+0000 7ff02effd640 10 monclient: dump: 2024-09-17T18:25:15.659 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:15.659 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:15.659 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.659 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.659 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:15.659 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:15.659 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:15.659 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:15.659 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:15.660 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:15.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.644+0000 7ff02effd640 10 monclient: _finish_auth 0 2024-09-17T18:25:15.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.644+0000 7ff02effd640 10 monclient: _check_auth_tickets 2024-09-17T18:25:15.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.644+0000 7ff02effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:25:15.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.644+0000 7ff02effd640 1 -- 172.21.15.121:0/3719990398 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0340070d0 con 0x7ff0400a1320 2024-09-17T18:25:15.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.644+0000 7ff044da3640 5 monclient: authenticate success, global_id 54143 2024-09-17T18:25:15.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.644+0000 7ff044da3640 10 monclient: _renew_subs 2024-09-17T18:25:15.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.644+0000 7ff044da3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:25:15.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.644+0000 7ff044da3640 1 -- 172.21.15.121:0/3719990398 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff040128cf0 con 0x7ff0400a1320 2024-09-17T18:25:15.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.644+0000 7ff02effd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:25:15.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.644+0000 7ff02effd640 1 -- 172.21.15.121:0/3719990398 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0340074c0 con 0x7ff0400a1320 2024-09-17T18:25:15.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.644+0000 7ff02effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:15.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.644+0000 7ff02effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:25:15.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.644+0000 7ff02effd640 10 monclient: dump: 2024-09-17T18:25:15.661 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:15.661 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:15.661 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.661 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.661 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:15.661 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:15.661 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:15.661 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:15.661 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:15.661 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:15.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.644+0000 7ff03cd99640 4 set_mon_vals no callback set 2024-09-17T18:25:15.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.644+0000 7ff044da3640 10 client.54143.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:25:15.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.644+0000 7ff044da3640 10 monclient: _renew_subs 2024-09-17T18:25:15.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.644+0000 7ff044da3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:25:15.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.644+0000 7ff044da3640 1 -- 172.21.15.121:0/3719990398 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff040128a10 con 0x7ff0400a1320 2024-09-17T18:25:15.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.644+0000 7ff044da3640 1 librados: init done 2024-09-17T18:25:15.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.645+0000 7ff02effd640 1 -- 172.21.15.121:0/3719990398 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7ff03401a070 con 0x7ff0400a1320 2024-09-17T18:25:15.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.645+0000 7ff044da3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:15.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.645+0000 7ff044da3640 1 --2- 172.21.15.121:0/3719990398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff04006f830 0x7ff04006fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:15.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.645+0000 7ff044da3640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:15.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.645+0000 7ff044da3640 1 -- 172.21.15.121:0/3719990398 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff0401348a0 con 0x7ff04006f830 2024-09-17T18:25:15.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.645+0000 7ff02effd640 1 --2- 172.21.15.121:0/3719990398 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff0200401f0 0x7ff0200426b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:15.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.645+0000 7ff02effd640 1 -- 172.21.15.121:0/3719990398 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(287..287 src has 1..287) ==== 5814+0+0 (secure 0 0 0) 0x7ff034052d50 con 0x7ff0400a1320 2024-09-17T18:25:15.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.645+0000 7ff02effd640 10 client.54143.objecter ms_dispatch 0x7ff040000b90 osd_map(287..287 src has 1..287) 2024-09-17T18:25:15.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.645+0000 7ff02effd640 3 client.54143.objecter handle_osd_map got epochs [287,287] > 0 2024-09-17T18:25:15.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.645+0000 7ff02effd640 3 client.54143.objecter handle_osd_map decoding full epoch 287 2024-09-17T18:25:15.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.645+0000 7ff02effd640 20 client.54143.objecter dump_active .. 0 homeless 2024-09-17T18:25:15.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.645+0000 7ff03e59c640 1 --2- 172.21.15.121:0/3719990398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff04006f830 0x7ff04006fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:15.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.645+0000 7ff03e59c640 10 monclient: get_auth_request con 0x7ff04006f830 auth_method 0 2024-09-17T18:25:15.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.645+0000 7ff03e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:15.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.645+0000 7ff03e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:15.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.645+0000 7ff03e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:15.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.645+0000 7ff03ed9d640 1 --2- 172.21.15.121:0/3719990398 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff0200401f0 0x7ff0200426b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:15.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.645+0000 7ff03ed9d640 10 monclient: get_auth_request con 0x7ff0200401f0 auth_method 0 2024-09-17T18:25:15.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.645+0000 7ff03e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:15.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.645+0000 7ff03e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:15.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.645+0000 7ff03e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:15.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.646+0000 7ff03e59c640 10 monclient(hunting): handle_auth_done global_id 24718 payload 824 2024-09-17T18:25:15.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.646+0000 7ff03e59c640 1 -- 172.21.15.121:0/3719990398 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff04009e290 con 0x7ff04006f830 2024-09-17T18:25:15.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.648+0000 7ff02effd640 10 client.54143.objecter ms_handle_connect 0x7ff04006f830 2024-09-17T18:25:15.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.648+0000 7ff03e59c640 1 -- 172.21.15.121:0/3719990398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff04006f830 msgr2=0x7ff04006fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:15.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.648+0000 7ff02effd640 10 client.54143.objecter resend_mon_ops 2024-09-17T18:25:15.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.648+0000 7ff03e59c640 1 --2- 172.21.15.121:0/3719990398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff04006f830 0x7ff04006fc10 secure :-1 s=READY pgs=7383 cs=0 l=1 rev1=1 crypto rx=0x7ff030034000 tx=0x7ff030004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:15.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.648+0000 7ff02effd640 10 client.54143.objecter ms_handle_connect 0x7ff0200401f0 2024-09-17T18:25:15.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.648+0000 7ff03ed9d640 1 --2- 172.21.15.121:0/3719990398 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff0200401f0 0x7ff0200426b0 secure :-1 s=READY pgs=3980 cs=0 l=1 rev1=1 crypto rx=0x7ff038004360 tx=0x7ff038007e70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:15.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.648+0000 7ff02effd640 1 -- 172.21.15.121:0/3719990398 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff030043070 con 0x7ff04006f830 2024-09-17T18:25:15.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.648+0000 7ff02effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:15.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.648+0000 7ff02effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:25:15.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.648+0000 7ff02effd640 10 monclient: dump: 2024-09-17T18:25:15.666 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:15.667 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:15.667 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.667 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.667 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:15.667 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:15.667 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:15.667 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:15.667 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:15.667 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:15.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.648+0000 7ff02effd640 1 -- 172.21.15.121:0/3719990398 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff030038040 con 0x7ff04006f830 2024-09-17T18:25:15.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.648+0000 7ff02effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:15.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.648+0000 7ff02effd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:25:15.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.648+0000 7ff02effd640 1 -- 172.21.15.121:0/3719990398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff04006f830 msgr2=0x7ff04006fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:15.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.648+0000 7ff02effd640 1 --2- 172.21.15.121:0/3719990398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff04006f830 0x7ff04006fc10 secure :-1 s=READY pgs=7383 cs=0 l=1 rev1=1 crypto rx=0x7ff030034000 tx=0x7ff030004a40 comp rx=0 tx=0).stop 2024-09-17T18:25:15.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.648+0000 7ff02effd640 1 -- 172.21.15.121:0/3719990398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff04006f830 msgr2=0x7ff04006fc10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:25:15.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.648+0000 7ff02effd640 1 --2- 172.21.15.121:0/3719990398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff04006f830 0x7ff04006fc10 secure :-1 s=CLOSED pgs=7383 cs=0 l=1 rev1=1 crypto rx=0x7ff030034000 tx=0x7ff030004a40 comp rx=0 tx=0).stop 2024-09-17T18:25:15.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.654+0000 7ff044da3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:25:15.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.654+0000 7ff044da3640 1 --2- 172.21.15.121:0/3719990398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff040070150 0x7ff04011f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:15.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.654+0000 7ff044da3640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:15.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.654+0000 7ff044da3640 1 -- 172.21.15.121:0/3719990398 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff0400a2080 con 0x7ff040070150 2024-09-17T18:25:15.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.655+0000 7ff03e59c640 1 --2- 172.21.15.121:0/3719990398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff040070150 0x7ff04011f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:15.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.655+0000 7ff03e59c640 10 monclient: get_auth_request con 0x7ff040070150 auth_method 0 2024-09-17T18:25:15.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.655+0000 7ff03e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:15.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.655+0000 7ff03e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:15.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.655+0000 7ff03e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:15.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.655+0000 7ff03e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:15.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.655+0000 7ff03e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:15.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.655+0000 7ff03e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:15.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.655+0000 7ff03e59c640 10 monclient(hunting): handle_auth_done global_id 24721 payload 824 2024-09-17T18:25:15.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.655+0000 7ff03e59c640 1 -- 172.21.15.121:0/3719990398 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff04006cd40 con 0x7ff040070150 2024-09-17T18:25:15.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.657+0000 7ff03e59c640 1 -- 172.21.15.121:0/3719990398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff040070150 msgr2=0x7ff04011f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:15.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.657+0000 7ff02effd640 10 client.54143.objecter ms_handle_connect 0x7ff040070150 2024-09-17T18:25:15.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.657+0000 7ff03e59c640 1 --2- 172.21.15.121:0/3719990398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff040070150 0x7ff04011f6b0 secure :-1 s=READY pgs=7384 cs=0 l=1 rev1=1 crypto rx=0x7ff030000f80 tx=0x7ff030030000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:15.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.657+0000 7ff02effd640 10 client.54143.objecter resend_mon_ops 2024-09-17T18:25:15.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.658+0000 7ff02effd640 1 -- 172.21.15.121:0/3719990398 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff030037040 con 0x7ff040070150 2024-09-17T18:25:15.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.658+0000 7ff02effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:15.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.658+0000 7ff02effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:25:15.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.658+0000 7ff02effd640 10 monclient: dump: 2024-09-17T18:25:15.669 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:15.669 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:15.669 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.670 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.670 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:15.670 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:15.670 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:15.670 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:15.670 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:15.670 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:15.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.658+0000 7ff02effd640 1 -- 172.21.15.121:0/3719990398 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7ff030004460 con 0x7ff040070150 2024-09-17T18:25:15.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.658+0000 7ff02effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:25:15.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.658+0000 7ff02effd640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:25:15.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.658+0000 7ff02effd640 1 -- 172.21.15.121:0/3719990398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff040070150 msgr2=0x7ff04011f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:15.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.658+0000 7ff02effd640 1 --2- 172.21.15.121:0/3719990398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff040070150 0x7ff04011f6b0 secure :-1 s=READY pgs=7384 cs=0 l=1 rev1=1 crypto rx=0x7ff030000f80 tx=0x7ff030030000 comp rx=0 tx=0).stop 2024-09-17T18:25:15.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.658+0000 7ff02effd640 1 -- 172.21.15.121:0/3719990398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff040070150 msgr2=0x7ff04011f6b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:25:15.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.658+0000 7ff02effd640 1 --2- 172.21.15.121:0/3719990398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff040070150 0x7ff04011f6b0 secure :-1 s=CLOSED pgs=7384 cs=0 l=1 rev1=1 crypto rx=0x7ff030000f80 tx=0x7ff030030000 comp rx=0 tx=0).stop 2024-09-17T18:25:15.671 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:25:15.671 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:25:15.671 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:25:15.671 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:25:15.671 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 26, 2024-09-17T18:25:15.671 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 388604, 2024-09-17T18:25:15.671 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:25:15.671 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:25:15.672 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:25:15.672 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:25:15.672 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:25:15.672 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 7, 2024-09-17T18:25:15.672 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:25:15.672 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:25:15.672 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:25:15.672 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:25:15.672 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:25:15.672 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:25:15.672 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:25:15.672 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:25:15.672 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:25:15.672 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:25:15.673 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:25:15.673 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:25:15.673 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:25:15.673 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:25:15.673 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:25:15.673 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:25:15.673 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:25:15.673 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:25:15.673 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:25:15.673 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:25:15.673 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:25:15.673 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:25:15.673 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:25:15.674 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:25:15.674 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:25:15.674 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:25:15.674 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:25:15.674 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:25:15.674 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:15.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:15.674 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:25:15.674 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:25:15.674 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:25:15.674 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:25:15.674 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:25:15.674 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:25:15.675 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:25:15.675 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:25:15.675 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:25:15.675 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:25:15.675 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:25:15.675 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:25:15.675 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:25:15.675 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:25:15.675 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:25:15.675 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:25:15.675 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:25:15.675 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:25:15.675 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:25:15.676 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:25:15.676 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:25:15.676 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:25:15.676 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:25:15.676 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:25:15.676 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:25:15.676 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:25:15.676 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:25:15.676 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:25:15.676 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:25:15.676 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:25:15.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:15.676 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:25:15.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:15.677 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:25:15.677 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:25:15.677 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:25:15.677 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:25:15.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:15.677 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:25:15.677 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:25:15.677 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:15.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:15.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:15.677 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:25:15.677 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:25:15.677 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:15.678 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:15.678 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:15.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:15.678 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:25:15.678 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:25:15.678 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:25:15.678 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:25:15.678 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:25:15.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:15.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:15.678 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:25:15.678 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:25:15.678 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:25:15.678 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:25:15.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:15.679 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:25:15.679 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:25:15.679 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:15.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:15.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:15.679 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:25:15.679 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:25:15.679 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:15.679 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:15.679 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:15.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:15.679 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:25:15.679 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:25:15.679 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:25:15.680 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:25:15.680 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:25:15.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:15.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:15.680 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:25:15.680 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:25:15.680 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:25:15.680 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:25:15.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:15.680 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:25:15.680 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:25:15.680 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:15.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:15.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:15.681 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:25:15.681 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:25:15.681 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:15.681 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:15.681 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:15.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:15.681 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:25:15.681 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:25:15.681 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:25:15.681 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:25:15.681 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:25:15.681 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:15.681 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:15.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:15.681 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:25:15.682 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:25:15.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:15.682 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:25:15.682 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:25:15.682 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:25:15.682 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:15.682 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:25:15.682 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:25:15.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:15.682 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:25:15.682 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:25:15.682 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 4 2024-09-17T18:25:15.682 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:15.682 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:25:15.682 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:25:15.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:15.683 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:25:15.683 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:25:15.683 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 7 2024-09-17T18:25:15.683 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:15.683 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:25:15.683 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": [ 2024-09-17T18:25:15.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:15.683 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:25:15.683 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:25:15.683 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:25:15.683 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:15.683 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:15.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:15.683 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:25:15.683 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:25:15.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.658+0000 7ff044da3640 10 librados: watch_flush enter 2024-09-17T18:25:15.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.658+0000 7ff044da3640 10 librados: watch_flush exit 2024-09-17T18:25:15.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.658+0000 7ff044da3640 20 client.54143.objecter shutdown clearing up homeless session... 2024-09-17T18:25:15.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.658+0000 7ff044da3640 10 client.54143.objecter successfully canceled tick 2024-09-17T18:25:15.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.658+0000 7ff044da3640 1 -- 172.21.15.121:0/3719990398 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff0200401f0 msgr2=0x7ff0200426b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:15.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.658+0000 7ff044da3640 1 --2- 172.21.15.121:0/3719990398 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff0200401f0 0x7ff0200426b0 secure :-1 s=READY pgs=3980 cs=0 l=1 rev1=1 crypto rx=0x7ff038004360 tx=0x7ff038007e70 comp rx=0 tx=0).stop 2024-09-17T18:25:15.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.658+0000 7ff044da3640 10 monclient: shutdown 2024-09-17T18:25:15.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.658+0000 7ff044da3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:25:15.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.658+0000 7ff044da3640 1 -- 172.21.15.121:0/3719990398 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0400a1320 msgr2=0x7ff0401ce060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:15.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.658+0000 7ff044da3640 1 --2- 172.21.15.121:0/3719990398 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0400a1320 0x7ff0401ce060 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff03400c3e0 tx=0x7ff03400c8a0 comp rx=0 tx=0).stop 2024-09-17T18:25:15.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.658+0000 7ff03ed9d640 1 -- 172.21.15.121:0/3719990398 reap_dead start 2024-09-17T18:25:15.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.658+0000 7ff044da3640 1 -- 172.21.15.121:0/3719990398 shutdown_connections 2024-09-17T18:25:15.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.658+0000 7ff044da3640 1 --2- 172.21.15.121:0/3719990398 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0400a1320 0x7ff0401ce060 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:15.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.658+0000 7ff044da3640 1 -- 172.21.15.121:0/3719990398 >> 172.21.15.121:0/3719990398 conn(0x7ff0400a7830 msgr2=0x7ff0400a53d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:25:15.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.658+0000 7ff044da3640 1 -- 172.21.15.121:0/3719990398 shutdown_connections 2024-09-17T18:25:15.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.658+0000 7ff044da3640 1 -- 172.21.15.121:0/3719990398 wait complete. 2024-09-17T18:25:15.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.658+0000 7ff044da3640 1 librados: shutdown 2024-09-17T18:25:15.686 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:25:15.687 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-17T18:25:15.687 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-17T18:25:15.687 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-17T18:25:15.687 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-17T18:25:15.687 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-17T18:25:15.687 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T18:25:15.698 INFO:tasks.ceph.mon.c:Stopped 2024-09-17T18:25:15.698 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-17T18:25:15.698 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:25:15.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.832+0000 7fc8d2b6b640 10 monclient: get_monmap_and_config 2024-09-17T18:25:15.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.832+0000 7fc8d2b6b640 10 monclient: build_initial_monmap 2024-09-17T18:25:15.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.832+0000 7fc8d2b6b640 10 monclient: monmap: 2024-09-17T18:25:15.835 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:25:15.835 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:15.836 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:25:15.833992+0000 2024-09-17T18:25:15.836 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:25:15.833992+0000 2024-09-17T18:25:15.836 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:25:15.836 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:15.836 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:25:15.836 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:25:15.836 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:25:15.836 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:25:15.836 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:25:15.836 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:15.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.833+0000 7fc8d2b6b640 1 Processor -- start 2024-09-17T18:25:15.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.833+0000 7fc8d2b6b640 1 -- start start 2024-09-17T18:25:15.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.833+0000 7fc8d2b6b640 10 monclient: init 2024-09-17T18:25:15.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.833+0000 7fc8d2b6b640 10 monclient: _reopen_session rank -1 2024-09-17T18:25:15.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.833+0000 7fc8d2b6b640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-17T18:25:15.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.833+0000 7fc8d2b6b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc8cc0a8620 addr v1:172.21.15.173:6789/0 2024-09-17T18:25:15.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.833+0000 7fc8d2b6b640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc8cc0a6520 0x7fc8cc13dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:15.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.833+0000 7fc8d2b6b640 10 monclient(hunting): picked mon.noname-a con 0x7fc8cc0a6520 addr v2:172.21.15.121:3300/0 2024-09-17T18:25:15.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.833+0000 7fc8d2b6b640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fc8cc0a6900 0x7fc8cc140150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:15.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.833+0000 7fc8d2b6b640 10 monclient(hunting): picked mon.noname-c con 0x7fc8cc0a6900 addr v2:172.21.15.173:3300/0 2024-09-17T18:25:15.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.833+0000 7fc8d2b6b640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8cc09f0e0 con 0x7fc8cc0a8620 2024-09-17T18:25:15.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.833+0000 7fc8d2b6b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:15.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.833+0000 7fc8d2b6b640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fc8cc0a8a00 con 0x7fc8cc0a6520 2024-09-17T18:25:15.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.833+0000 7fc8d2b6b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:15.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.833+0000 7fc8d2b6b640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fc8cc09cf50 con 0x7fc8cc0a6900 2024-09-17T18:25:15.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.833+0000 7fc8d2b6b640 10 monclient(hunting): _renew_subs 2024-09-17T18:25:15.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.833+0000 7fc8d2b6b640 10 monclient(hunting): authenticate will time out at 1119.550663s 2024-09-17T18:25:15.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.833+0000 7fc8cbfff640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc8cc0a6520 0x7fc8cc13dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:15.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.833+0000 7fc8cbfff640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc8cc0a6520 0x7fc8cc13dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44628/0 (socket says 172.21.15.121:44628) 2024-09-17T18:25:15.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.833+0000 7fc8cbfff640 1 -- 172.21.15.121:0/3991161362 learned_addr learned my addr 172.21.15.121:0/3991161362 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:25:15.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.833+0000 7fc8cbfff640 10 monclient(hunting): get_auth_request con 0x7fc8cc0a6520 auth_method 0 2024-09-17T18:25:15.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.833+0000 7fc8cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:15.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.833+0000 7fc8cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:15.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.833+0000 7fc8cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:15.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.833+0000 7fc8cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:15.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.833+0000 7fc8cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:15.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.833+0000 7fc8cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:15.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.833+0000 7fc8d1108640 1 --2- 172.21.15.121:0/3991161362 >> v2:172.21.15.173:3300/0 conn(0x7fc8cc0a6900 0x7fc8cc140150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:15.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.833+0000 7fc8d1108640 1 -- 172.21.15.121:0/3991161362 >> v2:172.21.15.173:3300/0 conn(0x7fc8cc0a6900 msgr2=0x7fc8cc140150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T18:25:15.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.833+0000 7fc8d1108640 1 -- 172.21.15.121:0/3991161362 >> v2:172.21.15.173:3300/0 conn(0x7fc8cc0a6900 msgr2=0x7fc8cc140150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:25:15.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.833+0000 7fc8d1108640 1 --2- 172.21.15.121:0/3991161362 >> v2:172.21.15.173:3300/0 conn(0x7fc8cc0a6900 0x7fc8cc140150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:25:15.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.833+0000 7fc8d1108640 1 --2- 172.21.15.121:0/3991161362 >> v2:172.21.15.173:3300/0 conn(0x7fc8cc0a6900 0x7fc8cc140150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:25:15.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.834+0000 7fc8cbfff640 10 monclient(hunting): handle_auth_done global_id 44382 payload 274 2024-09-17T18:25:15.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.834+0000 7fc8cbfff640 1 -- 172.21.15.121:0/3991161362 >> v2:172.21.15.173:3300/0 conn(0x7fc8cc0a6900 msgr2=0x7fc8cc140150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:25:15.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.834+0000 7fc8cbfff640 1 --2- 172.21.15.121:0/3991161362 >> v2:172.21.15.173:3300/0 conn(0x7fc8cc0a6900 0x7fc8cc140150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:15.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.834+0000 7fc8cbfff640 1 -- 172.21.15.121:0/3991161362 >> v1:172.21.15.173:6789/0 conn(0x7fc8cc0a8620 legacy=0x7fc8cc0a5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:25:15.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.834+0000 7fc8cbfff640 10 monclient: _finish_hunting 0 2024-09-17T18:25:15.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.834+0000 7fc8cbfff640 1 monclient: found mon.noname-a 2024-09-17T18:25:15.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.834+0000 7fc8cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:25:15.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.834+0000 7fc8cbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:25:15.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.834+0000 7fc8cbfff640 1 -- 172.21.15.121:0/3991161362 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8cc070900 con 0x7fc8cc0a6520 2024-09-17T18:25:15.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.836+0000 7fc8cbfff640 1 -- 172.21.15.121:0/3991161362 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc8cc0a6520 msgr2=0x7fc8cc13dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:15.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.836+0000 7fc8cbfff640 1 --2- 172.21.15.121:0/3991161362 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc8cc0a6520 0x7fc8cc13dc00 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc09cba0 tx=0x7fc8b802a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89fa41caf20503c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:15.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.836+0000 7fc8cb7fe640 1 -- 172.21.15.121:0/3991161362 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8b8037070 con 0x7fc8cc0a6520 2024-09-17T18:25:15.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.836+0000 7fc8cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:15.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.836+0000 7fc8cb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:25:15.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.836+0000 7fc8cb7fe640 10 monclient: dump: 2024-09-17T18:25:15.849 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:15.849 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:15.849 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.849 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.849 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:15.849 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:15.849 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:15.849 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:15.849 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:15.849 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:15.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.836+0000 7fc8cb7fe640 10 monclient: _finish_auth 0 2024-09-17T18:25:15.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.836+0000 7fc8cb7fe640 10 monclient: _check_auth_tickets 2024-09-17T18:25:15.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.836+0000 7fc8cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:25:15.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.836+0000 7fc8d2b6b640 5 monclient: authenticate success, global_id 44382 2024-09-17T18:25:15.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.836+0000 7fc8cb7fe640 1 -- 172.21.15.121:0/3991161362 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8b8032640 con 0x7fc8cc0a6520 2024-09-17T18:25:15.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.836+0000 7fc8d2b6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:25:15.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.837+0000 7fc8cb7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:25:15.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.837+0000 7fc8d2b6b640 10 monclient: get_monmap_and_config success 2024-09-17T18:25:15.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.837+0000 7fc8cb7fe640 1 -- 172.21.15.121:0/3991161362 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8b8032a10 con 0x7fc8cc0a6520 2024-09-17T18:25:15.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.837+0000 7fc8d2b6b640 4 set_mon_vals no callback set 2024-09-17T18:25:15.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.837+0000 7fc8cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:15.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.837+0000 7fc8cb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:25:15.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.837+0000 7fc8cb7fe640 10 monclient: dump: 2024-09-17T18:25:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:15.851 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.837+0000 7fc8d2b6b640 10 monclient: shutdown 2024-09-17T18:25:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.837+0000 7fc8d2b6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:25:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.837+0000 7fc8d2b6b640 1 -- 172.21.15.121:0/3991161362 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc8cc0a6520 msgr2=0x7fc8cc13dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.837+0000 7fc8d2b6b640 1 --2- 172.21.15.121:0/3991161362 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc8cc0a6520 0x7fc8cc13dc00 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc09cba0 tx=0x7fc8b802a0b0 comp rx=0 tx=0).stop 2024-09-17T18:25:15.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.837+0000 7fc8d2b6b640 1 -- 172.21.15.121:0/3991161362 shutdown_connections 2024-09-17T18:25:15.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.837+0000 7fc8d2b6b640 1 --2- 172.21.15.121:0/3991161362 >> v2:172.21.15.173:3300/0 conn(0x7fc8cc0a6900 0x7fc8cc140150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:15.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.837+0000 7fc8d2b6b640 1 --2- 172.21.15.121:0/3991161362 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc8cc0a6520 0x7fc8cc13dc00 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:15.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.837+0000 7fc8d2b6b640 1 -- 172.21.15.121:0/3991161362 >> 172.21.15.121:0/3991161362 conn(0x7fc8cc0abcb0 msgr2=0x7fc8cc0a7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:25:15.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.837+0000 7fc8d2b6b640 1 -- 172.21.15.121:0/3991161362 shutdown_connections 2024-09-17T18:25:15.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.837+0000 7fc8d2b6b640 1 -- 172.21.15.121:0/3991161362 wait complete. 2024-09-17T18:25:15.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.838+0000 7fc8d2b6b640 10 monclient: build_initial_monmap 2024-09-17T18:25:15.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.838+0000 7fc8d2b6b640 10 monclient: monmap: 2024-09-17T18:25:15.852 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:25:15.852 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:25:15.852 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:25:15.852 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:25:15.852 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:25:15.853 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:15.853 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:25:15.853 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:25:15.853 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:25:15.853 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:15.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.838+0000 7fc8d2b6b640 1 librados: starting msgr at 2024-09-17T18:25:15.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.838+0000 7fc8d2b6b640 1 librados: starting objecter 2024-09-17T18:25:15.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.838+0000 7fc8d2b6b640 1 Processor -- start 2024-09-17T18:25:15.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.838+0000 7fc8d2b6b640 1 -- start start 2024-09-17T18:25:15.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.838+0000 7fc8d2b6b640 1 librados: setting wanted keys 2024-09-17T18:25:15.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.838+0000 7fc8d2b6b640 1 librados: calling monclient init 2024-09-17T18:25:15.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.838+0000 7fc8d2b6b640 10 monclient: init 2024-09-17T18:25:15.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.839+0000 7fc8d2b6b640 10 monclient: _reopen_session rank -1 2024-09-17T18:25:15.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.839+0000 7fc8d2b6b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T18:25:15.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.839+0000 7fc8d2b6b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc8cc0a6520 0x7fc8cc1cac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:15.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.839+0000 7fc8d2b6b640 10 monclient(hunting): picked mon.noname-c con 0x7fc8cc0a6520 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:25:15.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.839+0000 7fc8d2b6b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc8cc0a6900 0x7fc8cc128ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:15.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.839+0000 7fc8d2b6b640 10 monclient(hunting): picked mon.noname-b con 0x7fc8cc0a6900 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:25:15.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.839+0000 7fc8d2b6b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc8cc0a8620 0x7fc8cc13fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:15.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.839+0000 7fc8d2b6b640 10 monclient(hunting): picked mon.noname-a con 0x7fc8cc0a8620 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:25:15.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.839+0000 7fc8d2b6b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:15.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.839+0000 7fc8d2b6b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc8cc0a7490 con 0x7fc8cc0a8620 2024-09-17T18:25:15.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.839+0000 7fc8d2b6b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:15.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.839+0000 7fc8d2b6b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc8cc09d930 con 0x7fc8cc0a6900 2024-09-17T18:25:15.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.839+0000 7fc8d2b6b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:15.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.839+0000 7fc8d2b6b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc8cc0a7610 con 0x7fc8cc0a6520 2024-09-17T18:25:15.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.839+0000 7fc8d2b6b640 10 monclient(hunting): _renew_subs 2024-09-17T18:25:15.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.839+0000 7fc8d2b6b640 10 monclient(hunting): authenticate will time out at 1119.556105s 2024-09-17T18:25:15.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.839+0000 7fc8d0907640 1 -- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc8cc0a6520 msgr2=0x7fc8cc1cac00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:25:15.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.839+0000 7fc8d0907640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc8cc0a6520 0x7fc8cc1cac00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:25:15.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.839+0000 7fc8d1108640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc8cc0a8620 0x7fc8cc13fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:15.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.839+0000 7fc8d1108640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc8cc0a8620 0x7fc8cc13fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44642/0 (socket says 172.21.15.121:44642) 2024-09-17T18:25:15.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.839+0000 7fc8d1108640 1 -- 172.21.15.121:0/3290584504 learned_addr learned my addr 172.21.15.121:0/3290584504 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:25:15.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.839+0000 7fc8d1108640 10 monclient(hunting): get_auth_request con 0x7fc8cc0a8620 auth_method 0 2024-09-17T18:25:15.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.839+0000 7fc8d1108640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:15.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.839+0000 7fc8d1108640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:15.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.839+0000 7fc8d1108640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:15.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.839+0000 7fc8d1108640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:15.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.839+0000 7fc8d1108640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:15.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.839+0000 7fc8d1108640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:15.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.839+0000 7fc8d1108640 10 monclient(hunting): handle_auth_done global_id 44385 payload 824 2024-09-17T18:25:15.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.839+0000 7fc8d1108640 1 -- 172.21.15.121:0/3290584504 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc8cc0a6520 msgr2=0x7fc8cc1cac00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:25:15.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.839+0000 7fc8d1108640 1 --2- 172.21.15.121:0/3290584504 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc8cc0a6520 0x7fc8cc1cac00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:15.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.839+0000 7fc8cbfff640 1 --2- 172.21.15.121:0/3290584504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc8cc0a6900 0x7fc8cc128ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:15.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.839+0000 7fc8d1108640 1 -- 172.21.15.121:0/3290584504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc8cc0a6900 msgr2=0x7fc8cc128ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:15.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.839+0000 7fc8d1108640 1 --2- 172.21.15.121:0/3290584504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc8cc0a6900 0x7fc8cc128ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:15.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.839+0000 7fc8d1108640 10 monclient: _finish_hunting 0 2024-09-17T18:25:15.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.839+0000 7fc8d1108640 1 monclient: found mon.noname-a 2024-09-17T18:25:15.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.839+0000 7fc8d1108640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:25:15.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.839+0000 7fc8d1108640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:25:15.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.839+0000 7fc8d1108640 1 -- 172.21.15.121:0/3290584504 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8cc0a2600 con 0x7fc8cc0a8620 2024-09-17T18:25:15.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.841+0000 7fc8d1108640 1 -- 172.21.15.121:0/3290584504 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc8cc0a8620 msgr2=0x7fc8cc13fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:15.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.841+0000 7fc8d1108640 1 --2- 172.21.15.121:0/3290584504 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc8cc0a8620 0x7fc8cc13fc50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc8c400dc20 tx=0x7fc8c400b5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:15.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.841+0000 7fc8c97fa640 10 client.?.objecter ms_handle_connect 0x7fc8cc0a8620 2024-09-17T18:25:15.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.841+0000 7fc8c97fa640 10 client.?.objecter resend_mon_ops 2024-09-17T18:25:15.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.841+0000 7fc8d1108640 1 --2- 172.21.15.121:0/3290584504 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc8cc0a8620 0x7fc8cc13fc50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc8c400dc20 tx=0x7fc8c400b5d0 comp rx=0 tx=0).queue_received will delay after 1.7266e+09 on 0x7fc8c4009040 mon_map magic: 0 2024-09-17T18:25:15.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.841+0000 7fc8c97fa640 1 -- 172.21.15.121:0/3290584504 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c4009040 con 0x7fc8cc0a8620 2024-09-17T18:25:15.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.841+0000 7fc8c97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:15.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.841+0000 7fc8c97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:25:15.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.841+0000 7fc8c97fa640 10 monclient: dump: 2024-09-17T18:25:15.857 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:15.857 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:15.857 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.857 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.857 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:15.857 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:15.857 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:15.858 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:15.858 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:15.858 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:15.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.841+0000 7fc8c97fa640 10 monclient: _finish_auth 0 2024-09-17T18:25:15.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.841+0000 7fc8c97fa640 10 monclient: _check_auth_tickets 2024-09-17T18:25:15.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.841+0000 7fc8c97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:25:15.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.841+0000 7fc8c97fa640 1 -- 172.21.15.121:0/3290584504 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8c4003b90 con 0x7fc8cc0a8620 2024-09-17T18:25:15.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.841+0000 7fc8d2b6b640 5 monclient: authenticate success, global_id 44385 2024-09-17T18:25:15.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.841+0000 7fc8d2b6b640 10 monclient: _renew_subs 2024-09-17T18:25:15.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.841+0000 7fc8d2b6b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:15.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.841+0000 7fc8d2b6b640 1 -- 172.21.15.121:0/3290584504 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8cc127d40 con 0x7fc8cc0a8620 2024-09-17T18:25:15.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.841+0000 7fc8c97fa640 10 monclient: handle_config config(0 keys) 2024-09-17T18:25:15.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.841+0000 7fc8caffd640 4 set_mon_vals no callback set 2024-09-17T18:25:15.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.841+0000 7fc8d2b6b640 10 client.44385.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:25:15.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.841+0000 7fc8d2b6b640 10 monclient: _renew_subs 2024-09-17T18:25:15.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.841+0000 7fc8d2b6b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:15.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.841+0000 7fc8d2b6b640 1 -- 172.21.15.121:0/3290584504 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8cc129e40 con 0x7fc8cc0a8620 2024-09-17T18:25:15.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.841+0000 7fc8d2b6b640 1 librados: init done 2024-09-17T18:25:15.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.842+0000 7fc8d2b6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:15.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.842+0000 7fc8d2b6b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:15.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.842+0000 7fc8d2b6b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:15.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.842+0000 7fc8c97fa640 1 -- 172.21.15.121:0/3290584504 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c400de30 con 0x7fc8cc0a8620 2024-09-17T18:25:15.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.842+0000 7fc8d2b6b640 1 -- 172.21.15.121:0/3290584504 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8cc0a2a10 con 0x7fc8cc0a8620 2024-09-17T18:25:15.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.842+0000 7fc8c97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:15.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.842+0000 7fc8c97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:25:15.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.842+0000 7fc8c97fa640 10 monclient: dump: 2024-09-17T18:25:15.859 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:15.860 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:15.860 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.860 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:15.860 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:15.860 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:15.860 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:15.860 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:15.860 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:15.860 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:15.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.843+0000 7fc8c97fa640 1 -- 172.21.15.121:0/3290584504 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fc8c4003fc0 con 0x7fc8cc0a8620 2024-09-17T18:25:15.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.843+0000 7fc8c97fa640 1 --2- 172.21.15.121:0/3290584504 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc8b0040120 0x7fc8b00425e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:15.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.843+0000 7fc8d0907640 1 --2- 172.21.15.121:0/3290584504 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc8b0040120 0x7fc8b00425e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:15.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.843+0000 7fc8d0907640 10 monclient: get_auth_request con 0x7fc8b0040120 auth_method 0 2024-09-17T18:25:15.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.845+0000 7fc8c97fa640 10 client.44385.objecter ms_handle_connect 0x7fc8b0040120 2024-09-17T18:25:15.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.845+0000 7fc8d0907640 1 --2- 172.21.15.121:0/3290584504 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc8b0040120 0x7fc8b00425e0 secure :-1 s=READY pgs=3981 cs=0 l=1 rev1=1 crypto rx=0x7fc8c0003770 tx=0x7fc8c0007a10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:15.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.847+0000 7fc8c97fa640 1 -- 172.21.15.121:0/3290584504 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(287..287 src has 1..287) ==== 5814+0+0 (secure 0 0 0) 0x7fc8c4053eb0 con 0x7fc8cc0a8620 2024-09-17T18:25:15.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.847+0000 7fc8c97fa640 10 client.44385.objecter ms_dispatch 0x7fc8cc000b90 osd_map(287..287 src has 1..287) 2024-09-17T18:25:15.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.847+0000 7fc8c97fa640 3 client.44385.objecter handle_osd_map got epochs [287,287] > 0 2024-09-17T18:25:15.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.847+0000 7fc8c97fa640 3 client.44385.objecter handle_osd_map decoding full epoch 287 2024-09-17T18:25:15.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:15.848+0000 7fc8c97fa640 20 client.44385.objecter dump_active .. 0 homeless 2024-09-17T18:25:16.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:16.465+0000 7fc8c97fa640 1 -- 172.21.15.121:0/3290584504 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc8cc0a2a10 con 0x7fc8cc0a8620 2024-09-17T18:25:16.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:16.465+0000 7fc8c97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:16.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:16.465+0000 7fc8c97fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:25:16.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:16.708+0000 7fc8d2b6b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:25:16.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:16.708+0000 7fc8d2b6b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:25:16.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:16.708+0000 7fc8d2b6b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:16.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:16.708+0000 7fc8d2b6b640 1 -- 172.21.15.121:0/3290584504 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc8cc0a4c90 con 0x7fc8cc0a8620 2024-09-17T18:25:16.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:16.708+0000 7fc8c97fa640 1 -- 172.21.15.121:0/3290584504 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fc8c4015030 con 0x7fc8cc0a8620 2024-09-17T18:25:16.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:16.708+0000 7fc8c97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:25:16.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:16.708+0000 7fc8c97fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:25:16.711 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:25:16.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:16.710+0000 7fc8d2b6b640 10 librados: watch_flush enter 2024-09-17T18:25:16.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:16.710+0000 7fc8d2b6b640 10 librados: watch_flush exit 2024-09-17T18:25:16.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:16.710+0000 7fc8d2b6b640 20 client.44385.objecter shutdown clearing up homeless session... 2024-09-17T18:25:16.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:16.710+0000 7fc8d2b6b640 10 client.44385.objecter successfully canceled tick 2024-09-17T18:25:16.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:16.710+0000 7fc8d2b6b640 1 -- 172.21.15.121:0/3290584504 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc8b0040120 msgr2=0x7fc8b00425e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:16.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:16.710+0000 7fc8d2b6b640 1 --2- 172.21.15.121:0/3290584504 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc8b0040120 0x7fc8b00425e0 secure :-1 s=READY pgs=3981 cs=0 l=1 rev1=1 crypto rx=0x7fc8c0003770 tx=0x7fc8c0007a10 comp rx=0 tx=0).stop 2024-09-17T18:25:16.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:16.710+0000 7fc8d2b6b640 10 monclient: shutdown 2024-09-17T18:25:16.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:16.710+0000 7fc8d2b6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:25:16.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:16.710+0000 7fc8d2b6b640 1 -- 172.21.15.121:0/3290584504 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc8cc0a8620 msgr2=0x7fc8cc13fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:16.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:16.710+0000 7fc8d2b6b640 1 --2- 172.21.15.121:0/3290584504 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc8cc0a8620 0x7fc8cc13fc50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc8c400dc20 tx=0x7fc8c400b5d0 comp rx=0 tx=0).stop 2024-09-17T18:25:16.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:16.710+0000 7fc8d2b6b640 1 -- 172.21.15.121:0/3290584504 shutdown_connections 2024-09-17T18:25:16.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:16.710+0000 7fc8d2b6b640 1 --2- 172.21.15.121:0/3290584504 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc8b0040120 0x7fc8b00425e0 unknown :-1 s=CLOSED pgs=3981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:16.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:16.710+0000 7fc8d2b6b640 1 --2- 172.21.15.121:0/3290584504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc8cc0a6900 0x7fc8cc128ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:16.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:16.710+0000 7fc8d2b6b640 1 --2- 172.21.15.121:0/3290584504 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc8cc0a8620 0x7fc8cc13fc50 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:16.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:16.710+0000 7fc8d2b6b640 1 --2- 172.21.15.121:0/3290584504 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc8cc0a6520 0x7fc8cc1cac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:16.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:16.710+0000 7fc8d2b6b640 1 -- 172.21.15.121:0/3290584504 >> 172.21.15.121:0/3290584504 conn(0x7fc8cc0abcb0 msgr2=0x7fc8cc0a1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:25:16.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:16.711+0000 7fc8d2b6b640 1 -- 172.21.15.121:0/3290584504 shutdown_connections 2024-09-17T18:25:16.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:16.711+0000 7fc8d2b6b640 1 -- 172.21.15.121:0/3290584504 wait complete. 2024-09-17T18:25:16.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:16.711+0000 7fc8d2b6b640 1 librados: shutdown 2024-09-17T18:25:19.725 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:25:19.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.830+0000 7f597bfe8640 10 monclient: get_monmap_and_config 2024-09-17T18:25:19.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.830+0000 7f597bfe8640 10 monclient: build_initial_monmap 2024-09-17T18:25:19.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.830+0000 7f597bfe8640 10 monclient: monmap: 2024-09-17T18:25:19.833 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:25:19.833 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:19.833 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:25:19.831739+0000 2024-09-17T18:25:19.833 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:25:19.831739+0000 2024-09-17T18:25:19.833 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:25:19.833 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:19.833 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:25:19.833 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:25:19.834 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:25:19.834 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:25:19.834 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:25:19.834 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:19.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.830+0000 7f597bfe8640 1 Processor -- start 2024-09-17T18:25:19.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.831+0000 7f597bfe8640 1 -- start start 2024-09-17T18:25:19.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.831+0000 7f597bfe8640 10 monclient: init 2024-09-17T18:25:19.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.831+0000 7f597bfe8640 10 monclient: _reopen_session rank -1 2024-09-17T18:25:19.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.831+0000 7f597bfe8640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-17T18:25:19.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.831+0000 7f597bfe8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f59740a5c00 addr v1:172.21.15.121:6789/0 2024-09-17T18:25:19.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.831+0000 7f597bfe8640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f59740a5fe0 0x7f59740a63f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:19.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.831+0000 7f597bfe8640 10 monclient(hunting): picked mon.noname-a con 0x7f59740a5fe0 addr v2:172.21.15.121:3300/0 2024-09-17T18:25:19.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.831+0000 7f597bfe8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f597413dc00 addr v1:172.21.15.173:6789/0 2024-09-17T18:25:19.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.831+0000 7f597bfe8640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59740a8a80 con 0x7f59740a5c00 2024-09-17T18:25:19.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.831+0000 7f597bfe8640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59740a74d0 con 0x7f597413dc00 2024-09-17T18:25:19.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.831+0000 7f597bfe8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:19.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.831+0000 7f597bfe8640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f597409cb90 con 0x7f59740a5fe0 2024-09-17T18:25:19.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.831+0000 7f597bfe8640 10 monclient(hunting): _renew_subs 2024-09-17T18:25:19.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.831+0000 7f597bfe8640 10 monclient(hunting): authenticate will time out at 1123.548593s 2024-09-17T18:25:19.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.831+0000 7f5979583640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f59740a5fe0 0x7f59740a63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:19.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.831+0000 7f5979d84640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f59740a5c00 0x7f597409d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:59972/0 (socket says 172.21.15.121:59972) 2024-09-17T18:25:19.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.831+0000 7f5979583640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f59740a5fe0 0x7f59740a63f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40280/0 (socket says 172.21.15.121:40280) 2024-09-17T18:25:19.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.831+0000 7f5979d84640 1 -- 172.21.15.121:0/2869069515 learned_addr learned my addr 172.21.15.121:0/2869069515 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:25:19.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.831+0000 7f5979583640 10 monclient(hunting): get_auth_request con 0x7f59740a5fe0 auth_method 0 2024-09-17T18:25:19.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.831+0000 7f5979583640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:19.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.831+0000 7f5979583640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:19.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.831+0000 7f5979583640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:19.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.831+0000 7f5979583640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:19.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.831+0000 7f5979583640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:19.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.831+0000 7f5979583640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:19.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.831+0000 7f5979583640 10 monclient(hunting): handle_auth_done global_id 44388 payload 274 2024-09-17T18:25:19.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.831+0000 7f5979583640 1 -- 172.21.15.121:0/2869069515 >> v1:172.21.15.173:6789/0 conn(0x7f597413dc00 legacy=0x7f597413fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:25:19.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.831+0000 7f5979583640 1 -- 172.21.15.121:0/2869069515 >> v1:172.21.15.121:6789/0 conn(0x7f59740a5c00 legacy=0x7f597409d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:25:19.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.831+0000 7f5979583640 10 monclient: _finish_hunting 0 2024-09-17T18:25:19.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.831+0000 7f5979583640 1 monclient: found mon.noname-a 2024-09-17T18:25:19.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.831+0000 7f5979583640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:25:19.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.831+0000 7f5979583640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:25:19.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.831+0000 7f5979583640 1 -- 172.21.15.121:0/2869069515 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f597409ce70 con 0x7f59740a5fe0 2024-09-17T18:25:19.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.834+0000 7f5979583640 1 -- 172.21.15.121:0/2869069515 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59740a5fe0 msgr2=0x7f59740a63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:19.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.834+0000 7f5979583640 1 --2- 172.21.15.121:0/2869069515 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59740a5fe0 0x7f59740a63f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f59680097a0 tx=0x7f5968029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ba5a9cf43c8a27e server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:19.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.835+0000 7f5978d82640 1 -- 172.21.15.121:0/2869069515 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5968037070 con 0x7f59740a5fe0 2024-09-17T18:25:19.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.835+0000 7f5978d82640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:19.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.835+0000 7f5978d82640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:25:19.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.835+0000 7f5978d82640 10 monclient: dump: 2024-09-17T18:25:19.844 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:19.844 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:19.844 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:19.844 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:19.844 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:19.844 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:19.844 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:19.844 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:19.844 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:19.844 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:19.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.835+0000 7f5978d82640 10 monclient: _finish_auth 0 2024-09-17T18:25:19.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.835+0000 7f5978d82640 10 monclient: _check_auth_tickets 2024-09-17T18:25:19.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.835+0000 7f5978d82640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:25:19.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.835+0000 7f5978d82640 1 -- 172.21.15.121:0/2869069515 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5968032640 con 0x7f59740a5fe0 2024-09-17T18:25:19.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.835+0000 7f597bfe8640 5 monclient: authenticate success, global_id 44388 2024-09-17T18:25:19.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.835+0000 7f597bfe8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:25:19.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.835+0000 7f5978d82640 10 monclient: handle_config config(0 keys) 2024-09-17T18:25:19.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.835+0000 7f597bfe8640 10 monclient: get_monmap_and_config success 2024-09-17T18:25:19.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.835+0000 7f5978d82640 1 -- 172.21.15.121:0/2869069515 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5968032a10 con 0x7f59740a5fe0 2024-09-17T18:25:19.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.835+0000 7f5978d82640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:19.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.835+0000 7f597bfe8640 4 set_mon_vals no callback set 2024-09-17T18:25:19.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.835+0000 7f5978d82640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:25:19.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.835+0000 7f5978d82640 10 monclient: dump: 2024-09-17T18:25:19.845 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:19.845 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:19.845 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:19.845 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:19.846 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:19.846 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:19.846 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:19.846 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:19.846 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:19.846 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:19.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.835+0000 7f597bfe8640 10 monclient: shutdown 2024-09-17T18:25:19.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.835+0000 7f597bfe8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:25:19.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.835+0000 7f597bfe8640 1 -- 172.21.15.121:0/2869069515 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59740a5fe0 msgr2=0x7f59740a63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:19.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.835+0000 7f597bfe8640 1 --2- 172.21.15.121:0/2869069515 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59740a5fe0 0x7f59740a63f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f59680097a0 tx=0x7f5968029fd0 comp rx=0 tx=0).stop 2024-09-17T18:25:19.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.835+0000 7f597bfe8640 1 -- 172.21.15.121:0/2869069515 shutdown_connections 2024-09-17T18:25:19.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.835+0000 7f597bfe8640 1 --2- 172.21.15.121:0/2869069515 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59740a5fe0 0x7f59740a63f0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:19.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.835+0000 7f597bfe8640 1 -- 172.21.15.121:0/2869069515 >> 172.21.15.121:0/2869069515 conn(0x7f59740ab9b0 msgr2=0x7f59740a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:25:19.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.835+0000 7f597bfe8640 1 -- 172.21.15.121:0/2869069515 shutdown_connections 2024-09-17T18:25:19.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.835+0000 7f597bfe8640 1 -- 172.21.15.121:0/2869069515 wait complete. 2024-09-17T18:25:19.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.836+0000 7f597bfe8640 10 monclient: build_initial_monmap 2024-09-17T18:25:19.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.836+0000 7f597bfe8640 10 monclient: monmap: 2024-09-17T18:25:19.847 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:25:19.847 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:25:19.847 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:25:19.847 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:25:19.847 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:25:19.847 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:19.847 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:25:19.847 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:25:19.847 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:25:19.847 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:19.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.836+0000 7f597bfe8640 1 librados: starting msgr at 2024-09-17T18:25:19.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.836+0000 7f597bfe8640 1 librados: starting objecter 2024-09-17T18:25:19.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.836+0000 7f597bfe8640 1 Processor -- start 2024-09-17T18:25:19.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.836+0000 7f597bfe8640 1 -- start start 2024-09-17T18:25:19.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.836+0000 7f597bfe8640 1 librados: setting wanted keys 2024-09-17T18:25:19.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.836+0000 7f597bfe8640 1 librados: calling monclient init 2024-09-17T18:25:19.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.836+0000 7f597bfe8640 10 monclient: init 2024-09-17T18:25:19.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.837+0000 7f597bfe8640 10 monclient: _reopen_session rank -1 2024-09-17T18:25:19.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.837+0000 7f597bfe8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T18:25:19.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.837+0000 7f597bfe8640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f59740a5c00 0x7f597413cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:19.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.837+0000 7f597bfe8640 10 monclient(hunting): picked mon.noname-c con 0x7f59740a5c00 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:25:19.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.837+0000 7f597bfe8640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59740a5fe0 0x7f59741cde40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:19.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.837+0000 7f597bfe8640 10 monclient(hunting): picked mon.noname-a con 0x7f59740a5fe0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:25:19.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.837+0000 7f597bfe8640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f597413dc00 0x7f597413a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:19.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.837+0000 7f597bfe8640 10 monclient(hunting): picked mon.noname-b con 0x7f597413dc00 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:25:19.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.837+0000 7f597bfe8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:19.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.837+0000 7f597bfe8640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5974070550 con 0x7f59740a5fe0 2024-09-17T18:25:19.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.837+0000 7f597bfe8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:19.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.837+0000 7f597bfe8640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f59740703d0 con 0x7f597413dc00 2024-09-17T18:25:19.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.837+0000 7f597bfe8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:19.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.837+0000 7f597bfe8640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5974141710 con 0x7f59740a5c00 2024-09-17T18:25:19.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.837+0000 7f597bfe8640 10 monclient(hunting): _renew_subs 2024-09-17T18:25:19.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.837+0000 7f5979d84640 1 -- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f59740a5c00 msgr2=0x7f597413cdd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:25:19.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.837+0000 7f597bfe8640 10 monclient(hunting): authenticate will time out at 1123.554240s 2024-09-17T18:25:19.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.837+0000 7f5979d84640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f59740a5c00 0x7f597413cdd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:25:19.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.837+0000 7f5979583640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59740a5fe0 0x7f59741cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:19.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.837+0000 7f5979583640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59740a5fe0 0x7f59741cde40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40292/0 (socket says 172.21.15.121:40292) 2024-09-17T18:25:19.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.837+0000 7f5979583640 1 -- 172.21.15.121:0/4230251104 learned_addr learned my addr 172.21.15.121:0/4230251104 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:25:19.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.837+0000 7f5979583640 10 monclient(hunting): get_auth_request con 0x7f59740a5fe0 auth_method 0 2024-09-17T18:25:19.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.837+0000 7f5979583640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:19.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.837+0000 7f5979583640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:19.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.837+0000 7f5979583640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:19.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.837+0000 7f5979583640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:19.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.837+0000 7f5979583640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:19.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.837+0000 7f5979583640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:19.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.837+0000 7f5979583640 10 monclient(hunting): handle_auth_done global_id 44391 payload 824 2024-09-17T18:25:19.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.837+0000 7f5979583640 1 -- 172.21.15.121:0/4230251104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f59740a5c00 msgr2=0x7f597413cdd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:25:19.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.837+0000 7f5979583640 1 --2- 172.21.15.121:0/4230251104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f59740a5c00 0x7f597413cdd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:19.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.837+0000 7f5979583640 1 -- 172.21.15.121:0/4230251104 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f597413dc00 msgr2=0x7f597413a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:19.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.837+0000 7f597a585640 1 --2- 172.21.15.121:0/4230251104 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f597413dc00 0x7f597413a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:19.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.837+0000 7f5979583640 1 --2- 172.21.15.121:0/4230251104 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f597413dc00 0x7f597413a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:19.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.837+0000 7f5979583640 10 monclient: _finish_hunting 0 2024-09-17T18:25:19.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.837+0000 7f5979583640 1 monclient: found mon.noname-a 2024-09-17T18:25:19.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.837+0000 7f5979583640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:25:19.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.837+0000 7f5979583640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:25:19.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.837+0000 7f5979583640 1 -- 172.21.15.121:0/4230251104 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f597413d760 con 0x7f59740a5fe0 2024-09-17T18:25:19.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.837+0000 7f597a585640 10 monclient: get_auth_request con 0x7f597413dc00 auth_method 0 2024-09-17T18:25:19.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.837+0000 7f597a585640 1 --2- 172.21.15.121:0/4230251104 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f597413dc00 0x7f597413a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T18:25:19.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.839+0000 7f5979583640 1 -- 172.21.15.121:0/4230251104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59740a5fe0 msgr2=0x7f59741cde40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:19.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.839+0000 7f5979583640 1 --2- 172.21.15.121:0/4230251104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59740a5fe0 0x7f59741cde40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f59680097a0 tx=0x7f59680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:19.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.839+0000 7f59727fc640 10 client.?.objecter ms_handle_connect 0x7f59740a5fe0 2024-09-17T18:25:19.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.839+0000 7f59727fc640 10 client.?.objecter resend_mon_ops 2024-09-17T18:25:19.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.839+0000 7f59727fc640 1 -- 172.21.15.121:0/4230251104 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5968041070 con 0x7f59740a5fe0 2024-09-17T18:25:19.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.839+0000 7f59727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:19.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.839+0000 7f59727fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:25:19.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.839+0000 7f59727fc640 10 monclient: dump: 2024-09-17T18:25:19.855 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:19.855 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:19.855 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:19.855 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:19.856 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:19.856 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:19.856 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:19.856 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:19.856 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:19.856 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:19.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.839+0000 7f59727fc640 10 monclient: _finish_auth 0 2024-09-17T18:25:19.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.839+0000 7f59727fc640 10 monclient: _check_auth_tickets 2024-09-17T18:25:19.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.839+0000 7f59727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:25:19.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.840+0000 7f597bfe8640 5 monclient: authenticate success, global_id 44391 2024-09-17T18:25:19.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.840+0000 7f59727fc640 1 -- 172.21.15.121:0/4230251104 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f596803ca80 con 0x7f59740a5fe0 2024-09-17T18:25:19.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.840+0000 7f597bfe8640 10 monclient: _renew_subs 2024-09-17T18:25:19.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.840+0000 7f597bfe8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:19.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.840+0000 7f597bfe8640 1 -- 172.21.15.121:0/4230251104 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5974121980 con 0x7f59740a5fe0 2024-09-17T18:25:19.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.840+0000 7f59727fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:25:19.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.840+0000 7f59727fc640 1 -- 172.21.15.121:0/4230251104 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5968037040 con 0x7f59740a5fe0 2024-09-17T18:25:19.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.840+0000 7f59727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:19.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.840+0000 7f59727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:25:19.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.840+0000 7f59727fc640 10 monclient: dump: 2024-09-17T18:25:19.859 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:19.859 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:19.859 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:19.859 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:19.859 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:19.860 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:19.860 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:19.860 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:19.860 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:19.860 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:19.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.840+0000 7f597bfe8640 10 client.44391.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:25:19.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.840+0000 7f5973fff640 4 set_mon_vals no callback set 2024-09-17T18:25:19.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.840+0000 7f597bfe8640 10 monclient: _renew_subs 2024-09-17T18:25:19.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.840+0000 7f597bfe8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:19.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.840+0000 7f597bfe8640 1 -- 172.21.15.121:0/4230251104 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5974138ed0 con 0x7f59740a5fe0 2024-09-17T18:25:19.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.840+0000 7f597bfe8640 1 librados: init done 2024-09-17T18:25:19.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.841+0000 7f597bfe8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:19.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.841+0000 7f597bfe8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:19.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.841+0000 7f597bfe8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:19.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.841+0000 7f597bfe8640 1 -- 172.21.15.121:0/4230251104 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59740a63f0 con 0x7f59740a5fe0 2024-09-17T18:25:19.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.841+0000 7f59727fc640 1 -- 172.21.15.121:0/4230251104 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f5968044070 con 0x7f59740a5fe0 2024-09-17T18:25:19.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.841+0000 7f59727fc640 1 --2- 172.21.15.121:0/4230251104 >> v2:172.21.15.173:6812/2423786037 conn(0x7f5960040120 0x7f59600425e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:19.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.841+0000 7f59727fc640 1 -- 172.21.15.121:0/4230251104 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(287..287 src has 1..287) ==== 5814+0+0 (secure 0 0 0) 0x7f59680743b0 con 0x7f59740a5fe0 2024-09-17T18:25:19.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.841+0000 7f59727fc640 10 client.44391.objecter ms_dispatch 0x7f5974000b90 osd_map(287..287 src has 1..287) 2024-09-17T18:25:19.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.841+0000 7f59727fc640 3 client.44391.objecter handle_osd_map got epochs [287,287] > 0 2024-09-17T18:25:19.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.841+0000 7f59727fc640 3 client.44391.objecter handle_osd_map decoding full epoch 287 2024-09-17T18:25:19.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.841+0000 7f59727fc640 20 client.44391.objecter dump_active .. 0 homeless 2024-09-17T18:25:19.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.841+0000 7f5979d84640 1 --2- 172.21.15.121:0/4230251104 >> v2:172.21.15.173:6812/2423786037 conn(0x7f5960040120 0x7f59600425e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:19.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.841+0000 7f5979d84640 10 monclient: get_auth_request con 0x7f5960040120 auth_method 0 2024-09-17T18:25:19.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.844+0000 7f5979d84640 1 --2- 172.21.15.121:0/4230251104 >> v2:172.21.15.173:6812/2423786037 conn(0x7f5960040120 0x7f59600425e0 secure :-1 s=READY pgs=3982 cs=0 l=1 rev1=1 crypto rx=0x7f596c007fd0 tx=0x7f596c007b20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:19.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.844+0000 7f59727fc640 10 client.44391.objecter ms_handle_connect 0x7f5960040120 2024-09-17T18:25:19.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.845+0000 7f59727fc640 1 -- 172.21.15.121:0/4230251104 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f596803ce70 con 0x7f59740a5fe0 2024-09-17T18:25:19.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.845+0000 7f59727fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:19.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:19.846+0000 7f59727fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:25:20.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:20.087+0000 7f597bfe8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:25:20.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:20.087+0000 7f597bfe8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:25:20.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:20.087+0000 7f597bfe8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:20.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:20.087+0000 7f597bfe8640 1 -- 172.21.15.121:0/4230251104 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f597406cd40 con 0x7f59740a5fe0 2024-09-17T18:25:20.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:20.087+0000 7f59727fc640 1 -- 172.21.15.121:0/4230251104 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f5968032520 con 0x7f59740a5fe0 2024-09-17T18:25:20.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:20.087+0000 7f59727fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:25:20.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:20.087+0000 7f59727fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:25:20.091 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:25:20.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:20.089+0000 7f597bfe8640 10 librados: watch_flush enter 2024-09-17T18:25:20.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:20.089+0000 7f597bfe8640 10 librados: watch_flush exit 2024-09-17T18:25:20.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:20.089+0000 7f597bfe8640 20 client.44391.objecter shutdown clearing up homeless session... 2024-09-17T18:25:20.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:20.089+0000 7f597bfe8640 10 client.44391.objecter successfully canceled tick 2024-09-17T18:25:20.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:20.089+0000 7f597bfe8640 1 -- 172.21.15.121:0/4230251104 >> v2:172.21.15.173:6812/2423786037 conn(0x7f5960040120 msgr2=0x7f59600425e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:20.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:20.089+0000 7f597bfe8640 1 --2- 172.21.15.121:0/4230251104 >> v2:172.21.15.173:6812/2423786037 conn(0x7f5960040120 0x7f59600425e0 secure :-1 s=READY pgs=3982 cs=0 l=1 rev1=1 crypto rx=0x7f596c007fd0 tx=0x7f596c007b20 comp rx=0 tx=0).stop 2024-09-17T18:25:20.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:20.089+0000 7f597bfe8640 10 monclient: shutdown 2024-09-17T18:25:20.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:20.089+0000 7f597bfe8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:25:20.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:20.089+0000 7f597bfe8640 1 -- 172.21.15.121:0/4230251104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59740a5fe0 msgr2=0x7f59741cde40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:20.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:20.089+0000 7f597bfe8640 1 --2- 172.21.15.121:0/4230251104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59740a5fe0 0x7f59741cde40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f59680097a0 tx=0x7f59680046c0 comp rx=0 tx=0).stop 2024-09-17T18:25:20.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:20.089+0000 7f597bfe8640 1 -- 172.21.15.121:0/4230251104 shutdown_connections 2024-09-17T18:25:20.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:20.089+0000 7f597bfe8640 1 --2- 172.21.15.121:0/4230251104 >> v2:172.21.15.173:6812/2423786037 conn(0x7f5960040120 0x7f59600425e0 unknown :-1 s=CLOSED pgs=3982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:20.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:20.089+0000 7f597bfe8640 1 --2- 172.21.15.121:0/4230251104 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f597413dc00 0x7f597413a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:20.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:20.089+0000 7f597bfe8640 1 --2- 172.21.15.121:0/4230251104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59740a5fe0 0x7f59741cde40 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:20.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:20.089+0000 7f597bfe8640 1 --2- 172.21.15.121:0/4230251104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f59740a5c00 0x7f597413cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:20.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:20.089+0000 7f597bfe8640 1 -- 172.21.15.121:0/4230251104 >> 172.21.15.121:0/4230251104 conn(0x7f59740ab9b0 msgr2=0x7f59740a4660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:25:20.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:20.089+0000 7f597bfe8640 1 -- 172.21.15.121:0/4230251104 shutdown_connections 2024-09-17T18:25:20.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:20.089+0000 7f597bfe8640 1 -- 172.21.15.121:0/4230251104 wait complete. 2024-09-17T18:25:20.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:20.089+0000 7f597bfe8640 1 librados: shutdown 2024-09-17T18:25:23.103 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:25:23.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.208+0000 7fabe650e640 10 monclient: get_monmap_and_config 2024-09-17T18:25:23.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.208+0000 7fabe650e640 10 monclient: build_initial_monmap 2024-09-17T18:25:23.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.209+0000 7fabe650e640 10 monclient: monmap: 2024-09-17T18:25:23.212 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:25:23.212 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:23.212 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:25:23.210594+0000 2024-09-17T18:25:23.212 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:25:23.210594+0000 2024-09-17T18:25:23.212 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:25:23.212 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:23.212 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:25:23.212 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:25:23.212 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:25:23.212 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:25:23.213 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:25:23.213 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:23.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.209+0000 7fabe650e640 1 Processor -- start 2024-09-17T18:25:23.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.209+0000 7fabe650e640 1 -- start start 2024-09-17T18:25:23.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.209+0000 7fabe650e640 10 monclient: init 2024-09-17T18:25:23.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.210+0000 7fabe650e640 10 monclient: _reopen_session rank -1 2024-09-17T18:25:23.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.210+0000 7fabe650e640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-17T18:25:23.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.210+0000 7fabe650e640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fabe00a33f0 0x7fabe009e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:23.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.210+0000 7fabe650e640 10 monclient(hunting): picked mon.noname-c con 0x7fabe00a33f0 addr v2:172.21.15.173:3300/0 2024-09-17T18:25:23.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.210+0000 7fabe650e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fabe009da70 addr v1:172.21.15.173:6789/0 2024-09-17T18:25:23.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.210+0000 7fabe650e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fabe013a850 addr v1:172.21.15.121:6789/0 2024-09-17T18:25:23.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.210+0000 7fabe650e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabe0074b40 con 0x7fabe013a850 2024-09-17T18:25:23.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.210+0000 7fabe650e640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabe00a13c0 con 0x7fabe009da70 2024-09-17T18:25:23.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.210+0000 7fabe650e640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:23.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.210+0000 7fabe650e640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fabe0074550 con 0x7fabe00a33f0 2024-09-17T18:25:23.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.210+0000 7fabe650e640 10 monclient(hunting): _renew_subs 2024-09-17T18:25:23.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.210+0000 7fabe650e640 10 monclient(hunting): authenticate will time out at 1126.927419s 2024-09-17T18:25:23.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.210+0000 7fabe4aab640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fabe013a850 0x7fabe013cc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:59986/0 (socket says 172.21.15.121:59986) 2024-09-17T18:25:23.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.210+0000 7fabe4aab640 1 -- 172.21.15.121:0/1757082782 learned_addr learned my addr 172.21.15.121:0/1757082782 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:25:23.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.210+0000 7fabdffff640 1 --2- 172.21.15.121:0/1757082782 >> v2:172.21.15.173:3300/0 conn(0x7fabe00a33f0 0x7fabe009e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:23.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.210+0000 7fabdffff640 10 monclient(hunting): get_auth_request con 0x7fabe00a33f0 auth_method 0 2024-09-17T18:25:23.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.210+0000 7fabdffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:23.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.210+0000 7fabdffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:23.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.210+0000 7fabdffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:23.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.210+0000 7fabdffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:23.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.210+0000 7fabdffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:23.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.210+0000 7fabdffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:23.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.210+0000 7fabdffff640 10 monclient(hunting): handle_auth_done global_id 24724 payload 274 2024-09-17T18:25:23.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.210+0000 7fabdffff640 1 -- 172.21.15.121:0/1757082782 >> v1:172.21.15.173:6789/0 conn(0x7fabe009da70 legacy=0x7fabe009ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:25:23.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.210+0000 7fabdffff640 1 -- 172.21.15.121:0/1757082782 >> v1:172.21.15.121:6789/0 conn(0x7fabe013a850 legacy=0x7fabe013cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:25:23.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.210+0000 7fabdffff640 10 monclient: _finish_hunting 0 2024-09-17T18:25:23.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.210+0000 7fabdffff640 1 monclient: found mon.noname-c 2024-09-17T18:25:23.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.210+0000 7fabdffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:25:23.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.210+0000 7fabdffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.173:3300/0 2024-09-17T18:25:23.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.210+0000 7fabdffff640 1 -- 172.21.15.121:0/1757082782 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabe0077730 con 0x7fabe00a33f0 2024-09-17T18:25:23.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.213+0000 7fabdffff640 1 -- 172.21.15.121:0/1757082782 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fabe00a33f0 msgr2=0x7fabe009e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:23.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.213+0000 7fabdffff640 1 --2- 172.21.15.121:0/1757082782 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fabe00a33f0 0x7fabe009e550 secure :-1 s=READY pgs=7395 cs=0 l=1 rev1=1 crypto rx=0x7fabc80097a0 tx=0x7fabc8029fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=908f7b5cb59c5e10 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:23.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.214+0000 7fabdeffd640 1 -- 172.21.15.121:0/1757082782 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc8037070 con 0x7fabe00a33f0 2024-09-17T18:25:23.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.214+0000 7fabdeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:23.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.214+0000 7fabdeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:25:23.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.214+0000 7fabdeffd640 10 monclient: dump: 2024-09-17T18:25:23.224 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:23.224 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:23.224 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:23.224 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:23.224 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:23.224 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:23.225 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:23.225 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:23.225 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:23.225 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:23.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.214+0000 7fabdeffd640 10 monclient: _finish_auth 0 2024-09-17T18:25:23.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.214+0000 7fabdeffd640 10 monclient: _check_auth_tickets 2024-09-17T18:25:23.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.214+0000 7fabdeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:25:23.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.214+0000 7fabdeffd640 1 -- 172.21.15.121:0/1757082782 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabc8032640 con 0x7fabe00a33f0 2024-09-17T18:25:23.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.214+0000 7fabe650e640 5 monclient: authenticate success, global_id 24724 2024-09-17T18:25:23.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.214+0000 7fabe650e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:25:23.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.214+0000 7fabdeffd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:25:23.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.214+0000 7fabdeffd640 1 -- 172.21.15.121:0/1757082782 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc8032a10 con 0x7fabe00a33f0 2024-09-17T18:25:23.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.214+0000 7fabe650e640 10 monclient: get_monmap_and_config success 2024-09-17T18:25:23.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.214+0000 7fabe650e640 4 set_mon_vals no callback set 2024-09-17T18:25:23.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.214+0000 7fabdeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:23.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.214+0000 7fabdeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:25:23.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.214+0000 7fabdeffd640 10 monclient: dump: 2024-09-17T18:25:23.226 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:23.226 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:23.226 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:23.226 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:23.226 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:23.226 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:23.226 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:23.226 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:23.226 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:23.226 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:23.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.214+0000 7fabe650e640 10 monclient: shutdown 2024-09-17T18:25:23.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.214+0000 7fabe650e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:25:23.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.214+0000 7fabe650e640 1 -- 172.21.15.121:0/1757082782 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fabe00a33f0 msgr2=0x7fabe009e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:23.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.214+0000 7fabe650e640 1 --2- 172.21.15.121:0/1757082782 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fabe00a33f0 0x7fabe009e550 secure :-1 s=READY pgs=7395 cs=0 l=1 rev1=1 crypto rx=0x7fabc80097a0 tx=0x7fabc8029fd0 comp rx=0 tx=0).stop 2024-09-17T18:25:23.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.214+0000 7fabe650e640 1 -- 172.21.15.121:0/1757082782 shutdown_connections 2024-09-17T18:25:23.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.214+0000 7fabe650e640 1 --2- 172.21.15.121:0/1757082782 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fabe00a33f0 0x7fabe009e550 unknown :-1 s=CLOSED pgs=7395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:23.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.214+0000 7fabe650e640 1 -- 172.21.15.121:0/1757082782 >> 172.21.15.121:0/1757082782 conn(0x7fabe00a3b50 msgr2=0x7fabe00a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:25:23.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.214+0000 7fabe650e640 1 -- 172.21.15.121:0/1757082782 shutdown_connections 2024-09-17T18:25:23.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.214+0000 7fabe650e640 1 -- 172.21.15.121:0/1757082782 wait complete. 2024-09-17T18:25:23.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.215+0000 7fabe650e640 10 monclient: build_initial_monmap 2024-09-17T18:25:23.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.215+0000 7fabe650e640 10 monclient: monmap: 2024-09-17T18:25:23.227 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:25:23.227 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:25:23.227 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:25:23.227 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:25:23.227 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:25:23.228 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:23.228 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:25:23.228 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:25:23.228 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:25:23.228 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:23.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.215+0000 7fabe650e640 1 librados: starting msgr at 2024-09-17T18:25:23.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.215+0000 7fabe650e640 1 librados: starting objecter 2024-09-17T18:25:23.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.215+0000 7fabe650e640 1 Processor -- start 2024-09-17T18:25:23.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.215+0000 7fabe650e640 1 -- start start 2024-09-17T18:25:23.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.215+0000 7fabe650e640 1 librados: setting wanted keys 2024-09-17T18:25:23.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.215+0000 7fabe650e640 1 librados: calling monclient init 2024-09-17T18:25:23.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.215+0000 7fabe650e640 10 monclient: init 2024-09-17T18:25:23.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabe650e640 10 monclient: _reopen_session rank -1 2024-09-17T18:25:23.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabe650e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T18:25:23.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabe650e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fabe00a33f0 0x7fabe01cd860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:23.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabe650e640 10 monclient(hunting): picked mon.noname-a con 0x7fabe00a33f0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:25:23.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabe650e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fabe013a850 0x7fabe0136cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:23.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabe650e640 10 monclient(hunting): picked mon.noname-c con 0x7fabe013a850 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:25:23.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabe650e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fabe0121e80 0x7fabe0137210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:23.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabe650e640 10 monclient(hunting): picked mon.noname-b con 0x7fabe0121e80 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:25:23.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabe650e640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:23.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabe650e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fabe009f320 con 0x7fabe00a33f0 2024-09-17T18:25:23.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabe650e640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:23.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabe650e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fabe009f1a0 con 0x7fabe0121e80 2024-09-17T18:25:23.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabe650e640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:23.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabe650e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fabe013e360 con 0x7fabe013a850 2024-09-17T18:25:23.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabe650e640 10 monclient(hunting): _renew_subs 2024-09-17T18:25:23.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabe650e640 10 monclient(hunting): authenticate will time out at 1126.933396s 2024-09-17T18:25:23.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabdf7fe640 1 -- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fabe013a850 msgr2=0x7fabe0136cd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:25:23.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabdf7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fabe013a850 0x7fabe0136cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:25:23.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabdffff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fabe00a33f0 0x7fabe01cd860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:23.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabdffff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fabe00a33f0 0x7fabe01cd860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40308/0 (socket says 172.21.15.121:40308) 2024-09-17T18:25:23.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabdffff640 1 -- 172.21.15.121:0/2850179114 learned_addr learned my addr 172.21.15.121:0/2850179114 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:25:23.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabdffff640 10 monclient(hunting): get_auth_request con 0x7fabe00a33f0 auth_method 0 2024-09-17T18:25:23.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabdffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:23.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabdffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:23.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabdffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:23.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabe4aab640 1 --2- 172.21.15.121:0/2850179114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fabe0121e80 0x7fabe0137210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:23.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabdffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:23.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabdffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:23.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabdffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:23.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabe4aab640 10 monclient(hunting): get_auth_request con 0x7fabe0121e80 auth_method 0 2024-09-17T18:25:23.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabe4aab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:23.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabe4aab640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:23.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabe4aab640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:23.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabe4aab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:23.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabe4aab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:23.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabe4aab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:23.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabdffff640 10 monclient(hunting): handle_auth_done global_id 44394 payload 824 2024-09-17T18:25:23.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabdffff640 1 -- 172.21.15.121:0/2850179114 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fabe013a850 msgr2=0x7fabe0136cd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:25:23.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabdffff640 1 --2- 172.21.15.121:0/2850179114 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fabe013a850 0x7fabe0136cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:23.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabdffff640 1 -- 172.21.15.121:0/2850179114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fabe0121e80 msgr2=0x7fabe0137210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:23.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabdffff640 1 --2- 172.21.15.121:0/2850179114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fabe0121e80 0x7fabe0137210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:23.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabdffff640 10 monclient: _finish_hunting 0 2024-09-17T18:25:23.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabdffff640 1 monclient: found mon.noname-a 2024-09-17T18:25:23.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabdffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:25:23.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabdffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:25:23.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.216+0000 7fabdffff640 1 -- 172.21.15.121:0/2850179114 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabe01cdda0 con 0x7fabe00a33f0 2024-09-17T18:25:23.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.219+0000 7fabdffff640 1 -- 172.21.15.121:0/2850179114 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fabe00a33f0 msgr2=0x7fabe01cd860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:23.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.219+0000 7fabdffff640 1 --2- 172.21.15.121:0/2850179114 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fabe00a33f0 0x7fabe01cd860 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fabc80097a0 tx=0x7fabc80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:23.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.219+0000 7fabdcff9640 10 client.?.objecter ms_handle_connect 0x7fabe00a33f0 2024-09-17T18:25:23.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.219+0000 7fabdcff9640 10 client.?.objecter resend_mon_ops 2024-09-17T18:25:23.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.219+0000 7fabdcff9640 1 -- 172.21.15.121:0/2850179114 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc8041070 con 0x7fabe00a33f0 2024-09-17T18:25:23.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.219+0000 7fabdcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:23.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.219+0000 7fabdcff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:25:23.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.219+0000 7fabdcff9640 10 monclient: dump: 2024-09-17T18:25:23.232 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:23.233 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:23.233 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:23.233 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:23.233 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:23.233 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:23.233 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:23.233 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:23.233 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:23.233 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:23.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.219+0000 7fabdcff9640 10 monclient: _finish_auth 0 2024-09-17T18:25:23.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.219+0000 7fabdcff9640 10 monclient: _check_auth_tickets 2024-09-17T18:25:23.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.219+0000 7fabdcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:25:23.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.219+0000 7fabdcff9640 1 -- 172.21.15.121:0/2850179114 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabc803c880 con 0x7fabe00a33f0 2024-09-17T18:25:23.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.219+0000 7fabe650e640 5 monclient: authenticate success, global_id 44394 2024-09-17T18:25:23.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.219+0000 7fabe650e640 10 monclient: _renew_subs 2024-09-17T18:25:23.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.219+0000 7fabe650e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:23.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.219+0000 7fabe650e640 1 -- 172.21.15.121:0/2850179114 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabe009e240 con 0x7fabe00a33f0 2024-09-17T18:25:23.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.219+0000 7fabdcff9640 10 monclient: handle_config config(0 keys) 2024-09-17T18:25:23.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.219+0000 7fabdcff9640 1 -- 172.21.15.121:0/2850179114 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc8037040 con 0x7fabe00a33f0 2024-09-17T18:25:23.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.219+0000 7fabdcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:23.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.219+0000 7fabdcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:25:23.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.219+0000 7fabdcff9640 10 monclient: dump: 2024-09-17T18:25:23.234 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:23.234 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:23.234 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:23.234 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:23.234 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:23.234 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:23.234 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:23.234 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:23.235 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:23.235 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:23.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.219+0000 7fabe650e640 10 client.44394.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:25:23.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.219+0000 7fabe650e640 10 monclient: _renew_subs 2024-09-17T18:25:23.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.219+0000 7fabe650e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:23.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.219+0000 7fabde7fc640 4 set_mon_vals no callback set 2024-09-17T18:25:23.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.219+0000 7fabe650e640 1 -- 172.21.15.121:0/2850179114 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabe0123680 con 0x7fabe00a33f0 2024-09-17T18:25:23.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.219+0000 7fabe650e640 1 librados: init done 2024-09-17T18:25:23.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.220+0000 7fabe650e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:23.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.220+0000 7fabe650e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:23.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.220+0000 7fabe650e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:23.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.220+0000 7fabe650e640 1 -- 172.21.15.121:0/2850179114 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabe009e550 con 0x7fabe00a33f0 2024-09-17T18:25:23.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.220+0000 7fabdcff9640 1 -- 172.21.15.121:0/2850179114 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fabc8044070 con 0x7fabe00a33f0 2024-09-17T18:25:23.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.220+0000 7fabdcff9640 1 --2- 172.21.15.121:0/2850179114 >> v2:172.21.15.173:6812/2423786037 conn(0x7fabc0040150 0x7fabc0042610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:23.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.220+0000 7fabdcff9640 1 -- 172.21.15.121:0/2850179114 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(287..287 src has 1..287) ==== 5814+0+0 (secure 0 0 0) 0x7fabc8073ba0 con 0x7fabe00a33f0 2024-09-17T18:25:23.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.220+0000 7fabdcff9640 10 client.44394.objecter ms_dispatch 0x7fabe0000b90 osd_map(287..287 src has 1..287) 2024-09-17T18:25:23.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.220+0000 7fabdcff9640 3 client.44394.objecter handle_osd_map got epochs [287,287] > 0 2024-09-17T18:25:23.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.220+0000 7fabdcff9640 3 client.44394.objecter handle_osd_map decoding full epoch 287 2024-09-17T18:25:23.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.220+0000 7fabdcff9640 20 client.44394.objecter dump_active .. 0 homeless 2024-09-17T18:25:23.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.220+0000 7fabdf7fe640 1 --2- 172.21.15.121:0/2850179114 >> v2:172.21.15.173:6812/2423786037 conn(0x7fabc0040150 0x7fabc0042610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:23.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.220+0000 7fabdf7fe640 10 monclient: get_auth_request con 0x7fabc0040150 auth_method 0 2024-09-17T18:25:23.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.223+0000 7fabdf7fe640 1 --2- 172.21.15.121:0/2850179114 >> v2:172.21.15.173:6812/2423786037 conn(0x7fabc0040150 0x7fabc0042610 secure :-1 s=READY pgs=3983 cs=0 l=1 rev1=1 crypto rx=0x7fabe00a67e0 tx=0x7fabd0007a10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:23.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.223+0000 7fabdcff9640 10 client.44394.objecter ms_handle_connect 0x7fabc0040150 2024-09-17T18:25:23.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.225+0000 7fabdcff9640 1 -- 172.21.15.121:0/2850179114 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fabc804b020 con 0x7fabe00a33f0 2024-09-17T18:25:23.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.225+0000 7fabdcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:23.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.225+0000 7fabdcff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:25:23.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.464+0000 7fabe650e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:25:23.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.464+0000 7fabe650e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:25:23.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.464+0000 7fabe650e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:23.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.464+0000 7fabe650e640 1 -- 172.21.15.121:0/2850179114 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fabe009da70 con 0x7fabe00a33f0 2024-09-17T18:25:23.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.464+0000 7fabdcff9640 1 -- 172.21.15.121:0/2850179114 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7fabc803a360 con 0x7fabe00a33f0 2024-09-17T18:25:23.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.464+0000 7fabdcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:25:23.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.464+0000 7fabdcff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:25:23.468 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:25:23.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.466+0000 7fabe650e640 10 librados: watch_flush enter 2024-09-17T18:25:23.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.466+0000 7fabe650e640 10 librados: watch_flush exit 2024-09-17T18:25:23.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.466+0000 7fabe650e640 20 client.44394.objecter shutdown clearing up homeless session... 2024-09-17T18:25:23.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.466+0000 7fabe650e640 10 client.44394.objecter successfully canceled tick 2024-09-17T18:25:23.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.466+0000 7fabe650e640 1 -- 172.21.15.121:0/2850179114 >> v2:172.21.15.173:6812/2423786037 conn(0x7fabc0040150 msgr2=0x7fabc0042610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:23.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.466+0000 7fabe650e640 1 --2- 172.21.15.121:0/2850179114 >> v2:172.21.15.173:6812/2423786037 conn(0x7fabc0040150 0x7fabc0042610 secure :-1 s=READY pgs=3983 cs=0 l=1 rev1=1 crypto rx=0x7fabe00a67e0 tx=0x7fabd0007a10 comp rx=0 tx=0).stop 2024-09-17T18:25:23.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.466+0000 7fabe650e640 10 monclient: shutdown 2024-09-17T18:25:23.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.466+0000 7fabe650e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:25:23.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.466+0000 7fabe650e640 1 -- 172.21.15.121:0/2850179114 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fabe00a33f0 msgr2=0x7fabe01cd860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:23.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.466+0000 7fabe650e640 1 --2- 172.21.15.121:0/2850179114 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fabe00a33f0 0x7fabe01cd860 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fabc80097a0 tx=0x7fabc80046c0 comp rx=0 tx=0).stop 2024-09-17T18:25:23.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.466+0000 7fabe650e640 1 -- 172.21.15.121:0/2850179114 shutdown_connections 2024-09-17T18:25:23.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.466+0000 7fabe650e640 1 --2- 172.21.15.121:0/2850179114 >> v2:172.21.15.173:6812/2423786037 conn(0x7fabc0040150 0x7fabc0042610 unknown :-1 s=CLOSED pgs=3983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:23.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.466+0000 7fabe650e640 1 --2- 172.21.15.121:0/2850179114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fabe0121e80 0x7fabe0137210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:23.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.466+0000 7fabe650e640 1 --2- 172.21.15.121:0/2850179114 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fabe013a850 0x7fabe0136cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:23.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.466+0000 7fabe650e640 1 --2- 172.21.15.121:0/2850179114 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fabe00a33f0 0x7fabe01cd860 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:23.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.466+0000 7fabe650e640 1 -- 172.21.15.121:0/2850179114 >> 172.21.15.121:0/2850179114 conn(0x7fabe00a3b50 msgr2=0x7fabe00a5c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:25:23.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.466+0000 7fabe650e640 1 -- 172.21.15.121:0/2850179114 shutdown_connections 2024-09-17T18:25:23.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.466+0000 7fabe650e640 1 -- 172.21.15.121:0/2850179114 wait complete. 2024-09-17T18:25:23.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:23.466+0000 7fabe650e640 1 librados: shutdown 2024-09-17T18:25:26.481 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:25:26.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.586+0000 7f380c1c6640 10 monclient: get_monmap_and_config 2024-09-17T18:25:26.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.586+0000 7f380c1c6640 10 monclient: build_initial_monmap 2024-09-17T18:25:26.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.586+0000 7f380c1c6640 10 monclient: monmap: 2024-09-17T18:25:26.589 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:25:26.589 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:26.589 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:25:26.588052+0000 2024-09-17T18:25:26.589 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:25:26.588052+0000 2024-09-17T18:25:26.589 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:25:26.589 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:26.589 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:25:26.590 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:25:26.590 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:25:26.590 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:25:26.590 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:25:26.590 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:26.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.587+0000 7f380c1c6640 1 Processor -- start 2024-09-17T18:25:26.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.587+0000 7f380c1c6640 1 -- start start 2024-09-17T18:25:26.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.587+0000 7f380c1c6640 10 monclient: init 2024-09-17T18:25:26.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.587+0000 7f380c1c6640 10 monclient: _reopen_session rank -1 2024-09-17T18:25:26.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.587+0000 7f380c1c6640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-17T18:25:26.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.587+0000 7f380c1c6640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f38040a0a70 0x7f38041348f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:26.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.587+0000 7f380c1c6640 10 monclient(hunting): picked mon.noname-c con 0x7f38040a0a70 addr v2:172.21.15.173:3300/0 2024-09-17T18:25:26.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.587+0000 7f380c1c6640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f38040a0e50 0x7f38040a22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:26.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.587+0000 7f380c1c6640 10 monclient(hunting): picked mon.noname-a con 0x7f38040a0e50 addr v2:172.21.15.121:3300/0 2024-09-17T18:25:26.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.587+0000 7f380c1c6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f38040a1260 addr v1:172.21.15.173:6789/0 2024-09-17T18:25:26.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.587+0000 7f380c1c6640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f380409f5b0 con 0x7f38040a1260 2024-09-17T18:25:26.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.587+0000 7f380c1c6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:26.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.587+0000 7f380c1c6640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f38040a48b0 con 0x7f38040a0e50 2024-09-17T18:25:26.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.587+0000 7f380c1c6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:26.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.587+0000 7f380c1c6640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f380409ef80 con 0x7f38040a0a70 2024-09-17T18:25:26.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.587+0000 7f380c1c6640 10 monclient(hunting): _renew_subs 2024-09-17T18:25:26.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.587+0000 7f380c1c6640 10 monclient(hunting): authenticate will time out at 1130.304789s 2024-09-17T18:25:26.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.587+0000 7f3809761640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f38040a0e50 0x7f38040a22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:26.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.587+0000 7f3809761640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f38040a0e50 0x7f38040a22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40322/0 (socket says 172.21.15.121:40322) 2024-09-17T18:25:26.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.587+0000 7f3809761640 1 -- 172.21.15.121:0/2573695186 learned_addr learned my addr 172.21.15.121:0/2573695186 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:25:26.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.587+0000 7f3809761640 10 monclient(hunting): get_auth_request con 0x7f38040a0e50 auth_method 0 2024-09-17T18:25:26.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.587+0000 7f3809761640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:26.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.588+0000 7f3809761640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:26.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.588+0000 7f3809761640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:26.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.588+0000 7f3809761640 1 -- 172.21.15.121:0/2573695186 >> v2:172.21.15.121:3300/0 conn(0x7f38040a0e50 msgr2=0x7f38040a22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T18:25:26.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.588+0000 7f3809761640 1 -- 172.21.15.121:0/2573695186 >> v2:172.21.15.121:3300/0 conn(0x7f38040a0e50 msgr2=0x7f38040a22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:25:26.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.588+0000 7f3809761640 1 --2- 172.21.15.121:0/2573695186 >> v2:172.21.15.121:3300/0 conn(0x7f38040a0e50 0x7f38040a22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:25:26.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.588+0000 7f3809761640 1 --2- 172.21.15.121:0/2573695186 >> v2:172.21.15.121:3300/0 conn(0x7f38040a0e50 0x7f38040a22e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:25:26.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.588+0000 7f3809f62640 1 --2- 172.21.15.121:0/2573695186 >> v2:172.21.15.173:3300/0 conn(0x7f38040a0a70 0x7f38041348f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:26.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.588+0000 7f3809f62640 10 monclient(hunting): get_auth_request con 0x7f38040a0a70 auth_method 0 2024-09-17T18:25:26.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.588+0000 7f3809f62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:26.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.588+0000 7f3809f62640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:26.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.588+0000 7f3809f62640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:26.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.588+0000 7f3809f62640 1 -- 172.21.15.121:0/2573695186 >> v2:172.21.15.173:3300/0 conn(0x7f38040a0a70 msgr2=0x7f38041348f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:25:26.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.588+0000 7f3809f62640 1 -- 172.21.15.121:0/2573695186 >> v2:172.21.15.173:3300/0 conn(0x7f38040a0a70 msgr2=0x7f38041348f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:25:26.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.588+0000 7f3809f62640 1 --2- 172.21.15.121:0/2573695186 >> v2:172.21.15.173:3300/0 conn(0x7f38040a0a70 0x7f38041348f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:25:26.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.588+0000 7f3809f62640 1 --2- 172.21.15.121:0/2573695186 >> v2:172.21.15.173:3300/0 conn(0x7f38040a0a70 0x7f38041348f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:25:26.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.590+0000 7f380a763640 1 -- 172.21.15.121:0/2573695186 >> v1:172.21.15.173:6789/0 conn(0x7f38040a1260 legacy=0x7f38040a2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:26.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.789+0000 7f3809761640 1 --2- 172.21.15.121:0/2573695186 >> v2:172.21.15.121:3300/0 conn(0x7f38040a0e50 0x7f38040a22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:26.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.789+0000 7f3809761640 10 monclient(hunting): get_auth_request con 0x7f38040a0e50 auth_method 0 2024-09-17T18:25:26.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.789+0000 7f3809761640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:26.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.789+0000 7f3809761640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:26.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.789+0000 7f3809761640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:25:26.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.789+0000 7f3809761640 1 -- 172.21.15.121:0/2573695186 >> v2:172.21.15.121:3300/0 conn(0x7f38040a0e50 msgr2=0x7f38040a22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:25:26.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.789+0000 7f3809761640 1 -- 172.21.15.121:0/2573695186 >> v2:172.21.15.121:3300/0 conn(0x7f38040a0e50 msgr2=0x7f38040a22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:25:26.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.789+0000 7f3809761640 1 --2- 172.21.15.121:0/2573695186 >> v2:172.21.15.121:3300/0 conn(0x7f38040a0e50 0x7f38040a22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:25:26.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.789+0000 7f3809761640 1 --2- 172.21.15.121:0/2573695186 >> v2:172.21.15.121:3300/0 conn(0x7f38040a0e50 0x7f38040a22e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T18:25:26.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.789+0000 7f3809f62640 1 --2- 172.21.15.121:0/2573695186 >> v2:172.21.15.173:3300/0 conn(0x7f38040a0a70 0x7f38041348f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:26.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.789+0000 7f3809f62640 10 monclient(hunting): get_auth_request con 0x7f38040a0a70 auth_method 0 2024-09-17T18:25:26.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.789+0000 7f3809f62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:26.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.789+0000 7f3809f62640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:26.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.789+0000 7f3809f62640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:25:26.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.789+0000 7f3809f62640 1 -- 172.21.15.121:0/2573695186 >> v2:172.21.15.173:3300/0 conn(0x7f38040a0a70 msgr2=0x7f38041348f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T18:25:26.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.789+0000 7f3809f62640 1 -- 172.21.15.121:0/2573695186 >> v2:172.21.15.173:3300/0 conn(0x7f38040a0a70 msgr2=0x7f38041348f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:25:26.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.789+0000 7f3809f62640 1 --2- 172.21.15.121:0/2573695186 >> v2:172.21.15.173:3300/0 conn(0x7f38040a0a70 0x7f38041348f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:25:26.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:26.789+0000 7f3809f62640 1 --2- 172.21.15.121:0/2573695186 >> v2:172.21.15.173:3300/0 conn(0x7f38040a0a70 0x7f38041348f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T18:25:27.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:27.190+0000 7f3809761640 1 --2- 172.21.15.121:0/2573695186 >> v2:172.21.15.121:3300/0 conn(0x7f38040a0e50 0x7f38040a22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:27.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:27.190+0000 7f3809761640 10 monclient(hunting): get_auth_request con 0x7f38040a0e50 auth_method 0 2024-09-17T18:25:27.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:27.190+0000 7f3809761640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:27.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:27.190+0000 7f3809761640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:27.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:27.190+0000 7f3809761640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:25:27.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:27.190+0000 7f3809761640 1 -- 172.21.15.121:0/2573695186 >> v2:172.21.15.121:3300/0 conn(0x7f38040a0e50 msgr2=0x7f38040a22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:25:27.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:27.190+0000 7f3809761640 1 -- 172.21.15.121:0/2573695186 >> v2:172.21.15.121:3300/0 conn(0x7f38040a0e50 msgr2=0x7f38040a22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:25:27.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:27.190+0000 7f3809761640 1 --2- 172.21.15.121:0/2573695186 >> v2:172.21.15.121:3300/0 conn(0x7f38040a0e50 0x7f38040a22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:25:27.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:27.190+0000 7f3809761640 1 --2- 172.21.15.121:0/2573695186 >> v2:172.21.15.121:3300/0 conn(0x7f38040a0e50 0x7f38040a22e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T18:25:27.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:27.190+0000 7f3809f62640 1 --2- 172.21.15.121:0/2573695186 >> v2:172.21.15.173:3300/0 conn(0x7f38040a0a70 0x7f38041348f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:27.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:27.190+0000 7f3809f62640 10 monclient(hunting): get_auth_request con 0x7f38040a0a70 auth_method 0 2024-09-17T18:25:27.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:27.190+0000 7f3809f62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:27.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:27.190+0000 7f3809f62640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:27.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:27.190+0000 7f3809f62640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:25:27.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:27.190+0000 7f3809f62640 1 -- 172.21.15.121:0/2573695186 >> v2:172.21.15.173:3300/0 conn(0x7f38040a0a70 msgr2=0x7f38041348f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T18:25:27.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:27.190+0000 7f3809f62640 1 -- 172.21.15.121:0/2573695186 >> v2:172.21.15.173:3300/0 conn(0x7f38040a0a70 msgr2=0x7f38041348f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:25:27.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:27.190+0000 7f3809f62640 1 --2- 172.21.15.121:0/2573695186 >> v2:172.21.15.173:3300/0 conn(0x7f38040a0a70 0x7f38041348f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:25:27.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:27.190+0000 7f3809f62640 1 --2- 172.21.15.121:0/2573695186 >> v2:172.21.15.173:3300/0 conn(0x7f38040a0a70 0x7f38041348f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T18:25:27.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:27.991+0000 7f3809761640 1 --2- 172.21.15.121:0/2573695186 >> v2:172.21.15.121:3300/0 conn(0x7f38040a0e50 0x7f38040a22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:27.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:27.991+0000 7f3809761640 10 monclient(hunting): get_auth_request con 0x7f38040a0e50 auth_method 0 2024-09-17T18:25:27.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:27.991+0000 7f3809761640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:27.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:27.991+0000 7f3809761640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:27.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:27.991+0000 7f3809761640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:25:27.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:27.991+0000 7f3809761640 1 -- 172.21.15.121:0/2573695186 >> v2:172.21.15.121:3300/0 conn(0x7f38040a0e50 msgr2=0x7f38040a22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:25:27.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:27.991+0000 7f3809761640 1 -- 172.21.15.121:0/2573695186 >> v2:172.21.15.121:3300/0 conn(0x7f38040a0e50 msgr2=0x7f38040a22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:25:27.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:27.991+0000 7f3809761640 1 --2- 172.21.15.121:0/2573695186 >> v2:172.21.15.121:3300/0 conn(0x7f38040a0e50 0x7f38040a22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:25:27.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:27.991+0000 7f3809761640 1 --2- 172.21.15.121:0/2573695186 >> v2:172.21.15.121:3300/0 conn(0x7f38040a0e50 0x7f38040a22e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T18:25:27.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:27.991+0000 7f3809f62640 1 --2- 172.21.15.121:0/2573695186 >> v2:172.21.15.173:3300/0 conn(0x7f38040a0a70 0x7f38041348f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:27.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:27.991+0000 7f3809f62640 10 monclient(hunting): get_auth_request con 0x7f38040a0a70 auth_method 0 2024-09-17T18:25:27.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:27.991+0000 7f3809f62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:27.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:27.991+0000 7f3809f62640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:27.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:27.991+0000 7f3809f62640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:25:27.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:27.991+0000 7f3809f62640 1 -- 172.21.15.121:0/2573695186 >> v2:172.21.15.173:3300/0 conn(0x7f38040a0a70 msgr2=0x7f38041348f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:25:27.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:27.991+0000 7f3809f62640 1 -- 172.21.15.121:0/2573695186 >> v2:172.21.15.173:3300/0 conn(0x7f38040a0a70 msgr2=0x7f38041348f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:25:27.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:27.991+0000 7f3809f62640 1 --2- 172.21.15.121:0/2573695186 >> v2:172.21.15.173:3300/0 conn(0x7f38040a0a70 0x7f38041348f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:25:27.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:27.991+0000 7f3809f62640 1 --2- 172.21.15.121:0/2573695186 >> v2:172.21.15.173:3300/0 conn(0x7f38040a0a70 0x7f38041348f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T18:25:29.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:29.588+0000 7f37f77fe640 10 monclient(hunting): tick 2024-09-17T18:25:29.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:29.588+0000 7f37f77fe640 10 monclient(hunting): _check_auth_tickets 2024-09-17T18:25:29.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:29.588+0000 7f37f77fe640 1 monclient(hunting): continuing hunt 2024-09-17T18:25:29.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:29.588+0000 7f37f77fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T18:25:29.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:29.588+0000 7f37f77fe640 1 -- 172.21.15.121:0/2573695186 >> v2:172.21.15.173:3300/0 conn(0x7f38040a0a70 msgr2=0x7f38041348f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:25:29.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:29.588+0000 7f37f77fe640 1 --2- 172.21.15.121:0/2573695186 >> v2:172.21.15.173:3300/0 conn(0x7f38040a0a70 0x7f38041348f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:29.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:29.588+0000 7f37f77fe640 1 -- 172.21.15.121:0/2573695186 >> v2:172.21.15.121:3300/0 conn(0x7f38040a0e50 msgr2=0x7f38040a22e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:25:29.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:29.588+0000 7f37f77fe640 1 --2- 172.21.15.121:0/2573695186 >> v2:172.21.15.121:3300/0 conn(0x7f38040a0e50 0x7f38040a22e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:29.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:29.588+0000 7f37f77fe640 1 -- 172.21.15.121:0/2573695186 >> v1:172.21.15.173:6789/0 conn(0x7f38040a1260 legacy=0x7f38040a2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:29.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:29.588+0000 7f37f77fe640 10 monclient: _add_conns ranks=[0,2] 2024-09-17T18:25:29.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:29.588+0000 7f37f77fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f37e8003720 addr v1:172.21.15.121:6789/0 2024-09-17T18:25:29.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:29.588+0000 7f37f77fe640 1 --2- 172.21.15.121:0/2573695186 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37e8007410 0x7f37e80098b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:29.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:29.588+0000 7f37f77fe640 10 monclient(hunting): picked mon.noname-b con 0x7f37e8007410 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:25:29.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:29.588+0000 7f37f77fe640 1 -- 172.21.15.121:0/2573695186 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f380409eca0 con 0x7f37e8003720 2024-09-17T18:25:29.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:29.588+0000 7f37f77fe640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:29.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:29.588+0000 7f37f77fe640 1 -- 172.21.15.121:0/2573695186 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f38040a48b0 con 0x7f37e8007410 2024-09-17T18:25:29.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:29.588+0000 7f37f77fe640 10 monclient(hunting): _renew_subs 2024-09-17T18:25:29.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:29.588+0000 7f3809761640 1 -- 172.21.15.121:0/2573695186 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37e8007410 msgr2=0x7f37e80098b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:25:29.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:29.588+0000 7f3809761640 1 --2- 172.21.15.121:0/2573695186 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37e8007410 0x7f37e80098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:25:29.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:29.590+0000 7f3809f62640 1 -- 172.21.15.121:0/2573695186 >> v1:172.21.15.121:6789/0 conn(0x7f37e8003720 legacy=0x7f37e8005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:29.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:29.789+0000 7f3809761640 1 -- 172.21.15.121:0/2573695186 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37e8007410 msgr2=0x7f37e80098b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:25:29.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:29.789+0000 7f3809761640 1 --2- 172.21.15.121:0/2573695186 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37e8007410 0x7f37e80098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T18:25:30.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.190+0000 7f3809761640 1 -- 172.21.15.121:0/2573695186 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37e8007410 msgr2=0x7f37e80098b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:25:30.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.190+0000 7f3809761640 1 --2- 172.21.15.121:0/2573695186 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37e8007410 0x7f37e80098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T18:25:30.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.570+0000 7f3808f60640 1 -- 172.21.15.121:0/2573695186 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4185470980 0 0) 0x7f380409eca0 con 0x7f37e8003720 2024-09-17T18:25:30.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.570+0000 7f3808f60640 10 monclient(hunting): _negotiate 2024-09-17T18:25:30.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.570+0000 7f3808f60640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:30.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.570+0000 7f3808f60640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:30.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.570+0000 7f3808f60640 10 monclient(hunting): my global_id is 44397 2024-09-17T18:25:30.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.570+0000 7f3808f60640 1 -- 172.21.15.121:0/2573695186 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37f0003520 con 0x7f37e8003720 2024-09-17T18:25:30.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f3808f60640 1 -- 172.21.15.121:0/2573695186 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4231382392 0 0) 0x7f37f0003520 con 0x7f37e8003720 2024-09-17T18:25:30.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f3808f60640 1 -- 172.21.15.121:0/2573695186 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37e8007410 msgr2=0x7f37e80098b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:25:30.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f3808f60640 1 --2- 172.21.15.121:0/2573695186 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37e8007410 0x7f37e80098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:30.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f3808f60640 10 monclient: _finish_hunting 0 2024-09-17T18:25:30.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f3808f60640 1 monclient: found mon.noname-a-legacy 2024-09-17T18:25:30.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f3808f60640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:25:30.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f3808f60640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.121:6789/0 2024-09-17T18:25:30.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f3808f60640 1 -- 172.21.15.121:0/2573695186 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37e800b0f0 con 0x7f37e8003720 2024-09-17T18:25:30.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f3808f60640 10 monclient: _finish_auth 0 2024-09-17T18:25:30.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f3808f60640 10 monclient: _check_auth_tickets 2024-09-17T18:25:30.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f3808f60640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:25:30.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f3808f60640 1 -- 172.21.15.121:0/2573695186 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4002019445 0 0) 0x7f37f800d680 con 0x7f37e8003720 2024-09-17T18:25:30.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f380c1c6640 5 monclient: authenticate success, global_id 44397 2024-09-17T18:25:30.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f380c1c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:25:30.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f3808f60640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:30.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f3808f60640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T18:25:30.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f3808f60640 10 monclient: dump: 2024-09-17T18:25:30.577 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:30.577 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:30.577 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:30.578 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:30.578 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:30.578 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:30.578 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:30.578 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:30.578 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:30.578 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:30.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f3808f60640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] but i'm connected to v1:172.21.15.121:6789/0, reconnecting 2024-09-17T18:25:30.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f3808f60640 10 monclient: _reopen_session rank -1 2024-09-17T18:25:30.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f3808f60640 1 -- 172.21.15.121:0/2573695186 >> v1:172.21.15.121:6789/0 conn(0x7f37e8003720 legacy=0x7f37e8005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:30.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f3808f60640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T18:25:30.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f380a763640 1 -- 172.21.15.121:0/2573695186 reap_dead start 2024-09-17T18:25:30.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f3808f60640 1 --2- 172.21.15.121:0/2573695186 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37e8007410 0x7f37f00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:30.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f3808f60640 10 monclient(hunting): picked mon.c con 0x7f37e8007410 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:25:30.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f3808f60640 1 --2- 172.21.15.121:0/2573695186 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37f00040c0 0x7f37f0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:30.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f3808f60640 10 monclient(hunting): picked mon.b con 0x7f37f00040c0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:25:30.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f3808f60640 1 --2- 172.21.15.121:0/2573695186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37f0004e60 0x7f37f000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:30.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f3808f60640 10 monclient(hunting): picked mon.a con 0x7f37f0004e60 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:25:30.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f3808f60640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:30.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f3808f60640 1 -- 172.21.15.121:0/2573695186 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f38040a48b0 con 0x7f37f0004e60 2024-09-17T18:25:30.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f3808f60640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:30.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f3808f60640 1 -- 172.21.15.121:0/2573695186 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f37f0005650 con 0x7f37f00040c0 2024-09-17T18:25:30.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f3808f60640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:30.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f3808f60640 1 -- 172.21.15.121:0/2573695186 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f37f000bf50 con 0x7f37e8007410 2024-09-17T18:25:30.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f3808f60640 10 monclient(hunting): _renew_subs 2024-09-17T18:25:30.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f3808f60640 10 monclient(hunting): _finish_auth 0 2024-09-17T18:25:30.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f3809f62640 1 -- 172.21.15.121:0/2573695186 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37e8007410 msgr2=0x7f37f00039b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:25:30.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f3809f62640 1 --2- 172.21.15.121:0/2573695186 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37e8007410 0x7f37f00039b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:25:30.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f380c1c6640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-17T18:25:30.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f380a763640 1 --2- 172.21.15.121:0/2573695186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37f0004e60 0x7f37f000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:30.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f380a763640 10 monclient(hunting): get_auth_request con 0x7f37f0004e60 auth_method 0 2024-09-17T18:25:30.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f380a763640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:30.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f380a763640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:30.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f380a763640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:25:30.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f380a763640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:30.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f3809761640 1 --2- 172.21.15.121:0/2573695186 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37f00040c0 0x7f37f0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:30.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f380a763640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:30.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f380a763640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:25:30.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f3809761640 10 monclient(hunting): get_auth_request con 0x7f37f00040c0 auth_method 0 2024-09-17T18:25:30.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f3809761640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:30.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f3809761640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:30.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.571+0000 7f3809761640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:25:30.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.572+0000 7f3809761640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:30.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.572+0000 7f3809761640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:30.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.572+0000 7f3809761640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:25:30.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.572+0000 7f380a763640 10 monclient(hunting): handle_auth_done global_id 44397 payload 293 2024-09-17T18:25:30.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.572+0000 7f380a763640 1 -- 172.21.15.121:0/2573695186 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37e8007410 msgr2=0x7f37f00039b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:25:30.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.572+0000 7f380a763640 1 --2- 172.21.15.121:0/2573695186 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37e8007410 0x7f37f00039b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:30.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.572+0000 7f380a763640 1 -- 172.21.15.121:0/2573695186 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37f00040c0 msgr2=0x7f37f0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:25:30.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.572+0000 7f380a763640 1 --2- 172.21.15.121:0/2573695186 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37f00040c0 0x7f37f0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:30.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.572+0000 7f380a763640 10 monclient: _finish_hunting 0 2024-09-17T18:25:30.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.572+0000 7f380a763640 1 monclient: found mon.a 2024-09-17T18:25:30.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.572+0000 7f380a763640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:25:30.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.572+0000 7f380a763640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:30.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.572+0000 7f380a763640 1 -- 172.21.15.121:0/2573695186 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37f0003520 con 0x7f37f0004e60 2024-09-17T18:25:30.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.572+0000 7f380a763640 10 monclient: _finish_auth 0 2024-09-17T18:25:30.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.572+0000 7f380a763640 10 monclient: _check_auth_tickets 2024-09-17T18:25:30.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.572+0000 7f380a763640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:25:30.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.574+0000 7f380a763640 1 -- 172.21.15.121:0/2573695186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37f0004e60 msgr2=0x7f37f000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:30.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.574+0000 7f380a763640 1 --2- 172.21.15.121:0/2573695186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37f0004e60 0x7f37f000b9c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f38040a1640 tx=0x7f37f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e15d0d8a42d74131 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:30.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.575+0000 7f3808f60640 1 -- 172.21.15.121:0/2573695186 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37fc001c60 con 0x7f37f0004e60 2024-09-17T18:25:30.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.575+0000 7f3808f60640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:30.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.575+0000 7f3808f60640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:25:30.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.575+0000 7f3808f60640 10 monclient: dump: 2024-09-17T18:25:30.587 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:30.587 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:30.587 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:30.587 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:30.588 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:30.588 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:30.588 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:30.588 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:30.588 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:30.588 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:30.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.575+0000 7f3808f60640 1 -- 172.21.15.121:0/2573695186 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37fc02e070 con 0x7f37f0004e60 2024-09-17T18:25:30.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.575+0000 7f3808f60640 10 monclient: handle_config config(0 keys) 2024-09-17T18:25:30.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.575+0000 7f3808f60640 1 -- 172.21.15.121:0/2573695186 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37fc005b40 con 0x7f37f0004e60 2024-09-17T18:25:30.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.575+0000 7f380c1c6640 10 monclient: get_monmap_and_config success 2024-09-17T18:25:30.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.575+0000 7f380c1c6640 4 set_mon_vals no callback set 2024-09-17T18:25:30.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.575+0000 7f3808f60640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:30.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.575+0000 7f3808f60640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:25:30.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.575+0000 7f3808f60640 10 monclient: dump: 2024-09-17T18:25:30.588 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:30.589 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:30.589 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:30.589 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:30.589 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:30.589 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:30.589 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:30.589 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:30.589 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:30.589 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:30.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.576+0000 7f380c1c6640 10 monclient: shutdown 2024-09-17T18:25:30.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.576+0000 7f380c1c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:25:30.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.576+0000 7f380c1c6640 1 -- 172.21.15.121:0/2573695186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37f0004e60 msgr2=0x7f37f000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:30.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.576+0000 7f380c1c6640 1 --2- 172.21.15.121:0/2573695186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37f0004e60 0x7f37f000b9c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f38040a1640 tx=0x7f37f0005830 comp rx=0 tx=0).stop 2024-09-17T18:25:30.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.576+0000 7f380c1c6640 1 -- 172.21.15.121:0/2573695186 shutdown_connections 2024-09-17T18:25:30.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.576+0000 7f380c1c6640 1 --2- 172.21.15.121:0/2573695186 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37f00040c0 0x7f37f0004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:30.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.576+0000 7f380c1c6640 1 --2- 172.21.15.121:0/2573695186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37f0004e60 0x7f37f000b9c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:30.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.576+0000 7f380c1c6640 1 --2- 172.21.15.121:0/2573695186 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37e8007410 0x7f37f00039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:30.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.576+0000 7f380c1c6640 1 -- 172.21.15.121:0/2573695186 >> 172.21.15.121:0/2573695186 conn(0x7f38040a3b50 msgr2=0x7f38040a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:25:30.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.576+0000 7f380c1c6640 1 -- 172.21.15.121:0/2573695186 shutdown_connections 2024-09-17T18:25:30.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.576+0000 7f380c1c6640 1 -- 172.21.15.121:0/2573695186 wait complete. 2024-09-17T18:25:30.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.577+0000 7f380c1c6640 10 monclient: build_initial_monmap 2024-09-17T18:25:30.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.577+0000 7f380c1c6640 10 monclient: monmap: 2024-09-17T18:25:30.590 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:25:30.590 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:25:30.590 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:25:30.590 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:25:30.590 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:25:30.590 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:30.591 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:25:30.591 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:25:30.591 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:25:30.591 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:30.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.577+0000 7f380c1c6640 1 librados: starting msgr at 2024-09-17T18:25:30.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.577+0000 7f380c1c6640 1 librados: starting objecter 2024-09-17T18:25:30.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.577+0000 7f380c1c6640 1 Processor -- start 2024-09-17T18:25:30.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.577+0000 7f380c1c6640 1 -- start start 2024-09-17T18:25:30.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.577+0000 7f380c1c6640 1 librados: setting wanted keys 2024-09-17T18:25:30.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.577+0000 7f380c1c6640 1 librados: calling monclient init 2024-09-17T18:25:30.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.577+0000 7f380c1c6640 10 monclient: init 2024-09-17T18:25:30.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.577+0000 7f380c1c6640 10 monclient: _reopen_session rank -1 2024-09-17T18:25:30.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.577+0000 7f380c1c6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T18:25:30.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.577+0000 7f380c1c6640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37e8007410 0x7f38041cd040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:30.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.577+0000 7f380c1c6640 10 monclient(hunting): picked mon.noname-b con 0x7f37e8007410 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:25:30.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.578+0000 7f380c1c6640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37f00040c0 0x7f3804133b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:30.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.578+0000 7f380c1c6640 10 monclient(hunting): picked mon.noname-a con 0x7f37f00040c0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:25:30.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.578+0000 7f380c1c6640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37f0004e60 0x7f3804134090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:30.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.578+0000 7f380c1c6640 10 monclient(hunting): picked mon.noname-c con 0x7f37f0004e60 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:25:30.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.578+0000 7f380c1c6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:30.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.578+0000 7f380c1c6640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f380409de60 con 0x7f37f00040c0 2024-09-17T18:25:30.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.578+0000 7f380c1c6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:30.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.578+0000 7f380c1c6640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f380409dce0 con 0x7f37e8007410 2024-09-17T18:25:30.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.578+0000 7f380c1c6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:30.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.578+0000 7f380c1c6640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f38041348f0 con 0x7f37f0004e60 2024-09-17T18:25:30.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.578+0000 7f380c1c6640 10 monclient(hunting): _renew_subs 2024-09-17T18:25:30.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.578+0000 7f380c1c6640 10 monclient(hunting): authenticate will time out at 1134.295078s 2024-09-17T18:25:30.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.578+0000 7f380a763640 1 -- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37f0004e60 msgr2=0x7f3804134090 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:25:30.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.578+0000 7f380a763640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37f0004e60 0x7f3804134090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:25:30.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.578+0000 7f3809761640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37f00040c0 0x7f3804133b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:30.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.578+0000 7f3809761640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37f00040c0 0x7f3804133b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38784/0 (socket says 172.21.15.121:38784) 2024-09-17T18:25:30.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.578+0000 7f3809761640 1 -- 172.21.15.121:0/4257429294 learned_addr learned my addr 172.21.15.121:0/4257429294 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:25:30.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.578+0000 7f3809761640 10 monclient(hunting): get_auth_request con 0x7f37f00040c0 auth_method 0 2024-09-17T18:25:30.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.578+0000 7f3809761640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:30.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.578+0000 7f3809761640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:30.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.578+0000 7f3809761640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:30.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.578+0000 7f3809f62640 1 --2- 172.21.15.121:0/4257429294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37e8007410 0x7f38041cd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:30.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.578+0000 7f3809f62640 10 monclient(hunting): get_auth_request con 0x7f37e8007410 auth_method 0 2024-09-17T18:25:30.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.578+0000 7f3809f62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:30.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.578+0000 7f3809f62640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:30.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.578+0000 7f3809f62640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:30.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.578+0000 7f3809761640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:30.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.578+0000 7f3809761640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:30.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.578+0000 7f3809761640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:30.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.578+0000 7f3809f62640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:30.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.578+0000 7f3809f62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:30.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.578+0000 7f3809f62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:30.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.578+0000 7f3809761640 10 monclient(hunting): handle_auth_done global_id 44400 payload 824 2024-09-17T18:25:30.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.578+0000 7f3809761640 1 -- 172.21.15.121:0/4257429294 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37f0004e60 msgr2=0x7f3804134090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:25:30.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.578+0000 7f3809761640 1 --2- 172.21.15.121:0/4257429294 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37f0004e60 0x7f3804134090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:30.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.578+0000 7f3809761640 1 -- 172.21.15.121:0/4257429294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37e8007410 msgr2=0x7f38041cd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:30.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.578+0000 7f3809761640 1 --2- 172.21.15.121:0/4257429294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37e8007410 0x7f38041cd040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:30.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.578+0000 7f3809761640 10 monclient: _finish_hunting 0 2024-09-17T18:25:30.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.578+0000 7f3809761640 1 monclient: found mon.noname-a 2024-09-17T18:25:30.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.578+0000 7f3809761640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:25:30.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.578+0000 7f3809761640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:25:30.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.578+0000 7f3809761640 1 -- 172.21.15.121:0/4257429294 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38040a2f50 con 0x7f37f00040c0 2024-09-17T18:25:30.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.580+0000 7f3809761640 1 -- 172.21.15.121:0/4257429294 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37f00040c0 msgr2=0x7f3804133b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:30.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.580+0000 7f3809761640 1 --2- 172.21.15.121:0/4257429294 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37f00040c0 0x7f3804133b50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f380000cdf0 tx=0x7f380000b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:30.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.580+0000 7f37f67fc640 10 client.?.objecter ms_handle_connect 0x7f37f00040c0 2024-09-17T18:25:30.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.580+0000 7f37f67fc640 10 client.?.objecter resend_mon_ops 2024-09-17T18:25:30.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.580+0000 7f37f67fc640 1 -- 172.21.15.121:0/4257429294 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f380000de60 con 0x7f37f00040c0 2024-09-17T18:25:30.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.580+0000 7f37f67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:30.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.581+0000 7f37f67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:25:30.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.581+0000 7f37f67fc640 10 monclient: dump: 2024-09-17T18:25:30.595 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:30.595 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:30.595 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:30.595 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:30.596 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:30.596 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:30.596 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:30.596 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:30.596 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:30.596 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:30.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.581+0000 7f37f67fc640 10 monclient: _finish_auth 0 2024-09-17T18:25:30.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.581+0000 7f37f67fc640 10 monclient: _check_auth_tickets 2024-09-17T18:25:30.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.581+0000 7f37f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:25:30.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.581+0000 7f380c1c6640 5 monclient: authenticate success, global_id 44400 2024-09-17T18:25:30.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.581+0000 7f380c1c6640 10 monclient: _renew_subs 2024-09-17T18:25:30.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.581+0000 7f37f67fc640 1 -- 172.21.15.121:0/4257429294 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3800004030 con 0x7f37f00040c0 2024-09-17T18:25:30.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.581+0000 7f380c1c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:30.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.581+0000 7f380c1c6640 1 -- 172.21.15.121:0/4257429294 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3804125370 con 0x7f37f00040c0 2024-09-17T18:25:30.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.581+0000 7f37f67fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:25:30.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.581+0000 7f37f67fc640 1 -- 172.21.15.121:0/4257429294 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38000043e0 con 0x7f37f00040c0 2024-09-17T18:25:30.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.581+0000 7f37f67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:30.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.581+0000 7f37f67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:25:30.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.581+0000 7f37f67fc640 10 monclient: dump: 2024-09-17T18:25:30.597 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:30.597 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:30.597 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:30.597 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:30.597 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:30.597 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:30.597 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:30.597 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:30.597 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:30.597 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:30.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.581+0000 7f37f7fff640 4 set_mon_vals no callback set 2024-09-17T18:25:30.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.581+0000 7f380c1c6640 10 client.44400.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:25:30.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.581+0000 7f380c1c6640 10 monclient: _renew_subs 2024-09-17T18:25:30.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.581+0000 7f380c1c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:30.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.581+0000 7f380c1c6640 1 -- 172.21.15.121:0/4257429294 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3804123e50 con 0x7f37f00040c0 2024-09-17T18:25:30.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.581+0000 7f380c1c6640 1 librados: init done 2024-09-17T18:25:30.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.582+0000 7f380c1c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:30.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.582+0000 7f37f67fc640 1 -- 172.21.15.121:0/4257429294 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f3800019810 con 0x7f37f00040c0 2024-09-17T18:25:30.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.582+0000 7f380c1c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:30.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.582+0000 7f380c1c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:30.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.582+0000 7f380c1c6640 1 -- 172.21.15.121:0/4257429294 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38041232f0 con 0x7f37f00040c0 2024-09-17T18:25:30.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.582+0000 7f37f67fc640 1 --2- 172.21.15.121:0/4257429294 >> v2:172.21.15.173:6812/2423786037 conn(0x7f37e8040a60 0x7f37e8042f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:30.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.582+0000 7f37f67fc640 1 -- 172.21.15.121:0/4257429294 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(288..288 src has 1..288) ==== 6231+0+0 (secure 0 0 0) 0x7f3800054e10 con 0x7f37f00040c0 2024-09-17T18:25:30.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.582+0000 7f37f67fc640 10 client.44400.objecter ms_dispatch 0x7f3804000b90 osd_map(288..288 src has 1..288) 2024-09-17T18:25:30.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.582+0000 7f37f67fc640 3 client.44400.objecter handle_osd_map got epochs [288,288] > 0 2024-09-17T18:25:30.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.582+0000 7f37f67fc640 3 client.44400.objecter handle_osd_map decoding full epoch 288 2024-09-17T18:25:30.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.582+0000 7f37f67fc640 20 client.44400.objecter dump_active .. 0 homeless 2024-09-17T18:25:30.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.582+0000 7f3809f62640 1 --2- 172.21.15.121:0/4257429294 >> v2:172.21.15.173:6812/2423786037 conn(0x7f37e8040a60 0x7f37e8042f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:30.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.582+0000 7f3809f62640 10 monclient: get_auth_request con 0x7f37e8040a60 auth_method 0 2024-09-17T18:25:30.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.585+0000 7f3809f62640 1 --2- 172.21.15.121:0/4257429294 >> v2:172.21.15.173:6812/2423786037 conn(0x7f37e8040a60 0x7f37e8042f20 secure :-1 s=READY pgs=3984 cs=0 l=1 rev1=1 crypto rx=0x7f37f8000c00 tx=0x7f37f8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:30.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.585+0000 7f37f67fc640 10 client.44400.objecter ms_handle_connect 0x7f37e8040a60 2024-09-17T18:25:30.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.587+0000 7f37f67fc640 1 -- 172.21.15.121:0/4257429294 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f38040a2f50 con 0x7f37f00040c0 2024-09-17T18:25:30.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.587+0000 7f37f67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:30.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.587+0000 7f37f67fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:25:30.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.833+0000 7f380c1c6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:25:30.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.833+0000 7f380c1c6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:25:30.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.833+0000 7f380c1c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:30.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.833+0000 7f380c1c6640 1 -- 172.21.15.121:0/4257429294 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f38040a1a50 con 0x7f37f00040c0 2024-09-17T18:25:30.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.834+0000 7f37f67fc640 1 -- 172.21.15.121:0/4257429294 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f3800056030 con 0x7f37f00040c0 2024-09-17T18:25:30.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.834+0000 7f37f67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:25:30.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.834+0000 7f37f67fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:25:30.837 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":30,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:25:30.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.836+0000 7f380c1c6640 10 librados: watch_flush enter 2024-09-17T18:25:30.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.836+0000 7f380c1c6640 10 librados: watch_flush exit 2024-09-17T18:25:30.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.836+0000 7f380c1c6640 20 client.44400.objecter shutdown clearing up homeless session... 2024-09-17T18:25:30.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.836+0000 7f380c1c6640 10 client.44400.objecter successfully canceled tick 2024-09-17T18:25:30.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.836+0000 7f380c1c6640 1 -- 172.21.15.121:0/4257429294 >> v2:172.21.15.173:6812/2423786037 conn(0x7f37e8040a60 msgr2=0x7f37e8042f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:30.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.836+0000 7f380c1c6640 1 --2- 172.21.15.121:0/4257429294 >> v2:172.21.15.173:6812/2423786037 conn(0x7f37e8040a60 0x7f37e8042f20 secure :-1 s=READY pgs=3984 cs=0 l=1 rev1=1 crypto rx=0x7f37f8000c00 tx=0x7f37f8008040 comp rx=0 tx=0).stop 2024-09-17T18:25:30.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.836+0000 7f380c1c6640 10 monclient: shutdown 2024-09-17T18:25:30.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.836+0000 7f380c1c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:25:30.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.836+0000 7f380c1c6640 1 -- 172.21.15.121:0/4257429294 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37f00040c0 msgr2=0x7f3804133b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:30.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.836+0000 7f380c1c6640 1 --2- 172.21.15.121:0/4257429294 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37f00040c0 0x7f3804133b50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f380000cdf0 tx=0x7f380000b7a0 comp rx=0 tx=0).stop 2024-09-17T18:25:30.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.836+0000 7f380c1c6640 1 -- 172.21.15.121:0/4257429294 shutdown_connections 2024-09-17T18:25:30.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.836+0000 7f380c1c6640 1 --2- 172.21.15.121:0/4257429294 >> v2:172.21.15.173:6812/2423786037 conn(0x7f37e8040a60 0x7f37e8042f20 unknown :-1 s=CLOSED pgs=3984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:30.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.836+0000 7f380c1c6640 1 --2- 172.21.15.121:0/4257429294 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37f0004e60 0x7f3804134090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:30.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.836+0000 7f380c1c6640 1 --2- 172.21.15.121:0/4257429294 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37f00040c0 0x7f3804133b50 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:30.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.836+0000 7f380c1c6640 1 --2- 172.21.15.121:0/4257429294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37e8007410 0x7f38041cd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:30.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.836+0000 7f380c1c6640 1 -- 172.21.15.121:0/4257429294 >> 172.21.15.121:0/4257429294 conn(0x7f38040a3b50 msgr2=0x7f38040a1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:25:30.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.836+0000 7f380c1c6640 1 -- 172.21.15.121:0/4257429294 shutdown_connections 2024-09-17T18:25:30.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.836+0000 7f380c1c6640 1 -- 172.21.15.121:0/4257429294 wait complete. 2024-09-17T18:25:30.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.836+0000 7f380c1c6640 1 librados: shutdown 2024-09-17T18:25:30.850 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-17T18:25:30.850 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-17T18:25:30.850 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T18:25:30.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.946+0000 7fa717b52640 10 monclient: get_monmap_and_config 2024-09-17T18:25:30.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.946+0000 7fa717b52640 10 monclient: build_initial_monmap 2024-09-17T18:25:30.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.946+0000 7fa717b52640 10 monclient: monmap: 2024-09-17T18:25:30.948 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:25:30.948 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:30.948 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:25:30.946687+0000 2024-09-17T18:25:30.948 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:25:30.946687+0000 2024-09-17T18:25:30.948 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:25:30.948 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:30.948 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:25:30.948 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:25:30.948 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:25:30.948 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:25:30.949 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:25:30.949 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:30.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.946+0000 7fa717b52640 1 Processor -- start 2024-09-17T18:25:30.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.946+0000 7fa717b52640 1 -- start start 2024-09-17T18:25:30.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.946+0000 7fa717b52640 10 monclient: init 2024-09-17T18:25:30.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.946+0000 7fa717b52640 10 monclient: _reopen_session rank -1 2024-09-17T18:25:30.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.946+0000 7fa717b52640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-17T18:25:30.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.946+0000 7fa717b52640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa7100700f0 0x7fa710134e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:30.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.946+0000 7fa717b52640 10 monclient(hunting): picked mon.noname-a con 0x7fa7100700f0 addr v2:172.21.15.121:3300/0 2024-09-17T18:25:30.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.946+0000 7fa717b52640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa71009e810 0x7fa710138360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:30.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.946+0000 7fa717b52640 10 monclient(hunting): picked mon.noname-b con 0x7fa71009e810 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:25:30.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.946+0000 7fa717b52640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa71009ec60 addr v1:172.21.15.173:6789/0 2024-09-17T18:25:30.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.946+0000 7fa717b52640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa71009ca60 con 0x7fa71009ec60 2024-09-17T18:25:30.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.946+0000 7fa717b52640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:30.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.946+0000 7fa717b52640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fa7100704d0 con 0x7fa7100700f0 2024-09-17T18:25:30.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.946+0000 7fa717b52640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:30.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.946+0000 7fa717b52640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa71009cd20 con 0x7fa71009e810 2024-09-17T18:25:30.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.946+0000 7fa717b52640 10 monclient(hunting): _renew_subs 2024-09-17T18:25:30.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.946+0000 7fa717b52640 10 monclient(hunting): authenticate will time out at 1134.663393s 2024-09-17T18:25:30.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.946+0000 7fa7150ed640 1 -- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa71009e810 msgr2=0x7fa710138360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:25:30.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.946+0000 7fa7150ed640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa71009e810 0x7fa710138360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:25:30.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.946+0000 7fa7158ee640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa7100700f0 0x7fa710134e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:30.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.946+0000 7fa7158ee640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa7100700f0 0x7fa710134e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38790/0 (socket says 172.21.15.121:38790) 2024-09-17T18:25:30.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.946+0000 7fa7158ee640 1 -- 172.21.15.121:0/2606764928 learned_addr learned my addr 172.21.15.121:0/2606764928 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:25:30.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.946+0000 7fa7158ee640 10 monclient(hunting): get_auth_request con 0x7fa7100700f0 auth_method 0 2024-09-17T18:25:30.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.946+0000 7fa7158ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:30.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.946+0000 7fa7158ee640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:30.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.946+0000 7fa7158ee640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:30.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.946+0000 7fa7158ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:30.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.946+0000 7fa7158ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:30.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.946+0000 7fa7158ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:30.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.946+0000 7fa7158ee640 10 monclient(hunting): handle_auth_done global_id 44403 payload 274 2024-09-17T18:25:30.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.946+0000 7fa7158ee640 1 -- 172.21.15.121:0/2606764928 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa71009e810 msgr2=0x7fa710138360 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:25:30.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.946+0000 7fa7158ee640 1 --2- 172.21.15.121:0/2606764928 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa71009e810 0x7fa710138360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:30.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.946+0000 7fa7158ee640 1 -- 172.21.15.121:0/2606764928 >> v1:172.21.15.173:6789/0 conn(0x7fa71009ec60 legacy=0x7fa7101388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:25:30.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.946+0000 7fa7158ee640 10 monclient: _finish_hunting 0 2024-09-17T18:25:30.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.946+0000 7fa7158ee640 1 monclient: found mon.noname-a 2024-09-17T18:25:30.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.946+0000 7fa7158ee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:25:30.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.946+0000 7fa7158ee640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:25:30.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.946+0000 7fa7158ee640 1 -- 172.21.15.121:0/2606764928 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa71009d000 con 0x7fa7100700f0 2024-09-17T18:25:30.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.949+0000 7fa7158ee640 1 -- 172.21.15.121:0/2606764928 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7100700f0 msgr2=0x7fa710134e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:30.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.949+0000 7fa7158ee640 1 --2- 172.21.15.121:0/2606764928 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7100700f0 0x7fa710134e50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa708009870 tx=0x7fa70802a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8378e4b25dfb83da server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:30.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.949+0000 7fa7148ec640 1 -- 172.21.15.121:0/2606764928 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa708037070 con 0x7fa7100700f0 2024-09-17T18:25:30.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.949+0000 7fa7148ec640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:30.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.949+0000 7fa7148ec640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:25:30.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.949+0000 7fa7148ec640 10 monclient: dump: 2024-09-17T18:25:30.959 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:30.960 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:30.960 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:30.960 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:30.960 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:30.960 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:30.960 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:30.960 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:30.960 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:30.960 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:30.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.949+0000 7fa7148ec640 10 monclient: _finish_auth 0 2024-09-17T18:25:30.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.949+0000 7fa7148ec640 10 monclient: _check_auth_tickets 2024-09-17T18:25:30.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.949+0000 7fa7148ec640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:25:30.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.949+0000 7fa717b52640 5 monclient: authenticate success, global_id 44403 2024-09-17T18:25:30.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.949+0000 7fa7148ec640 1 -- 172.21.15.121:0/2606764928 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa708032640 con 0x7fa7100700f0 2024-09-17T18:25:30.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.949+0000 7fa717b52640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:25:30.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.949+0000 7fa7148ec640 10 monclient: handle_config config(0 keys) 2024-09-17T18:25:30.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.949+0000 7fa717b52640 10 monclient: get_monmap_and_config success 2024-09-17T18:25:30.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.949+0000 7fa717b52640 4 set_mon_vals no callback set 2024-09-17T18:25:30.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.949+0000 7fa7148ec640 1 -- 172.21.15.121:0/2606764928 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa708032a10 con 0x7fa7100700f0 2024-09-17T18:25:30.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.949+0000 7fa7148ec640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:30.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.949+0000 7fa7148ec640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:25:30.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.949+0000 7fa7148ec640 10 monclient: dump: 2024-09-17T18:25:30.961 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:30.961 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:30.961 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:30.961 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:30.962 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:30.962 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:30.962 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:30.962 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:30.962 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:30.962 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:30.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.950+0000 7fa717b52640 10 monclient: shutdown 2024-09-17T18:25:30.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.950+0000 7fa717b52640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:25:30.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.950+0000 7fa717b52640 1 -- 172.21.15.121:0/2606764928 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7100700f0 msgr2=0x7fa710134e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:30.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.950+0000 7fa717b52640 1 --2- 172.21.15.121:0/2606764928 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7100700f0 0x7fa710134e50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa708009870 tx=0x7fa70802a0d0 comp rx=0 tx=0).stop 2024-09-17T18:25:30.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.950+0000 7fa717b52640 1 -- 172.21.15.121:0/2606764928 shutdown_connections 2024-09-17T18:25:30.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.950+0000 7fa717b52640 1 --2- 172.21.15.121:0/2606764928 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa71009e810 0x7fa710138360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:30.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.950+0000 7fa717b52640 1 --2- 172.21.15.121:0/2606764928 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7100700f0 0x7fa710134e50 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:30.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.950+0000 7fa717b52640 1 -- 172.21.15.121:0/2606764928 >> 172.21.15.121:0/2606764928 conn(0x7fa7100ab830 msgr2=0x7fa7100a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:25:30.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.950+0000 7fa717b52640 1 -- 172.21.15.121:0/2606764928 shutdown_connections 2024-09-17T18:25:30.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.950+0000 7fa717b52640 1 -- 172.21.15.121:0/2606764928 wait complete. 2024-09-17T18:25:30.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.951+0000 7fa717b52640 10 monclient: build_initial_monmap 2024-09-17T18:25:30.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.951+0000 7fa717b52640 10 monclient: monmap: 2024-09-17T18:25:30.963 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:25:30.963 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:25:30.963 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:25:30.963 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:25:30.963 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:25:30.963 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:30.963 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:25:30.963 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:25:30.963 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:25:30.963 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:30.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.951+0000 7fa717b52640 1 librados: starting msgr at 2024-09-17T18:25:30.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.951+0000 7fa717b52640 1 librados: starting objecter 2024-09-17T18:25:30.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.951+0000 7fa717b52640 1 Processor -- start 2024-09-17T18:25:30.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.951+0000 7fa717b52640 1 -- start start 2024-09-17T18:25:30.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.951+0000 7fa717b52640 1 librados: setting wanted keys 2024-09-17T18:25:30.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.951+0000 7fa717b52640 1 librados: calling monclient init 2024-09-17T18:25:30.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.951+0000 7fa717b52640 10 monclient: init 2024-09-17T18:25:30.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.951+0000 7fa717b52640 10 monclient: _reopen_session rank -1 2024-09-17T18:25:30.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.951+0000 7fa717b52640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T18:25:30.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.951+0000 7fa717b52640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7100700f0 0x7fa71012ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:30.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.951+0000 7fa717b52640 10 monclient(hunting): picked mon.noname-c con 0x7fa7100700f0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:25:30.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.951+0000 7fa717b52640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa71009e810 0x7fa71012f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:30.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.951+0000 7fa717b52640 10 monclient(hunting): picked mon.noname-b con 0x7fa71009e810 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:25:30.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.951+0000 7fa717b52640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa71009ec60 0x7fa7101cd590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:30.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.951+0000 7fa717b52640 10 monclient(hunting): picked mon.noname-a con 0x7fa71009ec60 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:25:30.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.951+0000 7fa717b52640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:30.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.951+0000 7fa717b52640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa710135510 con 0x7fa71009ec60 2024-09-17T18:25:30.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.951+0000 7fa717b52640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:30.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.951+0000 7fa717b52640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa710135390 con 0x7fa71009e810 2024-09-17T18:25:30.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.951+0000 7fa717b52640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:30.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.951+0000 7fa717b52640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa710138fb0 con 0x7fa7100700f0 2024-09-17T18:25:30.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.951+0000 7fa717b52640 10 monclient(hunting): _renew_subs 2024-09-17T18:25:30.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.951+0000 7fa7158ee640 1 -- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7100700f0 msgr2=0x7fa71012ece0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:25:30.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.951+0000 7fa717b52640 10 monclient(hunting): authenticate will time out at 1134.668957s 2024-09-17T18:25:30.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.951+0000 7fa7158ee640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7100700f0 0x7fa71012ece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:25:30.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.952+0000 7fa7160ef640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa71009ec60 0x7fa7101cd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:30.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.952+0000 7fa7160ef640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa71009ec60 0x7fa7101cd590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38794/0 (socket says 172.21.15.121:38794) 2024-09-17T18:25:30.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.952+0000 7fa7160ef640 1 -- 172.21.15.121:0/3756413634 learned_addr learned my addr 172.21.15.121:0/3756413634 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:25:30.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.952+0000 7fa7160ef640 10 monclient(hunting): get_auth_request con 0x7fa71009ec60 auth_method 0 2024-09-17T18:25:30.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.952+0000 7fa7160ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:30.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.952+0000 7fa7160ef640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:30.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.952+0000 7fa7160ef640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:30.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.952+0000 7fa7160ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:30.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.952+0000 7fa7160ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:30.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.952+0000 7fa7160ef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:30.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.952+0000 7fa7150ed640 1 --2- 172.21.15.121:0/3756413634 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa71009e810 0x7fa71012f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:30.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.952+0000 7fa7160ef640 10 monclient(hunting): handle_auth_done global_id 44406 payload 824 2024-09-17T18:25:30.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.952+0000 7fa7160ef640 1 -- 172.21.15.121:0/3756413634 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7100700f0 msgr2=0x7fa71012ece0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:25:30.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.952+0000 7fa7160ef640 1 --2- 172.21.15.121:0/3756413634 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7100700f0 0x7fa71012ece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:30.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.952+0000 7fa7160ef640 1 -- 172.21.15.121:0/3756413634 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa71009e810 msgr2=0x7fa71012f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:30.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.952+0000 7fa7160ef640 1 --2- 172.21.15.121:0/3756413634 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa71009e810 0x7fa71012f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:30.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.952+0000 7fa7160ef640 10 monclient: _finish_hunting 0 2024-09-17T18:25:30.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.952+0000 7fa7160ef640 1 monclient: found mon.noname-a 2024-09-17T18:25:30.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.952+0000 7fa7160ef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:25:30.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.952+0000 7fa7160ef640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:25:30.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.952+0000 7fa7160ef640 1 -- 172.21.15.121:0/3756413634 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa71012f760 con 0x7fa71009ec60 2024-09-17T18:25:30.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.952+0000 7fa7150ed640 10 monclient: get_auth_request con 0x7fa71009e810 auth_method 0 2024-09-17T18:25:30.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.952+0000 7fa7150ed640 1 --2- 172.21.15.121:0/3756413634 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa71009e810 0x7fa71012f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T18:25:30.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.954+0000 7fa7160ef640 1 -- 172.21.15.121:0/3756413634 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa71009ec60 msgr2=0x7fa7101cd590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:30.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.954+0000 7fa7160ef640 1 --2- 172.21.15.121:0/3756413634 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa71009ec60 0x7fa7101cd590 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa70000c780 tx=0x7fa70000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:30.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.954+0000 7fa70e7fc640 10 client.?.objecter ms_handle_connect 0x7fa71009ec60 2024-09-17T18:25:30.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.954+0000 7fa70e7fc640 10 client.?.objecter resend_mon_ops 2024-09-17T18:25:30.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.954+0000 7fa70e7fc640 1 -- 172.21.15.121:0/3756413634 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa70000ac60 con 0x7fa71009ec60 2024-09-17T18:25:30.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.954+0000 7fa70e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:30.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.954+0000 7fa70e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:25:30.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.954+0000 7fa70e7fc640 10 monclient: dump: 2024-09-17T18:25:30.970 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:30.970 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:30.970 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:30.970 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:30.970 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:30.970 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:30.970 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:30.970 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:30.970 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:30.970 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:30.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.954+0000 7fa70e7fc640 10 monclient: _finish_auth 0 2024-09-17T18:25:30.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.954+0000 7fa70e7fc640 10 monclient: _check_auth_tickets 2024-09-17T18:25:30.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.954+0000 7fa70e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:25:30.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.954+0000 7fa70e7fc640 1 -- 172.21.15.121:0/3756413634 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa70000ae00 con 0x7fa71009ec60 2024-09-17T18:25:30.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.954+0000 7fa717b52640 5 monclient: authenticate success, global_id 44406 2024-09-17T18:25:30.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.954+0000 7fa717b52640 10 monclient: _renew_subs 2024-09-17T18:25:30.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.954+0000 7fa717b52640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:30.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.954+0000 7fa717b52640 1 -- 172.21.15.121:0/3756413634 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7100a0fa0 con 0x7fa71009ec60 2024-09-17T18:25:30.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.954+0000 7fa70e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:25:30.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.954+0000 7fa70e7fc640 1 -- 172.21.15.121:0/3756413634 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa700004ba0 con 0x7fa71009ec60 2024-09-17T18:25:30.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.954+0000 7fa70e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:30.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.954+0000 7fa70e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:25:30.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.954+0000 7fa70e7fc640 10 monclient: dump: 2024-09-17T18:25:30.971 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:30.971 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:30.971 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:30.971 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:30.972 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:30.972 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:30.972 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:30.972 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:30.972 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:30.972 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:30.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.954+0000 7fa717b52640 10 client.44406.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:25:30.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.954+0000 7fa70ffff640 4 set_mon_vals no callback set 2024-09-17T18:25:30.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.954+0000 7fa717b52640 10 monclient: _renew_subs 2024-09-17T18:25:30.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.954+0000 7fa717b52640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:30.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.954+0000 7fa717b52640 1 -- 172.21.15.121:0/3756413634 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7101348a0 con 0x7fa71009ec60 2024-09-17T18:25:30.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.954+0000 7fa717b52640 1 librados: init done 2024-09-17T18:25:30.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.955+0000 7fa717b52640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:30.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.955+0000 7fa717b52640 1 --2- 172.21.15.121:0/3756413634 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7101cdad0 0x7fa7101cdeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:30.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.955+0000 7fa717b52640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:30.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.955+0000 7fa717b52640 1 -- 172.21.15.121:0/3756413634 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa7101ce3f0 con 0x7fa7101cdad0 2024-09-17T18:25:30.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.955+0000 7fa70e7fc640 1 -- 172.21.15.121:0/3756413634 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fa700005030 con 0x7fa71009ec60 2024-09-17T18:25:30.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.955+0000 7fa7158ee640 1 --2- 172.21.15.121:0/3756413634 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7101cdad0 0x7fa7101cdeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:30.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.955+0000 7fa7158ee640 10 monclient: get_auth_request con 0x7fa7101cdad0 auth_method 0 2024-09-17T18:25:30.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.955+0000 7fa7158ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:30.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.955+0000 7fa7158ee640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:30.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.955+0000 7fa7158ee640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:30.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.955+0000 7fa7158ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:30.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.955+0000 7fa7158ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:30.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.955+0000 7fa7158ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:30.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.955+0000 7fa7158ee640 10 monclient(hunting): handle_auth_done global_id 44409 payload 824 2024-09-17T18:25:30.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.955+0000 7fa70e7fc640 1 --2- 172.21.15.121:0/3756413634 >> v2:172.21.15.173:6812/2423786037 conn(0x7fa6fc040120 0x7fa6fc0425e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:30.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.955+0000 7fa7158ee640 1 -- 172.21.15.121:0/3756413634 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa710134e50 con 0x7fa7101cdad0 2024-09-17T18:25:30.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.956+0000 7fa70e7fc640 1 -- 172.21.15.121:0/3756413634 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(288..288 src has 1..288) ==== 6231+0+0 (secure 0 0 0) 0x7fa700053e50 con 0x7fa71009ec60 2024-09-17T18:25:30.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.956+0000 7fa70e7fc640 10 client.44406.objecter ms_dispatch 0x7fa710000b90 osd_map(288..288 src has 1..288) 2024-09-17T18:25:30.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.956+0000 7fa70e7fc640 3 client.44406.objecter handle_osd_map got epochs [288,288] > 0 2024-09-17T18:25:30.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.956+0000 7fa70e7fc640 3 client.44406.objecter handle_osd_map decoding full epoch 288 2024-09-17T18:25:30.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.956+0000 7fa70e7fc640 20 client.44406.objecter dump_active .. 0 homeless 2024-09-17T18:25:30.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.956+0000 7fa7150ed640 1 --2- 172.21.15.121:0/3756413634 >> v2:172.21.15.173:6812/2423786037 conn(0x7fa6fc040120 0x7fa6fc0425e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:30.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.956+0000 7fa7150ed640 10 monclient: get_auth_request con 0x7fa6fc040120 auth_method 0 2024-09-17T18:25:30.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.958+0000 7fa70e7fc640 10 client.44406.objecter ms_handle_connect 0x7fa7101cdad0 2024-09-17T18:25:30.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.958+0000 7fa70e7fc640 10 client.44406.objecter resend_mon_ops 2024-09-17T18:25:30.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.958+0000 7fa7158ee640 1 -- 172.21.15.121:0/3756413634 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7101cdad0 msgr2=0x7fa7101cdeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:30.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.958+0000 7fa7158ee640 1 --2- 172.21.15.121:0/3756413634 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7101cdad0 0x7fa7101cdeb0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa7100a90b0 tx=0x7fa7080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:30.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.958+0000 7fa70e7fc640 1 -- 172.21.15.121:0/3756413634 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa708042070 con 0x7fa7101cdad0 2024-09-17T18:25:30.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.958+0000 7fa70e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:30.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.958+0000 7fa70e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:25:30.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.958+0000 7fa70e7fc640 10 monclient: dump: 2024-09-17T18:25:30.979 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:30.979 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:30.979 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:30.979 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:30.979 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:30.979 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:30.980 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:30.980 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:30.980 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:30.980 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:30.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.958+0000 7fa70e7fc640 1 -- 172.21.15.121:0/3756413634 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa70803e460 con 0x7fa7101cdad0 2024-09-17T18:25:30.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.958+0000 7fa70e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:30.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.958+0000 7fa70e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:25:30.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.958+0000 7fa70e7fc640 1 -- 172.21.15.121:0/3756413634 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7101cdad0 msgr2=0x7fa7101cdeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:30.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.958+0000 7fa70e7fc640 1 --2- 172.21.15.121:0/3756413634 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7101cdad0 0x7fa7101cdeb0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa7100a90b0 tx=0x7fa7080046c0 comp rx=0 tx=0).stop 2024-09-17T18:25:30.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.958+0000 7fa70e7fc640 1 -- 172.21.15.121:0/3756413634 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7101cdad0 msgr2=0x7fa7101cdeb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:25:30.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.958+0000 7fa70e7fc640 1 --2- 172.21.15.121:0/3756413634 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7101cdad0 0x7fa7101cdeb0 secure :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa7100a90b0 tx=0x7fa7080046c0 comp rx=0 tx=0).stop 2024-09-17T18:25:30.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.958+0000 7fa7150ed640 1 --2- 172.21.15.121:0/3756413634 >> v2:172.21.15.173:6812/2423786037 conn(0x7fa6fc040120 0x7fa6fc0425e0 secure :-1 s=READY pgs=3985 cs=0 l=1 rev1=1 crypto rx=0x7fa704006fd0 tx=0x7fa704008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:30.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.958+0000 7fa70e7fc640 10 client.44406.objecter ms_handle_connect 0x7fa6fc040120 2024-09-17T18:25:30.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.965+0000 7fa717b52640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:25:30.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.965+0000 7fa717b52640 1 --2- 172.21.15.121:0/3756413634 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa71012fb40 0x7fa71012ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:30.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.965+0000 7fa717b52640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:30.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.965+0000 7fa717b52640 1 -- 172.21.15.121:0/3756413634 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa71009f9c0 con 0x7fa71012fb40 2024-09-17T18:25:30.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.965+0000 7fa7158ee640 1 --2- 172.21.15.121:0/3756413634 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa71012fb40 0x7fa71012ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:30.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.965+0000 7fa7158ee640 10 monclient: get_auth_request con 0x7fa71012fb40 auth_method 0 2024-09-17T18:25:30.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.965+0000 7fa7158ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:30.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.965+0000 7fa7158ee640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:30.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.965+0000 7fa7158ee640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:30.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.965+0000 7fa7158ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:30.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.965+0000 7fa7158ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:30.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.965+0000 7fa7158ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:30.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.965+0000 7fa7158ee640 10 monclient(hunting): handle_auth_done global_id 44412 payload 824 2024-09-17T18:25:30.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.965+0000 7fa7158ee640 1 -- 172.21.15.121:0/3756413634 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa71006a660 con 0x7fa71012fb40 2024-09-17T18:25:30.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.967+0000 7fa7158ee640 1 -- 172.21.15.121:0/3756413634 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa71012fb40 msgr2=0x7fa71012ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:30.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.967+0000 7fa70e7fc640 10 client.44406.objecter ms_handle_connect 0x7fa71012fb40 2024-09-17T18:25:30.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.967+0000 7fa70e7fc640 10 client.44406.objecter resend_mon_ops 2024-09-17T18:25:30.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.967+0000 7fa7158ee640 1 --2- 172.21.15.121:0/3756413634 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa71012fb40 0x7fa71012ff50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa708000c00 tx=0x7fa708008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:30.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.967+0000 7fa70e7fc640 1 -- 172.21.15.121:0/3756413634 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa708037040 con 0x7fa71012fb40 2024-09-17T18:25:30.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.967+0000 7fa70e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:30.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.967+0000 7fa70e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:25:30.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.967+0000 7fa70e7fc640 10 monclient: dump: 2024-09-17T18:25:30.982 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:30.982 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:30.982 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:30.982 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:30.982 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:30.982 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:30.983 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:30.983 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:30.983 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:30.983 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:30.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.967+0000 7fa70e7fc640 1 -- 172.21.15.121:0/3756413634 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7fa70803e610 con 0x7fa71012fb40 2024-09-17T18:25:30.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.967+0000 7fa70e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:25:30.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.967+0000 7fa70e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:25:30.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.967+0000 7fa70e7fc640 1 -- 172.21.15.121:0/3756413634 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa71012fb40 msgr2=0x7fa71012ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:30.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.967+0000 7fa70e7fc640 1 --2- 172.21.15.121:0/3756413634 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa71012fb40 0x7fa71012ff50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa708000c00 tx=0x7fa708008030 comp rx=0 tx=0).stop 2024-09-17T18:25:30.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.967+0000 7fa70e7fc640 1 -- 172.21.15.121:0/3756413634 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa71012fb40 msgr2=0x7fa71012ff50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:25:30.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.967+0000 7fa70e7fc640 1 --2- 172.21.15.121:0/3756413634 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa71012fb40 0x7fa71012ff50 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:30.984 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:25:30.984 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:25:30.984 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:25:30.984 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "leader", 2024-09-17T18:25:30.984 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 30, 2024-09-17T18:25:30.984 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 156632, 2024-09-17T18:25:30.984 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:25:30.984 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:25:30.984 INFO:teuthology.orchestra.run.smithi121.stdout: 1 2024-09-17T18:25:30.984 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:25:30.984 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 0, 2024-09-17T18:25:30.984 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:25:30.984 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:25:30.985 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:25:30.985 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:25:30.985 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:25:30.985 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:25:30.985 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:25:30.985 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:25:30.985 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:25:30.985 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:25:30.985 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:25:30.985 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:25:30.985 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:25:30.985 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:25:30.985 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:25:30.986 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:25:30.986 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:25:30.986 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:25:30.986 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:25:30.986 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:25:30.986 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:25:30.986 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:25:30.986 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:25:30.986 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:25:30.986 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:25:30.986 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:25:30.986 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:25:30.986 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:25:30.986 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:30.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:30.987 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:25:30.987 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:25:30.987 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:25:30.987 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:25:30.987 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:25:30.987 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:25:30.987 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:25:30.987 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:25:30.987 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:25:30.987 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:25:30.987 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:25:30.987 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:25:30.987 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:25:30.987 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:25:30.988 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:25:30.988 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:25:30.988 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:25:30.988 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:25:30.988 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:25:30.988 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:25:30.988 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:25:30.988 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:25:30.988 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:25:30.988 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:25:30.988 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:25:30.988 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:25:30.988 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:25:30.988 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:25:30.989 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:25:30.989 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:25:30.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:30.989 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:25:30.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:30.989 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:25:30.989 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:25:30.989 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:25:30.989 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:25:30.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:30.989 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:25:30.989 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:25:30.989 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:30.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:30.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:30.990 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:25:30.990 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:25:30.990 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:30.990 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:30.990 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:30.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:30.990 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:25:30.990 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:25:30.990 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:25:30.990 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:25:30.990 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:25:30.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:30.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:30.990 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:25:30.991 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:25:30.991 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:25:30.991 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:25:30.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:30.991 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:25:30.991 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:25:30.991 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:30.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:30.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:30.991 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:25:30.991 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:25:30.991 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:30.991 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:30.991 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:30.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:30.992 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:25:30.992 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:25:30.992 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:25:30.992 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:25:30.992 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:25:30.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:30.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:30.992 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:25:30.992 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:25:30.992 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:25:30.992 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:25:30.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:30.992 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:25:30.993 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:25:30.993 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:30.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:30.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:30.993 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:25:30.993 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:25:30.993 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:30.993 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:30.993 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:30.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:30.993 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:25:30.993 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:25:30.993 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:25:30.993 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:25:30.994 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:25:30.994 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:30.994 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:30.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:30.994 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:25:30.994 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:25:30.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:30.994 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:25:30.994 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:25:30.994 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:25:30.994 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:30.994 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:25:30.994 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:25:30.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:30.995 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:25:30.995 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:25:30.995 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-17T18:25:30.995 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:30.995 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:25:30.995 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:25:30.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:30.995 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:25:30.995 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:25:30.995 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-17T18:25:30.995 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:30.995 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:30.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:30.995 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:25:30.996 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:25:30.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.967+0000 7fa717b52640 10 librados: watch_flush enter 2024-09-17T18:25:30.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.967+0000 7fa717b52640 10 librados: watch_flush exit 2024-09-17T18:25:30.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.967+0000 7fa717b52640 20 client.44406.objecter shutdown clearing up homeless session... 2024-09-17T18:25:30.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.967+0000 7fa717b52640 10 client.44406.objecter successfully canceled tick 2024-09-17T18:25:30.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.967+0000 7fa717b52640 1 -- 172.21.15.121:0/3756413634 >> v2:172.21.15.173:6812/2423786037 conn(0x7fa6fc040120 msgr2=0x7fa6fc0425e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:30.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.967+0000 7fa717b52640 1 --2- 172.21.15.121:0/3756413634 >> v2:172.21.15.173:6812/2423786037 conn(0x7fa6fc040120 0x7fa6fc0425e0 secure :-1 s=READY pgs=3985 cs=0 l=1 rev1=1 crypto rx=0x7fa704006fd0 tx=0x7fa704008040 comp rx=0 tx=0).stop 2024-09-17T18:25:30.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.967+0000 7fa717b52640 10 monclient: shutdown 2024-09-17T18:25:30.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.967+0000 7fa717b52640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:25:30.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.967+0000 7fa717b52640 1 -- 172.21.15.121:0/3756413634 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa71009ec60 msgr2=0x7fa7101cd590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:30.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.967+0000 7fa717b52640 1 --2- 172.21.15.121:0/3756413634 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa71009ec60 0x7fa7101cd590 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa70000c780 tx=0x7fa70000cc40 comp rx=0 tx=0).stop 2024-09-17T18:25:30.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.967+0000 7fa7160ef640 1 -- 172.21.15.121:0/3756413634 reap_dead start 2024-09-17T18:25:30.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.967+0000 7fa7160ef640 1 -- 172.21.15.121:0/3756413634 reap_dead start 2024-09-17T18:25:30.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.967+0000 7fa717b52640 1 -- 172.21.15.121:0/3756413634 shutdown_connections 2024-09-17T18:25:30.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.967+0000 7fa717b52640 1 -- 172.21.15.121:0/3756413634 >> 172.21.15.121:0/3756413634 conn(0x7fa7100ab830 msgr2=0x7fa710137f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:25:30.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.968+0000 7fa717b52640 1 -- 172.21.15.121:0/3756413634 shutdown_connections 2024-09-17T18:25:30.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.968+0000 7fa717b52640 1 -- 172.21.15.121:0/3756413634 wait complete. 2024-09-17T18:25:30.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:30.968+0000 7fa717b52640 1 librados: shutdown 2024-09-17T18:25:30.999 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T18:25:30.999 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-17T18:25:30.999 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T18:25:31.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f575b38a640 10 monclient: get_monmap_and_config 2024-09-17T18:25:31.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f575b38a640 10 monclient: build_initial_monmap 2024-09-17T18:25:31.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f575b38a640 10 monclient: monmap: 2024-09-17T18:25:31.097 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:25:31.097 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:31.098 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:25:31.095767+0000 2024-09-17T18:25:31.098 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:25:31.095767+0000 2024-09-17T18:25:31.098 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:25:31.098 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:31.098 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:25:31.098 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:25:31.099 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:25:31.099 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:25:31.099 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:25:31.099 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:31.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f575b38a640 1 Processor -- start 2024-09-17T18:25:31.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f575b38a640 1 -- start start 2024-09-17T18:25:31.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f575b38a640 10 monclient: init 2024-09-17T18:25:31.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f575b38a640 10 monclient: _reopen_session rank -1 2024-09-17T18:25:31.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f575b38a640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-17T18:25:31.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f575b38a640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f57540700f0 0x7f5754134e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:31.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f575b38a640 10 monclient(hunting): picked mon.noname-c con 0x7f57540700f0 addr v2:172.21.15.173:3300/0 2024-09-17T18:25:31.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f575b38a640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f575409e810 0x7f5754138360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:31.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f575b38a640 10 monclient(hunting): picked mon.noname-a con 0x7f575409e810 addr v2:172.21.15.121:3300/0 2024-09-17T18:25:31.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f575b38a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f575409ec60 0x7f57541388a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:31.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f575b38a640 10 monclient(hunting): picked mon.noname-b con 0x7f575409ec60 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:25:31.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f575b38a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:31.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f575b38a640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f57540704d0 con 0x7f575409e810 2024-09-17T18:25:31.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f575b38a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:31.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f575b38a640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f575409ca60 con 0x7f57540700f0 2024-09-17T18:25:31.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f575b38a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:31.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f575b38a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f575409cd20 con 0x7f575409ec60 2024-09-17T18:25:31.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f575b38a640 10 monclient(hunting): _renew_subs 2024-09-17T18:25:31.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f575b38a640 10 monclient(hunting): authenticate will time out at 1134.812483s 2024-09-17T18:25:31.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f5759927640 1 -- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f575409ec60 msgr2=0x7f57541388a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:25:31.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f5759927640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f575409ec60 0x7f57541388a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:25:31.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f5758925640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f575409e810 0x7f5754138360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:31.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f5758925640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f575409e810 0x7f5754138360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38830/0 (socket says 172.21.15.121:38830) 2024-09-17T18:25:31.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f5758925640 1 -- 172.21.15.121:0/3534958567 learned_addr learned my addr 172.21.15.121:0/3534958567 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:25:31.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f5758925640 10 monclient(hunting): get_auth_request con 0x7f575409e810 auth_method 0 2024-09-17T18:25:31.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f5758925640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:31.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f5758925640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:31.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f5758925640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:31.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f5758925640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:31.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f5758925640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:31.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f5758925640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:31.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f5758925640 10 monclient(hunting): handle_auth_done global_id 44415 payload 274 2024-09-17T18:25:31.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f5759126640 1 --2- 172.21.15.121:0/3534958567 >> v2:172.21.15.173:3300/0 conn(0x7f57540700f0 0x7f5754134e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:31.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f5758925640 1 -- 172.21.15.121:0/3534958567 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f575409ec60 msgr2=0x7f57541388a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:25:31.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f5758925640 1 --2- 172.21.15.121:0/3534958567 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f575409ec60 0x7f57541388a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:31.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f5758925640 1 -- 172.21.15.121:0/3534958567 >> v2:172.21.15.173:3300/0 conn(0x7f57540700f0 msgr2=0x7f5754134e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:25:31.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f5758925640 1 --2- 172.21.15.121:0/3534958567 >> v2:172.21.15.173:3300/0 conn(0x7f57540700f0 0x7f5754134e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:31.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f5758925640 10 monclient: _finish_hunting 0 2024-09-17T18:25:31.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f5758925640 1 monclient: found mon.noname-a 2024-09-17T18:25:31.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f5758925640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:25:31.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f5758925640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:25:31.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.095+0000 7f5758925640 1 -- 172.21.15.121:0/3534958567 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f575409d000 con 0x7f575409e810 2024-09-17T18:25:31.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.098+0000 7f5758925640 1 -- 172.21.15.121:0/3534958567 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f575409e810 msgr2=0x7f5754138360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:31.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.098+0000 7f5758925640 1 --2- 172.21.15.121:0/3534958567 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f575409e810 0x7f5754138360 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5754077b10 tx=0x7f574802a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b623a010e15a1ce server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:31.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.098+0000 7f5753fff640 1 -- 172.21.15.121:0/3534958567 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5748037070 con 0x7f575409e810 2024-09-17T18:25:31.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.098+0000 7f5753fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:31.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.098+0000 7f5753fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:25:31.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.098+0000 7f5753fff640 10 monclient: dump: 2024-09-17T18:25:31.124 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:31.124 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:31.124 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:31.124 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:31.124 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:31.125 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:31.125 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:31.125 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:31.125 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:31.125 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:31.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.098+0000 7f5753fff640 10 monclient: _finish_auth 0 2024-09-17T18:25:31.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.098+0000 7f5753fff640 10 monclient: _check_auth_tickets 2024-09-17T18:25:31.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.098+0000 7f5753fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:25:31.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.098+0000 7f575b38a640 5 monclient: authenticate success, global_id 44415 2024-09-17T18:25:31.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.098+0000 7f5753fff640 1 -- 172.21.15.121:0/3534958567 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5748032640 con 0x7f575409e810 2024-09-17T18:25:31.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.098+0000 7f575b38a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:25:31.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.098+0000 7f5753fff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:25:31.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.098+0000 7f575b38a640 10 monclient: get_monmap_and_config success 2024-09-17T18:25:31.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.098+0000 7f5753fff640 1 -- 172.21.15.121:0/3534958567 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5748032a10 con 0x7f575409e810 2024-09-17T18:25:31.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.098+0000 7f5753fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:31.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.098+0000 7f575b38a640 4 set_mon_vals no callback set 2024-09-17T18:25:31.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.098+0000 7f5753fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:25:31.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.098+0000 7f5753fff640 10 monclient: dump: 2024-09-17T18:25:31.127 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:31.128 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:31.128 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:31.128 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:31.128 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:31.128 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:31.128 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:31.129 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:31.129 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:31.129 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:31.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.099+0000 7f575b38a640 10 monclient: shutdown 2024-09-17T18:25:31.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.099+0000 7f575b38a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:25:31.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.099+0000 7f575b38a640 1 -- 172.21.15.121:0/3534958567 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f575409e810 msgr2=0x7f5754138360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:31.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.099+0000 7f575b38a640 1 --2- 172.21.15.121:0/3534958567 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f575409e810 0x7f5754138360 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5754077b10 tx=0x7f574802a0d0 comp rx=0 tx=0).stop 2024-09-17T18:25:31.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.099+0000 7f575b38a640 1 -- 172.21.15.121:0/3534958567 shutdown_connections 2024-09-17T18:25:31.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.099+0000 7f575b38a640 1 --2- 172.21.15.121:0/3534958567 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f575409ec60 0x7f57541388a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:31.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.099+0000 7f575b38a640 1 --2- 172.21.15.121:0/3534958567 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f575409e810 0x7f5754138360 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:31.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.099+0000 7f575b38a640 1 --2- 172.21.15.121:0/3534958567 >> v2:172.21.15.173:3300/0 conn(0x7f57540700f0 0x7f5754134e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:31.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.099+0000 7f575b38a640 1 -- 172.21.15.121:0/3534958567 >> 172.21.15.121:0/3534958567 conn(0x7f57540ab830 msgr2=0x7f57540a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:25:31.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.099+0000 7f575b38a640 1 -- 172.21.15.121:0/3534958567 shutdown_connections 2024-09-17T18:25:31.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.099+0000 7f575b38a640 1 -- 172.21.15.121:0/3534958567 wait complete. 2024-09-17T18:25:31.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.100+0000 7f575b38a640 10 monclient: build_initial_monmap 2024-09-17T18:25:31.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.100+0000 7f575b38a640 10 monclient: monmap: 2024-09-17T18:25:31.130 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:25:31.130 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:25:31.131 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:25:31.131 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:25:31.131 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:25:31.131 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:31.131 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:25:31.131 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:25:31.131 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:25:31.132 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:31.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.100+0000 7f575b38a640 1 librados: starting msgr at 2024-09-17T18:25:31.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.100+0000 7f575b38a640 1 librados: starting objecter 2024-09-17T18:25:31.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.100+0000 7f575b38a640 1 Processor -- start 2024-09-17T18:25:31.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.100+0000 7f575b38a640 1 -- start start 2024-09-17T18:25:31.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.100+0000 7f575b38a640 1 librados: setting wanted keys 2024-09-17T18:25:31.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.100+0000 7f575b38a640 1 librados: calling monclient init 2024-09-17T18:25:31.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.100+0000 7f575b38a640 10 monclient: init 2024-09-17T18:25:31.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.100+0000 7f575b38a640 10 monclient: _reopen_session rank -1 2024-09-17T18:25:31.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.100+0000 7f575b38a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T18:25:31.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.100+0000 7f575b38a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f57540700f0 0x7f57541cd580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:31.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.100+0000 7f575b38a640 10 monclient(hunting): picked mon.noname-c con 0x7f57540700f0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:25:31.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.100+0000 7f575b38a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f575409e810 0x7f5754131c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:31.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.100+0000 7f575b38a640 10 monclient(hunting): picked mon.noname-a con 0x7f575409e810 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:25:31.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.100+0000 7f575b38a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f575409ec60 0x7f57540a0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:31.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.100+0000 7f575b38a640 10 monclient(hunting): picked mon.noname-b con 0x7f575409ec60 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:25:31.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.100+0000 7f5759126640 1 -- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f57540700f0 msgr2=0x7f57541cd580 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:3301/0 2024-09-17T18:25:31.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.100+0000 7f575b38a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:31.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.100+0000 7f575b38a640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5754135510 con 0x7f575409e810 2024-09-17T18:25:31.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.100+0000 7f575b38a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:31.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.100+0000 7f5759126640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f57540700f0 0x7f57541cd580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:25:31.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.100+0000 7f575b38a640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5754135390 con 0x7f575409ec60 2024-09-17T18:25:31.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.100+0000 7f575b38a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:31.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.100+0000 7f575b38a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5754138de0 con 0x7f57540700f0 2024-09-17T18:25:31.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.100+0000 7f575b38a640 10 monclient(hunting): _renew_subs 2024-09-17T18:25:31.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.100+0000 7f575b38a640 10 monclient(hunting): authenticate will time out at 1134.817919s 2024-09-17T18:25:31.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.100+0000 7f5758925640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f575409e810 0x7f5754131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:31.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.100+0000 7f5758925640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f575409e810 0x7f5754131c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38846/0 (socket says 172.21.15.121:38846) 2024-09-17T18:25:31.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.101+0000 7f5758925640 1 -- 172.21.15.121:0/1695035753 learned_addr learned my addr 172.21.15.121:0/1695035753 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:25:31.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.101+0000 7f5758925640 10 monclient(hunting): get_auth_request con 0x7f575409e810 auth_method 0 2024-09-17T18:25:31.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.101+0000 7f5758925640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:31.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.101+0000 7f5758925640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:31.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.101+0000 7f5758925640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:31.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.101+0000 7f5758925640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:31.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.101+0000 7f5758925640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:31.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.101+0000 7f5758925640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:31.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.101+0000 7f5758925640 10 monclient(hunting): handle_auth_done global_id 44418 payload 824 2024-09-17T18:25:31.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.101+0000 7f5758925640 1 -- 172.21.15.121:0/1695035753 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f57540700f0 msgr2=0x7f57541cd580 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:25:31.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.101+0000 7f5758925640 1 --2- 172.21.15.121:0/1695035753 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f57540700f0 0x7f57541cd580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:31.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.101+0000 7f5758925640 1 -- 172.21.15.121:0/1695035753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f575409ec60 msgr2=0x7f57540a0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:31.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.101+0000 7f5758925640 1 --2- 172.21.15.121:0/1695035753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f575409ec60 0x7f57540a0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:31.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.101+0000 7f5758925640 10 monclient: _finish_hunting 0 2024-09-17T18:25:31.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.101+0000 7f5758925640 1 monclient: found mon.noname-a 2024-09-17T18:25:31.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.101+0000 7f5758925640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:25:31.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.101+0000 7f5758925640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:25:31.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.101+0000 7f5758925640 1 -- 172.21.15.121:0/1695035753 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57541230a0 con 0x7f575409e810 2024-09-17T18:25:31.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.103+0000 7f5758925640 1 -- 172.21.15.121:0/1695035753 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f575409e810 msgr2=0x7f5754131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:31.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.103+0000 7f5758925640 1 --2- 172.21.15.121:0/1695035753 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f575409e810 0x7f5754131c60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5748009960 tx=0x7f57480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:31.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.103+0000 7f5751ffb640 10 client.?.objecter ms_handle_connect 0x7f575409e810 2024-09-17T18:25:31.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.103+0000 7f5751ffb640 10 client.?.objecter resend_mon_ops 2024-09-17T18:25:31.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.103+0000 7f5751ffb640 1 -- 172.21.15.121:0/1695035753 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5748037040 con 0x7f575409e810 2024-09-17T18:25:31.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.103+0000 7f5751ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:31.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.103+0000 7f5758925640 1 --2- 172.21.15.121:0/1695035753 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f575409e810 0x7f5754131c60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5748009960 tx=0x7f57480046c0 comp rx=0 tx=0).queue_received will delay after 1.7266e+09 on 0x7f574803bc00 mon_map magic: 0 2024-09-17T18:25:31.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.103+0000 7f5751ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:25:31.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.103+0000 7f5751ffb640 10 monclient: dump: 2024-09-17T18:25:31.140 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:31.141 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:31.141 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:31.141 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:31.141 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:31.141 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:31.141 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:31.141 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:31.142 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:31.142 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:31.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.103+0000 7f5751ffb640 10 monclient: _finish_auth 0 2024-09-17T18:25:31.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.103+0000 7f5751ffb640 10 monclient: _check_auth_tickets 2024-09-17T18:25:31.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.103+0000 7f5751ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:25:31.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.103+0000 7f5751ffb640 1 -- 172.21.15.121:0/1695035753 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5748042070 con 0x7f575409e810 2024-09-17T18:25:31.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.103+0000 7f575b38a640 5 monclient: authenticate success, global_id 44418 2024-09-17T18:25:31.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.103+0000 7f575b38a640 10 monclient: _renew_subs 2024-09-17T18:25:31.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.103+0000 7f575b38a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:31.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.103+0000 7f575b38a640 1 -- 172.21.15.121:0/1695035753 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5754121e60 con 0x7f575409e810 2024-09-17T18:25:31.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.103+0000 7f5751ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:25:31.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.103+0000 7f57537fe640 4 set_mon_vals no callback set 2024-09-17T18:25:31.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.103+0000 7f575b38a640 10 client.44418.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:25:31.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.103+0000 7f575b38a640 10 monclient: _renew_subs 2024-09-17T18:25:31.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.103+0000 7f575b38a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:25:31.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.103+0000 7f575b38a640 1 -- 172.21.15.121:0/1695035753 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57541347a0 con 0x7f575409e810 2024-09-17T18:25:31.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.103+0000 7f575b38a640 1 librados: init done 2024-09-17T18:25:31.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.104+0000 7f575b38a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:31.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.104+0000 7f575b38a640 1 --2- 172.21.15.121:0/1695035753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f57541321a0 0x7f57540a07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:31.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.104+0000 7f575b38a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:31.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.104+0000 7f575b38a640 1 -- 172.21.15.121:0/1695035753 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f57541235c0 con 0x7f57541321a0 2024-09-17T18:25:31.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.104+0000 7f5751ffb640 1 -- 172.21.15.121:0/1695035753 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f574803bc00 con 0x7f575409e810 2024-09-17T18:25:31.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.104+0000 7f5751ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:31.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.104+0000 7f5751ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:25:31.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.104+0000 7f5751ffb640 10 monclient: dump: 2024-09-17T18:25:31.145 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:31.145 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:31.146 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:31.146 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:31.146 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:31.146 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:31.146 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:31.146 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:31.146 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:31.147 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:31.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.104+0000 7f5751ffb640 1 -- 172.21.15.121:0/1695035753 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f5748047460 con 0x7f575409e810 2024-09-17T18:25:31.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.104+0000 7f5751ffb640 1 --2- 172.21.15.121:0/1695035753 >> v2:172.21.15.173:6812/2423786037 conn(0x7f5740040120 0x7f57400425e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:31.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.105+0000 7f5759126640 1 --2- 172.21.15.121:0/1695035753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f57541321a0 0x7f57540a07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:31.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.105+0000 7f5759126640 10 monclient: get_auth_request con 0x7f57541321a0 auth_method 0 2024-09-17T18:25:31.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.105+0000 7f5759126640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:31.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.105+0000 7f5759126640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:31.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.105+0000 7f5759126640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:31.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.105+0000 7f5759126640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:31.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.105+0000 7f5759126640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:31.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.105+0000 7f5759927640 1 --2- 172.21.15.121:0/1695035753 >> v2:172.21.15.173:6812/2423786037 conn(0x7f5740040120 0x7f57400425e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:31.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.105+0000 7f5759126640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:31.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.105+0000 7f5759927640 10 monclient: get_auth_request con 0x7f5740040120 auth_method 0 2024-09-17T18:25:31.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.105+0000 7f5759126640 10 monclient(hunting): handle_auth_done global_id 24733 payload 824 2024-09-17T18:25:31.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.105+0000 7f5759126640 1 -- 172.21.15.121:0/1695035753 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5754134e50 con 0x7f57541321a0 2024-09-17T18:25:31.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.107+0000 7f5759927640 1 --2- 172.21.15.121:0/1695035753 >> v2:172.21.15.173:6812/2423786037 conn(0x7f5740040120 0x7f57400425e0 secure :-1 s=READY pgs=3986 cs=0 l=1 rev1=1 crypto rx=0x7f5744003d00 tx=0x7f5744009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:31.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.107+0000 7f5751ffb640 10 client.44418.objecter ms_handle_connect 0x7f5740040120 2024-09-17T18:25:31.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.107+0000 7f5751ffb640 10 client.44418.objecter ms_handle_connect 0x7f57541321a0 2024-09-17T18:25:31.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.107+0000 7f5751ffb640 10 client.44418.objecter resend_mon_ops 2024-09-17T18:25:31.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.107+0000 7f5759126640 1 -- 172.21.15.121:0/1695035753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f57541321a0 msgr2=0x7f57540a07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:31.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.107+0000 7f5759126640 1 --2- 172.21.15.121:0/1695035753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f57541321a0 0x7f57540a07b0 secure :-1 s=READY pgs=7399 cs=0 l=1 rev1=1 crypto rx=0x7f574c00c4e0 tx=0x7f574c00c9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:31.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.107+0000 7f5751ffb640 1 -- 172.21.15.121:0/1695035753 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f574c015070 con 0x7f57541321a0 2024-09-17T18:25:31.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.107+0000 7f5751ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:31.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.107+0000 7f5751ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:25:31.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.107+0000 7f5751ffb640 10 monclient: dump: 2024-09-17T18:25:31.150 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:31.150 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:31.150 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:31.150 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:31.150 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:31.150 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:31.150 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:31.150 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:31.150 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:31.150 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:31.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.108+0000 7f5751ffb640 1 -- 172.21.15.121:0/1695035753 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f574c004220 con 0x7f57541321a0 2024-09-17T18:25:31.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.108+0000 7f5751ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:25:31.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.108+0000 7f5751ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:25:31.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.108+0000 7f5751ffb640 1 -- 172.21.15.121:0/1695035753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f57541321a0 msgr2=0x7f57540a07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:31.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.108+0000 7f5751ffb640 1 --2- 172.21.15.121:0/1695035753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f57541321a0 0x7f57540a07b0 secure :-1 s=READY pgs=7399 cs=0 l=1 rev1=1 crypto rx=0x7f574c00c4e0 tx=0x7f574c00c9a0 comp rx=0 tx=0).stop 2024-09-17T18:25:31.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.108+0000 7f5751ffb640 1 -- 172.21.15.121:0/1695035753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f57541321a0 msgr2=0x7f57540a07b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:25:31.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.108+0000 7f5751ffb640 1 --2- 172.21.15.121:0/1695035753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f57541321a0 0x7f57540a07b0 secure :-1 s=CLOSED pgs=7399 cs=0 l=1 rev1=1 crypto rx=0x7f574c00c4e0 tx=0x7f574c00c9a0 comp rx=0 tx=0).stop 2024-09-17T18:25:31.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.115+0000 7f575b38a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:25:31.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.115+0000 7f575b38a640 1 --2- 172.21.15.121:0/1695035753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f57540a0cf0 0x7f57540a23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:25:31.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.115+0000 7f575b38a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:25:31.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.115+0000 7f575b38a640 1 -- 172.21.15.121:0/1695035753 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f57540a10d0 con 0x7f57540a0cf0 2024-09-17T18:25:31.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.115+0000 7f5759126640 1 --2- 172.21.15.121:0/1695035753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f57540a0cf0 0x7f57540a23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:25:31.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.115+0000 7f5759126640 10 monclient: get_auth_request con 0x7f57540a0cf0 auth_method 0 2024-09-17T18:25:31.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.115+0000 7f5759126640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:25:31.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.115+0000 7f5759126640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:25:31.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.115+0000 7f5759126640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:25:31.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.115+0000 7f5759126640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:25:31.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.115+0000 7f5759126640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:25:31.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.115+0000 7f5759126640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:25:31.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.115+0000 7f5759126640 10 monclient(hunting): handle_auth_done global_id 24736 payload 824 2024-09-17T18:25:31.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.115+0000 7f5759126640 1 -- 172.21.15.121:0/1695035753 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f57540a1ce0 con 0x7f57540a0cf0 2024-09-17T18:25:31.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.118+0000 7f5759126640 1 -- 172.21.15.121:0/1695035753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f57540a0cf0 msgr2=0x7f57540a23f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:25:31.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.118+0000 7f5751ffb640 10 client.44418.objecter ms_handle_connect 0x7f57540a0cf0 2024-09-17T18:25:31.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.118+0000 7f5751ffb640 10 client.44418.objecter resend_mon_ops 2024-09-17T18:25:31.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.118+0000 7f5759126640 1 --2- 172.21.15.121:0/1695035753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f57540a0cf0 0x7f57540a23f0 secure :-1 s=READY pgs=7400 cs=0 l=1 rev1=1 crypto rx=0x7f574c016000 tx=0x7f574c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:25:31.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.118+0000 7f5751ffb640 1 -- 172.21.15.121:0/1695035753 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f574c003c30 con 0x7f57540a0cf0 2024-09-17T18:25:31.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.118+0000 7f5751ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:25:31.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.118+0000 7f5751ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:25:31.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.118+0000 7f5751ffb640 10 monclient: dump: 2024-09-17T18:25:31.153 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:25:31.154 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:25:31.154 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:31.154 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:25:31.154 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:25:31.154 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:25:31.154 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:25:31.154 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:25:31.154 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:25:31.154 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:25:31.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.118+0000 7f5751ffb640 1 -- 172.21.15.121:0/1695035753 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f574c003c30 con 0x7f57540a0cf0 2024-09-17T18:25:31.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.118+0000 7f5751ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:25:31.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.118+0000 7f5751ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:25:31.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.118+0000 7f5751ffb640 1 -- 172.21.15.121:0/1695035753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f57540a0cf0 msgr2=0x7f57540a23f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:31.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.118+0000 7f5751ffb640 1 --2- 172.21.15.121:0/1695035753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f57540a0cf0 0x7f57540a23f0 secure :-1 s=READY pgs=7400 cs=0 l=1 rev1=1 crypto rx=0x7f574c016000 tx=0x7f574c004730 comp rx=0 tx=0).stop 2024-09-17T18:25:31.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.118+0000 7f5751ffb640 1 -- 172.21.15.121:0/1695035753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f57540a0cf0 msgr2=0x7f57540a23f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:25:31.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.118+0000 7f5751ffb640 1 --2- 172.21.15.121:0/1695035753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f57540a0cf0 0x7f57540a23f0 unknown :-1 s=CLOSED pgs=7400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:25:31.155 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:25:31.155 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:25:31.155 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:25:31.155 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:25:31.155 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 30, 2024-09-17T18:25:31.155 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 404064, 2024-09-17T18:25:31.155 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:25:31.155 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:25:31.155 INFO:teuthology.orchestra.run.smithi121.stdout: 1 2024-09-17T18:25:31.155 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:25:31.155 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 0, 2024-09-17T18:25:31.155 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:25:31.156 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:25:31.156 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:25:31.156 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:25:31.156 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:25:31.156 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:25:31.156 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:25:31.156 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:25:31.156 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:25:31.156 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:25:31.156 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:25:31.156 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:25:31.156 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:25:31.156 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:25:31.156 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:25:31.157 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:25:31.157 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:25:31.157 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:25:31.157 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:25:31.157 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:25:31.157 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:25:31.157 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:25:31.157 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:25:31.157 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:25:31.157 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:25:31.157 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:25:31.157 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:25:31.157 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:25:31.157 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:31.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:31.158 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:25:31.158 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:25:31.158 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:25:31.158 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:25:31.158 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:25:31.158 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:25:31.158 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:25:31.158 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:25:31.158 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:25:31.158 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:25:31.158 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:25:31.158 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:25:31.158 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:25:31.158 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:25:31.158 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:25:31.159 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:25:31.159 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:25:31.159 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:25:31.159 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:25:31.159 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:25:31.159 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:25:31.159 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:25:31.159 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:25:31.159 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:25:31.159 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:25:31.159 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:25:31.159 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:25:31.159 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:25:31.159 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:25:31.159 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:25:31.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:31.160 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:25:31.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:31.160 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:25:31.160 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:25:31.160 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:25:31.160 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:25:31.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:31.160 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:25:31.160 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:25:31.160 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:31.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:31.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:31.160 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:25:31.161 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:25:31.161 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:31.161 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:31.161 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:31.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:31.161 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:25:31.161 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:25:31.161 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:25:31.161 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:25:31.161 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:25:31.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:31.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:31.161 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:25:31.161 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:25:31.161 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:25:31.162 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:25:31.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:31.162 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:25:31.162 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:25:31.162 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:31.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:31.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:31.162 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:25:31.162 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:25:31.162 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:31.162 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:31.162 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:31.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:31.162 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:25:31.162 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:25:31.162 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:25:31.163 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:25:31.163 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:25:31.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:31.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:31.163 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:25:31.163 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:25:31.163 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:25:31.163 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:25:31.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:31.163 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:25:31.163 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:25:31.163 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:31.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:31.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:31.164 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:25:31.164 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:25:31.164 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:25:31.164 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:31.164 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:31.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:31.164 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:25:31.164 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:25:31.164 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:25:31.164 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:25:31.164 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:25:31.164 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:31.164 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:31.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:31.164 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:25:31.165 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:25:31.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:31.165 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:25:31.165 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:25:31.165 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:25:31.165 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:31.165 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:25:31.165 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:25:31.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:31.165 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:25:31.165 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:25:31.165 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 4 2024-09-17T18:25:31.165 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:31.166 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:25:31.166 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:25:31.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:31.166 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:25:31.166 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:25:31.166 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 7 2024-09-17T18:25:31.166 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:31.166 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:25:31.166 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": [ 2024-09-17T18:25:31.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:25:31.166 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:25:31.166 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:25:31.166 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:25:31.166 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:25:31.166 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:25:31.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:25:31.167 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:25:31.167 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:25:31.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.118+0000 7f575b38a640 10 librados: watch_flush enter 2024-09-17T18:25:31.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.118+0000 7f575b38a640 10 librados: watch_flush exit 2024-09-17T18:25:31.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.118+0000 7f575b38a640 20 client.44418.objecter shutdown clearing up homeless session... 2024-09-17T18:25:31.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.118+0000 7f575b38a640 10 client.44418.objecter successfully canceled tick 2024-09-17T18:25:31.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.118+0000 7f575b38a640 1 -- 172.21.15.121:0/1695035753 >> v2:172.21.15.173:6812/2423786037 conn(0x7f5740040120 msgr2=0x7f57400425e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:31.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.118+0000 7f575b38a640 1 --2- 172.21.15.121:0/1695035753 >> v2:172.21.15.173:6812/2423786037 conn(0x7f5740040120 0x7f57400425e0 secure :-1 s=READY pgs=3986 cs=0 l=1 rev1=1 crypto rx=0x7f5744003d00 tx=0x7f5744009210 comp rx=0 tx=0).stop 2024-09-17T18:25:31.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.118+0000 7f575b38a640 10 monclient: shutdown 2024-09-17T18:25:31.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.118+0000 7f575b38a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:25:31.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.118+0000 7f575b38a640 1 -- 172.21.15.121:0/1695035753 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f575409e810 msgr2=0x7f5754131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:25:31.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.118+0000 7f575b38a640 1 --2- 172.21.15.121:0/1695035753 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f575409e810 0x7f5754131c60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5748009960 tx=0x7f57480046c0 comp rx=0 tx=0).stop 2024-09-17T18:25:31.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.118+0000 7f5759927640 1 -- 172.21.15.121:0/1695035753 reap_dead start 2024-09-17T18:25:31.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.118+0000 7f5751ffb640 1 -- 172.21.15.121:0/1695035753 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(288..288 src has 1..288) ==== 6231+0+0 (unknown 0 0 0) 0x7f57480743b0 con 0x7f575409e810 2024-09-17T18:25:31.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.118+0000 7f5751ffb640 10 client.44418.objecter ms_dispatch 0x7f5754000b90 osd_map(288..288 src has 1..288) 2024-09-17T18:25:31.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.118+0000 7f5751ffb640 10 librados: disconnected, discarding osd_map(288..288 src has 1..288) 2024-09-17T18:25:31.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.118+0000 7f5759927640 1 -- 172.21.15.121:0/1695035753 reap_dead start 2024-09-17T18:25:31.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.118+0000 7f575b38a640 1 -- 172.21.15.121:0/1695035753 shutdown_connections 2024-09-17T18:25:31.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.118+0000 7f575b38a640 1 -- 172.21.15.121:0/1695035753 >> 172.21.15.121:0/1695035753 conn(0x7f57540ab830 msgr2=0x7f5754133510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:25:31.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.118+0000 7f575b38a640 1 -- 172.21.15.121:0/1695035753 shutdown_connections 2024-09-17T18:25:31.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.118+0000 7f575b38a640 1 -- 172.21.15.121:0/1695035753 wait complete. 2024-09-17T18:25:31.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:25:31.118+0000 7f575b38a640 1 librados: shutdown 2024-09-17T18:25:31.170 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T18:25:31.170 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-17T18:25:49.051 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ 2024-09-17T18:25:49.051 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (2433 ms) 2024-09-17T18:25:49.052 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-09-17T18:25:49.052 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (6990 ms) 2024-09-17T18:25:49.052 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-09-17T18:25:49.052 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (7803 ms) 2024-09-17T18:25:49.052 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-09-17T18:25:49.052 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (6229 ms) 2024-09-17T18:25:49.052 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-09-17T18:25:49.052 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (21197 ms) 2024-09-17T18:25:49.052 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-09-17T18:25:49.052 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (6779 ms) 2024-09-17T18:25:49.052 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-09-17T18:25:49.052 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7664 ms) 2024-09-17T18:25:49.053 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (182918 ms total) 2024-09-17T18:25:49.053 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: 2024-09-17T18:25:49.053 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-09-17T18:25:49.053 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (348420 ms total) 2024-09-17T18:25:49.053 INFO:tasks.workunit.client.0.smithi173.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-09-17T18:27:01.170 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-17T18:27:01.171 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-17T18:27:01.171 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-17T18:27:01.175 INFO:tasks.ceph.mon.c:Started 2024-09-17T18:27:01.175 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T18:27:01.175 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:27:01.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.301+0000 7fc1766a9640 10 monclient: get_monmap_and_config 2024-09-17T18:27:01.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.301+0000 7fc1766a9640 10 monclient: build_initial_monmap 2024-09-17T18:27:01.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.301+0000 7fc1766a9640 10 monclient: monmap: 2024-09-17T18:27:01.303 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:27:01.303 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:01.303 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:27:01.301871+0000 2024-09-17T18:27:01.303 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:27:01.301871+0000 2024-09-17T18:27:01.303 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:27:01.303 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:01.303 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:27:01.303 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:27:01.304 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:27:01.304 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:27:01.304 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:27:01.304 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:01.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.302+0000 7fc1766a9640 1 Processor -- start 2024-09-17T18:27:01.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.302+0000 7fc1766a9640 1 -- start start 2024-09-17T18:27:01.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.302+0000 7fc1766a9640 10 monclient: init 2024-09-17T18:27:01.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.302+0000 7fc1766a9640 10 monclient: _reopen_session rank -1 2024-09-17T18:27:01.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.302+0000 7fc1766a9640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-17T18:27:01.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.302+0000 7fc1766a9640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc1700a5c00 0x7fc17009d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:01.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.302+0000 7fc1766a9640 10 monclient(hunting): picked mon.noname-a con 0x7fc1700a5c00 addr v2:172.21.15.121:3300/0 2024-09-17T18:27:01.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.302+0000 7fc1766a9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc1700a5fe0 addr v1:172.21.15.173:6789/0 2024-09-17T18:27:01.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.302+0000 7fc1766a9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc17013dc00 addr v1:172.21.15.121:6789/0 2024-09-17T18:27:01.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.302+0000 7fc1766a9640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1700a8a80 con 0x7fc17013dc00 2024-09-17T18:27:01.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.302+0000 7fc1766a9640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc17009de00 con 0x7fc1700a5fe0 2024-09-17T18:27:01.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.302+0000 7fc1766a9640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:01.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.302+0000 7fc1766a9640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fc1700a7110 con 0x7fc1700a5c00 2024-09-17T18:27:01.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.302+0000 7fc1766a9640 10 monclient(hunting): _renew_subs 2024-09-17T18:27:01.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.302+0000 7fc1766a9640 10 monclient(hunting): authenticate will time out at 1225.018669s 2024-09-17T18:27:01.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.302+0000 7fc16ffff640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc1700a5c00 0x7fc17009d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:01.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.302+0000 7fc174c46640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fc17013dc00 0x7fc17013fff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:39712/0 (socket says 172.21.15.121:39712) 2024-09-17T18:27:01.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.302+0000 7fc174c46640 1 -- 172.21.15.121:0/3795698152 learned_addr learned my addr 172.21.15.121:0/3795698152 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:27:01.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.302+0000 7fc16ffff640 10 monclient(hunting): get_auth_request con 0x7fc1700a5c00 auth_method 0 2024-09-17T18:27:01.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.302+0000 7fc16ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:01.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.302+0000 7fc16ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:01.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.302+0000 7fc16ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:01.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.302+0000 7fc16ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:01.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.302+0000 7fc16ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:01.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.302+0000 7fc16ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:01.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.303+0000 7fc16ffff640 10 monclient(hunting): handle_auth_done global_id 44433 payload 274 2024-09-17T18:27:01.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.303+0000 7fc16ffff640 1 -- 172.21.15.121:0/3795698152 >> v1:172.21.15.173:6789/0 conn(0x7fc1700a5fe0 legacy=0x7fc1700a63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:27:01.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.303+0000 7fc16ffff640 1 -- 172.21.15.121:0/3795698152 >> v1:172.21.15.121:6789/0 conn(0x7fc17013dc00 legacy=0x7fc17013fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:27:01.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.303+0000 7fc16ffff640 10 monclient: _finish_hunting 0 2024-09-17T18:27:01.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.303+0000 7fc16ffff640 1 monclient: found mon.noname-a 2024-09-17T18:27:01.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.303+0000 7fc16ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:27:01.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.303+0000 7fc16ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:27:01.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.303+0000 7fc16ffff640 1 -- 172.21.15.121:0/3795698152 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1700a74d0 con 0x7fc1700a5c00 2024-09-17T18:27:01.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.305+0000 7fc16ffff640 1 -- 172.21.15.121:0/3795698152 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1700a5c00 msgr2=0x7fc17009d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:01.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.305+0000 7fc16ffff640 1 --2- 172.21.15.121:0/3795698152 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1700a5c00 0x7fc17009d8c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc17009dfd0 tx=0x7fc160029fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=820adf0ea74b36a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:01.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.305+0000 7fc16effd640 1 -- 172.21.15.121:0/3795698152 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc160037070 con 0x7fc1700a5c00 2024-09-17T18:27:01.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.305+0000 7fc16effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:01.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.305+0000 7fc16effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:27:01.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.305+0000 7fc16effd640 10 monclient: dump: 2024-09-17T18:27:01.315 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:01.315 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:01.315 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:01.315 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:01.315 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:01.315 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:01.315 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:01.315 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:01.315 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:01.315 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:01.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.305+0000 7fc16effd640 10 monclient: _finish_auth 0 2024-09-17T18:27:01.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.305+0000 7fc16effd640 10 monclient: _check_auth_tickets 2024-09-17T18:27:01.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.305+0000 7fc16effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:27:01.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.305+0000 7fc1766a9640 5 monclient: authenticate success, global_id 44433 2024-09-17T18:27:01.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.305+0000 7fc16effd640 1 -- 172.21.15.121:0/3795698152 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc160032440 con 0x7fc1700a5c00 2024-09-17T18:27:01.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.305+0000 7fc1766a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:27:01.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.306+0000 7fc16effd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:27:01.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.306+0000 7fc1766a9640 10 monclient: get_monmap_and_config success 2024-09-17T18:27:01.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.306+0000 7fc16effd640 1 -- 172.21.15.121:0/3795698152 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc160032810 con 0x7fc1700a5c00 2024-09-17T18:27:01.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.306+0000 7fc1766a9640 4 set_mon_vals no callback set 2024-09-17T18:27:01.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.306+0000 7fc16effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:01.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.306+0000 7fc16effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:27:01.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.306+0000 7fc16effd640 10 monclient: dump: 2024-09-17T18:27:01.316 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:01.316 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:01.316 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:01.316 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:01.316 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:01.316 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:01.316 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:01.317 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:01.317 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:01.317 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:01.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.306+0000 7fc1766a9640 10 monclient: shutdown 2024-09-17T18:27:01.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.306+0000 7fc1766a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:27:01.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.306+0000 7fc1766a9640 1 -- 172.21.15.121:0/3795698152 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1700a5c00 msgr2=0x7fc17009d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:01.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.306+0000 7fc1766a9640 1 --2- 172.21.15.121:0/3795698152 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1700a5c00 0x7fc17009d8c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc17009dfd0 tx=0x7fc160029fb0 comp rx=0 tx=0).stop 2024-09-17T18:27:01.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.306+0000 7fc1766a9640 1 -- 172.21.15.121:0/3795698152 shutdown_connections 2024-09-17T18:27:01.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.306+0000 7fc1766a9640 1 --2- 172.21.15.121:0/3795698152 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1700a5c00 0x7fc17009d8c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:01.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.306+0000 7fc1766a9640 1 -- 172.21.15.121:0/3795698152 >> 172.21.15.121:0/3795698152 conn(0x7fc1700ab9b0 msgr2=0x7fc1700a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:27:01.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.306+0000 7fc1766a9640 1 -- 172.21.15.121:0/3795698152 shutdown_connections 2024-09-17T18:27:01.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.306+0000 7fc1766a9640 1 -- 172.21.15.121:0/3795698152 wait complete. 2024-09-17T18:27:01.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.307+0000 7fc1766a9640 10 monclient: build_initial_monmap 2024-09-17T18:27:01.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.307+0000 7fc1766a9640 10 monclient: monmap: 2024-09-17T18:27:01.317 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:27:01.318 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:27:01.318 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:27:01.318 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:27:01.318 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:27:01.318 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:01.318 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:27:01.318 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:27:01.318 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:27:01.318 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:01.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.307+0000 7fc1766a9640 1 librados: starting msgr at 2024-09-17T18:27:01.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.307+0000 7fc1766a9640 1 librados: starting objecter 2024-09-17T18:27:01.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.307+0000 7fc1766a9640 1 Processor -- start 2024-09-17T18:27:01.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.307+0000 7fc1766a9640 1 -- start start 2024-09-17T18:27:01.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.307+0000 7fc1766a9640 1 librados: setting wanted keys 2024-09-17T18:27:01.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.307+0000 7fc1766a9640 1 librados: calling monclient init 2024-09-17T18:27:01.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.307+0000 7fc1766a9640 10 monclient: init 2024-09-17T18:27:01.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc1766a9640 10 monclient: _reopen_session rank -1 2024-09-17T18:27:01.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc1766a9640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T18:27:01.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc1766a9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1700a5c00 0x7fc17013cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:01.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc1766a9640 10 monclient(hunting): picked mon.noname-c con 0x7fc1700a5c00 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:27:01.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc1766a9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1700a5fe0 0x7fc1701cdcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:01.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc1766a9640 10 monclient(hunting): picked mon.noname-a con 0x7fc1700a5fe0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:27:01.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc1766a9640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc17013dc00 0x7fc17013a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:01.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc1766a9640 10 monclient(hunting): picked mon.noname-b con 0x7fc17013dc00 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:27:01.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc1766a9640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:01.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc1766a9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc1700703d0 con 0x7fc1700a5fe0 2024-09-17T18:27:01.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc1766a9640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:01.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc1766a9640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc17009ce70 con 0x7fc17013dc00 2024-09-17T18:27:01.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc1766a9640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:01.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc1766a9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc170070550 con 0x7fc1700a5c00 2024-09-17T18:27:01.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc1766a9640 10 monclient(hunting): _renew_subs 2024-09-17T18:27:01.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc1766a9640 10 monclient(hunting): authenticate will time out at 1225.024271s 2024-09-17T18:27:01.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc16ffff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1700a5c00 0x7fc17013cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:01.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc16f7fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1700a5fe0 0x7fc1701cdcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:01.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc16ffff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1700a5c00 0x7fc17013cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:35528/0 (socket says 172.21.15.121:35528) 2024-09-17T18:27:01.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc16f7fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1700a5fe0 0x7fc1701cdcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41204/0 (socket says 172.21.15.121:41204) 2024-09-17T18:27:01.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc16ffff640 1 -- 172.21.15.121:0/3221241390 learned_addr learned my addr 172.21.15.121:0/3221241390 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:27:01.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc16ffff640 10 monclient(hunting): get_auth_request con 0x7fc1700a5c00 auth_method 0 2024-09-17T18:27:01.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc16ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:01.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc16ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:01.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc16ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:01.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc16f7fe640 10 monclient(hunting): get_auth_request con 0x7fc1700a5fe0 auth_method 0 2024-09-17T18:27:01.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc16f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:01.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc16f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:01.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc16f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:01.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc16ffff640 1 -- 172.21.15.121:0/3221241390 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1700a5c00 msgr2=0x7fc17013cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T18:27:01.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc16ffff640 1 -- 172.21.15.121:0/3221241390 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1700a5c00 msgr2=0x7fc17013cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:27:01.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc16ffff640 1 --2- 172.21.15.121:0/3221241390 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1700a5c00 0x7fc17013cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:27:01.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc16f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:01.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc16f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:01.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc16f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:01.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc16ffff640 1 --2- 172.21.15.121:0/3221241390 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1700a5c00 0x7fc17013cd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:27:01.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc174c46640 1 --2- 172.21.15.121:0/3221241390 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc17013dc00 0x7fc17013a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:01.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc174c46640 10 monclient(hunting): get_auth_request con 0x7fc17013dc00 auth_method 0 2024-09-17T18:27:01.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc174c46640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:01.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc174c46640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:01.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc174c46640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:01.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc16f7fe640 10 monclient(hunting): handle_auth_done global_id 44436 payload 824 2024-09-17T18:27:01.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc16f7fe640 1 -- 172.21.15.121:0/3221241390 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1700a5c00 msgr2=0x7fc17013cd00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:27:01.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc16f7fe640 1 --2- 172.21.15.121:0/3221241390 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1700a5c00 0x7fc17013cd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:01.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc16f7fe640 1 -- 172.21.15.121:0/3221241390 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc17013dc00 msgr2=0x7fc17013a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:01.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc16f7fe640 1 --2- 172.21.15.121:0/3221241390 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc17013dc00 0x7fc17013a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:01.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc16f7fe640 10 monclient: _finish_hunting 0 2024-09-17T18:27:01.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc16f7fe640 1 monclient: found mon.noname-a 2024-09-17T18:27:01.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc16f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:27:01.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc16f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:27:01.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.308+0000 7fc16f7fe640 1 -- 172.21.15.121:0/3221241390 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1700a23f0 con 0x7fc1700a5fe0 2024-09-17T18:27:01.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.310+0000 7fc16f7fe640 1 -- 172.21.15.121:0/3221241390 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1700a5fe0 msgr2=0x7fc1701cdcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:01.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.310+0000 7fc16f7fe640 1 --2- 172.21.15.121:0/3221241390 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1700a5fe0 0x7fc1701cdcc0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc16400c410 tx=0x7fc16400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:01.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.310+0000 7fc16cff9640 10 client.?.objecter ms_handle_connect 0x7fc1700a5fe0 2024-09-17T18:27:01.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.310+0000 7fc16cff9640 10 client.?.objecter resend_mon_ops 2024-09-17T18:27:01.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.311+0000 7fc16cff9640 1 -- 172.21.15.121:0/3221241390 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc164017070 con 0x7fc1700a5fe0 2024-09-17T18:27:01.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.311+0000 7fc16cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:01.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.311+0000 7fc16cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:27:01.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.311+0000 7fc16cff9640 10 monclient: dump: 2024-09-17T18:27:01.322 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:01.322 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:01.323 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:01.323 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:01.323 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:01.323 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:01.323 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:01.323 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:01.323 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:01.323 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:01.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.311+0000 7fc16cff9640 10 monclient: _finish_auth 0 2024-09-17T18:27:01.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.311+0000 7fc16cff9640 10 monclient: _check_auth_tickets 2024-09-17T18:27:01.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.311+0000 7fc16cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:27:01.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.311+0000 7fc16cff9640 1 -- 172.21.15.121:0/3221241390 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc164005230 con 0x7fc1700a5fe0 2024-09-17T18:27:01.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.311+0000 7fc1766a9640 5 monclient: authenticate success, global_id 44436 2024-09-17T18:27:01.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.311+0000 7fc1766a9640 10 monclient: _renew_subs 2024-09-17T18:27:01.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.311+0000 7fc1766a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:27:01.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.311+0000 7fc1766a9640 1 -- 172.21.15.121:0/3221241390 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc170121e70 con 0x7fc1700a5fe0 2024-09-17T18:27:01.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.311+0000 7fc16cff9640 10 monclient: handle_config config(0 keys) 2024-09-17T18:27:01.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.311+0000 7fc16cff9640 1 -- 172.21.15.121:0/3221241390 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1640117a0 con 0x7fc1700a5fe0 2024-09-17T18:27:01.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.311+0000 7fc16cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:01.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.311+0000 7fc16cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:27:01.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.311+0000 7fc16cff9640 10 monclient: dump: 2024-09-17T18:27:01.324 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:01.324 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:01.324 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:01.324 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:01.324 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:01.324 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:01.324 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:01.325 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:01.325 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:01.325 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:01.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.311+0000 7fc1766a9640 10 client.44436.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:27:01.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.311+0000 7fc1766a9640 10 monclient: _renew_subs 2024-09-17T18:27:01.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.311+0000 7fc16e7fc640 4 set_mon_vals no callback set 2024-09-17T18:27:01.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.311+0000 7fc1766a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:27:01.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.311+0000 7fc1766a9640 1 -- 172.21.15.121:0/3221241390 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1701411a0 con 0x7fc1700a5fe0 2024-09-17T18:27:01.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.311+0000 7fc1766a9640 1 librados: init done 2024-09-17T18:27:01.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.311+0000 7fc1766a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:27:01.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.311+0000 7fc1766a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:27:01.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.311+0000 7fc1766a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:27:01.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.312+0000 7fc16cff9640 1 -- 172.21.15.121:0/3221241390 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fc16401a070 con 0x7fc1700a5fe0 2024-09-17T18:27:01.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.311+0000 7fc1766a9640 1 -- 172.21.15.121:0/3221241390 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc17009d8c0 con 0x7fc1700a5fe0 2024-09-17T18:27:01.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.312+0000 7fc16cff9640 1 --2- 172.21.15.121:0/3221241390 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc15c0401f0 0x7fc15c0426b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:01.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.312+0000 7fc16cff9640 1 -- 172.21.15.121:0/3221241390 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(341..341 src has 177..341) ==== 5358+0+0 (secure 0 0 0) 0x7fc164052910 con 0x7fc1700a5fe0 2024-09-17T18:27:01.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.312+0000 7fc16cff9640 10 client.44436.objecter ms_dispatch 0x7fc170000b90 osd_map(341..341 src has 177..341) 2024-09-17T18:27:01.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.312+0000 7fc16cff9640 3 client.44436.objecter handle_osd_map got epochs [341,341] > 0 2024-09-17T18:27:01.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.312+0000 7fc16cff9640 3 client.44436.objecter handle_osd_map decoding full epoch 341 2024-09-17T18:27:01.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.312+0000 7fc16cff9640 20 client.44436.objecter dump_active .. 0 homeless 2024-09-17T18:27:01.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.312+0000 7fc16ffff640 1 --2- 172.21.15.121:0/3221241390 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc15c0401f0 0x7fc15c0426b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:01.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.312+0000 7fc16ffff640 10 monclient: get_auth_request con 0x7fc15c0401f0 auth_method 0 2024-09-17T18:27:01.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.315+0000 7fc16ffff640 1 --2- 172.21.15.121:0/3221241390 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc15c0401f0 0x7fc15c0426b0 secure :-1 s=READY pgs=3990 cs=0 l=1 rev1=1 crypto rx=0x7fc160009560 tx=0x7fc16002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:01.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.315+0000 7fc16cff9640 10 client.44436.objecter ms_handle_connect 0x7fc15c0401f0 2024-09-17T18:27:01.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.317+0000 7fc16cff9640 1 -- 172.21.15.121:0/3221241390 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc1700a23f0 con 0x7fc1700a5fe0 2024-09-17T18:27:01.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.317+0000 7fc16cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:27:01.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.317+0000 7fc16cff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:27:01.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.566+0000 7fc1766a9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:27:01.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.566+0000 7fc1766a9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:27:01.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.566+0000 7fc1766a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:27:01.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.566+0000 7fc1766a9640 1 -- 172.21.15.121:0/3221241390 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc170138e00 con 0x7fc1700a5fe0 2024-09-17T18:27:01.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.566+0000 7fc16cff9640 1 -- 172.21.15.121:0/3221241390 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7fc1640102e0 con 0x7fc1700a5fe0 2024-09-17T18:27:01.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.566+0000 7fc16cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:27:01.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.566+0000 7fc16cff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:27:01.568 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":30,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":91,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:27:01.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.568+0000 7fc1766a9640 10 librados: watch_flush enter 2024-09-17T18:27:01.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.568+0000 7fc1766a9640 10 librados: watch_flush exit 2024-09-17T18:27:01.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.568+0000 7fc1766a9640 20 client.44436.objecter shutdown clearing up homeless session... 2024-09-17T18:27:01.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.568+0000 7fc1766a9640 10 client.44436.objecter successfully canceled tick 2024-09-17T18:27:01.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.568+0000 7fc1766a9640 1 -- 172.21.15.121:0/3221241390 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc15c0401f0 msgr2=0x7fc15c0426b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:01.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.568+0000 7fc1766a9640 1 --2- 172.21.15.121:0/3221241390 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc15c0401f0 0x7fc15c0426b0 secure :-1 s=READY pgs=3990 cs=0 l=1 rev1=1 crypto rx=0x7fc160009560 tx=0x7fc16002e000 comp rx=0 tx=0).stop 2024-09-17T18:27:01.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.568+0000 7fc1766a9640 10 monclient: shutdown 2024-09-17T18:27:01.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.568+0000 7fc1766a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:27:01.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.568+0000 7fc1766a9640 1 -- 172.21.15.121:0/3221241390 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1700a5fe0 msgr2=0x7fc1701cdcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:01.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.568+0000 7fc1766a9640 1 --2- 172.21.15.121:0/3221241390 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1700a5fe0 0x7fc1701cdcc0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc16400c410 tx=0x7fc16400c8d0 comp rx=0 tx=0).stop 2024-09-17T18:27:01.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.569+0000 7fc1766a9640 1 -- 172.21.15.121:0/3221241390 shutdown_connections 2024-09-17T18:27:01.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.569+0000 7fc1766a9640 1 --2- 172.21.15.121:0/3221241390 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc15c0401f0 0x7fc15c0426b0 unknown :-1 s=CLOSED pgs=3990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:01.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.569+0000 7fc1766a9640 1 --2- 172.21.15.121:0/3221241390 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc17013dc00 0x7fc17013a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:01.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.569+0000 7fc1766a9640 1 --2- 172.21.15.121:0/3221241390 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1700a5fe0 0x7fc1701cdcc0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:01.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.569+0000 7fc1766a9640 1 --2- 172.21.15.121:0/3221241390 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1700a5c00 0x7fc17013cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:01.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.569+0000 7fc1766a9640 1 -- 172.21.15.121:0/3221241390 >> 172.21.15.121:0/3221241390 conn(0x7fc1700ab9b0 msgr2=0x7fc1700a99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:27:01.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.569+0000 7fc1766a9640 1 -- 172.21.15.121:0/3221241390 shutdown_connections 2024-09-17T18:27:01.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.569+0000 7fc1766a9640 1 -- 172.21.15.121:0/3221241390 wait complete. 2024-09-17T18:27:01.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:01.569+0000 7fc1766a9640 1 librados: shutdown 2024-09-17T18:27:04.582 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:27:04.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.691+0000 7f712c9d0640 10 monclient: get_monmap_and_config 2024-09-17T18:27:04.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.691+0000 7f712c9d0640 10 monclient: build_initial_monmap 2024-09-17T18:27:04.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.691+0000 7f712c9d0640 10 monclient: monmap: 2024-09-17T18:27:04.694 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:27:04.694 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:04.694 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:27:04.691962+0000 2024-09-17T18:27:04.694 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:27:04.691962+0000 2024-09-17T18:27:04.694 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:27:04.695 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:04.695 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:27:04.695 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:27:04.695 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:27:04.695 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:27:04.695 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:27:04.695 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:04.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.691+0000 7f712c9d0640 1 Processor -- start 2024-09-17T18:27:04.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.691+0000 7f712c9d0640 1 -- start start 2024-09-17T18:27:04.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.691+0000 7f712c9d0640 10 monclient: init 2024-09-17T18:27:04.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.692+0000 7f712c9d0640 10 monclient: _reopen_session rank -1 2024-09-17T18:27:04.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.692+0000 7f712c9d0640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-17T18:27:04.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.692+0000 7f712c9d0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f71280a33f0 addr v1:172.21.15.173:6789/0 2024-09-17T18:27:04.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.692+0000 7f712c9d0640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f712809da70 0x7f712809ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:04.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.692+0000 7f712c9d0640 10 monclient(hunting): picked mon.noname-c con 0x7f712809da70 addr v2:172.21.15.173:3300/0 2024-09-17T18:27:04.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.692+0000 7f712c9d0640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f712813a850 0x7f712813cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:04.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.692+0000 7f712c9d0640 10 monclient(hunting): picked mon.noname-b con 0x7f712813a850 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:27:04.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.692+0000 7f712c9d0640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7128074b40 con 0x7f71280a33f0 2024-09-17T18:27:04.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.692+0000 7f712c9d0640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:04.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.692+0000 7f712c9d0640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f7128074550 con 0x7f712809da70 2024-09-17T18:27:04.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.692+0000 7f712c9d0640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:04.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.692+0000 7f712c9d0640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f71280a37d0 con 0x7f712813a850 2024-09-17T18:27:04.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.692+0000 7f712c9d0640 10 monclient(hunting): _renew_subs 2024-09-17T18:27:04.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.692+0000 7f712c9d0640 10 monclient(hunting): authenticate will time out at 1228.408656s 2024-09-17T18:27:04.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.692+0000 7f7126d9d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f712813a850 0x7f712813cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:04.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.692+0000 7f7126d9d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f712813a850 0x7f712813cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:35530/0 (socket says 172.21.15.121:35530) 2024-09-17T18:27:04.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.692+0000 7f7126d9d640 1 -- 172.21.15.121:0/3622356132 learned_addr learned my addr 172.21.15.121:0/3622356132 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:27:04.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.692+0000 7f7126d9d640 10 monclient(hunting): get_auth_request con 0x7f712813a850 auth_method 0 2024-09-17T18:27:04.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.692+0000 7f7126d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:04.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.692+0000 7f7126d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:04.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.692+0000 7f7126d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:04.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.692+0000 7f7126d9d640 1 -- 172.21.15.121:0/3622356132 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f712813a850 msgr2=0x7f712813cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T18:27:04.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.692+0000 7f7126d9d640 1 -- 172.21.15.121:0/3622356132 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f712813a850 msgr2=0x7f712813cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:27:04.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.692+0000 7f7126d9d640 1 --2- 172.21.15.121:0/3622356132 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f712813a850 0x7f712813cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:27:04.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.692+0000 7f7126d9d640 1 --2- 172.21.15.121:0/3622356132 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f712813a850 0x7f712813cc40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:27:04.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.692+0000 7f7125d9b640 1 --2- 172.21.15.121:0/3622356132 >> v2:172.21.15.173:3300/0 conn(0x7f712809da70 0x7f712809ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:04.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.693+0000 7f7125d9b640 10 monclient(hunting): get_auth_request con 0x7f712809da70 auth_method 0 2024-09-17T18:27:04.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.693+0000 7f7125d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:04.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.693+0000 7f7125d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:04.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.693+0000 7f7125d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:04.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.693+0000 7f7125d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:04.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.693+0000 7f7125d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:04.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.693+0000 7f7125d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:04.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.693+0000 7f7125d9b640 10 monclient(hunting): handle_auth_done global_id 24754 payload 274 2024-09-17T18:27:04.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.693+0000 7f7125d9b640 1 -- 172.21.15.121:0/3622356132 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f712813a850 msgr2=0x7f712813cc40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:27:04.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.693+0000 7f7125d9b640 1 --2- 172.21.15.121:0/3622356132 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f712813a850 0x7f712813cc40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:04.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.693+0000 7f7125d9b640 1 -- 172.21.15.121:0/3622356132 >> v1:172.21.15.173:6789/0 conn(0x7f71280a33f0 legacy=0x7f712809e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:27:04.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.693+0000 7f7125d9b640 10 monclient: _finish_hunting 0 2024-09-17T18:27:04.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.693+0000 7f7125d9b640 1 monclient: found mon.noname-c 2024-09-17T18:27:04.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.693+0000 7f7125d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:27:04.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.693+0000 7f7125d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.173:3300/0 2024-09-17T18:27:04.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.693+0000 7f7125d9b640 1 -- 172.21.15.121:0/3622356132 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71280a13c0 con 0x7f712809da70 2024-09-17T18:27:04.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.696+0000 7f7125d9b640 1 -- 172.21.15.121:0/3622356132 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f712809da70 msgr2=0x7f712809ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:04.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.696+0000 7f7125d9b640 1 --2- 172.21.15.121:0/3622356132 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f712809da70 0x7f712809ec60 secure :-1 s=READY pgs=7421 cs=0 l=1 rev1=1 crypto rx=0x7f711c009dc0 tx=0x7f711c02a4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e6b339ee66912e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:04.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.696+0000 7f712559a640 1 -- 172.21.15.121:0/3622356132 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f711c007950 con 0x7f712809da70 2024-09-17T18:27:04.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.696+0000 7f712559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:04.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.696+0000 7f712559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:27:04.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.696+0000 7f712559a640 10 monclient: dump: 2024-09-17T18:27:04.713 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:04.713 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:04.713 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:04.713 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:04.713 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:04.714 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:04.714 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:04.714 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:04.714 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:04.714 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:04.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.696+0000 7f712559a640 10 monclient: _finish_auth 0 2024-09-17T18:27:04.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.696+0000 7f712559a640 10 monclient: _check_auth_tickets 2024-09-17T18:27:04.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.696+0000 7f712559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:27:04.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.696+0000 7f712559a640 1 -- 172.21.15.121:0/3622356132 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f711c007af0 con 0x7f712809da70 2024-09-17T18:27:04.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.696+0000 7f712559a640 10 monclient: handle_config config(0 keys) 2024-09-17T18:27:04.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.696+0000 7f712559a640 1 -- 172.21.15.121:0/3622356132 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f711c031400 con 0x7f712809da70 2024-09-17T18:27:04.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.696+0000 7f712559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:04.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.696+0000 7f712559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:27:04.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.696+0000 7f712559a640 10 monclient: dump: 2024-09-17T18:27:04.715 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:04.715 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:04.715 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:04.715 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:04.715 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:04.715 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:04.715 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:04.715 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:04.715 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:04.715 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:04.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.696+0000 7f712c9d0640 5 monclient: authenticate success, global_id 24754 2024-09-17T18:27:04.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.696+0000 7f712c9d0640 10 monclient: get_monmap_and_config success 2024-09-17T18:27:04.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.696+0000 7f712c9d0640 4 set_mon_vals no callback set 2024-09-17T18:27:04.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.697+0000 7f712c9d0640 10 monclient: shutdown 2024-09-17T18:27:04.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.697+0000 7f712c9d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:27:04.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.697+0000 7f712c9d0640 1 -- 172.21.15.121:0/3622356132 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f712809da70 msgr2=0x7f712809ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:04.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.697+0000 7f712c9d0640 1 --2- 172.21.15.121:0/3622356132 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f712809da70 0x7f712809ec60 secure :-1 s=READY pgs=7421 cs=0 l=1 rev1=1 crypto rx=0x7f711c009dc0 tx=0x7f711c02a4d0 comp rx=0 tx=0).stop 2024-09-17T18:27:04.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.697+0000 7f712c9d0640 1 -- 172.21.15.121:0/3622356132 shutdown_connections 2024-09-17T18:27:04.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.697+0000 7f712c9d0640 1 --2- 172.21.15.121:0/3622356132 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f712813a850 0x7f712813cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:04.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.697+0000 7f712c9d0640 1 --2- 172.21.15.121:0/3622356132 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f712809da70 0x7f712809ec60 unknown :-1 s=CLOSED pgs=7421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:04.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.697+0000 7f712c9d0640 1 -- 172.21.15.121:0/3622356132 >> 172.21.15.121:0/3622356132 conn(0x7f71280a3b50 msgr2=0x7f71280a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:27:04.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.697+0000 7f712c9d0640 1 -- 172.21.15.121:0/3622356132 shutdown_connections 2024-09-17T18:27:04.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.697+0000 7f712c9d0640 1 -- 172.21.15.121:0/3622356132 wait complete. 2024-09-17T18:27:04.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.698+0000 7f712c9d0640 10 monclient: build_initial_monmap 2024-09-17T18:27:04.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.698+0000 7f712c9d0640 10 monclient: monmap: 2024-09-17T18:27:04.716 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:27:04.716 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:27:04.717 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:27:04.717 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:27:04.717 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:27:04.717 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:04.717 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:27:04.717 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:27:04.717 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:27:04.717 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:04.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.698+0000 7f712c9d0640 1 librados: starting msgr at 2024-09-17T18:27:04.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.698+0000 7f712c9d0640 1 librados: starting objecter 2024-09-17T18:27:04.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.698+0000 7f712c9d0640 1 Processor -- start 2024-09-17T18:27:04.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.698+0000 7f712c9d0640 1 -- start start 2024-09-17T18:27:04.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.698+0000 7f712c9d0640 1 librados: setting wanted keys 2024-09-17T18:27:04.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.698+0000 7f712c9d0640 1 librados: calling monclient init 2024-09-17T18:27:04.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.698+0000 7f712c9d0640 10 monclient: init 2024-09-17T18:27:04.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f712c9d0640 10 monclient: _reopen_session rank -1 2024-09-17T18:27:04.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f712c9d0640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:27:04.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f712c9d0640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71280a33f0 0x7f71281c9c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:04.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f712c9d0640 10 monclient(hunting): picked mon.noname-a con 0x7f71280a33f0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:27:04.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f712c9d0640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f712813a850 0x7f71281ca190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:04.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f712c9d0640 10 monclient(hunting): picked mon.noname-b con 0x7f712813a850 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:27:04.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f712c9d0640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7128072410 0x7f7128072830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:04.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f712c9d0640 10 monclient(hunting): picked mon.noname-c con 0x7f7128072410 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:27:04.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f712c9d0640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:04.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f712c9d0640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f712809f320 con 0x7f71280a33f0 2024-09-17T18:27:04.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f712c9d0640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:04.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f712c9d0640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f712809f1a0 con 0x7f712813a850 2024-09-17T18:27:04.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f712c9d0640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:04.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f712c9d0640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f71280a22e0 con 0x7f7128072410 2024-09-17T18:27:04.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f712c9d0640 10 monclient(hunting): _renew_subs 2024-09-17T18:27:04.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f712c9d0640 10 monclient(hunting): authenticate will time out at 1228.415359s 2024-09-17T18:27:04.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f712659c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71280a33f0 0x7f71281c9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:04.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f7126d9d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7128072410 0x7f7128072830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:04.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f712659c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71280a33f0 0x7f71281c9c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41234/0 (socket says 172.21.15.121:41234) 2024-09-17T18:27:04.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f712659c640 1 -- 172.21.15.121:0/3938381892 learned_addr learned my addr 172.21.15.121:0/3938381892 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:27:04.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f7126d9d640 10 monclient(hunting): get_auth_request con 0x7f7128072410 auth_method 0 2024-09-17T18:27:04.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f7126d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:04.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f7126d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:04.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f7126d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:04.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f712659c640 10 monclient(hunting): get_auth_request con 0x7f71280a33f0 auth_method 0 2024-09-17T18:27:04.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f712659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:04.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f712659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:04.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f712659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:04.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f7125d9b640 1 --2- 172.21.15.121:0/3938381892 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f712813a850 0x7f71281ca190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:04.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f7125d9b640 10 monclient(hunting): get_auth_request con 0x7f712813a850 auth_method 0 2024-09-17T18:27:04.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f7125d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:04.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f7125d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:04.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f7125d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:04.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f7126d9d640 1 -- 172.21.15.121:0/3938381892 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7128072410 msgr2=0x7f7128072830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-17T18:27:04.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f7126d9d640 1 -- 172.21.15.121:0/3938381892 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7128072410 msgr2=0x7f7128072830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:27:04.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f7126d9d640 1 --2- 172.21.15.121:0/3938381892 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7128072410 0x7f7128072830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:27:04.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f712659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:04.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f712659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:04.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f712659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:04.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f7126d9d640 1 --2- 172.21.15.121:0/3938381892 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7128072410 0x7f7128072830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:27:04.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f7125d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:04.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f7125d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:04.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f7125d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:04.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f712659c640 10 monclient(hunting): handle_auth_done global_id 44439 payload 824 2024-09-17T18:27:04.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f712659c640 1 -- 172.21.15.121:0/3938381892 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7128072410 msgr2=0x7f7128072830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:27:04.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f712659c640 1 --2- 172.21.15.121:0/3938381892 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7128072410 0x7f7128072830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:04.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f712659c640 1 -- 172.21.15.121:0/3938381892 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f712813a850 msgr2=0x7f71281ca190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:04.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f712659c640 1 --2- 172.21.15.121:0/3938381892 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f712813a850 0x7f71281ca190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:04.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f712659c640 10 monclient: _finish_hunting 0 2024-09-17T18:27:04.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f712659c640 1 monclient: found mon.noname-a 2024-09-17T18:27:04.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f712659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:27:04.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f712659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:27:04.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f712659c640 1 -- 172.21.15.121:0/3938381892 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f712811e7b0 con 0x7f71280a33f0 2024-09-17T18:27:04.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.699+0000 7f7125d9b640 1 --2- 172.21.15.121:0/3938381892 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f712813a850 0x7f71281ca190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:27:04.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.702+0000 7f712659c640 1 -- 172.21.15.121:0/3938381892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71280a33f0 msgr2=0x7f71281c9c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:04.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.702+0000 7f712659c640 1 --2- 172.21.15.121:0/3938381892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71280a33f0 0x7f71281c9c50 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f712000c430 tx=0x7f712000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:04.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.702+0000 7f7116ffd640 10 client.?.objecter ms_handle_connect 0x7f71280a33f0 2024-09-17T18:27:04.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.702+0000 7f7116ffd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:27:04.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.702+0000 7f7116ffd640 1 -- 172.21.15.121:0/3938381892 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71200119a0 con 0x7f71280a33f0 2024-09-17T18:27:04.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.702+0000 7f7116ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:04.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.702+0000 7f7116ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:27:04.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.702+0000 7f7116ffd640 10 monclient: dump: 2024-09-17T18:27:04.723 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:04.723 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:04.723 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:04.723 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:04.723 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:04.723 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:04.723 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:04.723 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:04.723 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:04.724 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:04.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.702+0000 7f7116ffd640 10 monclient: _finish_auth 0 2024-09-17T18:27:04.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.702+0000 7f7116ffd640 10 monclient: _check_auth_tickets 2024-09-17T18:27:04.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.702+0000 7f7116ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:27:04.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.702+0000 7f7116ffd640 1 -- 172.21.15.121:0/3938381892 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7120017070 con 0x7f71280a33f0 2024-09-17T18:27:04.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.702+0000 7f712c9d0640 5 monclient: authenticate success, global_id 44439 2024-09-17T18:27:04.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.702+0000 7f712c9d0640 10 monclient: _renew_subs 2024-09-17T18:27:04.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.702+0000 7f712c9d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:27:04.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.702+0000 7f712c9d0640 1 -- 172.21.15.121:0/3938381892 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f712811d980 con 0x7f71280a33f0 2024-09-17T18:27:04.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.702+0000 7f7116ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:27:04.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.702+0000 7f7116ffd640 1 -- 172.21.15.121:0/3938381892 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7120011d80 con 0x7f71280a33f0 2024-09-17T18:27:04.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.702+0000 7f7116ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:04.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.702+0000 7f7116ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:27:04.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.702+0000 7f7116ffd640 10 monclient: dump: 2024-09-17T18:27:04.725 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:04.725 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:04.725 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:04.725 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:04.725 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:04.725 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:04.725 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:04.725 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:04.725 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:04.725 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:04.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.702+0000 7f7124d99640 4 set_mon_vals no callback set 2024-09-17T18:27:04.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.702+0000 7f712c9d0640 10 client.44439.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:27:04.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.702+0000 7f712c9d0640 10 monclient: _renew_subs 2024-09-17T18:27:04.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.702+0000 7f712c9d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:27:04.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.702+0000 7f712c9d0640 1 -- 172.21.15.121:0/3938381892 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f712811fce0 con 0x7f71280a33f0 2024-09-17T18:27:04.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.702+0000 7f712c9d0640 1 librados: init done 2024-09-17T18:27:04.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.703+0000 7f712c9d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:27:04.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.703+0000 7f712c9d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:27:04.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.703+0000 7f712c9d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:27:04.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.703+0000 7f7116ffd640 1 -- 172.21.15.121:0/3938381892 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f712001c460 con 0x7f71280a33f0 2024-09-17T18:27:04.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.703+0000 7f712c9d0640 1 -- 172.21.15.121:0/3938381892 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f712809e550 con 0x7f71280a33f0 2024-09-17T18:27:04.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.703+0000 7f7116ffd640 1 --2- 172.21.15.121:0/3938381892 >> v2:172.21.15.173:6812/2423786037 conn(0x7f7108040170 0x7f7108042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:04.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.703+0000 7f7116ffd640 1 -- 172.21.15.121:0/3938381892 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(341..341 src has 177..341) ==== 5358+0+0 (secure 0 0 0) 0x7f7120053590 con 0x7f71280a33f0 2024-09-17T18:27:04.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.703+0000 7f7116ffd640 10 client.44439.objecter ms_dispatch 0x7f7128000b90 osd_map(341..341 src has 177..341) 2024-09-17T18:27:04.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.703+0000 7f7116ffd640 3 client.44439.objecter handle_osd_map got epochs [341,341] > 0 2024-09-17T18:27:04.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.703+0000 7f7116ffd640 3 client.44439.objecter handle_osd_map decoding full epoch 341 2024-09-17T18:27:04.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.703+0000 7f7116ffd640 20 client.44439.objecter dump_active .. 0 homeless 2024-09-17T18:27:04.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.703+0000 7f7125d9b640 1 --2- 172.21.15.121:0/3938381892 >> v2:172.21.15.173:6812/2423786037 conn(0x7f7108040170 0x7f7108042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:04.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.703+0000 7f7125d9b640 10 monclient: get_auth_request con 0x7f7108040170 auth_method 0 2024-09-17T18:27:04.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.706+0000 7f7116ffd640 10 client.44439.objecter ms_handle_connect 0x7f7108040170 2024-09-17T18:27:04.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.706+0000 7f7125d9b640 1 --2- 172.21.15.121:0/3938381892 >> v2:172.21.15.173:6812/2423786037 conn(0x7f7108040170 0x7f7108042630 secure :-1 s=READY pgs=3991 cs=0 l=1 rev1=1 crypto rx=0x7f711c031000 tx=0x7f711c02f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:04.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.708+0000 7f7116ffd640 1 -- 172.21.15.121:0/3938381892 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7120051d50 con 0x7f71280a33f0 2024-09-17T18:27:04.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.708+0000 7f7116ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:27:04.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.708+0000 7f7116ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:27:04.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.946+0000 7f712c9d0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:27:04.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.946+0000 7f712c9d0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:27:04.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.946+0000 7f712c9d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:27:04.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.946+0000 7f712c9d0640 1 -- 172.21.15.121:0/3938381892 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f712813a630 con 0x7f71280a33f0 2024-09-17T18:27:04.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.947+0000 7f7116ffd640 1 -- 172.21.15.121:0/3938381892 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f712001c930 con 0x7f71280a33f0 2024-09-17T18:27:04.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.947+0000 7f7116ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:27:04.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.947+0000 7f7116ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:27:04.949 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":30,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":94,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:27:04.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.948+0000 7f712c9d0640 10 librados: watch_flush enter 2024-09-17T18:27:04.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.948+0000 7f712c9d0640 10 librados: watch_flush exit 2024-09-17T18:27:04.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.948+0000 7f712c9d0640 20 client.44439.objecter shutdown clearing up homeless session... 2024-09-17T18:27:04.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.948+0000 7f712c9d0640 10 client.44439.objecter successfully canceled tick 2024-09-17T18:27:04.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.949+0000 7f712c9d0640 1 -- 172.21.15.121:0/3938381892 >> v2:172.21.15.173:6812/2423786037 conn(0x7f7108040170 msgr2=0x7f7108042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:04.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.949+0000 7f712c9d0640 1 --2- 172.21.15.121:0/3938381892 >> v2:172.21.15.173:6812/2423786037 conn(0x7f7108040170 0x7f7108042630 secure :-1 s=READY pgs=3991 cs=0 l=1 rev1=1 crypto rx=0x7f711c031000 tx=0x7f711c02f000 comp rx=0 tx=0).stop 2024-09-17T18:27:04.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.949+0000 7f712c9d0640 10 monclient: shutdown 2024-09-17T18:27:04.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.949+0000 7f712c9d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:27:04.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.949+0000 7f712c9d0640 1 -- 172.21.15.121:0/3938381892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71280a33f0 msgr2=0x7f71281c9c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:04.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.949+0000 7f712c9d0640 1 --2- 172.21.15.121:0/3938381892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71280a33f0 0x7f71281c9c50 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f712000c430 tx=0x7f712000c8f0 comp rx=0 tx=0).stop 2024-09-17T18:27:04.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.949+0000 7f712c9d0640 1 -- 172.21.15.121:0/3938381892 shutdown_connections 2024-09-17T18:27:04.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.949+0000 7f712c9d0640 1 --2- 172.21.15.121:0/3938381892 >> v2:172.21.15.173:6812/2423786037 conn(0x7f7108040170 0x7f7108042630 unknown :-1 s=CLOSED pgs=3991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:04.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.949+0000 7f712c9d0640 1 --2- 172.21.15.121:0/3938381892 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f712813a850 0x7f71281ca190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:04.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.949+0000 7f712c9d0640 1 --2- 172.21.15.121:0/3938381892 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7128072410 0x7f7128072830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:04.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.949+0000 7f712c9d0640 1 --2- 172.21.15.121:0/3938381892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71280a33f0 0x7f71281c9c50 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:04.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.949+0000 7f712c9d0640 1 -- 172.21.15.121:0/3938381892 >> 172.21.15.121:0/3938381892 conn(0x7f71280a3b50 msgr2=0x7f71280a26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:27:04.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.949+0000 7f712c9d0640 1 -- 172.21.15.121:0/3938381892 shutdown_connections 2024-09-17T18:27:04.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.949+0000 7f712c9d0640 1 -- 172.21.15.121:0/3938381892 wait complete. 2024-09-17T18:27:04.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:04.949+0000 7f712c9d0640 1 librados: shutdown 2024-09-17T18:27:07.962 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:27:08.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.069+0000 7ff94749b640 10 monclient: get_monmap_and_config 2024-09-17T18:27:08.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.069+0000 7ff94749b640 10 monclient: build_initial_monmap 2024-09-17T18:27:08.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.069+0000 7ff94749b640 10 monclient: monmap: 2024-09-17T18:27:08.071 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:27:08.071 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:08.071 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:27:08.069639+0000 2024-09-17T18:27:08.071 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:27:08.069639+0000 2024-09-17T18:27:08.071 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:27:08.071 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:08.071 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:27:08.071 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:27:08.071 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:27:08.071 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:27:08.071 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:27:08.071 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:08.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.069+0000 7ff94749b640 1 Processor -- start 2024-09-17T18:27:08.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.069+0000 7ff94749b640 1 -- start start 2024-09-17T18:27:08.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.069+0000 7ff94749b640 10 monclient: init 2024-09-17T18:27:08.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.070+0000 7ff94749b640 10 monclient: _reopen_session rank -1 2024-09-17T18:27:08.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.070+0000 7ff94749b640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-17T18:27:08.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.070+0000 7ff94749b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff9400a5c00 addr v1:172.21.15.173:6789/0 2024-09-17T18:27:08.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.070+0000 7ff94749b640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff9400a5fe0 0x7ff9400a63f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:08.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.070+0000 7ff94749b640 10 monclient(hunting): picked mon.noname-a con 0x7ff9400a5fe0 addr v2:172.21.15.121:3300/0 2024-09-17T18:27:08.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.070+0000 7ff94749b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff94013dc00 addr v1:172.21.15.121:6789/0 2024-09-17T18:27:08.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.070+0000 7ff94749b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9400a8a80 con 0x7ff94013dc00 2024-09-17T18:27:08.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.070+0000 7ff94749b640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9400a74d0 con 0x7ff9400a5c00 2024-09-17T18:27:08.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.070+0000 7ff94749b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:08.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.070+0000 7ff94749b640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7ff94009cb90 con 0x7ff9400a5fe0 2024-09-17T18:27:08.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.070+0000 7ff94749b640 10 monclient(hunting): _renew_subs 2024-09-17T18:27:08.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.070+0000 7ff94749b640 10 monclient(hunting): authenticate will time out at 1231.786554s 2024-09-17T18:27:08.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.070+0000 7ff944a36640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff9400a5fe0 0x7ff9400a63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:08.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.070+0000 7ff945a38640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7ff94013dc00 0x7ff94013fff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:39714/0 (socket says 172.21.15.121:39714) 2024-09-17T18:27:08.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.070+0000 7ff945a38640 1 -- 172.21.15.121:0/3098313027 learned_addr learned my addr 172.21.15.121:0/3098313027 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:27:08.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.070+0000 7ff944a36640 10 monclient(hunting): get_auth_request con 0x7ff9400a5fe0 auth_method 0 2024-09-17T18:27:08.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.070+0000 7ff944a36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:08.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.070+0000 7ff944a36640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:08.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.070+0000 7ff944a36640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:08.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.070+0000 7ff944a36640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:08.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.070+0000 7ff944a36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:08.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.070+0000 7ff944a36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:08.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.070+0000 7ff944a36640 10 monclient(hunting): handle_auth_done global_id 44442 payload 274 2024-09-17T18:27:08.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.070+0000 7ff944a36640 1 -- 172.21.15.121:0/3098313027 >> v1:172.21.15.173:6789/0 conn(0x7ff9400a5c00 legacy=0x7ff94009d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:27:08.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.070+0000 7ff944a36640 1 -- 172.21.15.121:0/3098313027 >> v1:172.21.15.121:6789/0 conn(0x7ff94013dc00 legacy=0x7ff94013fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:27:08.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.070+0000 7ff944a36640 10 monclient: _finish_hunting 0 2024-09-17T18:27:08.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.070+0000 7ff944a36640 1 monclient: found mon.noname-a 2024-09-17T18:27:08.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.070+0000 7ff944a36640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:27:08.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.070+0000 7ff944a36640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:27:08.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.070+0000 7ff944a36640 1 -- 172.21.15.121:0/3098313027 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff94009ce70 con 0x7ff9400a5fe0 2024-09-17T18:27:08.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.073+0000 7ff944a36640 1 -- 172.21.15.121:0/3098313027 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9400a5fe0 msgr2=0x7ff9400a63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:08.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.073+0000 7ff944a36640 1 --2- 172.21.15.121:0/3098313027 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9400a5fe0 0x7ff9400a63f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff9300097a0 tx=0x7ff930029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd3ee2ece24faed1 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:08.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.073+0000 7ff937fff640 1 -- 172.21.15.121:0/3098313027 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff930037070 con 0x7ff9400a5fe0 2024-09-17T18:27:08.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.073+0000 7ff937fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:08.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.074+0000 7ff937fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:27:08.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.074+0000 7ff937fff640 10 monclient: dump: 2024-09-17T18:27:08.080 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:08.080 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:08.080 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:08.080 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:08.080 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:08.080 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:08.080 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:08.081 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:08.081 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:08.081 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:08.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.074+0000 7ff937fff640 10 monclient: _finish_auth 0 2024-09-17T18:27:08.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.074+0000 7ff937fff640 10 monclient: _check_auth_tickets 2024-09-17T18:27:08.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.074+0000 7ff937fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:27:08.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.074+0000 7ff937fff640 1 -- 172.21.15.121:0/3098313027 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff930032640 con 0x7ff9400a5fe0 2024-09-17T18:27:08.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.074+0000 7ff94749b640 5 monclient: authenticate success, global_id 44442 2024-09-17T18:27:08.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.074+0000 7ff94749b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:27:08.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.074+0000 7ff937fff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:27:08.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.074+0000 7ff937fff640 1 -- 172.21.15.121:0/3098313027 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff930032a10 con 0x7ff9400a5fe0 2024-09-17T18:27:08.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.074+0000 7ff94749b640 10 monclient: get_monmap_and_config success 2024-09-17T18:27:08.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.074+0000 7ff94749b640 4 set_mon_vals no callback set 2024-09-17T18:27:08.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.074+0000 7ff937fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:08.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.074+0000 7ff937fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:27:08.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.074+0000 7ff937fff640 10 monclient: dump: 2024-09-17T18:27:08.082 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:08.082 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:08.082 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:08.082 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:08.082 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:08.082 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:08.082 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:08.082 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:08.082 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:08.082 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:08.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.074+0000 7ff94749b640 10 monclient: shutdown 2024-09-17T18:27:08.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.074+0000 7ff94749b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:27:08.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.074+0000 7ff94749b640 1 -- 172.21.15.121:0/3098313027 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9400a5fe0 msgr2=0x7ff9400a63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:08.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.074+0000 7ff94749b640 1 --2- 172.21.15.121:0/3098313027 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9400a5fe0 0x7ff9400a63f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff9300097a0 tx=0x7ff930029fd0 comp rx=0 tx=0).stop 2024-09-17T18:27:08.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.074+0000 7ff94749b640 1 -- 172.21.15.121:0/3098313027 shutdown_connections 2024-09-17T18:27:08.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.074+0000 7ff94749b640 1 --2- 172.21.15.121:0/3098313027 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9400a5fe0 0x7ff9400a63f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:08.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.074+0000 7ff94749b640 1 -- 172.21.15.121:0/3098313027 >> 172.21.15.121:0/3098313027 conn(0x7ff9400ab9b0 msgr2=0x7ff9400a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:27:08.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.074+0000 7ff94749b640 1 -- 172.21.15.121:0/3098313027 shutdown_connections 2024-09-17T18:27:08.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.074+0000 7ff94749b640 1 -- 172.21.15.121:0/3098313027 wait complete. 2024-09-17T18:27:08.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.075+0000 7ff94749b640 10 monclient: build_initial_monmap 2024-09-17T18:27:08.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.075+0000 7ff94749b640 10 monclient: monmap: 2024-09-17T18:27:08.083 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:27:08.083 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:27:08.083 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:27:08.083 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:27:08.083 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:27:08.084 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:08.084 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:27:08.084 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:27:08.084 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:27:08.084 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:08.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.075+0000 7ff94749b640 1 librados: starting msgr at 2024-09-17T18:27:08.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.075+0000 7ff94749b640 1 librados: starting objecter 2024-09-17T18:27:08.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.075+0000 7ff94749b640 1 Processor -- start 2024-09-17T18:27:08.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.075+0000 7ff94749b640 1 -- start start 2024-09-17T18:27:08.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.075+0000 7ff94749b640 1 librados: setting wanted keys 2024-09-17T18:27:08.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.075+0000 7ff94749b640 1 librados: calling monclient init 2024-09-17T18:27:08.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.075+0000 7ff94749b640 10 monclient: init 2024-09-17T18:27:08.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff94749b640 10 monclient: _reopen_session rank -1 2024-09-17T18:27:08.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff94749b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:27:08.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff94749b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9400a5c00 0x7ff94013cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:08.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff94749b640 10 monclient(hunting): picked mon.noname-a con 0x7ff9400a5c00 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:27:08.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff94749b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9400a5fe0 0x7ff9401cde40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:08.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff94749b640 10 monclient(hunting): picked mon.noname-b con 0x7ff9400a5fe0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:27:08.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff94749b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff94013dc00 0x7ff94013a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:08.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff94749b640 10 monclient(hunting): picked mon.noname-c con 0x7ff94013dc00 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:27:08.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff94749b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:08.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff94749b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff940070550 con 0x7ff9400a5c00 2024-09-17T18:27:08.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff94749b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:08.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff94749b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff9400703d0 con 0x7ff9400a5fe0 2024-09-17T18:27:08.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff94749b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:08.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff94749b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff940141710 con 0x7ff94013dc00 2024-09-17T18:27:08.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff94749b640 10 monclient(hunting): _renew_subs 2024-09-17T18:27:08.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff94749b640 10 monclient(hunting): authenticate will time out at 1231.792176s 2024-09-17T18:27:08.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff945237640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9400a5c00 0x7ff94013cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:08.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff945237640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9400a5c00 0x7ff94013cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41258/0 (socket says 172.21.15.121:41258) 2024-09-17T18:27:08.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff945237640 1 -- 172.21.15.121:0/1108516122 learned_addr learned my addr 172.21.15.121:0/1108516122 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:27:08.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff945237640 10 monclient(hunting): get_auth_request con 0x7ff9400a5c00 auth_method 0 2024-09-17T18:27:08.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff945237640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:08.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff945237640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:08.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff945237640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:08.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff945a38640 1 --2- 172.21.15.121:0/1108516122 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff94013dc00 0x7ff94013a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:08.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff945237640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:08.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff945237640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:08.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff945237640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:08.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff945a38640 10 monclient(hunting): get_auth_request con 0x7ff94013dc00 auth_method 0 2024-09-17T18:27:08.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff945a38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:08.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff945a38640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:08.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff945a38640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:08.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff944a36640 1 --2- 172.21.15.121:0/1108516122 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9400a5fe0 0x7ff9401cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:08.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff944a36640 10 monclient(hunting): get_auth_request con 0x7ff9400a5fe0 auth_method 0 2024-09-17T18:27:08.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff944a36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:08.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff944a36640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:08.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff944a36640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:08.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff945a38640 1 -- 172.21.15.121:0/1108516122 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff94013dc00 msgr2=0x7ff94013a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-17T18:27:08.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff945a38640 1 -- 172.21.15.121:0/1108516122 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff94013dc00 msgr2=0x7ff94013a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:27:08.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff945a38640 1 --2- 172.21.15.121:0/1108516122 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff94013dc00 0x7ff94013a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:27:08.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff945237640 10 monclient(hunting): handle_auth_done global_id 44445 payload 824 2024-09-17T18:27:08.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff945a38640 1 --2- 172.21.15.121:0/1108516122 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff94013dc00 0x7ff94013a7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:27:08.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff945237640 1 -- 172.21.15.121:0/1108516122 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff94013dc00 msgr2=0x7ff94013a7d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:27:08.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff945237640 1 --2- 172.21.15.121:0/1108516122 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff94013dc00 0x7ff94013a7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:08.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff945237640 1 -- 172.21.15.121:0/1108516122 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9400a5fe0 msgr2=0x7ff9401cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:08.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff945237640 1 --2- 172.21.15.121:0/1108516122 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9400a5fe0 0x7ff9401cde40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:08.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff945237640 10 monclient: _finish_hunting 0 2024-09-17T18:27:08.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff945237640 1 monclient: found mon.noname-a 2024-09-17T18:27:08.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff945237640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:27:08.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff945237640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:27:08.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff945237640 1 -- 172.21.15.121:0/1108516122 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff94013d780 con 0x7ff9400a5c00 2024-09-17T18:27:08.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.076+0000 7ff944a36640 1 --2- 172.21.15.121:0/1108516122 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9400a5fe0 0x7ff9401cde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T18:27:08.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.078+0000 7ff945237640 1 -- 172.21.15.121:0/1108516122 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9400a5c00 msgr2=0x7ff94013cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:08.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.078+0000 7ff945237640 1 --2- 172.21.15.121:0/1108516122 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9400a5c00 0x7ff94013cdf0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff93c00c7c0 tx=0x7ff93c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:08.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.078+0000 7ff935ffb640 10 client.?.objecter ms_handle_connect 0x7ff9400a5c00 2024-09-17T18:27:08.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.078+0000 7ff935ffb640 10 client.?.objecter resend_mon_ops 2024-09-17T18:27:08.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.078+0000 7ff935ffb640 1 -- 172.21.15.121:0/1108516122 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff93c016020 con 0x7ff9400a5c00 2024-09-17T18:27:08.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.078+0000 7ff935ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:08.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.078+0000 7ff935ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:27:08.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.078+0000 7ff935ffb640 10 monclient: dump: 2024-09-17T18:27:08.091 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:08.091 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:08.091 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:08.091 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:08.091 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:08.091 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:08.091 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:08.091 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:08.091 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:08.091 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:08.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.078+0000 7ff935ffb640 10 monclient: _finish_auth 0 2024-09-17T18:27:08.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.078+0000 7ff935ffb640 10 monclient: _check_auth_tickets 2024-09-17T18:27:08.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.078+0000 7ff935ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:27:08.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.078+0000 7ff935ffb640 1 -- 172.21.15.121:0/1108516122 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff93c00ac60 con 0x7ff9400a5c00 2024-09-17T18:27:08.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.078+0000 7ff94749b640 5 monclient: authenticate success, global_id 44445 2024-09-17T18:27:08.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.078+0000 7ff94749b640 10 monclient: _renew_subs 2024-09-17T18:27:08.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.078+0000 7ff94749b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:27:08.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.078+0000 7ff94749b640 1 -- 172.21.15.121:0/1108516122 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff94013c290 con 0x7ff9400a5c00 2024-09-17T18:27:08.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.078+0000 7ff935ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:27:08.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.078+0000 7ff935ffb640 1 -- 172.21.15.121:0/1108516122 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff93c012660 con 0x7ff9400a5c00 2024-09-17T18:27:08.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.078+0000 7ff935ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:08.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.078+0000 7ff935ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:27:08.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.078+0000 7ff935ffb640 10 monclient: dump: 2024-09-17T18:27:08.092 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:08.092 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:08.093 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:08.093 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:08.093 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:08.093 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:08.093 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:08.093 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:08.093 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:08.093 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:08.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.079+0000 7ff94749b640 10 client.44445.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:27:08.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.079+0000 7ff94749b640 10 monclient: _renew_subs 2024-09-17T18:27:08.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.078+0000 7ff9377fe640 4 set_mon_vals no callback set 2024-09-17T18:27:08.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.079+0000 7ff94749b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:27:08.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.079+0000 7ff94749b640 1 -- 172.21.15.121:0/1108516122 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9401c66c0 con 0x7ff9400a5c00 2024-09-17T18:27:08.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.079+0000 7ff94749b640 1 librados: init done 2024-09-17T18:27:08.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.080+0000 7ff935ffb640 1 -- 172.21.15.121:0/1108516122 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7ff93c019020 con 0x7ff9400a5c00 2024-09-17T18:27:08.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.080+0000 7ff94749b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:27:08.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.080+0000 7ff94749b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:27:08.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.080+0000 7ff94749b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:27:08.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.080+0000 7ff94749b640 1 -- 172.21.15.121:0/1108516122 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9400a63f0 con 0x7ff9400a5c00 2024-09-17T18:27:08.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.080+0000 7ff935ffb640 1 --2- 172.21.15.121:0/1108516122 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff924040170 0x7ff924042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:08.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.080+0000 7ff935ffb640 1 -- 172.21.15.121:0/1108516122 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(341..341 src has 177..341) ==== 5358+0+0 (secure 0 0 0) 0x7ff93c053500 con 0x7ff9400a5c00 2024-09-17T18:27:08.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.080+0000 7ff935ffb640 10 client.44445.objecter ms_dispatch 0x7ff940000b90 osd_map(341..341 src has 177..341) 2024-09-17T18:27:08.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.080+0000 7ff935ffb640 3 client.44445.objecter handle_osd_map got epochs [341,341] > 0 2024-09-17T18:27:08.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.080+0000 7ff935ffb640 3 client.44445.objecter handle_osd_map decoding full epoch 341 2024-09-17T18:27:08.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.080+0000 7ff935ffb640 20 client.44445.objecter dump_active .. 0 homeless 2024-09-17T18:27:08.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.080+0000 7ff944a36640 1 --2- 172.21.15.121:0/1108516122 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff924040170 0x7ff924042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:08.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.080+0000 7ff944a36640 10 monclient: get_auth_request con 0x7ff924040170 auth_method 0 2024-09-17T18:27:08.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.083+0000 7ff944a36640 1 --2- 172.21.15.121:0/1108516122 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff924040170 0x7ff924042630 secure :-1 s=READY pgs=3992 cs=0 l=1 rev1=1 crypto rx=0x7ff94009d210 tx=0x7ff93002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:08.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.083+0000 7ff935ffb640 10 client.44445.objecter ms_handle_connect 0x7ff924040170 2024-09-17T18:27:08.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.085+0000 7ff935ffb640 1 -- 172.21.15.121:0/1108516122 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff93c020070 con 0x7ff9400a5c00 2024-09-17T18:27:08.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.085+0000 7ff935ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:27:08.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.085+0000 7ff935ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:27:08.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.325+0000 7ff94749b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:27:08.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.325+0000 7ff94749b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:27:08.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.325+0000 7ff94749b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:27:08.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.325+0000 7ff94749b640 1 -- 172.21.15.121:0/1108516122 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff9401219a0 con 0x7ff9400a5c00 2024-09-17T18:27:08.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.325+0000 7ff935ffb640 1 -- 172.21.15.121:0/1108516122 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7ff93c00ae10 con 0x7ff9400a5c00 2024-09-17T18:27:08.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.325+0000 7ff935ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:27:08.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.325+0000 7ff935ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:27:08.328 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":30,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":97,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:27:08.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.327+0000 7ff94749b640 10 librados: watch_flush enter 2024-09-17T18:27:08.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.328+0000 7ff94749b640 10 librados: watch_flush exit 2024-09-17T18:27:08.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.328+0000 7ff94749b640 20 client.44445.objecter shutdown clearing up homeless session... 2024-09-17T18:27:08.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.328+0000 7ff94749b640 10 client.44445.objecter successfully canceled tick 2024-09-17T18:27:08.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.328+0000 7ff94749b640 1 -- 172.21.15.121:0/1108516122 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff924040170 msgr2=0x7ff924042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:08.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.328+0000 7ff94749b640 1 --2- 172.21.15.121:0/1108516122 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff924040170 0x7ff924042630 secure :-1 s=READY pgs=3992 cs=0 l=1 rev1=1 crypto rx=0x7ff94009d210 tx=0x7ff93002e000 comp rx=0 tx=0).stop 2024-09-17T18:27:08.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.328+0000 7ff94749b640 10 monclient: shutdown 2024-09-17T18:27:08.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.328+0000 7ff94749b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:27:08.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.328+0000 7ff94749b640 1 -- 172.21.15.121:0/1108516122 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9400a5c00 msgr2=0x7ff94013cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:08.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.328+0000 7ff94749b640 1 --2- 172.21.15.121:0/1108516122 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9400a5c00 0x7ff94013cdf0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff93c00c7c0 tx=0x7ff93c00cc80 comp rx=0 tx=0).stop 2024-09-17T18:27:08.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.328+0000 7ff94749b640 1 -- 172.21.15.121:0/1108516122 shutdown_connections 2024-09-17T18:27:08.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.328+0000 7ff94749b640 1 --2- 172.21.15.121:0/1108516122 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff924040170 0x7ff924042630 unknown :-1 s=CLOSED pgs=3992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:08.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.328+0000 7ff94749b640 1 --2- 172.21.15.121:0/1108516122 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9400a5fe0 0x7ff9401cde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:08.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.328+0000 7ff94749b640 1 --2- 172.21.15.121:0/1108516122 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff94013dc00 0x7ff94013a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:08.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.328+0000 7ff94749b640 1 --2- 172.21.15.121:0/1108516122 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9400a5c00 0x7ff94013cdf0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:08.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.328+0000 7ff94749b640 1 -- 172.21.15.121:0/1108516122 >> 172.21.15.121:0/1108516122 conn(0x7ff9400ab9b0 msgr2=0x7ff9400a4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:27:08.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.328+0000 7ff94749b640 1 -- 172.21.15.121:0/1108516122 shutdown_connections 2024-09-17T18:27:08.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.328+0000 7ff94749b640 1 -- 172.21.15.121:0/1108516122 wait complete. 2024-09-17T18:27:08.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:08.328+0000 7ff94749b640 1 librados: shutdown 2024-09-17T18:27:11.340 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:27:11.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.447+0000 7fce180e5640 10 monclient: get_monmap_and_config 2024-09-17T18:27:11.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.447+0000 7fce180e5640 10 monclient: build_initial_monmap 2024-09-17T18:27:11.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.447+0000 7fce180e5640 10 monclient: monmap: 2024-09-17T18:27:11.449 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:27:11.449 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:11.449 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:27:11.447705+0000 2024-09-17T18:27:11.449 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:27:11.447705+0000 2024-09-17T18:27:11.449 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:27:11.449 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:11.449 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:27:11.449 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:27:11.450 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:27:11.450 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:27:11.450 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:27:11.450 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:11.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.447+0000 7fce180e5640 1 Processor -- start 2024-09-17T18:27:11.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.447+0000 7fce180e5640 1 -- start start 2024-09-17T18:27:11.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.447+0000 7fce180e5640 10 monclient: init 2024-09-17T18:27:11.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.448+0000 7fce180e5640 10 monclient: _reopen_session rank -1 2024-09-17T18:27:11.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.448+0000 7fce180e5640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-17T18:27:11.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.448+0000 7fce180e5640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fce100a5c00 0x7fce1009d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:11.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.448+0000 7fce180e5640 10 monclient(hunting): picked mon.noname-c con 0x7fce100a5c00 addr v2:172.21.15.173:3300/0 2024-09-17T18:27:11.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.448+0000 7fce180e5640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fce100a5fe0 0x7fce100a63f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:11.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.448+0000 7fce180e5640 10 monclient(hunting): picked mon.noname-a con 0x7fce100a5fe0 addr v2:172.21.15.121:3300/0 2024-09-17T18:27:11.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.448+0000 7fce180e5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fce101389c0 addr v1:172.21.15.121:6789/0 2024-09-17T18:27:11.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.448+0000 7fce180e5640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce100a8a80 con 0x7fce101389c0 2024-09-17T18:27:11.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.448+0000 7fce180e5640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:11.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.448+0000 7fce180e5640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fce1009de00 con 0x7fce100a5fe0 2024-09-17T18:27:11.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.448+0000 7fce180e5640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:11.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.448+0000 7fce180e5640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fce100a74d0 con 0x7fce100a5c00 2024-09-17T18:27:11.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.448+0000 7fce180e5640 10 monclient(hunting): _renew_subs 2024-09-17T18:27:11.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.448+0000 7fce180e5640 10 monclient(hunting): authenticate will time out at 1235.164430s 2024-09-17T18:27:11.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.448+0000 7fce15680640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fce100a5fe0 0x7fce100a63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:11.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.448+0000 7fce16682640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fce101389c0 0x7fce10138dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:35546/0 (socket says 172.21.15.121:35546) 2024-09-17T18:27:11.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.448+0000 7fce16682640 1 -- 172.21.15.121:0/1088075951 learned_addr learned my addr 172.21.15.121:0/1088075951 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:27:11.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.448+0000 7fce15680640 10 monclient(hunting): get_auth_request con 0x7fce100a5fe0 auth_method 0 2024-09-17T18:27:11.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.448+0000 7fce15680640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:11.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.448+0000 7fce15680640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:11.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.448+0000 7fce15680640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:11.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.448+0000 7fce15680640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:11.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.448+0000 7fce15680640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:11.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.448+0000 7fce15e81640 1 --2- 172.21.15.121:0/1088075951 >> v2:172.21.15.173:3300/0 conn(0x7fce100a5c00 0x7fce1009d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:11.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.448+0000 7fce15680640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:11.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.448+0000 7fce15e81640 10 monclient(hunting): get_auth_request con 0x7fce100a5c00 auth_method 0 2024-09-17T18:27:11.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.448+0000 7fce15e81640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:11.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.448+0000 7fce15e81640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:11.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.448+0000 7fce15e81640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:11.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.448+0000 7fce15e81640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:11.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.448+0000 7fce15e81640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:11.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.448+0000 7fce15e81640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:11.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.448+0000 7fce15680640 10 monclient(hunting): handle_auth_done global_id 44448 payload 274 2024-09-17T18:27:11.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.448+0000 7fce15680640 1 -- 172.21.15.121:0/1088075951 >> v2:172.21.15.173:3300/0 conn(0x7fce100a5c00 msgr2=0x7fce1009d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:27:11.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.448+0000 7fce15680640 1 --2- 172.21.15.121:0/1088075951 >> v2:172.21.15.173:3300/0 conn(0x7fce100a5c00 0x7fce1009d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:11.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.448+0000 7fce15680640 1 -- 172.21.15.121:0/1088075951 >> v1:172.21.15.121:6789/0 conn(0x7fce101389c0 legacy=0x7fce10138dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:27:11.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.448+0000 7fce15680640 10 monclient: _finish_hunting 0 2024-09-17T18:27:11.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.448+0000 7fce15680640 1 monclient: found mon.noname-a 2024-09-17T18:27:11.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.448+0000 7fce15680640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:27:11.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.448+0000 7fce15680640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:27:11.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.448+0000 7fce15680640 1 -- 172.21.15.121:0/1088075951 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce1009cb90 con 0x7fce100a5fe0 2024-09-17T18:27:11.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.451+0000 7fce15680640 1 -- 172.21.15.121:0/1088075951 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce100a5fe0 msgr2=0x7fce100a63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:11.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.451+0000 7fce15680640 1 --2- 172.21.15.121:0/1088075951 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce100a5fe0 0x7fce100a63f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fce1006a3f0 tx=0x7fce0402a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=876b3acb841777c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:11.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.451+0000 7fce14e7f640 1 -- 172.21.15.121:0/1088075951 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce04037070 con 0x7fce100a5fe0 2024-09-17T18:27:11.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.451+0000 7fce14e7f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:11.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.451+0000 7fce14e7f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:27:11.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.451+0000 7fce14e7f640 10 monclient: dump: 2024-09-17T18:27:11.463 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:11.463 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:11.463 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:11.463 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:11.463 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:11.463 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:11.463 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:11.463 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:11.463 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:11.464 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:11.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.451+0000 7fce14e7f640 10 monclient: _finish_auth 0 2024-09-17T18:27:11.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.451+0000 7fce14e7f640 10 monclient: _check_auth_tickets 2024-09-17T18:27:11.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.451+0000 7fce14e7f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:27:11.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.451+0000 7fce14e7f640 1 -- 172.21.15.121:0/1088075951 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce04032440 con 0x7fce100a5fe0 2024-09-17T18:27:11.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.451+0000 7fce180e5640 5 monclient: authenticate success, global_id 44448 2024-09-17T18:27:11.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.451+0000 7fce180e5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:27:11.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.451+0000 7fce14e7f640 10 monclient: handle_config config(0 keys) 2024-09-17T18:27:11.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.451+0000 7fce180e5640 10 monclient: get_monmap_and_config success 2024-09-17T18:27:11.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.451+0000 7fce180e5640 4 set_mon_vals no callback set 2024-09-17T18:27:11.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.451+0000 7fce14e7f640 1 -- 172.21.15.121:0/1088075951 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce04032810 con 0x7fce100a5fe0 2024-09-17T18:27:11.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.451+0000 7fce14e7f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:11.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.451+0000 7fce14e7f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:27:11.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.451+0000 7fce14e7f640 10 monclient: dump: 2024-09-17T18:27:11.465 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:11.465 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:11.465 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:11.465 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:11.465 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:11.465 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:11.465 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:11.465 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:11.465 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:11.465 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:11.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.452+0000 7fce180e5640 10 monclient: shutdown 2024-09-17T18:27:11.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.452+0000 7fce180e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:27:11.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.452+0000 7fce180e5640 1 -- 172.21.15.121:0/1088075951 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce100a5fe0 msgr2=0x7fce100a63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:11.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.452+0000 7fce180e5640 1 --2- 172.21.15.121:0/1088075951 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce100a5fe0 0x7fce100a63f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fce1006a3f0 tx=0x7fce0402a000 comp rx=0 tx=0).stop 2024-09-17T18:27:11.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.452+0000 7fce180e5640 1 -- 172.21.15.121:0/1088075951 shutdown_connections 2024-09-17T18:27:11.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.452+0000 7fce180e5640 1 --2- 172.21.15.121:0/1088075951 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce100a5fe0 0x7fce100a63f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:11.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.452+0000 7fce180e5640 1 --2- 172.21.15.121:0/1088075951 >> v2:172.21.15.173:3300/0 conn(0x7fce100a5c00 0x7fce1009d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:11.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.452+0000 7fce180e5640 1 -- 172.21.15.121:0/1088075951 >> 172.21.15.121:0/1088075951 conn(0x7fce100ab9b0 msgr2=0x7fce100a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:27:11.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.452+0000 7fce180e5640 1 -- 172.21.15.121:0/1088075951 shutdown_connections 2024-09-17T18:27:11.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.452+0000 7fce180e5640 1 -- 172.21.15.121:0/1088075951 wait complete. 2024-09-17T18:27:11.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.453+0000 7fce180e5640 10 monclient: build_initial_monmap 2024-09-17T18:27:11.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.453+0000 7fce180e5640 10 monclient: monmap: 2024-09-17T18:27:11.466 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:27:11.467 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:27:11.467 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:27:11.467 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:27:11.467 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:27:11.467 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:11.467 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:27:11.467 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:27:11.467 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:27:11.467 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:11.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.453+0000 7fce180e5640 1 librados: starting msgr at 2024-09-17T18:27:11.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.453+0000 7fce180e5640 1 librados: starting objecter 2024-09-17T18:27:11.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.453+0000 7fce180e5640 1 Processor -- start 2024-09-17T18:27:11.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.453+0000 7fce180e5640 1 -- start start 2024-09-17T18:27:11.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.453+0000 7fce180e5640 1 librados: setting wanted keys 2024-09-17T18:27:11.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.453+0000 7fce180e5640 1 librados: calling monclient init 2024-09-17T18:27:11.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.453+0000 7fce180e5640 10 monclient: init 2024-09-17T18:27:11.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.453+0000 7fce180e5640 10 monclient: _reopen_session rank -1 2024-09-17T18:27:11.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.453+0000 7fce180e5640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T18:27:11.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.453+0000 7fce180e5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fce100a5c00 0x7fce101240a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:11.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.453+0000 7fce180e5640 10 monclient(hunting): picked mon.noname-c con 0x7fce100a5c00 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:27:11.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.453+0000 7fce180e5640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce100a5fe0 0x7fce101245e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:11.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.453+0000 7fce180e5640 10 monclient(hunting): picked mon.noname-b con 0x7fce100a5fe0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:27:11.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce180e5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce101389c0 0x7fce10124b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:11.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce180e5640 10 monclient(hunting): picked mon.noname-a con 0x7fce101389c0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:27:11.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce180e5640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:11.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce180e5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fce100703d0 con 0x7fce101389c0 2024-09-17T18:27:11.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce180e5640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:11.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce180e5640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fce100a6930 con 0x7fce100a5fe0 2024-09-17T18:27:11.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce180e5640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:11.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce180e5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fce10070550 con 0x7fce100a5c00 2024-09-17T18:27:11.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce180e5640 10 monclient(hunting): _renew_subs 2024-09-17T18:27:11.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce180e5640 10 monclient(hunting): authenticate will time out at 1235.170035s 2024-09-17T18:27:11.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce15e81640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fce100a5c00 0x7fce101240a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:11.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce16682640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce101389c0 0x7fce10124b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:11.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce15e81640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fce100a5c00 0x7fce101240a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:46450/0 (socket says 172.21.15.121:46450) 2024-09-17T18:27:11.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce15e81640 1 -- 172.21.15.121:0/1514304540 learned_addr learned my addr 172.21.15.121:0/1514304540 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:27:11.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce16682640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce101389c0 0x7fce10124b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56478/0 (socket says 172.21.15.121:56478) 2024-09-17T18:27:11.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce15e81640 10 monclient(hunting): get_auth_request con 0x7fce100a5c00 auth_method 0 2024-09-17T18:27:11.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce15e81640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:11.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce15e81640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:11.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce15e81640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:11.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce16682640 10 monclient(hunting): get_auth_request con 0x7fce101389c0 auth_method 0 2024-09-17T18:27:11.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce16682640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:11.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce16682640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:11.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce16682640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:11.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce16682640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:11.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce16682640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:11.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce16682640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:11.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce15680640 1 --2- 172.21.15.121:0/1514304540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce100a5fe0 0x7fce101245e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:11.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce15e81640 1 -- 172.21.15.121:0/1514304540 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fce100a5c00 msgr2=0x7fce101240a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T18:27:11.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce15e81640 1 -- 172.21.15.121:0/1514304540 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fce100a5c00 msgr2=0x7fce101240a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:27:11.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce15680640 10 monclient(hunting): get_auth_request con 0x7fce100a5fe0 auth_method 0 2024-09-17T18:27:11.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce15e81640 1 --2- 172.21.15.121:0/1514304540 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fce100a5c00 0x7fce101240a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:27:11.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce15680640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:11.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce15680640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:11.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce15680640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:11.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce15e81640 1 --2- 172.21.15.121:0/1514304540 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fce100a5c00 0x7fce101240a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:27:11.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce15680640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:11.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce15680640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:11.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce15680640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:11.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce16682640 10 monclient(hunting): handle_auth_done global_id 44451 payload 824 2024-09-17T18:27:11.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce16682640 1 -- 172.21.15.121:0/1514304540 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fce100a5c00 msgr2=0x7fce101240a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:27:11.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce16682640 1 --2- 172.21.15.121:0/1514304540 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fce100a5c00 0x7fce101240a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:11.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce16682640 1 -- 172.21.15.121:0/1514304540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce100a5fe0 msgr2=0x7fce101245e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:11.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce16682640 1 --2- 172.21.15.121:0/1514304540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce100a5fe0 0x7fce101245e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:11.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce16682640 10 monclient: _finish_hunting 0 2024-09-17T18:27:11.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce16682640 1 monclient: found mon.noname-a 2024-09-17T18:27:11.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce16682640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:27:11.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce16682640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:27:11.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.454+0000 7fce16682640 1 -- 172.21.15.121:0/1514304540 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce10127050 con 0x7fce101389c0 2024-09-17T18:27:11.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.456+0000 7fce16682640 1 -- 172.21.15.121:0/1514304540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce101389c0 msgr2=0x7fce10124b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:11.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.456+0000 7fce16682640 1 --2- 172.21.15.121:0/1514304540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce101389c0 0x7fce10124b20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fce0000caf0 tx=0x7fce0000cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:11.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.456+0000 7fce0e7fc640 10 client.?.objecter ms_handle_connect 0x7fce101389c0 2024-09-17T18:27:11.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.456+0000 7fce0e7fc640 10 client.?.objecter resend_mon_ops 2024-09-17T18:27:11.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.456+0000 7fce0e7fc640 1 -- 172.21.15.121:0/1514304540 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce0000ade0 con 0x7fce101389c0 2024-09-17T18:27:11.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.456+0000 7fce0e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:11.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.456+0000 7fce0e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:27:11.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.456+0000 7fce0e7fc640 10 monclient: dump: 2024-09-17T18:27:11.473 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:11.473 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:11.473 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:11.473 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:11.473 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:11.473 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:11.473 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:11.473 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:11.473 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:11.473 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:11.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.456+0000 7fce0e7fc640 10 monclient: _finish_auth 0 2024-09-17T18:27:11.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.456+0000 7fce0e7fc640 10 monclient: _check_auth_tickets 2024-09-17T18:27:11.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.456+0000 7fce0e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:27:11.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.456+0000 7fce0e7fc640 1 -- 172.21.15.121:0/1514304540 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce00012400 con 0x7fce101389c0 2024-09-17T18:27:11.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.456+0000 7fce180e5640 5 monclient: authenticate success, global_id 44451 2024-09-17T18:27:11.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.456+0000 7fce180e5640 10 monclient: _renew_subs 2024-09-17T18:27:11.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.456+0000 7fce180e5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:27:11.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.456+0000 7fce180e5640 1 -- 172.21.15.121:0/1514304540 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce10127be0 con 0x7fce101389c0 2024-09-17T18:27:11.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.456+0000 7fce0e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:27:11.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.456+0000 7fce0e7fc640 1 -- 172.21.15.121:0/1514304540 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce000127f0 con 0x7fce101389c0 2024-09-17T18:27:11.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.456+0000 7fce0e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:11.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.456+0000 7fce0e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:27:11.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.456+0000 7fce0e7fc640 10 monclient: dump: 2024-09-17T18:27:11.474 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:11.474 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:11.474 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:11.475 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:11.475 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:11.475 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:11.475 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:11.475 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:11.475 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:11.475 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:11.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.456+0000 7fce180e5640 10 client.44451.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:27:11.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.456+0000 7fce0ffff640 4 set_mon_vals no callback set 2024-09-17T18:27:11.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.456+0000 7fce180e5640 10 monclient: _renew_subs 2024-09-17T18:27:11.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.456+0000 7fce180e5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:27:11.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.456+0000 7fce180e5640 1 -- 172.21.15.121:0/1514304540 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce100a4d60 con 0x7fce101389c0 2024-09-17T18:27:11.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.456+0000 7fce180e5640 1 librados: init done 2024-09-17T18:27:11.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.457+0000 7fce180e5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:27:11.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.457+0000 7fce180e5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:27:11.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.457+0000 7fce180e5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:27:11.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.457+0000 7fce0e7fc640 1 -- 172.21.15.121:0/1514304540 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fce000129d0 con 0x7fce101389c0 2024-09-17T18:27:11.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.457+0000 7fce180e5640 1 -- 172.21.15.121:0/1514304540 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce1009d8c0 con 0x7fce101389c0 2024-09-17T18:27:11.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.458+0000 7fce0e7fc640 1 --2- 172.21.15.121:0/1514304540 >> v2:172.21.15.173:6812/2423786037 conn(0x7fcdfc040170 0x7fcdfc042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:11.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.458+0000 7fce0e7fc640 1 -- 172.21.15.121:0/1514304540 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(341..341 src has 177..341) ==== 5358+0+0 (secure 0 0 0) 0x7fce00053cc0 con 0x7fce101389c0 2024-09-17T18:27:11.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.458+0000 7fce0e7fc640 10 client.44451.objecter ms_dispatch 0x7fce10000b90 osd_map(341..341 src has 177..341) 2024-09-17T18:27:11.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.458+0000 7fce0e7fc640 3 client.44451.objecter handle_osd_map got epochs [341,341] > 0 2024-09-17T18:27:11.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.458+0000 7fce0e7fc640 3 client.44451.objecter handle_osd_map decoding full epoch 341 2024-09-17T18:27:11.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.458+0000 7fce0e7fc640 20 client.44451.objecter dump_active .. 0 homeless 2024-09-17T18:27:11.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.458+0000 7fce15e81640 1 --2- 172.21.15.121:0/1514304540 >> v2:172.21.15.173:6812/2423786037 conn(0x7fcdfc040170 0x7fcdfc042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:11.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.458+0000 7fce15e81640 10 monclient: get_auth_request con 0x7fcdfc040170 auth_method 0 2024-09-17T18:27:11.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.460+0000 7fce0e7fc640 10 client.44451.objecter ms_handle_connect 0x7fcdfc040170 2024-09-17T18:27:11.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.460+0000 7fce15e81640 1 --2- 172.21.15.121:0/1514304540 >> v2:172.21.15.173:6812/2423786037 conn(0x7fcdfc040170 0x7fcdfc042630 secure :-1 s=READY pgs=3993 cs=0 l=1 rev1=1 crypto rx=0x7fce080095c0 tx=0x7fce08008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:11.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.462+0000 7fce0e7fc640 1 -- 172.21.15.121:0/1514304540 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fce00052480 con 0x7fce101389c0 2024-09-17T18:27:11.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.462+0000 7fce0e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:27:11.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.462+0000 7fce0e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:27:11.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.707+0000 7fce180e5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:27:11.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.707+0000 7fce180e5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:27:11.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.707+0000 7fce180e5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:27:11.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.707+0000 7fce180e5640 1 -- 172.21.15.121:0/1514304540 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fce100008d0 con 0x7fce101389c0 2024-09-17T18:27:11.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.708+0000 7fce0e7fc640 1 -- 172.21.15.121:0/1514304540 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1564 (secure 0 0 0) 0x7fce0001bc00 con 0x7fce101389c0 2024-09-17T18:27:11.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.708+0000 7fce0e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:27:11.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.708+0000 7fce0e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:27:11.710 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":30,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":101,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:27:11.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.709+0000 7fce180e5640 10 librados: watch_flush enter 2024-09-17T18:27:11.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.710+0000 7fce180e5640 10 librados: watch_flush exit 2024-09-17T18:27:11.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.710+0000 7fce180e5640 20 client.44451.objecter shutdown clearing up homeless session... 2024-09-17T18:27:11.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.710+0000 7fce180e5640 10 client.44451.objecter successfully canceled tick 2024-09-17T18:27:11.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.710+0000 7fce180e5640 1 -- 172.21.15.121:0/1514304540 >> v2:172.21.15.173:6812/2423786037 conn(0x7fcdfc040170 msgr2=0x7fcdfc042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:11.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.710+0000 7fce180e5640 1 --2- 172.21.15.121:0/1514304540 >> v2:172.21.15.173:6812/2423786037 conn(0x7fcdfc040170 0x7fcdfc042630 secure :-1 s=READY pgs=3993 cs=0 l=1 rev1=1 crypto rx=0x7fce080095c0 tx=0x7fce08008040 comp rx=0 tx=0).stop 2024-09-17T18:27:11.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.710+0000 7fce180e5640 10 monclient: shutdown 2024-09-17T18:27:11.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.710+0000 7fce180e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:27:11.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.710+0000 7fce180e5640 1 -- 172.21.15.121:0/1514304540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce101389c0 msgr2=0x7fce10124b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:11.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.710+0000 7fce180e5640 1 --2- 172.21.15.121:0/1514304540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce101389c0 0x7fce10124b20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fce0000caf0 tx=0x7fce0000cef0 comp rx=0 tx=0).stop 2024-09-17T18:27:11.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.710+0000 7fce180e5640 1 -- 172.21.15.121:0/1514304540 shutdown_connections 2024-09-17T18:27:11.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.710+0000 7fce180e5640 1 --2- 172.21.15.121:0/1514304540 >> v2:172.21.15.173:6812/2423786037 conn(0x7fcdfc040170 0x7fcdfc042630 unknown :-1 s=CLOSED pgs=3993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:11.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.710+0000 7fce180e5640 1 --2- 172.21.15.121:0/1514304540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce100a5fe0 0x7fce101245e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:11.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.710+0000 7fce180e5640 1 --2- 172.21.15.121:0/1514304540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce101389c0 0x7fce10124b20 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:11.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.710+0000 7fce180e5640 1 --2- 172.21.15.121:0/1514304540 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fce100a5c00 0x7fce101240a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:11.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.710+0000 7fce180e5640 1 -- 172.21.15.121:0/1514304540 >> 172.21.15.121:0/1514304540 conn(0x7fce100ab9b0 msgr2=0x7fce101382a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:27:11.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.710+0000 7fce180e5640 1 -- 172.21.15.121:0/1514304540 shutdown_connections 2024-09-17T18:27:11.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.710+0000 7fce180e5640 1 -- 172.21.15.121:0/1514304540 wait complete. 2024-09-17T18:27:11.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:11.710+0000 7fce180e5640 1 librados: shutdown 2024-09-17T18:27:14.723 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:27:14.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.829+0000 7f87b5c50640 10 monclient: get_monmap_and_config 2024-09-17T18:27:14.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.829+0000 7f87b5c50640 10 monclient: build_initial_monmap 2024-09-17T18:27:14.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.829+0000 7f87b5c50640 10 monclient: monmap: 2024-09-17T18:27:14.831 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:27:14.831 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:14.831 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:27:14.829900+0000 2024-09-17T18:27:14.831 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:27:14.829900+0000 2024-09-17T18:27:14.831 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:27:14.831 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:14.831 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:27:14.832 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:27:14.832 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:27:14.832 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:27:14.832 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:27:14.832 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:14.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.830+0000 7f87b5c50640 1 Processor -- start 2024-09-17T18:27:14.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.830+0000 7f87b5c50640 1 -- start start 2024-09-17T18:27:14.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.830+0000 7f87b5c50640 10 monclient: init 2024-09-17T18:27:14.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.830+0000 7f87b5c50640 10 monclient: _reopen_session rank -1 2024-09-17T18:27:14.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.830+0000 7f87b5c50640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-17T18:27:14.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.830+0000 7f87b5c50640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f87b00a5c00 addr v1:172.21.15.121:6789/0 2024-09-17T18:27:14.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.830+0000 7f87b5c50640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f87b00a5fe0 addr v1:172.21.15.173:6789/0 2024-09-17T18:27:14.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.830+0000 7f87b5c50640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f87b013ec10 0x7f87b0141000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:14.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.830+0000 7f87b5c50640 10 monclient(hunting): picked mon.noname-c con 0x7f87b013ec10 addr v2:172.21.15.173:3300/0 2024-09-17T18:27:14.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.830+0000 7f87b5c50640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87b00a8a80 con 0x7f87b00a5c00 2024-09-17T18:27:14.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.830+0000 7f87b5c50640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87b00a74d0 con 0x7f87b00a5fe0 2024-09-17T18:27:14.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.830+0000 7f87b5c50640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:14.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.830+0000 7f87b5c50640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f87b00a7110 con 0x7f87b013ec10 2024-09-17T18:27:14.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.830+0000 7f87b5c50640 10 monclient(hunting): _renew_subs 2024-09-17T18:27:14.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.830+0000 7f87b5c50640 10 monclient(hunting): authenticate will time out at 1238.546747s 2024-09-17T18:27:14.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.830+0000 7f87af7fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f87b00a5c00 0x7f87b009d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:35554/0 (socket says 172.21.15.121:35554) 2024-09-17T18:27:14.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.830+0000 7f87af7fe640 1 -- 172.21.15.121:0/3696034451 learned_addr learned my addr 172.21.15.121:0/3696034451 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:27:14.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.831+0000 7f87affff640 1 --2- 172.21.15.121:0/3696034451 >> v2:172.21.15.173:3300/0 conn(0x7f87b013ec10 0x7f87b0141000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:14.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.831+0000 7f87affff640 10 monclient(hunting): get_auth_request con 0x7f87b013ec10 auth_method 0 2024-09-17T18:27:14.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.831+0000 7f87affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:14.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.831+0000 7f87affff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:14.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.831+0000 7f87affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:14.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.831+0000 7f87affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:14.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.831+0000 7f87affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:14.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.831+0000 7f87affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:14.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.831+0000 7f87affff640 10 monclient(hunting): handle_auth_done global_id 24769 payload 274 2024-09-17T18:27:14.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.831+0000 7f87affff640 1 -- 172.21.15.121:0/3696034451 >> v1:172.21.15.173:6789/0 conn(0x7f87b00a5fe0 legacy=0x7f87b00a63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:27:14.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.831+0000 7f87affff640 1 -- 172.21.15.121:0/3696034451 >> v1:172.21.15.121:6789/0 conn(0x7f87b00a5c00 legacy=0x7f87b009d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:27:14.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.831+0000 7f87affff640 10 monclient: _finish_hunting 0 2024-09-17T18:27:14.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.831+0000 7f87affff640 1 monclient: found mon.noname-c 2024-09-17T18:27:14.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.831+0000 7f87affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:27:14.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.831+0000 7f87affff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.173:3300/0 2024-09-17T18:27:14.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.831+0000 7f87affff640 1 -- 172.21.15.121:0/3696034451 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87b009cb90 con 0x7f87b013ec10 2024-09-17T18:27:14.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.834+0000 7f87affff640 1 -- 172.21.15.121:0/3696034451 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87b013ec10 msgr2=0x7f87b0141000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:14.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.834+0000 7f87affff640 1 --2- 172.21.15.121:0/3696034451 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87b013ec10 0x7f87b0141000 secure :-1 s=READY pgs=7425 cs=0 l=1 rev1=1 crypto rx=0x7f87a8009d90 tx=0x7f87a802a490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7095794e0849b646 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:14.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.834+0000 7f87ae7fc640 1 -- 172.21.15.121:0/3696034451 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a8007a30 con 0x7f87b013ec10 2024-09-17T18:27:14.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.834+0000 7f87ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:14.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.834+0000 7f87ae7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:27:14.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.834+0000 7f87ae7fc640 10 monclient: dump: 2024-09-17T18:27:14.843 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:14.843 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:14.843 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:14.843 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:14.844 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:14.844 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:14.844 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:14.844 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:14.844 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:14.844 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:14.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.834+0000 7f87ae7fc640 10 monclient: _finish_auth 0 2024-09-17T18:27:14.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.834+0000 7f87ae7fc640 10 monclient: _check_auth_tickets 2024-09-17T18:27:14.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.834+0000 7f87ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:27:14.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.834+0000 7f87ae7fc640 1 -- 172.21.15.121:0/3696034451 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87a8007bd0 con 0x7f87b013ec10 2024-09-17T18:27:14.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.834+0000 7f87b5c50640 5 monclient: authenticate success, global_id 24769 2024-09-17T18:27:14.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.834+0000 7f87b5c50640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:27:14.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.834+0000 7f87ae7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:27:14.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.834+0000 7f87b5c50640 10 monclient: get_monmap_and_config success 2024-09-17T18:27:14.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.834+0000 7f87ae7fc640 1 -- 172.21.15.121:0/3696034451 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a8031400 con 0x7f87b013ec10 2024-09-17T18:27:14.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.834+0000 7f87ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:14.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.834+0000 7f87b5c50640 4 set_mon_vals no callback set 2024-09-17T18:27:14.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.834+0000 7f87ae7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:27:14.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.834+0000 7f87ae7fc640 10 monclient: dump: 2024-09-17T18:27:14.845 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:14.845 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:14.845 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:14.845 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:14.845 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:14.845 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:14.845 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:14.846 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:14.846 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:14.846 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:14.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.835+0000 7f87b5c50640 10 monclient: shutdown 2024-09-17T18:27:14.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.835+0000 7f87b5c50640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:27:14.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.835+0000 7f87b5c50640 1 -- 172.21.15.121:0/3696034451 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87b013ec10 msgr2=0x7f87b0141000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:14.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.835+0000 7f87b5c50640 1 --2- 172.21.15.121:0/3696034451 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87b013ec10 0x7f87b0141000 secure :-1 s=READY pgs=7425 cs=0 l=1 rev1=1 crypto rx=0x7f87a8009d90 tx=0x7f87a802a490 comp rx=0 tx=0).stop 2024-09-17T18:27:14.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.835+0000 7f87b5c50640 1 -- 172.21.15.121:0/3696034451 shutdown_connections 2024-09-17T18:27:14.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.835+0000 7f87b5c50640 1 --2- 172.21.15.121:0/3696034451 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87b013ec10 0x7f87b0141000 unknown :-1 s=CLOSED pgs=7425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:14.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.835+0000 7f87b5c50640 1 -- 172.21.15.121:0/3696034451 >> 172.21.15.121:0/3696034451 conn(0x7f87b00ab9b0 msgr2=0x7f87b00a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:27:14.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.835+0000 7f87b5c50640 1 -- 172.21.15.121:0/3696034451 shutdown_connections 2024-09-17T18:27:14.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.835+0000 7f87b5c50640 1 -- 172.21.15.121:0/3696034451 wait complete. 2024-09-17T18:27:14.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.835+0000 7f87b5c50640 10 monclient: build_initial_monmap 2024-09-17T18:27:14.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.835+0000 7f87b5c50640 10 monclient: monmap: 2024-09-17T18:27:14.847 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:27:14.847 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:27:14.847 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:27:14.847 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:27:14.847 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:27:14.847 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:14.847 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:27:14.847 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:27:14.847 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:27:14.847 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:14.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.836+0000 7f87b5c50640 1 librados: starting msgr at 2024-09-17T18:27:14.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.836+0000 7f87b5c50640 1 librados: starting objecter 2024-09-17T18:27:14.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.836+0000 7f87b5c50640 1 Processor -- start 2024-09-17T18:27:14.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.836+0000 7f87b5c50640 1 -- start start 2024-09-17T18:27:14.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.836+0000 7f87b5c50640 1 librados: setting wanted keys 2024-09-17T18:27:14.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.836+0000 7f87b5c50640 1 librados: calling monclient init 2024-09-17T18:27:14.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.836+0000 7f87b5c50640 10 monclient: init 2024-09-17T18:27:14.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.836+0000 7f87b5c50640 10 monclient: _reopen_session rank -1 2024-09-17T18:27:14.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.836+0000 7f87b5c50640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T18:27:14.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.836+0000 7f87b5c50640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87b00a5c00 0x7f87b01385a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:14.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.836+0000 7f87b5c50640 10 monclient(hunting): picked mon.noname-c con 0x7f87b00a5c00 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:27:14.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.836+0000 7f87b5c50640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87b00a5fe0 0x7f87b0138ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:14.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.836+0000 7f87b5c50640 10 monclient(hunting): picked mon.noname-a con 0x7f87b00a5fe0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:27:14.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.836+0000 7f87b5c50640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87b013ec10 0x7f87b013b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:14.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.836+0000 7f87b5c50640 10 monclient(hunting): picked mon.noname-b con 0x7f87b013ec10 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:27:14.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.836+0000 7f87b5c50640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:14.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.836+0000 7f87b5c50640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f87b0141540 con 0x7f87b00a5fe0 2024-09-17T18:27:14.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.836+0000 7f87b5c50640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:14.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.836+0000 7f87b5c50640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f87b009ced0 con 0x7f87b013ec10 2024-09-17T18:27:14.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.836+0000 7f87b5c50640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:14.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.836+0000 7f87b5c50640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f87b01416c0 con 0x7f87b00a5c00 2024-09-17T18:27:14.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.836+0000 7f87b5c50640 10 monclient(hunting): _renew_subs 2024-09-17T18:27:14.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.836+0000 7f87b5c50640 10 monclient(hunting): authenticate will time out at 1238.552867s 2024-09-17T18:27:14.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.836+0000 7f87aeffd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87b00a5fe0 0x7f87b0138ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:14.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.836+0000 7f87aeffd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87b00a5fe0 0x7f87b0138ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56492/0 (socket says 172.21.15.121:56492) 2024-09-17T18:27:14.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.836+0000 7f87aeffd640 1 -- 172.21.15.121:0/2463067485 learned_addr learned my addr 172.21.15.121:0/2463067485 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:27:14.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.836+0000 7f87aeffd640 10 monclient(hunting): get_auth_request con 0x7f87b00a5fe0 auth_method 0 2024-09-17T18:27:14.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.836+0000 7f87af7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87b00a5c00 0x7f87b01385a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:14.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.837+0000 7f87aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:14.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.837+0000 7f87aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:14.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.837+0000 7f87aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:14.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.837+0000 7f87af7fe640 10 monclient(hunting): get_auth_request con 0x7f87b00a5c00 auth_method 0 2024-09-17T18:27:14.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.837+0000 7f87af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:14.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.837+0000 7f87af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:14.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.837+0000 7f87af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:14.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.837+0000 7f87aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:14.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.837+0000 7f87aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:14.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.837+0000 7f87aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:14.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.837+0000 7f87af7fe640 1 -- 172.21.15.121:0/2463067485 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87b00a5c00 msgr2=0x7f87b01385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T18:27:14.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.837+0000 7f87af7fe640 1 -- 172.21.15.121:0/2463067485 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87b00a5c00 msgr2=0x7f87b01385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:27:14.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.837+0000 7f87af7fe640 1 --2- 172.21.15.121:0/2463067485 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87b00a5c00 0x7f87b01385a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:27:14.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.837+0000 7f87af7fe640 1 --2- 172.21.15.121:0/2463067485 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87b00a5c00 0x7f87b01385a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:27:14.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.837+0000 7f87affff640 1 --2- 172.21.15.121:0/2463067485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87b013ec10 0x7f87b013b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:14.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.837+0000 7f87affff640 10 monclient(hunting): get_auth_request con 0x7f87b013ec10 auth_method 0 2024-09-17T18:27:14.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.837+0000 7f87affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:14.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.837+0000 7f87affff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:14.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.837+0000 7f87affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:14.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.837+0000 7f87aeffd640 10 monclient(hunting): handle_auth_done global_id 44454 payload 824 2024-09-17T18:27:14.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.837+0000 7f87aeffd640 1 -- 172.21.15.121:0/2463067485 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87b00a5c00 msgr2=0x7f87b01385a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:27:14.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.837+0000 7f87aeffd640 1 --2- 172.21.15.121:0/2463067485 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87b00a5c00 0x7f87b01385a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:14.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.837+0000 7f87aeffd640 1 -- 172.21.15.121:0/2463067485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87b013ec10 msgr2=0x7f87b013b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:14.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.837+0000 7f87aeffd640 1 --2- 172.21.15.121:0/2463067485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87b013ec10 0x7f87b013b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:14.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.837+0000 7f87aeffd640 10 monclient: _finish_hunting 0 2024-09-17T18:27:14.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.837+0000 7f87aeffd640 1 monclient: found mon.noname-a 2024-09-17T18:27:14.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.837+0000 7f87aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:27:14.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.837+0000 7f87aeffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:27:14.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.837+0000 7f87aeffd640 1 -- 172.21.15.121:0/2463067485 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87b0121b30 con 0x7f87b00a5fe0 2024-09-17T18:27:14.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.839+0000 7f87aeffd640 1 -- 172.21.15.121:0/2463067485 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87b00a5fe0 msgr2=0x7f87b0138ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:14.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.839+0000 7f87aeffd640 1 --2- 172.21.15.121:0/2463067485 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87b00a5fe0 0x7f87b0138ae0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f87a400c430 tx=0x7f87a400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:14.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.839+0000 7f8797fff640 10 client.?.objecter ms_handle_connect 0x7f87b00a5fe0 2024-09-17T18:27:14.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.839+0000 7f8797fff640 10 client.?.objecter resend_mon_ops 2024-09-17T18:27:14.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.839+0000 7f8797fff640 1 -- 172.21.15.121:0/2463067485 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a40119a0 con 0x7f87b00a5fe0 2024-09-17T18:27:14.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.839+0000 7f8797fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:14.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.839+0000 7f8797fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:27:14.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.839+0000 7f8797fff640 10 monclient: dump: 2024-09-17T18:27:14.853 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:14.853 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:14.853 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:14.853 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:14.853 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:14.853 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:14.853 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:14.853 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:14.853 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:14.853 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:14.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.839+0000 7f8797fff640 10 monclient: _finish_auth 0 2024-09-17T18:27:14.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.839+0000 7f8797fff640 10 monclient: _check_auth_tickets 2024-09-17T18:27:14.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.839+0000 7f8797fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:27:14.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.839+0000 7f87b5c50640 5 monclient: authenticate success, global_id 44454 2024-09-17T18:27:14.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.839+0000 7f87b5c50640 10 monclient: _renew_subs 2024-09-17T18:27:14.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.839+0000 7f87b5c50640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:27:14.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.839+0000 7f87b5c50640 1 -- 172.21.15.121:0/2463067485 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87b01411a0 con 0x7f87b00a5fe0 2024-09-17T18:27:14.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.839+0000 7f8797fff640 1 -- 172.21.15.121:0/2463067485 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87a4017070 con 0x7f87b00a5fe0 2024-09-17T18:27:14.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.839+0000 7f8797fff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:27:14.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.839+0000 7f8797fff640 1 -- 172.21.15.121:0/2463067485 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a4011d80 con 0x7f87b00a5fe0 2024-09-17T18:27:14.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.839+0000 7f8797fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:14.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.839+0000 7f8797fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:27:14.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.839+0000 7f8797fff640 10 monclient: dump: 2024-09-17T18:27:14.854 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:14.854 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:14.854 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:14.854 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:14.855 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:14.855 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:14.855 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:14.855 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:14.855 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:14.855 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:14.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.839+0000 7f87b5c50640 10 client.44454.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:27:14.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.839+0000 7f87adffb640 4 set_mon_vals no callback set 2024-09-17T18:27:14.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.839+0000 7f87b5c50640 10 monclient: _renew_subs 2024-09-17T18:27:14.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.839+0000 7f87b5c50640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:27:14.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.839+0000 7f87b5c50640 1 -- 172.21.15.121:0/2463067485 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87b01220f0 con 0x7f87b00a5fe0 2024-09-17T18:27:14.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.839+0000 7f87b5c50640 1 librados: init done 2024-09-17T18:27:14.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.840+0000 7f87b5c50640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:27:14.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.840+0000 7f87b5c50640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:27:14.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.840+0000 7f87b5c50640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:27:14.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.840+0000 7f87b5c50640 1 -- 172.21.15.121:0/2463067485 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87b009d8c0 con 0x7f87b00a5fe0 2024-09-17T18:27:14.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.840+0000 7f8797fff640 1 -- 172.21.15.121:0/2463067485 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f87a401c460 con 0x7f87b00a5fe0 2024-09-17T18:27:14.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.841+0000 7f8797fff640 1 --2- 172.21.15.121:0/2463067485 >> v2:172.21.15.173:6812/2423786037 conn(0x7f879c040170 0x7f879c042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:14.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.841+0000 7f8797fff640 1 -- 172.21.15.121:0/2463067485 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(341..341 src has 177..341) ==== 5358+0+0 (secure 0 0 0) 0x7f87a4053670 con 0x7f87b00a5fe0 2024-09-17T18:27:14.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.841+0000 7f8797fff640 10 client.44454.objecter ms_dispatch 0x7f87b0000b90 osd_map(341..341 src has 177..341) 2024-09-17T18:27:14.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.841+0000 7f8797fff640 3 client.44454.objecter handle_osd_map got epochs [341,341] > 0 2024-09-17T18:27:14.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.841+0000 7f8797fff640 3 client.44454.objecter handle_osd_map decoding full epoch 341 2024-09-17T18:27:14.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.841+0000 7f8797fff640 20 client.44454.objecter dump_active .. 0 homeless 2024-09-17T18:27:14.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.841+0000 7f87af7fe640 1 --2- 172.21.15.121:0/2463067485 >> v2:172.21.15.173:6812/2423786037 conn(0x7f879c040170 0x7f879c042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:14.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.841+0000 7f87af7fe640 10 monclient: get_auth_request con 0x7f879c040170 auth_method 0 2024-09-17T18:27:14.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.843+0000 7f8797fff640 10 client.44454.objecter ms_handle_connect 0x7f879c040170 2024-09-17T18:27:14.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.843+0000 7f87af7fe640 1 --2- 172.21.15.121:0/2463067485 >> v2:172.21.15.173:6812/2423786037 conn(0x7f879c040170 0x7f879c042630 secure :-1 s=READY pgs=3994 cs=0 l=1 rev1=1 crypto rx=0x7f87a0007fd0 tx=0x7f87a0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:14.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.845+0000 7f8797fff640 1 -- 172.21.15.121:0/2463067485 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f87a4051e30 con 0x7f87b00a5fe0 2024-09-17T18:27:14.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.845+0000 7f8797fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:27:14.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:14.845+0000 7f8797fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:27:15.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.085+0000 7f87b5c50640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:27:15.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.085+0000 7f87b5c50640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:27:15.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.085+0000 7f87b5c50640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:27:15.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.085+0000 7f87b5c50640 1 -- 172.21.15.121:0/2463067485 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f87b0121850 con 0x7f87b00a5fe0 2024-09-17T18:27:15.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.086+0000 7f8797fff640 1 -- 172.21.15.121:0/2463067485 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f87a401c930 con 0x7f87b00a5fe0 2024-09-17T18:27:15.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.086+0000 7f8797fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:27:15.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.086+0000 7f8797fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:27:15.088 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:27:15.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.088+0000 7f87b5c50640 10 librados: watch_flush enter 2024-09-17T18:27:15.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.088+0000 7f87b5c50640 10 librados: watch_flush exit 2024-09-17T18:27:15.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.088+0000 7f87b5c50640 20 client.44454.objecter shutdown clearing up homeless session... 2024-09-17T18:27:15.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.088+0000 7f87b5c50640 10 client.44454.objecter successfully canceled tick 2024-09-17T18:27:15.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.088+0000 7f87b5c50640 1 -- 172.21.15.121:0/2463067485 >> v2:172.21.15.173:6812/2423786037 conn(0x7f879c040170 msgr2=0x7f879c042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:15.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.088+0000 7f87b5c50640 1 --2- 172.21.15.121:0/2463067485 >> v2:172.21.15.173:6812/2423786037 conn(0x7f879c040170 0x7f879c042630 secure :-1 s=READY pgs=3994 cs=0 l=1 rev1=1 crypto rx=0x7f87a0007fd0 tx=0x7f87a0009040 comp rx=0 tx=0).stop 2024-09-17T18:27:15.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.088+0000 7f87b5c50640 10 monclient: shutdown 2024-09-17T18:27:15.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.088+0000 7f87b5c50640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:27:15.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.088+0000 7f87b5c50640 1 -- 172.21.15.121:0/2463067485 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87b00a5fe0 msgr2=0x7f87b0138ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:15.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.088+0000 7f87b5c50640 1 --2- 172.21.15.121:0/2463067485 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87b00a5fe0 0x7f87b0138ae0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f87a400c430 tx=0x7f87a400c8f0 comp rx=0 tx=0).stop 2024-09-17T18:27:15.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.088+0000 7f87b5c50640 1 -- 172.21.15.121:0/2463067485 shutdown_connections 2024-09-17T18:27:15.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.088+0000 7f87b5c50640 1 --2- 172.21.15.121:0/2463067485 >> v2:172.21.15.173:6812/2423786037 conn(0x7f879c040170 0x7f879c042630 unknown :-1 s=CLOSED pgs=3994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:15.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.088+0000 7f87b5c50640 1 --2- 172.21.15.121:0/2463067485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87b013ec10 0x7f87b013b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:15.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.088+0000 7f87b5c50640 1 --2- 172.21.15.121:0/2463067485 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87b00a5fe0 0x7f87b0138ae0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:15.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.088+0000 7f87b5c50640 1 --2- 172.21.15.121:0/2463067485 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87b00a5c00 0x7f87b01385a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:15.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.088+0000 7f87b5c50640 1 -- 172.21.15.121:0/2463067485 >> 172.21.15.121:0/2463067485 conn(0x7f87b00ab9b0 msgr2=0x7f87b00a9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:27:15.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.088+0000 7f87b5c50640 1 -- 172.21.15.121:0/2463067485 shutdown_connections 2024-09-17T18:27:15.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.088+0000 7f87b5c50640 1 -- 172.21.15.121:0/2463067485 wait complete. 2024-09-17T18:27:15.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.088+0000 7f87b5c50640 1 librados: shutdown 2024-09-17T18:27:15.100 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T18:27:15.100 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T18:27:15.101 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T18:27:15.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.196+0000 7f9864b64640 10 monclient: get_monmap_and_config 2024-09-17T18:27:15.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.196+0000 7f9864b64640 10 monclient: build_initial_monmap 2024-09-17T18:27:15.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.196+0000 7f9864b64640 10 monclient: monmap: 2024-09-17T18:27:15.198 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:27:15.198 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:15.198 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:27:15.196773+0000 2024-09-17T18:27:15.198 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:27:15.196773+0000 2024-09-17T18:27:15.198 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:27:15.198 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:15.198 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:27:15.198 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:27:15.199 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:27:15.199 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:27:15.199 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:27:15.199 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:15.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.196+0000 7f9864b64640 1 Processor -- start 2024-09-17T18:27:15.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.196+0000 7f9864b64640 1 -- start start 2024-09-17T18:27:15.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.196+0000 7f9864b64640 10 monclient: init 2024-09-17T18:27:15.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.197+0000 7f9864b64640 10 monclient: _reopen_session rank -1 2024-09-17T18:27:15.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.197+0000 7f9864b64640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-17T18:27:15.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.197+0000 7f9864b64640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f986009e550 addr v1:172.21.15.121:6789/0 2024-09-17T18:27:15.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.197+0000 7f9864b64640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f986009f040 addr v1:172.21.15.173:6789/0 2024-09-17T18:27:15.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.197+0000 7f9864b64640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f986013bf70 0x7f986013e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:15.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.197+0000 7f9864b64640 10 monclient(hunting): picked mon.noname-b con 0x7f986013bf70 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:27:15.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.197+0000 7f9864b64640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98600a4460 con 0x7f986009e550 2024-09-17T18:27:15.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.197+0000 7f9864b64640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98600a2b60 con 0x7f986009f040 2024-09-17T18:27:15.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.197+0000 7f9864b64640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:15.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.197+0000 7f9864b64640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f98600a4950 con 0x7f986013bf70 2024-09-17T18:27:15.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.197+0000 7f9864b64640 10 monclient(hunting): _renew_subs 2024-09-17T18:27:15.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.197+0000 7f9864b64640 10 monclient(hunting): authenticate will time out at 1238.913600s 2024-09-17T18:27:15.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.197+0000 7f985e59c640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f986009e550 0x7f986009e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:35568/0 (socket says 172.21.15.121:35568) 2024-09-17T18:27:15.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.197+0000 7f985e59c640 1 -- 172.21.15.121:0/2785410666 learned_addr learned my addr 172.21.15.121:0/2785410666 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:27:15.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.197+0000 7f985ed9d640 1 --2- 172.21.15.121:0/2785410666 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f986013bf70 0x7f986013e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:15.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.197+0000 7f985ed9d640 10 monclient(hunting): get_auth_request con 0x7f986013bf70 auth_method 0 2024-09-17T18:27:15.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.197+0000 7f985ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:15.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.197+0000 7f985ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:15.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.197+0000 7f985ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:15.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.197+0000 7f985ed9d640 1 -- 172.21.15.121:0/2785410666 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f986013bf70 msgr2=0x7f986013e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T18:27:15.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.197+0000 7f985ed9d640 1 -- 172.21.15.121:0/2785410666 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f986013bf70 msgr2=0x7f986013e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:27:15.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.197+0000 7f985ed9d640 1 --2- 172.21.15.121:0/2785410666 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f986013bf70 0x7f986013e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:27:15.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.197+0000 7f985ed9d640 1 --2- 172.21.15.121:0/2785410666 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f986013bf70 0x7f986013e360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:27:15.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.199+0000 7f985e59c640 1 -- 172.21.15.121:0/2785410666 >> v1:172.21.15.121:6789/0 conn(0x7f986009e550 legacy=0x7f986009e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:15.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985dd9b640 1 -- 172.21.15.121:0/2785410666 >> v1:172.21.15.173:6789/0 conn(0x7f986009f040 legacy=0x7f986013a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:15.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985d59a640 1 -- 172.21.15.121:0/2785410666 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3891019679 0 0) 0x7f98600a4460 con 0x7f986009e550 2024-09-17T18:27:15.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985d59a640 10 monclient(hunting): _negotiate 2024-09-17T18:27:15.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985d59a640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:15.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:15.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985d59a640 10 monclient(hunting): my global_id is 44457 2024-09-17T18:27:15.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985d59a640 1 -- 172.21.15.121:0/2785410666 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9848003520 con 0x7f986009e550 2024-09-17T18:27:15.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985d59a640 1 -- 172.21.15.121:0/2785410666 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1517298403 0 0) 0x7f9848003520 con 0x7f986009e550 2024-09-17T18:27:15.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985d59a640 1 -- 172.21.15.121:0/2785410666 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f986013bf70 msgr2=0x7f986013e360 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:27:15.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985d59a640 1 --2- 172.21.15.121:0/2785410666 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f986013bf70 0x7f986013e360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:15.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985d59a640 1 -- 172.21.15.121:0/2785410666 >> v1:172.21.15.173:6789/0 conn(0x7f986009f040 legacy=0x7f986013a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:15.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985d59a640 10 monclient: _finish_hunting 0 2024-09-17T18:27:15.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985d59a640 1 monclient: found mon.noname-a-legacy 2024-09-17T18:27:15.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:27:15.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985d59a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.121:6789/0 2024-09-17T18:27:15.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985d59a640 1 -- 172.21.15.121:0/2785410666 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f986009f840 con 0x7f986009e550 2024-09-17T18:27:15.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985d59a640 10 monclient: _finish_auth 0 2024-09-17T18:27:15.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985d59a640 10 monclient: _check_auth_tickets 2024-09-17T18:27:15.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:27:15.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985d59a640 1 -- 172.21.15.121:0/2785410666 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4002019445 0 0) 0x7f98540019a0 con 0x7f986009e550 2024-09-17T18:27:15.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:15.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985d59a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T18:27:15.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985d59a640 10 monclient: dump: 2024-09-17T18:27:15.214 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:15.214 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:15.214 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:15.214 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:15.214 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:15.215 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:15.215 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:15.215 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:15.215 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:15.215 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:15.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985d59a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] but i'm connected to v1:172.21.15.121:6789/0, reconnecting 2024-09-17T18:27:15.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985d59a640 10 monclient: _reopen_session rank -1 2024-09-17T18:27:15.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985d59a640 1 -- 172.21.15.121:0/2785410666 >> v1:172.21.15.121:6789/0 conn(0x7f986009e550 legacy=0x7f986009e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:15.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985d59a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:27:15.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985d59a640 1 --2- 172.21.15.121:0/2785410666 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9848003af0 0x7f9848003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:15.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985d59a640 10 monclient(hunting): picked mon.a con 0x7f9848003af0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:27:15.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985d59a640 1 --2- 172.21.15.121:0/2785410666 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9848004800 0x7f9848004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:15.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985d59a640 10 monclient(hunting): picked mon.b con 0x7f9848004800 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:27:15.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985d59a640 1 --2- 172.21.15.121:0/2785410666 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f986013bf70 0x7f984800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:15.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985d59a640 10 monclient(hunting): picked mon.c con 0x7f986013bf70 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:27:15.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985d59a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:15.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985d59a640 1 -- 172.21.15.121:0/2785410666 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9848005670 con 0x7f9848003af0 2024-09-17T18:27:15.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985d59a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:15.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985d59a640 1 -- 172.21.15.121:0/2785410666 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f984800bf30 con 0x7f9848004800 2024-09-17T18:27:15.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985d59a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:15.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985d59a640 1 -- 172.21.15.121:0/2785410666 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f984800c130 con 0x7f986013bf70 2024-09-17T18:27:15.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985d59a640 10 monclient(hunting): _renew_subs 2024-09-17T18:27:15.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985d59a640 10 monclient(hunting): _finish_auth 0 2024-09-17T18:27:15.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985e59c640 1 --2- 172.21.15.121:0/2785410666 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9848003af0 0x7f9848003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:15.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985e59c640 10 monclient(hunting): get_auth_request con 0x7f9848003af0 auth_method 0 2024-09-17T18:27:15.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:15.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:15.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:27:15.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985ed9d640 1 --2- 172.21.15.121:0/2785410666 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f986013bf70 0x7f984800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:15.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985ed9d640 10 monclient(hunting): get_auth_request con 0x7f986013bf70 auth_method 0 2024-09-17T18:27:15.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:15.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:15.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:27:15.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985dd9b640 1 --2- 172.21.15.121:0/2785410666 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9848004800 0x7f9848004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:15.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:15.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:15.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985e59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:27:15.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985dd9b640 10 monclient(hunting): get_auth_request con 0x7f9848004800 auth_method 0 2024-09-17T18:27:15.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:15.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:15.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985dd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:27:15.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:15.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:15.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:27:15.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:15.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:15.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:27:15.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985e59c640 10 monclient(hunting): handle_auth_done global_id 44457 payload 293 2024-09-17T18:27:15.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985e59c640 1 -- 172.21.15.121:0/2785410666 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f986013bf70 msgr2=0x7f984800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:27:15.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985e59c640 1 --2- 172.21.15.121:0/2785410666 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f986013bf70 0x7f984800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:15.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985e59c640 1 -- 172.21.15.121:0/2785410666 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9848004800 msgr2=0x7f9848004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:27:15.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985e59c640 1 --2- 172.21.15.121:0/2785410666 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9848004800 0x7f9848004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:15.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985e59c640 10 monclient: _finish_hunting 0 2024-09-17T18:27:15.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985e59c640 1 monclient: found mon.a 2024-09-17T18:27:15.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:27:15.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985e59c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:27:15.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985e59c640 1 -- 172.21.15.121:0/2785410666 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9848003520 con 0x7f9848003af0 2024-09-17T18:27:15.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985e59c640 10 monclient: _finish_auth 0 2024-09-17T18:27:15.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985e59c640 10 monclient: _check_auth_tickets 2024-09-17T18:27:15.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f985e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:27:15.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f9864b64640 5 monclient: authenticate success, global_id 44457 2024-09-17T18:27:15.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.200+0000 7f9864b64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:27:15.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.203+0000 7f985e59c640 1 -- 172.21.15.121:0/2785410666 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9848003af0 msgr2=0x7f9848003f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:15.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.203+0000 7f985e59c640 1 --2- 172.21.15.121:0/2785410666 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9848003af0 0x7f9848003f10 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f98540014d0 tx=0x7f9848005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c461456b8aa76d80 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:15.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.203+0000 7f985d59a640 1 -- 172.21.15.121:0/2785410666 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9854036070 con 0x7f9848003af0 2024-09-17T18:27:15.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.203+0000 7f985d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:15.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.203+0000 7f985d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:27:15.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.203+0000 7f985d59a640 10 monclient: dump: 2024-09-17T18:27:15.219 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:15.219 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:15.220 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:15.220 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:15.220 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:15.220 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:15.220 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:15.220 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:15.220 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:15.220 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:15.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.203+0000 7f985d59a640 1 -- 172.21.15.121:0/2785410666 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9854005af0 con 0x7f9848003af0 2024-09-17T18:27:15.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.203+0000 7f9864b64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:27:15.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.203+0000 7f985d59a640 10 monclient: handle_config config(0 keys) 2024-09-17T18:27:15.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.203+0000 7f985d59a640 1 -- 172.21.15.121:0/2785410666 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f985400c0e0 con 0x7f9848003af0 2024-09-17T18:27:15.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.203+0000 7f9864b64640 10 monclient: get_monmap_and_config success 2024-09-17T18:27:15.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.203+0000 7f9864b64640 4 set_mon_vals no callback set 2024-09-17T18:27:15.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.203+0000 7f985d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:15.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.203+0000 7f985d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:27:15.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.203+0000 7f985d59a640 10 monclient: dump: 2024-09-17T18:27:15.221 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:15.221 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:15.221 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:15.221 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:15.221 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:15.221 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:15.221 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:15.221 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:15.221 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:15.221 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:15.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.204+0000 7f9864b64640 10 monclient: shutdown 2024-09-17T18:27:15.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.204+0000 7f9864b64640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:27:15.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.204+0000 7f9864b64640 1 -- 172.21.15.121:0/2785410666 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9848003af0 msgr2=0x7f9848003f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:15.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.204+0000 7f9864b64640 1 --2- 172.21.15.121:0/2785410666 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9848003af0 0x7f9848003f10 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f98540014d0 tx=0x7f9848005830 comp rx=0 tx=0).stop 2024-09-17T18:27:15.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.204+0000 7f985ed9d640 1 -- 172.21.15.121:0/2785410666 reap_dead start 2024-09-17T18:27:15.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.204+0000 7f9864b64640 1 -- 172.21.15.121:0/2785410666 shutdown_connections 2024-09-17T18:27:15.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.204+0000 7f9864b64640 1 -- 172.21.15.121:0/2785410666 >> 172.21.15.121:0/2785410666 conn(0x7f98600a7830 msgr2=0x7f98600a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:27:15.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.204+0000 7f9864b64640 1 -- 172.21.15.121:0/2785410666 shutdown_connections 2024-09-17T18:27:15.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.204+0000 7f9864b64640 1 -- 172.21.15.121:0/2785410666 wait complete. 2024-09-17T18:27:15.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.205+0000 7f9864b64640 10 monclient: build_initial_monmap 2024-09-17T18:27:15.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.205+0000 7f9864b64640 10 monclient: monmap: 2024-09-17T18:27:15.222 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:27:15.222 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:27:15.222 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:27:15.222 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:27:15.222 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:27:15.223 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:15.223 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:27:15.223 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:27:15.223 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:27:15.223 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:15.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.205+0000 7f9864b64640 1 librados: starting msgr at 2024-09-17T18:27:15.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.205+0000 7f9864b64640 1 librados: starting objecter 2024-09-17T18:27:15.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.205+0000 7f9864b64640 1 Processor -- start 2024-09-17T18:27:15.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.205+0000 7f9864b64640 1 -- start start 2024-09-17T18:27:15.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.205+0000 7f9864b64640 1 librados: setting wanted keys 2024-09-17T18:27:15.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.205+0000 7f9864b64640 1 librados: calling monclient init 2024-09-17T18:27:15.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.205+0000 7f9864b64640 10 monclient: init 2024-09-17T18:27:15.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.205+0000 7f9864b64640 10 monclient: _reopen_session rank -1 2024-09-17T18:27:15.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.205+0000 7f9864b64640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:27:15.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.205+0000 7f9864b64640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9848003af0 0x7f98601cd300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:15.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.205+0000 7f9864b64640 10 monclient(hunting): picked mon.noname-a con 0x7f9848003af0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:27:15.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.205+0000 7f9864b64640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9860138ae0 0x7f9860138f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:15.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.205+0000 7f9864b64640 10 monclient(hunting): picked mon.noname-b con 0x7f9860138ae0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:27:15.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.205+0000 7f9864b64640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f986006f3a0 0x7f986006f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:15.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.205+0000 7f9864b64640 10 monclient(hunting): picked mon.noname-c con 0x7f986006f3a0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:27:15.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.205+0000 7f9864b64640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:15.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.205+0000 7f9864b64640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f986009f490 con 0x7f9848003af0 2024-09-17T18:27:15.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.205+0000 7f9864b64640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:15.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.205+0000 7f9864b64640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f986009dce0 con 0x7f9860138ae0 2024-09-17T18:27:15.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.205+0000 7f9864b64640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:15.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.205+0000 7f9864b64640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f98601258d0 con 0x7f986006f3a0 2024-09-17T18:27:15.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.205+0000 7f9864b64640 10 monclient(hunting): _renew_subs 2024-09-17T18:27:15.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.205+0000 7f9864b64640 10 monclient(hunting): authenticate will time out at 1238.921941s 2024-09-17T18:27:15.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.205+0000 7f985e59c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9848003af0 0x7f98601cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:15.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.206+0000 7f985ed9d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f986006f3a0 0x7f986006f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:15.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.206+0000 7f985e59c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9848003af0 0x7f98601cd300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56514/0 (socket says 172.21.15.121:56514) 2024-09-17T18:27:15.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.206+0000 7f985e59c640 1 -- 172.21.15.121:0/991270127 learned_addr learned my addr 172.21.15.121:0/991270127 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:27:15.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.206+0000 7f985e59c640 10 monclient(hunting): get_auth_request con 0x7f9848003af0 auth_method 0 2024-09-17T18:27:15.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.206+0000 7f985e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:15.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.206+0000 7f985e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:15.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.206+0000 7f985e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:15.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.206+0000 7f985ed9d640 10 monclient(hunting): get_auth_request con 0x7f986006f3a0 auth_method 0 2024-09-17T18:27:15.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.206+0000 7f985ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:15.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.206+0000 7f985ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:15.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.206+0000 7f985ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:15.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.206+0000 7f985e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:15.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.206+0000 7f985e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:15.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.206+0000 7f985e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:15.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.206+0000 7f985ed9d640 1 -- 172.21.15.121:0/991270127 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f986006f3a0 msgr2=0x7f986006f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-17T18:27:15.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.206+0000 7f985ed9d640 1 -- 172.21.15.121:0/991270127 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f986006f3a0 msgr2=0x7f986006f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:27:15.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.206+0000 7f985ed9d640 1 --2- 172.21.15.121:0/991270127 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f986006f3a0 0x7f986006f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:27:15.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.206+0000 7f985ed9d640 1 --2- 172.21.15.121:0/991270127 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f986006f3a0 0x7f986006f820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:27:15.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.206+0000 7f985dd9b640 1 --2- 172.21.15.121:0/991270127 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9860138ae0 0x7f9860138f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:15.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.206+0000 7f985e59c640 10 monclient(hunting): handle_auth_done global_id 44460 payload 824 2024-09-17T18:27:15.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.206+0000 7f985e59c640 1 -- 172.21.15.121:0/991270127 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f986006f3a0 msgr2=0x7f986006f820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:27:15.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.206+0000 7f985e59c640 1 --2- 172.21.15.121:0/991270127 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f986006f3a0 0x7f986006f820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:15.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.206+0000 7f985e59c640 1 -- 172.21.15.121:0/991270127 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9860138ae0 msgr2=0x7f9860138f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:15.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.206+0000 7f985e59c640 1 --2- 172.21.15.121:0/991270127 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9860138ae0 0x7f9860138f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:15.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.206+0000 7f985e59c640 10 monclient: _finish_hunting 0 2024-09-17T18:27:15.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.206+0000 7f985e59c640 1 monclient: found mon.noname-a 2024-09-17T18:27:15.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.206+0000 7f985e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:27:15.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.206+0000 7f985e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:27:15.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.206+0000 7f985e59c640 1 -- 172.21.15.121:0/991270127 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98601274d0 con 0x7f9848003af0 2024-09-17T18:27:15.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.206+0000 7f985dd9b640 10 monclient: get_auth_request con 0x7f9860138ae0 auth_method 0 2024-09-17T18:27:15.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.206+0000 7f985dd9b640 1 --2- 172.21.15.121:0/991270127 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9860138ae0 0x7f9860138f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T18:27:15.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.208+0000 7f985e59c640 1 -- 172.21.15.121:0/991270127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9848003af0 msgr2=0x7f98601cd300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:15.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.208+0000 7f985e59c640 1 --2- 172.21.15.121:0/991270127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9848003af0 0x7f98601cd300 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f985400c420 tx=0x7f986009fa10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:15.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.208+0000 7f984effd640 10 client.?.objecter ms_handle_connect 0x7f9848003af0 2024-09-17T18:27:15.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.208+0000 7f984effd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:27:15.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.208+0000 7f984effd640 1 -- 172.21.15.121:0/991270127 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9854044070 con 0x7f9848003af0 2024-09-17T18:27:15.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.208+0000 7f984effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:15.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.208+0000 7f984effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:27:15.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.208+0000 7f984effd640 10 monclient: dump: 2024-09-17T18:27:15.227 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:15.227 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:15.227 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:15.228 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:15.228 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:15.228 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:15.228 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:15.228 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:15.228 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:15.228 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:15.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.208+0000 7f984effd640 10 monclient: _finish_auth 0 2024-09-17T18:27:15.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.208+0000 7f984effd640 10 monclient: _check_auth_tickets 2024-09-17T18:27:15.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.208+0000 7f984effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:27:15.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.208+0000 7f9864b64640 5 monclient: authenticate success, global_id 44460 2024-09-17T18:27:15.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.208+0000 7f984effd640 1 -- 172.21.15.121:0/991270127 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9854037040 con 0x7f9848003af0 2024-09-17T18:27:15.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.208+0000 7f984effd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:27:15.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.208+0000 7f984effd640 1 -- 172.21.15.121:0/991270127 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9854036070 con 0x7f9848003af0 2024-09-17T18:27:15.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.208+0000 7f984effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:15.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.208+0000 7f984effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:27:15.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.208+0000 7f984effd640 10 monclient: dump: 2024-09-17T18:27:15.229 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:15.229 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:15.229 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:15.229 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:15.229 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:15.229 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:15.229 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:15.229 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:15.229 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:15.229 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:15.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.208+0000 7f9864b64640 10 monclient: _renew_subs 2024-09-17T18:27:15.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.208+0000 7f985cd99640 4 set_mon_vals no callback set 2024-09-17T18:27:15.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.208+0000 7f9864b64640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:27:15.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.208+0000 7f9864b64640 1 -- 172.21.15.121:0/991270127 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9860070d70 con 0x7f9848003af0 2024-09-17T18:27:15.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.208+0000 7f9864b64640 10 client.44460.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:27:15.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.208+0000 7f9864b64640 10 monclient: _renew_subs 2024-09-17T18:27:15.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.208+0000 7f9864b64640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:27:15.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.208+0000 7f9864b64640 1 -- 172.21.15.121:0/991270127 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9860125cd0 con 0x7f9848003af0 2024-09-17T18:27:15.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.208+0000 7f9864b64640 1 librados: init done 2024-09-17T18:27:15.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.209+0000 7f9864b64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:27:15.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.209+0000 7f9864b64640 1 --2- 172.21.15.121:0/991270127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f986006fd60 0x7f986011d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:15.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.209+0000 7f9864b64640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:15.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.209+0000 7f9864b64640 1 -- 172.21.15.121:0/991270127 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f986009f610 con 0x7f986006fd60 2024-09-17T18:27:15.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.209+0000 7f985dd9b640 1 --2- 172.21.15.121:0/991270127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f986006fd60 0x7f986011d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:15.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.209+0000 7f984effd640 1 -- 172.21.15.121:0/991270127 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f9854004a20 con 0x7f9848003af0 2024-09-17T18:27:15.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.209+0000 7f985dd9b640 10 monclient: get_auth_request con 0x7f986006fd60 auth_method 0 2024-09-17T18:27:15.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.209+0000 7f985dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:15.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.209+0000 7f985dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:15.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.209+0000 7f985dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:15.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.209+0000 7f985dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:15.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.209+0000 7f985dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:15.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.209+0000 7f985dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:15.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.209+0000 7f985dd9b640 10 monclient(hunting): handle_auth_done global_id 44463 payload 824 2024-09-17T18:27:15.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.209+0000 7f985dd9b640 1 -- 172.21.15.121:0/991270127 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9860074900 con 0x7f986006fd60 2024-09-17T18:27:15.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.210+0000 7f984effd640 1 --2- 172.21.15.121:0/991270127 >> v2:172.21.15.173:6812/2423786037 conn(0x7f98400401a0 0x7f9840042660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:15.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.210+0000 7f984effd640 1 -- 172.21.15.121:0/991270127 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(341..341 src has 177..341) ==== 5358+0+0 (secure 0 0 0) 0x7f9854072a30 con 0x7f9848003af0 2024-09-17T18:27:15.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.210+0000 7f984effd640 10 client.44460.objecter ms_dispatch 0x7f9860000b90 osd_map(341..341 src has 177..341) 2024-09-17T18:27:15.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.210+0000 7f984effd640 3 client.44460.objecter handle_osd_map got epochs [341,341] > 0 2024-09-17T18:27:15.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.210+0000 7f984effd640 3 client.44460.objecter handle_osd_map decoding full epoch 341 2024-09-17T18:27:15.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.210+0000 7f984effd640 20 client.44460.objecter dump_active .. 0 homeless 2024-09-17T18:27:15.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.210+0000 7f985ed9d640 1 --2- 172.21.15.121:0/991270127 >> v2:172.21.15.173:6812/2423786037 conn(0x7f98400401a0 0x7f9840042660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:15.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.210+0000 7f985ed9d640 10 monclient: get_auth_request con 0x7f98400401a0 auth_method 0 2024-09-17T18:27:15.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.212+0000 7f984effd640 10 client.44460.objecter ms_handle_connect 0x7f986006fd60 2024-09-17T18:27:15.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.212+0000 7f985dd9b640 1 -- 172.21.15.121:0/991270127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f986006fd60 msgr2=0x7f986011d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:15.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.212+0000 7f984effd640 10 client.44460.objecter resend_mon_ops 2024-09-17T18:27:15.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.212+0000 7f985dd9b640 1 --2- 172.21.15.121:0/991270127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f986006fd60 0x7f986011d6a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9850006b60 tx=0x7f98500019c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:15.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.212+0000 7f984effd640 1 -- 172.21.15.121:0/991270127 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9850013780 con 0x7f986006fd60 2024-09-17T18:27:15.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.212+0000 7f984effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:15.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.212+0000 7f984effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:27:15.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.212+0000 7f984effd640 10 monclient: dump: 2024-09-17T18:27:15.235 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:15.235 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:15.235 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:15.235 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:15.235 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:15.235 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:15.235 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:15.235 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:15.235 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:15.235 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:15.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.212+0000 7f984effd640 1 -- 172.21.15.121:0/991270127 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9850019070 con 0x7f986006fd60 2024-09-17T18:27:15.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.212+0000 7f984effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:27:15.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.212+0000 7f984effd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:27:15.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.212+0000 7f984effd640 1 -- 172.21.15.121:0/991270127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f986006fd60 msgr2=0x7f986011d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:15.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.212+0000 7f984effd640 1 --2- 172.21.15.121:0/991270127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f986006fd60 0x7f986011d6a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9850006b60 tx=0x7f98500019c0 comp rx=0 tx=0).stop 2024-09-17T18:27:15.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.212+0000 7f984effd640 1 -- 172.21.15.121:0/991270127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f986006fd60 msgr2=0x7f986011d6a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:27:15.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.212+0000 7f984effd640 1 --2- 172.21.15.121:0/991270127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f986006fd60 0x7f986011d6a0 secure :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9850006b60 tx=0x7f98500019c0 comp rx=0 tx=0).stop 2024-09-17T18:27:15.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.213+0000 7f984effd640 10 client.44460.objecter ms_handle_connect 0x7f98400401a0 2024-09-17T18:27:15.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.213+0000 7f985ed9d640 1 --2- 172.21.15.121:0/991270127 >> v2:172.21.15.173:6812/2423786037 conn(0x7f98400401a0 0x7f9840042660 secure :-1 s=READY pgs=3995 cs=0 l=1 rev1=1 crypto rx=0x7f984800c300 tx=0x7f9858009770 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:15.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.218+0000 7f9864b64640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:27:15.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.218+0000 7f9864b64640 1 --2- 172.21.15.121:0/991270127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f986011dbe0 0x7f986011dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:15.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.218+0000 7f9864b64640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:15.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.218+0000 7f9864b64640 1 -- 172.21.15.121:0/991270127 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f986006cd40 con 0x7f986011dbe0 2024-09-17T18:27:15.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.218+0000 7f985dd9b640 1 --2- 172.21.15.121:0/991270127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f986011dbe0 0x7f986011dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:15.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.218+0000 7f985dd9b640 10 monclient: get_auth_request con 0x7f986011dbe0 auth_method 0 2024-09-17T18:27:15.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.218+0000 7f985dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:15.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.218+0000 7f985dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:15.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.218+0000 7f985dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:15.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.218+0000 7f985dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:15.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.218+0000 7f985dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:15.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.218+0000 7f985dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:15.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.218+0000 7f985dd9b640 10 monclient(hunting): handle_auth_done global_id 44466 payload 824 2024-09-17T18:27:15.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.218+0000 7f985dd9b640 1 -- 172.21.15.121:0/991270127 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9860139460 con 0x7f986011dbe0 2024-09-17T18:27:15.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.221+0000 7f985dd9b640 1 -- 172.21.15.121:0/991270127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f986011dbe0 msgr2=0x7f986011dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:15.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.221+0000 7f984effd640 10 client.44460.objecter ms_handle_connect 0x7f986011dbe0 2024-09-17T18:27:15.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.221+0000 7f984effd640 10 client.44460.objecter resend_mon_ops 2024-09-17T18:27:15.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.221+0000 7f985dd9b640 1 --2- 172.21.15.121:0/991270127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f986011dbe0 0x7f986011dff0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9850006b60 tx=0x7f98500014b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:15.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.221+0000 7f984effd640 1 -- 172.21.15.121:0/991270127 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9850013e20 con 0x7f986011dbe0 2024-09-17T18:27:15.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.221+0000 7f984effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:15.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.221+0000 7f984effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:27:15.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.221+0000 7f984effd640 10 monclient: dump: 2024-09-17T18:27:15.237 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:15.238 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:15.238 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:15.238 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:15.238 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:15.238 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:15.238 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:15.238 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:15.238 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:15.238 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:15.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.221+0000 7f984effd640 1 -- 172.21.15.121:0/991270127 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7f9850013e20 con 0x7f986011dbe0 2024-09-17T18:27:15.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.221+0000 7f984effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:27:15.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.221+0000 7f984effd640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:27:15.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.221+0000 7f984effd640 1 -- 172.21.15.121:0/991270127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f986011dbe0 msgr2=0x7f986011dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:15.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.221+0000 7f984effd640 1 --2- 172.21.15.121:0/991270127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f986011dbe0 0x7f986011dff0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9850006b60 tx=0x7f98500014b0 comp rx=0 tx=0).stop 2024-09-17T18:27:15.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.221+0000 7f984effd640 1 -- 172.21.15.121:0/991270127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f986011dbe0 msgr2=0x7f986011dff0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:27:15.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.221+0000 7f984effd640 1 --2- 172.21.15.121:0/991270127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f986011dbe0 0x7f986011dff0 secure :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9850006b60 tx=0x7f98500014b0 comp rx=0 tx=0).stop 2024-09-17T18:27:15.239 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:27:15.239 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:27:15.239 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:27:15.239 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "leader", 2024-09-17T18:27:15.239 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 32, 2024-09-17T18:27:15.239 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 260885, 2024-09-17T18:27:15.239 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:27:15.239 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:27:15.239 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:27:15.240 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:27:15.240 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:27:15.240 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 3, 2024-09-17T18:27:15.240 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:27:15.240 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:27:15.240 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:27:15.240 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:27:15.240 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:27:15.240 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:27:15.240 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:27:15.240 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:27:15.240 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:27:15.240 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:27:15.240 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:27:15.241 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:27:15.241 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:27:15.241 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:27:15.241 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:27:15.241 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:27:15.241 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:27:15.241 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:27:15.241 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:27:15.241 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:27:15.241 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:27:15.241 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:27:15.241 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:27:15.241 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:27:15.242 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:27:15.242 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:27:15.242 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:27:15.242 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:27:15.242 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:15.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:15.242 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:27:15.242 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:27:15.242 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:27:15.242 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:27:15.242 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:27:15.242 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:27:15.242 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:27:15.242 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:27:15.243 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:27:15.243 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:27:15.243 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:27:15.243 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:27:15.243 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:27:15.243 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:27:15.243 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:27:15.243 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:27:15.243 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:27:15.243 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:27:15.243 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:27:15.243 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:27:15.243 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:27:15.243 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:27:15.243 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:27:15.244 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:27:15.244 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:27:15.244 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:27:15.244 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:27:15.244 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:27:15.244 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:27:15.244 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:27:15.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:15.244 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:27:15.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:15.244 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:27:15.244 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:27:15.244 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:27:15.244 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:27:15.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:15.245 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:27:15.245 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:27:15.245 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:15.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:15.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:15.245 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:27:15.245 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:27:15.245 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:15.245 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:15.245 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:15.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:15.245 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:27:15.245 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:27:15.246 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:27:15.246 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:27:15.246 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:27:15.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:15.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:15.246 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:27:15.246 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:27:15.246 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:27:15.246 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:27:15.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:15.246 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:27:15.246 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:27:15.246 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:15.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:15.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:15.247 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:27:15.247 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:27:15.247 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:15.247 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:15.247 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:15.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:15.247 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:27:15.247 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:27:15.247 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:27:15.247 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:27:15.247 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:27:15.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:15.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:15.247 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:27:15.248 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:27:15.248 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:27:15.248 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:27:15.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:15.248 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:27:15.248 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:27:15.248 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:15.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:15.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:15.248 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:27:15.248 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:27:15.248 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:15.248 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:15.248 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:15.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:15.249 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:27:15.249 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:27:15.249 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:27:15.249 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:27:15.249 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:27:15.249 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:15.249 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:15.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:15.249 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:27:15.249 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:27:15.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:15.249 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:27:15.249 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:27:15.250 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:27:15.250 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:15.250 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:27:15.250 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:27:15.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:15.250 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:27:15.250 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:27:15.250 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-17T18:27:15.250 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:15.250 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:27:15.250 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:27:15.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:15.250 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:27:15.250 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:27:15.250 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-17T18:27:15.251 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:15.251 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:15.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:15.251 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:27:15.251 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:27:15.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.221+0000 7f9864b64640 10 librados: watch_flush enter 2024-09-17T18:27:15.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.221+0000 7f9864b64640 10 librados: watch_flush exit 2024-09-17T18:27:15.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.221+0000 7f9864b64640 20 client.44460.objecter shutdown clearing up homeless session... 2024-09-17T18:27:15.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.221+0000 7f9864b64640 10 client.44460.objecter successfully canceled tick 2024-09-17T18:27:15.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.221+0000 7f9864b64640 1 -- 172.21.15.121:0/991270127 >> v2:172.21.15.173:6812/2423786037 conn(0x7f98400401a0 msgr2=0x7f9840042660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:15.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.221+0000 7f9864b64640 1 --2- 172.21.15.121:0/991270127 >> v2:172.21.15.173:6812/2423786037 conn(0x7f98400401a0 0x7f9840042660 secure :-1 s=READY pgs=3995 cs=0 l=1 rev1=1 crypto rx=0x7f984800c300 tx=0x7f9858009770 comp rx=0 tx=0).stop 2024-09-17T18:27:15.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.221+0000 7f9864b64640 10 monclient: shutdown 2024-09-17T18:27:15.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.221+0000 7f9864b64640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:27:15.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.221+0000 7f9864b64640 1 -- 172.21.15.121:0/991270127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9848003af0 msgr2=0x7f98601cd300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:15.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.221+0000 7f9864b64640 1 --2- 172.21.15.121:0/991270127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9848003af0 0x7f98601cd300 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f985400c420 tx=0x7f986009fa10 comp rx=0 tx=0).stop 2024-09-17T18:27:15.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.221+0000 7f985ed9d640 1 -- 172.21.15.121:0/991270127 reap_dead start 2024-09-17T18:27:15.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.221+0000 7f985ed9d640 1 -- 172.21.15.121:0/991270127 reap_dead start 2024-09-17T18:27:15.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.221+0000 7f9864b64640 1 -- 172.21.15.121:0/991270127 shutdown_connections 2024-09-17T18:27:15.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.221+0000 7f9864b64640 1 -- 172.21.15.121:0/991270127 >> 172.21.15.121:0/991270127 conn(0x7f98600a7830 msgr2=0x7f98600a0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:27:15.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.221+0000 7f9864b64640 1 -- 172.21.15.121:0/991270127 shutdown_connections 2024-09-17T18:27:15.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.221+0000 7f9864b64640 1 -- 172.21.15.121:0/991270127 wait complete. 2024-09-17T18:27:15.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.222+0000 7f9864b64640 1 librados: shutdown 2024-09-17T18:27:15.254 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:27:15.254 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T18:27:15.254 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T18:27:15.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.351+0000 7f12359ad640 10 monclient: get_monmap_and_config 2024-09-17T18:27:15.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.351+0000 7f12359ad640 10 monclient: build_initial_monmap 2024-09-17T18:27:15.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.351+0000 7f12359ad640 10 monclient: monmap: 2024-09-17T18:27:15.354 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:27:15.354 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:15.354 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:27:15.352474+0000 2024-09-17T18:27:15.354 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:27:15.352474+0000 2024-09-17T18:27:15.354 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:27:15.354 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:15.354 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:27:15.354 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:27:15.354 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:27:15.354 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:27:15.354 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:27:15.354 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:15.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.352+0000 7f12359ad640 1 Processor -- start 2024-09-17T18:27:15.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.352+0000 7f12359ad640 1 -- start start 2024-09-17T18:27:15.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.352+0000 7f12359ad640 10 monclient: init 2024-09-17T18:27:15.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f12359ad640 10 monclient: _reopen_session rank -1 2024-09-17T18:27:15.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f12359ad640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-17T18:27:15.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f12359ad640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12300a3a00 0x7f12300a3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:15.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f12359ad640 10 monclient(hunting): picked mon.noname-b con 0x7f12300a3a00 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:27:15.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f12359ad640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f12300a4320 0x7f12301367c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:15.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f12359ad640 10 monclient(hunting): picked mon.noname-a con 0x7f12300a4320 addr v2:172.21.15.121:3300/0 2024-09-17T18:27:15.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f12359ad640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f1230136d00 0x7f1230137100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:15.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f12359ad640 10 monclient(hunting): picked mon.noname-c con 0x7f1230136d00 addr v2:172.21.15.173:3300/0 2024-09-17T18:27:15.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f12359ad640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:15.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f12359ad640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f123009ee00 con 0x7f12300a4320 2024-09-17T18:27:15.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f12359ad640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:15.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f12359ad640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f12300a0a00 con 0x7f1230136d00 2024-09-17T18:27:15.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f12359ad640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:15.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f12359ad640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1230076bc0 con 0x7f12300a3a00 2024-09-17T18:27:15.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f12359ad640 10 monclient(hunting): _renew_subs 2024-09-17T18:27:15.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f12359ad640 10 monclient(hunting): authenticate will time out at 1239.069094s 2024-09-17T18:27:15.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f122effd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12300a3a00 0x7f12300a3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:15.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f1227fff640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f12300a4320 0x7f12301367c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:15.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f122effd640 1 -- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12300a3a00 msgr2=0x7f12300a3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:27:15.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f1227fff640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f12300a4320 0x7f12301367c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56546/0 (socket says 172.21.15.121:56546) 2024-09-17T18:27:15.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f122effd640 1 -- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12300a3a00 msgr2=0x7f12300a3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:27:15.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f1227fff640 1 -- 172.21.15.121:0/1346230274 learned_addr learned my addr 172.21.15.121:0/1346230274 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:27:15.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f122effd640 1 --2- 172.21.15.121:0/1346230274 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12300a3a00 0x7f12300a3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:27:15.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f1227fff640 10 monclient(hunting): get_auth_request con 0x7f12300a4320 auth_method 0 2024-09-17T18:27:15.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f1227fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:15.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f1227fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:15.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f1227fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:15.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f122effd640 1 --2- 172.21.15.121:0/1346230274 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12300a3a00 0x7f12300a3de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:27:15.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f122f7fe640 1 --2- 172.21.15.121:0/1346230274 >> v2:172.21.15.173:3300/0 conn(0x7f1230136d00 0x7f1230137100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:15.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f1227fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:15.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f1227fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:15.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f1227fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:15.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f122f7fe640 10 monclient(hunting): get_auth_request con 0x7f1230136d00 auth_method 0 2024-09-17T18:27:15.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f122f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:15.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f122f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:15.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f122f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:15.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f122f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:15.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f122f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:15.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f122f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:15.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f1227fff640 10 monclient(hunting): handle_auth_done global_id 44469 payload 274 2024-09-17T18:27:15.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f1227fff640 1 -- 172.21.15.121:0/1346230274 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12300a3a00 msgr2=0x7f12300a3de0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:27:15.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f1227fff640 1 --2- 172.21.15.121:0/1346230274 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12300a3a00 0x7f12300a3de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:15.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f1227fff640 1 -- 172.21.15.121:0/1346230274 >> v2:172.21.15.173:3300/0 conn(0x7f1230136d00 msgr2=0x7f1230137100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:27:15.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f1227fff640 1 --2- 172.21.15.121:0/1346230274 >> v2:172.21.15.173:3300/0 conn(0x7f1230136d00 0x7f1230137100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:15.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f1227fff640 10 monclient: _finish_hunting 0 2024-09-17T18:27:15.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f1227fff640 1 monclient: found mon.noname-a 2024-09-17T18:27:15.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f1227fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:27:15.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f1227fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:27:15.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.353+0000 7f1227fff640 1 -- 172.21.15.121:0/1346230274 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12300a0040 con 0x7f12300a4320 2024-09-17T18:27:15.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.356+0000 7f1227fff640 1 -- 172.21.15.121:0/1346230274 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12300a4320 msgr2=0x7f12301367c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:15.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.356+0000 7f1227fff640 1 --2- 172.21.15.121:0/1346230274 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12300a4320 0x7f12301367c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1230077920 tx=0x7f121802a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e66e1809f223cf64 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:15.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.356+0000 7f122e7fc640 1 -- 172.21.15.121:0/1346230274 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1218037070 con 0x7f12300a4320 2024-09-17T18:27:15.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.356+0000 7f122e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:15.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.356+0000 7f122e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:27:15.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.356+0000 7f122e7fc640 10 monclient: dump: 2024-09-17T18:27:15.368 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:15.368 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:15.368 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:15.368 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:15.368 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:15.368 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:15.368 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:15.368 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:15.368 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:15.368 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:15.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.356+0000 7f122e7fc640 10 monclient: _finish_auth 0 2024-09-17T18:27:15.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.356+0000 7f122e7fc640 10 monclient: _check_auth_tickets 2024-09-17T18:27:15.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.356+0000 7f122e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:27:15.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.356+0000 7f122e7fc640 1 -- 172.21.15.121:0/1346230274 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1218032640 con 0x7f12300a4320 2024-09-17T18:27:15.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.356+0000 7f12359ad640 5 monclient: authenticate success, global_id 44469 2024-09-17T18:27:15.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.356+0000 7f12359ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:27:15.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.356+0000 7f122e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:27:15.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.356+0000 7f12359ad640 10 monclient: get_monmap_and_config success 2024-09-17T18:27:15.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.356+0000 7f122e7fc640 1 -- 172.21.15.121:0/1346230274 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1218032a10 con 0x7f12300a4320 2024-09-17T18:27:15.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.356+0000 7f12359ad640 4 set_mon_vals no callback set 2024-09-17T18:27:15.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.356+0000 7f122e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:15.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.356+0000 7f122e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:27:15.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.356+0000 7f122e7fc640 10 monclient: dump: 2024-09-17T18:27:15.369 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:15.369 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:15.369 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:15.370 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:15.370 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:15.370 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:15.370 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:15.370 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:15.370 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:15.370 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:15.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.357+0000 7f12359ad640 10 monclient: shutdown 2024-09-17T18:27:15.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.357+0000 7f12359ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:27:15.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.357+0000 7f12359ad640 1 -- 172.21.15.121:0/1346230274 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12300a4320 msgr2=0x7f12301367c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:15.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.357+0000 7f12359ad640 1 --2- 172.21.15.121:0/1346230274 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12300a4320 0x7f12301367c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1230077920 tx=0x7f121802a0b0 comp rx=0 tx=0).stop 2024-09-17T18:27:15.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.357+0000 7f12359ad640 1 -- 172.21.15.121:0/1346230274 shutdown_connections 2024-09-17T18:27:15.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.357+0000 7f12359ad640 1 --2- 172.21.15.121:0/1346230274 >> v2:172.21.15.173:3300/0 conn(0x7f1230136d00 0x7f1230137100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:15.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.357+0000 7f12359ad640 1 --2- 172.21.15.121:0/1346230274 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12300a4320 0x7f12301367c0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:15.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.357+0000 7f12359ad640 1 --2- 172.21.15.121:0/1346230274 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12300a3a00 0x7f12300a3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:15.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.357+0000 7f12359ad640 1 -- 172.21.15.121:0/1346230274 >> 172.21.15.121:0/1346230274 conn(0x7f12300a8680 msgr2=0x7f12300a4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:27:15.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.357+0000 7f12359ad640 1 -- 172.21.15.121:0/1346230274 shutdown_connections 2024-09-17T18:27:15.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.357+0000 7f12359ad640 1 -- 172.21.15.121:0/1346230274 wait complete. 2024-09-17T18:27:15.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.357+0000 7f12359ad640 10 monclient: build_initial_monmap 2024-09-17T18:27:15.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.357+0000 7f12359ad640 10 monclient: monmap: 2024-09-17T18:27:15.371 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:27:15.371 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:27:15.371 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:27:15.371 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:27:15.371 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:27:15.371 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:15.371 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:27:15.371 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:27:15.371 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:27:15.371 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:15.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.358+0000 7f12359ad640 1 librados: starting msgr at 2024-09-17T18:27:15.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.358+0000 7f12359ad640 1 librados: starting objecter 2024-09-17T18:27:15.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.358+0000 7f12359ad640 1 Processor -- start 2024-09-17T18:27:15.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.358+0000 7f12359ad640 1 -- start start 2024-09-17T18:27:15.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.358+0000 7f12359ad640 1 librados: setting wanted keys 2024-09-17T18:27:15.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.358+0000 7f12359ad640 1 librados: calling monclient init 2024-09-17T18:27:15.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.358+0000 7f12359ad640 10 monclient: init 2024-09-17T18:27:15.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.358+0000 7f12359ad640 10 monclient: _reopen_session rank -1 2024-09-17T18:27:15.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.358+0000 7f12359ad640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T18:27:15.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.358+0000 7f12359ad640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f12300a3a00 0x7f12301cba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:15.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.358+0000 7f12359ad640 10 monclient(hunting): picked mon.noname-b con 0x7f12300a3a00 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:27:15.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.358+0000 7f12359ad640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12300a4320 0x7f12301cbf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:15.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.358+0000 7f12359ad640 10 monclient(hunting): picked mon.noname-a con 0x7f12300a4320 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:27:15.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.358+0000 7f12359ad640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1230136d00 0x7f12301230b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:15.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.358+0000 7f12359ad640 10 monclient(hunting): picked mon.noname-c con 0x7f1230136d00 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:27:15.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.358+0000 7f12359ad640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:15.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.358+0000 7f12359ad640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f123009f7a0 con 0x7f12300a4320 2024-09-17T18:27:15.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.358+0000 7f12359ad640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:15.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.358+0000 7f12359ad640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f12300a4700 con 0x7f12300a3a00 2024-09-17T18:27:15.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.358+0000 7f12359ad640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:15.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.358+0000 7f12359ad640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f123009f1e0 con 0x7f1230136d00 2024-09-17T18:27:15.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.358+0000 7f12359ad640 10 monclient(hunting): _renew_subs 2024-09-17T18:27:15.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.358+0000 7f12359ad640 10 monclient(hunting): authenticate will time out at 1239.074861s 2024-09-17T18:27:15.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.358+0000 7f1227fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12300a4320 0x7f12301cbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:15.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.358+0000 7f122f7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1230136d00 0x7f12301230b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:15.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.358+0000 7f1227fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12300a4320 0x7f12301cbf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56558/0 (socket says 172.21.15.121:56558) 2024-09-17T18:27:15.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.358+0000 7f1227fff640 1 -- 172.21.15.121:0/2483460163 learned_addr learned my addr 172.21.15.121:0/2483460163 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:27:15.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.358+0000 7f1227fff640 10 monclient(hunting): get_auth_request con 0x7f12300a4320 auth_method 0 2024-09-17T18:27:15.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.358+0000 7f1227fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:15.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.358+0000 7f1227fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:15.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.358+0000 7f1227fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:15.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.358+0000 7f122f7fe640 10 monclient(hunting): get_auth_request con 0x7f1230136d00 auth_method 0 2024-09-17T18:27:15.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.358+0000 7f122f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:15.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.358+0000 7f122f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:15.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.358+0000 7f122f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:15.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.358+0000 7f122effd640 1 --2- 172.21.15.121:0/2483460163 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f12300a3a00 0x7f12301cba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:15.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.359+0000 7f122effd640 10 monclient(hunting): get_auth_request con 0x7f12300a3a00 auth_method 0 2024-09-17T18:27:15.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.359+0000 7f122effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:15.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.359+0000 7f122effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:15.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.359+0000 7f122effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:15.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.359+0000 7f1227fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:15.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.359+0000 7f1227fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:15.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.359+0000 7f1227fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:15.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.359+0000 7f122f7fe640 1 -- 172.21.15.121:0/2483460163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1230136d00 msgr2=0x7f12301230b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-17T18:27:15.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.359+0000 7f122f7fe640 1 -- 172.21.15.121:0/2483460163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1230136d00 msgr2=0x7f12301230b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:27:15.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.359+0000 7f122f7fe640 1 --2- 172.21.15.121:0/2483460163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1230136d00 0x7f12301230b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:27:15.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.359+0000 7f122f7fe640 1 --2- 172.21.15.121:0/2483460163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1230136d00 0x7f12301230b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:27:15.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.359+0000 7f122effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:15.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.359+0000 7f122effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:15.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.359+0000 7f122effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:15.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.359+0000 7f1227fff640 10 monclient(hunting): handle_auth_done global_id 44472 payload 824 2024-09-17T18:27:15.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.359+0000 7f1227fff640 1 -- 172.21.15.121:0/2483460163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1230136d00 msgr2=0x7f12301230b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:27:15.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.359+0000 7f1227fff640 1 --2- 172.21.15.121:0/2483460163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1230136d00 0x7f12301230b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:15.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.359+0000 7f1227fff640 1 -- 172.21.15.121:0/2483460163 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f12300a3a00 msgr2=0x7f12301cba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:15.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.359+0000 7f1227fff640 1 --2- 172.21.15.121:0/2483460163 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f12300a3a00 0x7f12301cba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:15.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.359+0000 7f1227fff640 10 monclient: _finish_hunting 0 2024-09-17T18:27:15.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.359+0000 7f1227fff640 1 monclient: found mon.noname-a 2024-09-17T18:27:15.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.359+0000 7f1227fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:27:15.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.359+0000 7f1227fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:27:15.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.359+0000 7f1227fff640 1 -- 172.21.15.121:0/2483460163 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1230136490 con 0x7f12300a4320 2024-09-17T18:27:15.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.361+0000 7f1227fff640 1 -- 172.21.15.121:0/2483460163 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12300a4320 msgr2=0x7f12301cbf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:15.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.361+0000 7f1227fff640 1 --2- 172.21.15.121:0/2483460163 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12300a4320 0x7f12301cbf90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1218000c00 tx=0x7f12180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:15.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.361+0000 7f122cff9640 10 client.?.objecter ms_handle_connect 0x7f12300a4320 2024-09-17T18:27:15.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.361+0000 7f122cff9640 10 client.?.objecter resend_mon_ops 2024-09-17T18:27:15.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.361+0000 7f122cff9640 1 -- 172.21.15.121:0/2483460163 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1218041020 con 0x7f12300a4320 2024-09-17T18:27:15.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.361+0000 7f122cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:15.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.361+0000 7f122cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:27:15.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.361+0000 7f122cff9640 10 monclient: dump: 2024-09-17T18:27:15.377 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:15.377 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:15.377 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:15.377 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:15.377 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:15.377 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:15.377 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:15.377 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:15.377 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:15.377 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:15.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.361+0000 7f122cff9640 10 monclient: _finish_auth 0 2024-09-17T18:27:15.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.361+0000 7f122cff9640 10 monclient: _check_auth_tickets 2024-09-17T18:27:15.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.361+0000 7f122cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:27:15.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.361+0000 7f122cff9640 1 -- 172.21.15.121:0/2483460163 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f121803d830 con 0x7f12300a4320 2024-09-17T18:27:15.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.361+0000 7f122cff9640 10 monclient: handle_config config(0 keys) 2024-09-17T18:27:15.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.361+0000 7f122cff9640 1 -- 172.21.15.121:0/2483460163 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1218037040 con 0x7f12300a4320 2024-09-17T18:27:15.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.361+0000 7f122cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:15.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.361+0000 7f122cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:27:15.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.361+0000 7f122cff9640 10 monclient: dump: 2024-09-17T18:27:15.378 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:15.378 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:15.378 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:15.378 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:15.378 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:15.378 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:15.378 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:15.378 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:15.379 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:15.379 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:15.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.361+0000 7f12359ad640 5 monclient: authenticate success, global_id 44472 2024-09-17T18:27:15.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.361+0000 7f12359ad640 10 monclient: _renew_subs 2024-09-17T18:27:15.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.361+0000 7f122dffb640 4 set_mon_vals no callback set 2024-09-17T18:27:15.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.361+0000 7f12359ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:27:15.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.361+0000 7f12359ad640 1 -- 172.21.15.121:0/2483460163 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1230124b70 con 0x7f12300a4320 2024-09-17T18:27:15.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.361+0000 7f12359ad640 10 client.44472.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:27:15.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.361+0000 7f12359ad640 10 monclient: _renew_subs 2024-09-17T18:27:15.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.361+0000 7f12359ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:27:15.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.361+0000 7f12359ad640 1 -- 172.21.15.121:0/2483460163 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1230126300 con 0x7f12300a4320 2024-09-17T18:27:15.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.361+0000 7f12359ad640 1 librados: init done 2024-09-17T18:27:15.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.362+0000 7f12359ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:27:15.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.362+0000 7f12359ad640 1 --2- 172.21.15.121:0/2483460163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12301235f0 0x7f12301239d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:15.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.362+0000 7f12359ad640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:15.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.362+0000 7f12359ad640 1 -- 172.21.15.121:0/2483460163 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1230126b50 con 0x7f12301235f0 2024-09-17T18:27:15.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.362+0000 7f122cff9640 1 -- 172.21.15.121:0/2483460163 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f1218044020 con 0x7f12300a4320 2024-09-17T18:27:15.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.362+0000 7f122effd640 1 --2- 172.21.15.121:0/2483460163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12301235f0 0x7f12301239d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:15.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.362+0000 7f122effd640 10 monclient: get_auth_request con 0x7f12301235f0 auth_method 0 2024-09-17T18:27:15.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.362+0000 7f122effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:15.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.362+0000 7f122effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:15.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.362+0000 7f122effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:15.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.362+0000 7f122effd640 1 -- 172.21.15.121:0/2483460163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12301235f0 msgr2=0x7f12301239d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T18:27:15.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.362+0000 7f122effd640 1 -- 172.21.15.121:0/2483460163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12301235f0 msgr2=0x7f12301239d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:27:15.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.362+0000 7f122effd640 1 --2- 172.21.15.121:0/2483460163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12301235f0 0x7f12301239d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:27:15.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.362+0000 7f122effd640 1 --2- 172.21.15.121:0/2483460163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12301235f0 0x7f12301239d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:27:15.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.362+0000 7f122cff9640 1 --2- 172.21.15.121:0/2483460163 >> v2:172.21.15.173:6812/2423786037 conn(0x7f12100401c0 0x7f1210042680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:15.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.362+0000 7f122cff9640 1 -- 172.21.15.121:0/2483460163 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(341..341 src has 177..341) ==== 5358+0+0 (secure 0 0 0) 0x7f1218073bb0 con 0x7f12300a4320 2024-09-17T18:27:15.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.362+0000 7f122cff9640 10 client.44472.objecter ms_dispatch 0x7f1230000b90 osd_map(341..341 src has 177..341) 2024-09-17T18:27:15.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.362+0000 7f122cff9640 3 client.44472.objecter handle_osd_map got epochs [341,341] > 0 2024-09-17T18:27:15.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.362+0000 7f122cff9640 3 client.44472.objecter handle_osd_map decoding full epoch 341 2024-09-17T18:27:15.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.363+0000 7f122cff9640 20 client.44472.objecter dump_active .. 0 homeless 2024-09-17T18:27:15.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.363+0000 7f122f7fe640 1 --2- 172.21.15.121:0/2483460163 >> v2:172.21.15.173:6812/2423786037 conn(0x7f12100401c0 0x7f1210042680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:15.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.363+0000 7f122f7fe640 10 monclient: get_auth_request con 0x7f12100401c0 auth_method 0 2024-09-17T18:27:15.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.365+0000 7f122cff9640 10 client.44472.objecter ms_handle_connect 0x7f12100401c0 2024-09-17T18:27:15.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.365+0000 7f122f7fe640 1 --2- 172.21.15.121:0/2483460163 >> v2:172.21.15.173:6812/2423786037 conn(0x7f12100401c0 0x7f1210042680 secure :-1 s=READY pgs=3996 cs=0 l=1 rev1=1 crypto rx=0x7f1228009b50 tx=0x7f122800b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:15.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.563+0000 7f122effd640 1 --2- 172.21.15.121:0/2483460163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12301235f0 0x7f12301239d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:15.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.563+0000 7f122effd640 10 monclient: get_auth_request con 0x7f12301235f0 auth_method 0 2024-09-17T18:27:15.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.563+0000 7f122effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:15.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.563+0000 7f122effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:15.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.563+0000 7f122effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:27:15.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.563+0000 7f122effd640 1 -- 172.21.15.121:0/2483460163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12301235f0 msgr2=0x7f12301239d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-17T18:27:15.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.563+0000 7f122effd640 1 -- 172.21.15.121:0/2483460163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12301235f0 msgr2=0x7f12301239d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:27:15.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.563+0000 7f122effd640 1 --2- 172.21.15.121:0/2483460163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12301235f0 0x7f12301239d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:27:15.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.563+0000 7f122effd640 1 --2- 172.21.15.121:0/2483460163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12301235f0 0x7f12301239d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T18:27:15.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.964+0000 7f122effd640 1 --2- 172.21.15.121:0/2483460163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12301235f0 0x7f12301239d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:15.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.964+0000 7f122effd640 10 monclient: get_auth_request con 0x7f12301235f0 auth_method 0 2024-09-17T18:27:15.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.964+0000 7f122effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:15.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.964+0000 7f122effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:15.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.964+0000 7f122effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:27:15.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.964+0000 7f122effd640 1 -- 172.21.15.121:0/2483460163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12301235f0 msgr2=0x7f12301239d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-17T18:27:15.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.964+0000 7f122effd640 1 -- 172.21.15.121:0/2483460163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12301235f0 msgr2=0x7f12301239d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:27:15.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.964+0000 7f122effd640 1 --2- 172.21.15.121:0/2483460163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12301235f0 0x7f12301239d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:27:15.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:15.964+0000 7f122effd640 1 --2- 172.21.15.121:0/2483460163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12301235f0 0x7f12301239d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T18:27:16.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.765+0000 7f122effd640 1 --2- 172.21.15.121:0/2483460163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12301235f0 0x7f12301239d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:16.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.765+0000 7f122effd640 10 monclient: get_auth_request con 0x7f12301235f0 auth_method 0 2024-09-17T18:27:16.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.765+0000 7f122effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:16.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.765+0000 7f122effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:16.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.765+0000 7f122effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:27:16.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.765+0000 7f122effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:16.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.765+0000 7f122effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:16.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.765+0000 7f122effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:16.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.765+0000 7f122effd640 10 monclient(hunting): handle_auth_done global_id 64100 payload 824 2024-09-17T18:27:16.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.765+0000 7f122effd640 1 -- 172.21.15.121:0/2483460163 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f12300a2330 con 0x7f12301235f0 2024-09-17T18:27:16.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.768+0000 7f122effd640 1 -- 172.21.15.121:0/2483460163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12301235f0 msgr2=0x7f12301239d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:16.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.768+0000 7f122effd640 1 --2- 172.21.15.121:0/2483460163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12301235f0 0x7f12301239d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f12200169e0 tx=0x7f1220016ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:16.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.768+0000 7f122cff9640 10 client.44472.objecter ms_handle_connect 0x7f12301235f0 2024-09-17T18:27:16.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.768+0000 7f122cff9640 10 client.44472.objecter resend_mon_ops 2024-09-17T18:27:16.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.768+0000 7f122cff9640 1 -- 172.21.15.121:0/2483460163 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1220022680 con 0x7f12301235f0 2024-09-17T18:27:16.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.768+0000 7f122cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:16.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.768+0000 7f122cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:27:16.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.768+0000 7f122cff9640 10 monclient: dump: 2024-09-17T18:27:16.770 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:16.770 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:16.771 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:16.771 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:16.771 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:16.771 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:16.771 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:16.771 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:16.771 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:16.771 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:16.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.768+0000 7f122cff9640 1 -- 172.21.15.121:0/2483460163 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1220022680 con 0x7f12301235f0 2024-09-17T18:27:16.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.768+0000 7f122cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:27:16.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.768+0000 7f122cff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:27:16.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.768+0000 7f122cff9640 1 -- 172.21.15.121:0/2483460163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12301235f0 msgr2=0x7f12301239d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:16.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.768+0000 7f122cff9640 1 --2- 172.21.15.121:0/2483460163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12301235f0 0x7f12301239d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f12200169e0 tx=0x7f1220016ea0 comp rx=0 tx=0).stop 2024-09-17T18:27:16.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.768+0000 7f122cff9640 1 -- 172.21.15.121:0/2483460163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12301235f0 msgr2=0x7f12301239d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:27:16.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.768+0000 7f122cff9640 1 --2- 172.21.15.121:0/2483460163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12301235f0 0x7f12301239d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:16.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.775+0000 7f12359ad640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:27:16.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.775+0000 7f12359ad640 1 --2- 172.21.15.121:0/2483460163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1230123f10 0x7f123011e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:16.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.775+0000 7f12359ad640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:16.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.775+0000 7f12359ad640 1 -- 172.21.15.121:0/2483460163 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f12300a4880 con 0x7f1230123f10 2024-09-17T18:27:16.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.775+0000 7f122effd640 1 --2- 172.21.15.121:0/2483460163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1230123f10 0x7f123011e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:16.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.775+0000 7f122effd640 10 monclient: get_auth_request con 0x7f1230123f10 auth_method 0 2024-09-17T18:27:16.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.775+0000 7f122effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:16.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.775+0000 7f122effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:16.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.775+0000 7f122effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:16.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.775+0000 7f122effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:16.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.775+0000 7f122effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:16.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.775+0000 7f122effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:16.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.775+0000 7f122effd640 10 monclient(hunting): handle_auth_done global_id 64103 payload 824 2024-09-17T18:27:16.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.775+0000 7f122effd640 1 -- 172.21.15.121:0/2483460163 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f123006a5f0 con 0x7f1230123f10 2024-09-17T18:27:16.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.777+0000 7f122effd640 1 -- 172.21.15.121:0/2483460163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1230123f10 msgr2=0x7f123011e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:16.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.778+0000 7f122effd640 1 --2- 172.21.15.121:0/2483460163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1230123f10 0x7f123011e640 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1220021000 tx=0x7f1220004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:16.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.778+0000 7f122cff9640 10 client.44472.objecter ms_handle_connect 0x7f1230123f10 2024-09-17T18:27:16.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.778+0000 7f122cff9640 10 client.44472.objecter resend_mon_ops 2024-09-17T18:27:16.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.780+0000 7f122cff9640 1 -- 172.21.15.121:0/2483460163 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1220022c40 con 0x7f1230123f10 2024-09-17T18:27:16.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.780+0000 7f122cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:16.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.780+0000 7f122cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:27:16.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.780+0000 7f122cff9640 10 monclient: dump: 2024-09-17T18:27:16.782 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:16.782 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:16.782 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:16.782 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:16.782 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:16.782 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:16.783 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:16.783 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:16.783 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:16.783 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:16.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.780+0000 7f122cff9640 1 -- 172.21.15.121:0/2483460163 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7f12200272c0 con 0x7f1230123f10 2024-09-17T18:27:16.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.780+0000 7f122cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:27:16.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.780+0000 7f122cff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:27:16.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.780+0000 7f122cff9640 1 -- 172.21.15.121:0/2483460163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1230123f10 msgr2=0x7f123011e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:16.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.780+0000 7f122cff9640 1 --2- 172.21.15.121:0/2483460163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1230123f10 0x7f123011e640 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1220021000 tx=0x7f1220004730 comp rx=0 tx=0).stop 2024-09-17T18:27:16.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.780+0000 7f122cff9640 1 -- 172.21.15.121:0/2483460163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1230123f10 msgr2=0x7f123011e640 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:27:16.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.780+0000 7f122cff9640 1 --2- 172.21.15.121:0/2483460163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1230123f10 0x7f123011e640 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1220021000 tx=0x7f1220004730 comp rx=0 tx=0).stop 2024-09-17T18:27:16.783 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:27:16.784 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:27:16.784 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:27:16.784 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:27:16.784 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 32, 2024-09-17T18:27:16.784 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 15512, 2024-09-17T18:27:16.784 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:27:16.784 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:27:16.784 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:27:16.784 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:27:16.784 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:27:16.784 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 5, 2024-09-17T18:27:16.784 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:27:16.785 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:27:16.785 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:27:16.785 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:27:16.785 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:27:16.785 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:27:16.785 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:27:16.785 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:27:16.785 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:27:16.785 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:27:16.785 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:27:16.785 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:27:16.785 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:27:16.785 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:27:16.786 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:27:16.786 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:27:16.786 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:27:16.786 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:27:16.786 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:27:16.786 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:27:16.786 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:27:16.786 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:27:16.786 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:27:16.786 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:27:16.786 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:27:16.786 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:27:16.786 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:27:16.787 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:27:16.787 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:16.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:16.787 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:27:16.787 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:27:16.787 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:27:16.787 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:27:16.787 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:27:16.787 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:27:16.787 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:27:16.787 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:27:16.787 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:27:16.787 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:27:16.788 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:27:16.788 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:27:16.788 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:27:16.788 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:27:16.788 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:27:16.788 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:27:16.788 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:27:16.788 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:27:16.788 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:27:16.788 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:27:16.788 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:27:16.788 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:27:16.788 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:27:16.788 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:27:16.789 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:27:16.789 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:27:16.789 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:27:16.789 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:27:16.789 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:27:16.789 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:27:16.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:16.789 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:27:16.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:16.789 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:27:16.789 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:27:16.789 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:27:16.789 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:27:16.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:16.789 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:27:16.790 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:27:16.790 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:16.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:16.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:16.790 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:27:16.790 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:27:16.790 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:16.790 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:16.790 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:16.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:16.790 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:27:16.790 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:27:16.790 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:27:16.790 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:27:16.791 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:27:16.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:16.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:16.791 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:27:16.791 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:27:16.791 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:27:16.791 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:27:16.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:16.791 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:27:16.791 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:27:16.791 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:16.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:16.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:16.791 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:27:16.792 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:27:16.792 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:16.792 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:16.792 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:16.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:16.792 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:27:16.792 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:27:16.792 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:27:16.792 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:27:16.792 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:27:16.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:16.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:16.792 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:27:16.792 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:27:16.792 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:27:16.792 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:27:16.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:16.793 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:27:16.793 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:27:16.793 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:16.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:16.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:16.793 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:27:16.793 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:27:16.793 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:16.793 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:16.793 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:16.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:16.793 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:27:16.793 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:27:16.793 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:27:16.793 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:27:16.794 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:27:16.794 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:16.794 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:16.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:16.794 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:27:16.794 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:27:16.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:16.794 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:27:16.794 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:27:16.794 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:27:16.794 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:16.794 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:27:16.794 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:27:16.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:16.794 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:27:16.795 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:27:16.795 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:27:16.795 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:16.795 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:16.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:16.795 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:27:16.795 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:27:16.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.780+0000 7f12359ad640 10 librados: watch_flush enter 2024-09-17T18:27:16.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.780+0000 7f12359ad640 10 librados: watch_flush exit 2024-09-17T18:27:16.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.780+0000 7f12359ad640 20 client.44472.objecter shutdown clearing up homeless session... 2024-09-17T18:27:16.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.780+0000 7f12359ad640 10 client.44472.objecter successfully canceled tick 2024-09-17T18:27:16.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.780+0000 7f12359ad640 1 -- 172.21.15.121:0/2483460163 >> v2:172.21.15.173:6812/2423786037 conn(0x7f12100401c0 msgr2=0x7f1210042680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:16.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.780+0000 7f12359ad640 1 --2- 172.21.15.121:0/2483460163 >> v2:172.21.15.173:6812/2423786037 conn(0x7f12100401c0 0x7f1210042680 secure :-1 s=READY pgs=3996 cs=0 l=1 rev1=1 crypto rx=0x7f1228009b50 tx=0x7f122800b040 comp rx=0 tx=0).stop 2024-09-17T18:27:16.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.780+0000 7f12359ad640 10 monclient: shutdown 2024-09-17T18:27:16.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.780+0000 7f12359ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:27:16.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.780+0000 7f12359ad640 1 -- 172.21.15.121:0/2483460163 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12300a4320 msgr2=0x7f12301cbf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:16.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.780+0000 7f12359ad640 1 --2- 172.21.15.121:0/2483460163 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12300a4320 0x7f12301cbf90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1218000c00 tx=0x7f12180046c0 comp rx=0 tx=0).stop 2024-09-17T18:27:16.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.780+0000 7f122f7fe640 1 -- 172.21.15.121:0/2483460163 reap_dead start 2024-09-17T18:27:16.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.781+0000 7f122f7fe640 1 -- 172.21.15.121:0/2483460163 reap_dead start 2024-09-17T18:27:16.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.781+0000 7f12359ad640 1 -- 172.21.15.121:0/2483460163 shutdown_connections 2024-09-17T18:27:16.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.781+0000 7f12359ad640 1 -- 172.21.15.121:0/2483460163 >> 172.21.15.121:0/2483460163 conn(0x7f12300a8680 msgr2=0x7f1230073b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:27:16.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.781+0000 7f12359ad640 1 -- 172.21.15.121:0/2483460163 shutdown_connections 2024-09-17T18:27:16.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.781+0000 7f12359ad640 1 -- 172.21.15.121:0/2483460163 wait complete. 2024-09-17T18:27:16.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.781+0000 7f12359ad640 1 librados: shutdown 2024-09-17T18:27:16.800 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:27:16.800 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T18:27:16.800 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T18:27:16.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.901+0000 7f889fd3d640 10 monclient: get_monmap_and_config 2024-09-17T18:27:16.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.901+0000 7f889fd3d640 10 monclient: build_initial_monmap 2024-09-17T18:27:16.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.901+0000 7f889fd3d640 10 monclient: monmap: 2024-09-17T18:27:16.903 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:27:16.903 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:16.903 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:27:16.902147+0000 2024-09-17T18:27:16.903 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:27:16.902147+0000 2024-09-17T18:27:16.903 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:27:16.904 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:16.904 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:27:16.904 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:27:16.904 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:27:16.904 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:27:16.904 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:27:16.904 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:16.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.902+0000 7f889fd3d640 1 Processor -- start 2024-09-17T18:27:16.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.902+0000 7f889fd3d640 1 -- start start 2024-09-17T18:27:16.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.902+0000 7f889fd3d640 10 monclient: init 2024-09-17T18:27:16.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.902+0000 7f889fd3d640 10 monclient: _reopen_session rank -1 2024-09-17T18:27:16.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.902+0000 7f889fd3d640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-17T18:27:16.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.902+0000 7f889fd3d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88980a0a00 0x7f88980a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:16.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.902+0000 7f889fd3d640 10 monclient(hunting): picked mon.noname-b con 0x7f88980a0a00 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:27:16.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.902+0000 7f889fd3d640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f88980a1320 0x7f8898134980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:16.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.902+0000 7f889fd3d640 10 monclient(hunting): picked mon.noname-c con 0x7f88980a1320 addr v2:172.21.15.173:3300/0 2024-09-17T18:27:16.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.902+0000 7f889fd3d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8898134ec0 addr v1:172.21.15.173:6789/0 2024-09-17T18:27:16.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.902+0000 7f889fd3d640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88980a3820 con 0x7f8898134ec0 2024-09-17T18:27:16.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.902+0000 7f889fd3d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:16.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.902+0000 7f889fd3d640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f88980a2ea0 con 0x7f88980a1320 2024-09-17T18:27:16.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.902+0000 7f889fd3d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:16.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.902+0000 7f889fd3d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f889809e800 con 0x7f88980a0a00 2024-09-17T18:27:16.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.902+0000 7f889fd3d640 10 monclient(hunting): _renew_subs 2024-09-17T18:27:16.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.902+0000 7f889fd3d640 10 monclient(hunting): authenticate will time out at 1240.618844s 2024-09-17T18:27:16.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.902+0000 7f889dad9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88980a0a00 0x7f88980a0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:16.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.902+0000 7f889dad9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88980a0a00 0x7f88980a0de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:46602/0 (socket says 172.21.15.121:46602) 2024-09-17T18:27:16.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.902+0000 7f889dad9640 1 -- 172.21.15.121:0/2304573755 learned_addr learned my addr 172.21.15.121:0/2304573755 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:27:16.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.902+0000 7f889dad9640 10 monclient(hunting): get_auth_request con 0x7f88980a0a00 auth_method 0 2024-09-17T18:27:16.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.902+0000 7f889dad9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:16.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.902+0000 7f889dad9640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:16.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.902+0000 7f889dad9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:16.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.903+0000 7f889dad9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:16.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.903+0000 7f889dad9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:16.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.903+0000 7f889dad9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:16.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.903+0000 7f889dad9640 10 monclient(hunting): handle_auth_done global_id 64106 payload 274 2024-09-17T18:27:16.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.903+0000 7f889dad9640 1 -- 172.21.15.121:0/2304573755 >> v2:172.21.15.173:3300/0 conn(0x7f88980a1320 msgr2=0x7f8898134980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:27:16.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.903+0000 7f889dad9640 1 --2- 172.21.15.121:0/2304573755 >> v2:172.21.15.173:3300/0 conn(0x7f88980a1320 0x7f8898134980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:16.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.903+0000 7f889dad9640 1 -- 172.21.15.121:0/2304573755 >> v1:172.21.15.173:6789/0 conn(0x7f8898134ec0 legacy=0x7f889813bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:27:16.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.903+0000 7f889dad9640 10 monclient: _finish_hunting 0 2024-09-17T18:27:16.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.903+0000 7f889dad9640 1 monclient: found mon.noname-b 2024-09-17T18:27:16.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.903+0000 7f889dad9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:27:16.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.903+0000 7f889dad9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:27:16.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.903+0000 7f889dad9640 1 -- 172.21.15.121:0/2304573755 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88980a2c30 con 0x7f88980a0a00 2024-09-17T18:27:16.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.906+0000 7f889dad9640 1 -- 172.21.15.121:0/2304573755 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88980a0a00 msgr2=0x7f88980a0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:16.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.906+0000 7f889dad9640 1 --2- 172.21.15.121:0/2304573755 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88980a0a00 0x7f88980a0de0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f889400aa10 tx=0x7f889402b460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37612bb2bb73f6d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:16.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.906+0000 7f889cad7640 1 -- 172.21.15.121:0/2304573755 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f889400b040 con 0x7f88980a0a00 2024-09-17T18:27:16.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.906+0000 7f889cad7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:16.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.906+0000 7f889cad7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:27:16.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.906+0000 7f889cad7640 10 monclient: dump: 2024-09-17T18:27:16.916 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:16.916 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:16.916 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:16.916 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:16.916 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:16.916 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:16.916 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:16.916 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:16.916 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:16.916 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:16.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.906+0000 7f889cad7640 10 monclient: _finish_auth 0 2024-09-17T18:27:16.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.906+0000 7f889cad7640 10 monclient: _check_auth_tickets 2024-09-17T18:27:16.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.906+0000 7f889cad7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:27:16.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.906+0000 7f889cad7640 1 -- 172.21.15.121:0/2304573755 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8894030a00 con 0x7f88980a0a00 2024-09-17T18:27:16.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.906+0000 7f889fd3d640 5 monclient: authenticate success, global_id 64106 2024-09-17T18:27:16.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.906+0000 7f889fd3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:27:16.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.906+0000 7f889cad7640 10 monclient: handle_config config(0 keys) 2024-09-17T18:27:16.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.906+0000 7f889fd3d640 10 monclient: get_monmap_and_config success 2024-09-17T18:27:16.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.906+0000 7f889cad7640 1 -- 172.21.15.121:0/2304573755 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8894030dd0 con 0x7f88980a0a00 2024-09-17T18:27:16.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.906+0000 7f889fd3d640 4 set_mon_vals no callback set 2024-09-17T18:27:16.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.906+0000 7f889cad7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:16.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.906+0000 7f889cad7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:27:16.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.906+0000 7f889cad7640 10 monclient: dump: 2024-09-17T18:27:16.918 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:16.918 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:16.918 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:16.918 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:16.918 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:16.918 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:16.918 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:16.918 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:16.918 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:16.918 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:16.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.906+0000 7f889fd3d640 10 monclient: shutdown 2024-09-17T18:27:16.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.906+0000 7f889fd3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:27:16.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.906+0000 7f889fd3d640 1 -- 172.21.15.121:0/2304573755 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88980a0a00 msgr2=0x7f88980a0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:16.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.906+0000 7f889fd3d640 1 --2- 172.21.15.121:0/2304573755 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88980a0a00 0x7f88980a0de0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f889400aa10 tx=0x7f889402b460 comp rx=0 tx=0).stop 2024-09-17T18:27:16.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.906+0000 7f889fd3d640 1 -- 172.21.15.121:0/2304573755 shutdown_connections 2024-09-17T18:27:16.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.906+0000 7f889fd3d640 1 --2- 172.21.15.121:0/2304573755 >> v2:172.21.15.173:3300/0 conn(0x7f88980a1320 0x7f8898134980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:16.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.906+0000 7f889fd3d640 1 --2- 172.21.15.121:0/2304573755 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88980a0a00 0x7f88980a0de0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:16.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.906+0000 7f889fd3d640 1 -- 172.21.15.121:0/2304573755 >> 172.21.15.121:0/2304573755 conn(0x7f88980a7830 msgr2=0x7f88980a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:27:16.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.906+0000 7f889fd3d640 1 -- 172.21.15.121:0/2304573755 shutdown_connections 2024-09-17T18:27:16.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.906+0000 7f889fd3d640 1 -- 172.21.15.121:0/2304573755 wait complete. 2024-09-17T18:27:16.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.907+0000 7f889fd3d640 10 monclient: build_initial_monmap 2024-09-17T18:27:16.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.907+0000 7f889fd3d640 10 monclient: monmap: 2024-09-17T18:27:16.919 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:27:16.919 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:27:16.919 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:27:16.919 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:27:16.919 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:27:16.919 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:16.919 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:27:16.920 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:27:16.920 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:27:16.920 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:16.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.907+0000 7f889fd3d640 1 librados: starting msgr at 2024-09-17T18:27:16.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.907+0000 7f889fd3d640 1 librados: starting objecter 2024-09-17T18:27:16.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.907+0000 7f889fd3d640 1 Processor -- start 2024-09-17T18:27:16.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.907+0000 7f889fd3d640 1 -- start start 2024-09-17T18:27:16.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.907+0000 7f889fd3d640 1 librados: setting wanted keys 2024-09-17T18:27:16.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.907+0000 7f889fd3d640 1 librados: calling monclient init 2024-09-17T18:27:16.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.907+0000 7f889fd3d640 10 monclient: init 2024-09-17T18:27:16.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889fd3d640 10 monclient: _reopen_session rank -1 2024-09-17T18:27:16.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889fd3d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T18:27:16.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889fd3d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88980a0a00 0x7f88981cd650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:16.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889fd3d640 10 monclient(hunting): picked mon.noname-c con 0x7f88980a0a00 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:27:16.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889fd3d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f88980a1320 0x7f8898133bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:16.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889fd3d640 10 monclient(hunting): picked mon.noname-a con 0x7f88980a1320 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:27:16.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889fd3d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8898134ec0 0x7f889806f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:16.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889fd3d640 10 monclient(hunting): picked mon.noname-b con 0x7f8898134ec0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:27:16.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889fd3d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:16.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889fd3d640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f889809de60 con 0x7f88980a1320 2024-09-17T18:27:16.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889fd3d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:16.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889fd3d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f889809dce0 con 0x7f8898134ec0 2024-09-17T18:27:16.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889fd3d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:16.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889fd3d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f88980a3120 con 0x7f88980a0a00 2024-09-17T18:27:16.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889fd3d640 10 monclient(hunting): _renew_subs 2024-09-17T18:27:16.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889fd3d640 10 monclient(hunting): authenticate will time out at 1240.624219s 2024-09-17T18:27:16.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889dad9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88980a0a00 0x7f88981cd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:16.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889d2d8640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f88980a1320 0x7f8898133bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:16.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889dad9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88980a0a00 0x7f88981cd650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:46618/0 (socket says 172.21.15.121:46618) 2024-09-17T18:27:16.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889d2d8640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f88980a1320 0x7f8898133bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56564/0 (socket says 172.21.15.121:56564) 2024-09-17T18:27:16.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889dad9640 1 -- 172.21.15.121:0/3011151331 learned_addr learned my addr 172.21.15.121:0/3011151331 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:27:16.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889dad9640 10 monclient(hunting): get_auth_request con 0x7f88980a0a00 auth_method 0 2024-09-17T18:27:16.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889dad9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:16.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889dad9640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:16.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889dad9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:16.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889d2d8640 10 monclient(hunting): get_auth_request con 0x7f88980a1320 auth_method 0 2024-09-17T18:27:16.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889d2d8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:16.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889d2d8640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:16.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889d2d8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:16.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889dad9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:16.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889dad9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:16.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889e2da640 1 --2- 172.21.15.121:0/3011151331 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8898134ec0 0x7f889806f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:16.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889dad9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:16.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889d2d8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:16.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889d2d8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:16.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889d2d8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:16.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889e2da640 10 monclient(hunting): get_auth_request con 0x7f8898134ec0 auth_method 0 2024-09-17T18:27:16.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889e2da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:16.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889e2da640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:16.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889e2da640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:16.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889e2da640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:16.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889e2da640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:16.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889e2da640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:16.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889dad9640 10 monclient(hunting): handle_auth_done global_id 64109 payload 824 2024-09-17T18:27:16.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889dad9640 1 -- 172.21.15.121:0/3011151331 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8898134ec0 msgr2=0x7f889806f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:16.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889dad9640 1 --2- 172.21.15.121:0/3011151331 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8898134ec0 0x7f889806f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:16.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889dad9640 1 -- 172.21.15.121:0/3011151331 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f88980a1320 msgr2=0x7f8898133bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:16.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889dad9640 1 --2- 172.21.15.121:0/3011151331 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f88980a1320 0x7f8898133bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:16.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889dad9640 10 monclient: _finish_hunting 0 2024-09-17T18:27:16.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889dad9640 1 monclient: found mon.noname-c 2024-09-17T18:27:16.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889dad9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:27:16.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889dad9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:27:16.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889dad9640 1 -- 172.21.15.121:0/3011151331 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8898070c90 con 0x7f88980a0a00 2024-09-17T18:27:16.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.908+0000 7f889d2d8640 1 --2- 172.21.15.121:0/3011151331 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f88980a1320 0x7f8898133bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:27:16.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.910+0000 7f889dad9640 1 -- 172.21.15.121:0/3011151331 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88980a0a00 msgr2=0x7f88981cd650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:16.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.910+0000 7f889dad9640 1 --2- 172.21.15.121:0/3011151331 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88980a0a00 0x7f88981cd650 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f889400a1c0 tx=0x7f88940046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:16.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.910+0000 7f888e7fc640 10 client.?.objecter ms_handle_connect 0x7f88980a0a00 2024-09-17T18:27:16.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.910+0000 7f888e7fc640 10 client.?.objecter resend_mon_ops 2024-09-17T18:27:16.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.911+0000 7f888e7fc640 1 -- 172.21.15.121:0/3011151331 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8894040020 con 0x7f88980a0a00 2024-09-17T18:27:16.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.911+0000 7f888e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:16.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.911+0000 7f888e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:27:16.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.911+0000 7f888e7fc640 10 monclient: dump: 2024-09-17T18:27:16.925 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:16.925 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:16.925 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:16.925 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:16.925 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:16.925 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:16.925 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:16.925 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:16.925 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:16.926 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:16.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.911+0000 7f888e7fc640 10 monclient: _finish_auth 0 2024-09-17T18:27:16.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.911+0000 7f888e7fc640 10 monclient: _check_auth_tickets 2024-09-17T18:27:16.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.911+0000 7f888e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:27:16.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.911+0000 7f888e7fc640 1 -- 172.21.15.121:0/3011151331 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8894032e10 con 0x7f88980a0a00 2024-09-17T18:27:16.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.911+0000 7f888e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:27:16.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.911+0000 7f888e7fc640 1 -- 172.21.15.121:0/3011151331 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f889403b670 con 0x7f88980a0a00 2024-09-17T18:27:16.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.911+0000 7f888e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:16.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.911+0000 7f888e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:27:16.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.911+0000 7f888e7fc640 10 monclient: dump: 2024-09-17T18:27:16.926 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:16.926 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:16.926 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:16.926 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:16.927 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:16.927 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:16.927 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:16.927 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:16.927 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:16.927 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:16.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.911+0000 7f889fd3d640 5 monclient: authenticate success, global_id 64109 2024-09-17T18:27:16.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.911+0000 7f889fd3d640 10 monclient: _renew_subs 2024-09-17T18:27:16.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.911+0000 7f889fd3d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:27:16.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.911+0000 7f888ffff640 4 set_mon_vals no callback set 2024-09-17T18:27:16.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.911+0000 7f889fd3d640 1 -- 172.21.15.121:0/3011151331 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88980a5dd0 con 0x7f88980a0a00 2024-09-17T18:27:16.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.911+0000 7f889fd3d640 10 client.64109.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:27:16.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.911+0000 7f889fd3d640 10 monclient: _renew_subs 2024-09-17T18:27:16.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.911+0000 7f889fd3d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:27:16.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.911+0000 7f889fd3d640 1 -- 172.21.15.121:0/3011151331 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f88981235d0 con 0x7f88980a0a00 2024-09-17T18:27:16.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.911+0000 7f889fd3d640 1 librados: init done 2024-09-17T18:27:16.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.911+0000 7f889fd3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:27:16.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.911+0000 7f889fd3d640 1 --2- 172.21.15.121:0/3011151331 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8898134130 0x7f889802d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:16.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.911+0000 7f889fd3d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:16.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.911+0000 7f889fd3d640 1 -- 172.21.15.121:0/3011151331 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8898121bc0 con 0x7f8898134130 2024-09-17T18:27:16.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.911+0000 7f888e7fc640 1 -- 172.21.15.121:0/3011151331 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f8894043020 con 0x7f88980a0a00 2024-09-17T18:27:16.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.912+0000 7f888e7fc640 1 --2- 172.21.15.121:0/3011151331 >> v2:172.21.15.173:6812/2423786037 conn(0x7f887c0401a0 0x7f887c042660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:16.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.912+0000 7f889d2d8640 1 --2- 172.21.15.121:0/3011151331 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8898134130 0x7f889802d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:16.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.912+0000 7f888e7fc640 1 -- 172.21.15.121:0/3011151331 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(341..341 src has 177..341) ==== 5358+0+0 (secure 0 0 0) 0x7f88940740e0 con 0x7f88980a0a00 2024-09-17T18:27:16.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.912+0000 7f888e7fc640 10 client.64109.objecter ms_dispatch 0x7f8898000b90 osd_map(341..341 src has 177..341) 2024-09-17T18:27:16.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.912+0000 7f888e7fc640 3 client.64109.objecter handle_osd_map got epochs [341,341] > 0 2024-09-17T18:27:16.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.912+0000 7f888e7fc640 3 client.64109.objecter handle_osd_map decoding full epoch 341 2024-09-17T18:27:16.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.912+0000 7f889d2d8640 10 monclient: get_auth_request con 0x7f8898134130 auth_method 0 2024-09-17T18:27:16.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.912+0000 7f889d2d8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:16.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.912+0000 7f889d2d8640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:16.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.912+0000 7f889d2d8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:16.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.912+0000 7f888e7fc640 20 client.64109.objecter dump_active .. 0 homeless 2024-09-17T18:27:16.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.912+0000 7f889d2d8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:16.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.912+0000 7f889d2d8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:16.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.912+0000 7f889d2d8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:16.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.912+0000 7f889e2da640 1 --2- 172.21.15.121:0/3011151331 >> v2:172.21.15.173:6812/2423786037 conn(0x7f887c0401a0 0x7f887c042660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:16.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.912+0000 7f889e2da640 10 monclient: get_auth_request con 0x7f887c0401a0 auth_method 0 2024-09-17T18:27:16.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.912+0000 7f889d2d8640 10 monclient(hunting): handle_auth_done global_id 24787 payload 824 2024-09-17T18:27:16.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.912+0000 7f889d2d8640 1 -- 172.21.15.121:0/3011151331 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f88981231a0 con 0x7f8898134130 2024-09-17T18:27:16.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.914+0000 7f888e7fc640 10 client.64109.objecter ms_handle_connect 0x7f8898134130 2024-09-17T18:27:16.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.914+0000 7f889d2d8640 1 -- 172.21.15.121:0/3011151331 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8898134130 msgr2=0x7f889802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:16.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.914+0000 7f888e7fc640 10 client.64109.objecter resend_mon_ops 2024-09-17T18:27:16.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.914+0000 7f889d2d8640 1 --2- 172.21.15.121:0/3011151331 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8898134130 0x7f889802d440 secure :-1 s=READY pgs=7427 cs=0 l=1 rev1=1 crypto rx=0x7f88981348e0 tx=0x7f888800ef70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:16.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.914+0000 7f888e7fc640 10 client.64109.objecter ms_handle_connect 0x7f887c0401a0 2024-09-17T18:27:16.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.914+0000 7f889e2da640 1 --2- 172.21.15.121:0/3011151331 >> v2:172.21.15.173:6812/2423786037 conn(0x7f887c0401a0 0x7f887c042660 secure :-1 s=READY pgs=3997 cs=0 l=1 rev1=1 crypto rx=0x7f8890001e70 tx=0x7f8890009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:16.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.915+0000 7f888e7fc640 1 -- 172.21.15.121:0/3011151331 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8888017070 con 0x7f8898134130 2024-09-17T18:27:16.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.915+0000 7f888e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:16.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.915+0000 7f888e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:27:16.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.915+0000 7f888e7fc640 10 monclient: dump: 2024-09-17T18:27:16.932 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:16.933 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:16.933 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:16.933 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:16.933 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:16.933 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:16.933 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:16.933 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:16.933 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:16.933 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:16.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.915+0000 7f888e7fc640 1 -- 172.21.15.121:0/3011151331 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f888800cd90 con 0x7f8898134130 2024-09-17T18:27:16.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.915+0000 7f888e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:27:16.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.915+0000 7f888e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:27:16.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.915+0000 7f888e7fc640 1 -- 172.21.15.121:0/3011151331 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8898134130 msgr2=0x7f889802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:16.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.915+0000 7f888e7fc640 1 --2- 172.21.15.121:0/3011151331 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8898134130 0x7f889802d440 secure :-1 s=READY pgs=7427 cs=0 l=1 rev1=1 crypto rx=0x7f88981348e0 tx=0x7f888800ef70 comp rx=0 tx=0).stop 2024-09-17T18:27:16.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.915+0000 7f888e7fc640 1 -- 172.21.15.121:0/3011151331 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8898134130 msgr2=0x7f889802d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:27:16.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.915+0000 7f888e7fc640 1 --2- 172.21.15.121:0/3011151331 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8898134130 0x7f889802d440 secure :-1 s=CLOSED pgs=7427 cs=0 l=1 rev1=1 crypto rx=0x7f88981348e0 tx=0x7f888800ef70 comp rx=0 tx=0).stop 2024-09-17T18:27:16.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.921+0000 7f889fd3d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:27:16.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.921+0000 7f889fd3d640 1 --2- 172.21.15.121:0/3011151331 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f889802d980 0x7f889802dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:16.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.921+0000 7f889fd3d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:16.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.921+0000 7f889fd3d640 1 -- 172.21.15.121:0/3011151331 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f88980a2030 con 0x7f889802d980 2024-09-17T18:27:16.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.921+0000 7f889d2d8640 1 --2- 172.21.15.121:0/3011151331 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f889802d980 0x7f889802dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:16.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.921+0000 7f889d2d8640 10 monclient: get_auth_request con 0x7f889802d980 auth_method 0 2024-09-17T18:27:16.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.921+0000 7f889d2d8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:16.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.921+0000 7f889d2d8640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:16.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.921+0000 7f889d2d8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:16.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.921+0000 7f889d2d8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:16.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.921+0000 7f889d2d8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:16.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.921+0000 7f889d2d8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:16.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.921+0000 7f889d2d8640 10 monclient(hunting): handle_auth_done global_id 24790 payload 824 2024-09-17T18:27:16.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.921+0000 7f889d2d8640 1 -- 172.21.15.121:0/3011151331 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f88981bce50 con 0x7f889802d980 2024-09-17T18:27:16.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.924+0000 7f889d2d8640 1 -- 172.21.15.121:0/3011151331 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f889802d980 msgr2=0x7f889802dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:16.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.924+0000 7f888e7fc640 10 client.64109.objecter ms_handle_connect 0x7f889802d980 2024-09-17T18:27:16.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.924+0000 7f888e7fc640 10 client.64109.objecter resend_mon_ops 2024-09-17T18:27:16.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.924+0000 7f889d2d8640 1 --2- 172.21.15.121:0/3011151331 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f889802d980 0x7f889802dd90 secure :-1 s=READY pgs=7428 cs=0 l=1 rev1=1 crypto rx=0x7f8888019000 tx=0x7f88880047d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:16.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.924+0000 7f888e7fc640 1 -- 172.21.15.121:0/3011151331 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8888017070 con 0x7f889802d980 2024-09-17T18:27:16.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.924+0000 7f888e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:16.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.924+0000 7f888e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:27:16.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.924+0000 7f888e7fc640 10 monclient: dump: 2024-09-17T18:27:16.935 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:16.935 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:16.935 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:16.935 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:16.935 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:16.936 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:16.936 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:16.936 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:16.936 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:16.936 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:16.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.924+0000 7f888e7fc640 1 -- 172.21.15.121:0/3011151331 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f8888013470 con 0x7f889802d980 2024-09-17T18:27:16.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.924+0000 7f888e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:27:16.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.924+0000 7f888e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:27:16.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.924+0000 7f888e7fc640 1 -- 172.21.15.121:0/3011151331 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f889802d980 msgr2=0x7f889802dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:16.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.924+0000 7f888e7fc640 1 --2- 172.21.15.121:0/3011151331 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f889802d980 0x7f889802dd90 secure :-1 s=READY pgs=7428 cs=0 l=1 rev1=1 crypto rx=0x7f8888019000 tx=0x7f88880047d0 comp rx=0 tx=0).stop 2024-09-17T18:27:16.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.924+0000 7f888e7fc640 1 -- 172.21.15.121:0/3011151331 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f889802d980 msgr2=0x7f889802dd90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:27:16.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.924+0000 7f888e7fc640 1 --2- 172.21.15.121:0/3011151331 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f889802d980 0x7f889802dd90 unknown :-1 s=CLOSED pgs=7428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:16.937 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:27:16.937 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:27:16.937 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:27:16.937 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:27:16.937 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 32, 2024-09-17T18:27:16.937 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 509869, 2024-09-17T18:27:16.937 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:27:16.937 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:27:16.937 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:27:16.937 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:27:16.937 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:27:16.937 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 5, 2024-09-17T18:27:16.938 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:27:16.938 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:27:16.938 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:27:16.938 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:27:16.938 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:27:16.938 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:27:16.938 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:27:16.938 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:27:16.938 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:27:16.938 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:27:16.938 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:27:16.938 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:27:16.938 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:27:16.939 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:27:16.939 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:27:16.939 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:27:16.939 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:27:16.939 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:27:16.939 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:27:16.939 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:27:16.939 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:27:16.939 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:27:16.939 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:27:16.939 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:27:16.939 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:27:16.939 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:27:16.939 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:27:16.939 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:27:16.940 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:16.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:16.940 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:27:16.940 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:27:16.940 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:27:16.940 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:27:16.940 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:27:16.940 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:27:16.940 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:27:16.940 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:27:16.940 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:27:16.940 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:27:16.940 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:27:16.941 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:27:16.941 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:27:16.941 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:27:16.941 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:27:16.941 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:27:16.941 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:27:16.941 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:27:16.941 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:27:16.941 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:27:16.941 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:27:16.941 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:27:16.941 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:27:16.941 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:27:16.942 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:27:16.942 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:27:16.942 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:27:16.942 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:27:16.942 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:27:16.942 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:27:16.942 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:16.942 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:27:16.942 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:16.942 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:27:16.942 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:27:16.942 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:27:16.942 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:27:16.942 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:16.943 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:27:16.943 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:27:16.943 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:16.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:16.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:16.943 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:27:16.943 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:27:16.943 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:16.943 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:16.943 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:16.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:16.943 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:27:16.943 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:27:16.943 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:27:16.944 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:27:16.944 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:27:16.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:16.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:16.944 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:27:16.944 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:27:16.944 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:27:16.944 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:27:16.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:16.944 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:27:16.944 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:27:16.944 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:16.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:16.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:16.944 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:27:16.945 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:27:16.945 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:16.945 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:16.945 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:16.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:16.945 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:27:16.945 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:27:16.945 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:27:16.945 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:27:16.945 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:27:16.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:16.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:16.945 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:27:16.946 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:27:16.946 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:27:16.946 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:27:16.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:16.946 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:27:16.946 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:27:16.946 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:16.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:16.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:16.946 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:27:16.946 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:27:16.946 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:16.946 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:16.946 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:16.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:16.947 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:27:16.947 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:27:16.947 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:27:16.947 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:27:16.947 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:27:16.947 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:16.947 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:16.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:16.947 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:27:16.947 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:27:16.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:16.947 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:27:16.947 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:27:16.948 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:27:16.948 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:16.948 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:27:16.948 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:27:16.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:16.948 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:27:16.948 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:27:16.948 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 4 2024-09-17T18:27:16.948 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:16.948 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:27:16.948 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:27:16.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:16.948 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:27:16.949 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:27:16.949 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 6 2024-09-17T18:27:16.949 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:16.949 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:27:16.949 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": [ 2024-09-17T18:27:16.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:16.949 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:27:16.949 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:27:16.949 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:27:16.949 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:16.949 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:16.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:16.949 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:27:16.950 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:27:16.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.924+0000 7f889fd3d640 10 librados: watch_flush enter 2024-09-17T18:27:16.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.924+0000 7f889fd3d640 10 librados: watch_flush exit 2024-09-17T18:27:16.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.924+0000 7f889fd3d640 20 client.64109.objecter shutdown clearing up homeless session... 2024-09-17T18:27:16.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.924+0000 7f889fd3d640 10 client.64109.objecter successfully canceled tick 2024-09-17T18:27:16.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.924+0000 7f889fd3d640 1 -- 172.21.15.121:0/3011151331 >> v2:172.21.15.173:6812/2423786037 conn(0x7f887c0401a0 msgr2=0x7f887c042660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:16.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.924+0000 7f889fd3d640 1 --2- 172.21.15.121:0/3011151331 >> v2:172.21.15.173:6812/2423786037 conn(0x7f887c0401a0 0x7f887c042660 secure :-1 s=READY pgs=3997 cs=0 l=1 rev1=1 crypto rx=0x7f8890001e70 tx=0x7f8890009040 comp rx=0 tx=0).stop 2024-09-17T18:27:16.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.924+0000 7f889fd3d640 10 monclient: shutdown 2024-09-17T18:27:16.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.924+0000 7f889fd3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:27:16.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.924+0000 7f889fd3d640 1 -- 172.21.15.121:0/3011151331 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88980a0a00 msgr2=0x7f88981cd650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:16.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.924+0000 7f889fd3d640 1 --2- 172.21.15.121:0/3011151331 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88980a0a00 0x7f88981cd650 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f889400a1c0 tx=0x7f88940046c0 comp rx=0 tx=0).stop 2024-09-17T18:27:16.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.924+0000 7f889e2da640 1 -- 172.21.15.121:0/3011151331 reap_dead start 2024-09-17T18:27:16.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.924+0000 7f889e2da640 1 -- 172.21.15.121:0/3011151331 reap_dead start 2024-09-17T18:27:16.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.924+0000 7f889fd3d640 1 -- 172.21.15.121:0/3011151331 shutdown_connections 2024-09-17T18:27:16.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.924+0000 7f889fd3d640 1 -- 172.21.15.121:0/3011151331 >> 172.21.15.121:0/3011151331 conn(0x7f88980a7830 msgr2=0x7f88981ba020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:27:16.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.925+0000 7f889fd3d640 1 -- 172.21.15.121:0/3011151331 shutdown_connections 2024-09-17T18:27:16.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.925+0000 7f889fd3d640 1 -- 172.21.15.121:0/3011151331 wait complete. 2024-09-17T18:27:16.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:16.925+0000 7f889fd3d640 1 librados: shutdown 2024-09-17T18:27:16.953 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:27:16.953 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-17T18:27:16.953 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-17T18:27:17.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.049+0000 7f8eedcbe640 10 monclient: get_monmap_and_config 2024-09-17T18:27:17.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.049+0000 7f8eedcbe640 10 monclient: build_initial_monmap 2024-09-17T18:27:17.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.049+0000 7f8eedcbe640 10 monclient: monmap: 2024-09-17T18:27:17.051 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:27:17.051 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:17.051 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:27:17.049764+0000 2024-09-17T18:27:17.051 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:27:17.049764+0000 2024-09-17T18:27:17.051 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:27:17.051 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:17.051 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:27:17.051 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:27:17.051 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:27:17.051 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:27:17.052 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:27:17.052 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:17.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.050+0000 7f8eedcbe640 1 Processor -- start 2024-09-17T18:27:17.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.050+0000 7f8eedcbe640 1 -- start start 2024-09-17T18:27:17.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.050+0000 7f8eedcbe640 10 monclient: init 2024-09-17T18:27:17.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.050+0000 7f8eedcbe640 10 monclient: _reopen_session rank -1 2024-09-17T18:27:17.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.050+0000 7f8eedcbe640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-17T18:27:17.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.050+0000 7f8eedcbe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8ee80a5c00 0x7f8ee809d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:17.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.050+0000 7f8eedcbe640 10 monclient(hunting): picked mon.noname-b con 0x7f8ee80a5c00 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:27:17.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.050+0000 7f8eedcbe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8ee80a5fe0 addr v1:172.21.15.121:6789/0 2024-09-17T18:27:17.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.050+0000 7f8eedcbe640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f8ee813dc00 0x7f8ee813fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:17.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.050+0000 7f8eedcbe640 10 monclient(hunting): picked mon.noname-c con 0x7f8ee813dc00 addr v2:172.21.15.173:3300/0 2024-09-17T18:27:17.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.050+0000 7f8eedcbe640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ee80a8a80 con 0x7f8ee80a5fe0 2024-09-17T18:27:17.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.050+0000 7f8eedcbe640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:17.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.050+0000 7f8eedcbe640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f8ee80a7110 con 0x7f8ee813dc00 2024-09-17T18:27:17.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.050+0000 7f8eedcbe640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:17.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.050+0000 7f8eedcbe640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8ee809de00 con 0x7f8ee80a5c00 2024-09-17T18:27:17.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.050+0000 7f8eedcbe640 10 monclient(hunting): _renew_subs 2024-09-17T18:27:17.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.050+0000 7f8eedcbe640 10 monclient(hunting): authenticate will time out at 1240.766614s 2024-09-17T18:27:17.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.050+0000 7f8edffff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f8ee80a5fe0 0x7f8ee80a63c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:35578/0 (socket says 172.21.15.121:35578) 2024-09-17T18:27:17.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.050+0000 7f8ee77fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8ee80a5c00 0x7f8ee809d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:17.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.050+0000 7f8edffff640 1 -- 172.21.15.121:0/1182384886 learned_addr learned my addr 172.21.15.121:0/1182384886 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:27:17.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.050+0000 7f8ee77fe640 10 monclient(hunting): get_auth_request con 0x7f8ee80a5c00 auth_method 0 2024-09-17T18:27:17.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.050+0000 7f8ee77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:17.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.050+0000 7f8ee77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:17.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.050+0000 7f8ee77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:17.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.050+0000 7f8ee77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:17.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.050+0000 7f8ee77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:17.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.050+0000 7f8ee77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:17.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.050+0000 7f8ee7fff640 1 --2- 172.21.15.121:0/1182384886 >> v2:172.21.15.173:3300/0 conn(0x7f8ee813dc00 0x7f8ee813fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:17.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.050+0000 7f8ee7fff640 10 monclient(hunting): get_auth_request con 0x7f8ee813dc00 auth_method 0 2024-09-17T18:27:17.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.050+0000 7f8ee7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:17.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.050+0000 7f8ee7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:17.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.050+0000 7f8ee7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:17.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.050+0000 7f8ee7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:17.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.050+0000 7f8ee7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:17.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.050+0000 7f8ee7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:17.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.050+0000 7f8ee77fe640 10 monclient(hunting): handle_auth_done global_id 64112 payload 274 2024-09-17T18:27:17.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.051+0000 7f8ee77fe640 1 -- 172.21.15.121:0/1182384886 >> v2:172.21.15.173:3300/0 conn(0x7f8ee813dc00 msgr2=0x7f8ee813fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:27:17.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.051+0000 7f8ee77fe640 1 --2- 172.21.15.121:0/1182384886 >> v2:172.21.15.173:3300/0 conn(0x7f8ee813dc00 0x7f8ee813fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:17.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.051+0000 7f8ee77fe640 1 -- 172.21.15.121:0/1182384886 >> v1:172.21.15.121:6789/0 conn(0x7f8ee80a5fe0 legacy=0x7f8ee80a63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:27:17.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.051+0000 7f8ee77fe640 10 monclient: _finish_hunting 0 2024-09-17T18:27:17.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.051+0000 7f8ee77fe640 1 monclient: found mon.noname-b 2024-09-17T18:27:17.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.051+0000 7f8ee77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:27:17.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.051+0000 7f8ee77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:27:17.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.051+0000 7f8ee77fe640 1 -- 172.21.15.121:0/1182384886 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ee80a74d0 con 0x7f8ee80a5c00 2024-09-17T18:27:17.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.053+0000 7f8ee77fe640 1 -- 172.21.15.121:0/1182384886 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8ee80a5c00 msgr2=0x7f8ee809d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:17.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.053+0000 7f8ee77fe640 1 --2- 172.21.15.121:0/1182384886 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8ee80a5c00 0x7f8ee809d8c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8ee809dfd0 tx=0x7f8ed8029fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7aabc899dd509097 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:17.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.054+0000 7f8ee6ffd640 1 -- 172.21.15.121:0/1182384886 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ed8037070 con 0x7f8ee80a5c00 2024-09-17T18:27:17.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.054+0000 7f8ee6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:17.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.054+0000 7f8ee6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:27:17.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.054+0000 7f8ee6ffd640 10 monclient: dump: 2024-09-17T18:27:17.063 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:17.063 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:17.063 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:17.063 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:17.064 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:17.064 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:17.064 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:17.064 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:17.064 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:17.064 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:17.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.054+0000 7f8ee6ffd640 10 monclient: _finish_auth 0 2024-09-17T18:27:17.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.054+0000 7f8ee6ffd640 10 monclient: _check_auth_tickets 2024-09-17T18:27:17.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.054+0000 7f8ee6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:27:17.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.054+0000 7f8ee6ffd640 1 -- 172.21.15.121:0/1182384886 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ed8032440 con 0x7f8ee80a5c00 2024-09-17T18:27:17.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.054+0000 7f8eedcbe640 5 monclient: authenticate success, global_id 64112 2024-09-17T18:27:17.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.054+0000 7f8eedcbe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:27:17.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.054+0000 7f8ee6ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:27:17.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.054+0000 7f8eedcbe640 10 monclient: get_monmap_and_config success 2024-09-17T18:27:17.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.054+0000 7f8ee6ffd640 1 -- 172.21.15.121:0/1182384886 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ed8032810 con 0x7f8ee80a5c00 2024-09-17T18:27:17.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.054+0000 7f8eedcbe640 4 set_mon_vals no callback set 2024-09-17T18:27:17.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.054+0000 7f8ee6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:17.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.054+0000 7f8ee6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:27:17.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.054+0000 7f8ee6ffd640 10 monclient: dump: 2024-09-17T18:27:17.065 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:17.065 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:17.065 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:17.065 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:17.065 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:17.065 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:17.065 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:17.065 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:17.065 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:17.065 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:17.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.054+0000 7f8eedcbe640 10 monclient: shutdown 2024-09-17T18:27:17.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.054+0000 7f8eedcbe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:27:17.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.054+0000 7f8eedcbe640 1 -- 172.21.15.121:0/1182384886 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8ee80a5c00 msgr2=0x7f8ee809d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:17.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.054+0000 7f8eedcbe640 1 --2- 172.21.15.121:0/1182384886 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8ee80a5c00 0x7f8ee809d8c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8ee809dfd0 tx=0x7f8ed8029fb0 comp rx=0 tx=0).stop 2024-09-17T18:27:17.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.054+0000 7f8eedcbe640 1 -- 172.21.15.121:0/1182384886 shutdown_connections 2024-09-17T18:27:17.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.054+0000 7f8eedcbe640 1 --2- 172.21.15.121:0/1182384886 >> v2:172.21.15.173:3300/0 conn(0x7f8ee813dc00 0x7f8ee813fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:17.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.054+0000 7f8eedcbe640 1 --2- 172.21.15.121:0/1182384886 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8ee80a5c00 0x7f8ee809d8c0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:17.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.054+0000 7f8eedcbe640 1 -- 172.21.15.121:0/1182384886 >> 172.21.15.121:0/1182384886 conn(0x7f8ee80ab9b0 msgr2=0x7f8ee80a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:27:17.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.054+0000 7f8eedcbe640 1 -- 172.21.15.121:0/1182384886 shutdown_connections 2024-09-17T18:27:17.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.054+0000 7f8eedcbe640 1 -- 172.21.15.121:0/1182384886 wait complete. 2024-09-17T18:27:17.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.055+0000 7f8eedcbe640 10 monclient: build_initial_monmap 2024-09-17T18:27:17.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.055+0000 7f8eedcbe640 10 monclient: monmap: 2024-09-17T18:27:17.067 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:27:17.067 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:27:17.067 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:27:17.067 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:27:17.067 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:27:17.067 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:17.067 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:27:17.067 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:27:17.067 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:27:17.067 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:17.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.055+0000 7f8eedcbe640 1 librados: starting msgr at 2024-09-17T18:27:17.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.055+0000 7f8eedcbe640 1 librados: starting objecter 2024-09-17T18:27:17.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.055+0000 7f8eedcbe640 1 Processor -- start 2024-09-17T18:27:17.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.055+0000 7f8eedcbe640 1 -- start start 2024-09-17T18:27:17.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.055+0000 7f8eedcbe640 1 librados: setting wanted keys 2024-09-17T18:27:17.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.055+0000 7f8eedcbe640 1 librados: calling monclient init 2024-09-17T18:27:17.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.055+0000 7f8eedcbe640 10 monclient: init 2024-09-17T18:27:17.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8eedcbe640 10 monclient: _reopen_session rank -1 2024-09-17T18:27:17.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8eedcbe640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:27:17.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8eedcbe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ee80a5c00 0x7f8ee81cd720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:17.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8eedcbe640 10 monclient(hunting): picked mon.noname-a con 0x7f8ee80a5c00 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:27:17.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8eedcbe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ee80a5fe0 0x7f8ee81cddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:17.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8eedcbe640 10 monclient(hunting): picked mon.noname-b con 0x7f8ee80a5fe0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:27:17.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8eedcbe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8ee813dc00 0x7f8ee813bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:17.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8eedcbe640 10 monclient(hunting): picked mon.noname-c con 0x7f8ee813dc00 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:27:17.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8eedcbe640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:17.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8eedcbe640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee8070550 con 0x7f8ee80a5c00 2024-09-17T18:27:17.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8eedcbe640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:17.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8eedcbe640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee80703d0 con 0x7f8ee80a5fe0 2024-09-17T18:27:17.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8eedcbe640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:17.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8eedcbe640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8ee8140530 con 0x7f8ee813dc00 2024-09-17T18:27:17.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8eedcbe640 10 monclient(hunting): _renew_subs 2024-09-17T18:27:17.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8eedcbe640 10 monclient(hunting): authenticate will time out at 1240.772109s 2024-09-17T18:27:17.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8ee77fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ee80a5c00 0x7f8ee81cd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:17.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8ee7fff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8ee813dc00 0x7f8ee813bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:17.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8ee7fff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8ee813dc00 0x7f8ee813bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:46638/0 (socket says 172.21.15.121:46638) 2024-09-17T18:27:17.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8ee77fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ee80a5c00 0x7f8ee81cd720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56578/0 (socket says 172.21.15.121:56578) 2024-09-17T18:27:17.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8ee7fff640 1 -- 172.21.15.121:0/4271264050 learned_addr learned my addr 172.21.15.121:0/4271264050 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:27:17.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8ee7fff640 10 monclient(hunting): get_auth_request con 0x7f8ee813dc00 auth_method 0 2024-09-17T18:27:17.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8ee7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:17.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8ee7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:17.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8ee7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:17.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8ee77fe640 10 monclient(hunting): get_auth_request con 0x7f8ee80a5c00 auth_method 0 2024-09-17T18:27:17.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8ee77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:17.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8ee77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:17.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8ee77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:17.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8ee7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:17.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8ee7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:17.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8ee7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:17.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8ee77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:17.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8ee77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:17.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8ee77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:17.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8edffff640 1 --2- 172.21.15.121:0/4271264050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ee80a5fe0 0x7f8ee81cddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:17.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8edffff640 10 monclient(hunting): get_auth_request con 0x7f8ee80a5fe0 auth_method 0 2024-09-17T18:27:17.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8edffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:17.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8edffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:17.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8edffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:17.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8ee7fff640 10 monclient(hunting): handle_auth_done global_id 64115 payload 824 2024-09-17T18:27:17.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8ee7fff640 1 -- 172.21.15.121:0/4271264050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ee80a5fe0 msgr2=0x7f8ee81cddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:17.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8ee7fff640 1 --2- 172.21.15.121:0/4271264050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ee80a5fe0 0x7f8ee81cddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:17.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8ee7fff640 1 -- 172.21.15.121:0/4271264050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ee80a5c00 msgr2=0x7f8ee81cd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:17.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8ee7fff640 1 --2- 172.21.15.121:0/4271264050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ee80a5c00 0x7f8ee81cd720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:17.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8ee7fff640 10 monclient: _finish_hunting 0 2024-09-17T18:27:17.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8ee7fff640 1 monclient: found mon.noname-c 2024-09-17T18:27:17.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8ee7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:27:17.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8ee7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:27:17.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8ee7fff640 1 -- 172.21.15.121:0/4271264050 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ee80a23f0 con 0x7f8ee813dc00 2024-09-17T18:27:17.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.056+0000 7f8edffff640 1 --2- 172.21.15.121:0/4271264050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ee80a5fe0 0x7f8ee81cddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T18:27:17.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.058+0000 7f8ee7fff640 1 -- 172.21.15.121:0/4271264050 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8ee813dc00 msgr2=0x7f8ee813bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:17.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.058+0000 7f8ee7fff640 1 --2- 172.21.15.121:0/4271264050 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8ee813dc00 0x7f8ee813bb30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8ee000efc0 tx=0x7f8ee000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:17.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.058+0000 7f8ee4ff9640 10 client.?.objecter ms_handle_connect 0x7f8ee813dc00 2024-09-17T18:27:17.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.058+0000 7f8ee4ff9640 10 client.?.objecter resend_mon_ops 2024-09-17T18:27:17.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.058+0000 7f8ee4ff9640 1 -- 172.21.15.121:0/4271264050 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee0018020 con 0x7f8ee813dc00 2024-09-17T18:27:17.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.058+0000 7f8ee4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:17.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.058+0000 7f8ee4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:27:17.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.058+0000 7f8ee4ff9640 10 monclient: dump: 2024-09-17T18:27:17.072 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:17.072 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:17.072 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:17.072 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:17.072 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:17.072 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:17.072 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:17.072 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:17.072 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:17.072 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:17.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.058+0000 7f8ee4ff9640 10 monclient: _finish_auth 0 2024-09-17T18:27:17.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.058+0000 7f8ee4ff9640 10 monclient: _check_auth_tickets 2024-09-17T18:27:17.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.058+0000 7f8ee4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:27:17.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.058+0000 7f8ee4ff9640 1 -- 172.21.15.121:0/4271264050 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ee0014680 con 0x7f8ee813dc00 2024-09-17T18:27:17.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.058+0000 7f8eedcbe640 5 monclient: authenticate success, global_id 64115 2024-09-17T18:27:17.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.058+0000 7f8eedcbe640 10 monclient: _renew_subs 2024-09-17T18:27:17.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.058+0000 7f8eedcbe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:27:17.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.058+0000 7f8eedcbe640 1 -- 172.21.15.121:0/4271264050 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ee8122b80 con 0x7f8ee813dc00 2024-09-17T18:27:17.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.058+0000 7f8ee4ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T18:27:17.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.058+0000 7f8ee4ff9640 1 -- 172.21.15.121:0/4271264050 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee0014a70 con 0x7f8ee813dc00 2024-09-17T18:27:17.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.058+0000 7f8ee4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:17.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.058+0000 7f8ee4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:27:17.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.058+0000 7f8ee4ff9640 10 monclient: dump: 2024-09-17T18:27:17.073 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:17.073 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:17.073 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:17.074 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:17.074 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:17.074 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:17.074 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:17.074 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:17.074 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:17.074 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:17.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.058+0000 7f8eedcbe640 10 client.64115.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:27:17.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.058+0000 7f8ee67fc640 4 set_mon_vals no callback set 2024-09-17T18:27:17.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.058+0000 7f8eedcbe640 10 monclient: _renew_subs 2024-09-17T18:27:17.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.058+0000 7f8eedcbe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:27:17.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.058+0000 7f8eedcbe640 1 -- 172.21.15.121:0/4271264050 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ee81390c0 con 0x7f8ee813dc00 2024-09-17T18:27:17.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.058+0000 7f8eedcbe640 1 librados: init done 2024-09-17T18:27:17.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.059+0000 7f8eedcbe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:27:17.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.059+0000 7f8eedcbe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:27:17.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.059+0000 7f8eedcbe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:27:17.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.059+0000 7f8eedcbe640 1 -- 172.21.15.121:0/4271264050 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ee80a1e90 con 0x7f8ee813dc00 2024-09-17T18:27:17.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.059+0000 7f8ee4ff9640 1 -- 172.21.15.121:0/4271264050 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f8ee0005d10 con 0x7f8ee813dc00 2024-09-17T18:27:17.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.060+0000 7f8ee4ff9640 1 --2- 172.21.15.121:0/4271264050 >> v2:172.21.15.173:6812/2423786037 conn(0x7f8ec80401a0 0x7f8ec8042660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:17.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.060+0000 7f8ee4ff9640 1 -- 172.21.15.121:0/4271264050 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(341..341 src has 177..341) ==== 5358+0+0 (secure 0 0 0) 0x7f8ee00546c0 con 0x7f8ee813dc00 2024-09-17T18:27:17.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.060+0000 7f8ee4ff9640 10 client.64115.objecter ms_dispatch 0x7f8ee8000b90 osd_map(341..341 src has 177..341) 2024-09-17T18:27:17.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.060+0000 7f8ee4ff9640 3 client.64115.objecter handle_osd_map got epochs [341,341] > 0 2024-09-17T18:27:17.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.060+0000 7f8ee4ff9640 3 client.64115.objecter handle_osd_map decoding full epoch 341 2024-09-17T18:27:17.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.060+0000 7f8ee4ff9640 20 client.64115.objecter dump_active .. 0 homeless 2024-09-17T18:27:17.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.060+0000 7f8ee77fe640 1 --2- 172.21.15.121:0/4271264050 >> v2:172.21.15.173:6812/2423786037 conn(0x7f8ec80401a0 0x7f8ec8042660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:17.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.060+0000 7f8ee77fe640 10 monclient: get_auth_request con 0x7f8ec80401a0 auth_method 0 2024-09-17T18:27:17.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.062+0000 7f8ee4ff9640 10 client.64115.objecter ms_handle_connect 0x7f8ec80401a0 2024-09-17T18:27:17.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.062+0000 7f8ee77fe640 1 --2- 172.21.15.121:0/4271264050 >> v2:172.21.15.173:6812/2423786037 conn(0x7f8ec80401a0 0x7f8ec8042660 secure :-1 s=READY pgs=3998 cs=0 l=1 rev1=1 crypto rx=0x7f8ed8034000 tx=0x7f8ed802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:17.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.064+0000 7f8ee4ff9640 1 -- 172.21.15.121:0/4271264050 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8ee0022030 con 0x7f8ee813dc00 2024-09-17T18:27:17.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.064+0000 7f8ee4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:27:17.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.064+0000 7f8ee4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:27:17.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.314+0000 7f8eedcbe640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-17T18:27:17.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.314+0000 7f8eedcbe640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-17T18:27:17.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.314+0000 7f8eedcbe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:27:17.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.314+0000 7f8eedcbe640 1 -- 172.21.15.121:0/4271264050 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f8ee809d8c0 con 0x7f8ee813dc00 2024-09-17T18:27:17.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.315+0000 7f8ee4ff9640 1 -- 172.21.15.121:0/4271264050 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f8ee0010040 con 0x7f8ee813dc00 2024-09-17T18:27:17.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.315+0000 7f8ee4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-17T18:27:17.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.315+0000 7f8ee4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:27:17.318 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:17.316+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:17.318 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:17.316+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=12,kv=13,logm=59} crc {auth=1586802809,config=2586702516,health=1995790572,kv=1581938346,logm=2154234411}) 2024-09-17T18:27:17.318 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:17.316+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=14,config=2,health=12,kv=14,logm=58} crc {auth=1586802809,config=2586702516,health=1995790572,kv=2790417503,logm=2270419695}) 2024-09-17T18:27:17.318 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:17.316+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:17.318 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:17.316+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=12,kv=13,logm=59} crc {auth=1586802809,config=2586702516,health=1995790572,kv=1581938346,logm=2154234411}) 2024-09-17T18:27:17.318 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:17.316+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=13,config=1,health=11,kv=14,logm=61} crc {auth=3996957391,config=1228700967,health=4180235331,kv=2790417503,logm=779149171}) 2024-09-17T18:27:17.319 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:17.317+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:17.319 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:17.317+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3797393364}) 2024-09-17T18:27:17.319 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:17.317+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2950341780}) 2024-09-17T18:27:17.319 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:17.317+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:17.319 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:17.317+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3797393364}) 2024-09-17T18:27:17.319 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:17.317+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1975082667}) 2024-09-17T18:27:17.320 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:17.318+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:17.320 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:17.318+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4028049462}) 2024-09-17T18:27:17.320 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:17.318+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1369799162}) 2024-09-17T18:27:17.321 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:17.318+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:17.321 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:17.318+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4028049462}) 2024-09-17T18:27:17.321 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:17.318+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2892089913}) 2024-09-17T18:27:17.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.318+0000 7f8eedcbe640 10 librados: watch_flush enter 2024-09-17T18:27:17.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.319+0000 7f8eedcbe640 10 librados: watch_flush exit 2024-09-17T18:27:17.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.319+0000 7f8eedcbe640 20 client.64115.objecter shutdown clearing up homeless session... 2024-09-17T18:27:17.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.319+0000 7f8eedcbe640 10 client.64115.objecter successfully canceled tick 2024-09-17T18:27:17.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.319+0000 7f8eedcbe640 1 -- 172.21.15.121:0/4271264050 >> v2:172.21.15.173:6812/2423786037 conn(0x7f8ec80401a0 msgr2=0x7f8ec8042660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:17.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.319+0000 7f8eedcbe640 1 --2- 172.21.15.121:0/4271264050 >> v2:172.21.15.173:6812/2423786037 conn(0x7f8ec80401a0 0x7f8ec8042660 secure :-1 s=READY pgs=3998 cs=0 l=1 rev1=1 crypto rx=0x7f8ed8034000 tx=0x7f8ed802e000 comp rx=0 tx=0).stop 2024-09-17T18:27:17.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.319+0000 7f8eedcbe640 10 monclient: shutdown 2024-09-17T18:27:17.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.319+0000 7f8eedcbe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:27:17.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.319+0000 7f8eedcbe640 1 -- 172.21.15.121:0/4271264050 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8ee813dc00 msgr2=0x7f8ee813bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:17.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.319+0000 7f8eedcbe640 1 --2- 172.21.15.121:0/4271264050 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8ee813dc00 0x7f8ee813bb30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8ee000efc0 tx=0x7f8ee000c490 comp rx=0 tx=0).stop 2024-09-17T18:27:17.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.319+0000 7f8eedcbe640 1 -- 172.21.15.121:0/4271264050 shutdown_connections 2024-09-17T18:27:17.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.319+0000 7f8eedcbe640 1 --2- 172.21.15.121:0/4271264050 >> v2:172.21.15.173:6812/2423786037 conn(0x7f8ec80401a0 0x7f8ec8042660 unknown :-1 s=CLOSED pgs=3998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:17.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.319+0000 7f8eedcbe640 1 --2- 172.21.15.121:0/4271264050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ee80a5fe0 0x7f8ee81cddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:17.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.319+0000 7f8eedcbe640 1 --2- 172.21.15.121:0/4271264050 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8ee813dc00 0x7f8ee813bb30 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:17.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.319+0000 7f8eedcbe640 1 --2- 172.21.15.121:0/4271264050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ee80a5c00 0x7f8ee81cd720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:17.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.319+0000 7f8eedcbe640 1 -- 172.21.15.121:0/4271264050 >> 172.21.15.121:0/4271264050 conn(0x7f8ee80ab9b0 msgr2=0x7f8ee80a0e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:27:17.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.319+0000 7f8eedcbe640 1 -- 172.21.15.121:0/4271264050 shutdown_connections 2024-09-17T18:27:17.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.319+0000 7f8eedcbe640 1 -- 172.21.15.121:0/4271264050 wait complete. 2024-09-17T18:27:17.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:17.319+0000 7f8eedcbe640 1 librados: shutdown 2024-09-17T18:27:17.325 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:17.320+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:17.325 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:17.320+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1042295088}) 2024-09-17T18:27:17.325 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:17.320+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3537095900}) 2024-09-17T18:27:17.325 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:17.320+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:17.325 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:17.320+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1042295088}) 2024-09-17T18:27:17.325 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:17.320+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1897871653}) 2024-09-17T18:27:17.325 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:17.320+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:17.325 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:17.320+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=205568369}) 2024-09-17T18:27:17.325 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:17.320+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1699777664}) 2024-09-17T18:27:17.325 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:17.321+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:17.325 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:17.321+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=205568369}) 2024-09-17T18:27:17.326 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:17.321+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3585291718}) 2024-09-17T18:27:17.326 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:17.321+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:17.326 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:17.321+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3456403675}) 2024-09-17T18:27:17.326 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:17.321+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4060247655}) 2024-09-17T18:27:17.326 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:17.321+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:17.326 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:17.321+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3456403675}) 2024-09-17T18:27:17.326 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:17.321+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3598743487}) 2024-09-17T18:27:17.331 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-17T18:27:17.630 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:17.628+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:17.630 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:17.628+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1901790390}) 2024-09-17T18:27:17.631 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:17.628+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2855410754}) 2024-09-17T18:27:17.631 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:17.628+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:17.631 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:17.628+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1901790390}) 2024-09-17T18:27:17.631 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:17.628+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1885113481}) 2024-09-17T18:27:18.115 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.113+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.115 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.113+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3492331280}) 2024-09-17T18:27:18.115 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.113+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3074189474}) 2024-09-17T18:27:18.115 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.113+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.115 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.113+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3492331280}) 2024-09-17T18:27:18.115 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.113+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1865904450}) 2024-09-17T18:27:18.116 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.114+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.116 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.114+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2308263974}) 2024-09-17T18:27:18.116 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.114+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3225023518}) 2024-09-17T18:27:18.116 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.114+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.116 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.114+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2308263974}) 2024-09-17T18:27:18.116 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.114+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1932754355}) 2024-09-17T18:27:18.117 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.115+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.117 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.115+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2683092279}) 2024-09-17T18:27:18.117 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.115+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2205683059}) 2024-09-17T18:27:18.117 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.115+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.117 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.115+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2683092279}) 2024-09-17T18:27:18.117 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.115+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3731056277}) 2024-09-17T18:27:18.118 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.116+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.118 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.116+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=621978799}) 2024-09-17T18:27:18.118 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.116+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1636924385}) 2024-09-17T18:27:18.118 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.116+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.118 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.116+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=621978799}) 2024-09-17T18:27:18.118 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.116+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1944912574}) 2024-09-17T18:27:18.118 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.117+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.119 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.117+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3753360103}) 2024-09-17T18:27:18.119 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.117+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2493241346}) 2024-09-17T18:27:18.119 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.117+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.119 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.117+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3753360103}) 2024-09-17T18:27:18.119 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.117+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2516766925}) 2024-09-17T18:27:18.119 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.117+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.119 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.117+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1939307008}) 2024-09-17T18:27:18.120 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.117+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1787473188}) 2024-09-17T18:27:18.120 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.117+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.120 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.117+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1939307008}) 2024-09-17T18:27:18.120 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.117+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2081499725}) 2024-09-17T18:27:18.121 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.118+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.121 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.118+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4103103620}) 2024-09-17T18:27:18.121 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.118+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2407640438}) 2024-09-17T18:27:18.121 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.118+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.121 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.118+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4103103620}) 2024-09-17T18:27:18.121 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.118+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2019337657}) 2024-09-17T18:27:18.121 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.119+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.121 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.119+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=660837875}) 2024-09-17T18:27:18.121 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.119+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1214951204}) 2024-09-17T18:27:18.121 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.119+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.122 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.119+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=660837875}) 2024-09-17T18:27:18.122 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.119+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1684062924}) 2024-09-17T18:27:18.122 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.120+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.122 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.120+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1981161741}) 2024-09-17T18:27:18.122 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.120+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1615617037}) 2024-09-17T18:27:18.122 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.120+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.123 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.120+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1981161741}) 2024-09-17T18:27:18.123 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.120+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3597171554}) 2024-09-17T18:27:18.124 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.121+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.124 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.121+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=305008976}) 2024-09-17T18:27:18.124 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.121+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=831677679}) 2024-09-17T18:27:18.124 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.121+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.124 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.121+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=305008976}) 2024-09-17T18:27:18.124 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.121+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1116251098}) 2024-09-17T18:27:18.124 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.122+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.124 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.122+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3404299915}) 2024-09-17T18:27:18.124 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.122+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=70350203}) 2024-09-17T18:27:18.124 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.122+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.124 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.122+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3404299915}) 2024-09-17T18:27:18.124 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.122+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2738920204}) 2024-09-17T18:27:18.125 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.123+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.125 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.123+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2705364457}) 2024-09-17T18:27:18.126 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.123+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1939092363}) 2024-09-17T18:27:18.126 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.123+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.126 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.123+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2705364457}) 2024-09-17T18:27:18.126 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.123+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2498404952}) 2024-09-17T18:27:18.126 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.124+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.126 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.124+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=141657992}) 2024-09-17T18:27:18.126 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.124+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=304446215}) 2024-09-17T18:27:18.126 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.124+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.126 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.124+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=141657992}) 2024-09-17T18:27:18.126 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.124+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1788184005}) 2024-09-17T18:27:18.127 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.124+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.127 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.124+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1668744675}) 2024-09-17T18:27:18.127 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.124+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3802365213}) 2024-09-17T18:27:18.127 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.124+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.127 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.124+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1668744675}) 2024-09-17T18:27:18.128 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.124+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=419248253}) 2024-09-17T18:27:18.128 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.125+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.128 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.125+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1615506224}) 2024-09-17T18:27:18.128 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.125+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4239723058}) 2024-09-17T18:27:18.128 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.125+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.128 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.125+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1615506224}) 2024-09-17T18:27:18.128 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.125+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3498626499}) 2024-09-17T18:27:18.129 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.126+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.129 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.126+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2229606163}) 2024-09-17T18:27:18.129 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.126+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4129540661}) 2024-09-17T18:27:18.129 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.126+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.129 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.126+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2229606163}) 2024-09-17T18:27:18.129 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.126+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3422635098}) 2024-09-17T18:27:18.129 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.127+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.129 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.127+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2276273941}) 2024-09-17T18:27:18.129 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.127+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=649444164}) 2024-09-17T18:27:18.129 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.127+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.130 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.127+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2276273941}) 2024-09-17T18:27:18.130 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.127+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=175600813}) 2024-09-17T18:27:18.130 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.128+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.131 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.128+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3682116509}) 2024-09-17T18:27:18.131 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.128+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1659654405}) 2024-09-17T18:27:18.131 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.128+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.131 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.128+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3682116509}) 2024-09-17T18:27:18.131 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.128+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2439899024}) 2024-09-17T18:27:18.131 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.128+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.131 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.128+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1307336700}) 2024-09-17T18:27:18.131 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.128+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2119606015}) 2024-09-17T18:27:18.131 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.129+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.131 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.129+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1307336700}) 2024-09-17T18:27:18.131 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.129+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1073169971}) 2024-09-17T18:27:18.133 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.129+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.133 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.129+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2021142121}) 2024-09-17T18:27:18.133 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.129+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=79952820}) 2024-09-17T18:27:18.133 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.129+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.133 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.129+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2021142121}) 2024-09-17T18:27:18.133 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.129+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2215755760}) 2024-09-17T18:27:18.133 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.130+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.133 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.130+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2604229880}) 2024-09-17T18:27:18.133 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.130+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2999108422}) 2024-09-17T18:27:18.133 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.130+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.133 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.130+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2604229880}) 2024-09-17T18:27:18.133 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.130+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2613627173}) 2024-09-17T18:27:18.133 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.131+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.134 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.131+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3568846939}) 2024-09-17T18:27:18.134 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.131+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1434351030}) 2024-09-17T18:27:18.134 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.131+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.134 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.131+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3568846939}) 2024-09-17T18:27:18.134 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.131+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2629373800}) 2024-09-17T18:27:18.135 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.132+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.135 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.132+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=834421409}) 2024-09-17T18:27:18.135 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.132+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1779093528}) 2024-09-17T18:27:18.135 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.132+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.135 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.132+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=834421409}) 2024-09-17T18:27:18.135 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.132+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2280656663}) 2024-09-17T18:27:18.135 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.132+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.135 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.132+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3750136211}) 2024-09-17T18:27:18.136 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.132+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1018462104}) 2024-09-17T18:27:18.136 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.132+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.136 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.132+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3750136211}) 2024-09-17T18:27:18.136 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.132+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=641384030}) 2024-09-17T18:27:18.136 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.133+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.136 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.133+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=460544775}) 2024-09-17T18:27:18.136 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.133+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3727300290}) 2024-09-17T18:27:18.136 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.133+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.136 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.133+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=460544775}) 2024-09-17T18:27:18.136 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.133+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4084266126}) 2024-09-17T18:27:18.138 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.134+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.138 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.134+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2194904298}) 2024-09-17T18:27:18.138 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.134+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2220576359}) 2024-09-17T18:27:18.138 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.134+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.138 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.134+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2194904298}) 2024-09-17T18:27:18.138 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.134+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4279200994}) 2024-09-17T18:27:18.138 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.135+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.138 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.135+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=617009302}) 2024-09-17T18:27:18.138 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.135+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2473153018}) 2024-09-17T18:27:18.138 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.135+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.139 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.135+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=617009302}) 2024-09-17T18:27:18.139 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.135+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4252755338}) 2024-09-17T18:27:18.139 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.135+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.139 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.135+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2959013022}) 2024-09-17T18:27:18.139 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.135+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=267258422}) 2024-09-17T18:27:18.139 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.135+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.139 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.135+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2959013022}) 2024-09-17T18:27:18.139 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.135+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=772949024}) 2024-09-17T18:27:18.139 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.136+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.139 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.136+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1985512754}) 2024-09-17T18:27:18.139 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.136+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1057690269}) 2024-09-17T18:27:18.139 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.136+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.139 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.136+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1985512754}) 2024-09-17T18:27:18.139 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.136+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=489466033}) 2024-09-17T18:27:18.141 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.137+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.141 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.137+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2172127316}) 2024-09-17T18:27:18.141 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.137+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2978047605}) 2024-09-17T18:27:18.141 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.137+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.141 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.137+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2172127316}) 2024-09-17T18:27:18.141 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.137+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1007167335}) 2024-09-17T18:27:18.141 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.138+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.141 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.138+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=61,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=14} crc {logm=4268794761,mdsmap=761013554,mgr=3361498112,mgr_command_descs=1957073259,mgr_metadata=402068389,mgrstat=3693142462,mon_config_key=4093146309}) 2024-09-17T18:27:18.141 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.138+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=58,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=15,monmap=2,nvmeofgw=2} crc {logm=1564241813,mdsmap=761013554,mgr=1719742396,mgr_command_descs=1957073259,mgr_metadata=402068389,mgrstat=648583734,mon_config_key=1519263089,monmap=344458555,nvmeofgw=1596038039}) 2024-09-17T18:27:18.141 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.138+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.142 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.138+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=61,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=14} crc {logm=4268794761,mdsmap=761013554,mgr=3361498112,mgr_command_descs=1957073259,mgr_metadata=402068389,mgrstat=3693142462,mon_config_key=4093146309}) 2024-09-17T18:27:18.142 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.138+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=58,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=2} crc {logm=663889286,mdsmap=761013554,mgr=1675575104,mgr_command_descs=1957073259,mgr_metadata=402068389,mgrstat=3621774545,mon_config_key=1582663684,monmap=3140514009}) 2024-09-17T18:27:18.142 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.139+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.142 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.139+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=2,monmap=2,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=26,osdmap=61} crc {mon_config_key=3829049047,monmap=344458555,nvmeofgw=598337686,osd_metadata=794661279,osd_pg_creating=2225705617,osd_snap=1199849253,osdmap=2491347651}) 2024-09-17T18:27:18.142 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.139+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=2,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=56} crc {mon_config_key=3829049047,monmap=2277444362,nvmeofgw=598337686,osd_metadata=121115626,osd_pg_creating=2225705617,osd_snap=1511106774,osdmap=2369141546}) 2024-09-17T18:27:18.142 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.139+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.142 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.139+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=2,monmap=2,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=26,osdmap=61} crc {mon_config_key=3829049047,monmap=344458555,nvmeofgw=598337686,osd_metadata=794661279,osd_pg_creating=2225705617,osd_snap=1199849253,osdmap=2491347651}) 2024-09-17T18:27:18.142 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.139+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=2,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=28,osdmap=57} crc {mon_config_key=3829049047,monmap=2277444362,nvmeofgw=598337686,osd_metadata=121115626,osd_pg_creating=2225705617,osd_snap=3154474934,osdmap=1444626746}) 2024-09-17T18:27:18.143 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.140+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.143 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.140+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=779928238}) 2024-09-17T18:27:18.143 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.140+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=402147534}) 2024-09-17T18:27:18.143 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.140+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.143 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.140+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=779928238}) 2024-09-17T18:27:18.143 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.140+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1958859948}) 2024-09-17T18:27:18.149 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.147+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.149 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.147+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=93} crc {osdmap=4126086721}) 2024-09-17T18:27:18.149 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.147+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=91} crc {osdmap=1355447479}) 2024-09-17T18:27:18.149 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.147+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:27:18.149 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.147+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=93} crc {osdmap=4126086721}) 2024-09-17T18:27:18.149 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:27:18.147+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=94} crc {osdmap=2853991572}) 2024-09-17T18:27:18.332 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T18:27:18.332 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:27:18.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.431+0000 7fee6fb10640 10 monclient: get_monmap_and_config 2024-09-17T18:27:18.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.431+0000 7fee6fb10640 10 monclient: build_initial_monmap 2024-09-17T18:27:18.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.431+0000 7fee6fb10640 10 monclient: monmap: 2024-09-17T18:27:18.433 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:27:18.433 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:18.433 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:27:18.431882+0000 2024-09-17T18:27:18.433 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:27:18.431882+0000 2024-09-17T18:27:18.434 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:27:18.434 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:18.434 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:27:18.434 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:27:18.434 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:27:18.434 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:27:18.434 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:27:18.434 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:18.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.431+0000 7fee6fb10640 1 Processor -- start 2024-09-17T18:27:18.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6fb10640 1 -- start start 2024-09-17T18:27:18.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6fb10640 10 monclient: init 2024-09-17T18:27:18.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6fb10640 10 monclient: _reopen_session rank -1 2024-09-17T18:27:18.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6fb10640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-17T18:27:18.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6fb10640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fee680a5c00 0x7fee6809d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:18.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6fb10640 10 monclient(hunting): picked mon.noname-a con 0x7fee680a5c00 addr v2:172.21.15.121:3300/0 2024-09-17T18:27:18.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6fb10640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fee680a5fe0 addr v1:172.21.15.173:6789/0 2024-09-17T18:27:18.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6fb10640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee6813dc00 0x7fee6813fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:18.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6fb10640 10 monclient(hunting): picked mon.noname-b con 0x7fee6813dc00 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:27:18.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6fb10640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee680a8a80 con 0x7fee680a5fe0 2024-09-17T18:27:18.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6fb10640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:18.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6fb10640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fee680a7110 con 0x7fee680a5c00 2024-09-17T18:27:18.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6fb10640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:18.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6fb10640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fee6809de00 con 0x7fee6813dc00 2024-09-17T18:27:18.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6fb10640 10 monclient(hunting): _renew_subs 2024-09-17T18:27:18.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6fb10640 10 monclient(hunting): authenticate will time out at 1242.148581s 2024-09-17T18:27:18.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6e0ad640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee6813dc00 0x7fee6813fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:18.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6d8ac640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fee680a5c00 0x7fee6809d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:18.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6e0ad640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee6813dc00 0x7fee6813fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:46650/0 (socket says 172.21.15.121:46650) 2024-09-17T18:27:18.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6e0ad640 1 -- 172.21.15.121:0/2822132053 learned_addr learned my addr 172.21.15.121:0/2822132053 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:27:18.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6d8ac640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fee680a5c00 0x7fee6809d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56592/0 (socket says 172.21.15.121:56592) 2024-09-17T18:27:18.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6e0ad640 10 monclient(hunting): get_auth_request con 0x7fee6813dc00 auth_method 0 2024-09-17T18:27:18.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6e0ad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:18.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6e0ad640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:18.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6e0ad640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:18.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6d8ac640 10 monclient(hunting): get_auth_request con 0x7fee680a5c00 auth_method 0 2024-09-17T18:27:18.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6d8ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:18.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6d8ac640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:18.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6d8ac640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:18.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6e0ad640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:18.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6e0ad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:18.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6e0ad640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:18.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6d8ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:18.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6d8ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:18.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6d8ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:18.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6e0ad640 10 monclient(hunting): handle_auth_done global_id 64118 payload 274 2024-09-17T18:27:18.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6e0ad640 1 -- 172.21.15.121:0/2822132053 >> v2:172.21.15.121:3300/0 conn(0x7fee680a5c00 msgr2=0x7fee6809d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:27:18.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6e0ad640 1 --2- 172.21.15.121:0/2822132053 >> v2:172.21.15.121:3300/0 conn(0x7fee680a5c00 0x7fee6809d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:18.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6e0ad640 1 -- 172.21.15.121:0/2822132053 >> v1:172.21.15.173:6789/0 conn(0x7fee680a5fe0 legacy=0x7fee680a63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:27:18.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6e0ad640 10 monclient: _finish_hunting 0 2024-09-17T18:27:18.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6e0ad640 1 monclient: found mon.noname-b 2024-09-17T18:27:18.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6e0ad640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:27:18.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6e0ad640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:27:18.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.432+0000 7fee6e0ad640 1 -- 172.21.15.121:0/2822132053 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee680a74d0 con 0x7fee6813dc00 2024-09-17T18:27:18.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.435+0000 7fee6e0ad640 1 -- 172.21.15.121:0/2822132053 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee6813dc00 msgr2=0x7fee6813fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:18.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.435+0000 7fee6e0ad640 1 --2- 172.21.15.121:0/2822132053 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee6813dc00 0x7fee6813fff0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fee58009870 tx=0x7fee5802a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=522fa45da2e2b66d server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:18.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.435+0000 7fee6c8aa640 1 -- 172.21.15.121:0/2822132053 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee58037070 con 0x7fee6813dc00 2024-09-17T18:27:18.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.435+0000 7fee6c8aa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:18.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.435+0000 7fee6c8aa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:27:18.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.435+0000 7fee6c8aa640 10 monclient: dump: 2024-09-17T18:27:18.445 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:18.445 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:18.445 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:18.445 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:18.445 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:18.445 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:18.446 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:18.446 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:18.446 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:18.446 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:18.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.435+0000 7fee6c8aa640 10 monclient: _finish_auth 0 2024-09-17T18:27:18.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.435+0000 7fee6c8aa640 10 monclient: _check_auth_tickets 2024-09-17T18:27:18.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.435+0000 7fee6c8aa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:27:18.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.435+0000 7fee6c8aa640 1 -- 172.21.15.121:0/2822132053 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee58032440 con 0x7fee6813dc00 2024-09-17T18:27:18.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.435+0000 7fee6fb10640 5 monclient: authenticate success, global_id 64118 2024-09-17T18:27:18.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.435+0000 7fee6fb10640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:27:18.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.435+0000 7fee6c8aa640 10 monclient: handle_config config(0 keys) 2024-09-17T18:27:18.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.435+0000 7fee6c8aa640 1 -- 172.21.15.121:0/2822132053 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee58032810 con 0x7fee6813dc00 2024-09-17T18:27:18.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.435+0000 7fee6fb10640 10 monclient: get_monmap_and_config success 2024-09-17T18:27:18.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.435+0000 7fee6fb10640 4 set_mon_vals no callback set 2024-09-17T18:27:18.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.435+0000 7fee6c8aa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:18.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.435+0000 7fee6c8aa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:27:18.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.435+0000 7fee6c8aa640 10 monclient: dump: 2024-09-17T18:27:18.447 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:18.447 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:18.447 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:18.447 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:18.447 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:18.447 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:18.447 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:18.447 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:18.447 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:18.447 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:18.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.436+0000 7fee6fb10640 10 monclient: shutdown 2024-09-17T18:27:18.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.436+0000 7fee6fb10640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:27:18.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.436+0000 7fee6fb10640 1 -- 172.21.15.121:0/2822132053 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee6813dc00 msgr2=0x7fee6813fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:18.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.436+0000 7fee6fb10640 1 --2- 172.21.15.121:0/2822132053 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee6813dc00 0x7fee6813fff0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fee58009870 tx=0x7fee5802a0a0 comp rx=0 tx=0).stop 2024-09-17T18:27:18.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.436+0000 7fee6fb10640 1 -- 172.21.15.121:0/2822132053 shutdown_connections 2024-09-17T18:27:18.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.436+0000 7fee6fb10640 1 --2- 172.21.15.121:0/2822132053 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee6813dc00 0x7fee6813fff0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:18.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.436+0000 7fee6fb10640 1 --2- 172.21.15.121:0/2822132053 >> v2:172.21.15.121:3300/0 conn(0x7fee680a5c00 0x7fee6809d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:18.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.436+0000 7fee6fb10640 1 -- 172.21.15.121:0/2822132053 >> 172.21.15.121:0/2822132053 conn(0x7fee680ab9b0 msgr2=0x7fee680a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:27:18.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.436+0000 7fee6fb10640 1 -- 172.21.15.121:0/2822132053 shutdown_connections 2024-09-17T18:27:18.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.436+0000 7fee6fb10640 1 -- 172.21.15.121:0/2822132053 wait complete. 2024-09-17T18:27:18.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.437+0000 7fee6fb10640 10 monclient: build_initial_monmap 2024-09-17T18:27:18.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.437+0000 7fee6fb10640 10 monclient: monmap: 2024-09-17T18:27:18.448 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:27:18.448 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:27:18.449 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:27:18.449 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:27:18.449 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:27:18.449 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:18.449 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:27:18.449 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:27:18.449 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:27:18.449 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:18.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.437+0000 7fee6fb10640 1 librados: starting msgr at 2024-09-17T18:27:18.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.437+0000 7fee6fb10640 1 librados: starting objecter 2024-09-17T18:27:18.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.437+0000 7fee6fb10640 1 Processor -- start 2024-09-17T18:27:18.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.437+0000 7fee6fb10640 1 -- start start 2024-09-17T18:27:18.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.437+0000 7fee6fb10640 1 librados: setting wanted keys 2024-09-17T18:27:18.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.437+0000 7fee6fb10640 1 librados: calling monclient init 2024-09-17T18:27:18.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.437+0000 7fee6fb10640 10 monclient: init 2024-09-17T18:27:18.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.437+0000 7fee6fb10640 10 monclient: _reopen_session rank -1 2024-09-17T18:27:18.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.437+0000 7fee6fb10640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T18:27:18.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.437+0000 7fee6fb10640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee680a5c00 0x7fee68121dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:18.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.437+0000 7fee6fb10640 10 monclient(hunting): picked mon.noname-b con 0x7fee680a5c00 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:27:18.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.437+0000 7fee6fb10640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee680a5fe0 0x7fee6813c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:18.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.437+0000 7fee6fb10640 10 monclient(hunting): picked mon.noname-c con 0x7fee680a5fe0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:27:18.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.437+0000 7fee6fb10640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee6813dc00 0x7fee68138c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:18.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.437+0000 7fee6fb10640 10 monclient(hunting): picked mon.noname-a con 0x7fee6813dc00 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:27:18.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.437+0000 7fee6fb10640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:18.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.437+0000 7fee6fb10640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fee680703d0 con 0x7fee6813dc00 2024-09-17T18:27:18.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.437+0000 7fee6fb10640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:18.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.437+0000 7fee6fb10640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fee68070550 con 0x7fee680a5c00 2024-09-17T18:27:18.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.437+0000 7fee6fb10640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:18.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.437+0000 7fee6fb10640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fee68140530 con 0x7fee680a5fe0 2024-09-17T18:27:18.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.437+0000 7fee6fb10640 10 monclient(hunting): _renew_subs 2024-09-17T18:27:18.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.437+0000 7fee6fb10640 10 monclient(hunting): authenticate will time out at 1242.153854s 2024-09-17T18:27:18.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.437+0000 7fee6d0ab640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee680a5fe0 0x7fee6813c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:18.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.437+0000 7fee6e0ad640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee6813dc00 0x7fee68138c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:18.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.437+0000 7fee6d0ab640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee680a5fe0 0x7fee6813c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:46660/0 (socket says 172.21.15.121:46660) 2024-09-17T18:27:18.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.437+0000 7fee6e0ad640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee6813dc00 0x7fee68138c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56600/0 (socket says 172.21.15.121:56600) 2024-09-17T18:27:18.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.437+0000 7fee6d0ab640 1 -- 172.21.15.121:0/1921478897 learned_addr learned my addr 172.21.15.121:0/1921478897 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:27:18.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.437+0000 7fee6d0ab640 10 monclient(hunting): get_auth_request con 0x7fee680a5fe0 auth_method 0 2024-09-17T18:27:18.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.437+0000 7fee6d0ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:18.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.437+0000 7fee6d0ab640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:18.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.437+0000 7fee6d0ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:18.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.437+0000 7fee6e0ad640 10 monclient(hunting): get_auth_request con 0x7fee6813dc00 auth_method 0 2024-09-17T18:27:18.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.437+0000 7fee6e0ad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:18.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.437+0000 7fee6e0ad640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:18.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.437+0000 7fee6e0ad640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:18.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.438+0000 7fee6d0ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:18.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.438+0000 7fee6d0ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:18.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.438+0000 7fee6d0ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:18.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.438+0000 7fee6e0ad640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:18.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.438+0000 7fee6e0ad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:18.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.438+0000 7fee6e0ad640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:18.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.438+0000 7fee6d8ac640 1 --2- 172.21.15.121:0/1921478897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee680a5c00 0x7fee68121dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:18.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.438+0000 7fee6d8ac640 10 monclient(hunting): get_auth_request con 0x7fee680a5c00 auth_method 0 2024-09-17T18:27:18.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.438+0000 7fee6d8ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:18.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.438+0000 7fee6d8ac640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:18.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.438+0000 7fee6d8ac640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:18.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.438+0000 7fee6d0ab640 10 monclient(hunting): handle_auth_done global_id 64121 payload 824 2024-09-17T18:27:18.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.438+0000 7fee6d0ab640 1 -- 172.21.15.121:0/1921478897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee680a5c00 msgr2=0x7fee68121dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:18.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.438+0000 7fee6d0ab640 1 --2- 172.21.15.121:0/1921478897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee680a5c00 0x7fee68121dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:18.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.438+0000 7fee6d0ab640 1 -- 172.21.15.121:0/1921478897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee6813dc00 msgr2=0x7fee68138c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:18.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.438+0000 7fee6d0ab640 1 --2- 172.21.15.121:0/1921478897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee6813dc00 0x7fee68138c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:18.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.438+0000 7fee6d0ab640 10 monclient: _finish_hunting 0 2024-09-17T18:27:18.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.438+0000 7fee6d0ab640 1 monclient: found mon.noname-c 2024-09-17T18:27:18.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.438+0000 7fee6d0ab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:27:18.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.438+0000 7fee6d0ab640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:27:18.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.438+0000 7fee6d0ab640 1 -- 172.21.15.121:0/1921478897 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee6813d620 con 0x7fee680a5fe0 2024-09-17T18:27:18.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.438+0000 7fee6e0ad640 1 --2- 172.21.15.121:0/1921478897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee6813dc00 0x7fee68138c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:27:18.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.440+0000 7fee6d0ab640 1 -- 172.21.15.121:0/1921478897 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee680a5fe0 msgr2=0x7fee6813c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:18.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.440+0000 7fee6d0ab640 1 --2- 172.21.15.121:0/1921478897 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee680a5fe0 0x7fee6813c020 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fee5c00c7f0 tx=0x7fee5c00ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:18.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.440+0000 7fee667fc640 10 client.?.objecter ms_handle_connect 0x7fee680a5fe0 2024-09-17T18:27:18.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.440+0000 7fee667fc640 10 client.?.objecter resend_mon_ops 2024-09-17T18:27:18.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.440+0000 7fee667fc640 1 -- 172.21.15.121:0/1921478897 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee5c016020 con 0x7fee680a5fe0 2024-09-17T18:27:18.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.440+0000 7fee667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:18.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.440+0000 7fee667fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:27:18.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.440+0000 7fee667fc640 10 monclient: dump: 2024-09-17T18:27:18.454 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:18.454 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:18.454 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:18.454 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:18.454 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:18.454 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:18.454 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:18.454 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:18.454 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:18.454 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:18.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.440+0000 7fee667fc640 10 monclient: _finish_auth 0 2024-09-17T18:27:18.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.440+0000 7fee667fc640 10 monclient: _check_auth_tickets 2024-09-17T18:27:18.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.440+0000 7fee667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:27:18.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.440+0000 7fee667fc640 1 -- 172.21.15.121:0/1921478897 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee5c00ac00 con 0x7fee680a5fe0 2024-09-17T18:27:18.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.440+0000 7fee6fb10640 5 monclient: authenticate success, global_id 64121 2024-09-17T18:27:18.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.440+0000 7fee6fb10640 10 monclient: _renew_subs 2024-09-17T18:27:18.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.440+0000 7fee6fb10640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:27:18.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.440+0000 7fee6fb10640 1 -- 172.21.15.121:0/1921478897 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee680a23f0 con 0x7fee680a5fe0 2024-09-17T18:27:18.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.440+0000 7fee667fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:27:18.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.440+0000 7fee667fc640 1 -- 172.21.15.121:0/1921478897 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee5c012420 con 0x7fee680a5fe0 2024-09-17T18:27:18.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.440+0000 7fee667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:18.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.440+0000 7fee667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:27:18.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.440+0000 7fee667fc640 10 monclient: dump: 2024-09-17T18:27:18.455 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:18.455 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:18.455 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:18.455 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:18.455 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:18.455 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:18.456 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:18.456 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:18.456 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:18.456 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:18.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.440+0000 7fee6fb10640 10 client.64121.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:27:18.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.440+0000 7fee6fb10640 10 monclient: _renew_subs 2024-09-17T18:27:18.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.440+0000 7fee6fb10640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:27:18.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.440+0000 7fee6fb10640 1 -- 172.21.15.121:0/1921478897 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fee68139180 con 0x7fee680a5fe0 2024-09-17T18:27:18.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.440+0000 7fee6fb10640 1 librados: init done 2024-09-17T18:27:18.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.440+0000 7fee67fff640 4 set_mon_vals no callback set 2024-09-17T18:27:18.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.441+0000 7fee667fc640 1 -- 172.21.15.121:0/1921478897 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fee5c019020 con 0x7fee680a5fe0 2024-09-17T18:27:18.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.441+0000 7fee6fb10640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:27:18.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.441+0000 7fee6fb10640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:27:18.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.441+0000 7fee6fb10640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:27:18.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.441+0000 7fee6fb10640 1 -- 172.21.15.121:0/1921478897 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee680a1580 con 0x7fee680a5fe0 2024-09-17T18:27:18.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.441+0000 7fee667fc640 1 --2- 172.21.15.121:0/1921478897 >> v2:172.21.15.173:6812/2423786037 conn(0x7fee54040170 0x7fee54042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:18.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.441+0000 7fee667fc640 1 -- 172.21.15.121:0/1921478897 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(343..343 src has 177..343) ==== 5345+0+0 (secure 0 0 0) 0x7fee5c053440 con 0x7fee680a5fe0 2024-09-17T18:27:18.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.441+0000 7fee667fc640 10 client.64121.objecter ms_dispatch 0x7fee68000b90 osd_map(343..343 src has 177..343) 2024-09-17T18:27:18.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.441+0000 7fee667fc640 3 client.64121.objecter handle_osd_map got epochs [343,343] > 0 2024-09-17T18:27:18.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.441+0000 7fee667fc640 3 client.64121.objecter handle_osd_map decoding full epoch 343 2024-09-17T18:27:18.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.441+0000 7fee667fc640 20 client.64121.objecter dump_active .. 0 homeless 2024-09-17T18:27:18.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.442+0000 7fee6d8ac640 1 --2- 172.21.15.121:0/1921478897 >> v2:172.21.15.173:6812/2423786037 conn(0x7fee54040170 0x7fee54042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:18.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.442+0000 7fee6d8ac640 10 monclient: get_auth_request con 0x7fee54040170 auth_method 0 2024-09-17T18:27:18.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.444+0000 7fee667fc640 10 client.64121.objecter ms_handle_connect 0x7fee54040170 2024-09-17T18:27:18.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.444+0000 7fee6d8ac640 1 --2- 172.21.15.121:0/1921478897 >> v2:172.21.15.173:6812/2423786037 conn(0x7fee54040170 0x7fee54042630 secure :-1 s=READY pgs=3999 cs=0 l=1 rev1=1 crypto rx=0x7fee60009ad0 tx=0x7fee60008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:18.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.446+0000 7fee667fc640 1 -- 172.21.15.121:0/1921478897 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fee680a1580 con 0x7fee680a5fe0 2024-09-17T18:27:18.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.446+0000 7fee667fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:27:18.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.446+0000 7fee667fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:27:18.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.690+0000 7fee6fb10640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:27:18.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.690+0000 7fee6fb10640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:27:18.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.690+0000 7fee6fb10640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:27:18.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.690+0000 7fee6fb10640 1 -- 172.21.15.121:0/1921478897 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fee6809d8c0 con 0x7fee680a5fe0 2024-09-17T18:27:18.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.691+0000 7fee667fc640 1 -- 172.21.15.121:0/1921478897 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fee5c020070 con 0x7fee680a5fe0 2024-09-17T18:27:18.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.691+0000 7fee667fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:27:18.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.691+0000 7fee667fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:27:18.693 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:27:18.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.693+0000 7fee6fb10640 10 librados: watch_flush enter 2024-09-17T18:27:18.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.693+0000 7fee6fb10640 10 librados: watch_flush exit 2024-09-17T18:27:18.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.693+0000 7fee6fb10640 20 client.64121.objecter shutdown clearing up homeless session... 2024-09-17T18:27:18.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.693+0000 7fee6fb10640 10 client.64121.objecter successfully canceled tick 2024-09-17T18:27:18.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.693+0000 7fee6fb10640 1 -- 172.21.15.121:0/1921478897 >> v2:172.21.15.173:6812/2423786037 conn(0x7fee54040170 msgr2=0x7fee54042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:18.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.693+0000 7fee6fb10640 1 --2- 172.21.15.121:0/1921478897 >> v2:172.21.15.173:6812/2423786037 conn(0x7fee54040170 0x7fee54042630 secure :-1 s=READY pgs=3999 cs=0 l=1 rev1=1 crypto rx=0x7fee60009ad0 tx=0x7fee60008040 comp rx=0 tx=0).stop 2024-09-17T18:27:18.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.693+0000 7fee6fb10640 10 monclient: shutdown 2024-09-17T18:27:18.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.693+0000 7fee6fb10640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:27:18.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.693+0000 7fee6fb10640 1 -- 172.21.15.121:0/1921478897 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee680a5fe0 msgr2=0x7fee6813c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:18.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.693+0000 7fee6fb10640 1 --2- 172.21.15.121:0/1921478897 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee680a5fe0 0x7fee6813c020 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fee5c00c7f0 tx=0x7fee5c00ccb0 comp rx=0 tx=0).stop 2024-09-17T18:27:18.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.693+0000 7fee6fb10640 1 -- 172.21.15.121:0/1921478897 shutdown_connections 2024-09-17T18:27:18.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.693+0000 7fee6fb10640 1 --2- 172.21.15.121:0/1921478897 >> v2:172.21.15.173:6812/2423786037 conn(0x7fee54040170 0x7fee54042630 unknown :-1 s=CLOSED pgs=3999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:18.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.693+0000 7fee6fb10640 1 --2- 172.21.15.121:0/1921478897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee6813dc00 0x7fee68138c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:18.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.693+0000 7fee6fb10640 1 --2- 172.21.15.121:0/1921478897 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee680a5fe0 0x7fee6813c020 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:18.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.693+0000 7fee6fb10640 1 --2- 172.21.15.121:0/1921478897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee680a5c00 0x7fee68121dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:18.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.693+0000 7fee6fb10640 1 -- 172.21.15.121:0/1921478897 >> 172.21.15.121:0/1921478897 conn(0x7fee680ab9b0 msgr2=0x7fee680a0e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:27:18.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.693+0000 7fee6fb10640 1 -- 172.21.15.121:0/1921478897 shutdown_connections 2024-09-17T18:27:18.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.693+0000 7fee6fb10640 1 -- 172.21.15.121:0/1921478897 wait complete. 2024-09-17T18:27:18.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.693+0000 7fee6fb10640 1 librados: shutdown 2024-09-17T18:27:18.705 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T18:27:18.705 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-17T18:27:18.705 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T18:27:18.705 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T18:27:18.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.802+0000 7f699f5ec640 10 monclient: get_monmap_and_config 2024-09-17T18:27:18.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.802+0000 7f699f5ec640 10 monclient: build_initial_monmap 2024-09-17T18:27:18.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.802+0000 7f699f5ec640 10 monclient: monmap: 2024-09-17T18:27:18.804 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:27:18.804 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:18.804 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:27:18.803035+0000 2024-09-17T18:27:18.804 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:27:18.803035+0000 2024-09-17T18:27:18.804 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:27:18.804 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:18.804 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:27:18.805 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:27:18.805 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:27:18.805 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:27:18.805 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:27:18.805 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:18.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.802+0000 7f699f5ec640 1 Processor -- start 2024-09-17T18:27:18.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.803+0000 7f699f5ec640 1 -- start start 2024-09-17T18:27:18.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.803+0000 7f699f5ec640 10 monclient: init 2024-09-17T18:27:18.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.803+0000 7f699f5ec640 10 monclient: _reopen_session rank -1 2024-09-17T18:27:18.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.803+0000 7f699f5ec640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-17T18:27:18.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.803+0000 7f699f5ec640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f699809e8d0 addr v1:172.21.15.173:6789/0 2024-09-17T18:27:18.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.803+0000 7f699f5ec640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6998135c30 addr v1:172.21.15.121:6789/0 2024-09-17T18:27:18.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.803+0000 7f699f5ec640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6998136040 0x7f699813de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:18.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.803+0000 7f699f5ec640 10 monclient(hunting): picked mon.noname-b con 0x7f6998136040 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:27:18.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.803+0000 7f699f5ec640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69980a0780 con 0x7f6998135c30 2024-09-17T18:27:18.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.803+0000 7f699f5ec640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69980a0a60 con 0x7f699809e8d0 2024-09-17T18:27:18.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.803+0000 7f699f5ec640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:18.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.803+0000 7f699f5ec640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6998069d50 con 0x7f6998136040 2024-09-17T18:27:18.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.803+0000 7f699f5ec640 10 monclient(hunting): _renew_subs 2024-09-17T18:27:18.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.803+0000 7f699f5ec640 10 monclient(hunting): authenticate will time out at 1242.519723s 2024-09-17T18:27:18.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.803+0000 7f699db89640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6998136040 0x7f699813de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:18.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.803+0000 7f699cb87640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f6998135c30 0x7f699813a760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:35584/0 (socket says 172.21.15.121:35584) 2024-09-17T18:27:18.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.803+0000 7f699cb87640 1 -- 172.21.15.121:0/1368935129 learned_addr learned my addr 172.21.15.121:0/1368935129 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:27:18.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.803+0000 7f699db89640 10 monclient(hunting): get_auth_request con 0x7f6998136040 auth_method 0 2024-09-17T18:27:18.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.803+0000 7f699db89640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:18.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.803+0000 7f699db89640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:18.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.803+0000 7f699db89640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:18.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.803+0000 7f699db89640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:18.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.803+0000 7f699db89640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:18.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.803+0000 7f699db89640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:18.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.804+0000 7f699db89640 10 monclient(hunting): handle_auth_done global_id 64124 payload 274 2024-09-17T18:27:18.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.804+0000 7f699db89640 1 -- 172.21.15.121:0/1368935129 >> v1:172.21.15.173:6789/0 conn(0x7f699809e8d0 legacy=0x7f6998135520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:27:18.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.804+0000 7f699db89640 1 -- 172.21.15.121:0/1368935129 >> v1:172.21.15.121:6789/0 conn(0x7f6998135c30 legacy=0x7f699813a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:27:18.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.804+0000 7f699db89640 10 monclient: _finish_hunting 0 2024-09-17T18:27:18.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.804+0000 7f699db89640 1 monclient: found mon.noname-b 2024-09-17T18:27:18.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.804+0000 7f699db89640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:27:18.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.804+0000 7f699db89640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:27:18.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.804+0000 7f699db89640 1 -- 172.21.15.121:0/1368935129 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69980a0d40 con 0x7f6998136040 2024-09-17T18:27:18.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.806+0000 7f699db89640 1 -- 172.21.15.121:0/1368935129 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6998136040 msgr2=0x7f699813de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:18.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.806+0000 7f699db89640 1 --2- 172.21.15.121:0/1368935129 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6998136040 0x7f699813de90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6988009df0 tx=0x7f698802a4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e36491163153edbe server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:18.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.806+0000 7f6997fff640 1 -- 172.21.15.121:0/1368935129 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6988007a30 con 0x7f6998136040 2024-09-17T18:27:18.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.806+0000 7f6997fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:18.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.807+0000 7f6997fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:27:18.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.807+0000 7f6997fff640 10 monclient: dump: 2024-09-17T18:27:18.813 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:18.813 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:18.813 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:18.813 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:18.813 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:18.813 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:18.813 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:18.813 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:18.813 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:18.813 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:18.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.807+0000 7f6997fff640 10 monclient: _finish_auth 0 2024-09-17T18:27:18.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.807+0000 7f6997fff640 10 monclient: _check_auth_tickets 2024-09-17T18:27:18.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.807+0000 7f6997fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:27:18.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.807+0000 7f6997fff640 1 -- 172.21.15.121:0/1368935129 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6988007bd0 con 0x7f6998136040 2024-09-17T18:27:18.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.807+0000 7f699f5ec640 5 monclient: authenticate success, global_id 64124 2024-09-17T18:27:18.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.807+0000 7f699f5ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:27:18.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.807+0000 7f6997fff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:27:18.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.807+0000 7f699f5ec640 10 monclient: get_monmap_and_config success 2024-09-17T18:27:18.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.807+0000 7f6997fff640 1 -- 172.21.15.121:0/1368935129 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6988031400 con 0x7f6998136040 2024-09-17T18:27:18.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.807+0000 7f699f5ec640 4 set_mon_vals no callback set 2024-09-17T18:27:18.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.807+0000 7f6997fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:18.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.807+0000 7f6997fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:27:18.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.807+0000 7f6997fff640 10 monclient: dump: 2024-09-17T18:27:18.814 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:18.814 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:18.815 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:18.815 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:18.815 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:18.815 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:18.815 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:18.815 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:18.815 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:18.815 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:18.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.807+0000 7f699f5ec640 10 monclient: shutdown 2024-09-17T18:27:18.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.807+0000 7f699f5ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:27:18.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.807+0000 7f699f5ec640 1 -- 172.21.15.121:0/1368935129 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6998136040 msgr2=0x7f699813de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:18.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.807+0000 7f699f5ec640 1 --2- 172.21.15.121:0/1368935129 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6998136040 0x7f699813de90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6988009df0 tx=0x7f698802a4b0 comp rx=0 tx=0).stop 2024-09-17T18:27:18.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.807+0000 7f699f5ec640 1 -- 172.21.15.121:0/1368935129 shutdown_connections 2024-09-17T18:27:18.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.807+0000 7f699f5ec640 1 --2- 172.21.15.121:0/1368935129 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6998136040 0x7f699813de90 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:18.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.807+0000 7f699f5ec640 1 -- 172.21.15.121:0/1368935129 >> 172.21.15.121:0/1368935129 conn(0x7f69980a75b0 msgr2=0x7f69980a79c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:27:18.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.807+0000 7f699f5ec640 1 -- 172.21.15.121:0/1368935129 shutdown_connections 2024-09-17T18:27:18.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.807+0000 7f699f5ec640 1 -- 172.21.15.121:0/1368935129 wait complete. 2024-09-17T18:27:18.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.808+0000 7f699f5ec640 10 monclient: build_initial_monmap 2024-09-17T18:27:18.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.808+0000 7f699f5ec640 10 monclient: monmap: 2024-09-17T18:27:18.816 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:27:18.816 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:27:18.816 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:27:18.816 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:27:18.816 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:27:18.816 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:18.816 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:27:18.816 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:27:18.816 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:27:18.817 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:18.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.808+0000 7f699f5ec640 1 librados: starting msgr at 2024-09-17T18:27:18.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.808+0000 7f699f5ec640 1 librados: starting objecter 2024-09-17T18:27:18.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.808+0000 7f699f5ec640 1 Processor -- start 2024-09-17T18:27:18.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.808+0000 7f699f5ec640 1 -- start start 2024-09-17T18:27:18.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.808+0000 7f699f5ec640 1 librados: setting wanted keys 2024-09-17T18:27:18.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.808+0000 7f699f5ec640 1 librados: calling monclient init 2024-09-17T18:27:18.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.808+0000 7f699f5ec640 10 monclient: init 2024-09-17T18:27:18.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.808+0000 7f699f5ec640 10 monclient: _reopen_session rank -1 2024-09-17T18:27:18.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699f5ec640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T18:27:18.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699f5ec640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f699809e8d0 0x7f699813a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:18.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699f5ec640 10 monclient(hunting): picked mon.noname-a con 0x7f699809e8d0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:27:18.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699f5ec640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6998135c30 0x7f699813a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:18.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699f5ec640 10 monclient(hunting): picked mon.noname-c con 0x7f6998135c30 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:27:18.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699f5ec640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6998136040 0x7f699813ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:18.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699f5ec640 10 monclient(hunting): picked mon.noname-b con 0x7f6998136040 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:27:18.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699f5ec640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:18.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699f5ec640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f699809fe40 con 0x7f699809e8d0 2024-09-17T18:27:18.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699f5ec640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:18.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699f5ec640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f699809fcc0 con 0x7f6998136040 2024-09-17T18:27:18.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699f5ec640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:18.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699f5ec640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f699813e3d0 con 0x7f6998135c30 2024-09-17T18:27:18.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699f5ec640 10 monclient(hunting): _renew_subs 2024-09-17T18:27:18.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699f5ec640 10 monclient(hunting): authenticate will time out at 1242.525086s 2024-09-17T18:27:18.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699d388640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f699809e8d0 0x7f699813a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:18.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699d388640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f699809e8d0 0x7f699813a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56602/0 (socket says 172.21.15.121:56602) 2024-09-17T18:27:18.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699cb87640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6998135c30 0x7f699813a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:18.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699d388640 1 -- 172.21.15.121:0/3031595095 learned_addr learned my addr 172.21.15.121:0/3031595095 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:27:18.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699d388640 10 monclient(hunting): get_auth_request con 0x7f699809e8d0 auth_method 0 2024-09-17T18:27:18.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699d388640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:18.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699d388640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:18.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699d388640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:18.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699cb87640 10 monclient(hunting): get_auth_request con 0x7f6998135c30 auth_method 0 2024-09-17T18:27:18.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699cb87640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:18.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699cb87640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:18.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699cb87640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:18.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699d388640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:18.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699d388640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:18.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699d388640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:18.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699cb87640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:18.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699cb87640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:18.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699cb87640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:18.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699db89640 1 --2- 172.21.15.121:0/3031595095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6998136040 0x7f699813ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:18.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699d388640 10 monclient(hunting): handle_auth_done global_id 44487 payload 824 2024-09-17T18:27:18.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699d388640 1 -- 172.21.15.121:0/3031595095 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6998135c30 msgr2=0x7f699813a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:18.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699d388640 1 --2- 172.21.15.121:0/3031595095 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6998135c30 0x7f699813a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:18.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699d388640 1 -- 172.21.15.121:0/3031595095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6998136040 msgr2=0x7f699813ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:18.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699d388640 1 --2- 172.21.15.121:0/3031595095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6998136040 0x7f699813ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:18.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699d388640 10 monclient: _finish_hunting 0 2024-09-17T18:27:18.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699d388640 1 monclient: found mon.noname-a 2024-09-17T18:27:18.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699d388640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:27:18.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699d388640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:27:18.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699d388640 1 -- 172.21.15.121:0/3031595095 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f699812c670 con 0x7f699809e8d0 2024-09-17T18:27:18.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699db89640 10 monclient: get_auth_request con 0x7f6998136040 auth_method 0 2024-09-17T18:27:18.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699cb87640 1 --2- 172.21.15.121:0/3031595095 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6998135c30 0x7f699813a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:27:18.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.809+0000 7f699db89640 1 --2- 172.21.15.121:0/3031595095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6998136040 0x7f699813ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T18:27:18.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.811+0000 7f699d388640 1 -- 172.21.15.121:0/3031595095 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f699809e8d0 msgr2=0x7f699813a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:18.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.811+0000 7f699d388640 1 --2- 172.21.15.121:0/3031595095 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f699809e8d0 0x7f699813a0b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f699000c460 tx=0x7f699000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:18.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.811+0000 7f6995ffb640 10 client.?.objecter ms_handle_connect 0x7f699809e8d0 2024-09-17T18:27:18.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.811+0000 7f6995ffb640 10 client.?.objecter resend_mon_ops 2024-09-17T18:27:18.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.811+0000 7f6995ffb640 1 -- 172.21.15.121:0/3031595095 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6990016020 con 0x7f699809e8d0 2024-09-17T18:27:18.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.811+0000 7f6995ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:18.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.811+0000 7f6995ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:27:18.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.811+0000 7f6995ffb640 10 monclient: dump: 2024-09-17T18:27:18.825 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:18.825 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:18.825 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:18.825 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:18.825 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:18.826 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:18.826 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:18.826 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:18.826 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:18.826 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:18.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.811+0000 7f6995ffb640 10 monclient: _finish_auth 0 2024-09-17T18:27:18.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.811+0000 7f6995ffb640 10 monclient: _check_auth_tickets 2024-09-17T18:27:18.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.811+0000 7f6995ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:27:18.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.811+0000 7f699f5ec640 5 monclient: authenticate success, global_id 44487 2024-09-17T18:27:18.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.811+0000 7f6995ffb640 1 -- 172.21.15.121:0/3031595095 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6990012a80 con 0x7f699809e8d0 2024-09-17T18:27:18.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.811+0000 7f699f5ec640 10 monclient: _renew_subs 2024-09-17T18:27:18.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.811+0000 7f699f5ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:27:18.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.811+0000 7f699f5ec640 1 -- 172.21.15.121:0/3031595095 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f699813c1e0 con 0x7f699809e8d0 2024-09-17T18:27:18.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.811+0000 7f6995ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:27:18.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.811+0000 7f6995ffb640 1 -- 172.21.15.121:0/3031595095 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6990011380 con 0x7f699809e8d0 2024-09-17T18:27:18.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.811+0000 7f6995ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:18.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.811+0000 7f6995ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:27:18.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.811+0000 7f6995ffb640 10 monclient: dump: 2024-09-17T18:27:18.827 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:18.827 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:18.827 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:18.827 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:18.827 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:18.827 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:18.827 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:18.827 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:18.827 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:18.827 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:18.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.812+0000 7f699f5ec640 10 client.44487.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:27:18.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.812+0000 7f699f5ec640 10 monclient: _renew_subs 2024-09-17T18:27:18.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.811+0000 7f69977fe640 4 set_mon_vals no callback set 2024-09-17T18:27:18.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.812+0000 7f699f5ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:27:18.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.812+0000 7f699f5ec640 1 -- 172.21.15.121:0/3031595095 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f699813d3c0 con 0x7f699809e8d0 2024-09-17T18:27:18.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.812+0000 7f699f5ec640 1 librados: init done 2024-09-17T18:27:18.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.812+0000 7f699f5ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:27:18.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.812+0000 7f699f5ec640 1 --2- 172.21.15.121:0/3031595095 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f699813b070 0x7f6998136740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:18.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.812+0000 7f699f5ec640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:18.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.812+0000 7f699f5ec640 1 -- 172.21.15.121:0/3031595095 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f699809d210 con 0x7f699813b070 2024-09-17T18:27:18.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.812+0000 7f699cb87640 1 --2- 172.21.15.121:0/3031595095 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f699813b070 0x7f6998136740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:18.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.812+0000 7f699cb87640 10 monclient: get_auth_request con 0x7f699813b070 auth_method 0 2024-09-17T18:27:18.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.812+0000 7f699cb87640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:18.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.812+0000 7f699cb87640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:18.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.812+0000 7f699cb87640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:18.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.812+0000 7f699cb87640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:18.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.812+0000 7f699cb87640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:18.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.813+0000 7f699cb87640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:18.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.813+0000 7f6995ffb640 1 -- 172.21.15.121:0/3031595095 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f6990019020 con 0x7f699809e8d0 2024-09-17T18:27:18.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.813+0000 7f699cb87640 10 monclient(hunting): handle_auth_done global_id 44490 payload 824 2024-09-17T18:27:18.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.813+0000 7f699cb87640 1 -- 172.21.15.121:0/3031595095 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6998135a50 con 0x7f699813b070 2024-09-17T18:27:18.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.813+0000 7f6995ffb640 1 --2- 172.21.15.121:0/3031595095 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6984040170 0x7f6984042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:18.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.813+0000 7f6995ffb640 1 -- 172.21.15.121:0/3031595095 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(343..343 src has 177..343) ==== 5345+0+0 (secure 0 0 0) 0x7f6990053120 con 0x7f699809e8d0 2024-09-17T18:27:18.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.813+0000 7f6995ffb640 10 client.44487.objecter ms_dispatch 0x7f6998000b90 osd_map(343..343 src has 177..343) 2024-09-17T18:27:18.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.813+0000 7f6995ffb640 3 client.44487.objecter handle_osd_map got epochs [343,343] > 0 2024-09-17T18:27:18.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.813+0000 7f6995ffb640 3 client.44487.objecter handle_osd_map decoding full epoch 343 2024-09-17T18:27:18.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.813+0000 7f6995ffb640 20 client.44487.objecter dump_active .. 0 homeless 2024-09-17T18:27:18.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.813+0000 7f699db89640 1 --2- 172.21.15.121:0/3031595095 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6984040170 0x7f6984042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:18.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.813+0000 7f699db89640 10 monclient: get_auth_request con 0x7f6984040170 auth_method 0 2024-09-17T18:27:18.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.815+0000 7f699cb87640 1 -- 172.21.15.121:0/3031595095 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f699813b070 msgr2=0x7f6998136740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:18.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.815+0000 7f6995ffb640 10 client.44487.objecter ms_handle_connect 0x7f699813b070 2024-09-17T18:27:18.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.815+0000 7f6995ffb640 10 client.44487.objecter resend_mon_ops 2024-09-17T18:27:18.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.815+0000 7f699cb87640 1 --2- 172.21.15.121:0/3031595095 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f699813b070 0x7f6998136740 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f69980a5990 tx=0x7f698c007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:18.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.815+0000 7f6995ffb640 1 -- 172.21.15.121:0/3031595095 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f698c017020 con 0x7f699813b070 2024-09-17T18:27:18.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.815+0000 7f6995ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:18.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.815+0000 7f6995ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:27:18.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.815+0000 7f6995ffb640 10 monclient: dump: 2024-09-17T18:27:18.830 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:18.830 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:18.830 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:18.831 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:18.831 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:18.831 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:18.831 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:18.831 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:18.831 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:18.831 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:18.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.815+0000 7f6995ffb640 1 -- 172.21.15.121:0/3031595095 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f698c00fc10 con 0x7f699813b070 2024-09-17T18:27:18.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.815+0000 7f6995ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:27:18.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.815+0000 7f6995ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:27:18.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.815+0000 7f6995ffb640 1 -- 172.21.15.121:0/3031595095 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f699813b070 msgr2=0x7f6998136740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:18.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.815+0000 7f6995ffb640 1 --2- 172.21.15.121:0/3031595095 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f699813b070 0x7f6998136740 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f69980a5990 tx=0x7f698c007700 comp rx=0 tx=0).stop 2024-09-17T18:27:18.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.815+0000 7f6995ffb640 1 -- 172.21.15.121:0/3031595095 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f699813b070 msgr2=0x7f6998136740 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:27:18.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.815+0000 7f6995ffb640 1 --2- 172.21.15.121:0/3031595095 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f699813b070 0x7f6998136740 secure :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f69980a5990 tx=0x7f698c007700 comp rx=0 tx=0).stop 2024-09-17T18:27:18.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.816+0000 7f699db89640 1 --2- 172.21.15.121:0/3031595095 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6984040170 0x7f6984042630 secure :-1 s=READY pgs=4000 cs=0 l=1 rev1=1 crypto rx=0x7f6988009dc0 tx=0x7f698802f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:18.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.816+0000 7f6995ffb640 10 client.44487.objecter ms_handle_connect 0x7f6984040170 2024-09-17T18:27:18.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.821+0000 7f699f5ec640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:27:18.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.821+0000 7f699f5ec640 1 --2- 172.21.15.121:0/3031595095 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6998136c80 0x7f6998137090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:18.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.821+0000 7f699f5ec640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:18.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.821+0000 7f699f5ec640 1 -- 172.21.15.121:0/3031595095 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6998135520 con 0x7f6998136c80 2024-09-17T18:27:18.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.821+0000 7f699cb87640 1 --2- 172.21.15.121:0/3031595095 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6998136c80 0x7f6998137090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:18.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.821+0000 7f699cb87640 10 monclient: get_auth_request con 0x7f6998136c80 auth_method 0 2024-09-17T18:27:18.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.821+0000 7f699cb87640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:18.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.821+0000 7f699cb87640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:18.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.821+0000 7f699cb87640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:18.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.821+0000 7f699cb87640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:18.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.821+0000 7f699cb87640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:18.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.821+0000 7f699cb87640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:18.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.822+0000 7f699cb87640 10 monclient(hunting): handle_auth_done global_id 44493 payload 824 2024-09-17T18:27:18.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.822+0000 7f699cb87640 1 -- 172.21.15.121:0/3031595095 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f69980a1f90 con 0x7f6998136c80 2024-09-17T18:27:18.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.824+0000 7f6995ffb640 10 client.44487.objecter ms_handle_connect 0x7f6998136c80 2024-09-17T18:27:18.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.824+0000 7f6995ffb640 10 client.44487.objecter resend_mon_ops 2024-09-17T18:27:18.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.824+0000 7f699cb87640 1 -- 172.21.15.121:0/3031595095 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6998136c80 msgr2=0x7f6998137090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:18.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.824+0000 7f699cb87640 1 --2- 172.21.15.121:0/3031595095 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6998136c80 0x7f6998137090 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f698c018000 tx=0x7f698c004940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:18.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.824+0000 7f6995ffb640 1 -- 172.21.15.121:0/3031595095 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f698c017030 con 0x7f6998136c80 2024-09-17T18:27:18.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.824+0000 7f6995ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:18.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.824+0000 7f6995ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:27:18.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.824+0000 7f6995ffb640 10 monclient: dump: 2024-09-17T18:27:18.834 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:18.834 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:18.834 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:18.834 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:18.834 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:18.834 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:18.834 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:18.835 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:18.835 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:18.835 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:18.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.824+0000 7f6995ffb640 1 -- 172.21.15.121:0/3031595095 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7f698c0058b0 con 0x7f6998136c80 2024-09-17T18:27:18.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.824+0000 7f6995ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:27:18.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.824+0000 7f6995ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:27:18.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.824+0000 7f6995ffb640 1 -- 172.21.15.121:0/3031595095 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6998136c80 msgr2=0x7f6998137090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:18.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.824+0000 7f6995ffb640 1 --2- 172.21.15.121:0/3031595095 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6998136c80 0x7f6998137090 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f698c018000 tx=0x7f698c004940 comp rx=0 tx=0).stop 2024-09-17T18:27:18.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.824+0000 7f6995ffb640 1 -- 172.21.15.121:0/3031595095 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6998136c80 msgr2=0x7f6998137090 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:27:18.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.824+0000 7f6995ffb640 1 --2- 172.21.15.121:0/3031595095 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6998136c80 0x7f6998137090 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:18.835 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:27:18.835 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:27:18.836 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:27:18.836 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "leader", 2024-09-17T18:27:18.836 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 32, 2024-09-17T18:27:18.836 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 264488, 2024-09-17T18:27:18.836 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:27:18.836 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:27:18.836 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:27:18.836 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:27:18.836 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:27:18.836 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 7, 2024-09-17T18:27:18.836 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:27:18.836 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:27:18.836 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:27:18.836 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:27:18.837 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:27:18.837 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:27:18.837 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:27:18.837 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:27:18.837 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:27:18.837 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:27:18.837 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:27:18.837 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:27:18.837 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:27:18.837 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:27:18.837 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:27:18.837 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:27:18.838 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:27:18.838 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:27:18.838 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:27:18.838 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:27:18.838 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:27:18.838 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:27:18.838 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:27:18.838 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:27:18.838 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:27:18.838 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:27:18.838 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:27:18.838 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:27:18.838 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:18.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:18.838 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:27:18.839 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:27:18.839 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:27:18.839 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:27:18.839 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:27:18.839 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:27:18.839 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:27:18.839 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:27:18.839 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:27:18.839 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:27:18.839 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:27:18.839 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:27:18.839 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:27:18.839 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:27:18.839 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:27:18.839 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:27:18.840 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:27:18.840 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:27:18.840 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:27:18.840 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:27:18.840 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:27:18.840 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:27:18.840 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:27:18.840 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:27:18.840 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:27:18.840 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:27:18.840 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:27:18.840 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:27:18.840 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:27:18.840 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:27:18.841 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:18.841 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:27:18.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:18.841 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:27:18.841 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:27:18.841 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:27:18.841 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:27:18.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:18.841 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:27:18.841 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:27:18.841 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:18.841 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:18.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:18.841 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:27:18.842 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:27:18.842 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:18.842 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:18.842 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:18.842 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:18.842 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:27:18.842 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:27:18.842 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:27:18.842 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:27:18.842 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:27:18.842 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:18.842 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:18.842 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:27:18.842 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:27:18.843 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:27:18.843 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:27:18.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:18.843 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:27:18.843 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:27:18.843 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:18.843 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:18.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:18.843 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:27:18.843 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:27:18.843 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:18.843 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:18.843 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:18.843 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:18.843 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:27:18.843 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:27:18.844 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:27:18.844 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:27:18.844 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:27:18.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:18.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:18.844 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:27:18.844 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:27:18.844 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:27:18.844 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:27:18.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:18.844 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:27:18.844 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:27:18.844 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:18.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:18.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:18.845 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:27:18.845 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:27:18.845 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:18.845 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:18.845 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:18.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:18.845 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:27:18.845 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:27:18.845 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:27:18.845 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:27:18.845 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:27:18.845 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:18.845 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:18.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:18.845 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:27:18.846 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:27:18.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:18.846 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:27:18.846 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:27:18.846 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:27:18.846 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:18.846 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:27:18.846 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:27:18.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:18.846 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:27:18.846 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:27:18.846 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-17T18:27:18.846 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:18.846 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:27:18.847 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:27:18.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:18.847 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:27:18.847 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:27:18.847 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-17T18:27:18.847 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:18.847 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:18.847 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:18.847 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:27:18.847 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:27:18.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.824+0000 7f699f5ec640 10 librados: watch_flush enter 2024-09-17T18:27:18.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.824+0000 7f699f5ec640 10 librados: watch_flush exit 2024-09-17T18:27:18.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.824+0000 7f699f5ec640 20 client.44487.objecter shutdown clearing up homeless session... 2024-09-17T18:27:18.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.824+0000 7f699f5ec640 10 client.44487.objecter successfully canceled tick 2024-09-17T18:27:18.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.824+0000 7f699f5ec640 1 -- 172.21.15.121:0/3031595095 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6984040170 msgr2=0x7f6984042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:18.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.825+0000 7f699f5ec640 1 --2- 172.21.15.121:0/3031595095 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6984040170 0x7f6984042630 secure :-1 s=READY pgs=4000 cs=0 l=1 rev1=1 crypto rx=0x7f6988009dc0 tx=0x7f698802f000 comp rx=0 tx=0).stop 2024-09-17T18:27:18.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.825+0000 7f699f5ec640 10 monclient: shutdown 2024-09-17T18:27:18.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.825+0000 7f699f5ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:27:18.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.825+0000 7f699f5ec640 1 -- 172.21.15.121:0/3031595095 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f699809e8d0 msgr2=0x7f699813a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:18.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.825+0000 7f699f5ec640 1 --2- 172.21.15.121:0/3031595095 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f699809e8d0 0x7f699813a0b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f699000c460 tx=0x7f699000c920 comp rx=0 tx=0).stop 2024-09-17T18:27:18.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.825+0000 7f699db89640 1 -- 172.21.15.121:0/3031595095 reap_dead start 2024-09-17T18:27:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.825+0000 7f699db89640 1 -- 172.21.15.121:0/3031595095 reap_dead start 2024-09-17T18:27:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.825+0000 7f699f5ec640 1 -- 172.21.15.121:0/3031595095 shutdown_connections 2024-09-17T18:27:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.825+0000 7f699f5ec640 1 -- 172.21.15.121:0/3031595095 >> 172.21.15.121:0/3031595095 conn(0x7f69980a75b0 msgr2=0x7f6998074480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:27:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.825+0000 7f699f5ec640 1 -- 172.21.15.121:0/3031595095 shutdown_connections 2024-09-17T18:27:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.825+0000 7f699f5ec640 1 -- 172.21.15.121:0/3031595095 wait complete. 2024-09-17T18:27:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.825+0000 7f699f5ec640 1 librados: shutdown 2024-09-17T18:27:18.850 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:27:18.850 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T18:27:18.850 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T18:27:18.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.946+0000 7f77306c8640 10 monclient: get_monmap_and_config 2024-09-17T18:27:18.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.946+0000 7f77306c8640 10 monclient: build_initial_monmap 2024-09-17T18:27:18.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.946+0000 7f77306c8640 10 monclient: monmap: 2024-09-17T18:27:18.949 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:27:18.949 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:18.949 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:27:18.947261+0000 2024-09-17T18:27:18.949 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:27:18.947261+0000 2024-09-17T18:27:18.949 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:27:18.949 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:18.949 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:27:18.950 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:27:18.950 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:27:18.950 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:27:18.950 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:27:18.950 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:18.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.947+0000 7f77306c8640 1 Processor -- start 2024-09-17T18:27:18.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.947+0000 7f77306c8640 1 -- start start 2024-09-17T18:27:18.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.947+0000 7f77306c8640 10 monclient: init 2024-09-17T18:27:18.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.947+0000 7f77306c8640 10 monclient: _reopen_session rank -1 2024-09-17T18:27:18.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.947+0000 7f77306c8640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-17T18:27:18.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.947+0000 7f77306c8640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f772809e550 0x7f772809e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:18.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.947+0000 7f77306c8640 10 monclient(hunting): picked mon.noname-c con 0x7f772809e550 addr v2:172.21.15.173:3300/0 2024-09-17T18:27:18.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.947+0000 7f77306c8640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f772809ee70 0x7f7728139840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:18.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.947+0000 7f77306c8640 10 monclient(hunting): picked mon.noname-b con 0x7f772809ee70 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:27:18.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.947+0000 7f77306c8640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f772809f2c0 0x7f772813bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:18.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.947+0000 7f77306c8640 10 monclient(hunting): picked mon.noname-a con 0x7f772809f2c0 addr v2:172.21.15.121:3300/0 2024-09-17T18:27:18.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.947+0000 7f77306c8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:18.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.947+0000 7f77306c8640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f77280a4950 con 0x7f772809f2c0 2024-09-17T18:27:18.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.947+0000 7f77306c8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:18.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.947+0000 7f77306c8640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f77280a4460 con 0x7f772809e550 2024-09-17T18:27:18.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.947+0000 7f77306c8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:18.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.947+0000 7f77306c8640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f77280a2b60 con 0x7f772809ee70 2024-09-17T18:27:18.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.947+0000 7f77306c8640 10 monclient(hunting): _renew_subs 2024-09-17T18:27:18.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.947+0000 7f77306c8640 10 monclient(hunting): authenticate will time out at 1242.663913s 2024-09-17T18:27:18.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.948+0000 7f772ec65640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f772809f2c0 0x7f772813bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:18.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.948+0000 7f772dc63640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f772809ee70 0x7f7728139840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:18.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.948+0000 7f772ec65640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f772809f2c0 0x7f772813bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56624/0 (socket says 172.21.15.121:56624) 2024-09-17T18:27:18.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.948+0000 7f772ec65640 1 -- 172.21.15.121:0/1997340704 learned_addr learned my addr 172.21.15.121:0/1997340704 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:27:18.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.948+0000 7f772dc63640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f772809ee70 0x7f7728139840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:46696/0 (socket says 172.21.15.121:46696) 2024-09-17T18:27:18.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.948+0000 7f772ec65640 10 monclient(hunting): get_auth_request con 0x7f772809f2c0 auth_method 0 2024-09-17T18:27:18.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.948+0000 7f772ec65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:18.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.948+0000 7f772ec65640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:18.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.948+0000 7f772ec65640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:18.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.948+0000 7f772dc63640 10 monclient(hunting): get_auth_request con 0x7f772809ee70 auth_method 0 2024-09-17T18:27:18.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.948+0000 7f772dc63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:18.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.948+0000 7f772dc63640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:18.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.948+0000 7f772dc63640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:18.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.948+0000 7f772dc63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:18.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.948+0000 7f772dc63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:18.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.948+0000 7f772dc63640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:18.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.948+0000 7f772ec65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:18.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.948+0000 7f772ec65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:18.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.948+0000 7f772ec65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:18.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.948+0000 7f772e464640 1 --2- 172.21.15.121:0/1997340704 >> v2:172.21.15.173:3300/0 conn(0x7f772809e550 0x7f772809e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:18.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.948+0000 7f772e464640 10 monclient(hunting): get_auth_request con 0x7f772809e550 auth_method 0 2024-09-17T18:27:18.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.948+0000 7f772e464640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:18.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.948+0000 7f772e464640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:18.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.948+0000 7f772e464640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:18.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.948+0000 7f772dc63640 10 monclient(hunting): handle_auth_done global_id 64130 payload 274 2024-09-17T18:27:18.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.948+0000 7f772dc63640 1 -- 172.21.15.121:0/1997340704 >> v2:172.21.15.173:3300/0 conn(0x7f772809e550 msgr2=0x7f772809e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:27:18.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.948+0000 7f772dc63640 1 --2- 172.21.15.121:0/1997340704 >> v2:172.21.15.173:3300/0 conn(0x7f772809e550 0x7f772809e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:18.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.948+0000 7f772dc63640 1 -- 172.21.15.121:0/1997340704 >> v2:172.21.15.121:3300/0 conn(0x7f772809f2c0 msgr2=0x7f772813bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:27:18.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.948+0000 7f772dc63640 1 --2- 172.21.15.121:0/1997340704 >> v2:172.21.15.121:3300/0 conn(0x7f772809f2c0 0x7f772813bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:18.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.948+0000 7f772dc63640 10 monclient: _finish_hunting 0 2024-09-17T18:27:18.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.948+0000 7f772dc63640 1 monclient: found mon.noname-b 2024-09-17T18:27:18.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.948+0000 7f772dc63640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:27:18.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.948+0000 7f772dc63640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:27:18.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.948+0000 7f772dc63640 1 -- 172.21.15.121:0/1997340704 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f772809f840 con 0x7f772809ee70 2024-09-17T18:27:18.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.948+0000 7f772ec65640 1 --2- 172.21.15.121:0/1997340704 >> v2:172.21.15.121:3300/0 conn(0x7f772809f2c0 0x7f772813bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:27:18.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.951+0000 7f772dc63640 1 -- 172.21.15.121:0/1997340704 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f772809ee70 msgr2=0x7f7728139840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:18.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.951+0000 7f772dc63640 1 --2- 172.21.15.121:0/1997340704 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f772809ee70 0x7f7728139840 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f772809e0a0 tx=0x7f771802a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9abcd3a1dae37d18 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:18.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.951+0000 7f772d462640 1 -- 172.21.15.121:0/1997340704 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7718037070 con 0x7f772809ee70 2024-09-17T18:27:18.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.951+0000 7f772d462640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:18.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.951+0000 7f772d462640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:27:18.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.951+0000 7f772d462640 10 monclient: dump: 2024-09-17T18:27:18.964 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:18.964 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:18.964 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:18.964 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:18.964 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:18.964 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:18.964 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:18.964 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:18.964 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:18.965 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:18.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.951+0000 7f772d462640 10 monclient: _finish_auth 0 2024-09-17T18:27:18.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.951+0000 7f772d462640 10 monclient: _check_auth_tickets 2024-09-17T18:27:18.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.951+0000 7f772d462640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:27:18.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.951+0000 7f772d462640 1 -- 172.21.15.121:0/1997340704 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7718032440 con 0x7f772809ee70 2024-09-17T18:27:18.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.951+0000 7f77306c8640 5 monclient: authenticate success, global_id 64130 2024-09-17T18:27:18.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.951+0000 7f77306c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:27:18.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.951+0000 7f772d462640 10 monclient: handle_config config(0 keys) 2024-09-17T18:27:18.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.951+0000 7f77306c8640 10 monclient: get_monmap_and_config success 2024-09-17T18:27:18.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.951+0000 7f772d462640 1 -- 172.21.15.121:0/1997340704 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7718032810 con 0x7f772809ee70 2024-09-17T18:27:18.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.951+0000 7f772d462640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:18.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.951+0000 7f77306c8640 4 set_mon_vals no callback set 2024-09-17T18:27:18.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.951+0000 7f772d462640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:27:18.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.951+0000 7f772d462640 10 monclient: dump: 2024-09-17T18:27:18.966 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:18.966 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:18.966 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:18.966 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:18.966 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:18.966 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:18.966 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:18.966 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:18.966 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:18.966 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:18.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.951+0000 7f77306c8640 10 monclient: shutdown 2024-09-17T18:27:18.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.951+0000 7f77306c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:27:18.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.951+0000 7f77306c8640 1 -- 172.21.15.121:0/1997340704 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f772809ee70 msgr2=0x7f7728139840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:18.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.951+0000 7f77306c8640 1 --2- 172.21.15.121:0/1997340704 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f772809ee70 0x7f7728139840 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f772809e0a0 tx=0x7f771802a000 comp rx=0 tx=0).stop 2024-09-17T18:27:18.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.951+0000 7f77306c8640 1 -- 172.21.15.121:0/1997340704 shutdown_connections 2024-09-17T18:27:18.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.951+0000 7f77306c8640 1 --2- 172.21.15.121:0/1997340704 >> v2:172.21.15.121:3300/0 conn(0x7f772809f2c0 0x7f772813bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:18.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.951+0000 7f77306c8640 1 --2- 172.21.15.121:0/1997340704 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f772809ee70 0x7f7728139840 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:18.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.951+0000 7f77306c8640 1 --2- 172.21.15.121:0/1997340704 >> v2:172.21.15.173:3300/0 conn(0x7f772809e550 0x7f772809e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:18.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.951+0000 7f77306c8640 1 -- 172.21.15.121:0/1997340704 >> 172.21.15.121:0/1997340704 conn(0x7f77280a7830 msgr2=0x7f77280a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:27:18.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.951+0000 7f77306c8640 1 -- 172.21.15.121:0/1997340704 shutdown_connections 2024-09-17T18:27:18.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.951+0000 7f77306c8640 1 -- 172.21.15.121:0/1997340704 wait complete. 2024-09-17T18:27:18.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.952+0000 7f77306c8640 10 monclient: build_initial_monmap 2024-09-17T18:27:18.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.952+0000 7f77306c8640 10 monclient: monmap: 2024-09-17T18:27:18.967 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:27:18.968 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:27:18.968 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:27:18.968 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:27:18.968 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:27:18.968 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:18.968 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:27:18.968 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:27:18.968 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:27:18.968 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:18.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.952+0000 7f77306c8640 1 librados: starting msgr at 2024-09-17T18:27:18.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.952+0000 7f77306c8640 1 librados: starting objecter 2024-09-17T18:27:18.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.952+0000 7f77306c8640 1 Processor -- start 2024-09-17T18:27:18.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.952+0000 7f77306c8640 1 -- start start 2024-09-17T18:27:18.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.952+0000 7f77306c8640 1 librados: setting wanted keys 2024-09-17T18:27:18.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.952+0000 7f77306c8640 1 librados: calling monclient init 2024-09-17T18:27:18.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.952+0000 7f77306c8640 10 monclient: init 2024-09-17T18:27:18.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f77306c8640 10 monclient: _reopen_session rank -1 2024-09-17T18:27:18.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f77306c8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T18:27:18.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f77306c8640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f772809e550 0x7f77281c8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:18.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f77306c8640 10 monclient(hunting): picked mon.noname-a con 0x7f772809e550 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:27:18.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f77306c8640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f772809ee70 0x7f77281b88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:18.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f77306c8640 10 monclient(hunting): picked mon.noname-c con 0x7f772809ee70 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:27:18.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f77306c8640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f772809f2c0 0x7f77281b8e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:18.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f77306c8640 10 monclient(hunting): picked mon.noname-b con 0x7f772809f2c0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:27:18.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f77306c8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:18.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f77306c8640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f772809de60 con 0x7f772809e550 2024-09-17T18:27:18.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f77306c8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:18.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f77306c8640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f772809dce0 con 0x7f772809f2c0 2024-09-17T18:27:18.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f77306c8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:18.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f77306c8640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f77280747e0 con 0x7f772809ee70 2024-09-17T18:27:18.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f77306c8640 10 monclient(hunting): _renew_subs 2024-09-17T18:27:18.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f77306c8640 10 monclient(hunting): authenticate will time out at 1242.669215s 2024-09-17T18:27:18.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f772e464640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f772809e550 0x7f77281c8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:18.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f772e464640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f772809e550 0x7f77281c8ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56634/0 (socket says 172.21.15.121:56634) 2024-09-17T18:27:18.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f772e464640 1 -- 172.21.15.121:0/3437477043 learned_addr learned my addr 172.21.15.121:0/3437477043 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:27:18.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f772e464640 10 monclient(hunting): get_auth_request con 0x7f772809e550 auth_method 0 2024-09-17T18:27:18.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f772e464640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:18.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f772e464640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:18.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f772e464640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:18.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f772dc63640 1 --2- 172.21.15.121:0/3437477043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f772809ee70 0x7f77281b88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:18.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f772dc63640 10 monclient(hunting): get_auth_request con 0x7f772809ee70 auth_method 0 2024-09-17T18:27:18.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f772dc63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:18.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f772dc63640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:18.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f772dc63640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:18.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f772e464640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:18.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f772e464640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:18.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f772e464640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:18.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f772dc63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:18.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f772dc63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:18.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f772dc63640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:18.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f772ec65640 1 --2- 172.21.15.121:0/3437477043 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f772809f2c0 0x7f77281b8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:18.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f772ec65640 10 monclient(hunting): get_auth_request con 0x7f772809f2c0 auth_method 0 2024-09-17T18:27:18.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f772ec65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:18.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f772ec65640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:18.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f772ec65640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:18.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f772dc63640 10 monclient(hunting): handle_auth_done global_id 64133 payload 824 2024-09-17T18:27:18.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f772dc63640 1 -- 172.21.15.121:0/3437477043 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f772809f2c0 msgr2=0x7f77281b8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:18.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f772dc63640 1 --2- 172.21.15.121:0/3437477043 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f772809f2c0 0x7f77281b8e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:18.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f772dc63640 1 -- 172.21.15.121:0/3437477043 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f772809e550 msgr2=0x7f77281c8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:18.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f772dc63640 1 --2- 172.21.15.121:0/3437477043 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f772809e550 0x7f77281c8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:18.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f772dc63640 10 monclient: _finish_hunting 0 2024-09-17T18:27:18.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f772dc63640 1 monclient: found mon.noname-c 2024-09-17T18:27:18.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f772dc63640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:27:18.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f772dc63640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:27:18.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f772dc63640 1 -- 172.21.15.121:0/3437477043 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77281a6fb0 con 0x7f772809ee70 2024-09-17T18:27:18.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f772e464640 1 --2- 172.21.15.121:0/3437477043 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f772809e550 0x7f77281c8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:27:18.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.953+0000 7f772ec65640 1 --2- 172.21.15.121:0/3437477043 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f772809f2c0 0x7f77281b8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T18:27:18.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.955+0000 7f772dc63640 1 -- 172.21.15.121:0/3437477043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f772809ee70 msgr2=0x7f77281b88c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:18.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.955+0000 7f772dc63640 1 --2- 172.21.15.121:0/3437477043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f772809ee70 0x7f77281b88c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7718031000 tx=0x7f77180046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:18.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.955+0000 7f771effd640 10 client.?.objecter ms_handle_connect 0x7f772809ee70 2024-09-17T18:27:18.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.955+0000 7f771effd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:27:18.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.956+0000 7f771effd640 1 -- 172.21.15.121:0/3437477043 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7718041020 con 0x7f772809ee70 2024-09-17T18:27:18.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.956+0000 7f771effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:18.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.956+0000 7f771effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:27:18.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.956+0000 7f771effd640 10 monclient: dump: 2024-09-17T18:27:18.973 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:18.973 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:18.973 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:18.973 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:18.973 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:18.973 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:18.973 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:18.974 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.956+0000 7f771effd640 10 monclient: _finish_auth 0 2024-09-17T18:27:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.956+0000 7f771effd640 10 monclient: _check_auth_tickets 2024-09-17T18:27:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.956+0000 7f771effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:27:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.956+0000 7f771effd640 1 -- 172.21.15.121:0/3437477043 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f771803d8a0 con 0x7f772809ee70 2024-09-17T18:27:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.956+0000 7f77306c8640 5 monclient: authenticate success, global_id 64133 2024-09-17T18:27:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.956+0000 7f771effd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:27:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.956+0000 7f771effd640 1 -- 172.21.15.121:0/3437477043 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7718037040 con 0x7f772809ee70 2024-09-17T18:27:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.956+0000 7f771effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.956+0000 7f771effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:27:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.956+0000 7f771effd640 10 monclient: dump: 2024-09-17T18:27:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:18.975 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:18.975 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:18.975 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:18.975 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:18.975 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:18.975 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:18.975 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:18.975 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:18.975 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:18.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.956+0000 7f77306c8640 10 monclient: _renew_subs 2024-09-17T18:27:18.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.956+0000 7f772cc61640 4 set_mon_vals no callback set 2024-09-17T18:27:18.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.956+0000 7f77306c8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:27:18.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.956+0000 7f77306c8640 1 -- 172.21.15.121:0/3437477043 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77281c93e0 con 0x7f772809ee70 2024-09-17T18:27:18.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.956+0000 7f77306c8640 10 client.64133.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:27:18.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.956+0000 7f77306c8640 10 monclient: _renew_subs 2024-09-17T18:27:18.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.956+0000 7f77306c8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:27:18.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.956+0000 7f77306c8640 1 -- 172.21.15.121:0/3437477043 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f77281a6100 con 0x7f772809ee70 2024-09-17T18:27:18.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.956+0000 7f77306c8640 1 librados: init done 2024-09-17T18:27:18.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.956+0000 7f77306c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:27:18.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.956+0000 7f77306c8640 1 --2- 172.21.15.121:0/3437477043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f77281a3d20 0x7f772802d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:18.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.956+0000 7f77306c8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:18.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.956+0000 7f77306c8640 1 -- 172.21.15.121:0/3437477043 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f77281a4f30 con 0x7f77281a3d20 2024-09-17T18:27:18.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.957+0000 7f772e464640 1 --2- 172.21.15.121:0/3437477043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f77281a3d20 0x7f772802d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:18.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.957+0000 7f771effd640 1 -- 172.21.15.121:0/3437477043 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f7718044020 con 0x7f772809ee70 2024-09-17T18:27:18.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.957+0000 7f772e464640 10 monclient: get_auth_request con 0x7f77281a3d20 auth_method 0 2024-09-17T18:27:18.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.957+0000 7f772e464640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:18.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.957+0000 7f772e464640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:18.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.957+0000 7f772e464640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:18.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.957+0000 7f772e464640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:18.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.957+0000 7f772e464640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:18.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.957+0000 7f772e464640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:18.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.957+0000 7f772e464640 10 monclient(hunting): handle_auth_done global_id 64136 payload 824 2024-09-17T18:27:18.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.957+0000 7f771effd640 1 --2- 172.21.15.121:0/3437477043 >> v2:172.21.15.173:6812/2423786037 conn(0x7f770c0401c0 0x7f770c042680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:18.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.957+0000 7f772e464640 1 -- 172.21.15.121:0/3437477043 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7728071000 con 0x7f77281a3d20 2024-09-17T18:27:18.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.957+0000 7f771effd640 1 -- 172.21.15.121:0/3437477043 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(343..343 src has 177..343) ==== 5345+0+0 (secure 0 0 0) 0x7f7718073aa0 con 0x7f772809ee70 2024-09-17T18:27:18.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.957+0000 7f771effd640 10 client.64133.objecter ms_dispatch 0x7f7728000b90 osd_map(343..343 src has 177..343) 2024-09-17T18:27:18.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.957+0000 7f771effd640 3 client.64133.objecter handle_osd_map got epochs [343,343] > 0 2024-09-17T18:27:18.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.957+0000 7f771effd640 3 client.64133.objecter handle_osd_map decoding full epoch 343 2024-09-17T18:27:18.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.957+0000 7f771effd640 20 client.64133.objecter dump_active .. 0 homeless 2024-09-17T18:27:18.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.957+0000 7f772ec65640 1 --2- 172.21.15.121:0/3437477043 >> v2:172.21.15.173:6812/2423786037 conn(0x7f770c0401c0 0x7f770c042680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:18.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.957+0000 7f772ec65640 10 monclient: get_auth_request con 0x7f770c0401c0 auth_method 0 2024-09-17T18:27:18.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.959+0000 7f771effd640 10 client.64133.objecter ms_handle_connect 0x7f77281a3d20 2024-09-17T18:27:18.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.959+0000 7f771effd640 10 client.64133.objecter resend_mon_ops 2024-09-17T18:27:18.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.959+0000 7f772e464640 1 -- 172.21.15.121:0/3437477043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f77281a3d20 msgr2=0x7f772802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:18.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.959+0000 7f772e464640 1 --2- 172.21.15.121:0/3437477043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f77281a3d20 0x7f772802d440 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7724010c80 tx=0x7f772400e5a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:18.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.959+0000 7f771effd640 1 -- 172.21.15.121:0/3437477043 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7724018070 con 0x7f77281a3d20 2024-09-17T18:27:18.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.959+0000 7f771effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:18.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.959+0000 7f771effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:27:18.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.959+0000 7f771effd640 10 monclient: dump: 2024-09-17T18:27:18.980 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:18.980 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:18.981 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:18.981 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:18.981 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:18.981 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:18.981 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:18.981 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:18.981 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:18.981 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:18.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.959+0000 7f771effd640 1 -- 172.21.15.121:0/3437477043 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7724014640 con 0x7f77281a3d20 2024-09-17T18:27:18.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.959+0000 7f771effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:27:18.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.959+0000 7f771effd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:27:18.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.959+0000 7f771effd640 1 -- 172.21.15.121:0/3437477043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f77281a3d20 msgr2=0x7f772802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:18.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.959+0000 7f771effd640 1 --2- 172.21.15.121:0/3437477043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f77281a3d20 0x7f772802d440 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7724010c80 tx=0x7f772400e5a0 comp rx=0 tx=0).stop 2024-09-17T18:27:18.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.959+0000 7f771effd640 1 -- 172.21.15.121:0/3437477043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f77281a3d20 msgr2=0x7f772802d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:27:18.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.959+0000 7f771effd640 1 --2- 172.21.15.121:0/3437477043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f77281a3d20 0x7f772802d440 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7724010c80 tx=0x7f772400e5a0 comp rx=0 tx=0).stop 2024-09-17T18:27:18.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.960+0000 7f771effd640 10 client.64133.objecter ms_handle_connect 0x7f770c0401c0 2024-09-17T18:27:18.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.960+0000 7f772ec65640 1 --2- 172.21.15.121:0/3437477043 >> v2:172.21.15.173:6812/2423786037 conn(0x7f770c0401c0 0x7f770c042680 secure :-1 s=READY pgs=4001 cs=0 l=1 rev1=1 crypto rx=0x7f772000bea0 tx=0x7f772000d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:18.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.965+0000 7f77306c8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:27:18.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.965+0000 7f77306c8640 1 --2- 172.21.15.121:0/3437477043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f77281a4100 0x7f772802d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:18.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.965+0000 7f77306c8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:18.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.965+0000 7f77306c8640 1 -- 172.21.15.121:0/3437477043 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f772809e930 con 0x7f77281a4100 2024-09-17T18:27:18.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.966+0000 7f772e464640 1 --2- 172.21.15.121:0/3437477043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f77281a4100 0x7f772802d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:18.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.966+0000 7f772e464640 10 monclient: get_auth_request con 0x7f77281a4100 auth_method 0 2024-09-17T18:27:18.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.966+0000 7f772e464640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:18.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.966+0000 7f772e464640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:18.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.966+0000 7f772e464640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:18.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.966+0000 7f772e464640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:18.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.966+0000 7f772e464640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:18.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.966+0000 7f772e464640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:18.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.966+0000 7f772e464640 10 monclient(hunting): handle_auth_done global_id 64139 payload 824 2024-09-17T18:27:18.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.966+0000 7f772e464640 1 -- 172.21.15.121:0/3437477043 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f772806cd40 con 0x7f77281a4100 2024-09-17T18:27:18.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.968+0000 7f772e464640 1 -- 172.21.15.121:0/3437477043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f77281a4100 msgr2=0x7f772802d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:18.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.968+0000 7f771effd640 10 client.64133.objecter ms_handle_connect 0x7f77281a4100 2024-09-17T18:27:18.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.968+0000 7f771effd640 10 client.64133.objecter resend_mon_ops 2024-09-17T18:27:18.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.968+0000 7f772e464640 1 --2- 172.21.15.121:0/3437477043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f77281a4100 0x7f772802d980 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f772401a000 tx=0x7f77240046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:18.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.968+0000 7f771effd640 1 -- 172.21.15.121:0/3437477043 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f772401dcf0 con 0x7f77281a4100 2024-09-17T18:27:18.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.968+0000 7f771effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:18.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.968+0000 7f771effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:27:18.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.968+0000 7f771effd640 10 monclient: dump: 2024-09-17T18:27:18.983 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:18.983 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:18.983 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:18.983 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:18.984 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:18.984 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:18.984 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:18.984 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:18.984 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:18.984 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:18.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.968+0000 7f771effd640 1 -- 172.21.15.121:0/3437477043 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7f7724004030 con 0x7f77281a4100 2024-09-17T18:27:18.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.968+0000 7f771effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:27:18.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.968+0000 7f771effd640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:27:18.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.968+0000 7f771effd640 1 -- 172.21.15.121:0/3437477043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f77281a4100 msgr2=0x7f772802d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:18.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.968+0000 7f771effd640 1 --2- 172.21.15.121:0/3437477043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f77281a4100 0x7f772802d980 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f772401a000 tx=0x7f77240046c0 comp rx=0 tx=0).stop 2024-09-17T18:27:18.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.968+0000 7f771effd640 1 -- 172.21.15.121:0/3437477043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f77281a4100 msgr2=0x7f772802d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:27:18.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.968+0000 7f771effd640 1 --2- 172.21.15.121:0/3437477043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f77281a4100 0x7f772802d980 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f772401a000 tx=0x7f77240046c0 comp rx=0 tx=0).stop 2024-09-17T18:27:18.985 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:27:18.985 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:27:18.985 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:27:18.985 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:27:18.985 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 32, 2024-09-17T18:27:18.985 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 17700, 2024-09-17T18:27:18.985 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:27:18.985 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:27:18.985 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:27:18.985 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:27:18.986 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:27:18.986 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 7, 2024-09-17T18:27:18.986 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:27:18.986 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:27:18.986 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:27:18.986 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:27:18.986 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:27:18.986 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:27:18.986 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:27:18.986 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:27:18.986 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:27:18.986 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:27:18.986 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:27:18.986 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:27:18.987 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:27:18.987 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:27:18.987 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:27:18.987 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:27:18.987 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:27:18.987 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:27:18.987 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:27:18.987 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:27:18.987 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:27:18.987 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:27:18.987 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:27:18.987 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:27:18.987 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:27:18.987 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:27:18.988 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:27:18.988 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:27:18.988 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:18.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:18.988 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:27:18.988 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:27:18.988 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:27:18.988 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:27:18.988 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:27:18.988 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:27:18.988 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:27:18.988 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:27:18.988 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:27:18.989 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:27:18.989 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:27:18.989 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:27:18.989 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:27:18.989 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:27:18.989 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:27:18.989 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:27:18.989 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:27:18.989 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:27:18.989 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:27:18.989 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:27:18.989 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:27:18.989 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:27:18.989 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:27:18.990 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:27:18.990 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:27:18.990 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:27:18.990 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:27:18.990 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:27:18.990 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:27:18.990 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:27:18.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:18.990 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:27:18.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:18.990 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:27:18.990 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:27:18.990 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:27:18.991 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:27:18.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:18.991 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:27:18.991 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:27:18.991 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:18.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:18.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:18.991 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:27:18.991 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:27:18.991 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:18.991 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:18.991 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:18.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:18.991 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:27:18.991 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:27:18.992 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:27:18.992 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:27:18.992 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:27:18.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:18.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:18.992 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:27:18.992 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:27:18.992 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:27:18.992 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:27:18.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:18.992 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:27:18.992 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:27:18.992 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:18.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:18.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:18.993 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:27:18.993 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:27:18.993 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:18.993 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:18.993 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:18.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:18.993 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:27:18.993 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:27:18.993 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:27:18.993 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:27:18.993 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:27:18.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:18.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:18.994 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:27:18.994 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:27:18.994 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:27:18.994 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:27:18.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:18.994 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:27:18.994 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:27:18.994 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:18.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:18.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:18.994 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:27:18.994 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:27:18.994 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:18.995 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:18.995 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:18.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:18.995 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:27:18.995 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:27:18.995 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:27:18.995 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:27:18.995 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:27:18.995 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:18.995 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:18.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:18.995 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:27:18.995 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:27:18.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:18.996 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:27:18.996 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:27:18.996 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:27:18.996 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:18.996 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:27:18.996 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:27:18.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:18.996 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:27:18.996 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:27:18.996 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-17T18:27:18.996 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:18.996 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:18.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:18.996 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:27:18.996 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:27:18.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.969+0000 7f77306c8640 10 librados: watch_flush enter 2024-09-17T18:27:18.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.969+0000 7f77306c8640 10 librados: watch_flush exit 2024-09-17T18:27:18.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.969+0000 7f77306c8640 20 client.64133.objecter shutdown clearing up homeless session... 2024-09-17T18:27:18.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.969+0000 7f77306c8640 10 client.64133.objecter successfully canceled tick 2024-09-17T18:27:18.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.969+0000 7f77306c8640 1 -- 172.21.15.121:0/3437477043 >> v2:172.21.15.173:6812/2423786037 conn(0x7f770c0401c0 msgr2=0x7f770c042680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:18.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.969+0000 7f77306c8640 1 --2- 172.21.15.121:0/3437477043 >> v2:172.21.15.173:6812/2423786037 conn(0x7f770c0401c0 0x7f770c042680 secure :-1 s=READY pgs=4001 cs=0 l=1 rev1=1 crypto rx=0x7f772000bea0 tx=0x7f772000d040 comp rx=0 tx=0).stop 2024-09-17T18:27:18.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.969+0000 7f77306c8640 10 monclient: shutdown 2024-09-17T18:27:18.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.969+0000 7f77306c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:27:18.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.969+0000 7f77306c8640 1 -- 172.21.15.121:0/3437477043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f772809ee70 msgr2=0x7f77281b88c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:18.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.969+0000 7f77306c8640 1 --2- 172.21.15.121:0/3437477043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f772809ee70 0x7f77281b88c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7718031000 tx=0x7f77180046c0 comp rx=0 tx=0).stop 2024-09-17T18:27:18.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.969+0000 7f772ec65640 1 -- 172.21.15.121:0/3437477043 reap_dead start 2024-09-17T18:27:18.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.969+0000 7f772ec65640 1 -- 172.21.15.121:0/3437477043 reap_dead start 2024-09-17T18:27:18.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.969+0000 7f77306c8640 1 -- 172.21.15.121:0/3437477043 shutdown_connections 2024-09-17T18:27:18.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.969+0000 7f77306c8640 1 -- 172.21.15.121:0/3437477043 >> 172.21.15.121:0/3437477043 conn(0x7f77280a7830 msgr2=0x7f77280a5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:27:18.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.969+0000 7f77306c8640 1 -- 172.21.15.121:0/3437477043 shutdown_connections 2024-09-17T18:27:18.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.969+0000 7f77306c8640 1 -- 172.21.15.121:0/3437477043 wait complete. 2024-09-17T18:27:18.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:18.969+0000 7f77306c8640 1 librados: shutdown 2024-09-17T18:27:18.999 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:27:19.000 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T18:27:19.000 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T18:27:19.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.095+0000 7f41d38a9640 10 monclient: get_monmap_and_config 2024-09-17T18:27:19.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.095+0000 7f41d38a9640 10 monclient: build_initial_monmap 2024-09-17T18:27:19.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.095+0000 7f41d38a9640 10 monclient: monmap: 2024-09-17T18:27:19.097 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:27:19.097 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:19.097 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:27:19.096265+0000 2024-09-17T18:27:19.097 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:27:19.096265+0000 2024-09-17T18:27:19.097 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:27:19.097 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:19.098 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:27:19.098 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:27:19.098 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:27:19.098 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:27:19.098 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:27:19.098 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:19.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.096+0000 7f41d38a9640 1 Processor -- start 2024-09-17T18:27:19.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.096+0000 7f41d38a9640 1 -- start start 2024-09-17T18:27:19.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.096+0000 7f41d38a9640 10 monclient: init 2024-09-17T18:27:19.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.096+0000 7f41d38a9640 10 monclient: _reopen_session rank -1 2024-09-17T18:27:19.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.096+0000 7f41d38a9640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-17T18:27:19.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.096+0000 7f41d38a9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f41cc09e550 addr v1:172.21.15.121:6789/0 2024-09-17T18:27:19.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.096+0000 7f41d38a9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f41cc09f040 addr v1:172.21.15.173:6789/0 2024-09-17T18:27:19.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.096+0000 7f41d38a9640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f41cc13bf70 0x7f41cc13e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:19.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.096+0000 7f41d38a9640 10 monclient(hunting): picked mon.noname-a con 0x7f41cc13bf70 addr v2:172.21.15.121:3300/0 2024-09-17T18:27:19.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.096+0000 7f41d38a9640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41cc0a4460 con 0x7f41cc09e550 2024-09-17T18:27:19.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.096+0000 7f41d38a9640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41cc0a2b60 con 0x7f41cc09f040 2024-09-17T18:27:19.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.096+0000 7f41d38a9640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:19.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.096+0000 7f41d38a9640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f41cc0a4950 con 0x7f41cc13bf70 2024-09-17T18:27:19.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.096+0000 7f41d38a9640 10 monclient(hunting): _renew_subs 2024-09-17T18:27:19.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.096+0000 7f41d38a9640 10 monclient(hunting): authenticate will time out at 1242.812945s 2024-09-17T18:27:19.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.097+0000 7f41d1645640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f41cc09e550 0x7f41cc09e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:35888/0 (socket says 172.21.15.121:35888) 2024-09-17T18:27:19.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.097+0000 7f41d1645640 1 -- 172.21.15.121:0/2651214190 learned_addr learned my addr 172.21.15.121:0/2651214190 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:27:19.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.097+0000 7f41d1e46640 1 --2- 172.21.15.121:0/2651214190 >> v2:172.21.15.121:3300/0 conn(0x7f41cc13bf70 0x7f41cc13e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:19.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.097+0000 7f41d1e46640 10 monclient(hunting): get_auth_request con 0x7f41cc13bf70 auth_method 0 2024-09-17T18:27:19.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.097+0000 7f41d1e46640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:19.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.097+0000 7f41d1e46640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:19.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.097+0000 7f41d1e46640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:19.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.097+0000 7f41d1e46640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:19.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.097+0000 7f41d1e46640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:19.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.097+0000 7f41d1e46640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:19.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.097+0000 7f41d1e46640 10 monclient(hunting): handle_auth_done global_id 44502 payload 274 2024-09-17T18:27:19.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.097+0000 7f41d1e46640 1 -- 172.21.15.121:0/2651214190 >> v1:172.21.15.173:6789/0 conn(0x7f41cc09f040 legacy=0x7f41cc13a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:27:19.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.097+0000 7f41d1e46640 1 -- 172.21.15.121:0/2651214190 >> v1:172.21.15.121:6789/0 conn(0x7f41cc09e550 legacy=0x7f41cc09e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:27:19.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.097+0000 7f41d1e46640 10 monclient: _finish_hunting 0 2024-09-17T18:27:19.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.097+0000 7f41d1e46640 1 monclient: found mon.noname-a 2024-09-17T18:27:19.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.097+0000 7f41d1e46640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:27:19.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.097+0000 7f41d1e46640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:27:19.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.097+0000 7f41d1e46640 1 -- 172.21.15.121:0/2651214190 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41cc09f840 con 0x7f41cc13bf70 2024-09-17T18:27:19.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.100+0000 7f41d1e46640 1 -- 172.21.15.121:0/2651214190 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41cc13bf70 msgr2=0x7f41cc13e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:19.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.100+0000 7f41d1e46640 1 --2- 172.21.15.121:0/2651214190 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41cc13bf70 0x7f41cc13e360 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f41cc074790 tx=0x7f41c402c670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ca60a9abf6ece08 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:19.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.100+0000 7f41c3fff640 1 -- 172.21.15.121:0/2651214190 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c4007a30 con 0x7f41cc13bf70 2024-09-17T18:27:19.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.100+0000 7f41c3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:19.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.100+0000 7f41c3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:27:19.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.100+0000 7f41c3fff640 10 monclient: dump: 2024-09-17T18:27:19.110 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:19.110 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:19.110 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:19.110 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:19.110 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:19.110 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:19.110 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:19.110 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:19.110 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:19.110 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:19.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.100+0000 7f41c3fff640 10 monclient: _finish_auth 0 2024-09-17T18:27:19.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.100+0000 7f41c3fff640 10 monclient: _check_auth_tickets 2024-09-17T18:27:19.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.100+0000 7f41c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:27:19.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.100+0000 7f41d38a9640 5 monclient: authenticate success, global_id 44502 2024-09-17T18:27:19.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.100+0000 7f41c3fff640 1 -- 172.21.15.121:0/2651214190 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41c4007bd0 con 0x7f41cc13bf70 2024-09-17T18:27:19.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.100+0000 7f41d38a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:27:19.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.100+0000 7f41c3fff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:27:19.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.100+0000 7f41d38a9640 10 monclient: get_monmap_and_config success 2024-09-17T18:27:19.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.100+0000 7f41c3fff640 1 -- 172.21.15.121:0/2651214190 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c4033400 con 0x7f41cc13bf70 2024-09-17T18:27:19.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.100+0000 7f41d38a9640 4 set_mon_vals no callback set 2024-09-17T18:27:19.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.100+0000 7f41c3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:19.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.100+0000 7f41c3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:27:19.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.100+0000 7f41c3fff640 10 monclient: dump: 2024-09-17T18:27:19.111 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:19.111 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:19.111 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:19.112 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:19.112 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:19.112 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:19.112 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:19.112 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:19.112 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:19.112 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:19.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.100+0000 7f41d38a9640 10 monclient: shutdown 2024-09-17T18:27:19.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.100+0000 7f41d38a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:27:19.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.100+0000 7f41d38a9640 1 -- 172.21.15.121:0/2651214190 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41cc13bf70 msgr2=0x7f41cc13e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:19.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.100+0000 7f41d38a9640 1 --2- 172.21.15.121:0/2651214190 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41cc13bf70 0x7f41cc13e360 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f41cc074790 tx=0x7f41c402c670 comp rx=0 tx=0).stop 2024-09-17T18:27:19.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.100+0000 7f41d38a9640 1 -- 172.21.15.121:0/2651214190 shutdown_connections 2024-09-17T18:27:19.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.100+0000 7f41d38a9640 1 --2- 172.21.15.121:0/2651214190 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41cc13bf70 0x7f41cc13e360 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:19.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.100+0000 7f41d38a9640 1 -- 172.21.15.121:0/2651214190 >> 172.21.15.121:0/2651214190 conn(0x7f41cc0a7830 msgr2=0x7f41cc0a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:27:19.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.100+0000 7f41d38a9640 1 -- 172.21.15.121:0/2651214190 shutdown_connections 2024-09-17T18:27:19.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.100+0000 7f41d38a9640 1 -- 172.21.15.121:0/2651214190 wait complete. 2024-09-17T18:27:19.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.101+0000 7f41d38a9640 10 monclient: build_initial_monmap 2024-09-17T18:27:19.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.101+0000 7f41d38a9640 10 monclient: monmap: 2024-09-17T18:27:19.113 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:27:19.113 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:27:19.113 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:27:19.113 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:27:19.113 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:27:19.113 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:19.113 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:27:19.113 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:27:19.114 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:27:19.114 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:19.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.101+0000 7f41d38a9640 1 librados: starting msgr at 2024-09-17T18:27:19.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.101+0000 7f41d38a9640 1 librados: starting objecter 2024-09-17T18:27:19.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.101+0000 7f41d38a9640 1 Processor -- start 2024-09-17T18:27:19.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.101+0000 7f41d38a9640 1 -- start start 2024-09-17T18:27:19.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.101+0000 7f41d38a9640 1 librados: setting wanted keys 2024-09-17T18:27:19.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.101+0000 7f41d38a9640 1 librados: calling monclient init 2024-09-17T18:27:19.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.101+0000 7f41d38a9640 10 monclient: init 2024-09-17T18:27:19.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d38a9640 10 monclient: _reopen_session rank -1 2024-09-17T18:27:19.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d38a9640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T18:27:19.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d38a9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41cc09e550 0x7f41cc11f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:19.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d38a9640 10 monclient(hunting): picked mon.noname-a con 0x7f41cc09e550 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:27:19.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d38a9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41cc09f040 0x7f41cc1367c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:19.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d38a9640 10 monclient(hunting): picked mon.noname-c con 0x7f41cc09f040 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:27:19.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d38a9640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41cc13bf70 0x7f41cc11d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:19.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d38a9640 10 monclient(hunting): picked mon.noname-b con 0x7f41cc13bf70 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:27:19.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d38a9640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:19.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d38a9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f41cc09f490 con 0x7f41cc09e550 2024-09-17T18:27:19.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d38a9640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:19.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d38a9640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f41cc09dce0 con 0x7f41cc13bf70 2024-09-17T18:27:19.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d38a9640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:19.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d38a9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f41cc0a5c30 con 0x7f41cc09f040 2024-09-17T18:27:19.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d38a9640 10 monclient(hunting): _renew_subs 2024-09-17T18:27:19.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d38a9640 10 monclient(hunting): authenticate will time out at 1242.818217s 2024-09-17T18:27:19.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d1645640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41cc09e550 0x7f41cc11f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:19.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d1645640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41cc09e550 0x7f41cc11f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45748/0 (socket says 172.21.15.121:45748) 2024-09-17T18:27:19.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d1645640 1 -- 172.21.15.121:0/2000027748 learned_addr learned my addr 172.21.15.121:0/2000027748 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:27:19.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d1645640 10 monclient(hunting): get_auth_request con 0x7f41cc09e550 auth_method 0 2024-09-17T18:27:19.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d1645640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:19.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d1645640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:19.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d1645640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:19.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d0e44640 1 --2- 172.21.15.121:0/2000027748 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41cc09f040 0x7f41cc1367c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:19.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d0e44640 10 monclient(hunting): get_auth_request con 0x7f41cc09f040 auth_method 0 2024-09-17T18:27:19.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d0e44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:19.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d0e44640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:19.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d0e44640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:19.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d1645640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:19.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d1645640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:19.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d1645640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:19.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d0e44640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:19.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d0e44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:19.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d1e46640 1 --2- 172.21.15.121:0/2000027748 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41cc13bf70 0x7f41cc11d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:19.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d0e44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:19.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d1e46640 10 monclient(hunting): get_auth_request con 0x7f41cc13bf70 auth_method 0 2024-09-17T18:27:19.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d1e46640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:19.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d1e46640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:19.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d1e46640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:19.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d1e46640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:19.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d1e46640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:19.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d1e46640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:19.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d1645640 10 monclient(hunting): handle_auth_done global_id 44505 payload 824 2024-09-17T18:27:19.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d1645640 1 -- 172.21.15.121:0/2000027748 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41cc09f040 msgr2=0x7f41cc1367c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:19.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d1645640 1 --2- 172.21.15.121:0/2000027748 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41cc09f040 0x7f41cc1367c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:19.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d1645640 1 -- 172.21.15.121:0/2000027748 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41cc13bf70 msgr2=0x7f41cc11d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:19.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d1645640 1 --2- 172.21.15.121:0/2000027748 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41cc13bf70 0x7f41cc11d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:19.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d1645640 10 monclient: _finish_hunting 0 2024-09-17T18:27:19.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d1645640 1 monclient: found mon.noname-a 2024-09-17T18:27:19.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d1645640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:27:19.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d1645640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:27:19.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d1645640 1 -- 172.21.15.121:0/2000027748 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41cc137480 con 0x7f41cc09e550 2024-09-17T18:27:19.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.102+0000 7f41d0e44640 1 --2- 172.21.15.121:0/2000027748 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41cc09f040 0x7f41cc1367c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:27:19.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.104+0000 7f41d1645640 1 -- 172.21.15.121:0/2000027748 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41cc09e550 msgr2=0x7f41cc11f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:19.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.104+0000 7f41d1645640 1 --2- 172.21.15.121:0/2000027748 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41cc09e550 0x7f41cc11f970 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f41c800c460 tx=0x7f41c800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:19.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.104+0000 7f41c1ffb640 10 client.?.objecter ms_handle_connect 0x7f41cc09e550 2024-09-17T18:27:19.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.104+0000 7f41c1ffb640 10 client.?.objecter resend_mon_ops 2024-09-17T18:27:19.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.104+0000 7f41c1ffb640 1 -- 172.21.15.121:0/2000027748 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c8007070 con 0x7f41cc09e550 2024-09-17T18:27:19.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.104+0000 7f41c1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:19.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.104+0000 7f41c1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:27:19.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.104+0000 7f41c1ffb640 10 monclient: dump: 2024-09-17T18:27:19.119 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:19.119 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:19.119 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:19.119 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:19.119 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:19.119 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:19.119 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:19.119 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:19.119 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:19.119 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:19.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.104+0000 7f41c1ffb640 10 monclient: _finish_auth 0 2024-09-17T18:27:19.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.104+0000 7f41c1ffb640 10 monclient: _check_auth_tickets 2024-09-17T18:27:19.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.104+0000 7f41c1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:27:19.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.104+0000 7f41c1ffb640 1 -- 172.21.15.121:0/2000027748 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41c8007230 con 0x7f41cc09e550 2024-09-17T18:27:19.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.104+0000 7f41d38a9640 5 monclient: authenticate success, global_id 44505 2024-09-17T18:27:19.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.104+0000 7f41d38a9640 10 monclient: _renew_subs 2024-09-17T18:27:19.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.104+0000 7f41d38a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:27:19.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.104+0000 7f41d38a9640 1 -- 172.21.15.121:0/2000027748 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41cc09e240 con 0x7f41cc09e550 2024-09-17T18:27:19.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.104+0000 7f41c1ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:27:19.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.104+0000 7f41c1ffb640 1 -- 172.21.15.121:0/2000027748 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c8007620 con 0x7f41cc09e550 2024-09-17T18:27:19.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.104+0000 7f41c1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:19.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.104+0000 7f41c1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:27:19.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.104+0000 7f41c1ffb640 10 monclient: dump: 2024-09-17T18:27:19.120 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:19.120 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:19.120 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:19.120 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:19.120 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:19.121 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:19.121 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:19.121 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:19.121 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:19.121 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:19.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.104+0000 7f41c37fe640 4 set_mon_vals no callback set 2024-09-17T18:27:19.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.105+0000 7f41d38a9640 10 client.44505.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:27:19.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.105+0000 7f41d38a9640 10 monclient: _renew_subs 2024-09-17T18:27:19.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.105+0000 7f41d38a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:27:19.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.105+0000 7f41d38a9640 1 -- 172.21.15.121:0/2000027748 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41cc120e90 con 0x7f41cc09e550 2024-09-17T18:27:19.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.105+0000 7f41d38a9640 1 librados: init done 2024-09-17T18:27:19.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.106+0000 7f41d38a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:27:19.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.106+0000 7f41d38a9640 1 --2- 172.21.15.121:0/2000027748 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41cc136d00 0x7f41cc11dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:19.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.106+0000 7f41d38a9640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:19.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.106+0000 7f41d38a9640 1 -- 172.21.15.121:0/2000027748 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f41cc09f610 con 0x7f41cc136d00 2024-09-17T18:27:19.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.106+0000 7f41c1ffb640 1 -- 172.21.15.121:0/2000027748 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f41c801b400 con 0x7f41cc09e550 2024-09-17T18:27:19.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.106+0000 7f41c1ffb640 1 --2- 172.21.15.121:0/2000027748 >> v2:172.21.15.173:6812/2423786037 conn(0x7f41b00401c0 0x7f41b0042680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:19.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.106+0000 7f41c1ffb640 1 -- 172.21.15.121:0/2000027748 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(343..343 src has 177..343) ==== 5345+0+0 (secure 0 0 0) 0x7f41c8053350 con 0x7f41cc09e550 2024-09-17T18:27:19.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.106+0000 7f41c1ffb640 10 client.44505.objecter ms_dispatch 0x7f41cc000b90 osd_map(343..343 src has 177..343) 2024-09-17T18:27:19.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.106+0000 7f41c1ffb640 3 client.44505.objecter handle_osd_map got epochs [343,343] > 0 2024-09-17T18:27:19.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.106+0000 7f41c1ffb640 3 client.44505.objecter handle_osd_map decoding full epoch 343 2024-09-17T18:27:19.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.106+0000 7f41d0e44640 1 --2- 172.21.15.121:0/2000027748 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41cc136d00 0x7f41cc11dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:19.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.106+0000 7f41d0e44640 10 monclient: get_auth_request con 0x7f41cc136d00 auth_method 0 2024-09-17T18:27:19.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.106+0000 7f41d0e44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:19.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.106+0000 7f41d0e44640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:19.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.106+0000 7f41d0e44640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:19.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.106+0000 7f41c1ffb640 20 client.44505.objecter dump_active .. 0 homeless 2024-09-17T18:27:19.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.106+0000 7f41d1e46640 1 --2- 172.21.15.121:0/2000027748 >> v2:172.21.15.173:6812/2423786037 conn(0x7f41b00401c0 0x7f41b0042680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:19.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.106+0000 7f41d0e44640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:19.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.106+0000 7f41d0e44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:19.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.106+0000 7f41d0e44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:19.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.106+0000 7f41d1e46640 10 monclient: get_auth_request con 0x7f41b00401c0 auth_method 0 2024-09-17T18:27:19.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.106+0000 7f41d0e44640 10 monclient(hunting): handle_auth_done global_id 24811 payload 824 2024-09-17T18:27:19.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.106+0000 7f41d0e44640 1 -- 172.21.15.121:0/2000027748 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f41cc074900 con 0x7f41cc136d00 2024-09-17T18:27:19.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.108+0000 7f41d0e44640 1 -- 172.21.15.121:0/2000027748 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41cc136d00 msgr2=0x7f41cc11dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:19.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.108+0000 7f41c1ffb640 10 client.44505.objecter ms_handle_connect 0x7f41cc136d00 2024-09-17T18:27:19.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.108+0000 7f41d0e44640 1 --2- 172.21.15.121:0/2000027748 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41cc136d00 0x7f41cc11dbe0 secure :-1 s=READY pgs=7435 cs=0 l=1 rev1=1 crypto rx=0x7f41cc121380 tx=0x7f41bc00ff70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:19.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.108+0000 7f41c1ffb640 10 client.44505.objecter resend_mon_ops 2024-09-17T18:27:19.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.108+0000 7f41c1ffb640 10 client.44505.objecter ms_handle_connect 0x7f41b00401c0 2024-09-17T18:27:19.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.108+0000 7f41d1e46640 1 --2- 172.21.15.121:0/2000027748 >> v2:172.21.15.173:6812/2423786037 conn(0x7f41b00401c0 0x7f41b0042680 secure :-1 s=READY pgs=4002 cs=0 l=1 rev1=1 crypto rx=0x7f41c4036000 tx=0x7f41c4031000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:19.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.109+0000 7f41c1ffb640 1 -- 172.21.15.121:0/2000027748 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41bc018020 con 0x7f41cc136d00 2024-09-17T18:27:19.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.109+0000 7f41c1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:19.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.109+0000 7f41c1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:27:19.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.109+0000 7f41c1ffb640 10 monclient: dump: 2024-09-17T18:27:19.126 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:19.126 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:19.126 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:19.126 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:19.126 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:19.126 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:19.126 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:19.126 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:19.126 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:19.127 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:19.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.109+0000 7f41c1ffb640 1 -- 172.21.15.121:0/2000027748 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f41bc00dd10 con 0x7f41cc136d00 2024-09-17T18:27:19.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.109+0000 7f41c1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:27:19.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.109+0000 7f41c1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:27:19.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.109+0000 7f41c1ffb640 1 -- 172.21.15.121:0/2000027748 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41cc136d00 msgr2=0x7f41cc11dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:19.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.109+0000 7f41c1ffb640 1 --2- 172.21.15.121:0/2000027748 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41cc136d00 0x7f41cc11dbe0 secure :-1 s=READY pgs=7435 cs=0 l=1 rev1=1 crypto rx=0x7f41cc121380 tx=0x7f41bc00ff70 comp rx=0 tx=0).stop 2024-09-17T18:27:19.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.109+0000 7f41c1ffb640 1 -- 172.21.15.121:0/2000027748 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41cc136d00 msgr2=0x7f41cc11dbe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:27:19.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.109+0000 7f41c1ffb640 1 --2- 172.21.15.121:0/2000027748 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41cc136d00 0x7f41cc11dbe0 secure :-1 s=CLOSED pgs=7435 cs=0 l=1 rev1=1 crypto rx=0x7f41cc121380 tx=0x7f41bc00ff70 comp rx=0 tx=0).stop 2024-09-17T18:27:19.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.115+0000 7f41d38a9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:27:19.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.115+0000 7f41d38a9640 1 --2- 172.21.15.121:0/2000027748 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41cc11e120 0x7f41cc1379f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:19.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.115+0000 7f41d38a9640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:19.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.115+0000 7f41d38a9640 1 -- 172.21.15.121:0/2000027748 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f41cc11e530 con 0x7f41cc11e120 2024-09-17T18:27:19.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.115+0000 7f41d0e44640 1 --2- 172.21.15.121:0/2000027748 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41cc11e120 0x7f41cc1379f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:19.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.116+0000 7f41d0e44640 10 monclient: get_auth_request con 0x7f41cc11e120 auth_method 0 2024-09-17T18:27:19.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.116+0000 7f41d0e44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:19.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.116+0000 7f41d0e44640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:19.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.116+0000 7f41d0e44640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:19.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.116+0000 7f41d0e44640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:19.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.116+0000 7f41d0e44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:19.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.116+0000 7f41d0e44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:19.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.116+0000 7f41d0e44640 10 monclient(hunting): handle_auth_done global_id 24814 payload 824 2024-09-17T18:27:19.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.116+0000 7f41d0e44640 1 -- 172.21.15.121:0/2000027748 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f41cc06cd40 con 0x7f41cc11e120 2024-09-17T18:27:19.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.118+0000 7f41d0e44640 1 -- 172.21.15.121:0/2000027748 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41cc11e120 msgr2=0x7f41cc1379f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:19.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.118+0000 7f41d0e44640 1 --2- 172.21.15.121:0/2000027748 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41cc11e120 0x7f41cc1379f0 secure :-1 s=READY pgs=7436 cs=0 l=1 rev1=1 crypto rx=0x7f41cc121380 tx=0x7f41bc003760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:19.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.118+0000 7f41c1ffb640 10 client.44505.objecter ms_handle_connect 0x7f41cc11e120 2024-09-17T18:27:19.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.118+0000 7f41c1ffb640 10 client.44505.objecter resend_mon_ops 2024-09-17T18:27:19.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.119+0000 7f41c1ffb640 1 -- 172.21.15.121:0/2000027748 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41bc018070 con 0x7f41cc11e120 2024-09-17T18:27:19.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.119+0000 7f41c1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:19.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.119+0000 7f41c1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:27:19.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.119+0000 7f41c1ffb640 10 monclient: dump: 2024-09-17T18:27:19.129 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:19.129 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:19.129 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:19.129 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:19.129 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:19.129 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:19.129 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:19.129 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:19.129 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:19.129 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:19.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.119+0000 7f41c1ffb640 1 -- 172.21.15.121:0/2000027748 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f41bc020e50 con 0x7f41cc11e120 2024-09-17T18:27:19.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.119+0000 7f41c1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:27:19.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.119+0000 7f41c1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:27:19.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.119+0000 7f41c1ffb640 1 -- 172.21.15.121:0/2000027748 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41cc11e120 msgr2=0x7f41cc1379f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:19.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.119+0000 7f41c1ffb640 1 --2- 172.21.15.121:0/2000027748 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41cc11e120 0x7f41cc1379f0 secure :-1 s=READY pgs=7436 cs=0 l=1 rev1=1 crypto rx=0x7f41cc121380 tx=0x7f41bc003760 comp rx=0 tx=0).stop 2024-09-17T18:27:19.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.119+0000 7f41c1ffb640 1 -- 172.21.15.121:0/2000027748 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41cc11e120 msgr2=0x7f41cc1379f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:27:19.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.119+0000 7f41c1ffb640 1 --2- 172.21.15.121:0/2000027748 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41cc11e120 0x7f41cc1379f0 secure :-1 s=CLOSED pgs=7436 cs=0 l=1 rev1=1 crypto rx=0x7f41cc121380 tx=0x7f41bc003760 comp rx=0 tx=0).stop 2024-09-17T18:27:19.130 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:27:19.130 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:27:19.130 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:27:19.130 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:27:19.130 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 32, 2024-09-17T18:27:19.130 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 512065, 2024-09-17T18:27:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:27:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:27:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:27:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:27:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:27:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 7, 2024-09-17T18:27:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:27:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:27:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:27:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:27:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:27:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:27:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:27:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:27:19.132 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:27:19.132 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:27:19.132 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:27:19.132 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:27:19.132 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:27:19.132 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:27:19.132 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:27:19.132 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:27:19.132 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:27:19.132 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:27:19.132 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:27:19.132 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:27:19.132 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:27:19.132 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:27:19.133 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:27:19.133 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:27:19.133 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:27:19.133 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:27:19.133 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:27:19.133 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:27:19.133 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:19.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:19.133 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:27:19.133 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:27:19.133 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:27:19.133 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:27:19.133 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:27:19.133 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:27:19.134 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:27:19.134 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:27:19.134 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:27:19.134 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:27:19.134 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:27:19.134 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:27:19.134 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:27:19.134 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:27:19.134 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:27:19.134 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:27:19.134 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:27:19.134 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:27:19.134 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:27:19.135 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:27:19.135 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:27:19.135 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:27:19.135 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:27:19.135 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:27:19.135 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:27:19.135 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:27:19.135 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:27:19.135 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:27:19.135 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:27:19.135 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:27:19.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:19.135 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:27:19.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:19.135 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:27:19.136 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:27:19.136 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:27:19.136 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:27:19.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:19.136 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:27:19.136 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:27:19.136 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:19.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:19.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:19.136 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:27:19.136 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:27:19.136 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:19.136 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:19.136 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:19.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:19.137 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:27:19.137 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:27:19.137 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:27:19.137 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:27:19.137 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:27:19.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:19.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:19.137 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:27:19.137 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:27:19.137 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:27:19.137 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:27:19.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:19.137 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:27:19.138 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:27:19.138 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:19.138 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:19.138 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:19.138 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:27:19.138 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:27:19.138 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:19.138 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:19.138 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:19.138 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:19.138 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:27:19.138 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:27:19.138 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:27:19.139 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:27:19.139 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:27:19.139 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:19.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:19.139 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:27:19.139 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:27:19.139 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:27:19.139 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:27:19.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:19.139 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:27:19.139 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:27:19.139 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:19.139 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:19.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:19.139 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:27:19.140 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:27:19.140 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:19.140 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:19.140 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:19.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:19.140 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:27:19.140 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:27:19.140 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:27:19.140 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:27:19.140 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:27:19.140 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:19.140 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:19.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:19.141 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:27:19.141 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:27:19.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:19.141 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:27:19.141 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:27:19.141 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:27:19.141 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:19.141 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:27:19.141 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:27:19.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:19.141 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:27:19.141 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:27:19.141 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 4 2024-09-17T18:27:19.141 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:19.142 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:27:19.142 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:27:19.142 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:19.142 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:27:19.142 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:27:19.142 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 6 2024-09-17T18:27:19.142 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:19.142 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:27:19.142 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": [ 2024-09-17T18:27:19.142 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:19.142 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:27:19.142 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:27:19.142 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:27:19.142 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:19.142 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:19.143 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:19.143 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:27:19.143 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:27:19.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.120+0000 7f41d38a9640 10 librados: watch_flush enter 2024-09-17T18:27:19.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.120+0000 7f41d38a9640 10 librados: watch_flush exit 2024-09-17T18:27:19.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.120+0000 7f41d38a9640 20 client.44505.objecter shutdown clearing up homeless session... 2024-09-17T18:27:19.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.120+0000 7f41d38a9640 10 client.44505.objecter successfully canceled tick 2024-09-17T18:27:19.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.120+0000 7f41d38a9640 1 -- 172.21.15.121:0/2000027748 >> v2:172.21.15.173:6812/2423786037 conn(0x7f41b00401c0 msgr2=0x7f41b0042680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:19.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.120+0000 7f41d38a9640 1 --2- 172.21.15.121:0/2000027748 >> v2:172.21.15.173:6812/2423786037 conn(0x7f41b00401c0 0x7f41b0042680 secure :-1 s=READY pgs=4002 cs=0 l=1 rev1=1 crypto rx=0x7f41c4036000 tx=0x7f41c4031000 comp rx=0 tx=0).stop 2024-09-17T18:27:19.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.120+0000 7f41d38a9640 10 monclient: shutdown 2024-09-17T18:27:19.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.120+0000 7f41d38a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:27:19.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.120+0000 7f41d38a9640 1 -- 172.21.15.121:0/2000027748 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41cc09e550 msgr2=0x7f41cc11f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:19.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.120+0000 7f41d38a9640 1 --2- 172.21.15.121:0/2000027748 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41cc09e550 0x7f41cc11f970 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f41c800c460 tx=0x7f41c800c920 comp rx=0 tx=0).stop 2024-09-17T18:27:19.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.120+0000 7f41d1e46640 1 -- 172.21.15.121:0/2000027748 reap_dead start 2024-09-17T18:27:19.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.120+0000 7f41d1e46640 1 -- 172.21.15.121:0/2000027748 reap_dead start 2024-09-17T18:27:19.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.120+0000 7f41d38a9640 1 -- 172.21.15.121:0/2000027748 shutdown_connections 2024-09-17T18:27:19.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.120+0000 7f41d38a9640 1 -- 172.21.15.121:0/2000027748 >> 172.21.15.121:0/2000027748 conn(0x7f41cc0a7830 msgr2=0x7f41cc0a6530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:27:19.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.120+0000 7f41d38a9640 1 -- 172.21.15.121:0/2000027748 shutdown_connections 2024-09-17T18:27:19.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.120+0000 7f41d38a9640 1 -- 172.21.15.121:0/2000027748 wait complete. 2024-09-17T18:27:19.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:19.120+0000 7f41d38a9640 1 librados: shutdown 2024-09-17T18:27:19.146 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:27:19.146 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-17T18:27:19.146 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2024-09-17T18:27:19.146 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-17T18:27:19.146 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-17T18:27:19.147 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-17T18:27:19.147 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T18:27:19.167 INFO:tasks.ceph.mon.b:Stopped 2024-09-17T18:27:19.167 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-17T18:27:19.167 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-17T18:27:19.167 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-17T18:27:34.168 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-17T18:27:34.168 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-17T18:27:34.169 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-17T18:27:34.169 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:27:34.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.275+0000 7f534b87f640 10 monclient: get_monmap_and_config 2024-09-17T18:27:34.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.275+0000 7f534b87f640 10 monclient: build_initial_monmap 2024-09-17T18:27:34.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.275+0000 7f534b87f640 10 monclient: monmap: 2024-09-17T18:27:34.277 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:27:34.277 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:34.278 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:27:34.276186+0000 2024-09-17T18:27:34.278 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:27:34.276186+0000 2024-09-17T18:27:34.278 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:27:34.278 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:34.278 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:27:34.278 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:27:34.278 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:27:34.278 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:27:34.278 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:27:34.278 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:34.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.276+0000 7f534b87f640 1 Processor -- start 2024-09-17T18:27:34.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.276+0000 7f534b87f640 1 -- start start 2024-09-17T18:27:34.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.276+0000 7f534b87f640 10 monclient: init 2024-09-17T18:27:34.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.276+0000 7f534b87f640 10 monclient: _reopen_session rank -1 2024-09-17T18:27:34.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.276+0000 7f534b87f640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-17T18:27:34.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.276+0000 7f534b87f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53440a33f0 0x7f534409e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:34.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.276+0000 7f534b87f640 10 monclient(hunting): picked mon.noname-b con 0x7f53440a33f0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:27:34.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.276+0000 7f534b87f640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f534409da70 0x7f534409ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:34.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.276+0000 7f534b87f640 10 monclient(hunting): picked mon.noname-c con 0x7f534409da70 addr v2:172.21.15.173:3300/0 2024-09-17T18:27:34.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.276+0000 7f534b87f640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f534409efd0 0x7f534413b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:34.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.276+0000 7f534b87f640 10 monclient(hunting): picked mon.noname-a con 0x7f534409efd0 addr v2:172.21.15.121:3300/0 2024-09-17T18:27:34.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.276+0000 7f534b87f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:34.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.276+0000 7f534b87f640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f5344074550 con 0x7f534409efd0 2024-09-17T18:27:34.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.276+0000 7f534b87f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:34.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.276+0000 7f534b87f640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f53440a37d0 con 0x7f534409da70 2024-09-17T18:27:34.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.276+0000 7f534b87f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:34.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.276+0000 7f534b87f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5344074b40 con 0x7f53440a33f0 2024-09-17T18:27:34.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.276+0000 7f534b87f640 10 monclient(hunting): _renew_subs 2024-09-17T18:27:34.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.276+0000 7f534b87f640 10 monclient(hunting): authenticate will time out at 1257.992921s 2024-09-17T18:27:34.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.277+0000 7f5348e1a640 1 -- >> v2:172.21.15.173:3300/0 conn(0x7f534409da70 msgr2=0x7f534409ea90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.173:3300/0 2024-09-17T18:27:34.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.277+0000 7f5348e1a640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f534409da70 0x7f534409ea90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:27:34.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.277+0000 7f534961b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53440a33f0 0x7f534409e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:34.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.277+0000 7f5349e1c640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f534409efd0 0x7f534413b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:34.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.277+0000 7f534961b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53440a33f0 0x7f534409e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:36294/0 (socket says 172.21.15.121:36294) 2024-09-17T18:27:34.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.277+0000 7f534961b640 1 -- 172.21.15.121:0/3808690080 learned_addr learned my addr 172.21.15.121:0/3808690080 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:27:34.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.277+0000 7f5349e1c640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f534409efd0 0x7f534413b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41456/0 (socket says 172.21.15.121:41456) 2024-09-17T18:27:34.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.277+0000 7f534961b640 10 monclient(hunting): get_auth_request con 0x7f53440a33f0 auth_method 0 2024-09-17T18:27:34.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.277+0000 7f534961b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:34.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.277+0000 7f534961b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:34.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.277+0000 7f534961b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:34.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.277+0000 7f5349e1c640 10 monclient(hunting): get_auth_request con 0x7f534409efd0 auth_method 0 2024-09-17T18:27:34.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.277+0000 7f5349e1c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:34.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.277+0000 7f5349e1c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:34.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.277+0000 7f5349e1c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:34.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.277+0000 7f5349e1c640 1 -- 172.21.15.121:0/3808690080 >> v2:172.21.15.121:3300/0 conn(0x7f534409efd0 msgr2=0x7f534413b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T18:27:34.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.277+0000 7f534961b640 1 -- 172.21.15.121:0/3808690080 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53440a33f0 msgr2=0x7f534409e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:27:34.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.277+0000 7f5349e1c640 1 -- 172.21.15.121:0/3808690080 >> v2:172.21.15.121:3300/0 conn(0x7f534409efd0 msgr2=0x7f534413b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:27:34.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.277+0000 7f534961b640 1 -- 172.21.15.121:0/3808690080 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53440a33f0 msgr2=0x7f534409e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:27:34.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.277+0000 7f5349e1c640 1 --2- 172.21.15.121:0/3808690080 >> v2:172.21.15.121:3300/0 conn(0x7f534409efd0 0x7f534413b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:27:34.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.277+0000 7f534961b640 1 --2- 172.21.15.121:0/3808690080 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53440a33f0 0x7f534409e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:27:34.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.277+0000 7f5349e1c640 1 --2- 172.21.15.121:0/3808690080 >> v2:172.21.15.121:3300/0 conn(0x7f534409efd0 0x7f534413b850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:27:34.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.277+0000 7f534961b640 1 --2- 172.21.15.121:0/3808690080 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53440a33f0 0x7f534409e550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:27:34.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.478+0000 7f5348e1a640 1 -- 172.21.15.121:0/3808690080 >> v2:172.21.15.173:3300/0 conn(0x7f534409da70 msgr2=0x7f534409ea90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.173:3300/0 2024-09-17T18:27:34.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.478+0000 7f5348e1a640 1 --2- 172.21.15.121:0/3808690080 >> v2:172.21.15.173:3300/0 conn(0x7f534409da70 0x7f534409ea90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T18:27:34.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.478+0000 7f5349e1c640 1 --2- 172.21.15.121:0/3808690080 >> v2:172.21.15.121:3300/0 conn(0x7f534409efd0 0x7f534413b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:34.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.478+0000 7f534961b640 1 --2- 172.21.15.121:0/3808690080 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53440a33f0 0x7f534409e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:34.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.478+0000 7f5349e1c640 10 monclient(hunting): get_auth_request con 0x7f534409efd0 auth_method 0 2024-09-17T18:27:34.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.478+0000 7f5349e1c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:34.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.478+0000 7f5349e1c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:34.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.478+0000 7f5349e1c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:27:34.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.478+0000 7f534961b640 10 monclient(hunting): get_auth_request con 0x7f53440a33f0 auth_method 0 2024-09-17T18:27:34.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.478+0000 7f534961b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:34.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.478+0000 7f534961b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:34.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.478+0000 7f534961b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:27:34.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.479+0000 7f5349e1c640 1 -- 172.21.15.121:0/3808690080 >> v2:172.21.15.121:3300/0 conn(0x7f534409efd0 msgr2=0x7f534413b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T18:27:34.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.479+0000 7f5349e1c640 1 -- 172.21.15.121:0/3808690080 >> v2:172.21.15.121:3300/0 conn(0x7f534409efd0 msgr2=0x7f534413b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:27:34.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.479+0000 7f5349e1c640 1 --2- 172.21.15.121:0/3808690080 >> v2:172.21.15.121:3300/0 conn(0x7f534409efd0 0x7f534413b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:27:34.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.479+0000 7f534961b640 1 -- 172.21.15.121:0/3808690080 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53440a33f0 msgr2=0x7f534409e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:27:34.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.479+0000 7f534961b640 1 -- 172.21.15.121:0/3808690080 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53440a33f0 msgr2=0x7f534409e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:27:34.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.479+0000 7f534961b640 1 --2- 172.21.15.121:0/3808690080 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53440a33f0 0x7f534409e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:27:34.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.479+0000 7f5349e1c640 1 --2- 172.21.15.121:0/3808690080 >> v2:172.21.15.121:3300/0 conn(0x7f534409efd0 0x7f534413b850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T18:27:34.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.479+0000 7f534961b640 1 --2- 172.21.15.121:0/3808690080 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53440a33f0 0x7f534409e550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T18:27:34.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.879+0000 7f5348e1a640 1 -- 172.21.15.121:0/3808690080 >> v2:172.21.15.173:3300/0 conn(0x7f534409da70 msgr2=0x7f534409ea90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.173:3300/0 2024-09-17T18:27:34.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.879+0000 7f5348e1a640 1 --2- 172.21.15.121:0/3808690080 >> v2:172.21.15.173:3300/0 conn(0x7f534409da70 0x7f534409ea90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T18:27:34.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.880+0000 7f534961b640 1 --2- 172.21.15.121:0/3808690080 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53440a33f0 0x7f534409e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:34.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.881+0000 7f5349e1c640 1 --2- 172.21.15.121:0/3808690080 >> v2:172.21.15.121:3300/0 conn(0x7f534409efd0 0x7f534413b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:34.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.881+0000 7f534961b640 10 monclient(hunting): get_auth_request con 0x7f53440a33f0 auth_method 0 2024-09-17T18:27:34.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.881+0000 7f534961b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:34.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.881+0000 7f534961b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:34.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.881+0000 7f534961b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:27:34.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.881+0000 7f5349e1c640 10 monclient(hunting): get_auth_request con 0x7f534409efd0 auth_method 0 2024-09-17T18:27:34.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.881+0000 7f5349e1c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:34.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.881+0000 7f5349e1c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:34.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.881+0000 7f5349e1c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:27:34.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.881+0000 7f534961b640 1 -- 172.21.15.121:0/3808690080 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53440a33f0 msgr2=0x7f534409e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:27:34.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.881+0000 7f534961b640 1 -- 172.21.15.121:0/3808690080 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53440a33f0 msgr2=0x7f534409e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:27:34.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.881+0000 7f534961b640 1 --2- 172.21.15.121:0/3808690080 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53440a33f0 0x7f534409e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:27:34.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.881+0000 7f534961b640 1 --2- 172.21.15.121:0/3808690080 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53440a33f0 0x7f534409e550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T18:27:34.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.881+0000 7f5349e1c640 1 -- 172.21.15.121:0/3808690080 >> v2:172.21.15.121:3300/0 conn(0x7f534409efd0 msgr2=0x7f534413b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T18:27:34.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.881+0000 7f5349e1c640 1 -- 172.21.15.121:0/3808690080 >> v2:172.21.15.121:3300/0 conn(0x7f534409efd0 msgr2=0x7f534413b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:27:34.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.881+0000 7f5349e1c640 1 --2- 172.21.15.121:0/3808690080 >> v2:172.21.15.121:3300/0 conn(0x7f534409efd0 0x7f534413b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:27:34.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:34.881+0000 7f5349e1c640 1 --2- 172.21.15.121:0/3808690080 >> v2:172.21.15.121:3300/0 conn(0x7f534409efd0 0x7f534413b850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T18:27:35.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:35.680+0000 7f5348e1a640 1 -- 172.21.15.121:0/3808690080 >> v2:172.21.15.173:3300/0 conn(0x7f534409da70 msgr2=0x7f534409ea90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.173:3300/0 2024-09-17T18:27:35.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:35.680+0000 7f5348e1a640 1 --2- 172.21.15.121:0/3808690080 >> v2:172.21.15.173:3300/0 conn(0x7f534409da70 0x7f534409ea90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T18:27:35.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:35.682+0000 7f534961b640 1 --2- 172.21.15.121:0/3808690080 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53440a33f0 0x7f534409e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:35.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:35.682+0000 7f5349e1c640 1 --2- 172.21.15.121:0/3808690080 >> v2:172.21.15.121:3300/0 conn(0x7f534409efd0 0x7f534413b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:35.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:35.682+0000 7f534961b640 10 monclient(hunting): get_auth_request con 0x7f53440a33f0 auth_method 0 2024-09-17T18:27:35.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:35.682+0000 7f534961b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:35.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:35.682+0000 7f534961b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:35.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:35.682+0000 7f534961b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:27:35.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:35.682+0000 7f5349e1c640 10 monclient(hunting): get_auth_request con 0x7f534409efd0 auth_method 0 2024-09-17T18:27:35.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:35.682+0000 7f5349e1c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:35.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:35.682+0000 7f5349e1c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:35.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:35.682+0000 7f5349e1c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:27:35.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:35.682+0000 7f534961b640 1 -- 172.21.15.121:0/3808690080 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53440a33f0 msgr2=0x7f534409e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:27:35.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:35.682+0000 7f534961b640 1 -- 172.21.15.121:0/3808690080 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53440a33f0 msgr2=0x7f534409e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:27:35.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:35.682+0000 7f534961b640 1 --2- 172.21.15.121:0/3808690080 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53440a33f0 0x7f534409e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:27:35.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:35.682+0000 7f5349e1c640 1 -- 172.21.15.121:0/3808690080 >> v2:172.21.15.121:3300/0 conn(0x7f534409efd0 msgr2=0x7f534413b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T18:27:35.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:35.682+0000 7f5349e1c640 1 -- 172.21.15.121:0/3808690080 >> v2:172.21.15.121:3300/0 conn(0x7f534409efd0 msgr2=0x7f534413b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:27:35.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:35.682+0000 7f534961b640 1 --2- 172.21.15.121:0/3808690080 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53440a33f0 0x7f534409e550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T18:27:35.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:35.682+0000 7f5349e1c640 1 --2- 172.21.15.121:0/3808690080 >> v2:172.21.15.121:3300/0 conn(0x7f534409efd0 0x7f534413b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:27:35.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:35.682+0000 7f5349e1c640 1 --2- 172.21.15.121:0/3808690080 >> v2:172.21.15.121:3300/0 conn(0x7f534409efd0 0x7f534413b850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T18:27:37.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:37.276+0000 7f533affd640 10 monclient(hunting): tick 2024-09-17T18:27:37.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:37.276+0000 7f533affd640 10 monclient(hunting): _check_auth_tickets 2024-09-17T18:27:37.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:37.276+0000 7f533affd640 1 monclient(hunting): continuing hunt 2024-09-17T18:27:37.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:37.276+0000 7f533affd640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T18:27:37.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:37.276+0000 7f533affd640 1 -- 172.21.15.121:0/3808690080 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53440a33f0 msgr2=0x7f534409e550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:27:37.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:37.276+0000 7f533affd640 1 --2- 172.21.15.121:0/3808690080 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53440a33f0 0x7f534409e550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:37.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:37.276+0000 7f533affd640 1 -- 172.21.15.121:0/3808690080 >> v2:172.21.15.173:3300/0 conn(0x7f534409da70 msgr2=0x7f534409ea90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:27:37.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:37.276+0000 7f533affd640 1 --2- 172.21.15.121:0/3808690080 >> v2:172.21.15.173:3300/0 conn(0x7f534409da70 0x7f534409ea90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:37.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:37.276+0000 7f533affd640 1 -- 172.21.15.121:0/3808690080 >> v2:172.21.15.121:3300/0 conn(0x7f534409efd0 msgr2=0x7f534413b850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:27:37.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:37.276+0000 7f533affd640 1 --2- 172.21.15.121:0/3808690080 >> v2:172.21.15.121:3300/0 conn(0x7f534409efd0 0x7f534413b850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:37.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:37.276+0000 7f533affd640 10 monclient: _add_conns ranks=[1,0] 2024-09-17T18:27:37.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:37.276+0000 7f533affd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f53280036c0 addr v1:172.21.15.173:6789/0 2024-09-17T18:27:37.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:37.276+0000 7f533affd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f53280073f0 addr v1:172.21.15.121:6789/0 2024-09-17T18:27:37.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:37.276+0000 7f533affd640 1 -- 172.21.15.121:0/3808690080 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53440a13c0 con 0x7f53280073f0 2024-09-17T18:27:37.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:37.276+0000 7f533affd640 1 -- 172.21.15.121:0/3808690080 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f532800c130 con 0x7f53280036c0 2024-09-17T18:27:37.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:37.276+0000 7f533affd640 10 monclient(hunting): _renew_subs 2024-09-17T18:27:37.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:37.276+0000 7f534961b640 1 -- 172.21.15.121:0/3808690080 >> v1:172.21.15.173:6789/0 conn(0x7f53280036c0 legacy=0x7f5328005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.173:6789/0 2024-09-17T18:27:37.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:37.278+0000 7f5348e1a640 1 -- 172.21.15.121:0/3808690080 >> v1:172.21.15.121:6789/0 conn(0x7f53280073f0 legacy=0x7f53280098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:37.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:37.477+0000 7f534961b640 1 -- 172.21.15.121:0/3808690080 >> v1:172.21.15.173:6789/0 conn(0x7f53280036c0 legacy=0x7f5328005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.173:6789/0 2024-09-17T18:27:37.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:37.878+0000 7f534961b640 1 -- 172.21.15.121:0/3808690080 >> v1:172.21.15.173:6789/0 conn(0x7f53280036c0 legacy=0x7f5328005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.173:6789/0 2024-09-17T18:27:38.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:38.679+0000 7f534961b640 1 -- 172.21.15.121:0/3808690080 >> v1:172.21.15.173:6789/0 conn(0x7f53280036c0 legacy=0x7f5328005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.173:6789/0 2024-09-17T18:27:39.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.190+0000 7f533bfff640 1 -- 172.21.15.121:0/3808690080 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3141967394 0 0) 0x7f53440a13c0 con 0x7f53280073f0 2024-09-17T18:27:39.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.190+0000 7f533bfff640 10 monclient(hunting): _negotiate 2024-09-17T18:27:39.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.190+0000 7f533bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:39.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.190+0000 7f533bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:39.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.190+0000 7f533bfff640 10 monclient(hunting): my global_id is 44508 2024-09-17T18:27:39.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.190+0000 7f533bfff640 1 -- 172.21.15.121:0/3808690080 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5330003520 con 0x7f53280073f0 2024-09-17T18:27:39.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.191+0000 7f533bfff640 1 -- 172.21.15.121:0/3808690080 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2907753750 0 0) 0x7f5330003520 con 0x7f53280073f0 2024-09-17T18:27:39.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.191+0000 7f533bfff640 1 -- 172.21.15.121:0/3808690080 >> v1:172.21.15.173:6789/0 conn(0x7f53280036c0 legacy=0x7f5328005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:27:39.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.191+0000 7f533bfff640 10 monclient: _finish_hunting 0 2024-09-17T18:27:39.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.191+0000 7f533bfff640 1 monclient: found mon.noname-a-legacy 2024-09-17T18:27:39.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.191+0000 7f533bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:27:39.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.191+0000 7f533bfff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.121:6789/0 2024-09-17T18:27:39.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.191+0000 7f533bfff640 1 -- 172.21.15.121:0/3808690080 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f532800d330 con 0x7f53280073f0 2024-09-17T18:27:39.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.191+0000 7f533bfff640 10 monclient: _finish_auth 0 2024-09-17T18:27:39.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.191+0000 7f533bfff640 10 monclient: _check_auth_tickets 2024-09-17T18:27:39.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.191+0000 7f533bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:27:39.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.191+0000 7f533bfff640 1 -- 172.21.15.121:0/3808690080 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4002019445 0 0) 0x7f5334004500 con 0x7f53280073f0 2024-09-17T18:27:39.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.191+0000 7f533bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:39.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.191+0000 7f533bfff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T18:27:39.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.191+0000 7f533bfff640 10 monclient: dump: 2024-09-17T18:27:39.196 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:39.196 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:39.196 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:39.196 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:39.196 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:39.196 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:39.196 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:39.196 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:39.196 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:39.196 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:39.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.191+0000 7f533bfff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] but i'm connected to v1:172.21.15.121:6789/0, reconnecting 2024-09-17T18:27:39.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.191+0000 7f533bfff640 10 monclient: _reopen_session rank -1 2024-09-17T18:27:39.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.191+0000 7f533bfff640 1 -- 172.21.15.121:0/3808690080 >> v1:172.21.15.121:6789/0 conn(0x7f53280073f0 legacy=0x7f53280098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:39.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.191+0000 7f5349e1c640 1 -- 172.21.15.121:0/3808690080 reap_dead start 2024-09-17T18:27:39.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.192+0000 7f533bfff640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T18:27:39.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.192+0000 7f533bfff640 1 --2- 172.21.15.121:0/3808690080 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53300039e0 0x7f5330003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:39.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.192+0000 7f533bfff640 10 monclient(hunting): picked mon.c con 0x7f53300039e0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:27:39.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.192+0000 7f533bfff640 1 --2- 172.21.15.121:0/3808690080 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5330004800 0x7f5330004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:39.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.192+0000 7f533bfff640 10 monclient(hunting): picked mon.b con 0x7f5330004800 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:27:39.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.192+0000 7f533bfff640 1 --2- 172.21.15.121:0/3808690080 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53300053c0 0x7f533000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:39.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.192+0000 7f533bfff640 10 monclient(hunting): picked mon.a con 0x7f53300053c0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:27:39.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.192+0000 7f533bfff640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:39.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.192+0000 7f533bfff640 1 -- 172.21.15.121:0/3808690080 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f533000c220 con 0x7f53300053c0 2024-09-17T18:27:39.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.192+0000 7f533bfff640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:39.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.192+0000 7f533bfff640 1 -- 172.21.15.121:0/3808690080 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f533000c3d0 con 0x7f5330004800 2024-09-17T18:27:39.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.192+0000 7f533bfff640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:39.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.192+0000 7f533bfff640 1 -- 172.21.15.121:0/3808690080 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f533000c5d0 con 0x7f53300039e0 2024-09-17T18:27:39.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.193+0000 7f534961b640 1 --2- 172.21.15.121:0/3808690080 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53300039e0 0x7f5330003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:39.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.193+0000 7f5349e1c640 1 --2- 172.21.15.121:0/3808690080 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53300053c0 0x7f533000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:39.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.193+0000 7f533bfff640 10 monclient(hunting): _renew_subs 2024-09-17T18:27:39.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.193+0000 7f533bfff640 10 monclient(hunting): _finish_auth 0 2024-09-17T18:27:39.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.193+0000 7f5349e1c640 10 monclient(hunting): get_auth_request con 0x7f53300053c0 auth_method 0 2024-09-17T18:27:39.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.193+0000 7f5349e1c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:39.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.193+0000 7f5349e1c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:39.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.193+0000 7f5349e1c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:27:39.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.193+0000 7f534961b640 10 monclient(hunting): get_auth_request con 0x7f53300039e0 auth_method 0 2024-09-17T18:27:39.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.193+0000 7f534961b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:39.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.193+0000 7f534961b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:39.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.193+0000 7f534961b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:27:39.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.193+0000 7f534961b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:39.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.193+0000 7f534961b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:39.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.193+0000 7f534961b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:27:39.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.193+0000 7f5348e1a640 1 -- 172.21.15.121:0/3808690080 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5330004800 msgr2=0x7f5330004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.173:3300/0 2024-09-17T18:27:39.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.193+0000 7f5349e1c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:39.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.193+0000 7f5349e1c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:39.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.193+0000 7f5348e1a640 1 --2- 172.21.15.121:0/3808690080 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5330004800 0x7f5330004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:27:39.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.193+0000 7f5349e1c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:27:39.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.193+0000 7f534961b640 10 monclient(hunting): handle_auth_done global_id 44508 payload 293 2024-09-17T18:27:39.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.193+0000 7f534961b640 1 -- 172.21.15.121:0/3808690080 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5330004800 msgr2=0x7f5330004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:27:39.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.193+0000 7f534961b640 1 --2- 172.21.15.121:0/3808690080 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5330004800 0x7f5330004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:39.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.193+0000 7f534961b640 1 -- 172.21.15.121:0/3808690080 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53300053c0 msgr2=0x7f533000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:27:39.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.196+0000 7f534961b640 1 --2- 172.21.15.121:0/3808690080 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53300053c0 0x7f533000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:39.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.196+0000 7f534961b640 10 monclient: _finish_hunting 0 2024-09-17T18:27:39.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.196+0000 7f534961b640 1 monclient: found mon.c 2024-09-17T18:27:39.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.196+0000 7f534961b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:27:39.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.196+0000 7f534961b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:27:39.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.196+0000 7f534961b640 1 -- 172.21.15.121:0/3808690080 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5330003520 con 0x7f53300039e0 2024-09-17T18:27:39.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.196+0000 7f534961b640 10 monclient: _finish_auth 0 2024-09-17T18:27:39.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.196+0000 7f534961b640 10 monclient: _check_auth_tickets 2024-09-17T18:27:39.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.196+0000 7f534961b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:27:39.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.197+0000 7f5349e1c640 1 --2- 172.21.15.121:0/3808690080 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53300053c0 0x7f533000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:27:39.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.197+0000 7f534b87f640 5 monclient: authenticate success, global_id 44508 2024-09-17T18:27:39.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.197+0000 7f534b87f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:27:39.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.199+0000 7f534961b640 1 -- 172.21.15.121:0/3808690080 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53300039e0 msgr2=0x7f5330003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:39.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.199+0000 7f534961b640 1 --2- 172.21.15.121:0/3808690080 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53300039e0 0x7f5330003dc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5340016780 tx=0x7f5330005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c65543d1f65d6ca2 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:39.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.199+0000 7f533bfff640 1 -- 172.21.15.121:0/3808690080 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5340038070 con 0x7f53300039e0 2024-09-17T18:27:39.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.199+0000 7f533bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:39.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.199+0000 7f533bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:27:39.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.199+0000 7f533bfff640 10 monclient: dump: 2024-09-17T18:27:39.206 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:39.206 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:39.207 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:39.207 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:39.207 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:39.207 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:39.207 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:39.207 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:39.207 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:39.207 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:39.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.199+0000 7f534b87f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:27:39.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.199+0000 7f533bfff640 1 -- 172.21.15.121:0/3808690080 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5340030480 con 0x7f53300039e0 2024-09-17T18:27:39.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.199+0000 7f533bfff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:27:39.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.199+0000 7f534b87f640 10 monclient: get_monmap_and_config success 2024-09-17T18:27:39.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.199+0000 7f533bfff640 1 -- 172.21.15.121:0/3808690080 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f534000f040 con 0x7f53300039e0 2024-09-17T18:27:39.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.199+0000 7f534b87f640 4 set_mon_vals no callback set 2024-09-17T18:27:39.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.199+0000 7f533bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:39.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.199+0000 7f533bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:27:39.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.199+0000 7f533bfff640 10 monclient: dump: 2024-09-17T18:27:39.208 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:39.208 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:39.208 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:39.208 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:39.208 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:39.208 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:39.208 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:39.208 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:39.209 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:39.209 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:39.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.200+0000 7f534b87f640 10 monclient: shutdown 2024-09-17T18:27:39.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.200+0000 7f534b87f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:27:39.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.200+0000 7f534b87f640 1 -- 172.21.15.121:0/3808690080 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53300039e0 msgr2=0x7f5330003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:39.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.200+0000 7f534b87f640 1 --2- 172.21.15.121:0/3808690080 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53300039e0 0x7f5330003dc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5340016780 tx=0x7f5330005830 comp rx=0 tx=0).stop 2024-09-17T18:27:39.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.201+0000 7f534b87f640 1 -- 172.21.15.121:0/3808690080 shutdown_connections 2024-09-17T18:27:39.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.201+0000 7f534b87f640 1 --2- 172.21.15.121:0/3808690080 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5330004800 0x7f5330004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:39.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.201+0000 7f534b87f640 1 --2- 172.21.15.121:0/3808690080 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53300053c0 0x7f533000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:39.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.201+0000 7f534b87f640 1 --2- 172.21.15.121:0/3808690080 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53300039e0 0x7f5330003dc0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:39.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.201+0000 7f534b87f640 1 -- 172.21.15.121:0/3808690080 >> 172.21.15.121:0/3808690080 conn(0x7f53440a3b50 msgr2=0x7f53440a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:27:39.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.201+0000 7f534b87f640 1 -- 172.21.15.121:0/3808690080 shutdown_connections 2024-09-17T18:27:39.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.201+0000 7f534b87f640 1 -- 172.21.15.121:0/3808690080 wait complete. 2024-09-17T18:27:39.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.202+0000 7f534b87f640 10 monclient: build_initial_monmap 2024-09-17T18:27:39.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.202+0000 7f534b87f640 10 monclient: monmap: 2024-09-17T18:27:39.210 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:27:39.210 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:27:39.210 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:27:39.210 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:27:39.210 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:27:39.210 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:39.210 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:27:39.210 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:27:39.210 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:27:39.210 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:39.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.203+0000 7f534b87f640 1 librados: starting msgr at 2024-09-17T18:27:39.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.203+0000 7f534b87f640 1 librados: starting objecter 2024-09-17T18:27:39.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.203+0000 7f534b87f640 1 Processor -- start 2024-09-17T18:27:39.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.203+0000 7f534b87f640 1 -- start start 2024-09-17T18:27:39.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.203+0000 7f534b87f640 1 librados: setting wanted keys 2024-09-17T18:27:39.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.203+0000 7f534b87f640 1 librados: calling monclient init 2024-09-17T18:27:39.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.203+0000 7f534b87f640 10 monclient: init 2024-09-17T18:27:39.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.204+0000 7f534b87f640 10 monclient: _reopen_session rank -1 2024-09-17T18:27:39.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.204+0000 7f534b87f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T18:27:39.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.204+0000 7f534b87f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53300039e0 0x7f53441bc630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:39.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.204+0000 7f534b87f640 10 monclient(hunting): picked mon.noname-b con 0x7f53300039e0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:27:39.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.204+0000 7f534b87f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5330004800 0x7f53441bcb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:39.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.204+0000 7f534961b640 1 -- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53300039e0 msgr2=0x7f53441bc630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.173:3300/0 2024-09-17T18:27:39.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.204+0000 7f534961b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53300039e0 0x7f53441bc630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:27:39.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.204+0000 7f534b87f640 10 monclient(hunting): picked mon.noname-c con 0x7f5330004800 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:27:39.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.204+0000 7f534b87f640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53300053c0 0x7f53441a61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:39.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.204+0000 7f534b87f640 10 monclient(hunting): picked mon.noname-a con 0x7f53300053c0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:27:39.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.204+0000 7f534b87f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:39.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.204+0000 7f534b87f640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f534409e550 con 0x7f53300053c0 2024-09-17T18:27:39.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.204+0000 7f534b87f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:39.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.204+0000 7f534b87f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f534409f7b0 con 0x7f53300039e0 2024-09-17T18:27:39.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.204+0000 7f534b87f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:39.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.204+0000 7f534b87f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f534409e6d0 con 0x7f5330004800 2024-09-17T18:27:39.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.204+0000 7f534b87f640 10 monclient(hunting): _renew_subs 2024-09-17T18:27:39.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.204+0000 7f534b87f640 10 monclient(hunting): authenticate will time out at 1262.921901s 2024-09-17T18:27:39.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.204+0000 7f5348e1a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5330004800 0x7f53441bcb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:39.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.205+0000 7f5348e1a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5330004800 0x7f53441bcb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:54340/0 (socket says 172.21.15.121:54340) 2024-09-17T18:27:39.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.205+0000 7f5348e1a640 1 -- 172.21.15.121:0/1562870597 learned_addr learned my addr 172.21.15.121:0/1562870597 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:27:39.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.205+0000 7f5348e1a640 10 monclient(hunting): get_auth_request con 0x7f5330004800 auth_method 0 2024-09-17T18:27:39.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.205+0000 7f5348e1a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:39.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.205+0000 7f5348e1a640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:39.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.205+0000 7f5348e1a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:39.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.205+0000 7f5348e1a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:39.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.205+0000 7f5348e1a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:39.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.205+0000 7f5348e1a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:39.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.205+0000 7f5349e1c640 1 --2- 172.21.15.121:0/1562870597 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53300053c0 0x7f53441a61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:39.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.205+0000 7f5349e1c640 10 monclient(hunting): get_auth_request con 0x7f53300053c0 auth_method 0 2024-09-17T18:27:39.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.205+0000 7f5349e1c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:39.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.205+0000 7f5349e1c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:39.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.205+0000 7f5349e1c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:39.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.205+0000 7f5348e1a640 10 monclient(hunting): handle_auth_done global_id 64145 payload 824 2024-09-17T18:27:39.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.205+0000 7f5348e1a640 1 -- 172.21.15.121:0/1562870597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53300039e0 msgr2=0x7f53441bc630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:27:39.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.205+0000 7f5348e1a640 1 --2- 172.21.15.121:0/1562870597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53300039e0 0x7f53441bc630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:39.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.205+0000 7f5348e1a640 1 -- 172.21.15.121:0/1562870597 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53300053c0 msgr2=0x7f53441a61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:39.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.205+0000 7f5348e1a640 1 --2- 172.21.15.121:0/1562870597 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53300053c0 0x7f53441a61a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:39.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.205+0000 7f5348e1a640 10 monclient: _finish_hunting 0 2024-09-17T18:27:39.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.205+0000 7f5348e1a640 1 monclient: found mon.noname-c 2024-09-17T18:27:39.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.205+0000 7f5348e1a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:27:39.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.205+0000 7f5348e1a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:27:39.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.205+0000 7f5348e1a640 1 -- 172.21.15.121:0/1562870597 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f534409e240 con 0x7f5330004800 2024-09-17T18:27:39.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.205+0000 7f5349e1c640 1 --2- 172.21.15.121:0/1562870597 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53300053c0 0x7f53441a61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T18:27:39.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.207+0000 7f5348e1a640 1 -- 172.21.15.121:0/1562870597 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5330004800 msgr2=0x7f53441bcb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:39.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.207+0000 7f5348e1a640 1 --2- 172.21.15.121:0/1562870597 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5330004800 0x7f53441bcb70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f533400d900 tx=0x7f533400ddc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:39.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.207+0000 7f5339ffb640 10 client.?.objecter ms_handle_connect 0x7f5330004800 2024-09-17T18:27:39.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.207+0000 7f5339ffb640 10 client.?.objecter resend_mon_ops 2024-09-17T18:27:39.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.208+0000 7f5339ffb640 1 -- 172.21.15.121:0/1562870597 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5334004e00 con 0x7f5330004800 2024-09-17T18:27:39.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.208+0000 7f5339ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:39.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.208+0000 7f5339ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:27:39.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.208+0000 7f5339ffb640 10 monclient: dump: 2024-09-17T18:27:39.219 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:39.219 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:39.219 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:39.219 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:39.219 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:39.219 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:39.219 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:39.219 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:39.219 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:39.220 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:39.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.208+0000 7f5339ffb640 10 monclient: _finish_auth 0 2024-09-17T18:27:39.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.208+0000 7f5339ffb640 10 monclient: _check_auth_tickets 2024-09-17T18:27:39.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.208+0000 7f5339ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:27:39.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.208+0000 7f534b87f640 5 monclient: authenticate success, global_id 64145 2024-09-17T18:27:39.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.208+0000 7f5339ffb640 1 -- 172.21.15.121:0/1562870597 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f533400b7b0 con 0x7f5330004800 2024-09-17T18:27:39.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.208+0000 7f5339ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:27:39.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.208+0000 7f5339ffb640 1 -- 172.21.15.121:0/1562870597 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f533400bba0 con 0x7f5330004800 2024-09-17T18:27:39.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.208+0000 7f534b87f640 10 monclient: _renew_subs 2024-09-17T18:27:39.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.208+0000 7f534b87f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:27:39.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.208+0000 7f534b87f640 1 -- 172.21.15.121:0/1562870597 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53441a7c90 con 0x7f5330004800 2024-09-17T18:27:39.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.208+0000 7f5339ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:39.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.208+0000 7f5339ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:27:39.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.208+0000 7f533b7fe640 4 set_mon_vals no callback set 2024-09-17T18:27:39.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.208+0000 7f5339ffb640 10 monclient: dump: 2024-09-17T18:27:39.221 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:39.221 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:39.221 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:39.221 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:39.221 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:39.221 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:39.221 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:39.221 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:39.221 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:39.222 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:39.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.208+0000 7f534b87f640 10 client.64145.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:27:39.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.208+0000 7f534b87f640 10 monclient: _renew_subs 2024-09-17T18:27:39.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.208+0000 7f534b87f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:27:39.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.208+0000 7f534b87f640 1 -- 172.21.15.121:0/1562870597 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f53441a9750 con 0x7f5330004800 2024-09-17T18:27:39.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.208+0000 7f534b87f640 1 librados: init done 2024-09-17T18:27:39.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.209+0000 7f5339ffb640 1 -- 172.21.15.121:0/1562870597 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f5334005030 con 0x7f5330004800 2024-09-17T18:27:39.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.209+0000 7f5339ffb640 1 --2- 172.21.15.121:0/1562870597 >> v2:172.21.15.173:6812/2423786037 conn(0x7f53280409c0 0x7f5328042e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:39.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.209+0000 7f5339ffb640 1 -- 172.21.15.121:0/1562870597 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(344..344 src has 177..344) ==== 5358+0+0 (secure 0 0 0) 0x7f5334058060 con 0x7f5330004800 2024-09-17T18:27:39.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.209+0000 7f5339ffb640 10 client.64145.objecter ms_dispatch 0x7f5344000b90 osd_map(344..344 src has 177..344) 2024-09-17T18:27:39.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.209+0000 7f5339ffb640 3 client.64145.objecter handle_osd_map got epochs [344,344] > 0 2024-09-17T18:27:39.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.209+0000 7f5339ffb640 3 client.64145.objecter handle_osd_map decoding full epoch 344 2024-09-17T18:27:39.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.209+0000 7f5339ffb640 20 client.64145.objecter dump_active .. 0 homeless 2024-09-17T18:27:39.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.209+0000 7f534961b640 1 --2- 172.21.15.121:0/1562870597 >> v2:172.21.15.173:6812/2423786037 conn(0x7f53280409c0 0x7f5328042e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:39.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.209+0000 7f534961b640 10 monclient: get_auth_request con 0x7f53280409c0 auth_method 0 2024-09-17T18:27:39.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.209+0000 7f534b87f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:27:39.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.209+0000 7f534b87f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:27:39.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.209+0000 7f534b87f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:27:39.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.209+0000 7f534b87f640 1 -- 172.21.15.121:0/1562870597 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f534409ea60 con 0x7f5330004800 2024-09-17T18:27:39.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.212+0000 7f5339ffb640 10 client.64145.objecter ms_handle_connect 0x7f53280409c0 2024-09-17T18:27:39.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.212+0000 7f534961b640 1 --2- 172.21.15.121:0/1562870597 >> v2:172.21.15.173:6812/2423786037 conn(0x7f53280409c0 0x7f5328042e80 secure :-1 s=READY pgs=4003 cs=0 l=1 rev1=1 crypto rx=0x7f5340032000 tx=0x7f5340010040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:39.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.215+0000 7f5339ffb640 1 -- 172.21.15.121:0/1562870597 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f533401b470 con 0x7f5330004800 2024-09-17T18:27:39.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.215+0000 7f5339ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:27:39.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.215+0000 7f5339ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:27:39.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.473+0000 7f534b87f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:27:39.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.473+0000 7f534b87f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:27:39.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.473+0000 7f534b87f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:27:39.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.473+0000 7f534b87f640 1 -- 172.21.15.121:0/1562870597 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f534406ce60 con 0x7f5330004800 2024-09-17T18:27:39.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.473+0000 7f5339ffb640 1 -- 172.21.15.121:0/1562870597 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f53340055b0 con 0x7f5330004800 2024-09-17T18:27:39.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.473+0000 7f5339ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:27:39.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.473+0000 7f5339ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:27:39.477 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":34,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:27:39.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.476+0000 7f534b87f640 10 librados: watch_flush enter 2024-09-17T18:27:39.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.476+0000 7f534b87f640 10 librados: watch_flush exit 2024-09-17T18:27:39.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.476+0000 7f534b87f640 20 client.64145.objecter shutdown clearing up homeless session... 2024-09-17T18:27:39.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.476+0000 7f534b87f640 10 client.64145.objecter successfully canceled tick 2024-09-17T18:27:39.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.476+0000 7f534b87f640 1 -- 172.21.15.121:0/1562870597 >> v2:172.21.15.173:6812/2423786037 conn(0x7f53280409c0 msgr2=0x7f5328042e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:39.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.476+0000 7f534b87f640 1 --2- 172.21.15.121:0/1562870597 >> v2:172.21.15.173:6812/2423786037 conn(0x7f53280409c0 0x7f5328042e80 secure :-1 s=READY pgs=4003 cs=0 l=1 rev1=1 crypto rx=0x7f5340032000 tx=0x7f5340010040 comp rx=0 tx=0).stop 2024-09-17T18:27:39.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.476+0000 7f534b87f640 10 monclient: shutdown 2024-09-17T18:27:39.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.476+0000 7f534b87f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:27:39.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.476+0000 7f534b87f640 1 -- 172.21.15.121:0/1562870597 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5330004800 msgr2=0x7f53441bcb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:39.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.476+0000 7f534b87f640 1 --2- 172.21.15.121:0/1562870597 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5330004800 0x7f53441bcb70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f533400d900 tx=0x7f533400ddc0 comp rx=0 tx=0).stop 2024-09-17T18:27:39.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.476+0000 7f534b87f640 1 -- 172.21.15.121:0/1562870597 shutdown_connections 2024-09-17T18:27:39.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.476+0000 7f534b87f640 1 --2- 172.21.15.121:0/1562870597 >> v2:172.21.15.173:6812/2423786037 conn(0x7f53280409c0 0x7f5328042e80 unknown :-1 s=CLOSED pgs=4003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:39.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.476+0000 7f534b87f640 1 --2- 172.21.15.121:0/1562870597 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53300053c0 0x7f53441a61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:39.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.476+0000 7f534b87f640 1 --2- 172.21.15.121:0/1562870597 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5330004800 0x7f53441bcb70 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:39.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.476+0000 7f534b87f640 1 --2- 172.21.15.121:0/1562870597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53300039e0 0x7f53441bc630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:39.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.476+0000 7f534b87f640 1 -- 172.21.15.121:0/1562870597 >> 172.21.15.121:0/1562870597 conn(0x7f53440a3b50 msgr2=0x7f534406f400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:27:39.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.476+0000 7f534b87f640 1 -- 172.21.15.121:0/1562870597 shutdown_connections 2024-09-17T18:27:39.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.476+0000 7f534b87f640 1 -- 172.21.15.121:0/1562870597 wait complete. 2024-09-17T18:27:39.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.476+0000 7f534b87f640 1 librados: shutdown 2024-09-17T18:27:39.489 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-17T18:27:39.489 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-17T18:27:39.490 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T18:27:39.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.584+0000 7fd3d9cb6640 10 monclient: get_monmap_and_config 2024-09-17T18:27:39.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.584+0000 7fd3d9cb6640 10 monclient: build_initial_monmap 2024-09-17T18:27:39.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.584+0000 7fd3d9cb6640 10 monclient: monmap: 2024-09-17T18:27:39.587 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:27:39.587 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:39.588 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:27:39.586389+0000 2024-09-17T18:27:39.588 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:27:39.586389+0000 2024-09-17T18:27:39.588 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:27:39.588 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:39.588 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:27:39.588 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:27:39.588 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:27:39.588 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:27:39.588 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:27:39.588 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:39.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.585+0000 7fd3d9cb6640 1 Processor -- start 2024-09-17T18:27:39.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.585+0000 7fd3d9cb6640 1 -- start start 2024-09-17T18:27:39.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.585+0000 7fd3d9cb6640 10 monclient: init 2024-09-17T18:27:39.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.585+0000 7fd3d9cb6640 10 monclient: _reopen_session rank -1 2024-09-17T18:27:39.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.585+0000 7fd3d9cb6640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-17T18:27:39.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.586+0000 7fd3d9cb6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd3d409e550 addr v1:172.21.15.121:6789/0 2024-09-17T18:27:39.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.586+0000 7fd3d9cb6640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd3d409f040 0x7fd3d413a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:39.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.586+0000 7fd3d9cb6640 10 monclient(hunting): picked mon.noname-a con 0x7fd3d409f040 addr v2:172.21.15.121:3300/0 2024-09-17T18:27:39.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.586+0000 7fd3d9cb6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd3d413ad90 addr v1:172.21.15.173:6789/0 2024-09-17T18:27:39.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.586+0000 7fd3d9cb6640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3d40a4460 con 0x7fd3d409e550 2024-09-17T18:27:39.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.586+0000 7fd3d9cb6640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3d40a2b60 con 0x7fd3d413ad90 2024-09-17T18:27:39.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.586+0000 7fd3d9cb6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:39.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.586+0000 7fd3d9cb6640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fd3d40a4950 con 0x7fd3d409f040 2024-09-17T18:27:39.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.586+0000 7fd3d9cb6640 10 monclient(hunting): _renew_subs 2024-09-17T18:27:39.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.586+0000 7fd3d9cb6640 10 monclient(hunting): authenticate will time out at 1263.303078s 2024-09-17T18:27:39.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.586+0000 7fd3caffd640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd3d409f040 0x7fd3d413a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:39.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.586+0000 7fd3d37fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fd3d409e550 0x7fd3d409e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:56018/0 (socket says 172.21.15.121:56018) 2024-09-17T18:27:39.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.586+0000 7fd3caffd640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd3d409f040 0x7fd3d413a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42956/0 (socket says 172.21.15.121:42956) 2024-09-17T18:27:39.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.586+0000 7fd3d37fe640 1 -- 172.21.15.121:0/2067985579 learned_addr learned my addr 172.21.15.121:0/2067985579 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:27:39.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.586+0000 7fd3caffd640 10 monclient(hunting): get_auth_request con 0x7fd3d409f040 auth_method 0 2024-09-17T18:27:39.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.586+0000 7fd3caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:39.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.586+0000 7fd3caffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:39.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.586+0000 7fd3caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:39.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.586+0000 7fd3d3fff640 1 -- 172.21.15.121:0/2067985579 >> v1:172.21.15.173:6789/0 conn(0x7fd3d413ad90 legacy=0x7fd3d413d180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.173:6789/0 2024-09-17T18:27:39.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.586+0000 7fd3caffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:39.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.586+0000 7fd3caffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:39.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.586+0000 7fd3caffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:39.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.586+0000 7fd3caffd640 10 monclient(hunting): handle_auth_done global_id 44514 payload 274 2024-09-17T18:27:39.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.586+0000 7fd3caffd640 1 -- 172.21.15.121:0/2067985579 >> v1:172.21.15.173:6789/0 conn(0x7fd3d413ad90 legacy=0x7fd3d413d180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:27:39.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.586+0000 7fd3caffd640 1 -- 172.21.15.121:0/2067985579 >> v1:172.21.15.121:6789/0 conn(0x7fd3d409e550 legacy=0x7fd3d409e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:27:39.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.586+0000 7fd3caffd640 10 monclient: _finish_hunting 0 2024-09-17T18:27:39.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.586+0000 7fd3caffd640 1 monclient: found mon.noname-a 2024-09-17T18:27:39.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.586+0000 7fd3caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:27:39.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.586+0000 7fd3caffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:27:39.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.586+0000 7fd3caffd640 1 -- 172.21.15.121:0/2067985579 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3d409f840 con 0x7fd3d409f040 2024-09-17T18:27:39.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.589+0000 7fd3caffd640 1 -- 172.21.15.121:0/2067985579 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3d409f040 msgr2=0x7fd3d413a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:39.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.589+0000 7fd3caffd640 1 --2- 172.21.15.121:0/2067985579 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3d409f040 0x7fd3d413a850 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4077910 tx=0x7fd3bc029fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86a950f1915f4ffb server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:39.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.589+0000 7fd3d2ffd640 1 -- 172.21.15.121:0/2067985579 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc037070 con 0x7fd3d409f040 2024-09-17T18:27:39.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.589+0000 7fd3d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:39.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.589+0000 7fd3d2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:27:39.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.589+0000 7fd3d2ffd640 10 monclient: dump: 2024-09-17T18:27:39.600 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:39.600 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:39.600 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:39.600 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:39.600 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:39.600 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:39.600 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:39.600 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:39.600 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:39.600 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:39.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.589+0000 7fd3d2ffd640 10 monclient: _finish_auth 0 2024-09-17T18:27:39.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.589+0000 7fd3d2ffd640 10 monclient: _check_auth_tickets 2024-09-17T18:27:39.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.589+0000 7fd3d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:27:39.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.589+0000 7fd3d2ffd640 1 -- 172.21.15.121:0/2067985579 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3bc032440 con 0x7fd3d409f040 2024-09-17T18:27:39.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.589+0000 7fd3d9cb6640 5 monclient: authenticate success, global_id 44514 2024-09-17T18:27:39.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.589+0000 7fd3d9cb6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:27:39.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.589+0000 7fd3d2ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:27:39.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.589+0000 7fd3d9cb6640 10 monclient: get_monmap_and_config success 2024-09-17T18:27:39.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.589+0000 7fd3d2ffd640 1 -- 172.21.15.121:0/2067985579 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc032810 con 0x7fd3d409f040 2024-09-17T18:27:39.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.589+0000 7fd3d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:39.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.589+0000 7fd3d9cb6640 4 set_mon_vals no callback set 2024-09-17T18:27:39.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.589+0000 7fd3d2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:27:39.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.589+0000 7fd3d2ffd640 10 monclient: dump: 2024-09-17T18:27:39.601 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:39.602 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:39.602 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:39.602 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:39.602 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:39.602 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:39.602 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:39.602 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:39.602 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:39.602 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:39.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.589+0000 7fd3d9cb6640 10 monclient: shutdown 2024-09-17T18:27:39.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.589+0000 7fd3d9cb6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:27:39.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.589+0000 7fd3d9cb6640 1 -- 172.21.15.121:0/2067985579 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3d409f040 msgr2=0x7fd3d413a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:39.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.589+0000 7fd3d9cb6640 1 --2- 172.21.15.121:0/2067985579 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3d409f040 0x7fd3d413a850 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4077910 tx=0x7fd3bc029fb0 comp rx=0 tx=0).stop 2024-09-17T18:27:39.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.589+0000 7fd3d9cb6640 1 -- 172.21.15.121:0/2067985579 shutdown_connections 2024-09-17T18:27:39.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.589+0000 7fd3d9cb6640 1 --2- 172.21.15.121:0/2067985579 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3d409f040 0x7fd3d413a850 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:39.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.589+0000 7fd3d9cb6640 1 -- 172.21.15.121:0/2067985579 >> 172.21.15.121:0/2067985579 conn(0x7fd3d40a7830 msgr2=0x7fd3d40a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:27:39.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.589+0000 7fd3d9cb6640 1 -- 172.21.15.121:0/2067985579 shutdown_connections 2024-09-17T18:27:39.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.589+0000 7fd3d9cb6640 1 -- 172.21.15.121:0/2067985579 wait complete. 2024-09-17T18:27:39.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.590+0000 7fd3d9cb6640 10 monclient: build_initial_monmap 2024-09-17T18:27:39.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.590+0000 7fd3d9cb6640 10 monclient: monmap: 2024-09-17T18:27:39.603 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:27:39.603 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:27:39.603 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:27:39.603 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:27:39.603 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:27:39.604 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:39.604 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:27:39.604 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:27:39.604 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:27:39.604 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:39.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.590+0000 7fd3d9cb6640 1 librados: starting msgr at 2024-09-17T18:27:39.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.590+0000 7fd3d9cb6640 1 librados: starting objecter 2024-09-17T18:27:39.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.590+0000 7fd3d9cb6640 1 Processor -- start 2024-09-17T18:27:39.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.590+0000 7fd3d9cb6640 1 -- start start 2024-09-17T18:27:39.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.590+0000 7fd3d9cb6640 1 librados: setting wanted keys 2024-09-17T18:27:39.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.590+0000 7fd3d9cb6640 1 librados: calling monclient init 2024-09-17T18:27:39.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.590+0000 7fd3d9cb6640 10 monclient: init 2024-09-17T18:27:39.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3d9cb6640 10 monclient: _reopen_session rank -1 2024-09-17T18:27:39.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3d9cb6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T18:27:39.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3d9cb6640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3d409e550 0x7fd3d41ce1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:39.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3d9cb6640 10 monclient(hunting): picked mon.noname-c con 0x7fd3d409e550 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:27:39.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3d9cb6640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3d409f040 0x7fd3d41ce710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:39.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3d9cb6640 10 monclient(hunting): picked mon.noname-a con 0x7fd3d409f040 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:27:39.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3d9cb6640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3d413ad90 0x7fd3d406f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:39.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3d9cb6640 10 monclient(hunting): picked mon.noname-b con 0x7fd3d413ad90 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:27:39.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3d9cb6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:39.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3d9cb6640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d409f490 con 0x7fd3d409f040 2024-09-17T18:27:39.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3d9cb6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:39.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3d9cb6640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d409dce0 con 0x7fd3d413ad90 2024-09-17T18:27:39.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3d9cb6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:39.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3d9cb6640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd3d41247c0 con 0x7fd3d409e550 2024-09-17T18:27:39.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3d9cb6640 10 monclient(hunting): _renew_subs 2024-09-17T18:27:39.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3d9cb6640 10 monclient(hunting): authenticate will time out at 1263.308402s 2024-09-17T18:27:39.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3caffd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3d409f040 0x7fd3d41ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:39.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3d37fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3d409e550 0x7fd3d41ce1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:39.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3caffd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3d409f040 0x7fd3d41ce710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42970/0 (socket says 172.21.15.121:42970) 2024-09-17T18:27:39.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3caffd640 1 -- 172.21.15.121:0/914269877 learned_addr learned my addr 172.21.15.121:0/914269877 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:27:39.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3caffd640 10 monclient(hunting): get_auth_request con 0x7fd3d409f040 auth_method 0 2024-09-17T18:27:39.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:39.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3caffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:39.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:39.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3d37fe640 10 monclient(hunting): get_auth_request con 0x7fd3d409e550 auth_method 0 2024-09-17T18:27:39.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:39.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:39.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:39.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3d3fff640 1 -- 172.21.15.121:0/914269877 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3d413ad90 msgr2=0x7fd3d406f2f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.173:3300/0 2024-09-17T18:27:39.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3caffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:39.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3caffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:39.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3d3fff640 1 --2- 172.21.15.121:0/914269877 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3d413ad90 0x7fd3d406f2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:27:39.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3caffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:39.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:39.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:39.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:39.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3caffd640 10 monclient(hunting): handle_auth_done global_id 44517 payload 824 2024-09-17T18:27:39.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3caffd640 1 -- 172.21.15.121:0/914269877 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3d409e550 msgr2=0x7fd3d41ce1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:39.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3caffd640 1 --2- 172.21.15.121:0/914269877 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3d409e550 0x7fd3d41ce1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:39.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3caffd640 1 -- 172.21.15.121:0/914269877 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3d413ad90 msgr2=0x7fd3d406f2f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:27:39.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3caffd640 1 --2- 172.21.15.121:0/914269877 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3d413ad90 0x7fd3d406f2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:39.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3caffd640 10 monclient: _finish_hunting 0 2024-09-17T18:27:39.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3caffd640 1 monclient: found mon.noname-a 2024-09-17T18:27:39.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:27:39.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3caffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:27:39.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3caffd640 1 -- 172.21.15.121:0/914269877 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3d4125b80 con 0x7fd3d409f040 2024-09-17T18:27:39.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.591+0000 7fd3d37fe640 1 --2- 172.21.15.121:0/914269877 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3d409e550 0x7fd3d41ce1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:27:39.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.593+0000 7fd3caffd640 1 -- 172.21.15.121:0/914269877 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3d409f040 msgr2=0x7fd3d41ce710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:39.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.594+0000 7fd3caffd640 1 --2- 172.21.15.121:0/914269877 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3d409f040 0x7fd3d41ce710 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc009890 tx=0x7fd3bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:39.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.594+0000 7fd3d0ff9640 10 client.?.objecter ms_handle_connect 0x7fd3d409f040 2024-09-17T18:27:39.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.594+0000 7fd3d0ff9640 10 client.?.objecter resend_mon_ops 2024-09-17T18:27:39.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.594+0000 7fd3d0ff9640 1 -- 172.21.15.121:0/914269877 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc041070 con 0x7fd3d409f040 2024-09-17T18:27:39.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.594+0000 7fd3d0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:39.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.594+0000 7fd3d0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:27:39.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.594+0000 7fd3d0ff9640 10 monclient: dump: 2024-09-17T18:27:39.608 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:39.608 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:39.608 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:39.609 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:39.609 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:39.609 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:39.609 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:39.609 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:39.609 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:39.609 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:39.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.594+0000 7fd3d0ff9640 10 monclient: _finish_auth 0 2024-09-17T18:27:39.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.594+0000 7fd3d0ff9640 10 monclient: _check_auth_tickets 2024-09-17T18:27:39.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.594+0000 7fd3d0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:27:39.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.594+0000 7fd3d0ff9640 1 -- 172.21.15.121:0/914269877 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3bc03c890 con 0x7fd3d409f040 2024-09-17T18:27:39.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.594+0000 7fd3d9cb6640 5 monclient: authenticate success, global_id 44517 2024-09-17T18:27:39.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.594+0000 7fd3d9cb6640 10 monclient: _renew_subs 2024-09-17T18:27:39.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.594+0000 7fd3d9cb6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:27:39.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.594+0000 7fd3d9cb6640 1 -- 172.21.15.121:0/914269877 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3d409e240 con 0x7fd3d409f040 2024-09-17T18:27:39.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.594+0000 7fd3d0ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T18:27:39.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.594+0000 7fd3d0ff9640 1 -- 172.21.15.121:0/914269877 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc037040 con 0x7fd3d409f040 2024-09-17T18:27:39.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.594+0000 7fd3d0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:39.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.594+0000 7fd3d0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:27:39.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.594+0000 7fd3d0ff9640 10 monclient: dump: 2024-09-17T18:27:39.610 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:39.610 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:39.610 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:39.610 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:39.610 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:39.610 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:39.610 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:39.610 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:39.610 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:39.610 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:39.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.594+0000 7fd3d9cb6640 10 client.44517.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:27:39.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.594+0000 7fd3d9cb6640 10 monclient: _renew_subs 2024-09-17T18:27:39.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.594+0000 7fd3d9cb6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:27:39.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.594+0000 7fd3d9cb6640 1 -- 172.21.15.121:0/914269877 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3d41258a0 con 0x7fd3d409f040 2024-09-17T18:27:39.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.594+0000 7fd3d27fc640 4 set_mon_vals no callback set 2024-09-17T18:27:39.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.594+0000 7fd3d9cb6640 1 librados: init done 2024-09-17T18:27:39.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.595+0000 7fd3d9cb6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:27:39.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.595+0000 7fd3d9cb6640 1 --2- 172.21.15.121:0/914269877 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3d406f830 0x7fd3d406fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:39.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.595+0000 7fd3d9cb6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:39.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.595+0000 7fd3d9cb6640 1 -- 172.21.15.121:0/914269877 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d409f610 con 0x7fd3d406f830 2024-09-17T18:27:39.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.595+0000 7fd3d0ff9640 1 -- 172.21.15.121:0/914269877 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fd3bc044070 con 0x7fd3d409f040 2024-09-17T18:27:39.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.595+0000 7fd3d37fe640 1 --2- 172.21.15.121:0/914269877 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3d406f830 0x7fd3d406fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:39.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.595+0000 7fd3d37fe640 10 monclient: get_auth_request con 0x7fd3d406f830 auth_method 0 2024-09-17T18:27:39.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.595+0000 7fd3d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:39.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.595+0000 7fd3d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:39.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.595+0000 7fd3d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:39.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.595+0000 7fd3d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:39.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.595+0000 7fd3d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:39.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.595+0000 7fd3d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:39.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.595+0000 7fd3d0ff9640 1 --2- 172.21.15.121:0/914269877 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd3b4040150 0x7fd3b4042610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:39.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.595+0000 7fd3d37fe640 10 monclient(hunting): handle_auth_done global_id 44520 payload 824 2024-09-17T18:27:39.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.595+0000 7fd3d0ff9640 1 -- 172.21.15.121:0/914269877 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(345..345 src has 177..345) ==== 5358+0+0 (secure 0 0 0) 0x7fd3bc0739d0 con 0x7fd3d409f040 2024-09-17T18:27:39.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.595+0000 7fd3d0ff9640 10 client.44517.objecter ms_dispatch 0x7fd3d4000b90 osd_map(345..345 src has 177..345) 2024-09-17T18:27:39.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.595+0000 7fd3d37fe640 1 -- 172.21.15.121:0/914269877 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd3d4074900 con 0x7fd3d406f830 2024-09-17T18:27:39.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.595+0000 7fd3d0ff9640 3 client.44517.objecter handle_osd_map got epochs [345,345] > 0 2024-09-17T18:27:39.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.595+0000 7fd3d0ff9640 3 client.44517.objecter handle_osd_map decoding full epoch 345 2024-09-17T18:27:39.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.595+0000 7fd3d0ff9640 20 client.44517.objecter dump_active .. 0 homeless 2024-09-17T18:27:39.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.595+0000 7fd3d3fff640 1 --2- 172.21.15.121:0/914269877 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd3b4040150 0x7fd3b4042610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:39.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.595+0000 7fd3d3fff640 10 monclient: get_auth_request con 0x7fd3b4040150 auth_method 0 2024-09-17T18:27:39.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.597+0000 7fd3d37fe640 1 -- 172.21.15.121:0/914269877 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3d406f830 msgr2=0x7fd3d406fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:39.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.597+0000 7fd3d0ff9640 10 client.44517.objecter ms_handle_connect 0x7fd3d406f830 2024-09-17T18:27:39.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.597+0000 7fd3d0ff9640 10 client.44517.objecter resend_mon_ops 2024-09-17T18:27:39.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.597+0000 7fd3d37fe640 1 --2- 172.21.15.121:0/914269877 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3d406f830 0x7fd3d406fc10 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd3c4007270 tx=0x7fd3c4007730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:39.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.597+0000 7fd3d0ff9640 1 -- 172.21.15.121:0/914269877 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c4017020 con 0x7fd3d406f830 2024-09-17T18:27:39.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.597+0000 7fd3d0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:39.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.597+0000 7fd3d0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:27:39.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.597+0000 7fd3d0ff9640 10 monclient: dump: 2024-09-17T18:27:39.616 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:39.616 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:39.616 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:39.616 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:39.616 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:39.616 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:39.616 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:39.617 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:39.617 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:39.617 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:39.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.598+0000 7fd3d0ff9640 1 -- 172.21.15.121:0/914269877 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd3c400fd00 con 0x7fd3d406f830 2024-09-17T18:27:39.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.598+0000 7fd3d0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:27:39.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.598+0000 7fd3d0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:27:39.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.598+0000 7fd3d0ff9640 1 -- 172.21.15.121:0/914269877 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3d406f830 msgr2=0x7fd3d406fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:39.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.598+0000 7fd3d0ff9640 1 --2- 172.21.15.121:0/914269877 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3d406f830 0x7fd3d406fc10 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd3c4007270 tx=0x7fd3c4007730 comp rx=0 tx=0).stop 2024-09-17T18:27:39.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.598+0000 7fd3d0ff9640 1 -- 172.21.15.121:0/914269877 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3d406f830 msgr2=0x7fd3d406fc10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:27:39.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.598+0000 7fd3d0ff9640 1 --2- 172.21.15.121:0/914269877 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3d406f830 0x7fd3d406fc10 secure :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd3c4007270 tx=0x7fd3c4007730 comp rx=0 tx=0).stop 2024-09-17T18:27:39.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.598+0000 7fd3d0ff9640 10 client.44517.objecter ms_handle_connect 0x7fd3b4040150 2024-09-17T18:27:39.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.598+0000 7fd3d3fff640 1 --2- 172.21.15.121:0/914269877 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd3b4040150 0x7fd3b4042610 secure :-1 s=READY pgs=4004 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4135a40 tx=0x7fd3cc0091f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:39.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.604+0000 7fd3d9cb6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:27:39.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.604+0000 7fd3d9cb6640 1 --2- 172.21.15.121:0/914269877 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3d4070150 0x7fd3d4137a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:39.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.604+0000 7fd3d9cb6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:39.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.604+0000 7fd3d9cb6640 1 -- 172.21.15.121:0/914269877 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d4124f50 con 0x7fd3d4070150 2024-09-17T18:27:39.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.604+0000 7fd3d37fe640 1 --2- 172.21.15.121:0/914269877 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3d4070150 0x7fd3d4137a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:39.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.604+0000 7fd3d37fe640 10 monclient: get_auth_request con 0x7fd3d4070150 auth_method 0 2024-09-17T18:27:39.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.604+0000 7fd3d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:39.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.604+0000 7fd3d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:39.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.604+0000 7fd3d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:39.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.604+0000 7fd3d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:39.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.604+0000 7fd3d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:39.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.604+0000 7fd3d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:39.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.604+0000 7fd3d37fe640 10 monclient(hunting): handle_auth_done global_id 44523 payload 824 2024-09-17T18:27:39.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.604+0000 7fd3d37fe640 1 -- 172.21.15.121:0/914269877 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd3d4125520 con 0x7fd3d4070150 2024-09-17T18:27:39.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.606+0000 7fd3d37fe640 1 -- 172.21.15.121:0/914269877 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3d4070150 msgr2=0x7fd3d4137a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:39.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.606+0000 7fd3d0ff9640 10 client.44517.objecter ms_handle_connect 0x7fd3d4070150 2024-09-17T18:27:39.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.606+0000 7fd3d0ff9640 10 client.44517.objecter resend_mon_ops 2024-09-17T18:27:39.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.606+0000 7fd3d37fe640 1 --2- 172.21.15.121:0/914269877 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3d4070150 0x7fd3d4137a70 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd3c4007270 tx=0x7fd3c4004940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:39.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.606+0000 7fd3d0ff9640 1 -- 172.21.15.121:0/914269877 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c4017030 con 0x7fd3d4070150 2024-09-17T18:27:39.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.606+0000 7fd3d0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:39.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.606+0000 7fd3d0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:27:39.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.606+0000 7fd3d0ff9640 10 monclient: dump: 2024-09-17T18:27:39.619 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:39.619 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:39.619 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:39.619 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:39.619 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:39.619 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:39.619 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:39.620 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:39.620 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:39.620 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:39.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.606+0000 7fd3d0ff9640 1 -- 172.21.15.121:0/914269877 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7fd3c40058b0 con 0x7fd3d4070150 2024-09-17T18:27:39.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.606+0000 7fd3d0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:27:39.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.606+0000 7fd3d0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:27:39.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.606+0000 7fd3d0ff9640 1 -- 172.21.15.121:0/914269877 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3d4070150 msgr2=0x7fd3d4137a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:39.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.606+0000 7fd3d0ff9640 1 --2- 172.21.15.121:0/914269877 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3d4070150 0x7fd3d4137a70 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd3c4007270 tx=0x7fd3c4004940 comp rx=0 tx=0).stop 2024-09-17T18:27:39.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.606+0000 7fd3d0ff9640 1 -- 172.21.15.121:0/914269877 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3d4070150 msgr2=0x7fd3d4137a70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:27:39.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.606+0000 7fd3d0ff9640 1 --2- 172.21.15.121:0/914269877 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3d4070150 0x7fd3d4137a70 secure :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd3c4007270 tx=0x7fd3c4004940 comp rx=0 tx=0).stop 2024-09-17T18:27:39.621 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:27:39.621 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:27:39.621 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:27:39.621 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "leader", 2024-09-17T18:27:39.621 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 34, 2024-09-17T18:27:39.621 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 285271, 2024-09-17T18:27:39.621 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:27:39.622 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:27:39.622 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:27:39.622 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:27:39.622 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 0, 2024-09-17T18:27:39.622 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:27:39.622 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:27:39.622 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:27:39.622 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:27:39.622 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:27:39.622 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:27:39.622 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:27:39.622 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:27:39.622 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:27:39.622 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:27:39.622 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:27:39.623 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:27:39.623 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:27:39.623 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:27:39.623 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:27:39.623 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:27:39.623 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:27:39.623 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:27:39.623 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:27:39.623 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:27:39.623 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:27:39.623 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:27:39.623 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:27:39.623 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:27:39.623 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:27:39.624 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:27:39.624 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:27:39.624 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:27:39.624 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:39.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:39.624 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:27:39.624 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:27:39.624 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:27:39.624 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:27:39.624 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:27:39.624 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:27:39.624 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:27:39.624 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:27:39.625 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:27:39.625 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:27:39.625 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:27:39.625 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:27:39.625 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:27:39.625 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:27:39.625 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:27:39.625 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:27:39.625 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:27:39.625 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:27:39.625 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:27:39.625 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:27:39.625 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:27:39.625 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:27:39.625 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:27:39.626 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:27:39.626 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:27:39.626 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:27:39.626 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:27:39.626 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:27:39.626 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:27:39.626 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:27:39.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:39.626 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:27:39.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:39.626 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:27:39.626 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:27:39.626 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:27:39.627 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:27:39.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:39.627 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:27:39.627 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:27:39.627 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:39.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:39.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:39.627 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:27:39.627 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:27:39.627 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:39.627 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:39.627 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:39.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:39.627 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:27:39.628 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:27:39.628 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:27:39.628 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:27:39.628 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:27:39.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:39.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:39.628 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:27:39.628 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:27:39.628 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:27:39.628 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:27:39.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:39.628 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:27:39.629 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:27:39.629 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:39.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:39.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:39.629 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:27:39.629 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:27:39.629 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:39.629 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:39.629 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:39.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:39.629 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:27:39.629 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:27:39.629 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:27:39.629 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:27:39.630 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:27:39.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:39.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:39.630 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:27:39.630 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:27:39.630 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:27:39.630 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:27:39.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:39.630 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:27:39.630 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:27:39.630 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:39.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:39.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:39.630 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:27:39.630 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:27:39.631 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:39.631 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:39.631 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:39.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:39.631 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:27:39.631 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:27:39.631 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:27:39.631 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:27:39.631 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:27:39.631 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:39.631 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:39.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:39.631 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:27:39.631 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:27:39.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:39.632 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:27:39.632 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:27:39.632 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:27:39.632 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:39.632 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:27:39.632 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:27:39.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:39.632 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:27:39.632 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:27:39.632 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 3 2024-09-17T18:27:39.632 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:39.632 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:27:39.632 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:27:39.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:39.633 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:27:39.633 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:27:39.633 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 6 2024-09-17T18:27:39.633 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:39.633 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:39.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:39.633 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:27:39.633 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:27:39.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.607+0000 7fd3d9cb6640 10 librados: watch_flush enter 2024-09-17T18:27:39.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.607+0000 7fd3d9cb6640 10 librados: watch_flush exit 2024-09-17T18:27:39.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.607+0000 7fd3d9cb6640 20 client.44517.objecter shutdown clearing up homeless session... 2024-09-17T18:27:39.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.607+0000 7fd3d9cb6640 10 client.44517.objecter successfully canceled tick 2024-09-17T18:27:39.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.607+0000 7fd3d9cb6640 1 -- 172.21.15.121:0/914269877 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd3b4040150 msgr2=0x7fd3b4042610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:39.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.607+0000 7fd3d9cb6640 1 --2- 172.21.15.121:0/914269877 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd3b4040150 0x7fd3b4042610 secure :-1 s=READY pgs=4004 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4135a40 tx=0x7fd3cc0091f0 comp rx=0 tx=0).stop 2024-09-17T18:27:39.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.607+0000 7fd3d9cb6640 10 monclient: shutdown 2024-09-17T18:27:39.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.607+0000 7fd3d9cb6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:27:39.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.607+0000 7fd3d9cb6640 1 -- 172.21.15.121:0/914269877 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3d409f040 msgr2=0x7fd3d41ce710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:39.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.607+0000 7fd3d9cb6640 1 --2- 172.21.15.121:0/914269877 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3d409f040 0x7fd3d41ce710 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc009890 tx=0x7fd3bc0046c0 comp rx=0 tx=0).stop 2024-09-17T18:27:39.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.607+0000 7fd3d3fff640 1 -- 172.21.15.121:0/914269877 reap_dead start 2024-09-17T18:27:39.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.607+0000 7fd3d3fff640 1 -- 172.21.15.121:0/914269877 reap_dead start 2024-09-17T18:27:39.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.607+0000 7fd3d9cb6640 1 -- 172.21.15.121:0/914269877 shutdown_connections 2024-09-17T18:27:39.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.607+0000 7fd3d9cb6640 1 -- 172.21.15.121:0/914269877 >> 172.21.15.121:0/914269877 conn(0x7fd3d40a7830 msgr2=0x7fd3d40a1950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:27:39.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.607+0000 7fd3d9cb6640 1 -- 172.21.15.121:0/914269877 shutdown_connections 2024-09-17T18:27:39.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.607+0000 7fd3d9cb6640 1 -- 172.21.15.121:0/914269877 wait complete. 2024-09-17T18:27:39.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.607+0000 7fd3d9cb6640 1 librados: shutdown 2024-09-17T18:27:39.636 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T18:27:39.637 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-17T18:27:39.637 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T18:27:39.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.732+0000 7fcd4659b640 10 monclient: get_monmap_and_config 2024-09-17T18:27:39.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.732+0000 7fcd4659b640 10 monclient: build_initial_monmap 2024-09-17T18:27:39.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.732+0000 7fcd4659b640 10 monclient: monmap: 2024-09-17T18:27:39.735 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:27:39.736 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:39.736 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:27:39.734442+0000 2024-09-17T18:27:39.736 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:27:39.734442+0000 2024-09-17T18:27:39.736 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:27:39.736 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:39.736 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:27:39.736 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:27:39.736 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:27:39.736 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:27:39.736 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:27:39.736 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:39.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.733+0000 7fcd4659b640 1 Processor -- start 2024-09-17T18:27:39.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.733+0000 7fcd4659b640 1 -- start start 2024-09-17T18:27:39.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.733+0000 7fcd4659b640 10 monclient: init 2024-09-17T18:27:39.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.734+0000 7fcd4659b640 10 monclient: _reopen_session rank -1 2024-09-17T18:27:39.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.734+0000 7fcd4659b640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-17T18:27:39.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.734+0000 7fcd4659b640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fcd400a1dd0 0x7fcd4009be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:39.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.734+0000 7fcd4659b640 10 monclient(hunting): picked mon.noname-c con 0x7fcd400a1dd0 addr v2:172.21.15.173:3300/0 2024-09-17T18:27:39.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.734+0000 7fcd4659b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcd400a21b0 addr v1:172.21.15.121:6789/0 2024-09-17T18:27:39.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.734+0000 7fcd4659b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd4013b980 0x7fcd4013dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:39.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.734+0000 7fcd4659b640 10 monclient(hunting): picked mon.noname-b con 0x7fcd4013b980 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:27:39.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.734+0000 7fcd4659b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd400a3c20 con 0x7fcd400a21b0 2024-09-17T18:27:39.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.734+0000 7fcd4659b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:39.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.734+0000 7fcd4659b640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fcd400a11a0 con 0x7fcd400a1dd0 2024-09-17T18:27:39.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.734+0000 7fcd4659b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:39.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.734+0000 7fcd4659b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcd400a32a0 con 0x7fcd4013b980 2024-09-17T18:27:39.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.734+0000 7fcd4659b640 10 monclient(hunting): _renew_subs 2024-09-17T18:27:39.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.734+0000 7fcd4659b640 10 monclient(hunting): authenticate will time out at 1263.451145s 2024-09-17T18:27:39.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.734+0000 7fcd44b38640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd4013b980 0x7fcd4013dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:39.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.734+0000 7fcd3ffff640 1 -- >> v2:172.21.15.173:3300/0 conn(0x7fcd400a1dd0 msgr2=0x7fcd4009be50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.173:3300/0 2024-09-17T18:27:39.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.734+0000 7fcd3f7fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fcd400a21b0 0x7fcd400a2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:56032/0 (socket says 172.21.15.121:56032) 2024-09-17T18:27:39.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.734+0000 7fcd3f7fe640 1 -- 172.21.15.121:0/3691034020 learned_addr learned my addr 172.21.15.121:0/3691034020 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:27:39.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.734+0000 7fcd44b38640 10 monclient(hunting): get_auth_request con 0x7fcd4013b980 auth_method 0 2024-09-17T18:27:39.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.734+0000 7fcd3ffff640 1 --2- 172.21.15.121:0/3691034020 >> v2:172.21.15.173:3300/0 conn(0x7fcd400a1dd0 0x7fcd4009be50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:27:39.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.734+0000 7fcd44b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:39.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.734+0000 7fcd44b38640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:39.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.734+0000 7fcd44b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:39.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.734+0000 7fcd44b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:39.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.734+0000 7fcd44b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:39.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.734+0000 7fcd44b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:39.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.734+0000 7fcd44b38640 10 monclient(hunting): handle_auth_done global_id 64151 payload 274 2024-09-17T18:27:39.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.734+0000 7fcd44b38640 1 -- 172.21.15.121:0/3691034020 >> v2:172.21.15.173:3300/0 conn(0x7fcd400a1dd0 msgr2=0x7fcd4009be50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:27:39.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.734+0000 7fcd44b38640 1 --2- 172.21.15.121:0/3691034020 >> v2:172.21.15.173:3300/0 conn(0x7fcd400a1dd0 0x7fcd4009be50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:39.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.734+0000 7fcd44b38640 1 -- 172.21.15.121:0/3691034020 >> v1:172.21.15.121:6789/0 conn(0x7fcd400a21b0 legacy=0x7fcd400a2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:27:39.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.734+0000 7fcd44b38640 10 monclient: _finish_hunting 0 2024-09-17T18:27:39.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.734+0000 7fcd44b38640 1 monclient: found mon.noname-b 2024-09-17T18:27:39.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.734+0000 7fcd44b38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:27:39.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.734+0000 7fcd44b38640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:27:39.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.734+0000 7fcd44b38640 1 -- 172.21.15.121:0/3691034020 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd400a15e0 con 0x7fcd4013b980 2024-09-17T18:27:39.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.737+0000 7fcd44b38640 1 -- 172.21.15.121:0/3691034020 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd4013b980 msgr2=0x7fcd4013dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:39.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.737+0000 7fcd44b38640 1 --2- 172.21.15.121:0/3691034020 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd4013b980 0x7fcd4013dd70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcd400a1320 tx=0x7fcd3802a4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27042e36eadc2c38 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:39.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.737+0000 7fcd3effd640 1 -- 172.21.15.121:0/3691034020 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd38007a30 con 0x7fcd4013b980 2024-09-17T18:27:39.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.737+0000 7fcd3effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:39.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.737+0000 7fcd3effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:27:39.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.737+0000 7fcd3effd640 10 monclient: dump: 2024-09-17T18:27:39.748 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:39.748 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:39.748 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:39.748 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:39.748 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:39.748 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:39.748 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:39.748 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:39.748 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:39.748 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:39.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.737+0000 7fcd3effd640 10 monclient: _finish_auth 0 2024-09-17T18:27:39.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.737+0000 7fcd3effd640 10 monclient: _check_auth_tickets 2024-09-17T18:27:39.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.737+0000 7fcd3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:27:39.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.737+0000 7fcd3effd640 1 -- 172.21.15.121:0/3691034020 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd38007bd0 con 0x7fcd4013b980 2024-09-17T18:27:39.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.737+0000 7fcd4659b640 5 monclient: authenticate success, global_id 64151 2024-09-17T18:27:39.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.737+0000 7fcd4659b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:27:39.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.737+0000 7fcd3effd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:27:39.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.737+0000 7fcd4659b640 10 monclient: get_monmap_and_config success 2024-09-17T18:27:39.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.737+0000 7fcd3effd640 1 -- 172.21.15.121:0/3691034020 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd38031400 con 0x7fcd4013b980 2024-09-17T18:27:39.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.737+0000 7fcd3effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:39.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.737+0000 7fcd4659b640 4 set_mon_vals no callback set 2024-09-17T18:27:39.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.737+0000 7fcd3effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:27:39.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.737+0000 7fcd3effd640 10 monclient: dump: 2024-09-17T18:27:39.749 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:39.749 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:39.750 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:39.750 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:39.750 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:39.750 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:39.750 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:39.750 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:39.750 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:39.750 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:39.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.737+0000 7fcd4659b640 10 monclient: shutdown 2024-09-17T18:27:39.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.737+0000 7fcd4659b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:27:39.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.737+0000 7fcd4659b640 1 -- 172.21.15.121:0/3691034020 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd4013b980 msgr2=0x7fcd4013dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:39.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.737+0000 7fcd4659b640 1 --2- 172.21.15.121:0/3691034020 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd4013b980 0x7fcd4013dd70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcd400a1320 tx=0x7fcd3802a4b0 comp rx=0 tx=0).stop 2024-09-17T18:27:39.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.737+0000 7fcd4659b640 1 -- 172.21.15.121:0/3691034020 shutdown_connections 2024-09-17T18:27:39.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.737+0000 7fcd4659b640 1 --2- 172.21.15.121:0/3691034020 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd4013b980 0x7fcd4013dd70 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:39.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.737+0000 7fcd4659b640 1 --2- 172.21.15.121:0/3691034020 >> v2:172.21.15.173:3300/0 conn(0x7fcd400a1dd0 0x7fcd4009be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:39.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.737+0000 7fcd4659b640 1 -- 172.21.15.121:0/3691034020 >> 172.21.15.121:0/3691034020 conn(0x7fcd400a75b0 msgr2=0x7fcd400a79c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:27:39.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.738+0000 7fcd4659b640 1 -- 172.21.15.121:0/3691034020 shutdown_connections 2024-09-17T18:27:39.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.738+0000 7fcd4659b640 1 -- 172.21.15.121:0/3691034020 wait complete. 2024-09-17T18:27:39.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.738+0000 7fcd4659b640 10 monclient: build_initial_monmap 2024-09-17T18:27:39.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.738+0000 7fcd4659b640 10 monclient: monmap: 2024-09-17T18:27:39.751 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:27:39.751 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:27:39.751 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:27:39.751 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:27:39.751 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:27:39.751 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:39.751 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:27:39.751 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:27:39.751 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:27:39.752 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:39.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.738+0000 7fcd4659b640 1 librados: starting msgr at 2024-09-17T18:27:39.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.738+0000 7fcd4659b640 1 librados: starting objecter 2024-09-17T18:27:39.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.738+0000 7fcd4659b640 1 Processor -- start 2024-09-17T18:27:39.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd4659b640 1 -- start start 2024-09-17T18:27:39.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd4659b640 1 librados: setting wanted keys 2024-09-17T18:27:39.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd4659b640 1 librados: calling monclient init 2024-09-17T18:27:39.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd4659b640 10 monclient: init 2024-09-17T18:27:39.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd4659b640 10 monclient: _reopen_session rank -1 2024-09-17T18:27:39.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd4659b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:27:39.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd4659b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd400a1dd0 0x7fcd4009a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:39.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd4659b640 10 monclient(hunting): picked mon.noname-a con 0x7fcd400a1dd0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:27:39.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd4659b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd400a21b0 0x7fcd4009a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:39.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd4659b640 10 monclient(hunting): picked mon.noname-b con 0x7fcd400a21b0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:27:39.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd4659b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd4013b980 0x7fcd40073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:39.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd4659b640 10 monclient(hunting): picked mon.noname-c con 0x7fcd4013b980 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:27:39.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd4659b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:39.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd4659b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcd4013e430 con 0x7fcd400a1dd0 2024-09-17T18:27:39.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd4659b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:39.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd4659b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcd4013e2b0 con 0x7fcd400a21b0 2024-09-17T18:27:39.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd4659b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:39.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd4659b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcd4013e5b0 con 0x7fcd4013b980 2024-09-17T18:27:39.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd4659b640 10 monclient(hunting): _renew_subs 2024-09-17T18:27:39.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd4659b640 10 monclient(hunting): authenticate will time out at 1263.456586s 2024-09-17T18:27:39.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd44b38640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd4013b980 0x7fcd40073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:39.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd3ffff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd400a1dd0 0x7fcd4009a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:39.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd44b38640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd4013b980 0x7fcd40073180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:54374/0 (socket says 172.21.15.121:54374) 2024-09-17T18:27:39.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd44b38640 1 -- 172.21.15.121:0/3442996488 learned_addr learned my addr 172.21.15.121:0/3442996488 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:27:39.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd44b38640 10 monclient(hunting): get_auth_request con 0x7fcd4013b980 auth_method 0 2024-09-17T18:27:39.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd44b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:39.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd44b38640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:39.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd44b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:39.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd3ffff640 10 monclient(hunting): get_auth_request con 0x7fcd400a1dd0 auth_method 0 2024-09-17T18:27:39.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:39.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:39.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:39.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd3f7fe640 1 -- 172.21.15.121:0/3442996488 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd400a21b0 msgr2=0x7fcd4009a860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.173:3300/0 2024-09-17T18:27:39.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd3f7fe640 1 --2- 172.21.15.121:0/3442996488 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd400a21b0 0x7fcd4009a860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:27:39.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd44b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:39.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd44b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:39.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd44b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:39.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:39.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:39.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:39.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd44b38640 10 monclient(hunting): handle_auth_done global_id 64154 payload 824 2024-09-17T18:27:39.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd44b38640 1 -- 172.21.15.121:0/3442996488 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd400a21b0 msgr2=0x7fcd4009a860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:27:39.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd44b38640 1 --2- 172.21.15.121:0/3442996488 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd400a21b0 0x7fcd4009a860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:39.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd44b38640 1 -- 172.21.15.121:0/3442996488 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd400a1dd0 msgr2=0x7fcd4009a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:39.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd44b38640 1 --2- 172.21.15.121:0/3442996488 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd400a1dd0 0x7fcd4009a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:27:39.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd44b38640 10 monclient: _finish_hunting 0 2024-09-17T18:27:39.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd44b38640 1 monclient: found mon.noname-c 2024-09-17T18:27:39.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd44b38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:27:39.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd44b38640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:27:39.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.739+0000 7fcd44b38640 1 -- 172.21.15.121:0/3442996488 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd4009e5c0 con 0x7fcd4013b980 2024-09-17T18:27:39.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.742+0000 7fcd44b38640 1 -- 172.21.15.121:0/3442996488 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd4013b980 msgr2=0x7fcd40073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:39.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.742+0000 7fcd44b38640 1 --2- 172.21.15.121:0/3442996488 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd4013b980 0x7fcd40073180 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcd3802aaa0 tx=0x7fcd38004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:39.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.742+0000 7fcd3cff9640 10 client.?.objecter ms_handle_connect 0x7fcd4013b980 2024-09-17T18:27:39.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.742+0000 7fcd3cff9640 10 client.?.objecter resend_mon_ops 2024-09-17T18:27:39.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.742+0000 7fcd3cff9640 1 -- 172.21.15.121:0/3442996488 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd38040070 con 0x7fcd4013b980 2024-09-17T18:27:39.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.742+0000 7fcd3cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:39.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.742+0000 7fcd3cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:27:39.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.742+0000 7fcd3cff9640 10 monclient: dump: 2024-09-17T18:27:39.756 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:39.756 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:39.756 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:39.756 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:39.756 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:39.756 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:39.756 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:39.757 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:39.757 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:39.757 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:39.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.742+0000 7fcd3cff9640 10 monclient: _finish_auth 0 2024-09-17T18:27:39.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.742+0000 7fcd3cff9640 10 monclient: _check_auth_tickets 2024-09-17T18:27:39.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.742+0000 7fcd3cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:27:39.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.742+0000 7fcd4659b640 5 monclient: authenticate success, global_id 64154 2024-09-17T18:27:39.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.742+0000 7fcd4659b640 10 monclient: _renew_subs 2024-09-17T18:27:39.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.742+0000 7fcd3cff9640 1 -- 172.21.15.121:0/3442996488 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd38045660 con 0x7fcd4013b980 2024-09-17T18:27:39.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.742+0000 7fcd4659b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:27:39.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.742+0000 7fcd4659b640 1 -- 172.21.15.121:0/3442996488 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd4012db10 con 0x7fcd4013b980 2024-09-17T18:27:39.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.742+0000 7fcd3cff9640 10 monclient: handle_config config(0 keys) 2024-09-17T18:27:39.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.742+0000 7fcd3cff9640 1 -- 172.21.15.121:0/3442996488 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd380314c0 con 0x7fcd4013b980 2024-09-17T18:27:39.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.742+0000 7fcd3cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:39.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.742+0000 7fcd3cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:27:39.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.742+0000 7fcd3cff9640 10 monclient: dump: 2024-09-17T18:27:39.758 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:39.758 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:39.758 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:39.758 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:39.758 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:39.758 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:39.758 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:39.758 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:39.758 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:39.758 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:39.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.742+0000 7fcd4659b640 10 client.64154.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:27:39.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.742+0000 7fcd4659b640 10 monclient: _renew_subs 2024-09-17T18:27:39.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.742+0000 7fcd4659b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:27:39.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.742+0000 7fcd4659b640 1 -- 172.21.15.121:0/3442996488 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd4012c8d0 con 0x7fcd4013b980 2024-09-17T18:27:39.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.742+0000 7fcd4659b640 1 librados: init done 2024-09-17T18:27:39.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.742+0000 7fcd3e7fc640 4 set_mon_vals no callback set 2024-09-17T18:27:39.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.743+0000 7fcd4659b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:27:39.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.743+0000 7fcd4659b640 1 --2- 172.21.15.121:0/3442996488 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd400736c0 0x7fcd40073aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:39.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.743+0000 7fcd4659b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:39.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.743+0000 7fcd4659b640 1 -- 172.21.15.121:0/3442996488 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcd400a66e0 con 0x7fcd400736c0 2024-09-17T18:27:39.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.743+0000 7fcd3ffff640 1 --2- 172.21.15.121:0/3442996488 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd400736c0 0x7fcd40073aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:39.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.743+0000 7fcd3cff9640 1 -- 172.21.15.121:0/3442996488 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fcd38039040 con 0x7fcd4013b980 2024-09-17T18:27:39.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.743+0000 7fcd3ffff640 10 monclient: get_auth_request con 0x7fcd400736c0 auth_method 0 2024-09-17T18:27:39.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.743+0000 7fcd3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:39.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.743+0000 7fcd3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:39.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.743+0000 7fcd3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:39.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.743+0000 7fcd3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:39.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.743+0000 7fcd3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:39.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.743+0000 7fcd3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:39.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.743+0000 7fcd3ffff640 10 monclient(hunting): handle_auth_done global_id 64157 payload 824 2024-09-17T18:27:39.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.743+0000 7fcd3ffff640 1 -- 172.21.15.121:0/3442996488 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcd4009be50 con 0x7fcd400736c0 2024-09-17T18:27:39.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.743+0000 7fcd3cff9640 1 --2- 172.21.15.121:0/3442996488 >> v2:172.21.15.173:6812/2423786037 conn(0x7fcd2c040150 0x7fcd2c042610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:39.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.743+0000 7fcd3cff9640 1 -- 172.21.15.121:0/3442996488 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(345..345 src has 177..345) ==== 5358+0+0 (secure 0 0 0) 0x7fcd38075180 con 0x7fcd4013b980 2024-09-17T18:27:39.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.743+0000 7fcd3cff9640 10 client.64154.objecter ms_dispatch 0x7fcd40000b90 osd_map(345..345 src has 177..345) 2024-09-17T18:27:39.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.743+0000 7fcd3cff9640 3 client.64154.objecter handle_osd_map got epochs [345,345] > 0 2024-09-17T18:27:39.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.743+0000 7fcd3cff9640 3 client.64154.objecter handle_osd_map decoding full epoch 345 2024-09-17T18:27:39.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.743+0000 7fcd3cff9640 20 client.64154.objecter dump_active .. 0 homeless 2024-09-17T18:27:39.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.743+0000 7fcd3f7fe640 1 --2- 172.21.15.121:0/3442996488 >> v2:172.21.15.173:6812/2423786037 conn(0x7fcd2c040150 0x7fcd2c042610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:39.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.743+0000 7fcd3f7fe640 10 monclient: get_auth_request con 0x7fcd2c040150 auth_method 0 2024-09-17T18:27:39.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.745+0000 7fcd3cff9640 10 client.64154.objecter ms_handle_connect 0x7fcd400736c0 2024-09-17T18:27:39.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.745+0000 7fcd3cff9640 10 client.64154.objecter resend_mon_ops 2024-09-17T18:27:39.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.745+0000 7fcd3ffff640 1 -- 172.21.15.121:0/3442996488 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd400736c0 msgr2=0x7fcd40073aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:39.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.745+0000 7fcd3ffff640 1 --2- 172.21.15.121:0/3442996488 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd400736c0 0x7fcd40073aa0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcd30003930 tx=0x7fcd30009b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:39.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.745+0000 7fcd3cff9640 1 -- 172.21.15.121:0/3442996488 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd30016070 con 0x7fcd400736c0 2024-09-17T18:27:39.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.745+0000 7fcd3cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:39.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.745+0000 7fcd3cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:27:39.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.745+0000 7fcd3cff9640 10 monclient: dump: 2024-09-17T18:27:39.763 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:39.763 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:39.763 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:39.763 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:39.764 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:39.764 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:39.764 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:39.764 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:39.764 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:39.764 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:39.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.746+0000 7fcd3cff9640 1 -- 172.21.15.121:0/3442996488 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcd300128f0 con 0x7fcd400736c0 2024-09-17T18:27:39.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.746+0000 7fcd3cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:27:39.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.746+0000 7fcd3cff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:27:39.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.746+0000 7fcd3cff9640 1 -- 172.21.15.121:0/3442996488 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd400736c0 msgr2=0x7fcd40073aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:39.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.746+0000 7fcd3cff9640 1 --2- 172.21.15.121:0/3442996488 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd400736c0 0x7fcd40073aa0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcd30003930 tx=0x7fcd30009b00 comp rx=0 tx=0).stop 2024-09-17T18:27:39.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.746+0000 7fcd3cff9640 1 -- 172.21.15.121:0/3442996488 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd400736c0 msgr2=0x7fcd40073aa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:27:39.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.746+0000 7fcd3cff9640 1 --2- 172.21.15.121:0/3442996488 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd400736c0 0x7fcd40073aa0 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcd30003930 tx=0x7fcd30009b00 comp rx=0 tx=0).stop 2024-09-17T18:27:39.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.746+0000 7fcd3f7fe640 1 --2- 172.21.15.121:0/3442996488 >> v2:172.21.15.173:6812/2423786037 conn(0x7fcd2c040150 0x7fcd2c042610 secure :-1 s=READY pgs=4005 cs=0 l=1 rev1=1 crypto rx=0x7fcd34007fd0 tx=0x7fcd34007b20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:39.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.746+0000 7fcd3cff9640 10 client.64154.objecter ms_handle_connect 0x7fcd2c040150 2024-09-17T18:27:39.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.752+0000 7fcd4659b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:27:39.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.752+0000 7fcd4659b640 1 --2- 172.21.15.121:0/3442996488 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd40073fe0 0x7fcd4006ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:27:39.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.752+0000 7fcd4659b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:27:39.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.752+0000 7fcd4659b640 1 -- 172.21.15.121:0/3442996488 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcd4012e180 con 0x7fcd40073fe0 2024-09-17T18:27:39.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.752+0000 7fcd3ffff640 1 --2- 172.21.15.121:0/3442996488 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd40073fe0 0x7fcd4006ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:27:39.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.752+0000 7fcd3ffff640 10 monclient: get_auth_request con 0x7fcd40073fe0 auth_method 0 2024-09-17T18:27:39.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.752+0000 7fcd3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:27:39.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.752+0000 7fcd3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:27:39.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.752+0000 7fcd3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:27:39.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.752+0000 7fcd3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:27:39.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.752+0000 7fcd3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:27:39.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.752+0000 7fcd3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:27:39.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.752+0000 7fcd3ffff640 10 monclient(hunting): handle_auth_done global_id 64160 payload 824 2024-09-17T18:27:39.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.752+0000 7fcd3ffff640 1 -- 172.21.15.121:0/3442996488 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcd400768d0 con 0x7fcd40073fe0 2024-09-17T18:27:39.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.754+0000 7fcd3ffff640 1 -- 172.21.15.121:0/3442996488 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd40073fe0 msgr2=0x7fcd4006ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:27:39.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.754+0000 7fcd3ffff640 1 --2- 172.21.15.121:0/3442996488 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd40073fe0 0x7fcd4006ffb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcd30018000 tx=0x7fcd300059d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:27:39.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.754+0000 7fcd3cff9640 10 client.64154.objecter ms_handle_connect 0x7fcd40073fe0 2024-09-17T18:27:39.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.754+0000 7fcd3cff9640 10 client.64154.objecter resend_mon_ops 2024-09-17T18:27:39.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.754+0000 7fcd3cff9640 1 -- 172.21.15.121:0/3442996488 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd30016040 con 0x7fcd40073fe0 2024-09-17T18:27:39.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.754+0000 7fcd3cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:27:39.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.754+0000 7fcd3cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:27:39.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.754+0000 7fcd3cff9640 10 monclient: dump: 2024-09-17T18:27:39.766 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:27:39.766 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:27:39.766 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:39.766 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:27:39.766 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:27:39.767 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:27:39.767 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:27:39.767 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:27:39.767 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:27:39.767 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:27:39.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.755+0000 7fcd3cff9640 1 -- 172.21.15.121:0/3442996488 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4875 (secure 0 0 0) 0x7fcd3001d270 con 0x7fcd40073fe0 2024-09-17T18:27:39.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.755+0000 7fcd3cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:27:39.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.755+0000 7fcd3cff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:27:39.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.755+0000 7fcd3cff9640 1 -- 172.21.15.121:0/3442996488 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd40073fe0 msgr2=0x7fcd4006ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:39.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.755+0000 7fcd3cff9640 1 --2- 172.21.15.121:0/3442996488 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd40073fe0 0x7fcd4006ffb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcd30018000 tx=0x7fcd300059d0 comp rx=0 tx=0).stop 2024-09-17T18:27:39.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.755+0000 7fcd3cff9640 1 -- 172.21.15.121:0/3442996488 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd40073fe0 msgr2=0x7fcd4006ffb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:27:39.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.755+0000 7fcd3cff9640 1 --2- 172.21.15.121:0/3442996488 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd40073fe0 0x7fcd4006ffb0 secure :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcd30018000 tx=0x7fcd300059d0 comp rx=0 tx=0).stop 2024-09-17T18:27:39.768 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:27:39.768 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:27:39.768 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:27:39.768 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:27:39.768 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 34, 2024-09-17T18:27:39.768 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 38487, 2024-09-17T18:27:39.768 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:27:39.768 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:27:39.768 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:27:39.768 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:27:39.768 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 0, 2024-09-17T18:27:39.769 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:27:39.769 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:27:39.769 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:27:39.769 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:27:39.769 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:27:39.769 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:27:39.769 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:27:39.769 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:27:39.769 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:27:39.769 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:27:39.769 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:27:39.769 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:27:39.769 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:27:39.770 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:27:39.770 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:27:39.770 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:27:39.770 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:27:39.770 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:27:39.770 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:27:39.770 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:27:39.770 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:27:39.770 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:27:39.770 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:27:39.770 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:27:39.770 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:27:39.770 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:27:39.770 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:27:39.771 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:27:39.771 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:39.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:39.771 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:27:39.771 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:27:39.771 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:27:39.771 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:27:39.771 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:27:39.771 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:27:39.771 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:27:39.771 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:27:39.771 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:27:39.771 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:27:39.771 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:27:39.771 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:27:39.772 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:27:39.772 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:27:39.772 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:27:39.772 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:27:39.772 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:27:39.772 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:27:39.772 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:27:39.772 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:27:39.772 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:27:39.772 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:27:39.772 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:27:39.772 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:27:39.772 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:27:39.772 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:27:39.773 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:27:39.773 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:27:39.773 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:27:39.773 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:27:39.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:39.773 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:27:39.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:39.773 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:27:39.773 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:27:39.773 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:27:39.773 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:27:39.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:39.773 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:27:39.774 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:27:39.774 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:39.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:39.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:39.774 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:27:39.774 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:27:39.774 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:39.774 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:39.774 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:39.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:39.774 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:27:39.774 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:27:39.774 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:27:39.774 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:27:39.775 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:27:39.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:39.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:39.775 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:27:39.775 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:27:39.775 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:27:39.775 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:27:39.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:39.775 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:27:39.775 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:27:39.775 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:39.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:39.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:39.775 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:27:39.776 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:27:39.776 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:39.776 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:39.776 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:39.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:39.776 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:27:39.776 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:27:39.776 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:27:39.776 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:27:39.776 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:27:39.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:39.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:39.777 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:27:39.777 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:27:39.777 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:27:39.777 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:27:39.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:39.777 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:27:39.777 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:27:39.777 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:39.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:39.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:39.777 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:27:39.777 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:27:39.777 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:27:39.777 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:39.777 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:39.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:39.778 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:27:39.778 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:27:39.778 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:27:39.778 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:27:39.778 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:27:39.778 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:39.778 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:39.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:39.778 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:27:39.778 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:27:39.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:39.778 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:27:39.778 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:27:39.778 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:27:39.779 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:39.779 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:27:39.779 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:27:39.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:39.779 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:27:39.779 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:27:39.779 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 3 2024-09-17T18:27:39.779 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:39.779 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:27:39.779 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:27:39.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:39.779 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:27:39.779 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:27:39.779 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 3 2024-09-17T18:27:39.779 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:39.780 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:27:39.780 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": [ 2024-09-17T18:27:39.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:27:39.780 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:27:39.780 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:27:39.780 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:27:39.780 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:27:39.780 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:27:39.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:27:39.780 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:27:39.780 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:27:39.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.755+0000 7fcd4659b640 10 librados: watch_flush enter 2024-09-17T18:27:39.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.755+0000 7fcd4659b640 10 librados: watch_flush exit 2024-09-17T18:27:39.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.755+0000 7fcd4659b640 20 client.64154.objecter shutdown clearing up homeless session... 2024-09-17T18:27:39.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.755+0000 7fcd4659b640 10 client.64154.objecter successfully canceled tick 2024-09-17T18:27:39.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.755+0000 7fcd4659b640 1 -- 172.21.15.121:0/3442996488 >> v2:172.21.15.173:6812/2423786037 conn(0x7fcd2c040150 msgr2=0x7fcd2c042610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:39.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.755+0000 7fcd4659b640 1 --2- 172.21.15.121:0/3442996488 >> v2:172.21.15.173:6812/2423786037 conn(0x7fcd2c040150 0x7fcd2c042610 secure :-1 s=READY pgs=4005 cs=0 l=1 rev1=1 crypto rx=0x7fcd34007fd0 tx=0x7fcd34007b20 comp rx=0 tx=0).stop 2024-09-17T18:27:39.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.755+0000 7fcd4659b640 10 monclient: shutdown 2024-09-17T18:27:39.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.755+0000 7fcd4659b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:27:39.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.755+0000 7fcd4659b640 1 -- 172.21.15.121:0/3442996488 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd4013b980 msgr2=0x7fcd40073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:27:39.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.755+0000 7fcd4659b640 1 --2- 172.21.15.121:0/3442996488 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd4013b980 0x7fcd40073180 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcd3802aaa0 tx=0x7fcd38004a40 comp rx=0 tx=0).stop 2024-09-17T18:27:39.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.755+0000 7fcd44b38640 1 -- 172.21.15.121:0/3442996488 reap_dead start 2024-09-17T18:27:39.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.755+0000 7fcd44b38640 1 -- 172.21.15.121:0/3442996488 reap_dead start 2024-09-17T18:27:39.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.755+0000 7fcd4659b640 1 -- 172.21.15.121:0/3442996488 shutdown_connections 2024-09-17T18:27:39.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.755+0000 7fcd4659b640 1 -- 172.21.15.121:0/3442996488 >> 172.21.15.121:0/3442996488 conn(0x7fcd400a75b0 msgr2=0x7fcd4013aba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:27:39.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.755+0000 7fcd4659b640 1 -- 172.21.15.121:0/3442996488 shutdown_connections 2024-09-17T18:27:39.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.755+0000 7fcd4659b640 1 -- 172.21.15.121:0/3442996488 wait complete. 2024-09-17T18:27:39.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:27:39.755+0000 7fcd4659b640 1 librados: shutdown 2024-09-17T18:27:39.784 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T18:27:39.784 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-17T18:28:25.472 INFO:tasks.ceph.osd.4.smithi173.stderr:2024-09-17T18:28:25.470+0000 7fbfba792640 -1 snap_mapper.add_oid found existing snaps mapped on 300:f8587072:test-rados-api-smithi173-35399-80::foo:2, removing 2024-09-17T18:28:25.473 INFO:tasks.ceph.osd.0.smithi121.stderr:2024-09-17T18:28:25.470+0000 7f3287195640 -1 snap_mapper.add_oid found existing snaps mapped on 300:f8587072:test-rados-api-smithi173-35399-80::foo:2, removing 2024-09-17T18:28:55.416 INFO:tasks.workunit.client.0.smithi173.stdout: OK ] LibRadosTwoPoolsPP.ProxyRead (18771 ms) 2024-09-17T18:28:55.416 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-09-17T18:28:55.416 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (27426 ms) 2024-09-17T18:28:55.416 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-09-17T18:28:55.417 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3014 ms) 2024-09-17T18:28:55.417 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-09-17T18:28:55.417 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: require_osd_release = squid 2024-09-17T18:28:55.417 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (2765 ms) 2024-09-17T18:28:55.417 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-09-17T18:28:55.417 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (3644 ms) 2024-09-17T18:28:55.417 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-09-17T18:28:55.417 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: require_osd_release = squid 2024-09-17T18:28:55.417 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (3022 ms) 2024-09-17T18:28:55.417 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-09-17T18:28:55.417 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: require_osd_release = squid 2024-09-17T18:28:55.417 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (3338 ms) 2024-09-17T18:28:55.417 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-09-17T18:28:55.417 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: require_osd_release = squid 2024-09-17T18:28:55.418 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (39297 ms) 2024-09-17T18:28:55.418 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-09-17T18:28:55.418 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: require_osd_release = squid 2024-09-17T18:28:55.418 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (16719 ms) 2024-09-17T18:28:55.418 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-09-17T18:28:55.418 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: require_osd_release = squid 2024-09-17T18:28:55.418 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (3998 ms) 2024-09-17T18:28:55.418 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-09-17T18:28:55.418 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: require_osd_release = squid 2024-09-17T18:28:55.418 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (3012 ms) 2024-09-17T18:28:55.418 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-09-17T18:28:55.418 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: require_osd_release = squid 2024-09-17T18:28:55.418 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (24882 ms) 2024-09-17T18:28:55.418 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-09-17T18:28:55.418 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: require_osd_release = squid 2024-09-17T18:28:55.419 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (35198 ms) 2024-09-17T18:28:55.419 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-09-17T18:28:55.419 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: require_osd_release = squid 2024-09-17T18:28:55.419 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (5025 ms) 2024-09-17T18:28:55.419 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-09-17T18:28:55.419 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: require_osd_release = squid 2024-09-17T18:28:55.419 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (3915 ms) 2024-09-17T18:28:55.419 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-09-17T18:28:55.419 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: require_osd_release = squid 2024-09-17T18:28:55.419 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: waiting for scrubs... 2024-09-17T18:28:55.419 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: done waiting 2024-09-17T18:28:55.419 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (24280 ms) 2024-09-17T18:28:55.419 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-09-17T18:28:55.419 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: require_osd_release = squid 2024-09-17T18:28:55.419 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (9705 ms) 2024-09-17T18:28:55.420 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-09-17T18:28:55.420 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: require_osd_release = squid 2024-09-17T18:28:55.420 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (9001 ms) 2024-09-17T18:28:55.420 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-09-17T18:28:55.420 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: require_osd_release = squid 2024-09-17T18:28:55.420 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (9034 ms) 2024-09-17T18:28:55.420 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-09-17T18:29:09.785 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-17T18:29:09.785 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-17T18:29:09.785 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-17T18:29:09.789 INFO:tasks.ceph.mon.b:Started 2024-09-17T18:29:09.789 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-17T18:29:09.789 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-17T18:29:09.789 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-17T18:29:24.791 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-17T18:29:24.791 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-17T18:29:24.792 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T18:29:24.792 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:29:24.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.899+0000 7f41882b6640 10 monclient: get_monmap_and_config 2024-09-17T18:29:24.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.899+0000 7f41882b6640 10 monclient: build_initial_monmap 2024-09-17T18:29:24.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.899+0000 7f41882b6640 10 monclient: monmap: 2024-09-17T18:29:24.903 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:29:24.903 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:24.903 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:29:24.901563+0000 2024-09-17T18:29:24.903 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:29:24.901563+0000 2024-09-17T18:29:24.903 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:29:24.903 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:24.903 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:29:24.903 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:29:24.903 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:29:24.903 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:29:24.903 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:29:24.904 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:24.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.900+0000 7f41882b6640 1 Processor -- start 2024-09-17T18:29:24.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.900+0000 7f41882b6640 1 -- start start 2024-09-17T18:29:24.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.900+0000 7f41882b6640 10 monclient: init 2024-09-17T18:29:24.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.901+0000 7f41882b6640 10 monclient: _reopen_session rank -1 2024-09-17T18:29:24.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.901+0000 7f41882b6640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-17T18:29:24.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.901+0000 7f41882b6640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f41800a0a70 0x7f41801348f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:24.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.901+0000 7f41882b6640 10 monclient(hunting): picked mon.noname-c con 0x7f41800a0a70 addr v2:172.21.15.173:3300/0 2024-09-17T18:29:24.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.901+0000 7f41882b6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f41800a0e50 addr v1:172.21.15.173:6789/0 2024-09-17T18:29:24.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.901+0000 7f41882b6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f41800a1260 addr v1:172.21.15.121:6789/0 2024-09-17T18:29:24.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.901+0000 7f41882b6640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f418009f5b0 con 0x7f41800a1260 2024-09-17T18:29:24.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.901+0000 7f41882b6640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41800a3100 con 0x7f41800a0e50 2024-09-17T18:29:24.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.901+0000 7f41882b6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:24.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.901+0000 7f41882b6640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f41800a48b0 con 0x7f41800a0a70 2024-09-17T18:29:24.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.901+0000 7f41882b6640 10 monclient(hunting): _renew_subs 2024-09-17T18:29:24.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.901+0000 7f41882b6640 10 monclient(hunting): authenticate will time out at 1368.618359s 2024-09-17T18:29:24.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.901+0000 7f4186853640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f41800a1260 0x7f41800a29f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:43788/0 (socket says 172.21.15.121:43788) 2024-09-17T18:29:24.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.901+0000 7f4186853640 1 -- 172.21.15.121:0/2063961088 learned_addr learned my addr 172.21.15.121:0/2063961088 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:29:24.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.901+0000 7f4186052640 1 --2- 172.21.15.121:0/2063961088 >> v2:172.21.15.173:3300/0 conn(0x7f41800a0a70 0x7f41801348f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:24.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.901+0000 7f4186052640 10 monclient(hunting): get_auth_request con 0x7f41800a0a70 auth_method 0 2024-09-17T18:29:24.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.901+0000 7f4186052640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:24.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.901+0000 7f4186052640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:24.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.901+0000 7f4186052640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:24.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.901+0000 7f4185851640 1 -- 172.21.15.121:0/2063961088 >> v1:172.21.15.173:6789/0 conn(0x7f41800a0e50 legacy=0x7f41800a22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T18:29:24.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.901+0000 7f4185851640 1 -- 172.21.15.121:0/2063961088 >> v1:172.21.15.173:6789/0 conn(0x7f41800a0e50 legacy=0x7f41800a22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:29:24.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.901+0000 7f4185851640 1 --1- 172.21.15.121:0/2063961088 >> v1:172.21.15.173:6789/0 conn(0x7f41800a0e50 0x7f41800a22e0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-09-17T18:29:24.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.901+0000 7f4186052640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:24.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.901+0000 7f4186052640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:24.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.901+0000 7f4186052640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:24.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.902+0000 7f4186052640 10 monclient(hunting): handle_auth_done global_id 74098 payload 274 2024-09-17T18:29:24.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.902+0000 7f4186052640 1 -- 172.21.15.121:0/2063961088 >> v1:172.21.15.173:6789/0 conn(0x7f41800a0e50 legacy=0x7f41800a22e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:29:24.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.902+0000 7f4186052640 1 -- 172.21.15.121:0/2063961088 >> v1:172.21.15.121:6789/0 conn(0x7f41800a1260 legacy=0x7f41800a29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:29:24.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.902+0000 7f4186052640 10 monclient: _finish_hunting 0 2024-09-17T18:29:24.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.902+0000 7f4186052640 1 monclient: found mon.noname-c 2024-09-17T18:29:24.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.902+0000 7f4186052640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:29:24.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.902+0000 7f4186052640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.173:3300/0 2024-09-17T18:29:24.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.902+0000 7f4186052640 1 -- 172.21.15.121:0/2063961088 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f418009eca0 con 0x7f41800a0a70 2024-09-17T18:29:24.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.904+0000 7f4186052640 1 -- 172.21.15.121:0/2063961088 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41800a0a70 msgr2=0x7f41801348f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:24.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.904+0000 7f4186052640 1 --2- 172.21.15.121:0/2063961088 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41800a0a70 0x7f41801348f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4170009d90 tx=0x7f417002a490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1400517471c67f58 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:24.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.905+0000 7f4185050640 1 -- 172.21.15.121:0/2063961088 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4170007a30 con 0x7f41800a0a70 2024-09-17T18:29:24.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.905+0000 7f4185050640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:24.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.905+0000 7f4185050640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:29:24.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.905+0000 7f4185050640 10 monclient: dump: 2024-09-17T18:29:24.915 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:24.915 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:24.915 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:24.915 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:24.915 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:24.915 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:24.915 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:24.916 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:24.916 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:24.916 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:24.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.905+0000 7f4185050640 10 monclient: _finish_auth 0 2024-09-17T18:29:24.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.905+0000 7f4185050640 10 monclient: _check_auth_tickets 2024-09-17T18:29:24.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.905+0000 7f4185050640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:29:24.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.905+0000 7f4185050640 1 -- 172.21.15.121:0/2063961088 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4170007bd0 con 0x7f41800a0a70 2024-09-17T18:29:24.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.905+0000 7f4185050640 10 monclient: handle_config config(0 keys) 2024-09-17T18:29:24.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.905+0000 7f4185050640 1 -- 172.21.15.121:0/2063961088 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4170031400 con 0x7f41800a0a70 2024-09-17T18:29:24.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.905+0000 7f41882b6640 5 monclient: authenticate success, global_id 74098 2024-09-17T18:29:24.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.905+0000 7f41882b6640 10 monclient: get_monmap_and_config success 2024-09-17T18:29:24.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.905+0000 7f41882b6640 4 set_mon_vals no callback set 2024-09-17T18:29:24.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.905+0000 7f4185050640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:24.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.905+0000 7f4185050640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:29:24.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.905+0000 7f4185050640 10 monclient: dump: 2024-09-17T18:29:24.917 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:24.917 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:24.917 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:24.917 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:24.917 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:24.917 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:24.917 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:24.917 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:24.917 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:24.917 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:24.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.906+0000 7f41882b6640 10 monclient: shutdown 2024-09-17T18:29:24.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.906+0000 7f41882b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:29:24.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.906+0000 7f41882b6640 1 -- 172.21.15.121:0/2063961088 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41800a0a70 msgr2=0x7f41801348f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:24.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.906+0000 7f41882b6640 1 --2- 172.21.15.121:0/2063961088 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41800a0a70 0x7f41801348f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4170009d90 tx=0x7f417002a490 comp rx=0 tx=0).stop 2024-09-17T18:29:24.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.906+0000 7f41882b6640 1 -- 172.21.15.121:0/2063961088 shutdown_connections 2024-09-17T18:29:24.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.906+0000 7f41882b6640 1 --2- 172.21.15.121:0/2063961088 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41800a0a70 0x7f41801348f0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:24.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.906+0000 7f41882b6640 1 -- 172.21.15.121:0/2063961088 >> 172.21.15.121:0/2063961088 conn(0x7f41800a3b50 msgr2=0x7f41800a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:29:24.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.906+0000 7f41882b6640 1 -- 172.21.15.121:0/2063961088 shutdown_connections 2024-09-17T18:29:24.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.906+0000 7f41882b6640 1 -- 172.21.15.121:0/2063961088 wait complete. 2024-09-17T18:29:24.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.906+0000 7f41882b6640 10 monclient: build_initial_monmap 2024-09-17T18:29:24.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.906+0000 7f41882b6640 10 monclient: monmap: 2024-09-17T18:29:24.918 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:29:24.918 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:29:24.918 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:29:24.918 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:29:24.918 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:29:24.919 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:24.919 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:29:24.919 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:29:24.919 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:29:24.919 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:24.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f41882b6640 1 librados: starting msgr at 2024-09-17T18:29:24.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f41882b6640 1 librados: starting objecter 2024-09-17T18:29:24.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f41882b6640 1 Processor -- start 2024-09-17T18:29:24.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f41882b6640 1 -- start start 2024-09-17T18:29:24.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f41882b6640 1 librados: setting wanted keys 2024-09-17T18:29:24.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f41882b6640 1 librados: calling monclient init 2024-09-17T18:29:24.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f41882b6640 10 monclient: init 2024-09-17T18:29:24.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f41882b6640 10 monclient: _reopen_session rank -1 2024-09-17T18:29:24.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f41882b6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:29:24.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f41882b6640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41800a0a70 0x7f41801cba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:24.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f41882b6640 10 monclient(hunting): picked mon.noname-a con 0x7f41800a0a70 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:29:24.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f41882b6640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41800a0e50 0x7f41801cbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:24.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f41882b6640 10 monclient(hunting): picked mon.noname-b con 0x7f41800a0e50 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:29:24.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f41882b6640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41800a1260 0x7f4180123390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:24.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f41882b6640 10 monclient(hunting): picked mon.noname-c con 0x7f41800a1260 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:29:24.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f41882b6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:24.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f41882b6640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f418009de60 con 0x7f41800a0a70 2024-09-17T18:29:24.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f41882b6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:24.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f41882b6640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f418009dce0 con 0x7f41800a0e50 2024-09-17T18:29:24.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f41882b6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:24.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f41882b6640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f41800a1a50 con 0x7f41800a1260 2024-09-17T18:29:24.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f41882b6640 10 monclient(hunting): _renew_subs 2024-09-17T18:29:24.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f41882b6640 10 monclient(hunting): authenticate will time out at 1368.624722s 2024-09-17T18:29:24.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f4186052640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41800a0a70 0x7f41801cba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:24.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f4186853640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41800a1260 0x7f4180123390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:24.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f4186052640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41800a0a70 0x7f41801cba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:57976/0 (socket says 172.21.15.121:57976) 2024-09-17T18:29:24.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f4186052640 1 -- 172.21.15.121:0/1351702159 learned_addr learned my addr 172.21.15.121:0/1351702159 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:29:24.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f4186052640 10 monclient(hunting): get_auth_request con 0x7f41800a0a70 auth_method 0 2024-09-17T18:29:24.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f4186052640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:24.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f4186052640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:24.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f4186052640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:24.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f4185851640 1 --2- 172.21.15.121:0/1351702159 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41800a0e50 0x7f41801cbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:24.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f4186853640 10 monclient(hunting): get_auth_request con 0x7f41800a1260 auth_method 0 2024-09-17T18:29:24.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f4186853640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:24.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f4186853640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:24.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f4186853640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:24.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f4185851640 10 monclient(hunting): get_auth_request con 0x7f41800a0e50 auth_method 0 2024-09-17T18:29:24.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f4185851640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:24.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f4185851640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:24.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f4185851640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:24.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f4186853640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:24.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f4186853640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:24.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f4186853640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:24.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f4186052640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:24.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f4186052640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:24.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f4186052640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:24.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.907+0000 7f4185851640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:24.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.908+0000 7f4185851640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:24.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.908+0000 7f4185851640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:24.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.908+0000 7f4186853640 10 monclient(hunting): handle_auth_done global_id 64163 payload 824 2024-09-17T18:29:24.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.908+0000 7f4186853640 1 -- 172.21.15.121:0/1351702159 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41800a0e50 msgr2=0x7f41801cbfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:24.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.908+0000 7f4186853640 1 --2- 172.21.15.121:0/1351702159 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41800a0e50 0x7f41801cbfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:24.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.908+0000 7f4186853640 1 -- 172.21.15.121:0/1351702159 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41800a0a70 msgr2=0x7f41801cba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:24.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.908+0000 7f4186853640 1 --2- 172.21.15.121:0/1351702159 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41800a0a70 0x7f41801cba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:24.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.908+0000 7f4186853640 10 monclient: _finish_hunting 0 2024-09-17T18:29:24.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.908+0000 7f4186853640 1 monclient: found mon.noname-c 2024-09-17T18:29:24.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.908+0000 7f4186853640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:29:24.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.908+0000 7f4186853640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:29:24.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.908+0000 7f4186853640 1 -- 172.21.15.121:0/1351702159 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41801271b0 con 0x7f41800a1260 2024-09-17T18:29:24.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.908+0000 7f4186052640 1 --2- 172.21.15.121:0/1351702159 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41800a0a70 0x7f41801cba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:29:24.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.910+0000 7f4186853640 1 -- 172.21.15.121:0/1351702159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41800a1260 msgr2=0x7f4180123390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:24.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.910+0000 7f4186853640 1 --2- 172.21.15.121:0/1351702159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41800a1260 0x7f4180123390 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f417800c3e0 tx=0x7f417800c7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:24.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.910+0000 7f417effd640 10 client.?.objecter ms_handle_connect 0x7f41800a1260 2024-09-17T18:29:24.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.910+0000 7f417effd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:29:24.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.910+0000 7f417effd640 1 -- 172.21.15.121:0/1351702159 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4178017070 con 0x7f41800a1260 2024-09-17T18:29:24.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.910+0000 7f417effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:24.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.910+0000 7f417effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:29:24.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.910+0000 7f417effd640 10 monclient: dump: 2024-09-17T18:29:24.924 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:24.924 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:24.924 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:24.924 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:24.924 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:24.924 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:24.924 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:24.924 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:24.924 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:24.924 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:24.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.910+0000 7f417effd640 10 monclient: _finish_auth 0 2024-09-17T18:29:24.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.910+0000 7f417effd640 10 monclient: _check_auth_tickets 2024-09-17T18:29:24.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.910+0000 7f417effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:29:24.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.910+0000 7f417effd640 1 -- 172.21.15.121:0/1351702159 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41780070d0 con 0x7f41800a1260 2024-09-17T18:29:24.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.910+0000 7f41882b6640 5 monclient: authenticate success, global_id 64163 2024-09-17T18:29:24.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.910+0000 7f41882b6640 10 monclient: _renew_subs 2024-09-17T18:29:24.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.910+0000 7f41882b6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:29:24.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.910+0000 7f41882b6640 1 -- 172.21.15.121:0/1351702159 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4180126910 con 0x7f41800a1260 2024-09-17T18:29:24.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.910+0000 7f417effd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:29:24.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.910+0000 7f417effd640 1 -- 172.21.15.121:0/1351702159 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41780074c0 con 0x7f41800a1260 2024-09-17T18:29:24.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.910+0000 7f417effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:24.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.910+0000 7f417effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:29:24.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.910+0000 7f417effd640 10 monclient: dump: 2024-09-17T18:29:24.925 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:24.925 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:24.925 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:24.925 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:24.926 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:24.926 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:24.926 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:24.926 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:24.926 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:24.926 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:24.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.910+0000 7f41882b6640 10 client.64163.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:29:24.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.910+0000 7f41882b6640 10 monclient: _renew_subs 2024-09-17T18:29:24.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.910+0000 7f41882b6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:29:24.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.910+0000 7f41882b6640 1 -- 172.21.15.121:0/1351702159 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4180127be0 con 0x7f41800a1260 2024-09-17T18:29:24.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.910+0000 7f418484f640 4 set_mon_vals no callback set 2024-09-17T18:29:24.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.910+0000 7f41882b6640 1 librados: init done 2024-09-17T18:29:24.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.911+0000 7f417effd640 1 -- 172.21.15.121:0/1351702159 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f417801a070 con 0x7f41800a1260 2024-09-17T18:29:24.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.911+0000 7f41882b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:29:24.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.911+0000 7f41882b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:29:24.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.911+0000 7f41882b6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:29:24.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.911+0000 7f41882b6640 1 -- 172.21.15.121:0/1351702159 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41800a3540 con 0x7f41800a1260 2024-09-17T18:29:24.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.911+0000 7f417effd640 1 --2- 172.21.15.121:0/1351702159 >> v2:172.21.15.173:6812/2423786037 conn(0x7f416c040170 0x7f416c042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:24.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.911+0000 7f417effd640 1 -- 172.21.15.121:0/1351702159 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(409..409 src has 247..409) ==== 5358+0+0 (secure 0 0 0) 0x7f4178052ab0 con 0x7f41800a1260 2024-09-17T18:29:24.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.911+0000 7f417effd640 10 client.64163.objecter ms_dispatch 0x7f4180000b90 osd_map(409..409 src has 247..409) 2024-09-17T18:29:24.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.911+0000 7f417effd640 3 client.64163.objecter handle_osd_map got epochs [409,409] > 0 2024-09-17T18:29:24.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.911+0000 7f417effd640 3 client.64163.objecter handle_osd_map decoding full epoch 409 2024-09-17T18:29:24.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.911+0000 7f417effd640 20 client.64163.objecter dump_active .. 0 homeless 2024-09-17T18:29:24.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.911+0000 7f4186052640 1 --2- 172.21.15.121:0/1351702159 >> v2:172.21.15.173:6812/2423786037 conn(0x7f416c040170 0x7f416c042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:24.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.911+0000 7f4186052640 10 monclient: get_auth_request con 0x7f416c040170 auth_method 0 2024-09-17T18:29:24.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.914+0000 7f417effd640 10 client.64163.objecter ms_handle_connect 0x7f416c040170 2024-09-17T18:29:24.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.914+0000 7f4186052640 1 --2- 172.21.15.121:0/1351702159 >> v2:172.21.15.173:6812/2423786037 conn(0x7f416c040170 0x7f416c042630 secure :-1 s=READY pgs=4007 cs=0 l=1 rev1=1 crypto rx=0x7f4170009bd0 tx=0x7f417002f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:24.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.916+0000 7f417effd640 1 -- 172.21.15.121:0/1351702159 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f41800a3540 con 0x7f41800a1260 2024-09-17T18:29:24.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.916+0000 7f417effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:29:24.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:24.916+0000 7f417effd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:29:25.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.155+0000 7f41882b6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:29:25.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.155+0000 7f41882b6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:29:25.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.155+0000 7f41882b6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:29:25.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.155+0000 7f41882b6640 1 -- 172.21.15.121:0/1351702159 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f41801348f0 con 0x7f41800a1260 2024-09-17T18:29:25.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.155+0000 7f417effd640 1 -- 172.21.15.121:0/1351702159 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f4178051270 con 0x7f41800a1260 2024-09-17T18:29:25.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.155+0000 7f417effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:29:25.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.155+0000 7f417effd640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:29:25.158 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:29:25.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.157+0000 7f41882b6640 10 librados: watch_flush enter 2024-09-17T18:29:25.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.157+0000 7f41882b6640 10 librados: watch_flush exit 2024-09-17T18:29:25.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.157+0000 7f41882b6640 20 client.64163.objecter shutdown clearing up homeless session... 2024-09-17T18:29:25.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.157+0000 7f41882b6640 10 client.64163.objecter successfully canceled tick 2024-09-17T18:29:25.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.157+0000 7f41882b6640 1 -- 172.21.15.121:0/1351702159 >> v2:172.21.15.173:6812/2423786037 conn(0x7f416c040170 msgr2=0x7f416c042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:25.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.157+0000 7f41882b6640 1 --2- 172.21.15.121:0/1351702159 >> v2:172.21.15.173:6812/2423786037 conn(0x7f416c040170 0x7f416c042630 secure :-1 s=READY pgs=4007 cs=0 l=1 rev1=1 crypto rx=0x7f4170009bd0 tx=0x7f417002f000 comp rx=0 tx=0).stop 2024-09-17T18:29:25.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.157+0000 7f41882b6640 10 monclient: shutdown 2024-09-17T18:29:25.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.157+0000 7f41882b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:29:25.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.157+0000 7f41882b6640 1 -- 172.21.15.121:0/1351702159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41800a1260 msgr2=0x7f4180123390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:25.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.157+0000 7f41882b6640 1 --2- 172.21.15.121:0/1351702159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41800a1260 0x7f4180123390 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f417800c3e0 tx=0x7f417800c7e0 comp rx=0 tx=0).stop 2024-09-17T18:29:25.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.157+0000 7f41882b6640 1 -- 172.21.15.121:0/1351702159 shutdown_connections 2024-09-17T18:29:25.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.157+0000 7f41882b6640 1 --2- 172.21.15.121:0/1351702159 >> v2:172.21.15.173:6812/2423786037 conn(0x7f416c040170 0x7f416c042630 unknown :-1 s=CLOSED pgs=4007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:25.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.157+0000 7f41882b6640 1 --2- 172.21.15.121:0/1351702159 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41800a0e50 0x7f41801cbfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:25.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.157+0000 7f41882b6640 1 --2- 172.21.15.121:0/1351702159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41800a1260 0x7f4180123390 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:25.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.157+0000 7f41882b6640 1 --2- 172.21.15.121:0/1351702159 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41800a0a70 0x7f41801cba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:25.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.157+0000 7f41882b6640 1 -- 172.21.15.121:0/1351702159 >> 172.21.15.121:0/1351702159 conn(0x7f41800a3b50 msgr2=0x7f4180072e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:29:25.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.157+0000 7f41882b6640 1 -- 172.21.15.121:0/1351702159 shutdown_connections 2024-09-17T18:29:25.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.157+0000 7f41882b6640 1 -- 172.21.15.121:0/1351702159 wait complete. 2024-09-17T18:29:25.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.157+0000 7f41882b6640 1 librados: shutdown 2024-09-17T18:29:25.170 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T18:29:25.171 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T18:29:25.171 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T18:29:25.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.265+0000 7ff75b7cd640 10 monclient: get_monmap_and_config 2024-09-17T18:29:25.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.265+0000 7ff75b7cd640 10 monclient: build_initial_monmap 2024-09-17T18:29:25.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.265+0000 7ff75b7cd640 10 monclient: monmap: 2024-09-17T18:29:25.269 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:29:25.269 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:25.269 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:29:25.267275+0000 2024-09-17T18:29:25.269 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:29:25.267275+0000 2024-09-17T18:29:25.269 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:29:25.269 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:25.269 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:29:25.269 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:29:25.269 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:29:25.270 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:29:25.270 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:29:25.270 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:25.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.266+0000 7ff75b7cd640 1 Processor -- start 2024-09-17T18:29:25.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.266+0000 7ff75b7cd640 1 -- start start 2024-09-17T18:29:25.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.266+0000 7ff75b7cd640 10 monclient: init 2024-09-17T18:29:25.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.266+0000 7ff75b7cd640 10 monclient: _reopen_session rank -1 2024-09-17T18:29:25.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.266+0000 7ff75b7cd640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-17T18:29:25.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.266+0000 7ff75b7cd640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7ff75409da70 0x7ff7541361c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:25.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.266+0000 7ff75b7cd640 10 monclient(hunting): picked mon.noname-c con 0x7ff75409da70 addr v2:172.21.15.173:3300/0 2024-09-17T18:29:25.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.266+0000 7ff75b7cd640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff754136700 0x7ff754136b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:25.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.266+0000 7ff75b7cd640 10 monclient(hunting): picked mon.noname-a con 0x7ff754136700 addr v2:172.21.15.121:3300/0 2024-09-17T18:29:25.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.266+0000 7ff75b7cd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff754132cc0 0x7ff754133150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:25.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.266+0000 7ff75b7cd640 10 monclient(hunting): picked mon.noname-b con 0x7ff754132cc0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:29:25.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.266+0000 7ff75b7cd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:25.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.266+0000 7ff75b7cd640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7ff75409de50 con 0x7ff754136700 2024-09-17T18:29:25.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.266+0000 7ff75b7cd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:25.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.266+0000 7ff75b7cd640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7ff7540a0e20 con 0x7ff75409da70 2024-09-17T18:29:25.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.266+0000 7ff75b7cd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:25.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.266+0000 7ff75b7cd640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff7540a2d40 con 0x7ff754132cc0 2024-09-17T18:29:25.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.266+0000 7ff75b7cd640 10 monclient(hunting): _renew_subs 2024-09-17T18:29:25.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.266+0000 7ff75b7cd640 10 monclient(hunting): authenticate will time out at 1368.983912s 2024-09-17T18:29:25.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.267+0000 7ff758d68640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff754136700 0x7ff754136b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:25.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.267+0000 7ff759d6a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff754132cc0 0x7ff754133150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:25.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.267+0000 7ff759d6a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff754132cc0 0x7ff754133150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:47016/0 (socket says 172.21.15.121:47016) 2024-09-17T18:29:25.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.267+0000 7ff758d68640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff754136700 0x7ff754136b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:57986/0 (socket says 172.21.15.121:57986) 2024-09-17T18:29:25.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.267+0000 7ff759d6a640 1 -- 172.21.15.121:0/700215516 learned_addr learned my addr 172.21.15.121:0/700215516 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:29:25.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.267+0000 7ff759d6a640 10 monclient(hunting): get_auth_request con 0x7ff754132cc0 auth_method 0 2024-09-17T18:29:25.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.267+0000 7ff759d6a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:25.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.267+0000 7ff759d6a640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:25.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.267+0000 7ff759d6a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:25.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.267+0000 7ff758d68640 10 monclient(hunting): get_auth_request con 0x7ff754136700 auth_method 0 2024-09-17T18:29:25.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.267+0000 7ff758d68640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:25.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.267+0000 7ff758d68640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:25.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.267+0000 7ff758d68640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:25.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.267+0000 7ff759d6a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:25.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.267+0000 7ff759d6a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:25.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.267+0000 7ff759d6a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:25.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.267+0000 7ff758d68640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:25.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.267+0000 7ff758d68640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:25.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.267+0000 7ff758d68640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:25.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.267+0000 7ff759569640 1 --2- 172.21.15.121:0/700215516 >> v2:172.21.15.173:3300/0 conn(0x7ff75409da70 0x7ff7541361c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:25.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.267+0000 7ff759569640 10 monclient(hunting): get_auth_request con 0x7ff75409da70 auth_method 0 2024-09-17T18:29:25.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.267+0000 7ff759569640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:25.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.267+0000 7ff759569640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:25.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.267+0000 7ff759569640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:25.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.267+0000 7ff758d68640 10 monclient(hunting): handle_auth_done global_id 44532 payload 274 2024-09-17T18:29:25.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.267+0000 7ff758d68640 1 -- 172.21.15.121:0/700215516 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff754132cc0 msgr2=0x7ff754133150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:29:25.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.267+0000 7ff758d68640 1 --2- 172.21.15.121:0/700215516 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff754132cc0 0x7ff754133150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:25.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.267+0000 7ff758d68640 1 -- 172.21.15.121:0/700215516 >> v2:172.21.15.173:3300/0 conn(0x7ff75409da70 msgr2=0x7ff7541361c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:29:25.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.267+0000 7ff758d68640 1 --2- 172.21.15.121:0/700215516 >> v2:172.21.15.173:3300/0 conn(0x7ff75409da70 0x7ff7541361c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:25.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.267+0000 7ff758d68640 10 monclient: _finish_hunting 0 2024-09-17T18:29:25.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.267+0000 7ff758d68640 1 monclient: found mon.noname-a 2024-09-17T18:29:25.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.267+0000 7ff758d68640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:29:25.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.267+0000 7ff758d68640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:29:25.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.267+0000 7ff758d68640 1 -- 172.21.15.121:0/700215516 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7540a3020 con 0x7ff754136700 2024-09-17T18:29:25.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.267+0000 7ff759d6a640 1 --2- 172.21.15.121:0/700215516 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff754132cc0 0x7ff754133150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:29:25.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.267+0000 7ff759569640 1 --2- 172.21.15.121:0/700215516 >> v2:172.21.15.173:3300/0 conn(0x7ff75409da70 0x7ff7541361c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T18:29:25.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.270+0000 7ff758d68640 1 -- 172.21.15.121:0/700215516 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff754136700 msgr2=0x7ff754136b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:25.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.270+0000 7ff758d68640 1 --2- 172.21.15.121:0/700215516 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff754136700 0x7ff754136b30 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff75409e070 tx=0x7ff74402a010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cd06420377a5493 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:25.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.270+0000 7ff74bfff640 1 -- 172.21.15.121:0/700215516 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff744037070 con 0x7ff754136700 2024-09-17T18:29:25.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.270+0000 7ff74bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:25.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.270+0000 7ff74bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:29:25.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.270+0000 7ff74bfff640 10 monclient: dump: 2024-09-17T18:29:25.280 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:25.280 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:25.281 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.281 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.281 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:25.281 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:25.281 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:25.281 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:25.281 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:25.281 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:25.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.270+0000 7ff74bfff640 10 monclient: _finish_auth 0 2024-09-17T18:29:25.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.270+0000 7ff74bfff640 10 monclient: _check_auth_tickets 2024-09-17T18:29:25.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.270+0000 7ff74bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:29:25.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.270+0000 7ff74bfff640 1 -- 172.21.15.121:0/700215516 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff744032440 con 0x7ff754136700 2024-09-17T18:29:25.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.270+0000 7ff75b7cd640 5 monclient: authenticate success, global_id 44532 2024-09-17T18:29:25.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.270+0000 7ff75b7cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:29:25.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.270+0000 7ff74bfff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:29:25.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.270+0000 7ff75b7cd640 10 monclient: get_monmap_and_config success 2024-09-17T18:29:25.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.270+0000 7ff74bfff640 1 -- 172.21.15.121:0/700215516 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff744032810 con 0x7ff754136700 2024-09-17T18:29:25.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.270+0000 7ff75b7cd640 4 set_mon_vals no callback set 2024-09-17T18:29:25.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.270+0000 7ff74bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:25.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.270+0000 7ff74bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:29:25.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.270+0000 7ff74bfff640 10 monclient: dump: 2024-09-17T18:29:25.282 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:25.282 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:25.282 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.282 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.282 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:25.282 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:25.283 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:25.283 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:25.283 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:25.283 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:25.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.270+0000 7ff75b7cd640 10 monclient: shutdown 2024-09-17T18:29:25.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.270+0000 7ff75b7cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:29:25.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.270+0000 7ff75b7cd640 1 -- 172.21.15.121:0/700215516 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff754136700 msgr2=0x7ff754136b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:25.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.270+0000 7ff75b7cd640 1 --2- 172.21.15.121:0/700215516 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff754136700 0x7ff754136b30 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff75409e070 tx=0x7ff74402a010 comp rx=0 tx=0).stop 2024-09-17T18:29:25.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.270+0000 7ff75b7cd640 1 -- 172.21.15.121:0/700215516 shutdown_connections 2024-09-17T18:29:25.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.270+0000 7ff75b7cd640 1 --2- 172.21.15.121:0/700215516 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff754132cc0 0x7ff754133150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:25.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.270+0000 7ff75b7cd640 1 --2- 172.21.15.121:0/700215516 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff754136700 0x7ff754136b30 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:25.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.270+0000 7ff75b7cd640 1 --2- 172.21.15.121:0/700215516 >> v2:172.21.15.173:3300/0 conn(0x7ff75409da70 0x7ff7541361c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:25.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.270+0000 7ff75b7cd640 1 -- 172.21.15.121:0/700215516 >> 172.21.15.121:0/700215516 conn(0x7ff7540a7830 msgr2=0x7ff7540a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:29:25.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.270+0000 7ff75b7cd640 1 -- 172.21.15.121:0/700215516 shutdown_connections 2024-09-17T18:29:25.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.270+0000 7ff75b7cd640 1 -- 172.21.15.121:0/700215516 wait complete. 2024-09-17T18:29:25.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.271+0000 7ff75b7cd640 10 monclient: build_initial_monmap 2024-09-17T18:29:25.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.271+0000 7ff75b7cd640 10 monclient: monmap: 2024-09-17T18:29:25.284 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:29:25.284 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:29:25.284 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:29:25.284 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:29:25.284 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:29:25.284 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:25.284 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:29:25.284 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:29:25.284 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:29:25.284 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:25.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.271+0000 7ff75b7cd640 1 librados: starting msgr at 2024-09-17T18:29:25.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.271+0000 7ff75b7cd640 1 librados: starting objecter 2024-09-17T18:29:25.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.271+0000 7ff75b7cd640 1 Processor -- start 2024-09-17T18:29:25.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.271+0000 7ff75b7cd640 1 -- start start 2024-09-17T18:29:25.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.271+0000 7ff75b7cd640 1 librados: setting wanted keys 2024-09-17T18:29:25.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.271+0000 7ff75b7cd640 1 librados: calling monclient init 2024-09-17T18:29:25.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.271+0000 7ff75b7cd640 10 monclient: init 2024-09-17T18:29:25.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff75b7cd640 10 monclient: _reopen_session rank -1 2024-09-17T18:29:25.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff75b7cd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T18:29:25.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff75b7cd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff75409da70 0x7ff754073700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:25.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff75b7cd640 10 monclient(hunting): picked mon.noname-b con 0x7ff75409da70 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:29:25.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff75b7cd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff754132cc0 0x7ff75409e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:25.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff75b7cd640 10 monclient(hunting): picked mon.noname-c con 0x7ff754132cc0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:29:25.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff75b7cd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff754136700 0x7ff75409eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:25.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff75b7cd640 10 monclient(hunting): picked mon.noname-a con 0x7ff754136700 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:29:25.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff75b7cd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:25.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff75b7cd640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff754133690 con 0x7ff754136700 2024-09-17T18:29:25.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff75b7cd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:25.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff75b7cd640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff7540a3930 con 0x7ff75409da70 2024-09-17T18:29:25.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff75b7cd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:25.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff75b7cd640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff754133810 con 0x7ff754132cc0 2024-09-17T18:29:25.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff75b7cd640 10 monclient(hunting): _renew_subs 2024-09-17T18:29:25.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff75b7cd640 10 monclient(hunting): authenticate will time out at 1368.989191s 2024-09-17T18:29:25.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff758d68640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff754132cc0 0x7ff75409e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:25.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff759d6a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff754136700 0x7ff75409eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:25.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff758d68640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff754132cc0 0x7ff75409e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:47032/0 (socket says 172.21.15.121:47032) 2024-09-17T18:29:25.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff759d6a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff754136700 0x7ff75409eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:57998/0 (socket says 172.21.15.121:57998) 2024-09-17T18:29:25.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff758d68640 1 -- 172.21.15.121:0/3854979752 learned_addr learned my addr 172.21.15.121:0/3854979752 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:29:25.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff758d68640 10 monclient(hunting): get_auth_request con 0x7ff754132cc0 auth_method 0 2024-09-17T18:29:25.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff758d68640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:25.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff758d68640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:25.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff758d68640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:25.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff759d6a640 10 monclient(hunting): get_auth_request con 0x7ff754136700 auth_method 0 2024-09-17T18:29:25.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff759d6a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:25.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff759d6a640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:25.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff759d6a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:25.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff758d68640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:25.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff758d68640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:25.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff758d68640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:25.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff759d6a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:25.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff759d6a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:25.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff759d6a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:25.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff759569640 1 --2- 172.21.15.121:0/3854979752 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff75409da70 0x7ff754073700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:25.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff759569640 10 monclient(hunting): get_auth_request con 0x7ff75409da70 auth_method 0 2024-09-17T18:29:25.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff759569640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:25.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff759569640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:25.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff759569640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:25.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff759569640 1 -- 172.21.15.121:0/3854979752 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff75409da70 msgr2=0x7ff754073700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T18:29:25.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff759569640 1 -- 172.21.15.121:0/3854979752 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff75409da70 msgr2=0x7ff754073700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:29:25.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff759569640 1 --2- 172.21.15.121:0/3854979752 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff75409da70 0x7ff754073700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:29:25.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff759d6a640 10 monclient(hunting): handle_auth_done global_id 44535 payload 824 2024-09-17T18:29:25.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff759569640 1 --2- 172.21.15.121:0/3854979752 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff75409da70 0x7ff754073700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:29:25.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff759d6a640 1 -- 172.21.15.121:0/3854979752 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff754132cc0 msgr2=0x7ff75409e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:25.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff759d6a640 1 --2- 172.21.15.121:0/3854979752 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff754132cc0 0x7ff75409e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:25.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff759d6a640 1 -- 172.21.15.121:0/3854979752 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff75409da70 msgr2=0x7ff754073700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:29:25.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff759d6a640 1 --2- 172.21.15.121:0/3854979752 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff75409da70 0x7ff754073700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:25.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff759d6a640 10 monclient: _finish_hunting 0 2024-09-17T18:29:25.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff759d6a640 1 monclient: found mon.noname-a 2024-09-17T18:29:25.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff759d6a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:29:25.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff759d6a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:29:25.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff759d6a640 1 -- 172.21.15.121:0/3854979752 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff754071e80 con 0x7ff754136700 2024-09-17T18:29:25.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.272+0000 7ff758d68640 1 --2- 172.21.15.121:0/3854979752 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff754132cc0 0x7ff75409e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:29:25.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.274+0000 7ff759d6a640 1 -- 172.21.15.121:0/3854979752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff754136700 msgr2=0x7ff75409eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:25.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.274+0000 7ff759d6a640 1 --2- 172.21.15.121:0/3854979752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff754136700 0x7ff75409eac0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff74c00efc0 tx=0x7ff74c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:25.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.274+0000 7ff749ffb640 10 client.?.objecter ms_handle_connect 0x7ff754136700 2024-09-17T18:29:25.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.274+0000 7ff749ffb640 10 client.?.objecter resend_mon_ops 2024-09-17T18:29:25.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.274+0000 7ff749ffb640 1 -- 172.21.15.121:0/3854979752 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff74c00cd30 con 0x7ff754136700 2024-09-17T18:29:25.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.274+0000 7ff749ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:25.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.274+0000 7ff749ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:29:25.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.274+0000 7ff749ffb640 10 monclient: dump: 2024-09-17T18:29:25.294 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:25.294 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:25.294 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.294 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.294 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:25.294 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:25.294 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:25.294 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:25.294 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:25.294 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:25.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.274+0000 7ff749ffb640 10 monclient: _finish_auth 0 2024-09-17T18:29:25.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.274+0000 7ff749ffb640 10 monclient: _check_auth_tickets 2024-09-17T18:29:25.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.274+0000 7ff749ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:29:25.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.274+0000 7ff749ffb640 1 -- 172.21.15.121:0/3854979752 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff74c003990 con 0x7ff754136700 2024-09-17T18:29:25.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.274+0000 7ff75b7cd640 5 monclient: authenticate success, global_id 44535 2024-09-17T18:29:25.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.274+0000 7ff749ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:29:25.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.274+0000 7ff75b7cd640 10 monclient: _renew_subs 2024-09-17T18:29:25.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.274+0000 7ff749ffb640 1 -- 172.21.15.121:0/3854979752 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff74c003d80 con 0x7ff754136700 2024-09-17T18:29:25.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.274+0000 7ff75b7cd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:29:25.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.274+0000 7ff75b7cd640 1 -- 172.21.15.121:0/3854979752 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff754131ed0 con 0x7ff754136700 2024-09-17T18:29:25.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.274+0000 7ff749ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:25.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.274+0000 7ff749ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:29:25.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.274+0000 7ff74b7fe640 4 set_mon_vals no callback set 2024-09-17T18:29:25.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.274+0000 7ff749ffb640 10 monclient: dump: 2024-09-17T18:29:25.296 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:25.296 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:25.296 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.296 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.296 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:25.296 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:25.296 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:25.296 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:25.296 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:25.296 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:25.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.275+0000 7ff75b7cd640 10 client.44535.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:29:25.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.275+0000 7ff75b7cd640 10 monclient: _renew_subs 2024-09-17T18:29:25.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.275+0000 7ff75b7cd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:29:25.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.275+0000 7ff75b7cd640 1 -- 172.21.15.121:0/3854979752 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff75409e240 con 0x7ff754136700 2024-09-17T18:29:25.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.275+0000 7ff75b7cd640 1 librados: init done 2024-09-17T18:29:25.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.275+0000 7ff749ffb640 1 -- 172.21.15.121:0/3854979752 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7ff74c01b030 con 0x7ff754136700 2024-09-17T18:29:25.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.275+0000 7ff75b7cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:29:25.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.275+0000 7ff75b7cd640 1 --2- 172.21.15.121:0/3854979752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff75402e5c0 0x7ff75409f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:25.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.275+0000 7ff75b7cd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:25.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.275+0000 7ff75b7cd640 1 -- 172.21.15.121:0/3854979752 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff75402e9a0 con 0x7ff75402e5c0 2024-09-17T18:29:25.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.275+0000 7ff759569640 1 --2- 172.21.15.121:0/3854979752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff75402e5c0 0x7ff75409f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:25.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.276+0000 7ff759569640 10 monclient: get_auth_request con 0x7ff75402e5c0 auth_method 0 2024-09-17T18:29:25.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.276+0000 7ff759569640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:25.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.276+0000 7ff759569640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:25.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.276+0000 7ff759569640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:25.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.276+0000 7ff749ffb640 1 --2- 172.21.15.121:0/3854979752 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff738040220 0x7ff7380426e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:25.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.276+0000 7ff759569640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:25.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.276+0000 7ff759569640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:25.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.276+0000 7ff759569640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:25.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.276+0000 7ff749ffb640 1 -- 172.21.15.121:0/3854979752 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(409..409 src has 247..409) ==== 5358+0+0 (secure 0 0 0) 0x7ff74c053aa0 con 0x7ff754136700 2024-09-17T18:29:25.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.276+0000 7ff749ffb640 10 client.44535.objecter ms_dispatch 0x7ff754000b90 osd_map(409..409 src has 247..409) 2024-09-17T18:29:25.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.276+0000 7ff749ffb640 3 client.44535.objecter handle_osd_map got epochs [409,409] > 0 2024-09-17T18:29:25.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.276+0000 7ff749ffb640 3 client.44535.objecter handle_osd_map decoding full epoch 409 2024-09-17T18:29:25.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.276+0000 7ff749ffb640 20 client.44535.objecter dump_active .. 0 homeless 2024-09-17T18:29:25.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.276+0000 7ff759569640 10 monclient(hunting): handle_auth_done global_id 44538 payload 824 2024-09-17T18:29:25.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.276+0000 7ff759569640 1 -- 172.21.15.121:0/3854979752 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff754133150 con 0x7ff75402e5c0 2024-09-17T18:29:25.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.276+0000 7ff758d68640 1 --2- 172.21.15.121:0/3854979752 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff738040220 0x7ff7380426e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:25.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.276+0000 7ff758d68640 10 monclient: get_auth_request con 0x7ff738040220 auth_method 0 2024-09-17T18:29:25.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.278+0000 7ff749ffb640 10 client.44535.objecter ms_handle_connect 0x7ff75402e5c0 2024-09-17T18:29:25.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.278+0000 7ff759569640 1 -- 172.21.15.121:0/3854979752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff75402e5c0 msgr2=0x7ff75409f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:25.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.278+0000 7ff749ffb640 10 client.44535.objecter resend_mon_ops 2024-09-17T18:29:25.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.278+0000 7ff759569640 1 --2- 172.21.15.121:0/3854979752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff75402e5c0 0x7ff75409f000 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff750010790 tx=0x7ff750010c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:25.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.278+0000 7ff749ffb640 1 -- 172.21.15.121:0/3854979752 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff75001a020 con 0x7ff75402e5c0 2024-09-17T18:29:25.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.278+0000 7ff749ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:25.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.278+0000 7ff749ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:29:25.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.278+0000 7ff749ffb640 10 monclient: dump: 2024-09-17T18:29:25.299 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:25.299 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:25.299 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.299 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.299 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:25.299 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:25.300 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:25.300 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:25.300 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:25.300 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:25.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.278+0000 7ff749ffb640 10 client.44535.objecter ms_handle_connect 0x7ff738040220 2024-09-17T18:29:25.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.278+0000 7ff758d68640 1 --2- 172.21.15.121:0/3854979752 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff738040220 0x7ff7380426e0 secure :-1 s=READY pgs=4008 cs=0 l=1 rev1=1 crypto rx=0x7ff74402a050 tx=0x7ff74402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:25.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.278+0000 7ff749ffb640 1 -- 172.21.15.121:0/3854979752 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff75000ede0 con 0x7ff75402e5c0 2024-09-17T18:29:25.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.278+0000 7ff749ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:29:25.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.278+0000 7ff749ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:29:25.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.278+0000 7ff749ffb640 1 -- 172.21.15.121:0/3854979752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff75402e5c0 msgr2=0x7ff75409f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:25.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.278+0000 7ff749ffb640 1 --2- 172.21.15.121:0/3854979752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff75402e5c0 0x7ff75409f000 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff750010790 tx=0x7ff750010c50 comp rx=0 tx=0).stop 2024-09-17T18:29:25.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.278+0000 7ff749ffb640 1 -- 172.21.15.121:0/3854979752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff75402e5c0 msgr2=0x7ff75409f000 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:29:25.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.278+0000 7ff749ffb640 1 --2- 172.21.15.121:0/3854979752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff75402e5c0 0x7ff75409f000 secure :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff750010790 tx=0x7ff750010c50 comp rx=0 tx=0).stop 2024-09-17T18:29:25.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.284+0000 7ff75b7cd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:29:25.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.284+0000 7ff75b7cd640 1 --2- 172.21.15.121:0/3854979752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff754071820 0x7ff754063d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:25.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.284+0000 7ff75b7cd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:25.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.284+0000 7ff75b7cd640 1 -- 172.21.15.121:0/3854979752 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff7541bd5b0 con 0x7ff754071820 2024-09-17T18:29:25.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.285+0000 7ff759569640 1 --2- 172.21.15.121:0/3854979752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff754071820 0x7ff754063d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:25.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.285+0000 7ff759569640 10 monclient: get_auth_request con 0x7ff754071820 auth_method 0 2024-09-17T18:29:25.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.285+0000 7ff759569640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:25.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.285+0000 7ff759569640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:25.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.285+0000 7ff759569640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:25.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.285+0000 7ff759569640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:25.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.285+0000 7ff759569640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:25.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.285+0000 7ff759569640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:25.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.285+0000 7ff759569640 10 monclient(hunting): handle_auth_done global_id 44541 payload 824 2024-09-17T18:29:25.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.285+0000 7ff759569640 1 -- 172.21.15.121:0/3854979752 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff754071c00 con 0x7ff754071820 2024-09-17T18:29:25.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.287+0000 7ff749ffb640 10 client.44535.objecter ms_handle_connect 0x7ff754071820 2024-09-17T18:29:25.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.287+0000 7ff749ffb640 10 client.44535.objecter resend_mon_ops 2024-09-17T18:29:25.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.287+0000 7ff759569640 1 -- 172.21.15.121:0/3854979752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff754071820 msgr2=0x7ff754063d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:25.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.287+0000 7ff759569640 1 --2- 172.21.15.121:0/3854979752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff754071820 0x7ff754063d50 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ff75001b000 tx=0x7ff7500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:25.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.287+0000 7ff749ffb640 1 -- 172.21.15.121:0/3854979752 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff75001a040 con 0x7ff754071820 2024-09-17T18:29:25.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.287+0000 7ff749ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:25.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.287+0000 7ff749ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:29:25.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.287+0000 7ff749ffb640 10 monclient: dump: 2024-09-17T18:29:25.302 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:25.302 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:25.302 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.302 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.302 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:25.302 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:25.302 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:25.302 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:25.303 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:25.303 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:25.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.287+0000 7ff749ffb640 1 -- 172.21.15.121:0/3854979752 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7ff750015680 con 0x7ff754071820 2024-09-17T18:29:25.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.287+0000 7ff749ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:29:25.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.287+0000 7ff749ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:29:25.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.287+0000 7ff749ffb640 1 -- 172.21.15.121:0/3854979752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff754071820 msgr2=0x7ff754063d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:25.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.287+0000 7ff749ffb640 1 --2- 172.21.15.121:0/3854979752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff754071820 0x7ff754063d50 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ff75001b000 tx=0x7ff7500046c0 comp rx=0 tx=0).stop 2024-09-17T18:29:25.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.287+0000 7ff749ffb640 1 -- 172.21.15.121:0/3854979752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff754071820 msgr2=0x7ff754063d50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:29:25.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.287+0000 7ff749ffb640 1 --2- 172.21.15.121:0/3854979752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff754071820 0x7ff754063d50 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:25.303 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:29:25.303 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:29:25.303 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:29:25.303 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "leader", 2024-09-17T18:29:25.304 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 38, 2024-09-17T18:29:25.304 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 390952, 2024-09-17T18:29:25.304 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:29:25.304 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:29:25.304 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:29:25.304 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:29:25.304 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:29:25.304 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 0, 2024-09-17T18:29:25.304 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:29:25.304 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:29:25.304 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:29:25.304 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:29:25.304 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:29:25.304 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:29:25.305 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:29:25.305 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:29:25.305 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:29:25.305 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:29:25.305 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:29:25.305 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:29:25.305 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:29:25.305 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:29:25.305 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:29:25.305 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:29:25.305 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:29:25.305 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:29:25.305 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:29:25.305 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:29:25.305 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:29:25.305 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:29:25.306 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:29:25.306 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:29:25.306 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:29:25.306 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:29:25.306 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:29:25.306 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:29:25.306 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:25.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:25.306 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:29:25.306 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:29:25.306 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:29:25.306 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:29:25.306 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:29:25.306 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:29:25.306 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:29:25.306 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:29:25.307 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:29:25.307 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:29:25.307 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:29:25.307 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:29:25.307 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:29:25.307 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:29:25.307 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:29:25.307 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:29:25.307 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:29:25.307 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:29:25.307 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:29:25.307 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:29:25.307 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:29:25.307 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:29:25.307 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:29:25.308 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:29:25.308 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:29:25.308 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:29:25.308 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:29:25.308 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:29:25.308 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:29:25.308 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:29:25.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:25.308 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:29:25.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:25.308 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:29:25.308 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:29:25.308 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:29:25.308 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:29:25.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:25.309 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:29:25.309 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:29:25.309 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:25.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:25.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:25.309 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:29:25.309 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:29:25.309 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:25.309 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:25.309 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:25.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:25.309 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:29:25.309 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:29:25.309 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:29:25.310 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:29:25.310 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:29:25.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:25.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:25.310 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:29:25.310 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:29:25.310 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:29:25.310 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:29:25.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:25.310 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:29:25.310 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:29:25.310 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:25.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:25.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:25.310 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:29:25.311 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:29:25.311 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:25.311 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:25.311 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:25.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:25.311 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:29:25.311 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:29:25.311 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:29:25.311 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:29:25.311 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:29:25.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:25.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:25.311 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:29:25.311 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:29:25.311 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:29:25.311 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:29:25.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:25.312 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:29:25.312 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:29:25.312 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:25.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:25.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:25.312 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:29:25.312 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:29:25.312 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:25.312 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:25.312 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:25.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:25.312 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:29:25.312 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:29:25.312 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:29:25.312 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:29:25.313 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:29:25.313 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:25.313 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:25.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:25.313 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:29:25.313 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:29:25.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:25.313 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:29:25.313 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:29:25.313 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:29:25.313 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:25.313 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:29:25.313 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:29:25.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:25.313 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:29:25.313 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:29:25.314 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 4 2024-09-17T18:29:25.314 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:25.314 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:29:25.314 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:29:25.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:25.314 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:29:25.314 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:29:25.314 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 6 2024-09-17T18:29:25.314 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:25.314 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:25.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:25.314 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:29:25.314 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:29:25.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.287+0000 7ff75b7cd640 10 librados: watch_flush enter 2024-09-17T18:29:25.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.288+0000 7ff75b7cd640 10 librados: watch_flush exit 2024-09-17T18:29:25.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.288+0000 7ff75b7cd640 20 client.44535.objecter shutdown clearing up homeless session... 2024-09-17T18:29:25.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.288+0000 7ff75b7cd640 10 client.44535.objecter successfully canceled tick 2024-09-17T18:29:25.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.288+0000 7ff75b7cd640 1 -- 172.21.15.121:0/3854979752 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff738040220 msgr2=0x7ff7380426e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:25.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.288+0000 7ff75b7cd640 1 --2- 172.21.15.121:0/3854979752 >> v2:172.21.15.173:6812/2423786037 conn(0x7ff738040220 0x7ff7380426e0 secure :-1 s=READY pgs=4008 cs=0 l=1 rev1=1 crypto rx=0x7ff74402a050 tx=0x7ff74402e000 comp rx=0 tx=0).stop 2024-09-17T18:29:25.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.288+0000 7ff75b7cd640 10 monclient: shutdown 2024-09-17T18:29:25.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.288+0000 7ff75b7cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:29:25.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.288+0000 7ff75b7cd640 1 -- 172.21.15.121:0/3854979752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff754136700 msgr2=0x7ff75409eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:25.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.288+0000 7ff75b7cd640 1 --2- 172.21.15.121:0/3854979752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff754136700 0x7ff75409eac0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff74c00efc0 tx=0x7ff74c00c490 comp rx=0 tx=0).stop 2024-09-17T18:29:25.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.288+0000 7ff759d6a640 1 -- 172.21.15.121:0/3854979752 reap_dead start 2024-09-17T18:29:25.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.288+0000 7ff759d6a640 1 -- 172.21.15.121:0/3854979752 reap_dead start 2024-09-17T18:29:25.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.288+0000 7ff75b7cd640 1 -- 172.21.15.121:0/3854979752 shutdown_connections 2024-09-17T18:29:25.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.288+0000 7ff75b7cd640 1 -- 172.21.15.121:0/3854979752 >> 172.21.15.121:0/3854979752 conn(0x7ff7540a7830 msgr2=0x7ff7540a54f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:29:25.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.288+0000 7ff75b7cd640 1 -- 172.21.15.121:0/3854979752 shutdown_connections 2024-09-17T18:29:25.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.288+0000 7ff75b7cd640 1 -- 172.21.15.121:0/3854979752 wait complete. 2024-09-17T18:29:25.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.288+0000 7ff75b7cd640 1 librados: shutdown 2024-09-17T18:29:25.317 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:29:25.317 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T18:29:25.317 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T18:29:25.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.413+0000 7faabbc72640 10 monclient: get_monmap_and_config 2024-09-17T18:29:25.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.413+0000 7faabbc72640 10 monclient: build_initial_monmap 2024-09-17T18:29:25.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.413+0000 7faabbc72640 10 monclient: monmap: 2024-09-17T18:29:25.416 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:29:25.416 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:25.416 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:29:25.414620+0000 2024-09-17T18:29:25.416 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:29:25.414620+0000 2024-09-17T18:29:25.416 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:29:25.416 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:25.416 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:29:25.416 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:29:25.416 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:29:25.416 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:29:25.416 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:29:25.416 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:25.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.413+0000 7faabbc72640 1 Processor -- start 2024-09-17T18:29:25.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.413+0000 7faabbc72640 1 -- start start 2024-09-17T18:29:25.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.413+0000 7faabbc72640 10 monclient: init 2024-09-17T18:29:25.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.414+0000 7faabbc72640 10 monclient: _reopen_session rank -1 2024-09-17T18:29:25.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.414+0000 7faabbc72640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-17T18:29:25.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.414+0000 7faabbc72640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faab40700f0 0x7faab4134e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:25.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.414+0000 7faabbc72640 10 monclient(hunting): picked mon.noname-b con 0x7faab40700f0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:29:25.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.414+0000 7faabbc72640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faab409e810 addr v1:172.21.15.173:6789/0 2024-09-17T18:29:25.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.414+0000 7faabbc72640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7faab409ec30 0x7faab4138a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:25.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.414+0000 7faabbc72640 10 monclient(hunting): picked mon.noname-a con 0x7faab409ec30 addr v2:172.21.15.121:3300/0 2024-09-17T18:29:25.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.414+0000 7faabbc72640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faab409ca60 con 0x7faab409e810 2024-09-17T18:29:25.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.414+0000 7faabbc72640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:25.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.414+0000 7faabbc72640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7faab40704d0 con 0x7faab409ec30 2024-09-17T18:29:25.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.414+0000 7faabbc72640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:25.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.414+0000 7faabbc72640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7faab409cd20 con 0x7faab40700f0 2024-09-17T18:29:25.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.414+0000 7faabbc72640 10 monclient(hunting): _renew_subs 2024-09-17T18:29:25.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.414+0000 7faabbc72640 10 monclient(hunting): authenticate will time out at 1369.131338s 2024-09-17T18:29:25.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.414+0000 7faab9a0e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faab40700f0 0x7faab4134e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:25.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.414+0000 7faab9a0e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faab40700f0 0x7faab4134e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:47034/0 (socket says 172.21.15.121:47034) 2024-09-17T18:29:25.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.414+0000 7faab9a0e640 1 -- 172.21.15.121:0/3909275561 learned_addr learned my addr 172.21.15.121:0/3909275561 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:29:25.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.414+0000 7faab9a0e640 10 monclient(hunting): get_auth_request con 0x7faab40700f0 auth_method 0 2024-09-17T18:29:25.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.414+0000 7faab9a0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:25.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.414+0000 7faab9a0e640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:25.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.414+0000 7faab9a0e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:25.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.414+0000 7faaba20f640 1 --2- 172.21.15.121:0/3909275561 >> v2:172.21.15.121:3300/0 conn(0x7faab409ec30 0x7faab4138a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:25.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.414+0000 7faaba20f640 10 monclient(hunting): get_auth_request con 0x7faab409ec30 auth_method 0 2024-09-17T18:29:25.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.414+0000 7faaba20f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:25.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.414+0000 7faaba20f640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:25.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.414+0000 7faaba20f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:25.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.414+0000 7faab9a0e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:25.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.414+0000 7faab9a0e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:25.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.414+0000 7faab9a0e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:25.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.414+0000 7faaba20f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:25.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.414+0000 7faaba20f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:25.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.414+0000 7faaba20f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:25.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.414+0000 7faab9a0e640 10 monclient(hunting): handle_auth_done global_id 64172 payload 274 2024-09-17T18:29:25.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.414+0000 7faab9a0e640 1 -- 172.21.15.121:0/3909275561 >> v2:172.21.15.121:3300/0 conn(0x7faab409ec30 msgr2=0x7faab4138a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:29:25.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.414+0000 7faab9a0e640 1 --2- 172.21.15.121:0/3909275561 >> v2:172.21.15.121:3300/0 conn(0x7faab409ec30 0x7faab4138a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:25.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.414+0000 7faab9a0e640 1 -- 172.21.15.121:0/3909275561 >> v1:172.21.15.173:6789/0 conn(0x7faab409e810 legacy=0x7faab4138360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:29:25.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.414+0000 7faab9a0e640 10 monclient: _finish_hunting 0 2024-09-17T18:29:25.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.414+0000 7faab9a0e640 1 monclient: found mon.noname-b 2024-09-17T18:29:25.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.414+0000 7faab9a0e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:29:25.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.414+0000 7faab9a0e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:29:25.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.414+0000 7faab9a0e640 1 -- 172.21.15.121:0/3909275561 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faab409d000 con 0x7faab40700f0 2024-09-17T18:29:25.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.414+0000 7faaba20f640 1 --2- 172.21.15.121:0/3909275561 >> v2:172.21.15.121:3300/0 conn(0x7faab409ec30 0x7faab4138a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:29:25.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.417+0000 7faab9a0e640 1 -- 172.21.15.121:0/3909275561 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faab40700f0 msgr2=0x7faab4134e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:25.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.417+0000 7faab9a0e640 1 --2- 172.21.15.121:0/3909275561 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faab40700f0 0x7faab4134e50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7faab0009870 tx=0x7faab002a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcdd28c943d0bc42 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:25.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.421+0000 7faab8a0c640 1 -- 172.21.15.121:0/3909275561 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab0037070 con 0x7faab40700f0 2024-09-17T18:29:25.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.421+0000 7faab8a0c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:25.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.421+0000 7faab8a0c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:29:25.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.421+0000 7faab8a0c640 10 monclient: dump: 2024-09-17T18:29:25.428 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:25.428 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:25.428 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.428 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.428 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:25.429 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:25.429 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:25.429 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:25.429 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:25.429 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:25.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.421+0000 7faab8a0c640 10 monclient: _finish_auth 0 2024-09-17T18:29:25.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.421+0000 7faab8a0c640 10 monclient: _check_auth_tickets 2024-09-17T18:29:25.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.421+0000 7faab8a0c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:29:25.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.421+0000 7faab8a0c640 1 -- 172.21.15.121:0/3909275561 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faab0032640 con 0x7faab40700f0 2024-09-17T18:29:25.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.421+0000 7faabbc72640 5 monclient: authenticate success, global_id 64172 2024-09-17T18:29:25.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.421+0000 7faabbc72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:29:25.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.421+0000 7faab8a0c640 10 monclient: handle_config config(0 keys) 2024-09-17T18:29:25.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.421+0000 7faabbc72640 10 monclient: get_monmap_and_config success 2024-09-17T18:29:25.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.421+0000 7faab8a0c640 1 -- 172.21.15.121:0/3909275561 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab0032a10 con 0x7faab40700f0 2024-09-17T18:29:25.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.421+0000 7faab8a0c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:25.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.421+0000 7faabbc72640 4 set_mon_vals no callback set 2024-09-17T18:29:25.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.421+0000 7faab8a0c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:29:25.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.421+0000 7faab8a0c640 10 monclient: dump: 2024-09-17T18:29:25.430 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:25.430 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:25.430 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.430 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.430 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:25.430 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:25.430 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:25.430 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:25.430 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:25.430 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:25.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.422+0000 7faabbc72640 10 monclient: shutdown 2024-09-17T18:29:25.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.422+0000 7faabbc72640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:29:25.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.422+0000 7faabbc72640 1 -- 172.21.15.121:0/3909275561 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faab40700f0 msgr2=0x7faab4134e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:25.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.422+0000 7faabbc72640 1 --2- 172.21.15.121:0/3909275561 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faab40700f0 0x7faab4134e50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7faab0009870 tx=0x7faab002a130 comp rx=0 tx=0).stop 2024-09-17T18:29:25.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.422+0000 7faabbc72640 1 -- 172.21.15.121:0/3909275561 shutdown_connections 2024-09-17T18:29:25.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.422+0000 7faabbc72640 1 --2- 172.21.15.121:0/3909275561 >> v2:172.21.15.121:3300/0 conn(0x7faab409ec30 0x7faab4138a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:25.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.422+0000 7faabbc72640 1 --2- 172.21.15.121:0/3909275561 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faab40700f0 0x7faab4134e50 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:25.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.422+0000 7faabbc72640 1 -- 172.21.15.121:0/3909275561 >> 172.21.15.121:0/3909275561 conn(0x7faab40ab830 msgr2=0x7faab40a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:29:25.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.422+0000 7faabbc72640 1 -- 172.21.15.121:0/3909275561 shutdown_connections 2024-09-17T18:29:25.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.422+0000 7faabbc72640 1 -- 172.21.15.121:0/3909275561 wait complete. 2024-09-17T18:29:25.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.423+0000 7faabbc72640 10 monclient: build_initial_monmap 2024-09-17T18:29:25.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.423+0000 7faabbc72640 10 monclient: monmap: 2024-09-17T18:29:25.431 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:29:25.432 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:29:25.432 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:29:25.432 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:29:25.432 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:29:25.432 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:25.432 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:29:25.432 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:29:25.432 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:29:25.432 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:25.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.423+0000 7faabbc72640 1 librados: starting msgr at 2024-09-17T18:29:25.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.423+0000 7faabbc72640 1 librados: starting objecter 2024-09-17T18:29:25.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.423+0000 7faabbc72640 1 Processor -- start 2024-09-17T18:29:25.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.423+0000 7faabbc72640 1 -- start start 2024-09-17T18:29:25.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.423+0000 7faabbc72640 1 librados: setting wanted keys 2024-09-17T18:29:25.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.423+0000 7faabbc72640 1 librados: calling monclient init 2024-09-17T18:29:25.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.423+0000 7faabbc72640 10 monclient: init 2024-09-17T18:29:25.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.423+0000 7faabbc72640 10 monclient: _reopen_session rank -1 2024-09-17T18:29:25.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.423+0000 7faabbc72640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T18:29:25.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.423+0000 7faabbc72640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faab40700f0 0x7faab4131bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:25.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.423+0000 7faabbc72640 10 monclient(hunting): picked mon.noname-b con 0x7faab40700f0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:29:25.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.423+0000 7faabbc72640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faab409e810 0x7faab4132100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:25.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.423+0000 7faabbc72640 10 monclient(hunting): picked mon.noname-a con 0x7faab409e810 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:29:25.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.423+0000 7faabbc72640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faab409ec30 0x7faab40a0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:25.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.423+0000 7faabbc72640 10 monclient(hunting): picked mon.noname-c con 0x7faab409ec30 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:29:25.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.423+0000 7faabbc72640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:25.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.423+0000 7faabbc72640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7faab4135550 con 0x7faab409e810 2024-09-17T18:29:25.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.423+0000 7faabbc72640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:25.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.423+0000 7faabbc72640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7faab41353d0 con 0x7faab40700f0 2024-09-17T18:29:25.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.423+0000 7faabbc72640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:25.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.423+0000 7faabbc72640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7faab4138fb0 con 0x7faab409ec30 2024-09-17T18:29:25.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.423+0000 7faabbc72640 10 monclient(hunting): _renew_subs 2024-09-17T18:29:25.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.423+0000 7faabbc72640 10 monclient(hunting): authenticate will time out at 1369.140858s 2024-09-17T18:29:25.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.423+0000 7faaba20f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faab409ec30 0x7faab40a0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:25.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.423+0000 7faab920d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faab409e810 0x7faab4132100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:25.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.423+0000 7faaba20f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faab409ec30 0x7faab40a0210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:47044/0 (socket says 172.21.15.121:47044) 2024-09-17T18:29:25.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.423+0000 7faab920d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faab409e810 0x7faab4132100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58048/0 (socket says 172.21.15.121:58048) 2024-09-17T18:29:25.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.423+0000 7faaba20f640 1 -- 172.21.15.121:0/2550152643 learned_addr learned my addr 172.21.15.121:0/2550152643 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:29:25.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.423+0000 7faaba20f640 10 monclient(hunting): get_auth_request con 0x7faab409ec30 auth_method 0 2024-09-17T18:29:25.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.423+0000 7faaba20f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:25.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.423+0000 7faaba20f640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:25.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.423+0000 7faaba20f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:25.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.424+0000 7faab920d640 10 monclient(hunting): get_auth_request con 0x7faab409e810 auth_method 0 2024-09-17T18:29:25.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.424+0000 7faab920d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:25.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.424+0000 7faab920d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:25.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.424+0000 7faab920d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:25.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.424+0000 7faab9a0e640 1 --2- 172.21.15.121:0/2550152643 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faab40700f0 0x7faab4131bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:25.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.424+0000 7faab920d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:25.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.424+0000 7faab920d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:25.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.424+0000 7faab920d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:25.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.424+0000 7faaba20f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:25.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.424+0000 7faaba20f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:25.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.424+0000 7faaba20f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:25.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.424+0000 7faab9a0e640 10 monclient(hunting): get_auth_request con 0x7faab40700f0 auth_method 0 2024-09-17T18:29:25.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.424+0000 7faab9a0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:25.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.424+0000 7faab9a0e640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:25.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.424+0000 7faab9a0e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:25.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.424+0000 7faab9a0e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:25.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.424+0000 7faab9a0e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:25.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.424+0000 7faab9a0e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:25.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.424+0000 7faab920d640 10 monclient(hunting): handle_auth_done global_id 44547 payload 824 2024-09-17T18:29:25.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.424+0000 7faab920d640 1 -- 172.21.15.121:0/2550152643 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faab409ec30 msgr2=0x7faab40a0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:25.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.424+0000 7faab920d640 1 --2- 172.21.15.121:0/2550152643 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faab409ec30 0x7faab40a0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:25.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.424+0000 7faab920d640 1 -- 172.21.15.121:0/2550152643 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faab40700f0 msgr2=0x7faab4131bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:25.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.424+0000 7faab920d640 1 --2- 172.21.15.121:0/2550152643 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faab40700f0 0x7faab4131bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:25.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.424+0000 7faab920d640 10 monclient: _finish_hunting 0 2024-09-17T18:29:25.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.424+0000 7faab920d640 1 monclient: found mon.noname-a 2024-09-17T18:29:25.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.424+0000 7faab920d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:29:25.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.424+0000 7faab920d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:29:25.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.424+0000 7faab920d640 1 -- 172.21.15.121:0/2550152643 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faab4139450 con 0x7faab409e810 2024-09-17T18:29:25.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.424+0000 7faaba20f640 1 --2- 172.21.15.121:0/2550152643 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faab409ec30 0x7faab40a0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:29:25.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.426+0000 7faab920d640 1 -- 172.21.15.121:0/2550152643 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faab409e810 msgr2=0x7faab4132100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:25.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.426+0000 7faab920d640 1 --2- 172.21.15.121:0/2550152643 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faab409e810 0x7faab4132100 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7faaa400c410 tx=0x7faaa400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:25.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.426+0000 7faaaa7fc640 10 client.?.objecter ms_handle_connect 0x7faab409e810 2024-09-17T18:29:25.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.426+0000 7faaaa7fc640 10 client.?.objecter resend_mon_ops 2024-09-17T18:29:25.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.426+0000 7faaaa7fc640 1 -- 172.21.15.121:0/2550152643 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa4017070 con 0x7faab409e810 2024-09-17T18:29:25.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.426+0000 7faaaa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:25.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.426+0000 7faaaa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:29:25.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.426+0000 7faaaa7fc640 10 monclient: dump: 2024-09-17T18:29:25.441 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:25.442 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:25.442 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.442 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.442 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:25.442 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:25.442 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:25.442 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:25.442 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:25.442 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:25.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.426+0000 7faaaa7fc640 10 monclient: _finish_auth 0 2024-09-17T18:29:25.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.426+0000 7faaaa7fc640 10 monclient: _check_auth_tickets 2024-09-17T18:29:25.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.426+0000 7faaaa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:29:25.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.426+0000 7faaaa7fc640 1 -- 172.21.15.121:0/2550152643 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faaa4005230 con 0x7faab409e810 2024-09-17T18:29:25.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.426+0000 7faabbc72640 5 monclient: authenticate success, global_id 44547 2024-09-17T18:29:25.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.426+0000 7faaaa7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:29:25.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.426+0000 7faabbc72640 10 monclient: _renew_subs 2024-09-17T18:29:25.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.426+0000 7faaaa7fc640 1 -- 172.21.15.121:0/2550152643 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa40117a0 con 0x7faab409e810 2024-09-17T18:29:25.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.426+0000 7faabbc72640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:29:25.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.426+0000 7faabbc72640 1 -- 172.21.15.121:0/2550152643 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faab40a1be0 con 0x7faab409e810 2024-09-17T18:29:25.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.426+0000 7faaaa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:25.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.426+0000 7faaaa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:29:25.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.426+0000 7faaaa7fc640 10 monclient: dump: 2024-09-17T18:29:25.443 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:25.443 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:25.443 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.443 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.443 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:25.443 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:25.443 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:25.443 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:25.444 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:25.444 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:25.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.426+0000 7faaabfff640 4 set_mon_vals no callback set 2024-09-17T18:29:25.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.426+0000 7faabbc72640 10 client.44547.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:29:25.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.426+0000 7faabbc72640 10 monclient: _renew_subs 2024-09-17T18:29:25.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.426+0000 7faabbc72640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:29:25.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.426+0000 7faabbc72640 1 -- 172.21.15.121:0/2550152643 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faab4132640 con 0x7faab409e810 2024-09-17T18:29:25.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.426+0000 7faabbc72640 1 librados: init done 2024-09-17T18:29:25.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.427+0000 7faab920d640 1 --2- 172.21.15.121:0/2550152643 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faab409e810 0x7faab4132100 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7faaa400c410 tx=0x7faaa400c8d0 comp rx=0 tx=0).queue_received will delay after 1.7266e+09 on 0x7faaa40529b0 osd_map(410..410 src has 247..410) 2024-09-17T18:29:25.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.427+0000 7faaaa7fc640 1 -- 172.21.15.121:0/2550152643 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7faaa401a070 con 0x7faab409e810 2024-09-17T18:29:25.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.427+0000 7faabbc72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:29:25.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.427+0000 7faabbc72640 1 --2- 172.21.15.121:0/2550152643 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faab4139940 0x7faab40a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:25.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.427+0000 7faabbc72640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:25.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.427+0000 7faabbc72640 1 -- 172.21.15.121:0/2550152643 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7faab4134e50 con 0x7faab4139940 2024-09-17T18:29:25.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.427+0000 7faab9a0e640 1 --2- 172.21.15.121:0/2550152643 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faab4139940 0x7faab40a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:25.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.427+0000 7faab9a0e640 10 monclient: get_auth_request con 0x7faab4139940 auth_method 0 2024-09-17T18:29:25.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.427+0000 7faab9a0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:25.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.427+0000 7faab9a0e640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:25.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.427+0000 7faab9a0e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:25.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.427+0000 7faab9a0e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:25.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.427+0000 7faab9a0e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:25.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.427+0000 7faaaa7fc640 1 --2- 172.21.15.121:0/2550152643 >> v2:172.21.15.173:6812/2423786037 conn(0x7faa980401c0 0x7faa98042680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:25.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.427+0000 7faab9a0e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:25.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.427+0000 7faab9a0e640 10 monclient(hunting): handle_auth_done global_id 64178 payload 824 2024-09-17T18:29:25.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.427+0000 7faab9a0e640 1 -- 172.21.15.121:0/2550152643 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faab40008d0 con 0x7faab4139940 2024-09-17T18:29:25.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.427+0000 7faaba20f640 1 --2- 172.21.15.121:0/2550152643 >> v2:172.21.15.173:6812/2423786037 conn(0x7faa980401c0 0x7faa98042680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:25.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.428+0000 7faaba20f640 10 monclient: get_auth_request con 0x7faa980401c0 auth_method 0 2024-09-17T18:29:25.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.430+0000 7faab9a0e640 1 -- 172.21.15.121:0/2550152643 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faab4139940 msgr2=0x7faab40a0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:25.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.430+0000 7faaaa7fc640 10 client.44547.objecter ms_handle_connect 0x7faab4139940 2024-09-17T18:29:25.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.430+0000 7faab9a0e640 1 --2- 172.21.15.121:0/2550152643 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faab4139940 0x7faab40a0750 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7faab4070720 tx=0x7faab00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:25.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.430+0000 7faaaa7fc640 10 client.44547.objecter resend_mon_ops 2024-09-17T18:29:25.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.430+0000 7faaaa7fc640 1 -- 172.21.15.121:0/2550152643 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab0037040 con 0x7faab4139940 2024-09-17T18:29:25.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.430+0000 7faaaa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:25.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.430+0000 7faaaa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:29:25.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.430+0000 7faaaa7fc640 10 monclient: dump: 2024-09-17T18:29:25.446 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:25.446 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:25.446 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.446 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.446 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:25.446 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:25.446 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:25.446 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:25.446 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:25.446 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:25.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.430+0000 7faaaa7fc640 1 -- 172.21.15.121:0/2550152643 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7faab003c640 con 0x7faab4139940 2024-09-17T18:29:25.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.430+0000 7faaaa7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:29:25.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.430+0000 7faaaa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:29:25.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.430+0000 7faaaa7fc640 1 -- 172.21.15.121:0/2550152643 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faab4139940 msgr2=0x7faab40a0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:25.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.430+0000 7faaaa7fc640 1 --2- 172.21.15.121:0/2550152643 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faab4139940 0x7faab40a0750 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7faab4070720 tx=0x7faab00046c0 comp rx=0 tx=0).stop 2024-09-17T18:29:25.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.430+0000 7faaba20f640 1 --2- 172.21.15.121:0/2550152643 >> v2:172.21.15.173:6812/2423786037 conn(0x7faa980401c0 0x7faa98042680 secure :-1 s=READY pgs=4009 cs=0 l=1 rev1=1 crypto rx=0x7faaac009bd0 tx=0x7faaac00b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:25.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.430+0000 7faaaa7fc640 1 -- 172.21.15.121:0/2550152643 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faab4139940 msgr2=0x7faab40a0750 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:29:25.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.430+0000 7faaaa7fc640 1 --2- 172.21.15.121:0/2550152643 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faab4139940 0x7faab40a0750 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:25.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.430+0000 7faaaa7fc640 10 client.44547.objecter ms_handle_connect 0x7faa980401c0 2024-09-17T18:29:25.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.436+0000 7faabbc72640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:29:25.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.436+0000 7faabbc72640 1 --2- 172.21.15.121:0/2550152643 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faab40a0c90 0x7faab40a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:25.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.436+0000 7faabbc72640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:25.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.436+0000 7faabbc72640 1 -- 172.21.15.121:0/2550152643 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7faab413b1c0 con 0x7faab40a0c90 2024-09-17T18:29:25.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.436+0000 7faab9a0e640 1 --2- 172.21.15.121:0/2550152643 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faab40a0c90 0x7faab40a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:25.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.436+0000 7faab9a0e640 10 monclient: get_auth_request con 0x7faab40a0c90 auth_method 0 2024-09-17T18:29:25.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.436+0000 7faab9a0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:25.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.436+0000 7faab9a0e640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:25.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.436+0000 7faab9a0e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:25.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.436+0000 7faab9a0e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:25.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.436+0000 7faab9a0e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:25.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.436+0000 7faab9a0e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:25.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.437+0000 7faab9a0e640 10 monclient(hunting): handle_auth_done global_id 64181 payload 824 2024-09-17T18:29:25.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.437+0000 7faab9a0e640 1 -- 172.21.15.121:0/2550152643 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7faab40a15e0 con 0x7faab40a0c90 2024-09-17T18:29:25.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.439+0000 7faab9a0e640 1 -- 172.21.15.121:0/2550152643 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faab40a0c90 msgr2=0x7faab40a10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:25.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.439+0000 7faab9a0e640 1 --2- 172.21.15.121:0/2550152643 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faab40a0c90 0x7faab40a10a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7faab00090b0 tx=0x7faab0008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:25.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.439+0000 7faaaa7fc640 10 client.44547.objecter ms_handle_connect 0x7faab40a0c90 2024-09-17T18:29:25.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.439+0000 7faaaa7fc640 10 client.44547.objecter resend_mon_ops 2024-09-17T18:29:25.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.439+0000 7faaaa7fc640 1 -- 172.21.15.121:0/2550152643 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab0037030 con 0x7faab40a0c90 2024-09-17T18:29:25.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.439+0000 7faaaa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:25.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.439+0000 7faaaa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:29:25.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.439+0000 7faaaa7fc640 10 monclient: dump: 2024-09-17T18:29:25.450 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:25.450 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:25.450 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.450 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.450 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:25.450 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:25.450 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:25.450 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:25.450 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:25.450 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:25.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.439+0000 7faaaa7fc640 1 -- 172.21.15.121:0/2550152643 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7faab0047b70 con 0x7faab40a0c90 2024-09-17T18:29:25.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.439+0000 7faaaa7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:29:25.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.439+0000 7faaaa7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:29:25.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.439+0000 7faaaa7fc640 1 -- 172.21.15.121:0/2550152643 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faab40a0c90 msgr2=0x7faab40a10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:25.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.439+0000 7faaaa7fc640 1 --2- 172.21.15.121:0/2550152643 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faab40a0c90 0x7faab40a10a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7faab00090b0 tx=0x7faab0008030 comp rx=0 tx=0).stop 2024-09-17T18:29:25.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.439+0000 7faaaa7fc640 1 -- 172.21.15.121:0/2550152643 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faab40a0c90 msgr2=0x7faab40a10a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:29:25.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.439+0000 7faaaa7fc640 1 --2- 172.21.15.121:0/2550152643 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faab40a0c90 0x7faab40a10a0 secure :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7faab00090b0 tx=0x7faab0008030 comp rx=0 tx=0).stop 2024-09-17T18:29:25.451 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:29:25.451 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:29:25.451 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:29:25.451 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:29:25.451 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 38, 2024-09-17T18:29:25.451 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 144172, 2024-09-17T18:29:25.451 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:29:25.451 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:29:25.451 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:29:25.452 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:29:25.452 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:29:25.452 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 0, 2024-09-17T18:29:25.452 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:29:25.452 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:29:25.452 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:29:25.452 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:29:25.452 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:29:25.452 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:29:25.452 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:29:25.452 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:29:25.452 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:29:25.452 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:29:25.452 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:29:25.452 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:29:25.453 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:29:25.453 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:29:25.453 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:29:25.453 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:29:25.453 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:29:25.453 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:29:25.453 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:29:25.453 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:29:25.453 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:29:25.453 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:29:25.453 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:29:25.453 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:29:25.453 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:29:25.453 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:29:25.453 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:29:25.453 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:29:25.454 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:25.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:25.454 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:29:25.454 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:29:25.454 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:29:25.454 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:29:25.454 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:29:25.454 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:29:25.454 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:29:25.454 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:29:25.454 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:29:25.454 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:29:25.454 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:29:25.454 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:29:25.454 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:29:25.454 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:29:25.455 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:29:25.455 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:29:25.455 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:29:25.455 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:29:25.455 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:29:25.455 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:29:25.455 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:29:25.455 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:29:25.455 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:29:25.455 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:29:25.455 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:29:25.455 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:29:25.455 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:29:25.455 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:29:25.455 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:29:25.455 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:29:25.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:25.456 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:29:25.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:25.456 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:29:25.456 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:29:25.456 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:29:25.456 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:29:25.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:25.456 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:29:25.456 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:29:25.456 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:25.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:25.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:25.456 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:29:25.456 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:29:25.456 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:25.457 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:25.457 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:25.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:25.457 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:29:25.457 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:29:25.457 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:29:25.457 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:29:25.457 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:29:25.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:25.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:25.457 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:29:25.457 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:29:25.457 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:29:25.457 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:29:25.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:25.458 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:29:25.458 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:29:25.458 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:25.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:25.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:25.458 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:29:25.458 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:29:25.458 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:25.458 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:25.458 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:25.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:25.458 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:29:25.458 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:29:25.458 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:29:25.458 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:29:25.459 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:29:25.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:25.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:25.459 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:29:25.459 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:29:25.459 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:29:25.459 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:29:25.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:25.459 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:29:25.459 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:29:25.459 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:25.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:25.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:25.459 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:29:25.459 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:29:25.459 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:25.460 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:25.460 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:25.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:25.460 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:29:25.460 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:29:25.460 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:29:25.460 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:29:25.460 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:29:25.460 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:25.460 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:25.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:25.460 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:29:25.460 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:29:25.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:25.460 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:29:25.461 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:29:25.461 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:29:25.461 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:25.461 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:29:25.461 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:29:25.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:25.461 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:29:25.461 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:29:25.461 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-17T18:29:25.461 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:25.461 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:29:25.461 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:29:25.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:25.461 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:29:25.461 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:29:25.461 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-17T18:29:25.462 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:25.462 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:29:25.462 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": [ 2024-09-17T18:29:25.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:25.462 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:29:25.462 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:29:25.462 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:29:25.462 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:25.462 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:25.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:25.462 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:29:25.462 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:29:25.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.440+0000 7faabbc72640 10 librados: watch_flush enter 2024-09-17T18:29:25.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.440+0000 7faabbc72640 10 librados: watch_flush exit 2024-09-17T18:29:25.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.440+0000 7faabbc72640 20 client.44547.objecter shutdown clearing up homeless session... 2024-09-17T18:29:25.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.440+0000 7faabbc72640 10 client.44547.objecter successfully canceled tick 2024-09-17T18:29:25.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.440+0000 7faabbc72640 1 -- 172.21.15.121:0/2550152643 >> v2:172.21.15.173:6812/2423786037 conn(0x7faa980401c0 msgr2=0x7faa98042680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:25.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.440+0000 7faabbc72640 1 --2- 172.21.15.121:0/2550152643 >> v2:172.21.15.173:6812/2423786037 conn(0x7faa980401c0 0x7faa98042680 secure :-1 s=READY pgs=4009 cs=0 l=1 rev1=1 crypto rx=0x7faaac009bd0 tx=0x7faaac00b040 comp rx=0 tx=0).stop 2024-09-17T18:29:25.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.440+0000 7faabbc72640 10 monclient: shutdown 2024-09-17T18:29:25.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.440+0000 7faabbc72640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:29:25.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.440+0000 7faabbc72640 1 -- 172.21.15.121:0/2550152643 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faab409e810 msgr2=0x7faab4132100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:25.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.440+0000 7faabbc72640 1 --2- 172.21.15.121:0/2550152643 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faab409e810 0x7faab4132100 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7faaa400c410 tx=0x7faaa400c8d0 comp rx=0 tx=0).stop 2024-09-17T18:29:25.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.440+0000 7faaba20f640 1 -- 172.21.15.121:0/2550152643 reap_dead start 2024-09-17T18:29:25.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.440+0000 7faaaa7fc640 1 -- 172.21.15.121:0/2550152643 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(410..410 src has 247..410) ==== 5358+0+0 (unknown 0 0 0) 0x7faaa40529b0 con 0x7faab409e810 2024-09-17T18:29:25.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.440+0000 7faaaa7fc640 10 client.44547.objecter ms_dispatch 0x7faab4000b90 osd_map(410..410 src has 247..410) 2024-09-17T18:29:25.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.440+0000 7faaaa7fc640 10 librados: disconnected, discarding osd_map(410..410 src has 247..410) 2024-09-17T18:29:25.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.440+0000 7faaba20f640 1 -- 172.21.15.121:0/2550152643 reap_dead start 2024-09-17T18:29:25.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.440+0000 7faabbc72640 1 -- 172.21.15.121:0/2550152643 shutdown_connections 2024-09-17T18:29:25.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.440+0000 7faabbc72640 1 -- 172.21.15.121:0/2550152643 >> 172.21.15.121:0/2550152643 conn(0x7faab40ab830 msgr2=0x7faab4133ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:29:25.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.440+0000 7faabbc72640 1 -- 172.21.15.121:0/2550152643 shutdown_connections 2024-09-17T18:29:25.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.440+0000 7faabbc72640 1 -- 172.21.15.121:0/2550152643 wait complete. 2024-09-17T18:29:25.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.440+0000 7faabbc72640 1 librados: shutdown 2024-09-17T18:29:25.465 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:29:25.465 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T18:29:25.465 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T18:29:25.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.560+0000 7f407640c640 10 monclient: get_monmap_and_config 2024-09-17T18:29:25.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.560+0000 7f407640c640 10 monclient: build_initial_monmap 2024-09-17T18:29:25.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.560+0000 7f407640c640 10 monclient: monmap: 2024-09-17T18:29:25.563 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:29:25.563 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:25.563 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:29:25.562217+0000 2024-09-17T18:29:25.563 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:29:25.562217+0000 2024-09-17T18:29:25.563 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:29:25.563 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:25.564 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:29:25.564 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:29:25.564 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:29:25.564 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:29:25.564 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:29:25.564 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:25.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.561+0000 7f407640c640 1 Processor -- start 2024-09-17T18:29:25.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.561+0000 7f407640c640 1 -- start start 2024-09-17T18:29:25.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.561+0000 7f407640c640 10 monclient: init 2024-09-17T18:29:25.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.561+0000 7f407640c640 10 monclient: _reopen_session rank -1 2024-09-17T18:29:25.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.561+0000 7f407640c640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-17T18:29:25.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.561+0000 7f407640c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f40700a0a00 addr v1:172.21.15.121:6789/0 2024-09-17T18:29:25.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.561+0000 7f407640c640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f4070137c90 0x7f407013a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:25.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.561+0000 7f407640c640 10 monclient(hunting): picked mon.noname-c con 0x7f4070137c90 addr v2:172.21.15.173:3300/0 2024-09-17T18:29:25.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.561+0000 7f407640c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f407013a5c0 addr v1:172.21.15.173:6789/0 2024-09-17T18:29:25.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.561+0000 7f407640c640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40700a3820 con 0x7f40700a0a00 2024-09-17T18:29:25.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.561+0000 7f407640c640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40700a2ea0 con 0x7f407013a5c0 2024-09-17T18:29:25.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.561+0000 7f407640c640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:25.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.561+0000 7f407640c640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f407009e800 con 0x7f4070137c90 2024-09-17T18:29:25.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.561+0000 7f407640c640 10 monclient(hunting): _renew_subs 2024-09-17T18:29:25.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.561+0000 7f407640c640 10 monclient(hunting): authenticate will time out at 1369.278912s 2024-09-17T18:29:25.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.562+0000 7f406ffff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f40700a0a00 0x7f40700a0de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:43804/0 (socket says 172.21.15.121:43804) 2024-09-17T18:29:25.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.562+0000 7f406ffff640 1 -- 172.21.15.121:0/3605415659 learned_addr learned my addr 172.21.15.121:0/3605415659 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:29:25.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.562+0000 7f406f7fe640 1 --2- 172.21.15.121:0/3605415659 >> v2:172.21.15.173:3300/0 conn(0x7f4070137c90 0x7f407013a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:25.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.562+0000 7f406f7fe640 10 monclient(hunting): get_auth_request con 0x7f4070137c90 auth_method 0 2024-09-17T18:29:25.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.562+0000 7f406f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:25.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.562+0000 7f406f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:25.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.562+0000 7f406f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:25.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.562+0000 7f406f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:25.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.562+0000 7f406f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:25.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.562+0000 7f406f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:25.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406ffff640 1 -- 172.21.15.121:0/3605415659 >> v1:172.21.15.121:6789/0 conn(0x7f40700a0a00 legacy=0x7f40700a0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:25.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406effd640 1 -- 172.21.15.121:0/3605415659 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2703750419 0 0) 0x7f4060001450 con 0x7f40700a0a00 2024-09-17T18:29:25.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406effd640 10 monclient(hunting): _negotiate 2024-09-17T18:29:25.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:25.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:25.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406effd640 10 monclient(hunting): my global_id is 44550 2024-09-17T18:29:25.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406effd640 1 -- 172.21.15.121:0/3605415659 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4058003520 con 0x7f40700a0a00 2024-09-17T18:29:25.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f40749a9640 1 -- 172.21.15.121:0/3605415659 >> v1:172.21.15.173:6789/0 conn(0x7f407013a5c0 legacy=0x7f407013c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:25.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406effd640 1 -- 172.21.15.121:0/3605415659 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 825126127 0 0) 0x7f4058003520 con 0x7f40700a0a00 2024-09-17T18:29:25.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406effd640 1 -- 172.21.15.121:0/3605415659 >> v2:172.21.15.173:3300/0 conn(0x7f4070137c90 msgr2=0x7f407013a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:29:25.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406effd640 1 --2- 172.21.15.121:0/3605415659 >> v2:172.21.15.173:3300/0 conn(0x7f4070137c90 0x7f407013a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:25.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406effd640 1 -- 172.21.15.121:0/3605415659 >> v1:172.21.15.173:6789/0 conn(0x7f407013a5c0 legacy=0x7f407013c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:25.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406effd640 10 monclient: _finish_hunting 0 2024-09-17T18:29:25.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406effd640 1 monclient: found mon.noname-a-legacy 2024-09-17T18:29:25.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:29:25.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406effd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.121:6789/0 2024-09-17T18:29:25.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406effd640 1 -- 172.21.15.121:0/3605415659 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40700a2c30 con 0x7f40700a0a00 2024-09-17T18:29:25.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406effd640 10 monclient: _finish_auth 0 2024-09-17T18:29:25.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406effd640 10 monclient: _check_auth_tickets 2024-09-17T18:29:25.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:29:25.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406effd640 1 -- 172.21.15.121:0/3605415659 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4002019445 0 0) 0x7f4060001b70 con 0x7f40700a0a00 2024-09-17T18:29:25.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:25.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406effd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T18:29:25.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406effd640 10 monclient: dump: 2024-09-17T18:29:25.576 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:25.576 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:25.576 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.576 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.576 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:25.576 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:25.576 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:25.577 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:25.577 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:25.577 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:25.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406effd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] but i'm connected to v1:172.21.15.121:6789/0, reconnecting 2024-09-17T18:29:25.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406effd640 10 monclient: _reopen_session rank -1 2024-09-17T18:29:25.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406effd640 1 -- 172.21.15.121:0/3605415659 >> v1:172.21.15.121:6789/0 conn(0x7f40700a0a00 legacy=0x7f40700a0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:25.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406effd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:29:25.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406effd640 1 --2- 172.21.15.121:0/3605415659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4058003b40 0x7f4058003f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:25.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406effd640 10 monclient(hunting): picked mon.a con 0x7f4058003b40 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:29:25.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406effd640 1 --2- 172.21.15.121:0/3605415659 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4058004800 0x7f4058004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:25.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406effd640 10 monclient(hunting): picked mon.b con 0x7f4058004800 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:29:25.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406effd640 1 --2- 172.21.15.121:0/3605415659 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4058005410 0x7f405800ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:25.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406effd640 10 monclient(hunting): picked mon.c con 0x7f4058005410 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:29:25.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406effd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:25.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406effd640 1 -- 172.21.15.121:0/3605415659 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f407009e800 con 0x7f4058003b40 2024-09-17T18:29:25.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406effd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:25.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406effd640 1 -- 172.21.15.121:0/3605415659 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f405800c320 con 0x7f4058004800 2024-09-17T18:29:25.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406effd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:25.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406effd640 1 -- 172.21.15.121:0/3605415659 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f405800c520 con 0x7f4058005410 2024-09-17T18:29:25.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406effd640 10 monclient(hunting): _renew_subs 2024-09-17T18:29:25.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406effd640 10 monclient(hunting): _finish_auth 0 2024-09-17T18:29:25.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406ffff640 1 --2- 172.21.15.121:0/3605415659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4058003b40 0x7f4058003f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:25.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406ffff640 10 monclient(hunting): get_auth_request con 0x7f4058003b40 auth_method 0 2024-09-17T18:29:25.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:25.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:25.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:29:25.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f40749a9640 1 --2- 172.21.15.121:0/3605415659 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4058005410 0x7f405800ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:25.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f40749a9640 10 monclient(hunting): get_auth_request con 0x7f4058005410 auth_method 0 2024-09-17T18:29:25.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f40749a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:25.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f40749a9640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:25.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f40749a9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:29:25.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:25.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:25.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:29:25.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f40749a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:25.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f40749a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:25.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f40749a9640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:29:25.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406f7fe640 1 --2- 172.21.15.121:0/3605415659 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4058004800 0x7f4058004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:25.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406f7fe640 10 monclient(hunting): get_auth_request con 0x7f4058004800 auth_method 0 2024-09-17T18:29:25.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:25.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:25.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.564+0000 7f406f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:29:25.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.565+0000 7f406ffff640 10 monclient(hunting): handle_auth_done global_id 44550 payload 293 2024-09-17T18:29:25.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.565+0000 7f406ffff640 1 -- 172.21.15.121:0/3605415659 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4058005410 msgr2=0x7f405800ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:29:25.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.565+0000 7f406ffff640 1 --2- 172.21.15.121:0/3605415659 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4058005410 0x7f405800ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:25.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.565+0000 7f406ffff640 1 -- 172.21.15.121:0/3605415659 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4058004800 msgr2=0x7f4058004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:29:25.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.565+0000 7f406ffff640 1 --2- 172.21.15.121:0/3605415659 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4058004800 0x7f4058004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:25.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.565+0000 7f406ffff640 10 monclient: _finish_hunting 0 2024-09-17T18:29:25.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.565+0000 7f406ffff640 1 monclient: found mon.a 2024-09-17T18:29:25.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.565+0000 7f406ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:29:25.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.565+0000 7f406ffff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:29:25.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.565+0000 7f406ffff640 1 -- 172.21.15.121:0/3605415659 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4058003520 con 0x7f4058003b40 2024-09-17T18:29:25.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.565+0000 7f406ffff640 10 monclient: _finish_auth 0 2024-09-17T18:29:25.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.565+0000 7f406ffff640 10 monclient: _check_auth_tickets 2024-09-17T18:29:25.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.565+0000 7f406ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:29:25.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.565+0000 7f407640c640 5 monclient: authenticate success, global_id 44550 2024-09-17T18:29:25.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.565+0000 7f407640c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:29:25.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.565+0000 7f40749a9640 1 --2- 172.21.15.121:0/3605415659 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4058005410 0x7f405800ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:29:25.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.565+0000 7f406f7fe640 1 --2- 172.21.15.121:0/3605415659 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4058004800 0x7f4058004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T18:29:25.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.565+0000 7f40749a9640 1 -- 172.21.15.121:0/3605415659 reap_dead start 2024-09-17T18:29:25.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.567+0000 7f406ffff640 1 -- 172.21.15.121:0/3605415659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4058003b40 msgr2=0x7f4058003f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:25.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.567+0000 7f406ffff640 1 --2- 172.21.15.121:0/3605415659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4058003b40 0x7f4058003f60 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f40600016a0 tx=0x7f4058005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f419698b105e6fb3 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:25.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.567+0000 7f406effd640 1 -- 172.21.15.121:0/3605415659 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f406000d1c0 con 0x7f4058003b40 2024-09-17T18:29:25.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.567+0000 7f406effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:25.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.567+0000 7f406effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:29:25.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.567+0000 7f406effd640 10 monclient: dump: 2024-09-17T18:29:25.581 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:25.581 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:25.581 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.581 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.581 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:25.582 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:25.582 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:25.582 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:25.582 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:25.582 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:25.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.567+0000 7f407640c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:29:25.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.567+0000 7f406effd640 1 -- 172.21.15.121:0/3605415659 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4060033070 con 0x7f4058003b40 2024-09-17T18:29:25.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.567+0000 7f406effd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:29:25.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.567+0000 7f407640c640 10 monclient: get_monmap_and_config success 2024-09-17T18:29:25.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.567+0000 7f407640c640 4 set_mon_vals no callback set 2024-09-17T18:29:25.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.567+0000 7f406effd640 1 -- 172.21.15.121:0/3605415659 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4060005df0 con 0x7f4058003b40 2024-09-17T18:29:25.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.567+0000 7f406effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:25.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.567+0000 7f406effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:29:25.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.567+0000 7f406effd640 10 monclient: dump: 2024-09-17T18:29:25.583 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:25.583 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:25.583 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.583 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.583 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:25.583 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:25.583 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:25.583 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:25.583 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:25.583 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:25.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.568+0000 7f407640c640 10 monclient: shutdown 2024-09-17T18:29:25.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.568+0000 7f407640c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:29:25.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.568+0000 7f407640c640 1 -- 172.21.15.121:0/3605415659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4058003b40 msgr2=0x7f4058003f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:25.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.568+0000 7f407640c640 1 --2- 172.21.15.121:0/3605415659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4058003b40 0x7f4058003f60 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f40600016a0 tx=0x7f4058005830 comp rx=0 tx=0).stop 2024-09-17T18:29:25.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.568+0000 7f407640c640 1 -- 172.21.15.121:0/3605415659 shutdown_connections 2024-09-17T18:29:25.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.568+0000 7f407640c640 1 --2- 172.21.15.121:0/3605415659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4058003b40 0x7f4058003f60 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:25.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.568+0000 7f407640c640 1 -- 172.21.15.121:0/3605415659 >> 172.21.15.121:0/3605415659 conn(0x7f40700a7830 msgr2=0x7f40700a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:29:25.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.568+0000 7f407640c640 1 -- 172.21.15.121:0/3605415659 shutdown_connections 2024-09-17T18:29:25.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.568+0000 7f407640c640 1 -- 172.21.15.121:0/3605415659 wait complete. 2024-09-17T18:29:25.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.569+0000 7f407640c640 10 monclient: build_initial_monmap 2024-09-17T18:29:25.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.569+0000 7f407640c640 10 monclient: monmap: 2024-09-17T18:29:25.584 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:29:25.584 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:29:25.584 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:29:25.584 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:29:25.584 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:29:25.584 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:25.584 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:29:25.584 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:29:25.585 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:29:25.585 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:25.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.569+0000 7f407640c640 1 librados: starting msgr at 2024-09-17T18:29:25.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.569+0000 7f407640c640 1 librados: starting objecter 2024-09-17T18:29:25.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.569+0000 7f407640c640 1 Processor -- start 2024-09-17T18:29:25.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.569+0000 7f407640c640 1 -- start start 2024-09-17T18:29:25.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.569+0000 7f407640c640 1 librados: setting wanted keys 2024-09-17T18:29:25.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.569+0000 7f407640c640 1 librados: calling monclient init 2024-09-17T18:29:25.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.569+0000 7f407640c640 10 monclient: init 2024-09-17T18:29:25.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.569+0000 7f407640c640 10 monclient: _reopen_session rank -1 2024-09-17T18:29:25.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.569+0000 7f407640c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T18:29:25.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.569+0000 7f407640c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4058003b40 0x7f40701cbb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:25.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.569+0000 7f407640c640 10 monclient(hunting): picked mon.noname-b con 0x7f4058003b40 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:29:25.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.569+0000 7f407640c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40701cc070 0x7f407011f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:25.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.569+0000 7f407640c640 10 monclient(hunting): picked mon.noname-a con 0x7f40701cc070 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:29:25.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.569+0000 7f407640c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f40701cc4d0 0x7f407011f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:25.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.569+0000 7f407640c640 10 monclient(hunting): picked mon.noname-c con 0x7f40701cc4d0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:29:25.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.569+0000 7f407640c640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:25.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.569+0000 7f407640c640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f40700a3120 con 0x7f40701cc070 2024-09-17T18:29:25.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.569+0000 7f407640c640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:25.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.569+0000 7f407640c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f407009ecf0 con 0x7f4058003b40 2024-09-17T18:29:25.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.569+0000 7f407640c640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:25.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.569+0000 7f407640c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f40700a32a0 con 0x7f40701cc4d0 2024-09-17T18:29:25.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.569+0000 7f407640c640 10 monclient(hunting): _renew_subs 2024-09-17T18:29:25.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.570+0000 7f407640c640 10 monclient(hunting): authenticate will time out at 1369.287007s 2024-09-17T18:29:25.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.570+0000 7f40749a9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f40701cc4d0 0x7f407011f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:25.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.570+0000 7f406f7fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40701cc070 0x7f407011f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:25.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.570+0000 7f406f7fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40701cc070 0x7f407011f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58056/0 (socket says 172.21.15.121:58056) 2024-09-17T18:29:25.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.570+0000 7f40749a9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f40701cc4d0 0x7f407011f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:47078/0 (socket says 172.21.15.121:47078) 2024-09-17T18:29:25.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.570+0000 7f406f7fe640 1 -- 172.21.15.121:0/3480792842 learned_addr learned my addr 172.21.15.121:0/3480792842 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:29:25.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.570+0000 7f406f7fe640 10 monclient(hunting): get_auth_request con 0x7f40701cc070 auth_method 0 2024-09-17T18:29:25.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.570+0000 7f406f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:25.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.570+0000 7f406f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:25.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.570+0000 7f406f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:25.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.570+0000 7f40749a9640 10 monclient(hunting): get_auth_request con 0x7f40701cc4d0 auth_method 0 2024-09-17T18:29:25.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.570+0000 7f40749a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:25.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.570+0000 7f40749a9640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:25.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.570+0000 7f40749a9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:25.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.570+0000 7f406f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:25.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.570+0000 7f406f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:25.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.570+0000 7f406f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:25.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.570+0000 7f40749a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:25.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.570+0000 7f40749a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:25.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.570+0000 7f40749a9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:25.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.570+0000 7f406ffff640 1 --2- 172.21.15.121:0/3480792842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4058003b40 0x7f40701cbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:25.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.570+0000 7f406ffff640 10 monclient(hunting): get_auth_request con 0x7f4058003b40 auth_method 0 2024-09-17T18:29:25.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.570+0000 7f406ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:25.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.570+0000 7f406ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:25.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.570+0000 7f406ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:25.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.570+0000 7f40749a9640 10 monclient(hunting): handle_auth_done global_id 64184 payload 824 2024-09-17T18:29:25.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.570+0000 7f40749a9640 1 -- 172.21.15.121:0/3480792842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4058003b40 msgr2=0x7f40701cbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:25.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.570+0000 7f40749a9640 1 --2- 172.21.15.121:0/3480792842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4058003b40 0x7f40701cbb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:25.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.570+0000 7f40749a9640 1 -- 172.21.15.121:0/3480792842 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40701cc070 msgr2=0x7f407011f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:25.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.570+0000 7f40749a9640 1 --2- 172.21.15.121:0/3480792842 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40701cc070 0x7f407011f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:25.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.570+0000 7f40749a9640 10 monclient: _finish_hunting 0 2024-09-17T18:29:25.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.570+0000 7f40749a9640 1 monclient: found mon.noname-c 2024-09-17T18:29:25.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.570+0000 7f40749a9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:29:25.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.570+0000 7f40749a9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:29:25.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.570+0000 7f40749a9640 1 -- 172.21.15.121:0/3480792842 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4070135430 con 0x7f40701cc4d0 2024-09-17T18:29:25.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.570+0000 7f406f7fe640 1 --2- 172.21.15.121:0/3480792842 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40701cc070 0x7f407011f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:29:25.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.572+0000 7f40749a9640 1 -- 172.21.15.121:0/3480792842 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f40701cc4d0 msgr2=0x7f407011f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:25.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.572+0000 7f40749a9640 1 --2- 172.21.15.121:0/3480792842 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f40701cc4d0 0x7f407011f600 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f407009ef10 tx=0x7f406800d520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:25.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.572+0000 7f406cff9640 10 client.?.objecter ms_handle_connect 0x7f40701cc4d0 2024-09-17T18:29:25.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.572+0000 7f406cff9640 10 client.?.objecter resend_mon_ops 2024-09-17T18:29:25.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.572+0000 7f406cff9640 1 -- 172.21.15.121:0/3480792842 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4068001d90 con 0x7f40701cc4d0 2024-09-17T18:29:25.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.572+0000 7f406cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:25.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.572+0000 7f406cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:29:25.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.572+0000 7f406cff9640 10 monclient: dump: 2024-09-17T18:29:25.594 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:25.594 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:25.594 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.594 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.594 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:25.594 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:25.594 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:25.594 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:25.594 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:25.594 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:25.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.572+0000 7f406cff9640 10 monclient: _finish_auth 0 2024-09-17T18:29:25.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.572+0000 7f406cff9640 10 monclient: _check_auth_tickets 2024-09-17T18:29:25.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.572+0000 7f406cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:29:25.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.572+0000 7f407640c640 5 monclient: authenticate success, global_id 64184 2024-09-17T18:29:25.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.572+0000 7f406cff9640 1 -- 172.21.15.121:0/3480792842 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4068018070 con 0x7f40701cc4d0 2024-09-17T18:29:25.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.572+0000 7f407640c640 10 monclient: _renew_subs 2024-09-17T18:29:25.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.572+0000 7f407640c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:29:25.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.572+0000 7f407640c640 1 -- 172.21.15.121:0/3480792842 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4070122e40 con 0x7f40701cc4d0 2024-09-17T18:29:25.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.572+0000 7f406cff9640 10 monclient: handle_config config(0 keys) 2024-09-17T18:29:25.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.572+0000 7f406cff9640 1 -- 172.21.15.121:0/3480792842 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f406800db80 con 0x7f40701cc4d0 2024-09-17T18:29:25.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.572+0000 7f406cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:25.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.572+0000 7f406cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:29:25.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.572+0000 7f406cff9640 10 monclient: dump: 2024-09-17T18:29:25.595 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:25.595 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:25.595 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.595 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.595 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:25.595 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:25.595 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:25.595 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:25.595 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:25.596 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:25.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.572+0000 7f407640c640 10 client.64184.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:29:25.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.572+0000 7f407640c640 10 monclient: _renew_subs 2024-09-17T18:29:25.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.572+0000 7f406e7fc640 4 set_mon_vals no callback set 2024-09-17T18:29:25.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.572+0000 7f407640c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:29:25.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.572+0000 7f407640c640 1 -- 172.21.15.121:0/3480792842 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4070122b60 con 0x7f40701cc4d0 2024-09-17T18:29:25.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.572+0000 7f407640c640 1 librados: init done 2024-09-17T18:29:25.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.573+0000 7f407640c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:29:25.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.573+0000 7f407640c640 1 --2- 172.21.15.121:0/3480792842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f407011fb40 0x7f407011ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:25.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.573+0000 7f407640c640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:25.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.573+0000 7f407640c640 1 -- 172.21.15.121:0/3480792842 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f40700a1fa0 con 0x7f407011fb40 2024-09-17T18:29:25.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.573+0000 7f406cff9640 1 -- 172.21.15.121:0/3480792842 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f406800dd20 con 0x7f40701cc4d0 2024-09-17T18:29:25.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.573+0000 7f406cff9640 1 --2- 172.21.15.121:0/3480792842 >> v2:172.21.15.173:6812/2423786037 conn(0x7f405c0401d0 0x7f405c042690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:25.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.573+0000 7f406cff9640 1 -- 172.21.15.121:0/3480792842 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(410..410 src has 247..410) ==== 5358+0+0 (secure 0 0 0) 0x7f4068053580 con 0x7f40701cc4d0 2024-09-17T18:29:25.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.573+0000 7f406cff9640 10 client.64184.objecter ms_dispatch 0x7f4070000b90 osd_map(410..410 src has 247..410) 2024-09-17T18:29:25.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.573+0000 7f406ffff640 1 --2- 172.21.15.121:0/3480792842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f407011fb40 0x7f407011ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:25.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.573+0000 7f406cff9640 3 client.64184.objecter handle_osd_map got epochs [410,410] > 0 2024-09-17T18:29:25.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.573+0000 7f406cff9640 3 client.64184.objecter handle_osd_map decoding full epoch 410 2024-09-17T18:29:25.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.574+0000 7f406ffff640 10 monclient: get_auth_request con 0x7f407011fb40 auth_method 0 2024-09-17T18:29:25.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.574+0000 7f406ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:25.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.574+0000 7f406ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:25.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.574+0000 7f406ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:25.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.574+0000 7f406cff9640 20 client.64184.objecter dump_active .. 0 homeless 2024-09-17T18:29:25.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.574+0000 7f406ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:25.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.574+0000 7f406ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:25.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.574+0000 7f406ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:25.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.574+0000 7f406f7fe640 1 --2- 172.21.15.121:0/3480792842 >> v2:172.21.15.173:6812/2423786037 conn(0x7f405c0401d0 0x7f405c042690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:25.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.574+0000 7f406f7fe640 10 monclient: get_auth_request con 0x7f405c0401d0 auth_method 0 2024-09-17T18:29:25.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.574+0000 7f406ffff640 10 monclient(hunting): handle_auth_done global_id 74116 payload 824 2024-09-17T18:29:25.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.574+0000 7f406ffff640 1 -- 172.21.15.121:0/3480792842 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f40700a5c10 con 0x7f407011fb40 2024-09-17T18:29:25.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.576+0000 7f406ffff640 1 -- 172.21.15.121:0/3480792842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f407011fb40 msgr2=0x7f407011ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:25.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.576+0000 7f406cff9640 10 client.64184.objecter ms_handle_connect 0x7f407011fb40 2024-09-17T18:29:25.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.576+0000 7f406ffff640 1 --2- 172.21.15.121:0/3480792842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f407011fb40 0x7f407011ff20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4060032000 tx=0x7f40700a2e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:25.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.576+0000 7f406cff9640 10 client.64184.objecter resend_mon_ops 2024-09-17T18:29:25.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.576+0000 7f406cff9640 10 client.64184.objecter ms_handle_connect 0x7f405c0401d0 2024-09-17T18:29:25.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.576+0000 7f406f7fe640 1 --2- 172.21.15.121:0/3480792842 >> v2:172.21.15.173:6812/2423786037 conn(0x7f405c0401d0 0x7f405c042690 secure :-1 s=READY pgs=4010 cs=0 l=1 rev1=1 crypto rx=0x7f4064006f80 tx=0x7f4064008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:25.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.576+0000 7f406cff9640 1 -- 172.21.15.121:0/3480792842 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4060008170 con 0x7f407011fb40 2024-09-17T18:29:25.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.576+0000 7f406cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:25.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.576+0000 7f406cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:29:25.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.576+0000 7f406cff9640 10 monclient: dump: 2024-09-17T18:29:25.598 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:25.598 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:25.598 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.598 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.599 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:25.599 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:25.599 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:25.599 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:25.599 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:25.599 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:25.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.577+0000 7f406cff9640 1 -- 172.21.15.121:0/3480792842 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4060033030 con 0x7f407011fb40 2024-09-17T18:29:25.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.577+0000 7f406cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:29:25.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.577+0000 7f406cff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:29:25.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.577+0000 7f406cff9640 1 -- 172.21.15.121:0/3480792842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f407011fb40 msgr2=0x7f407011ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:25.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.577+0000 7f406cff9640 1 --2- 172.21.15.121:0/3480792842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f407011fb40 0x7f407011ff20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4060032000 tx=0x7f40700a2e00 comp rx=0 tx=0).stop 2024-09-17T18:29:25.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.577+0000 7f406cff9640 1 -- 172.21.15.121:0/3480792842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f407011fb40 msgr2=0x7f407011ff20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:29:25.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.577+0000 7f406cff9640 1 --2- 172.21.15.121:0/3480792842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f407011fb40 0x7f407011ff20 secure :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4060032000 tx=0x7f40700a2e00 comp rx=0 tx=0).stop 2024-09-17T18:29:25.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.583+0000 7f407640c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:29:25.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.583+0000 7f407640c640 1 --2- 172.21.15.121:0/3480792842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f40700712d0 0x7f40700716b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:25.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.583+0000 7f407640c640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:25.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.583+0000 7f407640c640 1 -- 172.21.15.121:0/3480792842 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4070120460 con 0x7f40700712d0 2024-09-17T18:29:25.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.583+0000 7f406ffff640 1 --2- 172.21.15.121:0/3480792842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f40700712d0 0x7f40700716b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:25.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.583+0000 7f406ffff640 10 monclient: get_auth_request con 0x7f40700712d0 auth_method 0 2024-09-17T18:29:25.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.583+0000 7f406ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:25.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.583+0000 7f406ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:25.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.583+0000 7f406ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:25.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.583+0000 7f406ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:25.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.583+0000 7f406ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:25.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.583+0000 7f406ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:25.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.583+0000 7f406ffff640 10 monclient(hunting): handle_auth_done global_id 74119 payload 824 2024-09-17T18:29:25.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.583+0000 7f406ffff640 1 -- 172.21.15.121:0/3480792842 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4070121640 con 0x7f40700712d0 2024-09-17T18:29:25.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.586+0000 7f406ffff640 1 -- 172.21.15.121:0/3480792842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f40700712d0 msgr2=0x7f40700716b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:25.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.586+0000 7f406cff9640 10 client.64184.objecter ms_handle_connect 0x7f40700712d0 2024-09-17T18:29:25.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.586+0000 7f406cff9640 10 client.64184.objecter resend_mon_ops 2024-09-17T18:29:25.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.586+0000 7f406ffff640 1 --2- 172.21.15.121:0/3480792842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f40700712d0 0x7f40700716b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f406000a000 tx=0x7f4060030000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:25.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.586+0000 7f406cff9640 1 -- 172.21.15.121:0/3480792842 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f406002ed50 con 0x7f40700712d0 2024-09-17T18:29:25.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.586+0000 7f406cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:25.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.586+0000 7f406cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:29:25.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.586+0000 7f406cff9640 10 monclient: dump: 2024-09-17T18:29:25.601 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:25.601 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:25.601 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.601 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.601 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:25.601 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:25.601 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:25.601 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:25.601 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:25.601 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:25.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.586+0000 7f406cff9640 1 -- 172.21.15.121:0/3480792842 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7f406002ed50 con 0x7f40700712d0 2024-09-17T18:29:25.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.586+0000 7f406cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:29:25.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.586+0000 7f406cff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:29:25.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.586+0000 7f406cff9640 1 -- 172.21.15.121:0/3480792842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f40700712d0 msgr2=0x7f40700716b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:25.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.586+0000 7f406cff9640 1 --2- 172.21.15.121:0/3480792842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f40700712d0 0x7f40700716b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f406000a000 tx=0x7f4060030000 comp rx=0 tx=0).stop 2024-09-17T18:29:25.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.586+0000 7f406cff9640 1 -- 172.21.15.121:0/3480792842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f40700712d0 msgr2=0x7f40700716b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:29:25.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.586+0000 7f406cff9640 1 --2- 172.21.15.121:0/3480792842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f40700712d0 0x7f40700716b0 secure :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f406000a000 tx=0x7f4060030000 comp rx=0 tx=0).stop 2024-09-17T18:29:25.602 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:29:25.602 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:29:25.602 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:29:25.602 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:29:25.602 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 38, 2024-09-17T18:29:25.602 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 15707, 2024-09-17T18:29:25.602 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:29:25.603 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:29:25.603 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:29:25.603 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:29:25.603 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:29:25.603 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 0, 2024-09-17T18:29:25.603 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:29:25.603 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:29:25.603 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:29:25.603 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:29:25.603 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:29:25.603 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:29:25.603 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:29:25.603 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:29:25.603 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:29:25.603 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:29:25.603 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:29:25.604 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:29:25.604 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:29:25.604 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:29:25.604 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:29:25.604 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:29:25.604 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:29:25.604 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:29:25.604 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:29:25.604 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:29:25.604 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:29:25.604 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:29:25.604 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:29:25.604 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:29:25.604 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:29:25.604 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:29:25.605 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:29:25.605 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:29:25.605 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:25.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:25.605 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:29:25.605 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:29:25.605 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:29:25.605 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:29:25.605 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:29:25.605 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:29:25.605 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:29:25.605 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:29:25.605 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:29:25.605 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:29:25.605 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:29:25.605 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:29:25.606 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:29:25.606 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:29:25.606 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:29:25.606 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:29:25.606 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:29:25.606 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:29:25.606 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:29:25.606 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:29:25.606 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:29:25.606 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:29:25.606 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:29:25.606 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:29:25.606 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:29:25.606 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:29:25.606 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:29:25.607 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:29:25.607 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:29:25.607 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:29:25.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:25.607 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:29:25.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:25.607 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:29:25.607 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:29:25.607 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:29:25.607 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:29:25.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:25.607 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:29:25.607 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:29:25.607 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:25.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:25.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:25.608 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:29:25.608 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:29:25.608 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:25.608 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:25.608 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:25.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:25.608 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:29:25.608 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:29:25.608 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:29:25.608 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:29:25.608 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:29:25.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:25.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:25.608 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:29:25.609 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:29:25.609 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:29:25.609 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:29:25.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:25.609 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:29:25.609 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:29:25.609 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:25.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:25.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:25.609 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:29:25.609 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:29:25.609 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:25.609 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:25.609 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:25.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:25.610 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:29:25.610 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:29:25.610 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:29:25.610 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:29:25.610 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:29:25.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:25.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:25.610 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:29:25.610 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:29:25.610 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:29:25.610 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:29:25.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:25.610 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:29:25.610 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:29:25.610 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:25.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:25.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:25.611 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:29:25.611 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:29:25.611 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:25.611 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:25.611 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:25.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:25.611 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:29:25.611 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:29:25.611 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:29:25.611 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:29:25.611 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:29:25.611 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:25.611 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:25.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:25.611 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:29:25.612 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:29:25.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:25.612 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:29:25.612 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:29:25.612 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:29:25.612 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:25.612 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:29:25.612 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:29:25.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:25.612 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:29:25.612 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:29:25.612 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:29:25.612 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:25.612 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:25.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:25.612 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:29:25.613 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:29:25.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.586+0000 7f407640c640 10 librados: watch_flush enter 2024-09-17T18:29:25.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.586+0000 7f407640c640 10 librados: watch_flush exit 2024-09-17T18:29:25.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.586+0000 7f407640c640 20 client.64184.objecter shutdown clearing up homeless session... 2024-09-17T18:29:25.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.586+0000 7f407640c640 10 client.64184.objecter successfully canceled tick 2024-09-17T18:29:25.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.586+0000 7f407640c640 1 -- 172.21.15.121:0/3480792842 >> v2:172.21.15.173:6812/2423786037 conn(0x7f405c0401d0 msgr2=0x7f405c042690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:25.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.586+0000 7f407640c640 1 --2- 172.21.15.121:0/3480792842 >> v2:172.21.15.173:6812/2423786037 conn(0x7f405c0401d0 0x7f405c042690 secure :-1 s=READY pgs=4010 cs=0 l=1 rev1=1 crypto rx=0x7f4064006f80 tx=0x7f4064008040 comp rx=0 tx=0).stop 2024-09-17T18:29:25.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.586+0000 7f407640c640 10 monclient: shutdown 2024-09-17T18:29:25.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.586+0000 7f407640c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:29:25.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.586+0000 7f407640c640 1 -- 172.21.15.121:0/3480792842 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f40701cc4d0 msgr2=0x7f407011f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:25.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.586+0000 7f407640c640 1 --2- 172.21.15.121:0/3480792842 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f40701cc4d0 0x7f407011f600 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f407009ef10 tx=0x7f406800d520 comp rx=0 tx=0).stop 2024-09-17T18:29:25.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.586+0000 7f40749a9640 1 -- 172.21.15.121:0/3480792842 reap_dead start 2024-09-17T18:29:25.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.586+0000 7f40749a9640 1 -- 172.21.15.121:0/3480792842 reap_dead start 2024-09-17T18:29:25.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.586+0000 7f407640c640 1 -- 172.21.15.121:0/3480792842 shutdown_connections 2024-09-17T18:29:25.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.586+0000 7f407640c640 1 -- 172.21.15.121:0/3480792842 >> 172.21.15.121:0/3480792842 conn(0x7f40700a7830 msgr2=0x7f40700a00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:29:25.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.587+0000 7f407640c640 1 -- 172.21.15.121:0/3480792842 shutdown_connections 2024-09-17T18:29:25.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.587+0000 7f407640c640 1 -- 172.21.15.121:0/3480792842 wait complete. 2024-09-17T18:29:25.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.587+0000 7f407640c640 1 librados: shutdown 2024-09-17T18:29:25.616 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:29:25.616 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-17T18:29:25.616 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-17T18:29:25.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.714+0000 7f9bed984640 10 monclient: get_monmap_and_config 2024-09-17T18:29:25.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.714+0000 7f9bed984640 10 monclient: build_initial_monmap 2024-09-17T18:29:25.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.714+0000 7f9bed984640 10 monclient: monmap: 2024-09-17T18:29:25.717 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:29:25.717 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:25.717 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:29:25.715994+0000 2024-09-17T18:29:25.717 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:29:25.715994+0000 2024-09-17T18:29:25.718 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:29:25.718 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:25.718 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:29:25.718 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:29:25.718 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:29:25.718 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:29:25.718 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:29:25.718 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:25.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.714+0000 7f9bed984640 1 Processor -- start 2024-09-17T18:29:25.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9bed984640 1 -- start start 2024-09-17T18:29:25.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9bed984640 10 monclient: init 2024-09-17T18:29:25.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9bed984640 10 monclient: _reopen_session rank -1 2024-09-17T18:29:25.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9bed984640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-17T18:29:25.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9bed984640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f9be80a5c00 0x7f9be809d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:25.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9bed984640 10 monclient(hunting): picked mon.noname-a con 0x7f9be80a5c00 addr v2:172.21.15.121:3300/0 2024-09-17T18:29:25.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9bed984640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9be80a5fe0 0x7f9be80a63c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:25.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9bed984640 10 monclient(hunting): picked mon.noname-b con 0x7f9be80a5fe0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:29:25.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9bed984640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9be81389c0 addr v1:172.21.15.173:6789/0 2024-09-17T18:29:25.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9bed984640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9be80a8a80 con 0x7f9be81389c0 2024-09-17T18:29:25.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9bed984640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:25.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9bed984640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f9be809de00 con 0x7f9be80a5c00 2024-09-17T18:29:25.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9bed984640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:25.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9bed984640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9be80a74d0 con 0x7f9be80a5fe0 2024-09-17T18:29:25.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9bed984640 10 monclient(hunting): _renew_subs 2024-09-17T18:29:25.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9bed984640 10 monclient(hunting): authenticate will time out at 1369.432605s 2024-09-17T18:29:25.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9be6ffd640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f9be80a5c00 0x7f9be809d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:25.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9be67fc640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9be80a5fe0 0x7f9be80a63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:25.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9be6ffd640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f9be80a5c00 0x7f9be809d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58064/0 (socket says 172.21.15.121:58064) 2024-09-17T18:29:25.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9be67fc640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9be80a5fe0 0x7f9be80a63c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:47092/0 (socket says 172.21.15.121:47092) 2024-09-17T18:29:25.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9be6ffd640 1 -- 172.21.15.121:0/2240309963 learned_addr learned my addr 172.21.15.121:0/2240309963 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:29:25.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9be6ffd640 10 monclient(hunting): get_auth_request con 0x7f9be80a5c00 auth_method 0 2024-09-17T18:29:25.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9be6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:25.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9be6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:25.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9be6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:25.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9be67fc640 10 monclient(hunting): get_auth_request con 0x7f9be80a5fe0 auth_method 0 2024-09-17T18:29:25.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9be67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:25.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9be67fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:25.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9be67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:25.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9be67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:25.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9be67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:25.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9be67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:25.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9be6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:25.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9be6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:25.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9be6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:25.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9be67fc640 10 monclient(hunting): handle_auth_done global_id 64187 payload 274 2024-09-17T18:29:25.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9be67fc640 1 -- 172.21.15.121:0/2240309963 >> v2:172.21.15.121:3300/0 conn(0x7f9be80a5c00 msgr2=0x7f9be809d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:29:25.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9be67fc640 1 --2- 172.21.15.121:0/2240309963 >> v2:172.21.15.121:3300/0 conn(0x7f9be80a5c00 0x7f9be809d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:25.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9be67fc640 1 -- 172.21.15.121:0/2240309963 >> v1:172.21.15.173:6789/0 conn(0x7f9be81389c0 legacy=0x7f9be8138e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:29:25.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9be67fc640 10 monclient: _finish_hunting 0 2024-09-17T18:29:25.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9be67fc640 1 monclient: found mon.noname-b 2024-09-17T18:29:25.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9be67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:29:25.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9be67fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:29:25.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9be67fc640 1 -- 172.21.15.121:0/2240309963 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9be809cb90 con 0x7f9be80a5fe0 2024-09-17T18:29:25.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.715+0000 7f9be6ffd640 1 --2- 172.21.15.121:0/2240309963 >> v2:172.21.15.121:3300/0 conn(0x7f9be80a5c00 0x7f9be809d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:29:25.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.718+0000 7f9be67fc640 1 -- 172.21.15.121:0/2240309963 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9be80a5fe0 msgr2=0x7f9be80a63c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:25.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.718+0000 7f9be67fc640 1 --2- 172.21.15.121:0/2240309963 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9be80a5fe0 0x7f9be80a63c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9bd00097a0 tx=0x7f9bd0029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb988e832b09feaa server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:25.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.718+0000 7f9be5ffb640 1 -- 172.21.15.121:0/2240309963 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd0037070 con 0x7f9be80a5fe0 2024-09-17T18:29:25.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.718+0000 7f9be5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:25.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.718+0000 7f9be5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:29:25.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.718+0000 7f9be5ffb640 10 monclient: dump: 2024-09-17T18:29:25.725 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:25.725 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:25.725 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.725 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.725 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:25.725 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:25.725 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:25.725 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:25.725 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:25.725 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:25.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.718+0000 7f9be5ffb640 10 monclient: _finish_auth 0 2024-09-17T18:29:25.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.718+0000 7f9be5ffb640 10 monclient: _check_auth_tickets 2024-09-17T18:29:25.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.718+0000 7f9be5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:29:25.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.718+0000 7f9be5ffb640 1 -- 172.21.15.121:0/2240309963 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bd0032440 con 0x7f9be80a5fe0 2024-09-17T18:29:25.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.718+0000 7f9bed984640 5 monclient: authenticate success, global_id 64187 2024-09-17T18:29:25.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.718+0000 7f9bed984640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:29:25.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.718+0000 7f9be5ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:29:25.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.718+0000 7f9be5ffb640 1 -- 172.21.15.121:0/2240309963 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd0032810 con 0x7f9be80a5fe0 2024-09-17T18:29:25.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.718+0000 7f9bed984640 10 monclient: get_monmap_and_config success 2024-09-17T18:29:25.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.718+0000 7f9bed984640 4 set_mon_vals no callback set 2024-09-17T18:29:25.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.718+0000 7f9be5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:25.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.718+0000 7f9be5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:29:25.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.718+0000 7f9be5ffb640 10 monclient: dump: 2024-09-17T18:29:25.726 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:25.726 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:25.726 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.726 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.726 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:25.727 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:25.727 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:25.727 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:25.727 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:25.727 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:25.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.719+0000 7f9bed984640 10 monclient: shutdown 2024-09-17T18:29:25.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.719+0000 7f9bed984640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:29:25.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.719+0000 7f9bed984640 1 -- 172.21.15.121:0/2240309963 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9be80a5fe0 msgr2=0x7f9be80a63c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:25.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.719+0000 7f9bed984640 1 --2- 172.21.15.121:0/2240309963 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9be80a5fe0 0x7f9be80a63c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9bd00097a0 tx=0x7f9bd0029fd0 comp rx=0 tx=0).stop 2024-09-17T18:29:25.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.719+0000 7f9bed984640 1 -- 172.21.15.121:0/2240309963 shutdown_connections 2024-09-17T18:29:25.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.719+0000 7f9bed984640 1 --2- 172.21.15.121:0/2240309963 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9be80a5fe0 0x7f9be80a63c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:25.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.719+0000 7f9bed984640 1 --2- 172.21.15.121:0/2240309963 >> v2:172.21.15.121:3300/0 conn(0x7f9be80a5c00 0x7f9be809d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:25.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.719+0000 7f9bed984640 1 -- 172.21.15.121:0/2240309963 >> 172.21.15.121:0/2240309963 conn(0x7f9be80ab9b0 msgr2=0x7f9be80a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:29:25.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.719+0000 7f9bed984640 1 -- 172.21.15.121:0/2240309963 shutdown_connections 2024-09-17T18:29:25.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.719+0000 7f9bed984640 1 -- 172.21.15.121:0/2240309963 wait complete. 2024-09-17T18:29:25.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.720+0000 7f9bed984640 10 monclient: build_initial_monmap 2024-09-17T18:29:25.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.720+0000 7f9bed984640 10 monclient: monmap: 2024-09-17T18:29:25.732 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:29:25.732 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:29:25.733 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:29:25.733 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:29:25.733 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:29:25.733 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:25.733 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:29:25.733 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:29:25.733 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:29:25.733 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:25.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.720+0000 7f9bed984640 1 librados: starting msgr at 2024-09-17T18:29:25.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.720+0000 7f9bed984640 1 librados: starting objecter 2024-09-17T18:29:25.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.720+0000 7f9bed984640 1 Processor -- start 2024-09-17T18:29:25.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.720+0000 7f9bed984640 1 -- start start 2024-09-17T18:29:25.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.720+0000 7f9bed984640 1 librados: setting wanted keys 2024-09-17T18:29:25.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.720+0000 7f9bed984640 1 librados: calling monclient init 2024-09-17T18:29:25.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.720+0000 7f9bed984640 10 monclient: init 2024-09-17T18:29:25.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9bed984640 10 monclient: _reopen_session rank -1 2024-09-17T18:29:25.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9bed984640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T18:29:25.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9bed984640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9be80a5c00 0x7f9be81cdc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:25.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9bed984640 10 monclient(hunting): picked mon.noname-a con 0x7f9be80a5c00 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:29:25.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9bed984640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9be80a5fe0 0x7f9be81ce190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:25.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9bed984640 10 monclient(hunting): picked mon.noname-c con 0x7f9be80a5fe0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:29:25.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9bed984640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9be81389c0 0x7f9be81253d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:25.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9bed984640 10 monclient(hunting): picked mon.noname-b con 0x7f9be81389c0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:29:25.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9bed984640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:25.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9bed984640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9be80a6900 con 0x7f9be80a5c00 2024-09-17T18:29:25.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9bed984640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:25.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9bed984640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9be80703d0 con 0x7f9be81389c0 2024-09-17T18:29:25.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9bed984640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:25.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9bed984640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9be80aa7f0 con 0x7f9be80a5fe0 2024-09-17T18:29:25.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9bed984640 10 monclient(hunting): _renew_subs 2024-09-17T18:29:25.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9bed984640 10 monclient(hunting): authenticate will time out at 1369.438111s 2024-09-17T18:29:25.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9be6ffd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9be80a5c00 0x7f9be81cdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:25.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9be67fc640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9be80a5fe0 0x7f9be81ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:25.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9be67fc640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9be80a5fe0 0x7f9be81ce190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:47106/0 (socket says 172.21.15.121:47106) 2024-09-17T18:29:25.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9be6ffd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9be80a5c00 0x7f9be81cdc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58074/0 (socket says 172.21.15.121:58074) 2024-09-17T18:29:25.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9be67fc640 1 -- 172.21.15.121:0/306106628 learned_addr learned my addr 172.21.15.121:0/306106628 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:29:25.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9be67fc640 10 monclient(hunting): get_auth_request con 0x7f9be80a5fe0 auth_method 0 2024-09-17T18:29:25.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9be67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:25.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9be67fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:25.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9be67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:25.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9be6ffd640 10 monclient(hunting): get_auth_request con 0x7f9be80a5c00 auth_method 0 2024-09-17T18:29:25.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9be6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:25.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9be6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:25.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9be6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:25.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9be67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:25.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9be67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:25.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9be67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:25.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9be6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:25.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9be6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:25.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9be6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:25.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9be77fe640 1 --2- 172.21.15.121:0/306106628 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9be81389c0 0x7f9be81253d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:25.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9be6ffd640 10 monclient(hunting): handle_auth_done global_id 44559 payload 824 2024-09-17T18:29:25.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9be6ffd640 1 -- 172.21.15.121:0/306106628 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9be80a5fe0 msgr2=0x7f9be81ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:25.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9be6ffd640 1 --2- 172.21.15.121:0/306106628 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9be80a5fe0 0x7f9be81ce190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:25.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9be6ffd640 1 -- 172.21.15.121:0/306106628 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9be81389c0 msgr2=0x7f9be81253d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:25.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9be6ffd640 1 --2- 172.21.15.121:0/306106628 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9be81389c0 0x7f9be81253d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:25.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9be6ffd640 10 monclient: _finish_hunting 0 2024-09-17T18:29:25.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9be6ffd640 1 monclient: found mon.noname-a 2024-09-17T18:29:25.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9be6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:29:25.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9be6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:29:25.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9be6ffd640 1 -- 172.21.15.121:0/306106628 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9be80a23f0 con 0x7f9be80a5c00 2024-09-17T18:29:25.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9be77fe640 10 monclient: get_auth_request con 0x7f9be81389c0 auth_method 0 2024-09-17T18:29:25.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9be77fe640 1 --2- 172.21.15.121:0/306106628 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9be81389c0 0x7f9be81253d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T18:29:25.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.721+0000 7f9be67fc640 1 --2- 172.21.15.121:0/306106628 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9be80a5fe0 0x7f9be81ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:29:25.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.723+0000 7f9be6ffd640 1 -- 172.21.15.121:0/306106628 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9be80a5c00 msgr2=0x7f9be81cdc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:25.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.723+0000 7f9be6ffd640 1 --2- 172.21.15.121:0/306106628 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9be80a5c00 0x7f9be81cdc50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9bd800e900 tx=0x7f9bd800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:25.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.723+0000 7f9bdffff640 10 client.?.objecter ms_handle_connect 0x7f9be80a5c00 2024-09-17T18:29:25.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.723+0000 7f9bdffff640 10 client.?.objecter resend_mon_ops 2024-09-17T18:29:25.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.723+0000 7f9bdffff640 1 -- 172.21.15.121:0/306106628 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd800cc10 con 0x7f9be80a5c00 2024-09-17T18:29:25.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.723+0000 7f9bdffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:25.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.723+0000 7f9bdffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:29:25.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.723+0000 7f9bdffff640 10 monclient: dump: 2024-09-17T18:29:25.737 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:25.737 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:25.737 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.738 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.738 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:25.738 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:25.738 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:25.738 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:25.738 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:25.738 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:25.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.723+0000 7f9bdffff640 10 monclient: _finish_auth 0 2024-09-17T18:29:25.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.723+0000 7f9bdffff640 10 monclient: _check_auth_tickets 2024-09-17T18:29:25.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.723+0000 7f9bdffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:29:25.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.723+0000 7f9bed984640 5 monclient: authenticate success, global_id 44559 2024-09-17T18:29:25.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.723+0000 7f9bdffff640 1 -- 172.21.15.121:0/306106628 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bd800cdb0 con 0x7f9be80a5c00 2024-09-17T18:29:25.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.723+0000 7f9bed984640 10 monclient: _renew_subs 2024-09-17T18:29:25.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.723+0000 7f9bed984640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:29:25.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.723+0000 7f9bed984640 1 -- 172.21.15.121:0/306106628 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9be8128f10 con 0x7f9be80a5c00 2024-09-17T18:29:25.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.723+0000 7f9bdffff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:29:25.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.723+0000 7f9bdffff640 1 -- 172.21.15.121:0/306106628 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd8004250 con 0x7f9be80a5c00 2024-09-17T18:29:25.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.723+0000 7f9bdffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:25.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.723+0000 7f9bdffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:29:25.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.723+0000 7f9bdffff640 10 monclient: dump: 2024-09-17T18:29:25.739 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:25.739 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:25.739 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.739 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:25.739 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:25.739 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:25.739 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:25.739 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:25.739 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:25.739 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:25.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.723+0000 7f9be57fa640 4 set_mon_vals no callback set 2024-09-17T18:29:25.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.723+0000 7f9bed984640 10 client.44559.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:29:25.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.723+0000 7f9bed984640 10 monclient: _renew_subs 2024-09-17T18:29:25.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.723+0000 7f9bed984640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:29:25.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.723+0000 7f9bed984640 1 -- 172.21.15.121:0/306106628 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9be81279f0 con 0x7f9be80a5c00 2024-09-17T18:29:25.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.723+0000 7f9bed984640 1 librados: init done 2024-09-17T18:29:25.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.724+0000 7f9bed984640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:29:25.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.724+0000 7f9bed984640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:29:25.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.724+0000 7f9bed984640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:29:25.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.724+0000 7f9bdffff640 1 -- 172.21.15.121:0/306106628 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f9bd8003610 con 0x7f9be80a5c00 2024-09-17T18:29:25.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.724+0000 7f9bed984640 1 -- 172.21.15.121:0/306106628 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9be80aa280 con 0x7f9be80a5c00 2024-09-17T18:29:25.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.725+0000 7f9bdffff640 1 --2- 172.21.15.121:0/306106628 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9bc80401a0 0x7f9bc8042660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:25.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.725+0000 7f9bdffff640 1 -- 172.21.15.121:0/306106628 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(410..410 src has 247..410) ==== 5358+0+0 (secure 0 0 0) 0x7f9bd8053710 con 0x7f9be80a5c00 2024-09-17T18:29:25.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.725+0000 7f9bdffff640 10 client.44559.objecter ms_dispatch 0x7f9be8000b90 osd_map(410..410 src has 247..410) 2024-09-17T18:29:25.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.725+0000 7f9bdffff640 3 client.44559.objecter handle_osd_map got epochs [410,410] > 0 2024-09-17T18:29:25.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.725+0000 7f9bdffff640 3 client.44559.objecter handle_osd_map decoding full epoch 410 2024-09-17T18:29:25.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.725+0000 7f9bdffff640 20 client.44559.objecter dump_active .. 0 homeless 2024-09-17T18:29:25.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.725+0000 7f9be67fc640 1 --2- 172.21.15.121:0/306106628 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9bc80401a0 0x7f9bc8042660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:25.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.725+0000 7f9be67fc640 10 monclient: get_auth_request con 0x7f9bc80401a0 auth_method 0 2024-09-17T18:29:25.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.727+0000 7f9bdffff640 10 client.44559.objecter ms_handle_connect 0x7f9bc80401a0 2024-09-17T18:29:25.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.727+0000 7f9be67fc640 1 --2- 172.21.15.121:0/306106628 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9bc80401a0 0x7f9bc8042660 secure :-1 s=READY pgs=4011 cs=0 l=1 rev1=1 crypto rx=0x7f9bd00097a0 tx=0x7f9bd002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:25.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.729+0000 7f9bdffff640 1 -- 172.21.15.121:0/306106628 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9be80aa280 con 0x7f9be80a5c00 2024-09-17T18:29:25.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.729+0000 7f9bdffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:29:25.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.729+0000 7f9bdffff640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:29:25.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.970+0000 7f9bed984640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-17T18:29:25.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.970+0000 7f9bed984640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-17T18:29:25.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.970+0000 7f9bed984640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:29:25.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.970+0000 7f9bed984640 1 -- 172.21.15.121:0/306106628 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f9be81395f0 con 0x7f9be80a5c00 2024-09-17T18:29:25.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.972+0000 7f9bdffff640 1 -- 172.21.15.121:0/306106628 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f9be80a23f0 con 0x7f9be80a5c00 2024-09-17T18:29:25.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.972+0000 7f9bdffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-17T18:29:25.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.972+0000 7f9bdffff640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:29:25.975 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.972+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:25.975 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.972+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=15,config=2,health=11,kv=13,logm=59} crc {auth=2990837629,config=2586702516,health=2069616219,kv=4292787700,logm=3384832719}) 2024-09-17T18:29:25.975 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.972+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=15,config=2,health=12,kv=13,logm=58} crc {auth=2990837629,config=2586702516,health=536953242,kv=2079569364,logm=1727405922}) 2024-09-17T18:29:25.975 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.972+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:25.975 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.972+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=15,config=2,health=11,kv=13,logm=59} crc {auth=2990837629,config=2586702516,health=2069616219,kv=4292787700,logm=3384832719}) 2024-09-17T18:29:25.975 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.972+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=15,config=2,health=10,kv=14,logm=59} crc {auth=2990837629,config=2586702516,health=2486131213,kv=2790417503,logm=827643269}) 2024-09-17T18:29:25.976 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.973+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:25.976 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.973+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=697202619}) 2024-09-17T18:29:25.976 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.973+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=280985650}) 2024-09-17T18:29:25.976 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.973+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:25.976 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.973+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=697202619}) 2024-09-17T18:29:25.977 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.973+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4122760854}) 2024-09-17T18:29:25.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.974+0000 7f9bed984640 10 librados: watch_flush enter 2024-09-17T18:29:25.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.974+0000 7f9bed984640 10 librados: watch_flush exit 2024-09-17T18:29:25.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.974+0000 7f9bed984640 20 client.44559.objecter shutdown clearing up homeless session... 2024-09-17T18:29:25.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.974+0000 7f9bed984640 10 client.44559.objecter successfully canceled tick 2024-09-17T18:29:25.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.974+0000 7f9bed984640 1 -- 172.21.15.121:0/306106628 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9bc80401a0 msgr2=0x7f9bc8042660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:25.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.974+0000 7f9bed984640 1 --2- 172.21.15.121:0/306106628 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9bc80401a0 0x7f9bc8042660 secure :-1 s=READY pgs=4011 cs=0 l=1 rev1=1 crypto rx=0x7f9bd00097a0 tx=0x7f9bd002e000 comp rx=0 tx=0).stop 2024-09-17T18:29:25.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.974+0000 7f9bed984640 10 monclient: shutdown 2024-09-17T18:29:25.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.974+0000 7f9bed984640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:29:25.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.974+0000 7f9bed984640 1 -- 172.21.15.121:0/306106628 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9be80a5c00 msgr2=0x7f9be81cdc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:25.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.974+0000 7f9bed984640 1 --2- 172.21.15.121:0/306106628 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9be80a5c00 0x7f9be81cdc50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9bd800e900 tx=0x7f9bd800edc0 comp rx=0 tx=0).stop 2024-09-17T18:29:25.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.974+0000 7f9bed984640 1 -- 172.21.15.121:0/306106628 shutdown_connections 2024-09-17T18:29:25.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.974+0000 7f9bed984640 1 --2- 172.21.15.121:0/306106628 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9bc80401a0 0x7f9bc8042660 unknown :-1 s=CLOSED pgs=4011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:25.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.974+0000 7f9bed984640 1 --2- 172.21.15.121:0/306106628 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9be81389c0 0x7f9be81253d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:25.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.974+0000 7f9bed984640 1 --2- 172.21.15.121:0/306106628 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9be80a5fe0 0x7f9be81ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:25.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.974+0000 7f9bed984640 1 --2- 172.21.15.121:0/306106628 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9be80a5c00 0x7f9be81cdc50 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:25.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.974+0000 7f9bed984640 1 -- 172.21.15.121:0/306106628 >> 172.21.15.121:0/306106628 conn(0x7f9be80ab9b0 msgr2=0x7f9be80a4800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:29:25.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.974+0000 7f9bed984640 1 -- 172.21.15.121:0/306106628 shutdown_connections 2024-09-17T18:29:25.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.974+0000 7f9bed984640 1 -- 172.21.15.121:0/306106628 wait complete. 2024-09-17T18:29:25.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:25.974+0000 7f9bed984640 1 librados: shutdown 2024-09-17T18:29:25.981 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.974+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:25.982 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.974+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3340927945}) 2024-09-17T18:29:25.982 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.974+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1430733712}) 2024-09-17T18:29:25.982 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.974+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:25.982 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.974+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3340927945}) 2024-09-17T18:29:25.982 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.974+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1145644064}) 2024-09-17T18:29:25.982 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.975+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:25.982 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.975+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3908852680}) 2024-09-17T18:29:25.982 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.975+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2962992726}) 2024-09-17T18:29:25.982 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.975+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:25.982 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.975+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3908852680}) 2024-09-17T18:29:25.982 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.975+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=278888567}) 2024-09-17T18:29:25.982 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.977+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:25.982 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.977+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=660323920}) 2024-09-17T18:29:25.982 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.977+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=115273556}) 2024-09-17T18:29:25.983 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.977+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:25.983 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.977+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=660323920}) 2024-09-17T18:29:25.983 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.977+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1771302472}) 2024-09-17T18:29:25.983 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.978+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:25.983 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.978+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1762939168}) 2024-09-17T18:29:25.983 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.978+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3775778141}) 2024-09-17T18:29:25.983 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.978+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:25.983 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.978+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1762939168}) 2024-09-17T18:29:25.983 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.978+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=669549564}) 2024-09-17T18:29:25.983 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.978+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:25.983 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.978+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1318203671}) 2024-09-17T18:29:25.983 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.978+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1539035819}) 2024-09-17T18:29:25.983 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.978+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:25.983 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.979+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1318203671}) 2024-09-17T18:29:25.983 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.979+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=318798284}) 2024-09-17T18:29:25.984 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.979+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:25.984 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.979+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2773392541}) 2024-09-17T18:29:25.984 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.979+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3805419531}) 2024-09-17T18:29:25.984 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.979+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:25.984 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.979+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2773392541}) 2024-09-17T18:29:25.984 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:25.980+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=106525068}) 2024-09-17T18:29:25.988 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-17T18:29:26.022 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.019+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:26.022 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.019+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2340177021}) 2024-09-17T18:29:26.022 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.019+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2716165546}) 2024-09-17T18:29:26.022 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.019+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:26.022 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.019+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2340177021}) 2024-09-17T18:29:26.022 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.019+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4112030940}) 2024-09-17T18:29:26.375 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.373+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:26.376 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.373+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=990336501}) 2024-09-17T18:29:26.376 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.373+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4014734252}) 2024-09-17T18:29:26.376 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.373+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:26.376 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.373+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=990336501}) 2024-09-17T18:29:26.376 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.373+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2270311084}) 2024-09-17T18:29:26.419 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.416+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:26.419 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.416+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2076909296}) 2024-09-17T18:29:26.419 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.416+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1426671600}) 2024-09-17T18:29:26.420 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.416+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:26.420 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.416+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2076909296}) 2024-09-17T18:29:26.420 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.416+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4290981993}) 2024-09-17T18:29:26.422 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.419+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:26.422 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.419+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=436979616}) 2024-09-17T18:29:26.422 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.419+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1013666875}) 2024-09-17T18:29:26.423 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.419+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:26.423 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.419+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=436979616}) 2024-09-17T18:29:26.423 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.419+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2660686379}) 2024-09-17T18:29:26.429 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.426+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:26.429 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.426+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=730347327}) 2024-09-17T18:29:26.429 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.426+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=839334840}) 2024-09-17T18:29:26.429 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.426+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:26.429 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.426+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=730347327}) 2024-09-17T18:29:26.429 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.426+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=520989625}) 2024-09-17T18:29:26.700 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.698+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:26.700 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.698+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2652043200}) 2024-09-17T18:29:26.700 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.698+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1197591102}) 2024-09-17T18:29:26.700 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.698+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:26.700 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.698+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2652043200}) 2024-09-17T18:29:26.700 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.698+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=357018323}) 2024-09-17T18:29:26.701 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.698+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:26.701 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.698+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1769388391}) 2024-09-17T18:29:26.701 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.698+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4119944364}) 2024-09-17T18:29:26.701 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.698+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:26.701 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.698+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1769388391}) 2024-09-17T18:29:26.702 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.698+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1031457556}) 2024-09-17T18:29:26.702 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.699+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:26.702 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.699+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2177687585}) 2024-09-17T18:29:26.702 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.699+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=366029768}) 2024-09-17T18:29:26.702 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.699+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:26.702 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.699+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2177687585}) 2024-09-17T18:29:26.702 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.699+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2849300479}) 2024-09-17T18:29:26.703 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.700+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:26.704 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.700+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=925070206}) 2024-09-17T18:29:26.704 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.700+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=568405944}) 2024-09-17T18:29:26.704 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.700+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:26.704 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.700+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=925070206}) 2024-09-17T18:29:26.704 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.700+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1531653714}) 2024-09-17T18:29:26.704 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.701+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:26.704 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.701+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=248394736}) 2024-09-17T18:29:26.704 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.701+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=923606806}) 2024-09-17T18:29:26.704 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.701+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:26.704 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.701+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=248394736}) 2024-09-17T18:29:26.704 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.701+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1818636368}) 2024-09-17T18:29:26.705 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.701+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:26.705 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.701+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3780236960}) 2024-09-17T18:29:26.705 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.701+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4023145533}) 2024-09-17T18:29:26.705 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.701+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:26.705 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.701+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3780236960}) 2024-09-17T18:29:26.705 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.701+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=266537200}) 2024-09-17T18:29:26.709 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.707+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:26.709 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.707+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=941435577}) 2024-09-17T18:29:26.709 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.707+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=642579506}) 2024-09-17T18:29:26.709 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.707+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:26.709 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.707+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=941435577}) 2024-09-17T18:29:26.709 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.707+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1872617727}) 2024-09-17T18:29:26.710 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.708+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:26.710 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.708+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2395569187}) 2024-09-17T18:29:26.710 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.708+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=213250330}) 2024-09-17T18:29:26.710 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.708+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:26.711 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.708+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2395569187}) 2024-09-17T18:29:26.711 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:26.708+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1304326332}) 2024-09-17T18:29:26.989 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T18:29:26.989 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:29:27.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.091+0000 7f039c885640 10 monclient: get_monmap_and_config 2024-09-17T18:29:27.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.091+0000 7f039c885640 10 monclient: build_initial_monmap 2024-09-17T18:29:27.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.091+0000 7f039c885640 10 monclient: monmap: 2024-09-17T18:29:27.094 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:29:27.094 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:27.094 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:29:27.092706+0000 2024-09-17T18:29:27.094 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:29:27.092706+0000 2024-09-17T18:29:27.094 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:29:27.094 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:27.094 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:29:27.094 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:29:27.094 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:29:27.094 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:29:27.094 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:29:27.094 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:27.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.091+0000 7f039c885640 1 Processor -- start 2024-09-17T18:29:27.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.091+0000 7f039c885640 1 -- start start 2024-09-17T18:29:27.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.091+0000 7f039c885640 10 monclient: init 2024-09-17T18:29:27.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.092+0000 7f039c885640 10 monclient: _reopen_session rank -1 2024-09-17T18:29:27.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.092+0000 7f039c885640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-17T18:29:27.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.092+0000 7f039c885640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f03980a5c00 0x7f039809d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:27.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.092+0000 7f039c885640 10 monclient(hunting): picked mon.noname-a con 0x7f03980a5c00 addr v2:172.21.15.121:3300/0 2024-09-17T18:29:27.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.092+0000 7f039c885640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f03980a5fe0 0x7f03980a63f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:27.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.092+0000 7f039c885640 10 monclient(hunting): picked mon.noname-c con 0x7f03980a5fe0 addr v2:172.21.15.173:3300/0 2024-09-17T18:29:27.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.092+0000 7f039c885640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f03981389c0 addr v1:172.21.15.173:6789/0 2024-09-17T18:29:27.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.092+0000 7f039c885640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03980a8a80 con 0x7f03981389c0 2024-09-17T18:29:27.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.092+0000 7f039c885640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:27.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.092+0000 7f039c885640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f039809de00 con 0x7f03980a5c00 2024-09-17T18:29:27.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.092+0000 7f039c885640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:27.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.092+0000 7f039c885640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f03980a74d0 con 0x7f03980a5fe0 2024-09-17T18:29:27.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.092+0000 7f039c885640 10 monclient(hunting): _renew_subs 2024-09-17T18:29:27.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.092+0000 7f039c885640 10 monclient(hunting): authenticate will time out at 1370.809506s 2024-09-17T18:29:27.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.092+0000 7f039659c640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f03980a5c00 0x7f039809d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:27.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.092+0000 7f039659c640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f03980a5c00 0x7f039809d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58088/0 (socket says 172.21.15.121:58088) 2024-09-17T18:29:27.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.092+0000 7f039659c640 1 -- 172.21.15.121:0/186538969 learned_addr learned my addr 172.21.15.121:0/186538969 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:29:27.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.092+0000 7f039659c640 10 monclient(hunting): get_auth_request con 0x7f03980a5c00 auth_method 0 2024-09-17T18:29:27.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.092+0000 7f039659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:27.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.092+0000 7f039659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:27.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.092+0000 7f039659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:27.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.092+0000 7f039659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:27.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.092+0000 7f039659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:27.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.092+0000 7f039659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:27.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.092+0000 7f0395d9b640 1 --2- 172.21.15.121:0/186538969 >> v2:172.21.15.173:3300/0 conn(0x7f03980a5fe0 0x7f03980a63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:27.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.092+0000 7f0395d9b640 10 monclient(hunting): get_auth_request con 0x7f03980a5fe0 auth_method 0 2024-09-17T18:29:27.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.092+0000 7f0395d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:27.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.092+0000 7f0395d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:27.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.092+0000 7f0395d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:27.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.092+0000 7f039659c640 10 monclient(hunting): handle_auth_done global_id 44562 payload 274 2024-09-17T18:29:27.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.092+0000 7f039659c640 1 -- 172.21.15.121:0/186538969 >> v2:172.21.15.173:3300/0 conn(0x7f03980a5fe0 msgr2=0x7f03980a63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:29:27.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.092+0000 7f039659c640 1 --2- 172.21.15.121:0/186538969 >> v2:172.21.15.173:3300/0 conn(0x7f03980a5fe0 0x7f03980a63f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.092+0000 7f039659c640 1 -- 172.21.15.121:0/186538969 >> v1:172.21.15.173:6789/0 conn(0x7f03981389c0 legacy=0x7f0398138dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:29:27.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.092+0000 7f039659c640 10 monclient: _finish_hunting 0 2024-09-17T18:29:27.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.092+0000 7f039659c640 1 monclient: found mon.noname-a 2024-09-17T18:29:27.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.092+0000 7f039659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:29:27.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.092+0000 7f039659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:29:27.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.092+0000 7f039659c640 1 -- 172.21.15.121:0/186538969 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f039809cb90 con 0x7f03980a5c00 2024-09-17T18:29:27.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.095+0000 7f039659c640 1 -- 172.21.15.121:0/186538969 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03980a5c00 msgr2=0x7f039809d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:27.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.095+0000 7f039659c640 1 --2- 172.21.15.121:0/186538969 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03980a5c00 0x7f039809d8c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f039806a560 tx=0x7f038c02a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd5cd005bb4b09c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:27.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.095+0000 7f039559a640 1 -- 172.21.15.121:0/186538969 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f038c036070 con 0x7f03980a5c00 2024-09-17T18:29:27.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.095+0000 7f039559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:27.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.095+0000 7f039559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:29:27.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.095+0000 7f039559a640 10 monclient: dump: 2024-09-17T18:29:27.106 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:27.107 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:27.107 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.107 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.107 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:27.107 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:27.107 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:27.107 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:27.107 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:27.107 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:27.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.095+0000 7f039559a640 10 monclient: _finish_auth 0 2024-09-17T18:29:27.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.095+0000 7f039559a640 10 monclient: _check_auth_tickets 2024-09-17T18:29:27.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.095+0000 7f039559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:29:27.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.095+0000 7f039559a640 1 -- 172.21.15.121:0/186538969 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f038c007800 con 0x7f03980a5c00 2024-09-17T18:29:27.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.095+0000 7f039c885640 5 monclient: authenticate success, global_id 44562 2024-09-17T18:29:27.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.095+0000 7f039c885640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:29:27.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.095+0000 7f039559a640 10 monclient: handle_config config(0 keys) 2024-09-17T18:29:27.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.095+0000 7f039c885640 10 monclient: get_monmap_and_config success 2024-09-17T18:29:27.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.095+0000 7f039c885640 4 set_mon_vals no callback set 2024-09-17T18:29:27.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.095+0000 7f039559a640 1 -- 172.21.15.121:0/186538969 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f038c007bd0 con 0x7f03980a5c00 2024-09-17T18:29:27.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.095+0000 7f039559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:27.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.095+0000 7f039559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:29:27.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.095+0000 7f039559a640 10 monclient: dump: 2024-09-17T18:29:27.108 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:27.108 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:27.108 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.108 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.109 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:27.109 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:27.109 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:27.109 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:27.109 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:27.109 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:27.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.096+0000 7f039c885640 10 monclient: shutdown 2024-09-17T18:29:27.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.096+0000 7f039c885640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:29:27.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.096+0000 7f039c885640 1 -- 172.21.15.121:0/186538969 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03980a5c00 msgr2=0x7f039809d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:27.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.096+0000 7f039c885640 1 --2- 172.21.15.121:0/186538969 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03980a5c00 0x7f039809d8c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f039806a560 tx=0x7f038c02a570 comp rx=0 tx=0).stop 2024-09-17T18:29:27.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.096+0000 7f039c885640 1 -- 172.21.15.121:0/186538969 shutdown_connections 2024-09-17T18:29:27.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.096+0000 7f039c885640 1 --2- 172.21.15.121:0/186538969 >> v2:172.21.15.173:3300/0 conn(0x7f03980a5fe0 0x7f03980a63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.096+0000 7f039c885640 1 --2- 172.21.15.121:0/186538969 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03980a5c00 0x7f039809d8c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.096+0000 7f039c885640 1 -- 172.21.15.121:0/186538969 >> 172.21.15.121:0/186538969 conn(0x7f03980ab9b0 msgr2=0x7f03980a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:29:27.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.096+0000 7f039c885640 1 -- 172.21.15.121:0/186538969 shutdown_connections 2024-09-17T18:29:27.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.096+0000 7f039c885640 1 -- 172.21.15.121:0/186538969 wait complete. 2024-09-17T18:29:27.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.097+0000 7f039c885640 10 monclient: build_initial_monmap 2024-09-17T18:29:27.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.097+0000 7f039c885640 10 monclient: monmap: 2024-09-17T18:29:27.110 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:29:27.110 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:29:27.110 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:29:27.110 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:29:27.110 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:29:27.110 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:27.110 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:29:27.110 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:29:27.110 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:29:27.110 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:27.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.097+0000 7f039c885640 1 librados: starting msgr at 2024-09-17T18:29:27.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.097+0000 7f039c885640 1 librados: starting objecter 2024-09-17T18:29:27.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.097+0000 7f039c885640 1 Processor -- start 2024-09-17T18:29:27.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.097+0000 7f039c885640 1 -- start start 2024-09-17T18:29:27.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.097+0000 7f039c885640 1 librados: setting wanted keys 2024-09-17T18:29:27.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.097+0000 7f039c885640 1 librados: calling monclient init 2024-09-17T18:29:27.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.097+0000 7f039c885640 10 monclient: init 2024-09-17T18:29:27.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f039c885640 10 monclient: _reopen_session rank -1 2024-09-17T18:29:27.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f039c885640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:29:27.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f039c885640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03980a5c00 0x7f0398125090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:27.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f039c885640 10 monclient(hunting): picked mon.noname-a con 0x7f03980a5c00 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:29:27.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f039c885640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03980a5fe0 0x7f03981255d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:27.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f039c885640 10 monclient(hunting): picked mon.noname-b con 0x7f03980a5fe0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:29:27.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f039c885640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03981389c0 0x7f0398125b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:27.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f039c885640 10 monclient(hunting): picked mon.noname-c con 0x7f03981389c0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:29:27.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f039c885640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:27.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f039c885640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f03980703d0 con 0x7f03980a5c00 2024-09-17T18:29:27.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f039c885640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:27.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f039c885640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f03980a6930 con 0x7f03980a5fe0 2024-09-17T18:29:27.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f039c885640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:27.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f039c885640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0398070550 con 0x7f03981389c0 2024-09-17T18:29:27.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f039c885640 10 monclient(hunting): _renew_subs 2024-09-17T18:29:27.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f039c885640 10 monclient(hunting): authenticate will time out at 1370.815155s 2024-09-17T18:29:27.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f039659c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03980a5c00 0x7f0398125090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:27.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f039659c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03980a5c00 0x7f0398125090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58102/0 (socket says 172.21.15.121:58102) 2024-09-17T18:29:27.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f039659c640 1 -- 172.21.15.121:0/1116708879 learned_addr learned my addr 172.21.15.121:0/1116708879 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:29:27.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f039659c640 10 monclient(hunting): get_auth_request con 0x7f03980a5c00 auth_method 0 2024-09-17T18:29:27.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f039659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:27.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f039659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:27.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f039659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:27.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f0396d9d640 1 --2- 172.21.15.121:0/1116708879 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03981389c0 0x7f0398125b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:27.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f0396d9d640 10 monclient(hunting): get_auth_request con 0x7f03981389c0 auth_method 0 2024-09-17T18:29:27.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f0396d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:27.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f0396d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:27.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f0396d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:27.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f039659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:27.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f039659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:27.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f039659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:27.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f0396d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:27.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f0396d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:27.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f0396d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:27.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f0395d9b640 1 --2- 172.21.15.121:0/1116708879 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03980a5fe0 0x7f03981255d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:27.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f0395d9b640 10 monclient(hunting): get_auth_request con 0x7f03980a5fe0 auth_method 0 2024-09-17T18:29:27.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f0395d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:27.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f0395d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:27.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f0395d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:27.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f0395d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:27.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f0395d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:27.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f0395d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:27.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f039659c640 10 monclient(hunting): handle_auth_done global_id 44565 payload 824 2024-09-17T18:29:27.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f039659c640 1 -- 172.21.15.121:0/1116708879 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03981389c0 msgr2=0x7f0398125b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:27.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f039659c640 1 --2- 172.21.15.121:0/1116708879 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03981389c0 0x7f0398125b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f039659c640 1 -- 172.21.15.121:0/1116708879 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03980a5fe0 msgr2=0x7f03981255d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:27.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f039659c640 1 --2- 172.21.15.121:0/1116708879 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03980a5fe0 0x7f03981255d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f039659c640 10 monclient: _finish_hunting 0 2024-09-17T18:29:27.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f039659c640 1 monclient: found mon.noname-a 2024-09-17T18:29:27.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f039659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:29:27.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f039659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:29:27.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.098+0000 7f039659c640 1 -- 172.21.15.121:0/1116708879 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0398128e20 con 0x7f03980a5c00 2024-09-17T18:29:27.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.100+0000 7f039659c640 1 -- 172.21.15.121:0/1116708879 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03980a5c00 msgr2=0x7f0398125090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:27.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.100+0000 7f039659c640 1 --2- 172.21.15.121:0/1116708879 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03980a5c00 0x7f0398125090 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f038c034040 tx=0x7f038c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:27.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.100+0000 7f0386ffd640 10 client.?.objecter ms_handle_connect 0x7f03980a5c00 2024-09-17T18:29:27.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.100+0000 7f0386ffd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:29:27.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.100+0000 7f0386ffd640 1 -- 172.21.15.121:0/1116708879 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f038c036050 con 0x7f03980a5c00 2024-09-17T18:29:27.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.100+0000 7f0386ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:27.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.100+0000 7f0386ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:29:27.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.100+0000 7f0386ffd640 10 monclient: dump: 2024-09-17T18:29:27.115 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:27.115 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:27.115 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.115 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.115 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:27.115 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:27.115 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:27.115 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:27.115 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:27.115 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:27.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.100+0000 7f0386ffd640 10 monclient: _finish_auth 0 2024-09-17T18:29:27.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.100+0000 7f0386ffd640 10 monclient: _check_auth_tickets 2024-09-17T18:29:27.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.100+0000 7f0386ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:29:27.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.100+0000 7f0386ffd640 1 -- 172.21.15.121:0/1116708879 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f038c042070 con 0x7f03980a5c00 2024-09-17T18:29:27.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.100+0000 7f039c885640 5 monclient: authenticate success, global_id 44565 2024-09-17T18:29:27.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.101+0000 7f039c885640 10 monclient: _renew_subs 2024-09-17T18:29:27.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.101+0000 7f039c885640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:29:27.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.101+0000 7f039c885640 1 -- 172.21.15.121:0/1116708879 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0398138760 con 0x7f03980a5c00 2024-09-17T18:29:27.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.101+0000 7f0386ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:29:27.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.101+0000 7f0386ffd640 1 -- 172.21.15.121:0/1116708879 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f038c031e10 con 0x7f03980a5c00 2024-09-17T18:29:27.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.101+0000 7f0386ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:27.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.101+0000 7f0386ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:29:27.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.101+0000 7f0386ffd640 10 monclient: dump: 2024-09-17T18:29:27.116 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:27.116 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:27.116 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.116 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.117 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:27.117 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:27.117 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:27.117 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:27.117 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:27.117 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:27.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.101+0000 7f039c885640 10 client.44565.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:29:27.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.101+0000 7f039c885640 10 monclient: _renew_subs 2024-09-17T18:29:27.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.101+0000 7f0394d99640 4 set_mon_vals no callback set 2024-09-17T18:29:27.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.101+0000 7f039c885640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:29:27.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.101+0000 7f039c885640 1 -- 172.21.15.121:0/1116708879 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0398127fb0 con 0x7f03980a5c00 2024-09-17T18:29:27.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.101+0000 7f039c885640 1 librados: init done 2024-09-17T18:29:27.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.102+0000 7f0386ffd640 1 -- 172.21.15.121:0/1116708879 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f038c047460 con 0x7f03980a5c00 2024-09-17T18:29:27.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.102+0000 7f039c885640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:29:27.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.102+0000 7f039c885640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:29:27.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.102+0000 7f039c885640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:29:27.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.102+0000 7f039c885640 1 -- 172.21.15.121:0/1116708879 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f039809d8c0 con 0x7f03980a5c00 2024-09-17T18:29:27.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.102+0000 7f0386ffd640 1 --2- 172.21.15.121:0/1116708879 >> v2:172.21.15.173:6812/2423786037 conn(0x7f03780401c0 0x7f0378042680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:27.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.102+0000 7f0386ffd640 1 -- 172.21.15.121:0/1116708879 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(411..411 src has 247..411) ==== 4973+0+0 (secure 0 0 0) 0x7f038c0745e0 con 0x7f03980a5c00 2024-09-17T18:29:27.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.102+0000 7f0386ffd640 10 client.44565.objecter ms_dispatch 0x7f0398000b90 osd_map(411..411 src has 247..411) 2024-09-17T18:29:27.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.102+0000 7f0386ffd640 3 client.44565.objecter handle_osd_map got epochs [411,411] > 0 2024-09-17T18:29:27.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.102+0000 7f0386ffd640 3 client.44565.objecter handle_osd_map decoding full epoch 411 2024-09-17T18:29:27.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.102+0000 7f0386ffd640 20 client.44565.objecter dump_active .. 0 homeless 2024-09-17T18:29:27.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.102+0000 7f0395d9b640 1 --2- 172.21.15.121:0/1116708879 >> v2:172.21.15.173:6812/2423786037 conn(0x7f03780401c0 0x7f0378042680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:27.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.102+0000 7f0395d9b640 10 monclient: get_auth_request con 0x7f03780401c0 auth_method 0 2024-09-17T18:29:27.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.105+0000 7f0386ffd640 10 client.44565.objecter ms_handle_connect 0x7f03780401c0 2024-09-17T18:29:27.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.105+0000 7f0395d9b640 1 --2- 172.21.15.121:0/1116708879 >> v2:172.21.15.173:6812/2423786037 conn(0x7f03780401c0 0x7f0378042680 secure :-1 s=READY pgs=4012 cs=0 l=1 rev1=1 crypto rx=0x7f0388006fd0 tx=0x7f0388008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:27.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.107+0000 7f0386ffd640 1 -- 172.21.15.121:0/1116708879 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f038c072f20 con 0x7f03980a5c00 2024-09-17T18:29:27.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.107+0000 7f0386ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:29:27.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.107+0000 7f0386ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:29:27.211 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.208+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.211 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.208+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2665815106}) 2024-09-17T18:29:27.211 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.208+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2120661261}) 2024-09-17T18:29:27.211 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.208+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.211 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.208+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2665815106}) 2024-09-17T18:29:27.211 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.208+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2335747648}) 2024-09-17T18:29:27.212 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.209+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.212 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.209+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3455482045}) 2024-09-17T18:29:27.212 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.209+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1969043744}) 2024-09-17T18:29:27.212 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.209+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.212 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.209+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3455482045}) 2024-09-17T18:29:27.212 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.209+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1094814088}) 2024-09-17T18:29:27.213 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.210+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.213 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.210+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4293036305}) 2024-09-17T18:29:27.213 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.210+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2322005735}) 2024-09-17T18:29:27.213 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.210+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.214 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.210+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4293036305}) 2024-09-17T18:29:27.214 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.210+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1410504516}) 2024-09-17T18:29:27.214 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.211+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.214 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.211+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1560070387}) 2024-09-17T18:29:27.215 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.211+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=307435130}) 2024-09-17T18:29:27.215 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.211+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.215 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.211+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1560070387}) 2024-09-17T18:29:27.215 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.211+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3178847506}) 2024-09-17T18:29:27.215 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.212+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.216 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.212+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2192320380}) 2024-09-17T18:29:27.216 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.212+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3117270266}) 2024-09-17T18:29:27.216 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.212+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.216 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.212+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2192320380}) 2024-09-17T18:29:27.216 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.212+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1014983197}) 2024-09-17T18:29:27.216 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.213+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.217 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.213+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=773817757}) 2024-09-17T18:29:27.217 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.213+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3553628953}) 2024-09-17T18:29:27.217 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.213+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.217 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.213+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=773817757}) 2024-09-17T18:29:27.217 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.213+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=842506444}) 2024-09-17T18:29:27.217 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.214+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.217 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.214+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3612925709}) 2024-09-17T18:29:27.218 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.214+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3840865389}) 2024-09-17T18:29:27.218 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.214+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.218 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.214+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3612925709}) 2024-09-17T18:29:27.218 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.214+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3447295524}) 2024-09-17T18:29:27.218 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.215+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.219 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.215+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1893678594}) 2024-09-17T18:29:27.219 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.215+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=511249931}) 2024-09-17T18:29:27.219 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.215+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.219 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.215+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1893678594}) 2024-09-17T18:29:27.219 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.215+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2839896076}) 2024-09-17T18:29:27.219 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.216+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.220 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.216+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1542769228}) 2024-09-17T18:29:27.220 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.216+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1118773679}) 2024-09-17T18:29:27.220 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.216+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.220 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.216+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1542769228}) 2024-09-17T18:29:27.220 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.216+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=257261243}) 2024-09-17T18:29:27.220 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.217+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.221 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.217+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2287936751}) 2024-09-17T18:29:27.221 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.217+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=8374416}) 2024-09-17T18:29:27.221 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.217+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.221 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.217+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2287936751}) 2024-09-17T18:29:27.221 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.217+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=580930723}) 2024-09-17T18:29:27.222 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.218+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.222 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.218+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2223678384}) 2024-09-17T18:29:27.222 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.218+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1816835839}) 2024-09-17T18:29:27.222 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.218+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.222 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.218+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2223678384}) 2024-09-17T18:29:27.223 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.218+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=702982037}) 2024-09-17T18:29:27.223 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.219+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.223 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.219+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1536766265}) 2024-09-17T18:29:27.223 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.219+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1804837164}) 2024-09-17T18:29:27.223 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.219+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.223 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.219+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1536766265}) 2024-09-17T18:29:27.223 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.219+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1313870553}) 2024-09-17T18:29:27.224 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.220+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.224 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.220+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3635872005}) 2024-09-17T18:29:27.224 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.220+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=197737949}) 2024-09-17T18:29:27.224 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.220+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.224 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.220+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3635872005}) 2024-09-17T18:29:27.224 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.220+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4231425498}) 2024-09-17T18:29:27.225 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.221+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.225 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.221+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2201485347}) 2024-09-17T18:29:27.225 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.221+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3260680010}) 2024-09-17T18:29:27.225 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.221+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.225 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.222+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2201485347}) 2024-09-17T18:29:27.225 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.222+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=28372292}) 2024-09-17T18:29:27.226 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.222+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.226 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.222+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3144016982}) 2024-09-17T18:29:27.226 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.223+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4126429152}) 2024-09-17T18:29:27.226 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.223+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.226 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.223+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3144016982}) 2024-09-17T18:29:27.226 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.223+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3005210727}) 2024-09-17T18:29:27.227 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.223+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.227 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.224+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2356117321}) 2024-09-17T18:29:27.228 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.224+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1054572851}) 2024-09-17T18:29:27.228 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.224+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.228 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.224+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2356117321}) 2024-09-17T18:29:27.228 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.224+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2397047072}) 2024-09-17T18:29:27.228 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.225+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.228 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.225+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3769827260}) 2024-09-17T18:29:27.228 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.225+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=820772882}) 2024-09-17T18:29:27.228 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.225+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.228 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.225+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3769827260}) 2024-09-17T18:29:27.228 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.225+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=804050878}) 2024-09-17T18:29:27.229 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.226+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.229 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.226+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=700485361}) 2024-09-17T18:29:27.230 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.226+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3386359870}) 2024-09-17T18:29:27.230 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.226+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.230 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.226+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=700485361}) 2024-09-17T18:29:27.230 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.226+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=598352480}) 2024-09-17T18:29:27.230 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.227+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.230 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.227+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=158341604}) 2024-09-17T18:29:27.230 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.227+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4117867096}) 2024-09-17T18:29:27.230 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.227+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.230 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.227+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=158341604}) 2024-09-17T18:29:27.230 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.227+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=42231302}) 2024-09-17T18:29:27.231 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.228+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.231 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.228+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1374681958}) 2024-09-17T18:29:27.231 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.228+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3631242307}) 2024-09-17T18:29:27.231 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.228+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.232 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.228+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1374681958}) 2024-09-17T18:29:27.232 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.228+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2594325666}) 2024-09-17T18:29:27.232 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.229+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.232 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.229+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1422267618}) 2024-09-17T18:29:27.232 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.229+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3171469145}) 2024-09-17T18:29:27.232 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.229+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.232 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.229+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1422267618}) 2024-09-17T18:29:27.232 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.229+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1607207952}) 2024-09-17T18:29:27.233 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.230+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.233 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.230+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1283100416}) 2024-09-17T18:29:27.233 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.230+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3032534747}) 2024-09-17T18:29:27.233 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.230+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.233 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.230+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1283100416}) 2024-09-17T18:29:27.233 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.230+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=994781929}) 2024-09-17T18:29:27.233 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.230+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.234 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.230+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2688850737}) 2024-09-17T18:29:27.234 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.230+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2910026569}) 2024-09-17T18:29:27.234 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.230+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.234 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.230+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2688850737}) 2024-09-17T18:29:27.234 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.230+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2554935051}) 2024-09-17T18:29:27.235 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.232+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.235 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.232+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=4,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=14,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=28,osdmap=19} crc {logm=3365061830,mdsmap=761013554,mgr=3361498112,mgr_command_descs=1957073259,mgr_metadata=402068389,mgrstat=464044374,mon_config_key=1753612445,monmap=2277444362,nvmeofgw=598337686,osd_metadata=284734903,osd_pg_creating=3383230014,osd_snap=3526543088,osdmap=2547766492}) 2024-09-17T18:29:27.235 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.232+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=4,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=16,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=17} crc {logm=332809135,mdsmap=761013554,mgr=3361498112,mgr_command_descs=1957073259,mgr_metadata=402068389,mgrstat=2776292112,mon_config_key=1071828679,monmap=344458555,nvmeofgw=598337686,osd_metadata=121115626,osd_pg_creating=3383230014,osd_snap=1066898354,osdmap=4152856518}) 2024-09-17T18:29:27.235 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.232+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.235 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.232+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=4,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=14,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=28,osdmap=19} crc {logm=3365061830,mdsmap=761013554,mgr=3361498112,mgr_command_descs=1957073259,mgr_metadata=402068389,mgrstat=464044374,mon_config_key=1753612445,monmap=2277444362,nvmeofgw=598337686,osd_metadata=284734903,osd_pg_creating=3383230014,osd_snap=3526543088,osdmap=2547766492}) 2024-09-17T18:29:27.235 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.232+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=3,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=14} crc {logm=4260789531,mdsmap=761013554,mgr=3361498112,mgr_command_descs=1957073259,mgr_metadata=402068389,mgrstat=2268908909,mon_config_key=1582663684,monmap=2277444362,nvmeofgw=598337686,osd_metadata=121115626,osd_pg_creating=3383230014,osd_snap=1161780173,osdmap=3187476591}) 2024-09-17T18:29:27.236 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.233+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.236 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.233+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2573250565}) 2024-09-17T18:29:27.236 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.233+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2793878416}) 2024-09-17T18:29:27.236 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.233+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.236 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.233+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2573250565}) 2024-09-17T18:29:27.236 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.233+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2484990158}) 2024-09-17T18:29:27.238 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.235+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.238 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.235+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3734725945}) 2024-09-17T18:29:27.238 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.235+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1996997625}) 2024-09-17T18:29:27.238 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.235+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.238 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.235+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3734725945}) 2024-09-17T18:29:27.238 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.235+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=253708594}) 2024-09-17T18:29:27.244 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.241+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.244 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.241+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=29} crc {osdmap=455550474}) 2024-09-17T18:29:27.244 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.241+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=32} crc {osdmap=271896870}) 2024-09-17T18:29:27.244 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.241+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:29:27.244 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.241+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=29} crc {osdmap=455550474}) 2024-09-17T18:29:27.244 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:29:27.241+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=29} crc {osdmap=2592731599}) 2024-09-17T18:29:27.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.357+0000 7f039c885640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:29:27.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.357+0000 7f039c885640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:29:27.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.357+0000 7f039c885640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:29:27.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.357+0000 7f039c885640 1 -- 172.21.15.121:0/1116708879 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f039806cd40 con 0x7f03980a5c00 2024-09-17T18:29:27.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.358+0000 7f0386ffd640 1 -- 172.21.15.121:0/1116708879 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f038c074020 con 0x7f03980a5c00 2024-09-17T18:29:27.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.358+0000 7f0386ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:29:27.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.358+0000 7f0386ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:29:27.361 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:29:27.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.360+0000 7f039c885640 10 librados: watch_flush enter 2024-09-17T18:29:27.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.360+0000 7f039c885640 10 librados: watch_flush exit 2024-09-17T18:29:27.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.360+0000 7f039c885640 20 client.44565.objecter shutdown clearing up homeless session... 2024-09-17T18:29:27.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.360+0000 7f039c885640 10 client.44565.objecter successfully canceled tick 2024-09-17T18:29:27.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.360+0000 7f039c885640 1 -- 172.21.15.121:0/1116708879 >> v2:172.21.15.173:6812/2423786037 conn(0x7f03780401c0 msgr2=0x7f0378042680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:27.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.360+0000 7f039c885640 1 --2- 172.21.15.121:0/1116708879 >> v2:172.21.15.173:6812/2423786037 conn(0x7f03780401c0 0x7f0378042680 secure :-1 s=READY pgs=4012 cs=0 l=1 rev1=1 crypto rx=0x7f0388006fd0 tx=0x7f0388008040 comp rx=0 tx=0).stop 2024-09-17T18:29:27.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.360+0000 7f039c885640 10 monclient: shutdown 2024-09-17T18:29:27.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.360+0000 7f039c885640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:29:27.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.360+0000 7f039c885640 1 -- 172.21.15.121:0/1116708879 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03980a5c00 msgr2=0x7f0398125090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:27.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.360+0000 7f039c885640 1 --2- 172.21.15.121:0/1116708879 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03980a5c00 0x7f0398125090 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f038c034040 tx=0x7f038c004a40 comp rx=0 tx=0).stop 2024-09-17T18:29:27.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.360+0000 7f039c885640 1 -- 172.21.15.121:0/1116708879 shutdown_connections 2024-09-17T18:29:27.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.360+0000 7f039c885640 1 --2- 172.21.15.121:0/1116708879 >> v2:172.21.15.173:6812/2423786037 conn(0x7f03780401c0 0x7f0378042680 unknown :-1 s=CLOSED pgs=4012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.360+0000 7f039c885640 1 --2- 172.21.15.121:0/1116708879 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03980a5fe0 0x7f03981255d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.360+0000 7f039c885640 1 --2- 172.21.15.121:0/1116708879 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03981389c0 0x7f0398125b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.360+0000 7f039c885640 1 --2- 172.21.15.121:0/1116708879 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03980a5c00 0x7f0398125090 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.360+0000 7f039c885640 1 -- 172.21.15.121:0/1116708879 >> 172.21.15.121:0/1116708879 conn(0x7f03980ab9b0 msgr2=0x7f0398138dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:29:27.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.360+0000 7f039c885640 1 -- 172.21.15.121:0/1116708879 shutdown_connections 2024-09-17T18:29:27.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.360+0000 7f039c885640 1 -- 172.21.15.121:0/1116708879 wait complete. 2024-09-17T18:29:27.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.360+0000 7f039c885640 1 librados: shutdown 2024-09-17T18:29:27.373 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T18:29:27.373 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-17T18:29:27.374 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T18:29:27.374 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T18:29:27.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.473+0000 7fcaa1744640 10 monclient: get_monmap_and_config 2024-09-17T18:29:27.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.473+0000 7fcaa1744640 10 monclient: build_initial_monmap 2024-09-17T18:29:27.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.473+0000 7fcaa1744640 10 monclient: monmap: 2024-09-17T18:29:27.476 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:29:27.476 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:27.476 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:29:27.474809+0000 2024-09-17T18:29:27.476 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:29:27.474809+0000 2024-09-17T18:29:27.476 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:29:27.476 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:27.476 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:29:27.476 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:29:27.476 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:29:27.476 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:29:27.477 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:29:27.477 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:27.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.473+0000 7fcaa1744640 1 Processor -- start 2024-09-17T18:29:27.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.473+0000 7fcaa1744640 1 -- start start 2024-09-17T18:29:27.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.473+0000 7fcaa1744640 10 monclient: init 2024-09-17T18:29:27.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.474+0000 7fcaa1744640 10 monclient: _reopen_session rank -1 2024-09-17T18:29:27.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.474+0000 7fcaa1744640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-17T18:29:27.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.474+0000 7fcaa1744640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fca9c0a5ca0 addr v1:172.21.15.173:6789/0 2024-09-17T18:29:27.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.474+0000 7fcaa1744640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fca9c0a6790 0x7fca9c13da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:27.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.474+0000 7fcaa1744640 10 monclient(hunting): picked mon.noname-c con 0x7fca9c0a6790 addr v2:172.21.15.173:3300/0 2024-09-17T18:29:27.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.474+0000 7fcaa1744640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca9c13dfd0 0x7fca9c1403c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:27.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.474+0000 7fcaa1744640 10 monclient(hunting): picked mon.noname-b con 0x7fca9c13dfd0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:29:27.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.474+0000 7fcaa1744640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca9c0a7b50 con 0x7fca9c0a5ca0 2024-09-17T18:29:27.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.474+0000 7fcaa1744640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:27.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.474+0000 7fcaa1744640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fca9c0704c0 con 0x7fca9c0a6790 2024-09-17T18:29:27.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.474+0000 7fcaa1744640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:27.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.474+0000 7fcaa1744640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fca9c0a5490 con 0x7fca9c13dfd0 2024-09-17T18:29:27.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.474+0000 7fcaa1744640 10 monclient(hunting): _renew_subs 2024-09-17T18:29:27.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.474+0000 7fcaa1744640 10 monclient(hunting): authenticate will time out at 1371.191412s 2024-09-17T18:29:27.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.474+0000 7fca9b7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca9c13dfd0 0x7fca9c1403c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:27.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.474+0000 7fca9b7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca9c13dfd0 0x7fca9c1403c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:47112/0 (socket says 172.21.15.121:47112) 2024-09-17T18:29:27.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.474+0000 7fca9b7fe640 1 -- 172.21.15.121:0/3029102565 learned_addr learned my addr 172.21.15.121:0/3029102565 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:29:27.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.474+0000 7fca9b7fe640 10 monclient(hunting): get_auth_request con 0x7fca9c13dfd0 auth_method 0 2024-09-17T18:29:27.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.474+0000 7fca9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:27.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.474+0000 7fca9b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:27.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.474+0000 7fca9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:27.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.474+0000 7fca9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:27.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.474+0000 7fca9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:27.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.474+0000 7fca9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:27.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.474+0000 7fca9a7fc640 1 --2- 172.21.15.121:0/3029102565 >> v2:172.21.15.173:3300/0 conn(0x7fca9c0a6790 0x7fca9c13da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:27.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.474+0000 7fca9a7fc640 10 monclient(hunting): get_auth_request con 0x7fca9c0a6790 auth_method 0 2024-09-17T18:29:27.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.474+0000 7fca9a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:27.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.474+0000 7fca9a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:27.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.474+0000 7fca9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:27.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.474+0000 7fca9b7fe640 10 monclient(hunting): handle_auth_done global_id 64196 payload 274 2024-09-17T18:29:27.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.474+0000 7fca9b7fe640 1 -- 172.21.15.121:0/3029102565 >> v2:172.21.15.173:3300/0 conn(0x7fca9c0a6790 msgr2=0x7fca9c13da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:29:27.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.474+0000 7fca9b7fe640 1 --2- 172.21.15.121:0/3029102565 >> v2:172.21.15.173:3300/0 conn(0x7fca9c0a6790 0x7fca9c13da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.474+0000 7fca9b7fe640 1 -- 172.21.15.121:0/3029102565 >> v1:172.21.15.173:6789/0 conn(0x7fca9c0a5ca0 legacy=0x7fca9c0a6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:29:27.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.474+0000 7fca9b7fe640 10 monclient: _finish_hunting 0 2024-09-17T18:29:27.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.474+0000 7fca9b7fe640 1 monclient: found mon.noname-b 2024-09-17T18:29:27.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.474+0000 7fca9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:29:27.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.474+0000 7fca9b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:29:27.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.474+0000 7fca9b7fe640 1 -- 172.21.15.121:0/3029102565 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca9c0a2a50 con 0x7fca9c13dfd0 2024-09-17T18:29:27.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.477+0000 7fca9b7fe640 1 -- 172.21.15.121:0/3029102565 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca9c13dfd0 msgr2=0x7fca9c1403c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:27.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.477+0000 7fca9b7fe640 1 --2- 172.21.15.121:0/3029102565 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca9c13dfd0 0x7fca9c1403c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fca94009870 tx=0x7fca9402a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b66db3975e9f389c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:27.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.477+0000 7fca99ffb640 1 -- 172.21.15.121:0/3029102565 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca94037070 con 0x7fca9c13dfd0 2024-09-17T18:29:27.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.477+0000 7fca99ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:27.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.477+0000 7fca99ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:29:27.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.477+0000 7fca99ffb640 10 monclient: dump: 2024-09-17T18:29:27.490 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:27.490 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:27.490 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.490 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.490 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:27.491 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:27.491 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:27.491 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:27.491 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:27.491 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:27.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.477+0000 7fca99ffb640 10 monclient: _finish_auth 0 2024-09-17T18:29:27.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.477+0000 7fca99ffb640 10 monclient: _check_auth_tickets 2024-09-17T18:29:27.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.477+0000 7fca99ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:29:27.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.477+0000 7fca99ffb640 1 -- 172.21.15.121:0/3029102565 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca9402ae50 con 0x7fca9c13dfd0 2024-09-17T18:29:27.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.477+0000 7fcaa1744640 5 monclient: authenticate success, global_id 64196 2024-09-17T18:29:27.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.477+0000 7fcaa1744640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:29:27.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.477+0000 7fca99ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:29:27.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.477+0000 7fcaa1744640 10 monclient: get_monmap_and_config success 2024-09-17T18:29:27.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.477+0000 7fcaa1744640 4 set_mon_vals no callback set 2024-09-17T18:29:27.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.477+0000 7fca99ffb640 1 -- 172.21.15.121:0/3029102565 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca94032860 con 0x7fca9c13dfd0 2024-09-17T18:29:27.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.477+0000 7fca99ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:27.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.477+0000 7fca99ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:29:27.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.477+0000 7fca99ffb640 10 monclient: dump: 2024-09-17T18:29:27.492 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:27.492 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:27.492 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.492 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.492 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:27.492 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:27.492 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:27.493 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:27.493 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:27.493 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:27.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.478+0000 7fcaa1744640 10 monclient: shutdown 2024-09-17T18:29:27.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.478+0000 7fcaa1744640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:29:27.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.478+0000 7fcaa1744640 1 -- 172.21.15.121:0/3029102565 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca9c13dfd0 msgr2=0x7fca9c1403c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:27.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.478+0000 7fcaa1744640 1 --2- 172.21.15.121:0/3029102565 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca9c13dfd0 0x7fca9c1403c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fca94009870 tx=0x7fca9402a130 comp rx=0 tx=0).stop 2024-09-17T18:29:27.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.478+0000 7fcaa1744640 1 -- 172.21.15.121:0/3029102565 shutdown_connections 2024-09-17T18:29:27.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.478+0000 7fcaa1744640 1 --2- 172.21.15.121:0/3029102565 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca9c13dfd0 0x7fca9c1403c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.478+0000 7fcaa1744640 1 --2- 172.21.15.121:0/3029102565 >> v2:172.21.15.173:3300/0 conn(0x7fca9c0a6790 0x7fca9c13da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.478+0000 7fcaa1744640 1 -- 172.21.15.121:0/3029102565 >> 172.21.15.121:0/3029102565 conn(0x7fca9c06dd10 msgr2=0x7fca9c0a7da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:29:27.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.478+0000 7fcaa1744640 1 -- 172.21.15.121:0/3029102565 shutdown_connections 2024-09-17T18:29:27.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.478+0000 7fcaa1744640 1 -- 172.21.15.121:0/3029102565 wait complete. 2024-09-17T18:29:27.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.478+0000 7fcaa1744640 10 monclient: build_initial_monmap 2024-09-17T18:29:27.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.478+0000 7fcaa1744640 10 monclient: monmap: 2024-09-17T18:29:27.494 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:29:27.494 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:29:27.494 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:29:27.494 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:29:27.494 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:29:27.494 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:27.494 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:29:27.494 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:29:27.494 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:29:27.494 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:27.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.479+0000 7fcaa1744640 1 librados: starting msgr at 2024-09-17T18:29:27.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.479+0000 7fcaa1744640 1 librados: starting objecter 2024-09-17T18:29:27.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.479+0000 7fcaa1744640 1 Processor -- start 2024-09-17T18:29:27.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.479+0000 7fcaa1744640 1 -- start start 2024-09-17T18:29:27.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.479+0000 7fcaa1744640 1 librados: setting wanted keys 2024-09-17T18:29:27.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.479+0000 7fcaa1744640 1 librados: calling monclient init 2024-09-17T18:29:27.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.479+0000 7fcaa1744640 10 monclient: init 2024-09-17T18:29:27.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.479+0000 7fcaa1744640 10 monclient: _reopen_session rank -1 2024-09-17T18:29:27.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.479+0000 7fcaa1744640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T18:29:27.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.479+0000 7fcaa1744640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fca9c0a5ca0 0x7fca9c0a0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:27.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.479+0000 7fcaa1744640 10 monclient(hunting): picked mon.noname-b con 0x7fca9c0a5ca0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:29:27.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.479+0000 7fcaa1744640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca9c0a6790 0x7fca9c137cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:27.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.479+0000 7fcaa1744640 10 monclient(hunting): picked mon.noname-a con 0x7fca9c0a6790 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:29:27.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.479+0000 7fcaa1744640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca9c13dfd0 0x7fca9c1381f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:27.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.479+0000 7fcaa1744640 10 monclient(hunting): picked mon.noname-c con 0x7fca9c13dfd0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:29:27.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.479+0000 7fcaa1744640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:27.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.479+0000 7fcaa1744640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fca9c0a5670 con 0x7fca9c0a6790 2024-09-17T18:29:27.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.479+0000 7fcaa1744640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:27.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.479+0000 7fcaa1744640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fca9c0a6eb0 con 0x7fca9c0a5ca0 2024-09-17T18:29:27.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.479+0000 7fcaa1744640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:27.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.479+0000 7fcaa1744640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fca9c0a8720 con 0x7fca9c13dfd0 2024-09-17T18:29:27.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.479+0000 7fcaa1744640 10 monclient(hunting): _renew_subs 2024-09-17T18:29:27.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.479+0000 7fcaa1744640 10 monclient(hunting): authenticate will time out at 1371.196817s 2024-09-17T18:29:27.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.479+0000 7fca9b7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca9c13dfd0 0x7fca9c1381f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:27.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.479+0000 7fca9a7fc640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca9c0a6790 0x7fca9c137cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:27.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.479+0000 7fca9b7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca9c13dfd0 0x7fca9c1381f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:47126/0 (socket says 172.21.15.121:47126) 2024-09-17T18:29:27.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.479+0000 7fca9b7fe640 1 -- 172.21.15.121:0/1119617910 learned_addr learned my addr 172.21.15.121:0/1119617910 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:29:27.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.479+0000 7fca9a7fc640 10 monclient(hunting): get_auth_request con 0x7fca9c0a6790 auth_method 0 2024-09-17T18:29:27.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.479+0000 7fca9a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:27.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.479+0000 7fca9a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:27.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.479+0000 7fca9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:27.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.479+0000 7fca9b7fe640 10 monclient(hunting): get_auth_request con 0x7fca9c13dfd0 auth_method 0 2024-09-17T18:29:27.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.479+0000 7fca9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:27.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.479+0000 7fca9b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:27.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.479+0000 7fca9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:27.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.480+0000 7fca9a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:27.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.480+0000 7fca9a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:27.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.480+0000 7fca9a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:27.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.480+0000 7fca9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:27.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.480+0000 7fca9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:27.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.480+0000 7fca9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:27.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.480+0000 7fca9affd640 1 --2- 172.21.15.121:0/1119617910 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fca9c0a5ca0 0x7fca9c0a0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:27.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.480+0000 7fca9affd640 10 monclient(hunting): get_auth_request con 0x7fca9c0a5ca0 auth_method 0 2024-09-17T18:29:27.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.480+0000 7fca9affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:27.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.480+0000 7fca9affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:27.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.480+0000 7fca9affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:27.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.480+0000 7fca9a7fc640 10 monclient(hunting): handle_auth_done global_id 44568 payload 824 2024-09-17T18:29:27.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.480+0000 7fca9a7fc640 1 -- 172.21.15.121:0/1119617910 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca9c13dfd0 msgr2=0x7fca9c1381f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:27.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.480+0000 7fca9a7fc640 1 --2- 172.21.15.121:0/1119617910 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca9c13dfd0 0x7fca9c1381f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.480+0000 7fca9a7fc640 1 -- 172.21.15.121:0/1119617910 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fca9c0a5ca0 msgr2=0x7fca9c0a0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:27.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.480+0000 7fca9a7fc640 1 --2- 172.21.15.121:0/1119617910 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fca9c0a5ca0 0x7fca9c0a0220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.480+0000 7fca9a7fc640 10 monclient: _finish_hunting 0 2024-09-17T18:29:27.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.480+0000 7fca9a7fc640 1 monclient: found mon.noname-a 2024-09-17T18:29:27.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.480+0000 7fca9a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:29:27.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.480+0000 7fca9a7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:29:27.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.480+0000 7fca9a7fc640 1 -- 172.21.15.121:0/1119617910 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca9c0a0cd0 con 0x7fca9c0a6790 2024-09-17T18:29:27.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.480+0000 7fca9b7fe640 1 --2- 172.21.15.121:0/1119617910 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca9c13dfd0 0x7fca9c1381f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:29:27.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.480+0000 7fca9affd640 1 --2- 172.21.15.121:0/1119617910 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fca9c0a5ca0 0x7fca9c0a0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T18:29:27.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.482+0000 7fca9a7fc640 1 -- 172.21.15.121:0/1119617910 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca9c0a6790 msgr2=0x7fca9c137cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:27.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.482+0000 7fca9a7fc640 1 --2- 172.21.15.121:0/1119617910 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca9c0a6790 0x7fca9c137cb0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fca8400d5f0 tx=0x7fca8400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:27.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.482+0000 7fca937fe640 10 client.?.objecter ms_handle_connect 0x7fca9c0a6790 2024-09-17T18:29:27.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.482+0000 7fca937fe640 10 client.?.objecter resend_mon_ops 2024-09-17T18:29:27.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.482+0000 7fca937fe640 1 -- 172.21.15.121:0/1119617910 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca84012800 con 0x7fca9c0a6790 2024-09-17T18:29:27.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.482+0000 7fca937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:27.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.482+0000 7fca937fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:29:27.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.482+0000 7fca937fe640 10 monclient: dump: 2024-09-17T18:29:27.499 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:27.499 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:27.499 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.499 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.499 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:27.499 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:27.499 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:27.499 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:27.500 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:27.500 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:27.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.482+0000 7fca937fe640 10 monclient: _finish_auth 0 2024-09-17T18:29:27.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.482+0000 7fca937fe640 10 monclient: _check_auth_tickets 2024-09-17T18:29:27.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.482+0000 7fca937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:29:27.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.482+0000 7fca937fe640 1 -- 172.21.15.121:0/1119617910 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca84019070 con 0x7fca9c0a6790 2024-09-17T18:29:27.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.482+0000 7fcaa1744640 5 monclient: authenticate success, global_id 44568 2024-09-17T18:29:27.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.482+0000 7fcaa1744640 10 monclient: _renew_subs 2024-09-17T18:29:27.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.482+0000 7fcaa1744640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:29:27.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.482+0000 7fcaa1744640 1 -- 172.21.15.121:0/1119617910 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca9c124760 con 0x7fca9c0a6790 2024-09-17T18:29:27.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.482+0000 7fca937fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:29:27.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.482+0000 7fca937fe640 1 -- 172.21.15.121:0/1119617910 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca84012be0 con 0x7fca9c0a6790 2024-09-17T18:29:27.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.482+0000 7fca937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:27.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.482+0000 7fca937fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:29:27.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.482+0000 7fca937fe640 10 monclient: dump: 2024-09-17T18:29:27.501 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:27.501 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:27.501 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.501 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.501 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:27.501 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:27.501 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:27.501 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:27.501 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:27.501 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:27.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.482+0000 7fcaa1744640 10 client.44568.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:29:27.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.482+0000 7fcaa1744640 10 monclient: _renew_subs 2024-09-17T18:29:27.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.482+0000 7fcaa1744640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:29:27.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.482+0000 7fcaa1744640 1 -- 172.21.15.121:0/1119617910 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca9c122f60 con 0x7fca9c0a6790 2024-09-17T18:29:27.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.482+0000 7fcaa1744640 1 librados: init done 2024-09-17T18:29:27.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.482+0000 7fca997fa640 4 set_mon_vals no callback set 2024-09-17T18:29:27.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.483+0000 7fcaa1744640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:29:27.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.483+0000 7fcaa1744640 1 --2- 172.21.15.121:0/1119617910 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca9c1cdd10 0x7fca9c02d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:27.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.483+0000 7fcaa1744640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:27.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.483+0000 7fcaa1744640 1 -- 172.21.15.121:0/1119617910 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fca9c122920 con 0x7fca9c1cdd10 2024-09-17T18:29:27.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.483+0000 7fca9affd640 1 --2- 172.21.15.121:0/1119617910 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca9c1cdd10 0x7fca9c02d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:27.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.483+0000 7fca9affd640 10 monclient: get_auth_request con 0x7fca9c1cdd10 auth_method 0 2024-09-17T18:29:27.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.483+0000 7fca9affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:27.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.483+0000 7fca9affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:27.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.483+0000 7fca9affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:27.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.483+0000 7fca937fe640 1 -- 172.21.15.121:0/1119617910 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fca8401e460 con 0x7fca9c0a6790 2024-09-17T18:29:27.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.483+0000 7fca9affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:27.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.483+0000 7fca9affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:27.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.483+0000 7fca9affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:27.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.483+0000 7fca9affd640 10 monclient(hunting): handle_auth_done global_id 44571 payload 824 2024-09-17T18:29:27.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.483+0000 7fca9affd640 1 -- 172.21.15.121:0/1119617910 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fca9c0a2490 con 0x7fca9c1cdd10 2024-09-17T18:29:27.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.483+0000 7fca937fe640 1 --2- 172.21.15.121:0/1119617910 >> v2:172.21.15.173:6812/2423786037 conn(0x7fca7c040170 0x7fca7c042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:27.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.483+0000 7fca937fe640 1 -- 172.21.15.121:0/1119617910 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(412..412 src has 247..412) ==== 5345+0+0 (secure 0 0 0) 0x7fca84053b70 con 0x7fca9c0a6790 2024-09-17T18:29:27.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.483+0000 7fca937fe640 10 client.44568.objecter ms_dispatch 0x7fca9c000b90 osd_map(412..412 src has 247..412) 2024-09-17T18:29:27.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.483+0000 7fca937fe640 3 client.44568.objecter handle_osd_map got epochs [412,412] > 0 2024-09-17T18:29:27.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.483+0000 7fca937fe640 3 client.44568.objecter handle_osd_map decoding full epoch 412 2024-09-17T18:29:27.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.484+0000 7fca937fe640 20 client.44568.objecter dump_active .. 0 homeless 2024-09-17T18:29:27.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.484+0000 7fca9b7fe640 1 --2- 172.21.15.121:0/1119617910 >> v2:172.21.15.173:6812/2423786037 conn(0x7fca7c040170 0x7fca7c042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:27.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.484+0000 7fca9b7fe640 10 monclient: get_auth_request con 0x7fca7c040170 auth_method 0 2024-09-17T18:29:27.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.486+0000 7fca937fe640 10 client.44568.objecter ms_handle_connect 0x7fca9c1cdd10 2024-09-17T18:29:27.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.486+0000 7fca937fe640 10 client.44568.objecter resend_mon_ops 2024-09-17T18:29:27.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.486+0000 7fca9affd640 1 -- 172.21.15.121:0/1119617910 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca9c1cdd10 msgr2=0x7fca9c02d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:27.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.486+0000 7fca9affd640 1 --2- 172.21.15.121:0/1119617910 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca9c1cdd10 0x7fca9c02d440 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fca8c00e890 tx=0x7fca8c00ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:27.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.486+0000 7fca937fe640 1 -- 172.21.15.121:0/1119617910 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca8c00cb50 con 0x7fca9c1cdd10 2024-09-17T18:29:27.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.486+0000 7fca937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:27.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.486+0000 7fca937fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:29:27.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.486+0000 7fca937fe640 10 monclient: dump: 2024-09-17T18:29:27.504 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:27.504 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:27.504 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.504 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.504 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:27.504 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:27.504 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:27.504 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:27.505 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:27.505 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:27.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.486+0000 7fca937fe640 1 -- 172.21.15.121:0/1119617910 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fca8c01a070 con 0x7fca9c1cdd10 2024-09-17T18:29:27.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.486+0000 7fca937fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:29:27.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.486+0000 7fca937fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:29:27.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.486+0000 7fca937fe640 1 -- 172.21.15.121:0/1119617910 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca9c1cdd10 msgr2=0x7fca9c02d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:27.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.486+0000 7fca937fe640 1 --2- 172.21.15.121:0/1119617910 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca9c1cdd10 0x7fca9c02d440 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fca8c00e890 tx=0x7fca8c00ed50 comp rx=0 tx=0).stop 2024-09-17T18:29:27.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.486+0000 7fca937fe640 1 -- 172.21.15.121:0/1119617910 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca9c1cdd10 msgr2=0x7fca9c02d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:29:27.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.486+0000 7fca937fe640 1 --2- 172.21.15.121:0/1119617910 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca9c1cdd10 0x7fca9c02d440 secure :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fca8c00e890 tx=0x7fca8c00ed50 comp rx=0 tx=0).stop 2024-09-17T18:29:27.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.486+0000 7fca9b7fe640 1 --2- 172.21.15.121:0/1119617910 >> v2:172.21.15.173:6812/2423786037 conn(0x7fca7c040170 0x7fca7c042630 secure :-1 s=READY pgs=4013 cs=0 l=1 rev1=1 crypto rx=0x7fca9402a170 tx=0x7fca9402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:27.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.486+0000 7fca937fe640 10 client.44568.objecter ms_handle_connect 0x7fca7c040170 2024-09-17T18:29:27.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.492+0000 7fcaa1744640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:29:27.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.492+0000 7fcaa1744640 1 --2- 172.21.15.121:0/1119617910 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca9c1ce0f0 0x7fca9c02d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:27.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.492+0000 7fcaa1744640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:27.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.492+0000 7fcaa1744640 1 -- 172.21.15.121:0/1119617910 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fca9c0a6080 con 0x7fca9c1ce0f0 2024-09-17T18:29:27.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.492+0000 7fca9affd640 1 --2- 172.21.15.121:0/1119617910 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca9c1ce0f0 0x7fca9c02d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:27.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.492+0000 7fca9affd640 10 monclient: get_auth_request con 0x7fca9c1ce0f0 auth_method 0 2024-09-17T18:29:27.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.492+0000 7fca9affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:27.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.492+0000 7fca9affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:27.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.492+0000 7fca9affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:27.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.492+0000 7fca9affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:27.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.492+0000 7fca9affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:27.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.492+0000 7fca9affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:27.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.492+0000 7fca9affd640 10 monclient(hunting): handle_auth_done global_id 44574 payload 824 2024-09-17T18:29:27.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.492+0000 7fca9affd640 1 -- 172.21.15.121:0/1119617910 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fca9c123160 con 0x7fca9c1ce0f0 2024-09-17T18:29:27.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.494+0000 7fca9affd640 1 -- 172.21.15.121:0/1119617910 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca9c1ce0f0 msgr2=0x7fca9c02d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:27.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.494+0000 7fca937fe640 10 client.44568.objecter ms_handle_connect 0x7fca9c1ce0f0 2024-09-17T18:29:27.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.494+0000 7fca937fe640 10 client.44568.objecter resend_mon_ops 2024-09-17T18:29:27.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.494+0000 7fca9affd640 1 --2- 172.21.15.121:0/1119617910 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca9c1ce0f0 0x7fca9c02d980 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fca8c019000 tx=0x7fca8c003ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:27.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.495+0000 7fca937fe640 1 -- 172.21.15.121:0/1119617910 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca8c025070 con 0x7fca9c1ce0f0 2024-09-17T18:29:27.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.495+0000 7fca937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:27.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.495+0000 7fca937fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:29:27.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.495+0000 7fca937fe640 10 monclient: dump: 2024-09-17T18:29:27.508 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:27.508 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:27.509 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.509 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.509 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:27.509 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:27.509 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:27.509 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:27.509 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:27.509 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:27.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.495+0000 7fca937fe640 1 -- 172.21.15.121:0/1119617910 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7fca8c006b50 con 0x7fca9c1ce0f0 2024-09-17T18:29:27.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.495+0000 7fca937fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:29:27.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.495+0000 7fca937fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:29:27.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.495+0000 7fca937fe640 1 -- 172.21.15.121:0/1119617910 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca9c1ce0f0 msgr2=0x7fca9c02d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:27.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.495+0000 7fca937fe640 1 --2- 172.21.15.121:0/1119617910 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca9c1ce0f0 0x7fca9c02d980 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fca8c019000 tx=0x7fca8c003ae0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.495+0000 7fca937fe640 1 -- 172.21.15.121:0/1119617910 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca9c1ce0f0 msgr2=0x7fca9c02d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:29:27.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.495+0000 7fca937fe640 1 --2- 172.21.15.121:0/1119617910 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca9c1ce0f0 0x7fca9c02d980 secure :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fca8c019000 tx=0x7fca8c003ae0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.510 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:29:27.510 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:29:27.510 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:29:27.510 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "leader", 2024-09-17T18:29:27.510 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 38, 2024-09-17T18:29:27.510 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 393160, 2024-09-17T18:29:27.510 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:29:27.510 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:29:27.510 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:29:27.510 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:29:27.510 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:29:27.510 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 2, 2024-09-17T18:29:27.511 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:29:27.511 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:29:27.511 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:29:27.511 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:29:27.511 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:29:27.511 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:29:27.511 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:29:27.511 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:29:27.511 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:29:27.511 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:29:27.511 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:29:27.511 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:29:27.511 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:29:27.511 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:29:27.511 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:29:27.512 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:29:27.512 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:29:27.512 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:29:27.512 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:29:27.512 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:29:27.512 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:29:27.512 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:29:27.512 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:29:27.512 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:29:27.512 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:29:27.512 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:29:27.512 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:29:27.512 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:29:27.512 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:27.512 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:27.513 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:29:27.513 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:29:27.513 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:29:27.513 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:29:27.513 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:29:27.513 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:29:27.513 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:29:27.513 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:29:27.513 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:29:27.513 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:29:27.513 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:29:27.513 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:29:27.513 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:29:27.513 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:29:27.513 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:29:27.513 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:29:27.514 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:29:27.514 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:29:27.514 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:29:27.514 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:29:27.514 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:29:27.514 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:29:27.514 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:29:27.514 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:29:27.514 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:29:27.514 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:29:27.514 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:29:27.514 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:29:27.514 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:29:27.514 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:29:27.514 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:27.515 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:29:27.515 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:27.515 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:29:27.515 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:29:27.515 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:29:27.515 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:29:27.515 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:27.515 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:29:27.515 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:29:27.515 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:27.515 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:27.515 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:27.515 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:29:27.516 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:29:27.516 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:27.516 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:27.516 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:27.516 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:27.516 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:29:27.516 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:29:27.516 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:29:27.516 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:29:27.516 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:29:27.516 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:27.516 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:27.516 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:29:27.517 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:29:27.517 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:29:27.517 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:29:27.517 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:27.517 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:29:27.517 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:29:27.517 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:27.517 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:27.517 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:27.517 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:29:27.517 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:29:27.517 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:27.517 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:27.517 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:27.517 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:27.518 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:29:27.518 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:29:27.518 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:29:27.518 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:29:27.518 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:29:27.518 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:27.518 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:27.518 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:29:27.518 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:29:27.518 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:29:27.518 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:29:27.518 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:27.518 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:29:27.518 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:29:27.519 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:27.519 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:27.519 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:27.519 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:29:27.519 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:29:27.519 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:27.519 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:27.519 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:27.519 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:27.519 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:29:27.519 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:29:27.519 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:29:27.519 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:29:27.519 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:29:27.520 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:27.520 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:27.520 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:27.520 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:29:27.520 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:29:27.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:27.520 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:29:27.520 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:29:27.520 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:29:27.520 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:27.520 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:29:27.520 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:29:27.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:27.520 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:29:27.520 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:29:27.521 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 4 2024-09-17T18:29:27.521 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:27.521 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:29:27.521 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:29:27.521 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:27.521 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:29:27.521 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:29:27.521 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 6 2024-09-17T18:29:27.521 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:27.521 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:27.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:27.521 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:29:27.521 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:29:27.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.495+0000 7fcaa1744640 10 librados: watch_flush enter 2024-09-17T18:29:27.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.495+0000 7fcaa1744640 10 librados: watch_flush exit 2024-09-17T18:29:27.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.495+0000 7fcaa1744640 20 client.44568.objecter shutdown clearing up homeless session... 2024-09-17T18:29:27.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.495+0000 7fcaa1744640 10 client.44568.objecter successfully canceled tick 2024-09-17T18:29:27.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.495+0000 7fcaa1744640 1 -- 172.21.15.121:0/1119617910 >> v2:172.21.15.173:6812/2423786037 conn(0x7fca7c040170 msgr2=0x7fca7c042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:27.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.495+0000 7fcaa1744640 1 --2- 172.21.15.121:0/1119617910 >> v2:172.21.15.173:6812/2423786037 conn(0x7fca7c040170 0x7fca7c042630 secure :-1 s=READY pgs=4013 cs=0 l=1 rev1=1 crypto rx=0x7fca9402a170 tx=0x7fca9402e000 comp rx=0 tx=0).stop 2024-09-17T18:29:27.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.495+0000 7fcaa1744640 10 monclient: shutdown 2024-09-17T18:29:27.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.495+0000 7fcaa1744640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:29:27.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.495+0000 7fcaa1744640 1 -- 172.21.15.121:0/1119617910 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca9c0a6790 msgr2=0x7fca9c137cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:27.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.495+0000 7fcaa1744640 1 --2- 172.21.15.121:0/1119617910 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca9c0a6790 0x7fca9c137cb0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fca8400d5f0 tx=0x7fca8400dab0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.495+0000 7fca9b7fe640 1 -- 172.21.15.121:0/1119617910 reap_dead start 2024-09-17T18:29:27.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.495+0000 7fca9b7fe640 1 -- 172.21.15.121:0/1119617910 reap_dead start 2024-09-17T18:29:27.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.495+0000 7fcaa1744640 1 -- 172.21.15.121:0/1119617910 shutdown_connections 2024-09-17T18:29:27.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.495+0000 7fcaa1744640 1 -- 172.21.15.121:0/1119617910 >> 172.21.15.121:0/1119617910 conn(0x7fca9c06dd10 msgr2=0x7fca9c138c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:29:27.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.495+0000 7fcaa1744640 1 -- 172.21.15.121:0/1119617910 shutdown_connections 2024-09-17T18:29:27.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.495+0000 7fcaa1744640 1 -- 172.21.15.121:0/1119617910 wait complete. 2024-09-17T18:29:27.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.495+0000 7fcaa1744640 1 librados: shutdown 2024-09-17T18:29:27.524 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:29:27.525 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T18:29:27.525 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T18:29:27.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.620+0000 7f3d7ec59640 10 monclient: get_monmap_and_config 2024-09-17T18:29:27.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.620+0000 7f3d7ec59640 10 monclient: build_initial_monmap 2024-09-17T18:29:27.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.620+0000 7f3d7ec59640 10 monclient: monmap: 2024-09-17T18:29:27.623 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:29:27.623 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:27.623 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:29:27.621677+0000 2024-09-17T18:29:27.623 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:29:27.621677+0000 2024-09-17T18:29:27.623 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:29:27.624 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:27.624 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:29:27.624 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:29:27.624 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:29:27.624 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:29:27.624 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:29:27.624 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:27.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.620+0000 7f3d7ec59640 1 Processor -- start 2024-09-17T18:29:27.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.620+0000 7f3d7ec59640 1 -- start start 2024-09-17T18:29:27.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.620+0000 7f3d7ec59640 10 monclient: init 2024-09-17T18:29:27.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d7ec59640 10 monclient: _reopen_session rank -1 2024-09-17T18:29:27.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d7ec59640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-17T18:29:27.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d7ec59640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d780700f0 0x7f3d78134e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:27.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d7ec59640 10 monclient(hunting): picked mon.noname-b con 0x7f3d780700f0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:29:27.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d7ec59640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f3d7809e810 0x7f3d78138360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:27.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d7ec59640 10 monclient(hunting): picked mon.noname-a con 0x7f3d7809e810 addr v2:172.21.15.121:3300/0 2024-09-17T18:29:27.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d7ec59640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3d7809ec30 addr v1:172.21.15.173:6789/0 2024-09-17T18:29:27.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d7ec59640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d7809ca60 con 0x7f3d7809ec30 2024-09-17T18:29:27.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d7ec59640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:27.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d7ec59640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f3d780704d0 con 0x7f3d7809e810 2024-09-17T18:29:27.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d7ec59640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:27.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d7ec59640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3d7809cd20 con 0x7f3d780700f0 2024-09-17T18:29:27.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d7ec59640 10 monclient(hunting): _renew_subs 2024-09-17T18:29:27.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d7ec59640 10 monclient(hunting): authenticate will time out at 1371.338431s 2024-09-17T18:29:27.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d7c9f5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d780700f0 0x7f3d78134e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:27.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d77fff640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f3d7809e810 0x7f3d78138360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:27.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d77fff640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f3d7809e810 0x7f3d78138360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58142/0 (socket says 172.21.15.121:58142) 2024-09-17T18:29:27.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d7c9f5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d780700f0 0x7f3d78134e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:47132/0 (socket says 172.21.15.121:47132) 2024-09-17T18:29:27.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d77fff640 1 -- 172.21.15.121:0/252719141 learned_addr learned my addr 172.21.15.121:0/252719141 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:29:27.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d7c9f5640 10 monclient(hunting): get_auth_request con 0x7f3d780700f0 auth_method 0 2024-09-17T18:29:27.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d7c9f5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:27.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d7c9f5640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:27.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d7c9f5640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:27.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d77fff640 10 monclient(hunting): get_auth_request con 0x7f3d7809e810 auth_method 0 2024-09-17T18:29:27.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:27.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d77fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:27.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:27.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:27.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:27.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:27.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d7c9f5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:27.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d7c9f5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:27.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d7c9f5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:27.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d77fff640 10 monclient(hunting): handle_auth_done global_id 44577 payload 274 2024-09-17T18:29:27.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d77fff640 1 -- 172.21.15.121:0/252719141 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d780700f0 msgr2=0x7f3d78134e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:29:27.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d77fff640 1 --2- 172.21.15.121:0/252719141 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d780700f0 0x7f3d78134e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d77fff640 1 -- 172.21.15.121:0/252719141 >> v1:172.21.15.173:6789/0 conn(0x7f3d7809ec30 legacy=0x7f3d781388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:29:27.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d77fff640 10 monclient: _finish_hunting 0 2024-09-17T18:29:27.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d77fff640 1 monclient: found mon.noname-a 2024-09-17T18:29:27.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d77fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:29:27.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d77fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:29:27.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d77fff640 1 -- 172.21.15.121:0/252719141 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d7809d000 con 0x7f3d7809e810 2024-09-17T18:29:27.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.621+0000 7f3d7c9f5640 1 --2- 172.21.15.121:0/252719141 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d780700f0 0x7f3d78134e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:29:27.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.624+0000 7f3d77fff640 1 -- 172.21.15.121:0/252719141 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d7809e810 msgr2=0x7f3d78138360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:27.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.624+0000 7f3d77fff640 1 --2- 172.21.15.121:0/252719141 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d7809e810 0x7f3d78138360 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3d64009870 tx=0x7f3d6402a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d178134e7e9a278b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:27.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.624+0000 7f3d777fe640 1 -- 172.21.15.121:0/252719141 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d64037070 con 0x7f3d7809e810 2024-09-17T18:29:27.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.624+0000 7f3d777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:27.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.624+0000 7f3d777fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:29:27.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.624+0000 7f3d777fe640 10 monclient: dump: 2024-09-17T18:29:27.636 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:27.636 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:27.637 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.637 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.637 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:27.637 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:27.637 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:27.637 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:27.637 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:27.637 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:27.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.624+0000 7f3d777fe640 10 monclient: _finish_auth 0 2024-09-17T18:29:27.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.624+0000 7f3d777fe640 10 monclient: _check_auth_tickets 2024-09-17T18:29:27.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.624+0000 7f3d777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:29:27.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.624+0000 7f3d777fe640 1 -- 172.21.15.121:0/252719141 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d64032640 con 0x7f3d7809e810 2024-09-17T18:29:27.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.624+0000 7f3d7ec59640 5 monclient: authenticate success, global_id 44577 2024-09-17T18:29:27.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.624+0000 7f3d7ec59640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:29:27.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.624+0000 7f3d777fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:29:27.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.624+0000 7f3d7ec59640 10 monclient: get_monmap_and_config success 2024-09-17T18:29:27.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.624+0000 7f3d777fe640 1 -- 172.21.15.121:0/252719141 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d64032a10 con 0x7f3d7809e810 2024-09-17T18:29:27.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.624+0000 7f3d777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:27.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.624+0000 7f3d7ec59640 4 set_mon_vals no callback set 2024-09-17T18:29:27.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.624+0000 7f3d777fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:29:27.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.624+0000 7f3d777fe640 10 monclient: dump: 2024-09-17T18:29:27.638 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:27.638 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:27.638 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.638 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.638 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:27.638 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:27.638 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:27.639 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:27.639 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:27.639 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:27.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.625+0000 7f3d7ec59640 10 monclient: shutdown 2024-09-17T18:29:27.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.625+0000 7f3d7ec59640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:29:27.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.625+0000 7f3d7ec59640 1 -- 172.21.15.121:0/252719141 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d7809e810 msgr2=0x7f3d78138360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:27.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.625+0000 7f3d7ec59640 1 --2- 172.21.15.121:0/252719141 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d7809e810 0x7f3d78138360 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3d64009870 tx=0x7f3d6402a130 comp rx=0 tx=0).stop 2024-09-17T18:29:27.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.625+0000 7f3d7ec59640 1 -- 172.21.15.121:0/252719141 shutdown_connections 2024-09-17T18:29:27.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.625+0000 7f3d7ec59640 1 --2- 172.21.15.121:0/252719141 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d7809e810 0x7f3d78138360 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.625+0000 7f3d7ec59640 1 --2- 172.21.15.121:0/252719141 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d780700f0 0x7f3d78134e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.625+0000 7f3d7ec59640 1 -- 172.21.15.121:0/252719141 >> 172.21.15.121:0/252719141 conn(0x7f3d780ab830 msgr2=0x7f3d780a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:29:27.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.625+0000 7f3d7ec59640 1 -- 172.21.15.121:0/252719141 shutdown_connections 2024-09-17T18:29:27.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.625+0000 7f3d7ec59640 1 -- 172.21.15.121:0/252719141 wait complete. 2024-09-17T18:29:27.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.626+0000 7f3d7ec59640 10 monclient: build_initial_monmap 2024-09-17T18:29:27.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.626+0000 7f3d7ec59640 10 monclient: monmap: 2024-09-17T18:29:27.639 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:29:27.640 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:29:27.640 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:29:27.640 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:29:27.640 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:29:27.640 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:27.640 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:29:27.640 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:29:27.640 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:29:27.640 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:27.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.626+0000 7f3d7ec59640 1 librados: starting msgr at 2024-09-17T18:29:27.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.626+0000 7f3d7ec59640 1 librados: starting objecter 2024-09-17T18:29:27.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.626+0000 7f3d7ec59640 1 Processor -- start 2024-09-17T18:29:27.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.626+0000 7f3d7ec59640 1 -- start start 2024-09-17T18:29:27.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.626+0000 7f3d7ec59640 1 librados: setting wanted keys 2024-09-17T18:29:27.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.626+0000 7f3d7ec59640 1 librados: calling monclient init 2024-09-17T18:29:27.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.626+0000 7f3d7ec59640 10 monclient: init 2024-09-17T18:29:27.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.626+0000 7f3d7ec59640 10 monclient: _reopen_session rank -1 2024-09-17T18:29:27.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.626+0000 7f3d7ec59640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T18:29:27.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.626+0000 7f3d7ec59640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d780700f0 0x7f3d78131bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:27.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.626+0000 7f3d7ec59640 10 monclient(hunting): picked mon.noname-c con 0x7f3d780700f0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:29:27.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.626+0000 7f3d7ec59640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d7809e810 0x7f3d78132100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:27.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.626+0000 7f3d7ec59640 10 monclient(hunting): picked mon.noname-b con 0x7f3d7809e810 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:29:27.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d7ec59640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d7809ec30 0x7f3d780a00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:27.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d7ec59640 10 monclient(hunting): picked mon.noname-a con 0x7f3d7809ec30 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:29:27.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d7ec59640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:27.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d7ec59640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3d78135550 con 0x7f3d7809ec30 2024-09-17T18:29:27.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d7ec59640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:27.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d7ec59640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3d781353d0 con 0x7f3d7809e810 2024-09-17T18:29:27.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d7ec59640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:27.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d7ec59640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3d78138fb0 con 0x7f3d780700f0 2024-09-17T18:29:27.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d7ec59640 10 monclient(hunting): _renew_subs 2024-09-17T18:29:27.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d7ec59640 10 monclient(hunting): authenticate will time out at 1371.344030s 2024-09-17T18:29:27.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d7c9f5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d780700f0 0x7f3d78131bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:27.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d7c9f5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d780700f0 0x7f3d78131bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:47140/0 (socket says 172.21.15.121:47140) 2024-09-17T18:29:27.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d7c9f5640 1 -- 172.21.15.121:0/3063439328 learned_addr learned my addr 172.21.15.121:0/3063439328 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:29:27.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d7c9f5640 10 monclient(hunting): get_auth_request con 0x7f3d780700f0 auth_method 0 2024-09-17T18:29:27.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d7d1f6640 1 --2- 172.21.15.121:0/3063439328 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d7809ec30 0x7f3d780a00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:27.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d7c9f5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:27.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d7c9f5640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:27.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d7c9f5640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:27.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d7d1f6640 10 monclient(hunting): get_auth_request con 0x7f3d7809ec30 auth_method 0 2024-09-17T18:29:27.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d7d1f6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:27.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d7d1f6640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:27.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d7d1f6640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:27.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d7c9f5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:27.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d7c9f5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:27.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d7c9f5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:27.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d7d1f6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:27.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d7d1f6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:27.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d7d1f6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:27.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d77fff640 1 --2- 172.21.15.121:0/3063439328 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d7809e810 0x7f3d78132100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:27.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d77fff640 10 monclient(hunting): get_auth_request con 0x7f3d7809e810 auth_method 0 2024-09-17T18:29:27.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:27.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d77fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:27.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:27.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:27.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:27.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:27.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d7d1f6640 10 monclient(hunting): handle_auth_done global_id 44580 payload 824 2024-09-17T18:29:27.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d7d1f6640 1 -- 172.21.15.121:0/3063439328 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d780700f0 msgr2=0x7f3d78131bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:27.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d7d1f6640 1 --2- 172.21.15.121:0/3063439328 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d780700f0 0x7f3d78131bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d7d1f6640 1 -- 172.21.15.121:0/3063439328 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d7809e810 msgr2=0x7f3d78132100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:27.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d7d1f6640 1 --2- 172.21.15.121:0/3063439328 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d7809e810 0x7f3d78132100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d7d1f6640 10 monclient: _finish_hunting 0 2024-09-17T18:29:27.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d7d1f6640 1 monclient: found mon.noname-a 2024-09-17T18:29:27.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d7d1f6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:29:27.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d7d1f6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:29:27.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d7d1f6640 1 -- 172.21.15.121:0/3063439328 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d780a1bb0 con 0x7f3d7809ec30 2024-09-17T18:29:27.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.627+0000 7f3d7c9f5640 1 --2- 172.21.15.121:0/3063439328 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d780700f0 0x7f3d78131bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:29:27.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.629+0000 7f3d7d1f6640 1 -- 172.21.15.121:0/3063439328 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d7809ec30 msgr2=0x7f3d780a00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:27.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.629+0000 7f3d7d1f6640 1 --2- 172.21.15.121:0/3063439328 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d7809ec30 0x7f3d780a00f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3d7000c780 tx=0x7f3d7000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:27.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.629+0000 7f3d757fa640 10 client.?.objecter ms_handle_connect 0x7f3d7809ec30 2024-09-17T18:29:27.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.629+0000 7f3d757fa640 10 client.?.objecter resend_mon_ops 2024-09-17T18:29:27.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.629+0000 7f3d757fa640 1 -- 172.21.15.121:0/3063439328 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d70016020 con 0x7f3d7809ec30 2024-09-17T18:29:27.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.629+0000 7f3d757fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:27.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.629+0000 7f3d757fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:29:27.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.629+0000 7f3d757fa640 10 monclient: dump: 2024-09-17T18:29:27.645 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:27.645 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:27.645 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.645 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.645 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:27.645 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:27.645 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:27.645 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:27.645 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:27.645 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:27.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.629+0000 7f3d757fa640 10 monclient: _finish_auth 0 2024-09-17T18:29:27.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.629+0000 7f3d757fa640 10 monclient: _check_auth_tickets 2024-09-17T18:29:27.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.629+0000 7f3d757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:29:27.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.629+0000 7f3d7ec59640 5 monclient: authenticate success, global_id 44580 2024-09-17T18:29:27.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.629+0000 7f3d757fa640 1 -- 172.21.15.121:0/3063439328 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d7000ac60 con 0x7f3d7809ec30 2024-09-17T18:29:27.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.629+0000 7f3d757fa640 10 monclient: handle_config config(0 keys) 2024-09-17T18:29:27.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.629+0000 7f3d757fa640 1 -- 172.21.15.121:0/3063439328 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d70012660 con 0x7f3d7809ec30 2024-09-17T18:29:27.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.629+0000 7f3d7ec59640 10 monclient: _renew_subs 2024-09-17T18:29:27.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.629+0000 7f3d7ec59640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:29:27.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.629+0000 7f3d7ec59640 1 -- 172.21.15.121:0/3063439328 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d78134c40 con 0x7f3d7809ec30 2024-09-17T18:29:27.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.629+0000 7f3d757fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:27.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.629+0000 7f3d757fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:29:27.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.629+0000 7f3d757fa640 10 monclient: dump: 2024-09-17T18:29:27.646 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:27.646 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:27.646 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.646 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.647 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:27.647 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:27.647 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:27.647 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:27.647 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:27.647 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:27.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.629+0000 7f3d76ffd640 4 set_mon_vals no callback set 2024-09-17T18:29:27.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.629+0000 7f3d7ec59640 10 client.44580.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:29:27.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.629+0000 7f3d7ec59640 10 monclient: _renew_subs 2024-09-17T18:29:27.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.629+0000 7f3d7ec59640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:29:27.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.629+0000 7f3d7ec59640 1 -- 172.21.15.121:0/3063439328 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d7813add0 con 0x7f3d7809ec30 2024-09-17T18:29:27.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.629+0000 7f3d7ec59640 1 librados: init done 2024-09-17T18:29:27.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.630+0000 7f3d7d1f6640 1 --2- 172.21.15.121:0/3063439328 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d7809ec30 0x7f3d780a00f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3d7000c780 tx=0x7f3d7000cc40 comp rx=0 tx=0).queue_received will delay after 1.7266e+09 on 0x7f3d70053650 osd_map(412..412 src has 247..412) 2024-09-17T18:29:27.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.630+0000 7f3d757fa640 1 -- 172.21.15.121:0/3063439328 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f3d70019020 con 0x7f3d7809ec30 2024-09-17T18:29:27.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.630+0000 7f3d7ec59640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:29:27.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.630+0000 7f3d7ec59640 1 --2- 172.21.15.121:0/3063439328 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d780a0630 0x7f3d780a0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:27.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.630+0000 7f3d7ec59640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:27.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.630+0000 7f3d7ec59640 1 -- 172.21.15.121:0/3063439328 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3d78134e50 con 0x7f3d780a0630 2024-09-17T18:29:27.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.630+0000 7f3d7c9f5640 1 --2- 172.21.15.121:0/3063439328 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d780a0630 0x7f3d780a0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:27.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.630+0000 7f3d7c9f5640 10 monclient: get_auth_request con 0x7f3d780a0630 auth_method 0 2024-09-17T18:29:27.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.630+0000 7f3d7c9f5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:27.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.630+0000 7f3d7c9f5640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:27.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.630+0000 7f3d7c9f5640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:27.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.630+0000 7f3d7c9f5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:27.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.630+0000 7f3d7c9f5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:27.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.630+0000 7f3d7c9f5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:27.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.630+0000 7f3d757fa640 1 --2- 172.21.15.121:0/3063439328 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3d5c040170 0x7f3d5c042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:27.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.631+0000 7f3d7c9f5640 10 monclient(hunting): handle_auth_done global_id 64208 payload 824 2024-09-17T18:29:27.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.631+0000 7f3d7c9f5640 1 -- 172.21.15.121:0/3063439328 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3d780008d0 con 0x7f3d780a0630 2024-09-17T18:29:27.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.631+0000 7f3d77fff640 1 --2- 172.21.15.121:0/3063439328 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3d5c040170 0x7f3d5c042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:27.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.631+0000 7f3d77fff640 10 monclient: get_auth_request con 0x7f3d5c040170 auth_method 0 2024-09-17T18:29:27.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.633+0000 7f3d757fa640 10 client.44580.objecter ms_handle_connect 0x7f3d780a0630 2024-09-17T18:29:27.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.633+0000 7f3d757fa640 10 client.44580.objecter resend_mon_ops 2024-09-17T18:29:27.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.633+0000 7f3d7c9f5640 1 -- 172.21.15.121:0/3063439328 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d780a0630 msgr2=0x7f3d780a0a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:27.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.633+0000 7f3d7c9f5640 1 --2- 172.21.15.121:0/3063439328 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d780a0630 0x7f3d780a0a10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c009ed0 tx=0x7f3d6c00f610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:27.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.633+0000 7f3d757fa640 1 -- 172.21.15.121:0/3063439328 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d6c014860 con 0x7f3d780a0630 2024-09-17T18:29:27.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.633+0000 7f3d757fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:27.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.633+0000 7f3d757fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:29:27.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.633+0000 7f3d757fa640 10 monclient: dump: 2024-09-17T18:29:27.649 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:27.649 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:27.649 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.650 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.650 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:27.650 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:27.650 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:27.650 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:27.650 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:27.650 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:27.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.633+0000 7f3d757fa640 1 -- 172.21.15.121:0/3063439328 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3d6c014860 con 0x7f3d780a0630 2024-09-17T18:29:27.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.633+0000 7f3d757fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:29:27.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.633+0000 7f3d757fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:29:27.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.633+0000 7f3d757fa640 1 -- 172.21.15.121:0/3063439328 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d780a0630 msgr2=0x7f3d780a0a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:27.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.633+0000 7f3d757fa640 1 --2- 172.21.15.121:0/3063439328 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d780a0630 0x7f3d780a0a10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c009ed0 tx=0x7f3d6c00f610 comp rx=0 tx=0).stop 2024-09-17T18:29:27.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.633+0000 7f3d757fa640 1 -- 172.21.15.121:0/3063439328 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d780a0630 msgr2=0x7f3d780a0a10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:29:27.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.633+0000 7f3d757fa640 1 --2- 172.21.15.121:0/3063439328 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d780a0630 0x7f3d780a0a10 secure :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c009ed0 tx=0x7f3d6c00f610 comp rx=0 tx=0).stop 2024-09-17T18:29:27.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.633+0000 7f3d77fff640 1 --2- 172.21.15.121:0/3063439328 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3d5c040170 0x7f3d5c042630 secure :-1 s=READY pgs=4014 cs=0 l=1 rev1=1 crypto rx=0x7f3d780a5870 tx=0x7f3d6402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:27.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.633+0000 7f3d757fa640 10 client.44580.objecter ms_handle_connect 0x7f3d5c040170 2024-09-17T18:29:27.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.640+0000 7f3d7ec59640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:29:27.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.640+0000 7f3d7ec59640 1 --2- 172.21.15.121:0/3063439328 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d780a0f50 0x7f3d780a2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:27.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.640+0000 7f3d7ec59640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:27.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.640+0000 7f3d7ec59640 1 -- 172.21.15.121:0/3063439328 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3d781398b0 con 0x7f3d780a0f50 2024-09-17T18:29:27.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.640+0000 7f3d7c9f5640 1 --2- 172.21.15.121:0/3063439328 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d780a0f50 0x7f3d780a2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:27.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.640+0000 7f3d7c9f5640 10 monclient: get_auth_request con 0x7f3d780a0f50 auth_method 0 2024-09-17T18:29:27.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.640+0000 7f3d7c9f5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:27.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.640+0000 7f3d7c9f5640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:27.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.640+0000 7f3d7c9f5640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:27.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.640+0000 7f3d7c9f5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:27.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.640+0000 7f3d7c9f5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:27.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.640+0000 7f3d7c9f5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:27.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.640+0000 7f3d7c9f5640 10 monclient(hunting): handle_auth_done global_id 64211 payload 824 2024-09-17T18:29:27.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.640+0000 7f3d7c9f5640 1 -- 172.21.15.121:0/3063439328 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3d78132640 con 0x7f3d780a0f50 2024-09-17T18:29:27.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.643+0000 7f3d7c9f5640 1 -- 172.21.15.121:0/3063439328 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d780a0f50 msgr2=0x7f3d780a2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:27.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.643+0000 7f3d7c9f5640 1 --2- 172.21.15.121:0/3063439328 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d780a0f50 0x7f3d780a2600 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c009ed0 tx=0x7f3d6c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:27.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.643+0000 7f3d757fa640 10 client.44580.objecter ms_handle_connect 0x7f3d780a0f50 2024-09-17T18:29:27.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.643+0000 7f3d757fa640 10 client.44580.objecter resend_mon_ops 2024-09-17T18:29:27.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.643+0000 7f3d757fa640 1 -- 172.21.15.121:0/3063439328 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d6c004220 con 0x7f3d780a0f50 2024-09-17T18:29:27.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.643+0000 7f3d757fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:27.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.643+0000 7f3d757fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:29:27.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.643+0000 7f3d757fa640 10 monclient: dump: 2024-09-17T18:29:27.654 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:27.654 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:27.654 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.654 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.654 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:27.654 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:27.654 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:27.654 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:27.654 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:27.654 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:27.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.643+0000 7f3d757fa640 1 -- 172.21.15.121:0/3063439328 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f3d6c004220 con 0x7f3d780a0f50 2024-09-17T18:29:27.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.643+0000 7f3d757fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:29:27.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.643+0000 7f3d757fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:29:27.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.643+0000 7f3d757fa640 1 -- 172.21.15.121:0/3063439328 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d780a0f50 msgr2=0x7f3d780a2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:27.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.643+0000 7f3d757fa640 1 --2- 172.21.15.121:0/3063439328 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d780a0f50 0x7f3d780a2600 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c009ed0 tx=0x7f3d6c0046c0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.643+0000 7f3d757fa640 1 -- 172.21.15.121:0/3063439328 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d780a0f50 msgr2=0x7f3d780a2600 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:29:27.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.643+0000 7f3d757fa640 1 --2- 172.21.15.121:0/3063439328 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d780a0f50 0x7f3d780a2600 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c009ed0 tx=0x7f3d6c0046c0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.655 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:29:27.655 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:29:27.655 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:29:27.655 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:29:27.655 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 38, 2024-09-17T18:29:27.655 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 146376, 2024-09-17T18:29:27.656 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:29:27.656 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:29:27.656 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:29:27.656 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:29:27.656 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:29:27.656 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 2, 2024-09-17T18:29:27.656 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:29:27.656 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:29:27.656 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:29:27.656 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:29:27.656 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:29:27.656 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:29:27.656 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:29:27.656 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:29:27.656 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:29:27.657 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:29:27.657 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:29:27.657 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:29:27.657 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:29:27.657 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:29:27.657 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:29:27.657 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:29:27.657 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:29:27.657 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:29:27.657 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:29:27.657 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:29:27.657 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:29:27.657 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:29:27.657 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:29:27.657 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:29:27.658 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:29:27.658 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:29:27.658 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:29:27.658 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:29:27.658 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:27.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:27.658 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:29:27.658 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:29:27.658 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:29:27.658 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:29:27.658 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:29:27.658 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:29:27.658 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:29:27.658 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:29:27.658 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:29:27.659 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:29:27.659 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:29:27.659 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:29:27.659 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:29:27.659 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:29:27.659 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:29:27.659 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:29:27.659 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:29:27.659 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:29:27.659 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:29:27.659 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:29:27.659 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:29:27.659 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:29:27.659 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:29:27.660 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:29:27.660 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:29:27.660 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:29:27.660 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:29:27.660 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:29:27.660 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:29:27.660 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:29:27.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:27.660 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:29:27.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:27.660 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:29:27.660 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:29:27.660 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:29:27.660 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:29:27.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:27.660 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:29:27.661 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:29:27.661 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:27.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:27.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:27.661 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:29:27.661 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:29:27.661 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:27.661 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:27.661 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:27.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:27.661 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:29:27.661 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:29:27.661 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:29:27.661 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:29:27.662 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:29:27.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:27.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:27.662 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:29:27.662 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:29:27.662 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:29:27.662 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:29:27.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:27.662 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:29:27.662 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:29:27.662 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:27.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:27.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:27.662 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:29:27.662 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:29:27.662 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:27.663 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:27.663 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:27.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:27.663 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:29:27.663 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:29:27.663 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:29:27.663 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:29:27.663 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:29:27.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:27.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:27.663 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:29:27.663 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:29:27.663 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:29:27.663 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:29:27.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:27.663 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:29:27.664 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:29:27.664 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:27.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:27.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:27.664 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:29:27.664 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:29:27.664 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:27.664 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:27.664 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:27.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:27.664 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:29:27.664 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:29:27.664 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:29:27.664 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:29:27.664 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:29:27.665 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:27.665 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:27.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:27.665 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:29:27.665 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:29:27.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:27.665 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:29:27.665 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:29:27.665 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:29:27.665 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:27.665 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:29:27.665 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:29:27.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:27.665 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:29:27.666 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:29:27.666 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-17T18:29:27.666 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:27.666 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:29:27.666 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:29:27.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:27.666 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:29:27.666 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:29:27.666 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-17T18:29:27.668 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:27.668 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:29:27.668 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": [ 2024-09-17T18:29:27.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:27.668 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:29:27.668 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:29:27.668 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:29:27.668 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:27.668 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:27.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:27.669 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:29:27.669 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:29:27.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.643+0000 7f3d7ec59640 10 librados: watch_flush enter 2024-09-17T18:29:27.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.643+0000 7f3d7ec59640 10 librados: watch_flush exit 2024-09-17T18:29:27.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.643+0000 7f3d7ec59640 20 client.44580.objecter shutdown clearing up homeless session... 2024-09-17T18:29:27.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.643+0000 7f3d7ec59640 10 client.44580.objecter successfully canceled tick 2024-09-17T18:29:27.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.643+0000 7f3d7ec59640 1 -- 172.21.15.121:0/3063439328 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3d5c040170 msgr2=0x7f3d5c042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:27.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.643+0000 7f3d7ec59640 1 --2- 172.21.15.121:0/3063439328 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3d5c040170 0x7f3d5c042630 secure :-1 s=READY pgs=4014 cs=0 l=1 rev1=1 crypto rx=0x7f3d780a5870 tx=0x7f3d6402e000 comp rx=0 tx=0).stop 2024-09-17T18:29:27.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.643+0000 7f3d7ec59640 10 monclient: shutdown 2024-09-17T18:29:27.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.643+0000 7f3d7ec59640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:29:27.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.643+0000 7f3d7ec59640 1 -- 172.21.15.121:0/3063439328 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d7809ec30 msgr2=0x7f3d780a00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:27.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.643+0000 7f3d7d1f6640 1 -- 172.21.15.121:0/3063439328 reap_dead start 2024-09-17T18:29:27.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.643+0000 7f3d7ec59640 1 --2- 172.21.15.121:0/3063439328 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d7809ec30 0x7f3d780a00f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3d7000c780 tx=0x7f3d7000cc40 comp rx=0 tx=0).stop 2024-09-17T18:29:27.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.643+0000 7f3d757fa640 1 -- 172.21.15.121:0/3063439328 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(412..412 src has 247..412) ==== 5345+0+0 (unknown 0 0 0) 0x7f3d70053650 con 0x7f3d7809ec30 2024-09-17T18:29:27.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.643+0000 7f3d757fa640 10 client.44580.objecter ms_dispatch 0x7f3d78000b90 osd_map(412..412 src has 247..412) 2024-09-17T18:29:27.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.643+0000 7f3d757fa640 10 librados: disconnected, discarding osd_map(412..412 src has 247..412) 2024-09-17T18:29:27.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.643+0000 7f3d7ec59640 1 -- 172.21.15.121:0/3063439328 shutdown_connections 2024-09-17T18:29:27.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.643+0000 7f3d7ec59640 1 --2- 172.21.15.121:0/3063439328 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d7809ec30 0x7f3d780a00f0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.643+0000 7f3d7ec59640 1 -- 172.21.15.121:0/3063439328 >> 172.21.15.121:0/3063439328 conn(0x7f3d780ab830 msgr2=0x7f3d78132e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:29:27.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.643+0000 7f3d7ec59640 1 -- 172.21.15.121:0/3063439328 shutdown_connections 2024-09-17T18:29:27.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.643+0000 7f3d7ec59640 1 -- 172.21.15.121:0/3063439328 wait complete. 2024-09-17T18:29:27.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.643+0000 7f3d7ec59640 1 librados: shutdown 2024-09-17T18:29:27.672 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:29:27.672 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T18:29:27.673 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T18:29:27.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.769+0000 7f69807e2640 10 monclient: get_monmap_and_config 2024-09-17T18:29:27.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.769+0000 7f69807e2640 10 monclient: build_initial_monmap 2024-09-17T18:29:27.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.769+0000 7f69807e2640 10 monclient: monmap: 2024-09-17T18:29:27.771 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:29:27.771 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:27.772 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:29:27.770444+0000 2024-09-17T18:29:27.772 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:29:27.770444+0000 2024-09-17T18:29:27.772 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:29:27.772 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:27.772 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:29:27.772 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:29:27.772 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:29:27.772 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:29:27.772 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:29:27.772 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:27.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.769+0000 7f69807e2640 1 Processor -- start 2024-09-17T18:29:27.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.769+0000 7f69807e2640 1 -- start start 2024-09-17T18:29:27.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.769+0000 7f69807e2640 10 monclient: init 2024-09-17T18:29:27.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.770+0000 7f69807e2640 10 monclient: _reopen_session rank -1 2024-09-17T18:29:27.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.770+0000 7f69807e2640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-09-17T18:29:27.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.770+0000 7f69807e2640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f697809da70 0x7f69781361c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:27.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.770+0000 7f69807e2640 10 monclient(hunting): picked mon.noname-c con 0x7f697809da70 addr v2:172.21.15.173:3300/0 2024-09-17T18:29:27.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.770+0000 7f69807e2640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6978136700 0x7f6978136b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:27.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.770+0000 7f69807e2640 10 monclient(hunting): picked mon.noname-b con 0x7f6978136700 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:29:27.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.770+0000 7f69807e2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6978132cc0 addr v1:172.21.15.121:6789/0 2024-09-17T18:29:27.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.770+0000 7f69807e2640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69780a2d40 con 0x7f6978132cc0 2024-09-17T18:29:27.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.770+0000 7f69807e2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:27.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.770+0000 7f69807e2640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f697809de50 con 0x7f697809da70 2024-09-17T18:29:27.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.770+0000 7f69807e2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:27.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.770+0000 7f69807e2640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f69780a0e20 con 0x7f6978136700 2024-09-17T18:29:27.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.770+0000 7f69807e2640 10 monclient(hunting): _renew_subs 2024-09-17T18:29:27.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.770+0000 7f69807e2640 10 monclient(hunting): authenticate will time out at 1371.487195s 2024-09-17T18:29:27.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.770+0000 7f697dd7d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6978136700 0x7f6978136b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:27.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.770+0000 7f697dd7d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6978136700 0x7f6978136b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:47150/0 (socket says 172.21.15.121:47150) 2024-09-17T18:29:27.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.770+0000 7f697ed7f640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f6978132cc0 0x7f6978133150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:43806/0 (socket says 172.21.15.121:43806) 2024-09-17T18:29:27.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.770+0000 7f697dd7d640 1 -- 172.21.15.121:0/2798307501 learned_addr learned my addr 172.21.15.121:0/2798307501 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:29:27.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.770+0000 7f697dd7d640 10 monclient(hunting): get_auth_request con 0x7f6978136700 auth_method 0 2024-09-17T18:29:27.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.770+0000 7f697dd7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:27.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.770+0000 7f697dd7d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:27.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.770+0000 7f697dd7d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:27.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.770+0000 7f697dd7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:27.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.770+0000 7f697dd7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:27.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.770+0000 7f697dd7d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:27.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.770+0000 7f697e57e640 1 --2- 172.21.15.121:0/2798307501 >> v2:172.21.15.173:3300/0 conn(0x7f697809da70 0x7f69781361c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:27.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.770+0000 7f697e57e640 10 monclient(hunting): get_auth_request con 0x7f697809da70 auth_method 0 2024-09-17T18:29:27.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.770+0000 7f697e57e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:27.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.770+0000 7f697e57e640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:27.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.770+0000 7f697e57e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:27.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.770+0000 7f697dd7d640 10 monclient(hunting): handle_auth_done global_id 64214 payload 274 2024-09-17T18:29:27.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.770+0000 7f697dd7d640 1 -- 172.21.15.121:0/2798307501 >> v2:172.21.15.173:3300/0 conn(0x7f697809da70 msgr2=0x7f69781361c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:29:27.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.770+0000 7f697dd7d640 1 --2- 172.21.15.121:0/2798307501 >> v2:172.21.15.173:3300/0 conn(0x7f697809da70 0x7f69781361c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.770+0000 7f697dd7d640 1 -- 172.21.15.121:0/2798307501 >> v1:172.21.15.121:6789/0 conn(0x7f6978132cc0 legacy=0x7f6978133150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:29:27.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.770+0000 7f697dd7d640 10 monclient: _finish_hunting 0 2024-09-17T18:29:27.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.770+0000 7f697dd7d640 1 monclient: found mon.noname-b 2024-09-17T18:29:27.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.770+0000 7f697dd7d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:29:27.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.770+0000 7f697dd7d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:29:27.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.770+0000 7f697dd7d640 1 -- 172.21.15.121:0/2798307501 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69780a3020 con 0x7f6978136700 2024-09-17T18:29:27.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.773+0000 7f697dd7d640 1 -- 172.21.15.121:0/2798307501 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6978136700 msgr2=0x7f6978136b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:27.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.773+0000 7f697dd7d640 1 --2- 172.21.15.121:0/2798307501 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6978136700 0x7f6978136b30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6968009870 tx=0x7f696802a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2ddd055d5bd5196 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:27.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.773+0000 7f697d57c640 1 -- 172.21.15.121:0/2798307501 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6968037070 con 0x7f6978136700 2024-09-17T18:29:27.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.773+0000 7f697d57c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:27.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.773+0000 7f697d57c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:29:27.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.773+0000 7f697d57c640 10 monclient: dump: 2024-09-17T18:29:27.785 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:27.785 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:27.785 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.785 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.785 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:27.785 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:27.785 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:27.785 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:27.786 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:27.786 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:27.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.773+0000 7f697d57c640 10 monclient: _finish_auth 0 2024-09-17T18:29:27.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.773+0000 7f697d57c640 10 monclient: _check_auth_tickets 2024-09-17T18:29:27.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.773+0000 7f697d57c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:29:27.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.773+0000 7f69807e2640 5 monclient: authenticate success, global_id 64214 2024-09-17T18:29:27.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.773+0000 7f69807e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:29:27.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.773+0000 7f697d57c640 1 -- 172.21.15.121:0/2798307501 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6968032440 con 0x7f6978136700 2024-09-17T18:29:27.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.773+0000 7f697d57c640 10 monclient: handle_config config(0 keys) 2024-09-17T18:29:27.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.773+0000 7f69807e2640 10 monclient: get_monmap_and_config success 2024-09-17T18:29:27.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.773+0000 7f69807e2640 4 set_mon_vals no callback set 2024-09-17T18:29:27.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.773+0000 7f697d57c640 1 -- 172.21.15.121:0/2798307501 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6968032810 con 0x7f6978136700 2024-09-17T18:29:27.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.773+0000 7f697d57c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:27.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.773+0000 7f697d57c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:29:27.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.773+0000 7f697d57c640 10 monclient: dump: 2024-09-17T18:29:27.787 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:27.787 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:27.787 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.787 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.787 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:27.787 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:27.787 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:27.787 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:27.787 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:27.787 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:27.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.773+0000 7f69807e2640 10 monclient: shutdown 2024-09-17T18:29:27.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.773+0000 7f69807e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:29:27.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.773+0000 7f69807e2640 1 -- 172.21.15.121:0/2798307501 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6978136700 msgr2=0x7f6978136b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:27.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.773+0000 7f69807e2640 1 --2- 172.21.15.121:0/2798307501 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6978136700 0x7f6978136b30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6968009870 tx=0x7f696802a0a0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.774+0000 7f69807e2640 1 -- 172.21.15.121:0/2798307501 shutdown_connections 2024-09-17T18:29:27.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.774+0000 7f69807e2640 1 --2- 172.21.15.121:0/2798307501 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6978136700 0x7f6978136b30 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.774+0000 7f69807e2640 1 --2- 172.21.15.121:0/2798307501 >> v2:172.21.15.173:3300/0 conn(0x7f697809da70 0x7f69781361c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.774+0000 7f69807e2640 1 -- 172.21.15.121:0/2798307501 >> 172.21.15.121:0/2798307501 conn(0x7f69780a7830 msgr2=0x7f69780a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:29:27.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.774+0000 7f69807e2640 1 -- 172.21.15.121:0/2798307501 shutdown_connections 2024-09-17T18:29:27.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.774+0000 7f69807e2640 1 -- 172.21.15.121:0/2798307501 wait complete. 2024-09-17T18:29:27.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.774+0000 7f69807e2640 10 monclient: build_initial_monmap 2024-09-17T18:29:27.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.774+0000 7f69807e2640 10 monclient: monmap: 2024-09-17T18:29:27.788 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:29:27.788 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:29:27.788 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:29:27.788 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:29:27.789 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:29:27.789 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:27.789 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:29:27.789 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:29:27.789 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:29:27.789 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:27.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.774+0000 7f69807e2640 1 librados: starting msgr at 2024-09-17T18:29:27.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.774+0000 7f69807e2640 1 librados: starting objecter 2024-09-17T18:29:27.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.774+0000 7f69807e2640 1 Processor -- start 2024-09-17T18:29:27.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f69807e2640 1 -- start start 2024-09-17T18:29:27.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f69807e2640 1 librados: setting wanted keys 2024-09-17T18:29:27.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f69807e2640 1 librados: calling monclient init 2024-09-17T18:29:27.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f69807e2640 10 monclient: init 2024-09-17T18:29:27.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f69807e2640 10 monclient: _reopen_session rank -1 2024-09-17T18:29:27.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f69807e2640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T18:29:27.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f69807e2640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f697809da70 0x7f69781cb670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:27.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f69807e2640 10 monclient(hunting): picked mon.noname-a con 0x7f697809da70 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:29:27.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f69807e2640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6978132cc0 0x7f697813d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:27.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f69807e2640 10 monclient(hunting): picked mon.noname-c con 0x7f6978132cc0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:29:27.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f69807e2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6978136700 0x7f697813dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:27.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f69807e2640 10 monclient(hunting): picked mon.noname-b con 0x7f6978136700 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:29:27.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f69807e2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:27.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f69807e2640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6978133860 con 0x7f697809da70 2024-09-17T18:29:27.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f69807e2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:27.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f69807e2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f69780a3930 con 0x7f6978136700 2024-09-17T18:29:27.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f69807e2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:27.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f69807e2640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f69781339e0 con 0x7f6978132cc0 2024-09-17T18:29:27.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f69807e2640 10 monclient(hunting): _renew_subs 2024-09-17T18:29:27.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f69807e2640 10 monclient(hunting): authenticate will time out at 1371.492464s 2024-09-17T18:29:27.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f697dd7d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6978132cc0 0x7f697813d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:27.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f697e57e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f697809da70 0x7f69781cb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:27.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f697dd7d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6978132cc0 0x7f697813d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:47154/0 (socket says 172.21.15.121:47154) 2024-09-17T18:29:27.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f697e57e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f697809da70 0x7f69781cb670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58156/0 (socket says 172.21.15.121:58156) 2024-09-17T18:29:27.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f697dd7d640 1 -- 172.21.15.121:0/3397801715 learned_addr learned my addr 172.21.15.121:0/3397801715 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:29:27.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f697dd7d640 10 monclient(hunting): get_auth_request con 0x7f6978132cc0 auth_method 0 2024-09-17T18:29:27.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f697dd7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:27.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f697dd7d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:27.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f697dd7d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:27.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f697e57e640 10 monclient(hunting): get_auth_request con 0x7f697809da70 auth_method 0 2024-09-17T18:29:27.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f697e57e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:27.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f697e57e640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:27.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f697e57e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:27.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f697dd7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:27.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f697dd7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:27.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f697dd7d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:27.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f697e57e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:27.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f697e57e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:27.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f697e57e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:27.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f697ed7f640 1 --2- 172.21.15.121:0/3397801715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6978136700 0x7f697813dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:27.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f697ed7f640 10 monclient(hunting): get_auth_request con 0x7f6978136700 auth_method 0 2024-09-17T18:29:27.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f697ed7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:27.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f697ed7f640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:27.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f697ed7f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:27.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f697dd7d640 10 monclient(hunting): handle_auth_done global_id 64217 payload 824 2024-09-17T18:29:27.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f697dd7d640 1 -- 172.21.15.121:0/3397801715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6978136700 msgr2=0x7f697813dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:27.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f697dd7d640 1 --2- 172.21.15.121:0/3397801715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6978136700 0x7f697813dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f697dd7d640 1 -- 172.21.15.121:0/3397801715 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f697809da70 msgr2=0x7f69781cb670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:27.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f697dd7d640 1 --2- 172.21.15.121:0/3397801715 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f697809da70 0x7f69781cb670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f697dd7d640 10 monclient: _finish_hunting 0 2024-09-17T18:29:27.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f697dd7d640 1 monclient: found mon.noname-c 2024-09-17T18:29:27.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f697dd7d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:29:27.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f697dd7d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:29:27.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f697dd7d640 1 -- 172.21.15.121:0/3397801715 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6978137e60 con 0x7f6978132cc0 2024-09-17T18:29:27.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.775+0000 7f697e57e640 1 --2- 172.21.15.121:0/3397801715 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f697809da70 0x7f69781cb670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:29:27.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.778+0000 7f697dd7d640 1 -- 172.21.15.121:0/3397801715 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6978132cc0 msgr2=0x7f697813d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:27.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.778+0000 7f697dd7d640 1 --2- 172.21.15.121:0/3397801715 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6978132cc0 0x7f697813d6b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6968000c00 tx=0x7f69680046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:27.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.778+0000 7f696effd640 10 client.?.objecter ms_handle_connect 0x7f6978132cc0 2024-09-17T18:29:27.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.778+0000 7f696effd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:29:27.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.778+0000 7f696effd640 1 -- 172.21.15.121:0/3397801715 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6968041030 con 0x7f6978132cc0 2024-09-17T18:29:27.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.778+0000 7f696effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:27.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.778+0000 7f696effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:29:27.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.778+0000 7f696effd640 10 monclient: dump: 2024-09-17T18:29:27.794 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:27.794 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:27.794 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.794 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.794 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:27.794 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:27.794 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:27.794 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:27.794 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:27.794 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:27.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.778+0000 7f696effd640 10 monclient: _finish_auth 0 2024-09-17T18:29:27.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.778+0000 7f696effd640 10 monclient: _check_auth_tickets 2024-09-17T18:29:27.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.778+0000 7f696effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:29:27.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.778+0000 7f696effd640 1 -- 172.21.15.121:0/3397801715 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f696803d630 con 0x7f6978132cc0 2024-09-17T18:29:27.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.778+0000 7f69807e2640 5 monclient: authenticate success, global_id 64217 2024-09-17T18:29:27.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.778+0000 7f69807e2640 10 monclient: _renew_subs 2024-09-17T18:29:27.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.778+0000 7f69807e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:29:27.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.778+0000 7f69807e2640 1 -- 172.21.15.121:0/3397801715 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6978138140 con 0x7f6978132cc0 2024-09-17T18:29:27.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.778+0000 7f696effd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:29:27.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.778+0000 7f696effd640 1 -- 172.21.15.121:0/3397801715 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6968037040 con 0x7f6978132cc0 2024-09-17T18:29:27.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.778+0000 7f696effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:27.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.778+0000 7f696effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:29:27.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.778+0000 7f696effd640 10 monclient: dump: 2024-09-17T18:29:27.795 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:27.795 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:27.795 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.795 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.796 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:27.796 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:27.796 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:27.796 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:27.796 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:27.796 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:27.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.778+0000 7f69807e2640 10 client.64217.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:29:27.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.778+0000 7f697cd7b640 4 set_mon_vals no callback set 2024-09-17T18:29:27.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.778+0000 7f69807e2640 10 monclient: _renew_subs 2024-09-17T18:29:27.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.778+0000 7f69807e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:29:27.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.778+0000 7f69807e2640 1 -- 172.21.15.121:0/3397801715 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f697813e130 con 0x7f6978132cc0 2024-09-17T18:29:27.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.778+0000 7f69807e2640 1 librados: init done 2024-09-17T18:29:27.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.778+0000 7f69807e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:29:27.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.778+0000 7f69807e2640 1 --2- 172.21.15.121:0/3397801715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6978071850 0x7f6978071c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:27.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.778+0000 7f69807e2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:27.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.778+0000 7f69807e2640 1 -- 172.21.15.121:0/3397801715 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f69781361c0 con 0x7f6978071850 2024-09-17T18:29:27.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.779+0000 7f697e57e640 1 --2- 172.21.15.121:0/3397801715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6978071850 0x7f6978071c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:27.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.779+0000 7f697e57e640 10 monclient: get_auth_request con 0x7f6978071850 auth_method 0 2024-09-17T18:29:27.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.779+0000 7f697e57e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:27.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.779+0000 7f697e57e640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:27.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.779+0000 7f697e57e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:27.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.779+0000 7f696effd640 1 -- 172.21.15.121:0/3397801715 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f6968044070 con 0x7f6978132cc0 2024-09-17T18:29:27.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.779+0000 7f697e57e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:27.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.779+0000 7f697e57e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:27.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.779+0000 7f697e57e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:27.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.779+0000 7f697e57e640 10 monclient(hunting): handle_auth_done global_id 74143 payload 824 2024-09-17T18:29:27.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.779+0000 7f696effd640 1 --2- 172.21.15.121:0/3397801715 >> v2:172.21.15.173:6812/2423786037 conn(0x7f695c040170 0x7f695c042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:27.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.779+0000 7f697e57e640 1 -- 172.21.15.121:0/3397801715 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f69780731a0 con 0x7f6978071850 2024-09-17T18:29:27.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.779+0000 7f696effd640 1 -- 172.21.15.121:0/3397801715 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(412..412 src has 247..412) ==== 5345+0+0 (secure 0 0 0) 0x7f6968073660 con 0x7f6978132cc0 2024-09-17T18:29:27.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.779+0000 7f696effd640 10 client.64217.objecter ms_dispatch 0x7f6978000b90 osd_map(412..412 src has 247..412) 2024-09-17T18:29:27.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.779+0000 7f696effd640 3 client.64217.objecter handle_osd_map got epochs [412,412] > 0 2024-09-17T18:29:27.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.779+0000 7f696effd640 3 client.64217.objecter handle_osd_map decoding full epoch 412 2024-09-17T18:29:27.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.779+0000 7f696effd640 20 client.64217.objecter dump_active .. 0 homeless 2024-09-17T18:29:27.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.779+0000 7f697ed7f640 1 --2- 172.21.15.121:0/3397801715 >> v2:172.21.15.173:6812/2423786037 conn(0x7f695c040170 0x7f695c042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:27.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.779+0000 7f697ed7f640 10 monclient: get_auth_request con 0x7f695c040170 auth_method 0 2024-09-17T18:29:27.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.781+0000 7f696effd640 10 client.64217.objecter ms_handle_connect 0x7f6978071850 2024-09-17T18:29:27.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.781+0000 7f697e57e640 1 -- 172.21.15.121:0/3397801715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6978071850 msgr2=0x7f6978071c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:27.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.781+0000 7f696effd640 10 client.64217.objecter resend_mon_ops 2024-09-17T18:29:27.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.781+0000 7f697e57e640 1 --2- 172.21.15.121:0/3397801715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6978071850 0x7f6978071c30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6974010b90 tx=0x7f697400e5a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:27.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.781+0000 7f696effd640 1 -- 172.21.15.121:0/3397801715 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6974018070 con 0x7f6978071850 2024-09-17T18:29:27.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.781+0000 7f696effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:27.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.781+0000 7f696effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:29:27.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.781+0000 7f696effd640 10 monclient: dump: 2024-09-17T18:29:27.799 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:27.799 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:27.799 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.799 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.799 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:27.799 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:27.799 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:27.799 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:27.799 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:27.799 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:27.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.782+0000 7f696effd640 10 client.64217.objecter ms_handle_connect 0x7f695c040170 2024-09-17T18:29:27.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.782+0000 7f697ed7f640 1 --2- 172.21.15.121:0/3397801715 >> v2:172.21.15.173:6812/2423786037 conn(0x7f695c040170 0x7f695c042630 secure :-1 s=READY pgs=4015 cs=0 l=1 rev1=1 crypto rx=0x7f69781cc750 tx=0x7f6970009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:27.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.782+0000 7f696effd640 1 -- 172.21.15.121:0/3397801715 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6974010e90 con 0x7f6978071850 2024-09-17T18:29:27.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.782+0000 7f696effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:29:27.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.782+0000 7f696effd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:29:27.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.782+0000 7f696effd640 1 -- 172.21.15.121:0/3397801715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6978071850 msgr2=0x7f6978071c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:27.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.782+0000 7f696effd640 1 --2- 172.21.15.121:0/3397801715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6978071850 0x7f6978071c30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6974010b90 tx=0x7f697400e5a0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.782+0000 7f696effd640 1 -- 172.21.15.121:0/3397801715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6978071850 msgr2=0x7f6978071c30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:29:27.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.782+0000 7f696effd640 1 --2- 172.21.15.121:0/3397801715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6978071850 0x7f6978071c30 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6974010b90 tx=0x7f697400e5a0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.788+0000 7f69807e2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:29:27.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.788+0000 7f69807e2640 1 --2- 172.21.15.121:0/3397801715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6978072170 0x7f697812d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:27.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.788+0000 7f69807e2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:27.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.788+0000 7f69807e2640 1 -- 172.21.15.121:0/3397801715 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f69780742e0 con 0x7f6978072170 2024-09-17T18:29:27.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.788+0000 7f697e57e640 1 --2- 172.21.15.121:0/3397801715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6978072170 0x7f697812d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:27.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.788+0000 7f697e57e640 10 monclient: get_auth_request con 0x7f6978072170 auth_method 0 2024-09-17T18:29:27.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.788+0000 7f697e57e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:27.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.788+0000 7f697e57e640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:27.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.788+0000 7f697e57e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:27.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.788+0000 7f697e57e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:27.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.788+0000 7f697e57e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:27.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.788+0000 7f697e57e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:27.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.788+0000 7f697e57e640 10 monclient(hunting): handle_auth_done global_id 74146 payload 824 2024-09-17T18:29:27.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.788+0000 7f697e57e640 1 -- 172.21.15.121:0/3397801715 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6978137980 con 0x7f6978072170 2024-09-17T18:29:27.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.792+0000 7f697e57e640 1 -- 172.21.15.121:0/3397801715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6978072170 msgr2=0x7f697812d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:27.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.792+0000 7f697e57e640 1 --2- 172.21.15.121:0/3397801715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6978072170 0x7f697812d9b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6974010b90 tx=0x7f69740046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:27.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.792+0000 7f696effd640 10 client.64217.objecter ms_handle_connect 0x7f6978072170 2024-09-17T18:29:27.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.792+0000 7f696effd640 10 client.64217.objecter resend_mon_ops 2024-09-17T18:29:27.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.792+0000 7f696effd640 1 -- 172.21.15.121:0/3397801715 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6974004230 con 0x7f6978072170 2024-09-17T18:29:27.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.792+0000 7f696effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:27.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.792+0000 7f696effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:29:27.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.792+0000 7f696effd640 10 monclient: dump: 2024-09-17T18:29:27.803 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:27.803 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:27.803 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.803 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.803 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:27.803 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:27.803 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:27.803 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:27.803 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:27.803 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:27.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.792+0000 7f696effd640 1 -- 172.21.15.121:0/3397801715 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7f6974004230 con 0x7f6978072170 2024-09-17T18:29:27.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.792+0000 7f696effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:29:27.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.792+0000 7f696effd640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:29:27.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.792+0000 7f696effd640 1 -- 172.21.15.121:0/3397801715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6978072170 msgr2=0x7f697812d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:27.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.792+0000 7f696effd640 1 --2- 172.21.15.121:0/3397801715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6978072170 0x7f697812d9b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6974010b90 tx=0x7f69740046c0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.792+0000 7f696effd640 1 -- 172.21.15.121:0/3397801715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6978072170 msgr2=0x7f697812d9b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:29:27.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.792+0000 7f696effd640 1 --2- 172.21.15.121:0/3397801715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6978072170 0x7f697812d9b0 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6974010b90 tx=0x7f69740046c0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.804 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:29:27.804 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:29:27.804 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:29:27.805 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:29:27.805 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 38, 2024-09-17T18:29:27.805 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 17912, 2024-09-17T18:29:27.805 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:29:27.805 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:29:27.805 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:29:27.805 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:29:27.805 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:29:27.805 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 2, 2024-09-17T18:29:27.805 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:29:27.805 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:29:27.805 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:29:27.805 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:29:27.805 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:29:27.806 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:29:27.806 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:29:27.806 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:29:27.806 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:29:27.806 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:29:27.806 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:29:27.806 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:29:27.806 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:29:27.806 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:29:27.806 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:29:27.806 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:29:27.806 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:29:27.806 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:29:27.806 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:29:27.807 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:29:27.807 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:29:27.807 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:29:27.807 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:29:27.807 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:29:27.807 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:29:27.807 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:29:27.807 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:29:27.807 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:29:27.807 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:27.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:27.807 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:29:27.807 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:29:27.807 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:29:27.807 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:29:27.807 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:29:27.808 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:29:27.808 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:29:27.808 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:29:27.808 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:29:27.808 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:29:27.808 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:29:27.808 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:29:27.808 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:29:27.808 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:29:27.808 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:29:27.808 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:29:27.808 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:29:27.808 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:29:27.808 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:29:27.808 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:29:27.809 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:29:27.809 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:29:27.809 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:29:27.809 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:29:27.809 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:29:27.809 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:29:27.809 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:29:27.809 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:29:27.809 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:29:27.809 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:29:27.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:27.809 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:29:27.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:27.809 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:29:27.810 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:29:27.810 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:29:27.810 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:29:27.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:27.810 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:29:27.810 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:29:27.810 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:27.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:27.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:27.810 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:29:27.810 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:29:27.810 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:27.810 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:27.810 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:27.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:27.810 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:29:27.811 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:29:27.811 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:29:27.811 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:29:27.811 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:29:27.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:27.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:27.811 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:29:27.811 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:29:27.811 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:29:27.811 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:29:27.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:27.811 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:29:27.811 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:29:27.811 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:27.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:27.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:27.812 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:29:27.812 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:29:27.812 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:27.812 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:27.812 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:27.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:27.812 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:29:27.812 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:29:27.812 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:29:27.812 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:29:27.812 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:29:27.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:27.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:27.813 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:29:27.813 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:29:27.813 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:29:27.813 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:29:27.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:27.813 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:29:27.813 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:29:27.813 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:27.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:27.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:27.813 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:29:27.813 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:29:27.813 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:27.813 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:27.813 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:27.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:27.814 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:29:27.814 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:29:27.814 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:29:27.814 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:29:27.814 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:29:27.814 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:27.814 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:27.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:27.814 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:29:27.814 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:29:27.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:27.815 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:29:27.815 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:29:27.815 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:29:27.815 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:27.815 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:29:27.815 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:29:27.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:27.815 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:29:27.815 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:29:27.815 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:29:27.815 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:27.815 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:27.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:27.816 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:29:27.816 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:29:27.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.792+0000 7f69807e2640 10 librados: watch_flush enter 2024-09-17T18:29:27.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.792+0000 7f69807e2640 10 librados: watch_flush exit 2024-09-17T18:29:27.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.792+0000 7f69807e2640 20 client.64217.objecter shutdown clearing up homeless session... 2024-09-17T18:29:27.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.792+0000 7f69807e2640 10 client.64217.objecter successfully canceled tick 2024-09-17T18:29:27.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.792+0000 7f69807e2640 1 -- 172.21.15.121:0/3397801715 >> v2:172.21.15.173:6812/2423786037 conn(0x7f695c040170 msgr2=0x7f695c042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:27.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.792+0000 7f69807e2640 1 --2- 172.21.15.121:0/3397801715 >> v2:172.21.15.173:6812/2423786037 conn(0x7f695c040170 0x7f695c042630 secure :-1 s=READY pgs=4015 cs=0 l=1 rev1=1 crypto rx=0x7f69781cc750 tx=0x7f6970009040 comp rx=0 tx=0).stop 2024-09-17T18:29:27.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.792+0000 7f69807e2640 10 monclient: shutdown 2024-09-17T18:29:27.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.792+0000 7f69807e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:29:27.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.792+0000 7f69807e2640 1 -- 172.21.15.121:0/3397801715 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6978132cc0 msgr2=0x7f697813d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:27.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.792+0000 7f69807e2640 1 --2- 172.21.15.121:0/3397801715 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6978132cc0 0x7f697813d6b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6968000c00 tx=0x7f69680046c0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.792+0000 7f697ed7f640 1 -- 172.21.15.121:0/3397801715 reap_dead start 2024-09-17T18:29:27.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.792+0000 7f697ed7f640 1 -- 172.21.15.121:0/3397801715 reap_dead start 2024-09-17T18:29:27.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.792+0000 7f69807e2640 1 -- 172.21.15.121:0/3397801715 shutdown_connections 2024-09-17T18:29:27.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.792+0000 7f69807e2640 1 -- 172.21.15.121:0/3397801715 >> 172.21.15.121:0/3397801715 conn(0x7f69780a7830 msgr2=0x7f69780a5610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:29:27.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.792+0000 7f69807e2640 1 -- 172.21.15.121:0/3397801715 shutdown_connections 2024-09-17T18:29:27.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.792+0000 7f69807e2640 1 -- 172.21.15.121:0/3397801715 wait complete. 2024-09-17T18:29:27.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.792+0000 7f69807e2640 1 librados: shutdown 2024-09-17T18:29:27.819 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:29:27.819 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-17T18:29:27.819 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-17T18:29:27.819 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-17T18:29:27.819 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b store 2024-09-17T18:29:27.819 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b sync_force --yes-i-really-mean-it 2024-09-17T18:29:27.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.915+0000 7f2c4ee33640 10 monclient: get_monmap_and_config 2024-09-17T18:29:27.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.915+0000 7f2c4ee33640 10 monclient: build_initial_monmap 2024-09-17T18:29:27.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.915+0000 7f2c4ee33640 10 monclient: monmap: 2024-09-17T18:29:27.918 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:29:27.918 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:27.918 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:29:27.916655+0000 2024-09-17T18:29:27.918 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:29:27.916655+0000 2024-09-17T18:29:27.918 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:29:27.918 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:27.918 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:29:27.918 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:29:27.918 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:29:27.918 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:29:27.918 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:29:27.918 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:27.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.915+0000 7f2c4ee33640 1 Processor -- start 2024-09-17T18:29:27.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.915+0000 7f2c4ee33640 1 -- start start 2024-09-17T18:29:27.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.915+0000 7f2c4ee33640 10 monclient: init 2024-09-17T18:29:27.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.916+0000 7f2c4ee33640 10 monclient: _reopen_session rank -1 2024-09-17T18:29:27.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.916+0000 7f2c4ee33640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-17T18:29:27.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.916+0000 7f2c4ee33640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f2c480a5c00 0x7f2c4809d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:27.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.916+0000 7f2c4ee33640 10 monclient(hunting): picked mon.noname-c con 0x7f2c480a5c00 addr v2:172.21.15.173:3300/0 2024-09-17T18:29:27.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.916+0000 7f2c4ee33640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2c480a5fe0 addr v1:172.21.15.173:6789/0 2024-09-17T18:29:27.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.916+0000 7f2c4ee33640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c4813dc70 0x7f2c48140060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:27.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.916+0000 7f2c4ee33640 10 monclient(hunting): picked mon.noname-b con 0x7f2c4813dc70 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:29:27.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.916+0000 7f2c4ee33640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c4809d510 con 0x7f2c480a5fe0 2024-09-17T18:29:27.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.916+0000 7f2c4ee33640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:27.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.916+0000 7f2c4ee33640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f2c4809cfe0 con 0x7f2c480a5c00 2024-09-17T18:29:27.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.916+0000 7f2c4ee33640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:27.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.916+0000 7f2c4ee33640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2c480a7480 con 0x7f2c4813dc70 2024-09-17T18:29:27.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.916+0000 7f2c4ee33640 10 monclient(hunting): _renew_subs 2024-09-17T18:29:27.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.916+0000 7f2c4ee33640 10 monclient(hunting): authenticate will time out at 1371.633315s 2024-09-17T18:29:27.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.916+0000 7f2c4d3d0640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c4813dc70 0x7f2c48140060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:27.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.916+0000 7f2c4d3d0640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c4813dc70 0x7f2c48140060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:47166/0 (socket says 172.21.15.121:47166) 2024-09-17T18:29:27.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.916+0000 7f2c4d3d0640 1 -- 172.21.15.121:0/1429282349 learned_addr learned my addr 172.21.15.121:0/1429282349 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:29:27.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.916+0000 7f2c4d3d0640 10 monclient(hunting): get_auth_request con 0x7f2c4813dc70 auth_method 0 2024-09-17T18:29:27.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.916+0000 7f2c4d3d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:27.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.916+0000 7f2c4d3d0640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:27.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.916+0000 7f2c4d3d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:27.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.916+0000 7f2c4cbcf640 1 --2- 172.21.15.121:0/1429282349 >> v2:172.21.15.173:3300/0 conn(0x7f2c480a5c00 0x7f2c4809d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:27.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.916+0000 7f2c4cbcf640 10 monclient(hunting): get_auth_request con 0x7f2c480a5c00 auth_method 0 2024-09-17T18:29:27.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.916+0000 7f2c4cbcf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:27.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.916+0000 7f2c4cbcf640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:27.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.916+0000 7f2c4cbcf640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:27.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.916+0000 7f2c4d3d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:27.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.916+0000 7f2c4d3d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:27.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.916+0000 7f2c4d3d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:27.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.916+0000 7f2c4d3d0640 10 monclient(hunting): handle_auth_done global_id 64220 payload 274 2024-09-17T18:29:27.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.916+0000 7f2c4d3d0640 1 -- 172.21.15.121:0/1429282349 >> v2:172.21.15.173:3300/0 conn(0x7f2c480a5c00 msgr2=0x7f2c4809d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:29:27.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.916+0000 7f2c4d3d0640 1 --2- 172.21.15.121:0/1429282349 >> v2:172.21.15.173:3300/0 conn(0x7f2c480a5c00 0x7f2c4809d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.916+0000 7f2c4d3d0640 1 -- 172.21.15.121:0/1429282349 >> v1:172.21.15.173:6789/0 conn(0x7f2c480a5fe0 legacy=0x7f2c480a63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:29:27.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.916+0000 7f2c4d3d0640 10 monclient: _finish_hunting 0 2024-09-17T18:29:27.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.916+0000 7f2c4d3d0640 1 monclient: found mon.noname-b 2024-09-17T18:29:27.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.916+0000 7f2c4d3d0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:29:27.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.916+0000 7f2c4d3d0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:29:27.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.916+0000 7f2c4d3d0640 1 -- 172.21.15.121:0/1429282349 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c480a8ae0 con 0x7f2c4813dc70 2024-09-17T18:29:27.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.919+0000 7f2c4d3d0640 1 -- 172.21.15.121:0/1429282349 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c4813dc70 msgr2=0x7f2c48140060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:27.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.919+0000 7f2c4d3d0640 1 --2- 172.21.15.121:0/1429282349 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c4813dc70 0x7f2c48140060 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2c40009870 tx=0x7f2c4002a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a5f2b7872c1e918 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:27.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.919+0000 7f2c3f7fe640 1 -- 172.21.15.121:0/1429282349 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c40037070 con 0x7f2c4813dc70 2024-09-17T18:29:27.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.919+0000 7f2c3f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:27.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.919+0000 7f2c3f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:29:27.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.919+0000 7f2c3f7fe640 10 monclient: dump: 2024-09-17T18:29:27.930 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:27.930 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:27.931 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.931 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.931 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:27.931 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:27.931 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:27.931 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:27.931 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:27.931 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:27.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.919+0000 7f2c3f7fe640 10 monclient: _finish_auth 0 2024-09-17T18:29:27.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.919+0000 7f2c3f7fe640 10 monclient: _check_auth_tickets 2024-09-17T18:29:27.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.919+0000 7f2c3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:29:27.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.919+0000 7f2c3f7fe640 1 -- 172.21.15.121:0/1429282349 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c40032440 con 0x7f2c4813dc70 2024-09-17T18:29:27.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.919+0000 7f2c4ee33640 5 monclient: authenticate success, global_id 64220 2024-09-17T18:29:27.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.919+0000 7f2c4ee33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:29:27.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.919+0000 7f2c3f7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:29:27.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.919+0000 7f2c4ee33640 10 monclient: get_monmap_and_config success 2024-09-17T18:29:27.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.919+0000 7f2c4ee33640 4 set_mon_vals no callback set 2024-09-17T18:29:27.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.919+0000 7f2c3f7fe640 1 -- 172.21.15.121:0/1429282349 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c40032810 con 0x7f2c4813dc70 2024-09-17T18:29:27.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.919+0000 7f2c3f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:27.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.919+0000 7f2c3f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:29:27.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.919+0000 7f2c3f7fe640 10 monclient: dump: 2024-09-17T18:29:27.932 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:27.932 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:27.932 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.932 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.932 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:27.932 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:27.932 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:27.933 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:27.933 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:27.933 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:27.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.920+0000 7f2c4ee33640 10 monclient: shutdown 2024-09-17T18:29:27.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.920+0000 7f2c4ee33640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:29:27.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.920+0000 7f2c4ee33640 1 -- 172.21.15.121:0/1429282349 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c4813dc70 msgr2=0x7f2c48140060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:27.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.920+0000 7f2c4ee33640 1 --2- 172.21.15.121:0/1429282349 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c4813dc70 0x7f2c48140060 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2c40009870 tx=0x7f2c4002a0a0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.920+0000 7f2c4ee33640 1 -- 172.21.15.121:0/1429282349 shutdown_connections 2024-09-17T18:29:27.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.920+0000 7f2c4ee33640 1 --2- 172.21.15.121:0/1429282349 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c4813dc70 0x7f2c48140060 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.920+0000 7f2c4ee33640 1 --2- 172.21.15.121:0/1429282349 >> v2:172.21.15.173:3300/0 conn(0x7f2c480a5c00 0x7f2c4809d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.920+0000 7f2c4ee33640 1 -- 172.21.15.121:0/1429282349 >> 172.21.15.121:0/1429282349 conn(0x7f2c480ab9e0 msgr2=0x7f2c480a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:29:27.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.920+0000 7f2c4ee33640 1 -- 172.21.15.121:0/1429282349 shutdown_connections 2024-09-17T18:29:27.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.920+0000 7f2c4ee33640 1 -- 172.21.15.121:0/1429282349 wait complete. 2024-09-17T18:29:27.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.920+0000 7f2c4ee33640 10 monclient: build_initial_monmap 2024-09-17T18:29:27.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.920+0000 7f2c4ee33640 10 monclient: monmap: 2024-09-17T18:29:27.933 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:29:27.934 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:29:27.934 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:29:27.934 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:29:27.934 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:29:27.934 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:27.934 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:29:27.934 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:29:27.934 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:29:27.934 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:27.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.921+0000 7f2c4ee33640 1 librados: starting msgr at 2024-09-17T18:29:27.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.921+0000 7f2c4ee33640 1 librados: starting objecter 2024-09-17T18:29:27.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.921+0000 7f2c4ee33640 1 Processor -- start 2024-09-17T18:29:27.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.921+0000 7f2c4ee33640 1 -- start start 2024-09-17T18:29:27.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.921+0000 7f2c4ee33640 1 librados: setting wanted keys 2024-09-17T18:29:27.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.921+0000 7f2c4ee33640 1 librados: calling monclient init 2024-09-17T18:29:27.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.921+0000 7f2c4ee33640 10 monclient: init 2024-09-17T18:29:27.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.921+0000 7f2c4ee33640 10 monclient: _reopen_session rank -1 2024-09-17T18:29:27.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.921+0000 7f2c4ee33640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T18:29:27.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.921+0000 7f2c4ee33640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c480a5c00 0x7f2c48138850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:27.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.921+0000 7f2c4ee33640 10 monclient(hunting): picked mon.noname-c con 0x7f2c480a5c00 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:29:27.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.921+0000 7f2c4ee33640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c480a5fe0 0x7f2c4813b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:27.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.921+0000 7f2c4ee33640 10 monclient(hunting): picked mon.noname-b con 0x7f2c480a5fe0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:29:27.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.921+0000 7f2c4ee33640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c4813dc70 0x7f2c4813bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:27.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.921+0000 7f2c4ee33640 10 monclient(hunting): picked mon.noname-a con 0x7f2c4813dc70 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:29:27.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.921+0000 7f2c4ee33640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:27.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.921+0000 7f2c4ee33640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2c48070480 con 0x7f2c4813dc70 2024-09-17T18:29:27.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.921+0000 7f2c4ee33640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:27.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.921+0000 7f2c4ee33640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2c48070300 con 0x7f2c480a5fe0 2024-09-17T18:29:27.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.921+0000 7f2c4ee33640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:27.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.921+0000 7f2c4ee33640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2c481405a0 con 0x7f2c480a5c00 2024-09-17T18:29:27.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.921+0000 7f2c4ee33640 10 monclient(hunting): _renew_subs 2024-09-17T18:29:27.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.921+0000 7f2c4ee33640 10 monclient(hunting): authenticate will time out at 1371.638769s 2024-09-17T18:29:27.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.921+0000 7f2c4cbcf640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c480a5c00 0x7f2c48138850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:27.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.921+0000 7f2c4cbcf640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c480a5c00 0x7f2c48138850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:47168/0 (socket says 172.21.15.121:47168) 2024-09-17T18:29:27.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.921+0000 7f2c4cbcf640 1 -- 172.21.15.121:0/2770655485 learned_addr learned my addr 172.21.15.121:0/2770655485 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:29:27.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.921+0000 7f2c4cbcf640 10 monclient(hunting): get_auth_request con 0x7f2c480a5c00 auth_method 0 2024-09-17T18:29:27.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.921+0000 7f2c4cbcf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:27.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.921+0000 7f2c4cbcf640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:27.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.921+0000 7f2c4cbcf640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:27.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.921+0000 7f2c4d3d0640 1 --2- 172.21.15.121:0/2770655485 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c4813dc70 0x7f2c4813bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:27.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.921+0000 7f2c4d3d0640 10 monclient(hunting): get_auth_request con 0x7f2c4813dc70 auth_method 0 2024-09-17T18:29:27.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.921+0000 7f2c4d3d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:27.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.921+0000 7f2c4d3d0640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:27.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.921+0000 7f2c4d3d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:27.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.921+0000 7f2c4cbcf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:27.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.921+0000 7f2c4cbcf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:27.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.921+0000 7f2c4cbcf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:27.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.921+0000 7f2c4d3d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:27.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.921+0000 7f2c4d3d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:27.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.921+0000 7f2c4d3d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:27.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.922+0000 7f2c4d3d0640 10 monclient(hunting): handle_auth_done global_id 44586 payload 824 2024-09-17T18:29:27.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.922+0000 7f2c3ffff640 1 --2- 172.21.15.121:0/2770655485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c480a5fe0 0x7f2c4813b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:27.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.922+0000 7f2c4d3d0640 1 -- 172.21.15.121:0/2770655485 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c480a5c00 msgr2=0x7f2c48138850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:27.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.922+0000 7f2c4d3d0640 1 --2- 172.21.15.121:0/2770655485 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c480a5c00 0x7f2c48138850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.922+0000 7f2c4d3d0640 1 -- 172.21.15.121:0/2770655485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c480a5fe0 msgr2=0x7f2c4813b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:27.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.922+0000 7f2c4d3d0640 1 --2- 172.21.15.121:0/2770655485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c480a5fe0 0x7f2c4813b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.922+0000 7f2c4d3d0640 10 monclient: _finish_hunting 0 2024-09-17T18:29:27.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.922+0000 7f2c4d3d0640 1 monclient: found mon.noname-a 2024-09-17T18:29:27.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.922+0000 7f2c4d3d0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:29:27.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.922+0000 7f2c4d3d0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:29:27.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.922+0000 7f2c4d3d0640 1 -- 172.21.15.121:0/2770655485 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c480aa6d0 con 0x7f2c4813dc70 2024-09-17T18:29:27.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.922+0000 7f2c4cbcf640 1 --2- 172.21.15.121:0/2770655485 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c480a5c00 0x7f2c48138850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:29:27.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.924+0000 7f2c4d3d0640 1 -- 172.21.15.121:0/2770655485 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c4813dc70 msgr2=0x7f2c4813bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:27.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.924+0000 7f2c4d3d0640 1 --2- 172.21.15.121:0/2770655485 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c4813dc70 0x7f2c4813bdc0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2c40031000 tx=0x7f2c400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:27.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.924+0000 7f2c3d7fa640 10 client.?.objecter ms_handle_connect 0x7f2c4813dc70 2024-09-17T18:29:27.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.924+0000 7f2c3d7fa640 10 client.?.objecter resend_mon_ops 2024-09-17T18:29:27.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.924+0000 7f2c3d7fa640 1 -- 172.21.15.121:0/2770655485 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c40037040 con 0x7f2c4813dc70 2024-09-17T18:29:27.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.924+0000 7f2c3d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:27.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.924+0000 7f2c3d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:29:27.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.924+0000 7f2c3d7fa640 10 monclient: dump: 2024-09-17T18:29:27.938 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:27.938 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:27.938 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.938 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.939 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:27.939 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:27.939 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:27.939 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:27.939 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:27.939 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:27.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.924+0000 7f2c3d7fa640 10 monclient: _finish_auth 0 2024-09-17T18:29:27.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.924+0000 7f2c3d7fa640 10 monclient: _check_auth_tickets 2024-09-17T18:29:27.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.924+0000 7f2c3d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:29:27.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.924+0000 7f2c3d7fa640 1 -- 172.21.15.121:0/2770655485 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c40042070 con 0x7f2c4813dc70 2024-09-17T18:29:27.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.924+0000 7f2c4ee33640 5 monclient: authenticate success, global_id 44586 2024-09-17T18:29:27.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.924+0000 7f2c3d7fa640 10 monclient: handle_config config(0 keys) 2024-09-17T18:29:27.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.924+0000 7f2c3d7fa640 1 -- 172.21.15.121:0/2770655485 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c4003b840 con 0x7f2c4813dc70 2024-09-17T18:29:27.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.924+0000 7f2c4ee33640 10 monclient: _renew_subs 2024-09-17T18:29:27.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.924+0000 7f2c4ee33640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:29:27.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.924+0000 7f2c4ee33640 1 -- 172.21.15.121:0/2770655485 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c48122ff0 con 0x7f2c4813dc70 2024-09-17T18:29:27.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.924+0000 7f2c3d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:27.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.924+0000 7f2c3d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:29:27.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.924+0000 7f2c3d7fa640 10 monclient: dump: 2024-09-17T18:29:27.940 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:27.940 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:27.940 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.940 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.940 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:27.940 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:27.940 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:27.940 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:27.940 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:27.940 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:27.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.924+0000 7f2c3effd640 4 set_mon_vals no callback set 2024-09-17T18:29:27.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.924+0000 7f2c4ee33640 10 client.44586.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:29:27.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.924+0000 7f2c4ee33640 10 monclient: _renew_subs 2024-09-17T18:29:27.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.924+0000 7f2c4ee33640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:29:27.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.924+0000 7f2c4ee33640 1 -- 172.21.15.121:0/2770655485 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c48121e30 con 0x7f2c4813dc70 2024-09-17T18:29:27.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.924+0000 7f2c4ee33640 1 librados: init done 2024-09-17T18:29:27.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.925+0000 7f2c4ee33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:29:27.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.925+0000 7f2c4ee33640 1 --2- 172.21.15.121:0/2770655485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c48138d90 0x7f2c4802d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:27.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.925+0000 7f2c4ee33640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:27.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.925+0000 7f2c4ee33640 1 -- 172.21.15.121:0/2770655485 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4809d930 con 0x7f2c48138d90 2024-09-17T18:29:27.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.925+0000 7f2c3d7fa640 1 -- 172.21.15.121:0/2770655485 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f2c4003ba20 con 0x7f2c4813dc70 2024-09-17T18:29:27.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.925+0000 7f2c4cbcf640 1 --2- 172.21.15.121:0/2770655485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c48138d90 0x7f2c4802d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:27.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.925+0000 7f2c4cbcf640 10 monclient: get_auth_request con 0x7f2c48138d90 auth_method 0 2024-09-17T18:29:27.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.925+0000 7f2c4cbcf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:27.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.925+0000 7f2c4cbcf640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:27.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.925+0000 7f2c4cbcf640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:27.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.925+0000 7f2c4cbcf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:27.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.925+0000 7f2c4cbcf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:27.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.925+0000 7f2c4cbcf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:27.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.925+0000 7f2c3d7fa640 1 --2- 172.21.15.121:0/2770655485 >> v2:172.21.15.173:6812/2423786037 conn(0x7f2c34040170 0x7f2c34042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:27.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.925+0000 7f2c3d7fa640 1 -- 172.21.15.121:0/2770655485 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(412..412 src has 247..412) ==== 5345+0+0 (secure 0 0 0) 0x7f2c40073bf0 con 0x7f2c4813dc70 2024-09-17T18:29:27.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.925+0000 7f2c3d7fa640 10 client.44586.objecter ms_dispatch 0x7f2c48000b90 osd_map(412..412 src has 247..412) 2024-09-17T18:29:27.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.925+0000 7f2c3d7fa640 3 client.44586.objecter handle_osd_map got epochs [412,412] > 0 2024-09-17T18:29:27.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.925+0000 7f2c3d7fa640 3 client.44586.objecter handle_osd_map decoding full epoch 412 2024-09-17T18:29:27.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.925+0000 7f2c4cbcf640 10 monclient(hunting): handle_auth_done global_id 74152 payload 824 2024-09-17T18:29:27.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.925+0000 7f2c3d7fa640 20 client.44586.objecter dump_active .. 0 homeless 2024-09-17T18:29:27.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.925+0000 7f2c4cbcf640 1 -- 172.21.15.121:0/2770655485 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2c48139820 con 0x7f2c48138d90 2024-09-17T18:29:27.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.926+0000 7f2c3ffff640 1 --2- 172.21.15.121:0/2770655485 >> v2:172.21.15.173:6812/2423786037 conn(0x7f2c34040170 0x7f2c34042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:27.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.926+0000 7f2c3ffff640 10 monclient: get_auth_request con 0x7f2c34040170 auth_method 0 2024-09-17T18:29:27.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.928+0000 7f2c3d7fa640 10 client.44586.objecter ms_handle_connect 0x7f2c48138d90 2024-09-17T18:29:27.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.928+0000 7f2c4cbcf640 1 -- 172.21.15.121:0/2770655485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c48138d90 msgr2=0x7f2c4802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:27.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.928+0000 7f2c3d7fa640 10 client.44586.objecter resend_mon_ops 2024-09-17T18:29:27.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.928+0000 7f2c4cbcf640 1 --2- 172.21.15.121:0/2770655485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c48138d90 0x7f2c4802d440 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2c44007590 tx=0x7f2c44007a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:27.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.928+0000 7f2c3d7fa640 10 client.44586.objecter ms_handle_connect 0x7f2c34040170 2024-09-17T18:29:27.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.928+0000 7f2c3ffff640 1 --2- 172.21.15.121:0/2770655485 >> v2:172.21.15.173:6812/2423786037 conn(0x7f2c34040170 0x7f2c34042630 secure :-1 s=READY pgs=4016 cs=0 l=1 rev1=1 crypto rx=0x7f2c38003d90 tx=0x7f2c38009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:27.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.928+0000 7f2c3d7fa640 1 -- 172.21.15.121:0/2770655485 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c44014880 con 0x7f2c48138d90 2024-09-17T18:29:27.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.928+0000 7f2c3d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:27.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.928+0000 7f2c3d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:29:27.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.928+0000 7f2c3d7fa640 10 monclient: dump: 2024-09-17T18:29:27.945 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:27.945 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:27.946 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.946 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.946 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:27.946 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:27.946 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:27.946 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:27.946 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:27.946 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:27.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.928+0000 7f2c3d7fa640 1 -- 172.21.15.121:0/2770655485 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2c4401a070 con 0x7f2c48138d90 2024-09-17T18:29:27.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.928+0000 7f2c3d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:29:27.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.928+0000 7f2c3d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:29:27.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.928+0000 7f2c3d7fa640 1 -- 172.21.15.121:0/2770655485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c48138d90 msgr2=0x7f2c4802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:27.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.928+0000 7f2c3d7fa640 1 --2- 172.21.15.121:0/2770655485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c48138d90 0x7f2c4802d440 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2c44007590 tx=0x7f2c44007a50 comp rx=0 tx=0).stop 2024-09-17T18:29:27.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.928+0000 7f2c3d7fa640 1 -- 172.21.15.121:0/2770655485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c48138d90 msgr2=0x7f2c4802d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:29:27.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.928+0000 7f2c3d7fa640 1 --2- 172.21.15.121:0/2770655485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c48138d90 0x7f2c4802d440 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2c44007590 tx=0x7f2c44007a50 comp rx=0 tx=0).stop 2024-09-17T18:29:27.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.936+0000 7f2c4ee33640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-17T18:29:27.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.936+0000 7f2c4ee33640 1 --2- 172.21.15.121:0/2770655485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c4802d980 0x7f2c4802dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:27.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.936+0000 7f2c4ee33640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:27.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.936+0000 7f2c4ee33640 1 -- 172.21.15.121:0/2770655485 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2c480a6da0 con 0x7f2c4802d980 2024-09-17T18:29:27.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.936+0000 7f2c4cbcf640 1 --2- 172.21.15.121:0/2770655485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c4802d980 0x7f2c4802dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:27.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.936+0000 7f2c4cbcf640 10 monclient: get_auth_request con 0x7f2c4802d980 auth_method 0 2024-09-17T18:29:27.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.936+0000 7f2c4cbcf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:27.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.936+0000 7f2c4cbcf640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:27.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.936+0000 7f2c4cbcf640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:27.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.936+0000 7f2c4cbcf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:27.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.936+0000 7f2c4cbcf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:27.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.936+0000 7f2c4cbcf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:27.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.936+0000 7f2c4cbcf640 10 monclient(hunting): handle_auth_done global_id 74155 payload 824 2024-09-17T18:29:27.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.936+0000 7f2c4cbcf640 1 -- 172.21.15.121:0/2770655485 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7f2c480a3d10 con 0x7f2c4802d980 2024-09-17T18:29:27.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.938+0000 7f2c4cbcf640 1 -- 172.21.15.121:0/2770655485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c4802d980 msgr2=0x7f2c4802dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:27.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.938+0000 7f2c3d7fa640 10 client.44586.objecter ms_handle_connect 0x7f2c4802d980 2024-09-17T18:29:27.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.938+0000 7f2c3d7fa640 10 client.44586.objecter resend_mon_ops 2024-09-17T18:29:27.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.938+0000 7f2c4cbcf640 1 --2- 172.21.15.121:0/2770655485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c4802d980 0x7f2c4802dd60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2c44019000 tx=0x7f2c44004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:27.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.939+0000 7f2c3d7fa640 1 -- 172.21.15.121:0/2770655485 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c4401fde0 con 0x7f2c4802d980 2024-09-17T18:29:27.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.939+0000 7f2c3d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:27.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.939+0000 7f2c3d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:29:27.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.939+0000 7f2c3d7fa640 10 monclient: dump: 2024-09-17T18:29:27.948 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:27.948 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:27.948 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.948 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:27.948 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:27.949 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:27.949 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:27.949 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:27.949 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:27.949 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:27.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.939+0000 7f2c3d7fa640 1 -- 172.21.15.121:0/2770655485 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (secure 0 0 0) 0x7f2c4401fde0 con 0x7f2c4802d980 2024-09-17T18:29:27.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.939+0000 7f2c3d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-17T18:29:27.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.939+0000 7f2c3d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:29:27.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.939+0000 7f2c3d7fa640 1 -- 172.21.15.121:0/2770655485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c4802d980 msgr2=0x7f2c4802dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:27.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.939+0000 7f2c3d7fa640 1 --2- 172.21.15.121:0/2770655485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c4802d980 0x7f2c4802dd60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2c44019000 tx=0x7f2c44004a40 comp rx=0 tx=0).stop 2024-09-17T18:29:27.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.939+0000 7f2c3d7fa640 1 -- 172.21.15.121:0/2770655485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c4802d980 msgr2=0x7f2c4802dd60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:29:27.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.939+0000 7f2c3d7fa640 1 --2- 172.21.15.121:0/2770655485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c4802d980 0x7f2c4802dd60 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2c44019000 tx=0x7f2c44004a40 comp rx=0 tx=0).stop 2024-09-17T18:29:27.950 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:29:27.950 INFO:teuthology.orchestra.run.smithi121.stdout: "ret": 0, 2024-09-17T18:29:27.950 INFO:teuthology.orchestra.run.smithi121.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-09-17T18:29:27.950 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:29:27.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.939+0000 7f2c4ee33640 10 librados: watch_flush enter 2024-09-17T18:29:27.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.940+0000 7f2c4ee33640 10 librados: watch_flush exit 2024-09-17T18:29:27.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.940+0000 7f2c4ee33640 20 client.44586.objecter shutdown clearing up homeless session... 2024-09-17T18:29:27.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.940+0000 7f2c4ee33640 10 client.44586.objecter successfully canceled tick 2024-09-17T18:29:27.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.940+0000 7f2c4ee33640 1 -- 172.21.15.121:0/2770655485 >> v2:172.21.15.173:6812/2423786037 conn(0x7f2c34040170 msgr2=0x7f2c34042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:27.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.940+0000 7f2c4ee33640 1 --2- 172.21.15.121:0/2770655485 >> v2:172.21.15.173:6812/2423786037 conn(0x7f2c34040170 0x7f2c34042630 secure :-1 s=READY pgs=4016 cs=0 l=1 rev1=1 crypto rx=0x7f2c38003d90 tx=0x7f2c38009040 comp rx=0 tx=0).stop 2024-09-17T18:29:27.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.940+0000 7f2c4ee33640 10 monclient: shutdown 2024-09-17T18:29:27.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.940+0000 7f2c4ee33640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:29:27.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.940+0000 7f2c4ee33640 1 -- 172.21.15.121:0/2770655485 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c4813dc70 msgr2=0x7f2c4813bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:27.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.940+0000 7f2c4ee33640 1 --2- 172.21.15.121:0/2770655485 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c4813dc70 0x7f2c4813bdc0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2c40031000 tx=0x7f2c400046c0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.940+0000 7f2c4d3d0640 1 -- 172.21.15.121:0/2770655485 reap_dead start 2024-09-17T18:29:27.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.940+0000 7f2c4ee33640 1 -- 172.21.15.121:0/2770655485 shutdown_connections 2024-09-17T18:29:27.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.940+0000 7f2c4ee33640 1 --2- 172.21.15.121:0/2770655485 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c4813dc70 0x7f2c4813bdc0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:27.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.940+0000 7f2c4ee33640 1 -- 172.21.15.121:0/2770655485 >> 172.21.15.121:0/2770655485 conn(0x7f2c480ab9e0 msgr2=0x7f2c480a1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:29:27.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.940+0000 7f2c4ee33640 1 -- 172.21.15.121:0/2770655485 shutdown_connections 2024-09-17T18:29:27.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.940+0000 7f2c4ee33640 1 -- 172.21.15.121:0/2770655485 wait complete. 2024-09-17T18:29:27.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:27.940+0000 7f2c4ee33640 1 librados: shutdown 2024-09-17T18:29:27.953 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-17T18:29:27.953 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-17T18:29:27.953 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T18:29:27.966 INFO:tasks.ceph.mon.b:Stopped 2024-09-17T18:29:27.966 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-17T18:29:27.966 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:29:28.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.060+0000 7fd3f707f640 10 monclient: get_monmap_and_config 2024-09-17T18:29:28.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.060+0000 7fd3f707f640 10 monclient: build_initial_monmap 2024-09-17T18:29:28.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.060+0000 7fd3f707f640 10 monclient: monmap: 2024-09-17T18:29:28.062 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:29:28.063 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:28.063 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:29:28.061149+0000 2024-09-17T18:29:28.063 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:29:28.061149+0000 2024-09-17T18:29:28.063 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:29:28.063 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:28.063 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:29:28.063 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:29:28.063 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:29:28.063 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:29:28.063 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:29:28.063 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:28.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.060+0000 7fd3f707f640 1 Processor -- start 2024-09-17T18:29:28.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.060+0000 7fd3f707f640 1 -- start start 2024-09-17T18:29:28.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.060+0000 7fd3f707f640 10 monclient: init 2024-09-17T18:29:28.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.060+0000 7fd3f707f640 10 monclient: _reopen_session rank -1 2024-09-17T18:29:28.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.060+0000 7fd3f707f640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-17T18:29:28.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.060+0000 7fd3f707f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd3f00a0a70 addr v1:172.21.15.173:6789/0 2024-09-17T18:29:28.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.060+0000 7fd3f707f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3f01348f0 0x7fd3f0139c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:28.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.060+0000 7fd3f707f640 10 monclient(hunting): picked mon.noname-b con 0x7fd3f01348f0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:29:28.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.060+0000 7fd3f707f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd3f013a150 addr v1:172.21.15.121:6789/0 2024-09-17T18:29:28.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.060+0000 7fd3f707f640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3f009f5b0 con 0x7fd3f013a150 2024-09-17T18:29:28.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.060+0000 7fd3f707f640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3f00a1560 con 0x7fd3f00a0a70 2024-09-17T18:29:28.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.060+0000 7fd3f707f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:28.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.060+0000 7fd3f707f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd3f009ef80 con 0x7fd3f01348f0 2024-09-17T18:29:28.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.060+0000 7fd3f707f640 10 monclient(hunting): _renew_subs 2024-09-17T18:29:28.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.060+0000 7fd3f707f640 10 monclient(hunting): authenticate will time out at 1371.777801s 2024-09-17T18:29:28.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.060+0000 7fd3e7fff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3f01348f0 0x7fd3f0139c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:28.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.060+0000 7fd3f561c640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fd3f013a150 0x7fd3f013c540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:43822/0 (socket says 172.21.15.121:43822) 2024-09-17T18:29:28.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.060+0000 7fd3f561c640 1 -- 172.21.15.121:0/1311653877 learned_addr learned my addr 172.21.15.121:0/1311653877 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:29:28.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.060+0000 7fd3e7fff640 10 monclient(hunting): get_auth_request con 0x7fd3f01348f0 auth_method 0 2024-09-17T18:29:28.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.060+0000 7fd3e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:28.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.060+0000 7fd3e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:28.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.060+0000 7fd3e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:28.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.060+0000 7fd3f4e1b640 1 -- 172.21.15.121:0/1311653877 >> v1:172.21.15.173:6789/0 conn(0x7fd3f00a0a70 legacy=0x7fd3f00a0e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.173:6789/0 2024-09-17T18:29:28.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.061+0000 7fd3e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:28.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.061+0000 7fd3e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:28.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.061+0000 7fd3e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:28.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.061+0000 7fd3e7fff640 10 monclient(hunting): handle_auth_done global_id 64226 payload 274 2024-09-17T18:29:28.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.061+0000 7fd3e7fff640 1 -- 172.21.15.121:0/1311653877 >> v1:172.21.15.173:6789/0 conn(0x7fd3f00a0a70 legacy=0x7fd3f00a0e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:29:28.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.061+0000 7fd3e7fff640 1 -- 172.21.15.121:0/1311653877 >> v1:172.21.15.121:6789/0 conn(0x7fd3f013a150 legacy=0x7fd3f013c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:29:28.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.061+0000 7fd3e7fff640 10 monclient: _finish_hunting 0 2024-09-17T18:29:28.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.061+0000 7fd3e7fff640 1 monclient: found mon.noname-b 2024-09-17T18:29:28.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.061+0000 7fd3e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:29:28.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.061+0000 7fd3e7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:29:28.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.061+0000 7fd3e7fff640 1 -- 172.21.15.121:0/1311653877 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3f009eca0 con 0x7fd3f01348f0 2024-09-17T18:29:28.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.063+0000 7fd3e7fff640 1 -- 172.21.15.121:0/1311653877 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3f01348f0 msgr2=0x7fd3f0139c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:28.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.063+0000 7fd3e7fff640 1 --2- 172.21.15.121:0/1311653877 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3f01348f0 0x7fd3f0139c10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd3e000a970 tx=0x7fd3e002b360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d16bf889c967dd32 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:28.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.064+0000 7fd3e77fe640 1 -- 172.21.15.121:0/1311653877 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3e0038070 con 0x7fd3f01348f0 2024-09-17T18:29:28.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.064+0000 7fd3e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:28.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.064+0000 7fd3e77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:29:28.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.064+0000 7fd3e77fe640 10 monclient: dump: 2024-09-17T18:29:28.075 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:28.075 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:28.075 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:28.075 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:28.075 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:28.075 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:28.075 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:28.076 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:28.076 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:28.076 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:28.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.064+0000 7fd3e77fe640 10 monclient: _finish_auth 0 2024-09-17T18:29:28.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.064+0000 7fd3e77fe640 10 monclient: _check_auth_tickets 2024-09-17T18:29:28.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.064+0000 7fd3e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:29:28.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.064+0000 7fd3f707f640 5 monclient: authenticate success, global_id 64226 2024-09-17T18:29:28.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.064+0000 7fd3e77fe640 1 -- 172.21.15.121:0/1311653877 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3e00306b0 con 0x7fd3f01348f0 2024-09-17T18:29:28.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.064+0000 7fd3f707f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:29:28.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.064+0000 7fd3e77fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:29:28.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.064+0000 7fd3f707f640 10 monclient: get_monmap_and_config success 2024-09-17T18:29:28.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.064+0000 7fd3e77fe640 1 -- 172.21.15.121:0/1311653877 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3e000b040 con 0x7fd3f01348f0 2024-09-17T18:29:28.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.064+0000 7fd3f707f640 4 set_mon_vals no callback set 2024-09-17T18:29:28.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.064+0000 7fd3e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:28.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.064+0000 7fd3e77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:29:28.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.064+0000 7fd3e77fe640 10 monclient: dump: 2024-09-17T18:29:28.077 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:28.077 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:28.077 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:28.077 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:28.077 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:28.077 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:28.077 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:28.077 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:28.077 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:28.077 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:28.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.064+0000 7fd3f707f640 10 monclient: shutdown 2024-09-17T18:29:28.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.064+0000 7fd3f707f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:29:28.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.064+0000 7fd3f707f640 1 -- 172.21.15.121:0/1311653877 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3f01348f0 msgr2=0x7fd3f0139c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:28.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.064+0000 7fd3f707f640 1 --2- 172.21.15.121:0/1311653877 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3f01348f0 0x7fd3f0139c10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd3e000a970 tx=0x7fd3e002b360 comp rx=0 tx=0).stop 2024-09-17T18:29:28.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.064+0000 7fd3f707f640 1 -- 172.21.15.121:0/1311653877 shutdown_connections 2024-09-17T18:29:28.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.064+0000 7fd3f707f640 1 --2- 172.21.15.121:0/1311653877 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3f01348f0 0x7fd3f0139c10 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:28.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.064+0000 7fd3f707f640 1 -- 172.21.15.121:0/1311653877 >> 172.21.15.121:0/1311653877 conn(0x7fd3f00a3b50 msgr2=0x7fd3f00a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:29:28.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.064+0000 7fd3f707f640 1 -- 172.21.15.121:0/1311653877 shutdown_connections 2024-09-17T18:29:28.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.064+0000 7fd3f707f640 1 -- 172.21.15.121:0/1311653877 wait complete. 2024-09-17T18:29:28.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.065+0000 7fd3f707f640 10 monclient: build_initial_monmap 2024-09-17T18:29:28.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.065+0000 7fd3f707f640 10 monclient: monmap: 2024-09-17T18:29:28.078 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:29:28.078 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:29:28.079 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:29:28.079 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:29:28.079 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:29:28.079 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:28.079 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:29:28.079 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:29:28.079 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:29:28.079 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:28.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.065+0000 7fd3f707f640 1 librados: starting msgr at 2024-09-17T18:29:28.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.065+0000 7fd3f707f640 1 librados: starting objecter 2024-09-17T18:29:28.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.065+0000 7fd3f707f640 1 Processor -- start 2024-09-17T18:29:28.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.065+0000 7fd3f707f640 1 -- start start 2024-09-17T18:29:28.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.065+0000 7fd3f707f640 1 librados: setting wanted keys 2024-09-17T18:29:28.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.065+0000 7fd3f707f640 1 librados: calling monclient init 2024-09-17T18:29:28.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.065+0000 7fd3f707f640 10 monclient: init 2024-09-17T18:29:28.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f707f640 10 monclient: _reopen_session rank -1 2024-09-17T18:29:28.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f707f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:29:28.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f707f640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3f00a0a70 0x7fd3f01334d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:28.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f707f640 10 monclient(hunting): picked mon.noname-a con 0x7fd3f00a0a70 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:29:28.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f707f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3f01348f0 0x7fd3f0133a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:28.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f707f640 10 monclient(hunting): picked mon.noname-b con 0x7fd3f01348f0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:29:28.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f707f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3f013a150 0x7fd3f011d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:28.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f707f640 10 monclient(hunting): picked mon.noname-c con 0x7fd3f013a150 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:29:28.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f707f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:28.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f707f640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd3f009de60 con 0x7fd3f00a0a70 2024-09-17T18:29:28.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f707f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:28.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f707f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd3f009dce0 con 0x7fd3f01348f0 2024-09-17T18:29:28.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f707f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:28.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f707f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd3f00a1a50 con 0x7fd3f013a150 2024-09-17T18:29:28.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f707f640 10 monclient(hunting): _renew_subs 2024-09-17T18:29:28.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f707f640 10 monclient(hunting): authenticate will time out at 1371.783194s 2024-09-17T18:29:28.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f4e1b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3f00a0a70 0x7fd3f01334d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:28.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f561c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3f013a150 0x7fd3f011d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:28.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f4e1b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3f00a0a70 0x7fd3f01334d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58184/0 (socket says 172.21.15.121:58184) 2024-09-17T18:29:28.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f4e1b640 1 -- 172.21.15.121:0/630603808 learned_addr learned my addr 172.21.15.121:0/630603808 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:29:28.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f561c640 10 monclient(hunting): get_auth_request con 0x7fd3f013a150 auth_method 0 2024-09-17T18:29:28.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f561c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:28.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f561c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:28.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f561c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:28.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f4e1b640 10 monclient(hunting): get_auth_request con 0x7fd3f00a0a70 auth_method 0 2024-09-17T18:29:28.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f4e1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:28.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f4e1b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:28.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f4e1b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:28.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3e7fff640 1 -- 172.21.15.121:0/630603808 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3f01348f0 msgr2=0x7fd3f0133a10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.173:3300/0 2024-09-17T18:29:28.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3e7fff640 1 --2- 172.21.15.121:0/630603808 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3f01348f0 0x7fd3f0133a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:29:28.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f561c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:28.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f561c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:28.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f561c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:28.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f4e1b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:28.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f4e1b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:28.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f4e1b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:28.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f561c640 10 monclient(hunting): handle_auth_done global_id 64229 payload 824 2024-09-17T18:29:28.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f561c640 1 -- 172.21.15.121:0/630603808 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3f01348f0 msgr2=0x7fd3f0133a10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:29:28.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f561c640 1 --2- 172.21.15.121:0/630603808 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3f01348f0 0x7fd3f0133a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:28.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f561c640 1 -- 172.21.15.121:0/630603808 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3f00a0a70 msgr2=0x7fd3f01334d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:28.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f561c640 1 --2- 172.21.15.121:0/630603808 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3f00a0a70 0x7fd3f01334d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:28.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f561c640 10 monclient: _finish_hunting 0 2024-09-17T18:29:28.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f561c640 1 monclient: found mon.noname-c 2024-09-17T18:29:28.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f561c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:29:28.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f561c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:29:28.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.066+0000 7fd3f561c640 1 -- 172.21.15.121:0/630603808 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3f009e240 con 0x7fd3f013a150 2024-09-17T18:29:28.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.068+0000 7fd3f561c640 1 -- 172.21.15.121:0/630603808 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3f013a150 msgr2=0x7fd3f011d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:28.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.068+0000 7fd3f561c640 1 --2- 172.21.15.121:0/630603808 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3f013a150 0x7fd3f011d900 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd3e800c3e0 tx=0x7fd3e800c7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:28.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.068+0000 7fd3e57fa640 10 client.?.objecter ms_handle_connect 0x7fd3f013a150 2024-09-17T18:29:28.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.068+0000 7fd3e57fa640 10 client.?.objecter resend_mon_ops 2024-09-17T18:29:28.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.069+0000 7fd3e57fa640 1 -- 172.21.15.121:0/630603808 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3e8017070 con 0x7fd3f013a150 2024-09-17T18:29:28.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.069+0000 7fd3e57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:28.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.069+0000 7fd3e57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:29:28.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.069+0000 7fd3e57fa640 10 monclient: dump: 2024-09-17T18:29:28.084 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:28.084 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:28.084 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:28.084 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:28.084 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:28.084 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:28.084 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:28.084 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:28.084 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:28.084 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:28.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.069+0000 7fd3e57fa640 10 monclient: _finish_auth 0 2024-09-17T18:29:28.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.069+0000 7fd3e57fa640 10 monclient: _check_auth_tickets 2024-09-17T18:29:28.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.069+0000 7fd3e57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:29:28.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.069+0000 7fd3f707f640 5 monclient: authenticate success, global_id 64229 2024-09-17T18:29:28.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.069+0000 7fd3e57fa640 1 -- 172.21.15.121:0/630603808 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3e80070d0 con 0x7fd3f013a150 2024-09-17T18:29:28.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.069+0000 7fd3e57fa640 10 monclient: handle_config config(0 keys) 2024-09-17T18:29:28.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.069+0000 7fd3e57fa640 1 -- 172.21.15.121:0/630603808 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3e80074c0 con 0x7fd3f013a150 2024-09-17T18:29:28.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.069+0000 7fd3f707f640 10 monclient: _renew_subs 2024-09-17T18:29:28.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.069+0000 7fd3f707f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:29:28.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.069+0000 7fd3f707f640 1 -- 172.21.15.121:0/630603808 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3f0121220 con 0x7fd3f013a150 2024-09-17T18:29:28.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.069+0000 7fd3e57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:28.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.069+0000 7fd3e57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:29:28.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.069+0000 7fd3e57fa640 10 monclient: dump: 2024-09-17T18:29:28.085 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:28.085 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:28.085 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:28.086 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:28.086 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:28.086 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:28.086 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:28.086 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:28.086 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:28.086 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:28.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.069+0000 7fd3e6ffd640 4 set_mon_vals no callback set 2024-09-17T18:29:28.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.069+0000 7fd3f707f640 10 client.64229.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:29:28.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.069+0000 7fd3f707f640 10 monclient: _renew_subs 2024-09-17T18:29:28.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.069+0000 7fd3f707f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:29:28.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.069+0000 7fd3f707f640 1 -- 172.21.15.121:0/630603808 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3f00a4e50 con 0x7fd3f013a150 2024-09-17T18:29:28.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.069+0000 7fd3f707f640 1 librados: init done 2024-09-17T18:29:28.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.070+0000 7fd3e57fa640 1 -- 172.21.15.121:0/630603808 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fd3e801a070 con 0x7fd3f013a150 2024-09-17T18:29:28.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.070+0000 7fd3e57fa640 1 --2- 172.21.15.121:0/630603808 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd3dc040170 0x7fd3dc042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:28.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.070+0000 7fd3f707f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:29:28.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.070+0000 7fd3f707f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:29:28.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.070+0000 7fd3f707f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:29:28.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.070+0000 7fd3f707f640 1 -- 172.21.15.121:0/630603808 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3f00a0e50 con 0x7fd3f013a150 2024-09-17T18:29:28.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.070+0000 7fd3e57fa640 1 -- 172.21.15.121:0/630603808 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(412..412 src has 247..412) ==== 5345+0+0 (secure 0 0 0) 0x7fd3e8052a90 con 0x7fd3f013a150 2024-09-17T18:29:28.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.070+0000 7fd3e57fa640 10 client.64229.objecter ms_dispatch 0x7fd3f0000b90 osd_map(412..412 src has 247..412) 2024-09-17T18:29:28.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.070+0000 7fd3e57fa640 3 client.64229.objecter handle_osd_map got epochs [412,412] > 0 2024-09-17T18:29:28.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.070+0000 7fd3e57fa640 3 client.64229.objecter handle_osd_map decoding full epoch 412 2024-09-17T18:29:28.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.070+0000 7fd3e57fa640 20 client.64229.objecter dump_active .. 0 homeless 2024-09-17T18:29:28.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.070+0000 7fd3f4e1b640 1 --2- 172.21.15.121:0/630603808 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd3dc040170 0x7fd3dc042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:28.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.070+0000 7fd3f4e1b640 10 monclient: get_auth_request con 0x7fd3dc040170 auth_method 0 2024-09-17T18:29:28.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.073+0000 7fd3f4e1b640 1 --2- 172.21.15.121:0/630603808 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd3dc040170 0x7fd3dc042630 secure :-1 s=READY pgs=4017 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec0094b0 tx=0x7fd3ec009400 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:28.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.073+0000 7fd3e57fa640 1 -- 172.21.15.121:0/630603808 DispatchQueue::entry inject delay of 0.002000 2024-09-17T18:29:28.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.075+0000 7fd3e57fa640 10 client.64229.objecter ms_handle_connect 0x7fd3dc040170 2024-09-17T18:29:28.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.075+0000 7fd3e57fa640 1 -- 172.21.15.121:0/630603808 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd3f009e240 con 0x7fd3f013a150 2024-09-17T18:29:28.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.075+0000 7fd3e57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:29:28.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.075+0000 7fd3e57fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:29:28.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.320+0000 7fd3f707f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:29:28.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.320+0000 7fd3f707f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:29:28.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.320+0000 7fd3f707f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:29:28.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.320+0000 7fd3f707f640 1 -- 172.21.15.121:0/630603808 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd3f00008d0 con 0x7fd3f013a150 2024-09-17T18:29:28.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.320+0000 7fd3e57fa640 1 -- 172.21.15.121:0/630603808 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fd3e801bb60 con 0x7fd3f013a150 2024-09-17T18:29:28.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.320+0000 7fd3e57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:29:28.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.320+0000 7fd3e57fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:29:28.324 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:29:28.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.322+0000 7fd3f707f640 10 librados: watch_flush enter 2024-09-17T18:29:28.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.322+0000 7fd3f707f640 10 librados: watch_flush exit 2024-09-17T18:29:28.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.322+0000 7fd3f707f640 20 client.64229.objecter shutdown clearing up homeless session... 2024-09-17T18:29:28.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.322+0000 7fd3f707f640 10 client.64229.objecter successfully canceled tick 2024-09-17T18:29:28.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.322+0000 7fd3f707f640 1 -- 172.21.15.121:0/630603808 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd3dc040170 msgr2=0x7fd3dc042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:28.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.322+0000 7fd3f707f640 1 --2- 172.21.15.121:0/630603808 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd3dc040170 0x7fd3dc042630 secure :-1 s=READY pgs=4017 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec0094b0 tx=0x7fd3ec009400 comp rx=0 tx=0).stop 2024-09-17T18:29:28.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.322+0000 7fd3f707f640 10 monclient: shutdown 2024-09-17T18:29:28.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.322+0000 7fd3f707f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:29:28.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.322+0000 7fd3f707f640 1 -- 172.21.15.121:0/630603808 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3f013a150 msgr2=0x7fd3f011d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:28.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.322+0000 7fd3f707f640 1 --2- 172.21.15.121:0/630603808 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3f013a150 0x7fd3f011d900 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd3e800c3e0 tx=0x7fd3e800c7e0 comp rx=0 tx=0).stop 2024-09-17T18:29:28.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.323+0000 7fd3f707f640 1 -- 172.21.15.121:0/630603808 shutdown_connections 2024-09-17T18:29:28.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.323+0000 7fd3f707f640 1 --2- 172.21.15.121:0/630603808 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd3dc040170 0x7fd3dc042630 unknown :-1 s=CLOSED pgs=4017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:28.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.323+0000 7fd3f707f640 1 --2- 172.21.15.121:0/630603808 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3f01348f0 0x7fd3f0133a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:28.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.323+0000 7fd3f707f640 1 --2- 172.21.15.121:0/630603808 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3f013a150 0x7fd3f011d900 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:28.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.323+0000 7fd3f707f640 1 --2- 172.21.15.121:0/630603808 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3f00a0a70 0x7fd3f01334d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:28.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.323+0000 7fd3f707f640 1 -- 172.21.15.121:0/630603808 >> 172.21.15.121:0/630603808 conn(0x7fd3f00a3b50 msgr2=0x7fd3f00a6370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:29:28.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.323+0000 7fd3f707f640 1 -- 172.21.15.121:0/630603808 shutdown_connections 2024-09-17T18:29:28.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.323+0000 7fd3f707f640 1 -- 172.21.15.121:0/630603808 wait complete. 2024-09-17T18:29:28.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:28.323+0000 7fd3f707f640 1 librados: shutdown 2024-09-17T18:29:31.336 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:29:31.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.442+0000 7fe918831640 10 monclient: get_monmap_and_config 2024-09-17T18:29:31.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.442+0000 7fe918831640 10 monclient: build_initial_monmap 2024-09-17T18:29:31.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.442+0000 7fe918831640 10 monclient: monmap: 2024-09-17T18:29:31.444 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:29:31.444 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:31.444 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:29:31.443071+0000 2024-09-17T18:29:31.444 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:29:31.443071+0000 2024-09-17T18:29:31.444 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:29:31.445 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:31.445 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:29:31.445 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:29:31.445 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:29:31.445 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:29:31.445 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:29:31.445 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:31.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.443+0000 7fe918831640 1 Processor -- start 2024-09-17T18:29:31.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.443+0000 7fe918831640 1 -- start start 2024-09-17T18:29:31.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.443+0000 7fe918831640 10 monclient: init 2024-09-17T18:29:31.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.443+0000 7fe918831640 10 monclient: _reopen_session rank -1 2024-09-17T18:29:31.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.443+0000 7fe918831640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-17T18:29:31.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.443+0000 7fe918831640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fe9141386b0 0x7fe9140707d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:31.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.443+0000 7fe918831640 10 monclient(hunting): picked mon.noname-c con 0x7fe9141386b0 addr v2:172.21.15.173:3300/0 2024-09-17T18:29:31.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.443+0000 7fe918831640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe91406ec00 addr v1:172.21.15.173:6789/0 2024-09-17T18:29:31.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.443+0000 7fe918831640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe9141391a0 addr v1:172.21.15.121:6789/0 2024-09-17T18:29:31.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.443+0000 7fe918831640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9140a5570 con 0x7fe9141391a0 2024-09-17T18:29:31.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.443+0000 7fe918831640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe91406f830 con 0x7fe91406ec00 2024-09-17T18:29:31.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.443+0000 7fe918831640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:31.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.443+0000 7fe918831640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fe91406fb10 con 0x7fe9141386b0 2024-09-17T18:29:31.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.443+0000 7fe918831640 10 monclient(hunting): _renew_subs 2024-09-17T18:29:31.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.443+0000 7fe918831640 10 monclient(hunting): authenticate will time out at 1375.159828s 2024-09-17T18:29:31.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.443+0000 7fe91259c640 1 -- >> v2:172.21.15.173:3300/0 conn(0x7fe9141386b0 msgr2=0x7fe9140707d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.173:3300/0 2024-09-17T18:29:31.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.443+0000 7fe911d9b640 1 -- >> v1:172.21.15.173:6789/0 conn(0x7fe91406ec00 legacy=0x7fe914138a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.173:6789/0 2024-09-17T18:29:31.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.443+0000 7fe912d9d640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fe9141391a0 0x7fe91413fbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:45900/0 (socket says 172.21.15.121:45900) 2024-09-17T18:29:31.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.443+0000 7fe912d9d640 1 -- 172.21.15.121:0/1349456196 learned_addr learned my addr 172.21.15.121:0/1349456196 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:29:31.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.443+0000 7fe91259c640 1 --2- 172.21.15.121:0/1349456196 >> v2:172.21.15.173:3300/0 conn(0x7fe9141386b0 0x7fe9140707d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:29:31.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe912d9d640 1 -- 172.21.15.121:0/1349456196 >> v1:172.21.15.121:6789/0 conn(0x7fe9141391a0 legacy=0x7fe91413fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:31.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe91159a640 1 -- 172.21.15.121:0/1349456196 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2737991718 0 0) 0x7fe9140a5570 con 0x7fe9141391a0 2024-09-17T18:29:31.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe91159a640 10 monclient(hunting): _negotiate 2024-09-17T18:29:31.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe91159a640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:31.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe91159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:31.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe91159a640 10 monclient(hunting): my global_id is 44592 2024-09-17T18:29:31.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe91159a640 1 -- 172.21.15.121:0/1349456196 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe900003520 con 0x7fe9141391a0 2024-09-17T18:29:31.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe91159a640 1 -- 172.21.15.121:0/1349456196 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 808656530 0 0) 0x7fe900003520 con 0x7fe9141391a0 2024-09-17T18:29:31.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe91159a640 1 -- 172.21.15.121:0/1349456196 >> v2:172.21.15.173:3300/0 conn(0x7fe9141386b0 msgr2=0x7fe9140707d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:29:31.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe91159a640 1 --2- 172.21.15.121:0/1349456196 >> v2:172.21.15.173:3300/0 conn(0x7fe9141386b0 0x7fe9140707d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:31.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe91159a640 1 -- 172.21.15.121:0/1349456196 >> v1:172.21.15.173:6789/0 conn(0x7fe91406ec00 legacy=0x7fe914138a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:29:31.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe91159a640 10 monclient: _finish_hunting 0 2024-09-17T18:29:31.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe91159a640 1 monclient: found mon.noname-a-legacy 2024-09-17T18:29:31.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe91159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:29:31.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe91159a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.121:6789/0 2024-09-17T18:29:31.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe91159a640 1 -- 172.21.15.121:0/1349456196 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe914070d10 con 0x7fe9141391a0 2024-09-17T18:29:31.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe91159a640 10 monclient: _finish_auth 0 2024-09-17T18:29:31.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe91159a640 10 monclient: _check_auth_tickets 2024-09-17T18:29:31.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe91159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:29:31.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe918831640 5 monclient: authenticate success, global_id 44592 2024-09-17T18:29:31.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe91159a640 1 -- 172.21.15.121:0/1349456196 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4002019445 0 0) 0x7fe9040019a0 con 0x7fe9141391a0 2024-09-17T18:29:31.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe918831640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:29:31.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe91159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:31.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe91159a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T18:29:31.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe91159a640 10 monclient: dump: 2024-09-17T18:29:31.461 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:31.461 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:31.461 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:31.461 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:31.461 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:31.461 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:31.461 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:31.462 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:31.462 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:31.462 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:31.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe91159a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] but i'm connected to v1:172.21.15.121:6789/0, reconnecting 2024-09-17T18:29:31.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe91159a640 10 monclient: _reopen_session rank -1 2024-09-17T18:29:31.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe91159a640 1 -- 172.21.15.121:0/1349456196 >> v1:172.21.15.121:6789/0 conn(0x7fe9141391a0 legacy=0x7fe91413fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:31.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe91159a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T18:29:31.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe91159a640 1 --2- 172.21.15.121:0/1349456196 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9000039b0 0x7fe900003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:31.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe91159a640 10 monclient(hunting): picked mon.b con 0x7fe9000039b0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:29:31.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe91159a640 1 --2- 172.21.15.121:0/1349456196 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe900004800 0x7fe900004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:31.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe91159a640 10 monclient(hunting): picked mon.a con 0x7fe900004800 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:29:31.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe91159a640 1 --2- 172.21.15.121:0/1349456196 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe9000053c0 0x7fe90000ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:31.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe91159a640 10 monclient(hunting): picked mon.c con 0x7fe9000053c0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:29:31.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe91159a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:31.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe91159a640 1 -- 172.21.15.121:0/1349456196 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe91406fb10 con 0x7fe900004800 2024-09-17T18:29:31.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe91159a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:31.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe91159a640 1 -- 172.21.15.121:0/1349456196 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe90000c300 con 0x7fe9000039b0 2024-09-17T18:29:31.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe91159a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:31.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe91159a640 1 -- 172.21.15.121:0/1349456196 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe90000c500 con 0x7fe9000053c0 2024-09-17T18:29:31.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe91159a640 10 monclient(hunting): _renew_subs 2024-09-17T18:29:31.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe91159a640 10 monclient(hunting): _finish_auth 0 2024-09-17T18:29:31.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe918831640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-17T18:29:31.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe911d9b640 1 --2- 172.21.15.121:0/1349456196 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe900004800 0x7fe900004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:31.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe91259c640 1 -- 172.21.15.121:0/1349456196 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9000039b0 msgr2=0x7fe900003db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.173:3300/0 2024-09-17T18:29:31.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe91259c640 1 --2- 172.21.15.121:0/1349456196 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9000039b0 0x7fe900003db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:29:31.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe911d9b640 10 monclient(hunting): get_auth_request con 0x7fe900004800 auth_method 0 2024-09-17T18:29:31.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe911d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:31.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe911d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:31.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe911d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:29:31.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe912d9d640 1 --2- 172.21.15.121:0/1349456196 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe9000053c0 0x7fe90000ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:31.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe912d9d640 10 monclient(hunting): get_auth_request con 0x7fe9000053c0 auth_method 0 2024-09-17T18:29:31.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe912d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:31.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe912d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:31.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe912d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:29:31.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe911d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:31.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe911d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:31.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe911d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:29:31.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe912d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:31.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe912d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:31.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe912d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:29:31.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe911d9b640 10 monclient(hunting): handle_auth_done global_id 44592 payload 293 2024-09-17T18:29:31.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe911d9b640 1 -- 172.21.15.121:0/1349456196 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe9000053c0 msgr2=0x7fe90000ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:29:31.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe911d9b640 1 --2- 172.21.15.121:0/1349456196 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe9000053c0 0x7fe90000ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:31.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe911d9b640 1 -- 172.21.15.121:0/1349456196 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9000039b0 msgr2=0x7fe900003db0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:29:31.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe911d9b640 1 --2- 172.21.15.121:0/1349456196 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9000039b0 0x7fe900003db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:31.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe911d9b640 10 monclient: _finish_hunting 0 2024-09-17T18:29:31.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe911d9b640 1 monclient: found mon.a 2024-09-17T18:29:31.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe911d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:29:31.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe911d9b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:29:31.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe911d9b640 1 -- 172.21.15.121:0/1349456196 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe900003520 con 0x7fe900004800 2024-09-17T18:29:31.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe911d9b640 10 monclient: _finish_auth 0 2024-09-17T18:29:31.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe911d9b640 10 monclient: _check_auth_tickets 2024-09-17T18:29:31.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe911d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:29:31.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.446+0000 7fe912d9d640 1 -- 172.21.15.121:0/1349456196 reap_dead start 2024-09-17T18:29:31.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.449+0000 7fe911d9b640 1 -- 172.21.15.121:0/1349456196 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe900004800 msgr2=0x7fe900004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:31.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.449+0000 7fe911d9b640 1 --2- 172.21.15.121:0/1349456196 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe900004800 0x7fe900004c80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc009900 tx=0x7fe900005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82ef0867aa67b644 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:31.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.449+0000 7fe91159a640 1 -- 172.21.15.121:0/1349456196 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8fc009500 con 0x7fe900004800 2024-09-17T18:29:31.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.449+0000 7fe91159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:31.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.449+0000 7fe91159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:29:31.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.449+0000 7fe91159a640 10 monclient: dump: 2024-09-17T18:29:31.467 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:31.467 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:31.467 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:31.467 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:31.467 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:31.467 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:31.467 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:31.467 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:31.467 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:31.467 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:31.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.449+0000 7fe918831640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:29:31.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.449+0000 7fe91159a640 1 -- 172.21.15.121:0/1349456196 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8fc02e070 con 0x7fe900004800 2024-09-17T18:29:31.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.449+0000 7fe91159a640 10 monclient: handle_config config(0 keys) 2024-09-17T18:29:31.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.449+0000 7fe918831640 10 monclient: get_monmap_and_config success 2024-09-17T18:29:31.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.449+0000 7fe918831640 4 set_mon_vals no callback set 2024-09-17T18:29:31.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.449+0000 7fe91159a640 1 -- 172.21.15.121:0/1349456196 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8fc02a910 con 0x7fe900004800 2024-09-17T18:29:31.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.449+0000 7fe91159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:31.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.449+0000 7fe91159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:29:31.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.449+0000 7fe91159a640 10 monclient: dump: 2024-09-17T18:29:31.468 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:31.468 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:31.468 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:31.469 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:31.469 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:31.469 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:31.469 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:31.469 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:31.469 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:31.469 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:31.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.450+0000 7fe918831640 10 monclient: shutdown 2024-09-17T18:29:31.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.450+0000 7fe918831640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:29:31.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.450+0000 7fe918831640 1 -- 172.21.15.121:0/1349456196 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe900004800 msgr2=0x7fe900004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:31.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.450+0000 7fe918831640 1 --2- 172.21.15.121:0/1349456196 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe900004800 0x7fe900004c80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc009900 tx=0x7fe900005830 comp rx=0 tx=0).stop 2024-09-17T18:29:31.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.450+0000 7fe918831640 1 -- 172.21.15.121:0/1349456196 shutdown_connections 2024-09-17T18:29:31.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.450+0000 7fe918831640 1 --2- 172.21.15.121:0/1349456196 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe900004800 0x7fe900004c80 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:31.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.450+0000 7fe918831640 1 -- 172.21.15.121:0/1349456196 >> 172.21.15.121:0/1349456196 conn(0x7fe9140a91e0 msgr2=0x7fe9140a95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:29:31.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.450+0000 7fe918831640 1 -- 172.21.15.121:0/1349456196 shutdown_connections 2024-09-17T18:29:31.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.450+0000 7fe918831640 1 -- 172.21.15.121:0/1349456196 wait complete. 2024-09-17T18:29:31.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.451+0000 7fe918831640 10 monclient: build_initial_monmap 2024-09-17T18:29:31.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.451+0000 7fe918831640 10 monclient: monmap: 2024-09-17T18:29:31.470 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:29:31.470 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:29:31.470 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:29:31.470 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:29:31.470 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:29:31.470 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:31.470 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:29:31.471 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:29:31.471 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:29:31.471 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:31.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.451+0000 7fe918831640 1 librados: starting msgr at 2024-09-17T18:29:31.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.451+0000 7fe918831640 1 librados: starting objecter 2024-09-17T18:29:31.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.451+0000 7fe918831640 1 Processor -- start 2024-09-17T18:29:31.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.451+0000 7fe918831640 1 -- start start 2024-09-17T18:29:31.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.451+0000 7fe918831640 1 librados: setting wanted keys 2024-09-17T18:29:31.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.451+0000 7fe918831640 1 librados: calling monclient init 2024-09-17T18:29:31.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.451+0000 7fe918831640 10 monclient: init 2024-09-17T18:29:31.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe918831640 10 monclient: _reopen_session rank -1 2024-09-17T18:29:31.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe918831640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T18:29:31.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe918831640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe900004800 0x7fe9140a03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:31.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe918831640 10 monclient(hunting): picked mon.noname-c con 0x7fe900004800 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:29:31.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe918831640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9140a0910 0x7fe9140a0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:31.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe918831640 10 monclient(hunting): picked mon.noname-b con 0x7fe9140a0910 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:29:31.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe918831640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9141cd9c0 0x7fe9141cde70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:31.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe918831640 10 monclient(hunting): picked mon.noname-a con 0x7fe9141cd9c0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:29:31.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe918831640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:31.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe918831640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe914070b60 con 0x7fe9141cd9c0 2024-09-17T18:29:31.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe918831640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:31.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe918831640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe9140709e0 con 0x7fe9140a0910 2024-09-17T18:29:31.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe918831640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:31.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe918831640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe914138a90 con 0x7fe900004800 2024-09-17T18:29:31.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe918831640 10 monclient(hunting): _renew_subs 2024-09-17T18:29:31.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe918831640 10 monclient(hunting): authenticate will time out at 1375.168098s 2024-09-17T18:29:31.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe91259c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe900004800 0x7fe9140a03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:31.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe912d9d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9141cd9c0 0x7fe9141cde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:31.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe91259c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe900004800 0x7fe9140a03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:58458/0 (socket says 172.21.15.121:58458) 2024-09-17T18:29:31.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe91259c640 1 -- 172.21.15.121:0/3124600241 learned_addr learned my addr 172.21.15.121:0/3124600241 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:29:31.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe91259c640 10 monclient(hunting): get_auth_request con 0x7fe900004800 auth_method 0 2024-09-17T18:29:31.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe91259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:31.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe91259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:31.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe91259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:31.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe912d9d640 10 monclient(hunting): get_auth_request con 0x7fe9141cd9c0 auth_method 0 2024-09-17T18:29:31.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe912d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:31.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe912d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:31.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe912d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:31.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe911d9b640 1 -- 172.21.15.121:0/3124600241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9140a0910 msgr2=0x7fe9140a0d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.173:3300/0 2024-09-17T18:29:31.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe911d9b640 1 --2- 172.21.15.121:0/3124600241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9140a0910 0x7fe9140a0d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:29:31.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe91259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:31.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe91259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:31.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe91259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:31.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe912d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:31.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe912d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:31.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe912d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:31.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe912d9d640 10 monclient(hunting): handle_auth_done global_id 44595 payload 824 2024-09-17T18:29:31.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe912d9d640 1 -- 172.21.15.121:0/3124600241 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe900004800 msgr2=0x7fe9140a03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:31.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe912d9d640 1 --2- 172.21.15.121:0/3124600241 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe900004800 0x7fe9140a03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:31.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe912d9d640 1 -- 172.21.15.121:0/3124600241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9140a0910 msgr2=0x7fe9140a0d70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:29:31.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe912d9d640 1 --2- 172.21.15.121:0/3124600241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9140a0910 0x7fe9140a0d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:31.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe912d9d640 10 monclient: _finish_hunting 0 2024-09-17T18:29:31.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe912d9d640 1 monclient: found mon.noname-a 2024-09-17T18:29:31.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe912d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:29:31.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe912d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:29:31.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe912d9d640 1 -- 172.21.15.121:0/3124600241 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9140a2a60 con 0x7fe9141cd9c0 2024-09-17T18:29:31.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.452+0000 7fe91259c640 1 --2- 172.21.15.121:0/3124600241 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe900004800 0x7fe9140a03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:29:31.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.454+0000 7fe912d9d640 1 -- 172.21.15.121:0/3124600241 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9141cd9c0 msgr2=0x7fe9141cde70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:31.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.454+0000 7fe912d9d640 1 --2- 172.21.15.121:0/3124600241 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9141cd9c0 0x7fe9141cde70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe9140a5ac0 tx=0x7fe9040074f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:31.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.454+0000 7fe90affd640 10 client.?.objecter ms_handle_connect 0x7fe9141cd9c0 2024-09-17T18:29:31.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.454+0000 7fe90affd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:29:31.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.454+0000 7fe90affd640 1 -- 172.21.15.121:0/3124600241 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe90400f910 con 0x7fe9141cd9c0 2024-09-17T18:29:31.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.454+0000 7fe90affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:31.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.454+0000 7fe90affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:29:31.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.454+0000 7fe90affd640 10 monclient: dump: 2024-09-17T18:29:31.476 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:31.476 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:31.476 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:31.476 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:31.476 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:31.476 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:31.476 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:31.476 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:31.476 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:31.476 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:31.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.454+0000 7fe90affd640 10 monclient: _finish_auth 0 2024-09-17T18:29:31.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.454+0000 7fe90affd640 10 monclient: _check_auth_tickets 2024-09-17T18:29:31.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.454+0000 7fe90affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:29:31.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.454+0000 7fe90affd640 1 -- 172.21.15.121:0/3124600241 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe904018070 con 0x7fe9141cd9c0 2024-09-17T18:29:31.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.454+0000 7fe918831640 5 monclient: authenticate success, global_id 44595 2024-09-17T18:29:31.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.454+0000 7fe918831640 10 monclient: _renew_subs 2024-09-17T18:29:31.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.454+0000 7fe918831640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:29:31.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.454+0000 7fe918831640 1 -- 172.21.15.121:0/3124600241 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9140a2d40 con 0x7fe9141cd9c0 2024-09-17T18:29:31.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.454+0000 7fe90affd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:29:31.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.454+0000 7fe90affd640 1 -- 172.21.15.121:0/3124600241 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe90400fd10 con 0x7fe9141cd9c0 2024-09-17T18:29:31.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.454+0000 7fe90affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:31.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.454+0000 7fe90affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:29:31.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.454+0000 7fe90affd640 10 monclient: dump: 2024-09-17T18:29:31.478 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:31.478 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:31.478 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:31.478 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:31.478 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:31.478 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:31.478 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:31.478 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:31.478 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:31.478 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:31.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.455+0000 7fe918831640 10 client.44595.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:29:31.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.455+0000 7fe918831640 10 monclient: _renew_subs 2024-09-17T18:29:31.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.455+0000 7fe918831640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:29:31.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.454+0000 7fe910d99640 4 set_mon_vals no callback set 2024-09-17T18:29:31.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.455+0000 7fe918831640 1 -- 172.21.15.121:0/3124600241 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9140a87b0 con 0x7fe9141cd9c0 2024-09-17T18:29:31.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.455+0000 7fe918831640 1 librados: init done 2024-09-17T18:29:31.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.455+0000 7fe90affd640 1 -- 172.21.15.121:0/3124600241 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fe90401d460 con 0x7fe9141cd9c0 2024-09-17T18:29:31.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.455+0000 7fe918831640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:29:31.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.455+0000 7fe918831640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:29:31.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.455+0000 7fe918831640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:29:31.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.455+0000 7fe918831640 1 -- 172.21.15.121:0/3124600241 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9140a1980 con 0x7fe9141cd9c0 2024-09-17T18:29:31.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.456+0000 7fe90affd640 1 --2- 172.21.15.121:0/3124600241 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe8f4040120 0x7fe8f40425e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:31.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.456+0000 7fe90affd640 1 -- 172.21.15.121:0/3124600241 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(412..412 src has 247..412) ==== 5345+0+0 (secure 0 0 0) 0x7fe904053450 con 0x7fe9141cd9c0 2024-09-17T18:29:31.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.456+0000 7fe90affd640 10 client.44595.objecter ms_dispatch 0x7fe914000b90 osd_map(412..412 src has 247..412) 2024-09-17T18:29:31.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.456+0000 7fe90affd640 3 client.44595.objecter handle_osd_map got epochs [412,412] > 0 2024-09-17T18:29:31.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.456+0000 7fe90affd640 3 client.44595.objecter handle_osd_map decoding full epoch 412 2024-09-17T18:29:31.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.456+0000 7fe90affd640 20 client.44595.objecter dump_active .. 0 homeless 2024-09-17T18:29:31.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.456+0000 7fe91259c640 1 --2- 172.21.15.121:0/3124600241 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe8f4040120 0x7fe8f40425e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:31.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.456+0000 7fe91259c640 10 monclient: get_auth_request con 0x7fe8f4040120 auth_method 0 2024-09-17T18:29:31.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.459+0000 7fe91259c640 1 --2- 172.21.15.121:0/3124600241 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe8f4040120 0x7fe8f40425e0 secure :-1 s=READY pgs=4018 cs=0 l=1 rev1=1 crypto rx=0x7fe90c0094b0 tx=0x7fe90c009400 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:31.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.459+0000 7fe90affd640 10 client.44595.objecter ms_handle_connect 0x7fe8f4040120 2024-09-17T18:29:31.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.460+0000 7fe90affd640 1 -- 172.21.15.121:0/3124600241 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe904051c10 con 0x7fe9141cd9c0 2024-09-17T18:29:31.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.460+0000 7fe90affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:29:31.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.460+0000 7fe90affd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:29:31.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.706+0000 7fe918831640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:29:31.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.706+0000 7fe918831640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:29:31.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.706+0000 7fe918831640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:29:31.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.706+0000 7fe918831640 1 -- 172.21.15.121:0/3124600241 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe914138e70 con 0x7fe9141cd9c0 2024-09-17T18:29:31.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.706+0000 7fe90affd640 1 -- 172.21.15.121:0/3124600241 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fe904012d40 con 0x7fe9141cd9c0 2024-09-17T18:29:31.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.706+0000 7fe90affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:29:31.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.706+0000 7fe90affd640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:29:31.709 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:29:31.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.709+0000 7fe918831640 10 librados: watch_flush enter 2024-09-17T18:29:31.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.709+0000 7fe918831640 10 librados: watch_flush exit 2024-09-17T18:29:31.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.709+0000 7fe918831640 20 client.44595.objecter shutdown clearing up homeless session... 2024-09-17T18:29:31.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.709+0000 7fe918831640 10 client.44595.objecter successfully canceled tick 2024-09-17T18:29:31.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.709+0000 7fe918831640 1 -- 172.21.15.121:0/3124600241 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe8f4040120 msgr2=0x7fe8f40425e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:31.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.709+0000 7fe918831640 1 --2- 172.21.15.121:0/3124600241 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe8f4040120 0x7fe8f40425e0 secure :-1 s=READY pgs=4018 cs=0 l=1 rev1=1 crypto rx=0x7fe90c0094b0 tx=0x7fe90c009400 comp rx=0 tx=0).stop 2024-09-17T18:29:31.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.709+0000 7fe918831640 10 monclient: shutdown 2024-09-17T18:29:31.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.709+0000 7fe918831640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:29:31.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.709+0000 7fe918831640 1 -- 172.21.15.121:0/3124600241 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9141cd9c0 msgr2=0x7fe9141cde70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:31.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.709+0000 7fe918831640 1 --2- 172.21.15.121:0/3124600241 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9141cd9c0 0x7fe9141cde70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe9140a5ac0 tx=0x7fe9040074f0 comp rx=0 tx=0).stop 2024-09-17T18:29:31.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.709+0000 7fe918831640 1 -- 172.21.15.121:0/3124600241 shutdown_connections 2024-09-17T18:29:31.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.709+0000 7fe918831640 1 --2- 172.21.15.121:0/3124600241 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe8f4040120 0x7fe8f40425e0 unknown :-1 s=CLOSED pgs=4018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:31.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.709+0000 7fe918831640 1 --2- 172.21.15.121:0/3124600241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9140a0910 0x7fe9140a0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:31.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.709+0000 7fe918831640 1 --2- 172.21.15.121:0/3124600241 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9141cd9c0 0x7fe9141cde70 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:31.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.709+0000 7fe918831640 1 --2- 172.21.15.121:0/3124600241 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe900004800 0x7fe9140a03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:31.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.709+0000 7fe918831640 1 -- 172.21.15.121:0/3124600241 >> 172.21.15.121:0/3124600241 conn(0x7fe9140a91e0 msgr2=0x7fe9140a6fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:29:31.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.709+0000 7fe918831640 1 -- 172.21.15.121:0/3124600241 shutdown_connections 2024-09-17T18:29:31.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.709+0000 7fe918831640 1 -- 172.21.15.121:0/3124600241 wait complete. 2024-09-17T18:29:31.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:31.709+0000 7fe918831640 1 librados: shutdown 2024-09-17T18:29:34.722 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:29:34.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.829+0000 7f096287f640 10 monclient: get_monmap_and_config 2024-09-17T18:29:34.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.829+0000 7f096287f640 10 monclient: build_initial_monmap 2024-09-17T18:29:34.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.829+0000 7f096287f640 10 monclient: monmap: 2024-09-17T18:29:34.831 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:29:34.831 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:34.831 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:29:34.829938+0000 2024-09-17T18:29:34.831 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:29:34.829938+0000 2024-09-17T18:29:34.831 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:29:34.831 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:34.831 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:29:34.832 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:29:34.832 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:29:34.832 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:29:34.832 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:29:34.832 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:34.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.830+0000 7f096287f640 1 Processor -- start 2024-09-17T18:29:34.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.830+0000 7f096287f640 1 -- start start 2024-09-17T18:29:34.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.830+0000 7f096287f640 10 monclient: init 2024-09-17T18:29:34.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.830+0000 7f096287f640 10 monclient: _reopen_session rank -1 2024-09-17T18:29:34.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.830+0000 7f096287f640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-17T18:29:34.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.830+0000 7f096287f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f095c073c70 0x7f095c070140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:34.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.830+0000 7f096287f640 10 monclient(hunting): picked mon.noname-b con 0x7f095c073c70 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:29:34.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.830+0000 7f096287f640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f095c074050 0x7f095c1352a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:34.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.830+0000 7f096287f640 10 monclient(hunting): picked mon.noname-c con 0x7f095c074050 addr v2:172.21.15.173:3300/0 2024-09-17T18:29:34.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.830+0000 7f096287f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f095c074490 addr v1:172.21.15.173:6789/0 2024-09-17T18:29:34.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.830+0000 7f096287f640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f095c0712c0 con 0x7f095c074490 2024-09-17T18:29:34.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.830+0000 7f096287f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:34.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.830+0000 7f096287f640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f095c070f00 con 0x7f095c074050 2024-09-17T18:29:34.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.830+0000 7f096287f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:34.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.830+0000 7f096287f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f095c071ea0 con 0x7f095c073c70 2024-09-17T18:29:34.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.830+0000 7f096287f640 10 monclient(hunting): _renew_subs 2024-09-17T18:29:34.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.830+0000 7f096287f640 10 monclient(hunting): authenticate will time out at 1378.546768s 2024-09-17T18:29:34.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.830+0000 7f095bfff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f095c073c70 0x7f095c070140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:34.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.830+0000 7f095b7fe640 1 -- >> v2:172.21.15.173:3300/0 conn(0x7f095c074050 msgr2=0x7f095c1352a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.173:3300/0 2024-09-17T18:29:34.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.830+0000 7f095bfff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f095c073c70 0x7f095c070140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:58468/0 (socket says 172.21.15.121:58468) 2024-09-17T18:29:34.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.830+0000 7f095bfff640 1 -- 172.21.15.121:0/3183074620 learned_addr learned my addr 172.21.15.121:0/3183074620 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:29:34.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.830+0000 7f095bfff640 10 monclient(hunting): get_auth_request con 0x7f095c073c70 auth_method 0 2024-09-17T18:29:34.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.830+0000 7f095bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:34.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.830+0000 7f095bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:34.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.830+0000 7f095bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:34.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.830+0000 7f095b7fe640 1 --2- 172.21.15.121:0/3183074620 >> v2:172.21.15.173:3300/0 conn(0x7f095c074050 0x7f095c1352a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:29:34.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.830+0000 7f0960e1c640 1 -- 172.21.15.121:0/3183074620 >> v1:172.21.15.173:6789/0 conn(0x7f095c074490 legacy=0x7f095c1357e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.173:6789/0 2024-09-17T18:29:34.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.830+0000 7f095affd640 1 -- 172.21.15.121:0/3183074620 DispatchQueue::entry inject delay of 0.002000 2024-09-17T18:29:34.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.830+0000 7f095bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:34.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.830+0000 7f095bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:34.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.830+0000 7f095bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:34.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.831+0000 7f095bfff640 10 monclient(hunting): handle_auth_done global_id 64235 payload 274 2024-09-17T18:29:34.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.831+0000 7f095bfff640 1 -- 172.21.15.121:0/3183074620 >> v2:172.21.15.173:3300/0 conn(0x7f095c074050 msgr2=0x7f095c1352a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:29:34.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.831+0000 7f095bfff640 1 --2- 172.21.15.121:0/3183074620 >> v2:172.21.15.173:3300/0 conn(0x7f095c074050 0x7f095c1352a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:34.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.831+0000 7f095bfff640 1 -- 172.21.15.121:0/3183074620 >> v1:172.21.15.173:6789/0 conn(0x7f095c074490 legacy=0x7f095c1357e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:29:34.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.831+0000 7f095bfff640 10 monclient: _finish_hunting 0 2024-09-17T18:29:34.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.831+0000 7f095bfff640 1 monclient: found mon.noname-b 2024-09-17T18:29:34.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.831+0000 7f095bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:29:34.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.831+0000 7f095bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:29:34.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.831+0000 7f095bfff640 1 -- 172.21.15.121:0/3183074620 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f095c072180 con 0x7f095c073c70 2024-09-17T18:29:34.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.833+0000 7f095bfff640 1 -- 172.21.15.121:0/3183074620 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f095c073c70 msgr2=0x7f095c070140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:34.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.833+0000 7f095bfff640 1 --2- 172.21.15.121:0/3183074620 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f095c073c70 0x7f095c070140 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f094c009870 tx=0x7f094c02a0d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebb36f0073af1b8d server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:34.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.834+0000 7f095affd640 1 -- 172.21.15.121:0/3183074620 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f094c037070 con 0x7f095c073c70 2024-09-17T18:29:34.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.834+0000 7f095affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:34.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.834+0000 7f095affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:29:34.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.834+0000 7f095affd640 10 monclient: dump: 2024-09-17T18:29:34.843 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:34.843 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:34.843 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:34.844 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:34.844 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:34.844 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:34.844 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:34.844 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:34.844 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:34.844 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:34.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.834+0000 7f095affd640 10 monclient: _finish_auth 0 2024-09-17T18:29:34.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.834+0000 7f095affd640 10 monclient: _check_auth_tickets 2024-09-17T18:29:34.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.834+0000 7f095affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:29:34.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.834+0000 7f096287f640 5 monclient: authenticate success, global_id 64235 2024-09-17T18:29:34.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.834+0000 7f095affd640 1 -- 172.21.15.121:0/3183074620 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f094c032640 con 0x7f095c073c70 2024-09-17T18:29:34.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.834+0000 7f096287f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:29:34.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.834+0000 7f095affd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:29:34.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.834+0000 7f096287f640 10 monclient: get_monmap_and_config success 2024-09-17T18:29:34.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.834+0000 7f095affd640 1 -- 172.21.15.121:0/3183074620 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f094c032a10 con 0x7f095c073c70 2024-09-17T18:29:34.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.834+0000 7f096287f640 4 set_mon_vals no callback set 2024-09-17T18:29:34.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.834+0000 7f095affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:34.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.834+0000 7f095affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:29:34.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.834+0000 7f095affd640 10 monclient: dump: 2024-09-17T18:29:34.845 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:34.846 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:34.846 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:34.846 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:34.846 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:34.846 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:34.846 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:34.846 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:34.846 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:34.846 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:34.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.834+0000 7f096287f640 10 monclient: shutdown 2024-09-17T18:29:34.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.834+0000 7f096287f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:29:34.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.834+0000 7f096287f640 1 -- 172.21.15.121:0/3183074620 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f095c073c70 msgr2=0x7f095c070140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:34.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.834+0000 7f096287f640 1 --2- 172.21.15.121:0/3183074620 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f095c073c70 0x7f095c070140 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f094c009870 tx=0x7f094c02a0d0 comp rx=0 tx=0).stop 2024-09-17T18:29:34.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.834+0000 7f096287f640 1 -- 172.21.15.121:0/3183074620 shutdown_connections 2024-09-17T18:29:34.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.834+0000 7f096287f640 1 --2- 172.21.15.121:0/3183074620 >> v2:172.21.15.173:3300/0 conn(0x7f095c074050 0x7f095c1352a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:34.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.834+0000 7f096287f640 1 --2- 172.21.15.121:0/3183074620 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f095c073c70 0x7f095c070140 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:34.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.834+0000 7f096287f640 1 -- 172.21.15.121:0/3183074620 >> 172.21.15.121:0/3183074620 conn(0x7f095c076f00 msgr2=0x7f095c077310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:29:34.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.834+0000 7f096287f640 1 -- 172.21.15.121:0/3183074620 shutdown_connections 2024-09-17T18:29:34.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.834+0000 7f096287f640 1 -- 172.21.15.121:0/3183074620 wait complete. 2024-09-17T18:29:34.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.835+0000 7f096287f640 10 monclient: build_initial_monmap 2024-09-17T18:29:34.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.835+0000 7f096287f640 10 monclient: monmap: 2024-09-17T18:29:34.847 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:29:34.848 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:29:34.848 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:29:34.848 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:29:34.848 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:29:34.848 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:34.848 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:29:34.848 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:29:34.848 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:29:34.848 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:34.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.835+0000 7f096287f640 1 librados: starting msgr at 2024-09-17T18:29:34.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.835+0000 7f096287f640 1 librados: starting objecter 2024-09-17T18:29:34.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.835+0000 7f096287f640 1 Processor -- start 2024-09-17T18:29:34.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.835+0000 7f096287f640 1 -- start start 2024-09-17T18:29:34.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.835+0000 7f096287f640 1 librados: setting wanted keys 2024-09-17T18:29:34.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.835+0000 7f096287f640 1 librados: calling monclient init 2024-09-17T18:29:34.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.835+0000 7f096287f640 10 monclient: init 2024-09-17T18:29:34.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f096287f640 10 monclient: _reopen_session rank -1 2024-09-17T18:29:34.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f096287f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:29:34.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f096287f640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f095c073c70 0x7f095c134440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:34.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f096287f640 10 monclient(hunting): picked mon.noname-a con 0x7f095c073c70 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:29:34.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f096287f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f095c074050 0x7f095c134980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:34.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f096287f640 10 monclient(hunting): picked mon.noname-b con 0x7f095c074050 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:29:34.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f096287f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f095c074490 0x7f095c121010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:34.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f096287f640 10 monclient(hunting): picked mon.noname-c con 0x7f095c074490 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:29:34.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f096287f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:34.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f096287f640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f095c06d6a0 con 0x7f095c073c70 2024-09-17T18:29:34.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f096287f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:34.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f096287f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f095c06d820 con 0x7f095c074050 2024-09-17T18:29:34.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f096287f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:34.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f096287f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f095c135ef0 con 0x7f095c074490 2024-09-17T18:29:34.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f096287f640 10 monclient(hunting): _renew_subs 2024-09-17T18:29:34.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f096287f640 10 monclient(hunting): authenticate will time out at 1378.552465s 2024-09-17T18:29:34.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f095bfff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f095c073c70 0x7f095c134440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:34.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f0960e1c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f095c074490 0x7f095c121010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:34.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f095bfff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f095c073c70 0x7f095c134440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34386/0 (socket says 172.21.15.121:34386) 2024-09-17T18:29:34.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f0960e1c640 10 monclient(hunting): get_auth_request con 0x7f095c074490 auth_method 0 2024-09-17T18:29:34.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f095bfff640 1 -- 172.21.15.121:0/1179219764 learned_addr learned my addr 172.21.15.121:0/1179219764 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:29:34.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f0960e1c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:34.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f0960e1c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:34.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f0960e1c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:34.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f095bfff640 10 monclient(hunting): get_auth_request con 0x7f095c073c70 auth_method 0 2024-09-17T18:29:34.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f095bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:34.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f095bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:34.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f095bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:34.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f095b7fe640 1 -- 172.21.15.121:0/1179219764 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f095c074050 msgr2=0x7f095c134980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.173:3300/0 2024-09-17T18:29:34.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f095b7fe640 1 --2- 172.21.15.121:0/1179219764 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f095c074050 0x7f095c134980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:29:34.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f0960e1c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:34.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f0960e1c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:34.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f0960e1c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:34.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f095bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:34.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f095bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:34.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f095bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:34.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f0960e1c640 10 monclient(hunting): handle_auth_done global_id 64238 payload 824 2024-09-17T18:29:34.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f0960e1c640 1 -- 172.21.15.121:0/1179219764 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f095c074050 msgr2=0x7f095c134980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:29:34.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f0960e1c640 1 --2- 172.21.15.121:0/1179219764 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f095c074050 0x7f095c134980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:34.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f0960e1c640 1 -- 172.21.15.121:0/1179219764 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f095c073c70 msgr2=0x7f095c134440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:34.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f0960e1c640 1 --2- 172.21.15.121:0/1179219764 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f095c073c70 0x7f095c134440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:34.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f0960e1c640 10 monclient: _finish_hunting 0 2024-09-17T18:29:34.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f0960e1c640 1 monclient: found mon.noname-c 2024-09-17T18:29:34.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f0960e1c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:29:34.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f0960e1c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:29:34.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f0960e1c640 1 -- 172.21.15.121:0/1179219764 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f095c074c80 con 0x7f095c074490 2024-09-17T18:29:34.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.836+0000 7f095bfff640 1 --2- 172.21.15.121:0/1179219764 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f095c073c70 0x7f095c134440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:29:34.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.839+0000 7f0960e1c640 1 -- 172.21.15.121:0/1179219764 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f095c074490 msgr2=0x7f095c121010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:34.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.839+0000 7f0960e1c640 1 --2- 172.21.15.121:0/1179219764 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f095c074490 0x7f095c121010 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0954003cb0 tx=0x7f0954009ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:34.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.839+0000 7f0958ff9640 10 client.?.objecter ms_handle_connect 0x7f095c074490 2024-09-17T18:29:34.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.839+0000 7f0958ff9640 10 client.?.objecter resend_mon_ops 2024-09-17T18:29:34.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.839+0000 7f0958ff9640 1 -- 172.21.15.121:0/1179219764 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0954015470 con 0x7f095c074490 2024-09-17T18:29:34.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.839+0000 7f0958ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:34.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.839+0000 7f0958ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:29:34.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.839+0000 7f0958ff9640 10 monclient: dump: 2024-09-17T18:29:34.854 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:34.854 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:34.854 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:34.855 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:34.855 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:34.855 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:34.855 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:34.855 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:34.855 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:34.855 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:34.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.839+0000 7f0958ff9640 10 monclient: _finish_auth 0 2024-09-17T18:29:34.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.839+0000 7f0958ff9640 10 monclient: _check_auth_tickets 2024-09-17T18:29:34.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.839+0000 7f0958ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:29:34.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.839+0000 7f0958ff9640 1 -- 172.21.15.121:0/1179219764 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0954010070 con 0x7f095c074490 2024-09-17T18:29:34.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.839+0000 7f096287f640 5 monclient: authenticate success, global_id 64238 2024-09-17T18:29:34.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.839+0000 7f096287f640 10 monclient: _renew_subs 2024-09-17T18:29:34.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.839+0000 7f096287f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:29:34.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.839+0000 7f096287f640 1 -- 172.21.15.121:0/1179219764 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f095c124870 con 0x7f095c074490 2024-09-17T18:29:34.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.839+0000 7f0958ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T18:29:34.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.839+0000 7f0958ff9640 1 -- 172.21.15.121:0/1179219764 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0954015850 con 0x7f095c074490 2024-09-17T18:29:34.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.839+0000 7f0958ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:34.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.839+0000 7f0958ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:29:34.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.839+0000 7f0958ff9640 10 monclient: dump: 2024-09-17T18:29:34.856 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:34.856 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:34.857 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:34.857 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:34.857 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:34.857 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:34.857 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:34.857 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:34.857 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:34.857 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:34.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.839+0000 7f096287f640 10 client.64238.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:29:34.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.839+0000 7f095a7fc640 4 set_mon_vals no callback set 2024-09-17T18:29:34.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.839+0000 7f096287f640 10 monclient: _renew_subs 2024-09-17T18:29:34.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.839+0000 7f096287f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:29:34.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.839+0000 7f096287f640 1 -- 172.21.15.121:0/1179219764 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f095c134ec0 con 0x7f095c074490 2024-09-17T18:29:34.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.839+0000 7f096287f640 1 librados: init done 2024-09-17T18:29:34.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.840+0000 7f096287f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:29:34.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.840+0000 7f096287f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:29:34.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.840+0000 7f096287f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:29:34.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.840+0000 7f0958ff9640 1 -- 172.21.15.121:0/1179219764 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f0954015af0 con 0x7f095c074490 2024-09-17T18:29:34.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.840+0000 7f096287f640 1 -- 172.21.15.121:0/1179219764 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f095c070140 con 0x7f095c074490 2024-09-17T18:29:34.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.840+0000 7f0958ff9640 1 --2- 172.21.15.121:0/1179219764 >> v2:172.21.15.173:6812/2423786037 conn(0x7f09480401a0 0x7f0948042660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:34.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.840+0000 7f0958ff9640 1 -- 172.21.15.121:0/1179219764 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(412..412 src has 247..412) ==== 5345+0+0 (secure 0 0 0) 0x7f0954053db0 con 0x7f095c074490 2024-09-17T18:29:34.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.840+0000 7f0958ff9640 10 client.64238.objecter ms_dispatch 0x7f095c000b90 osd_map(412..412 src has 247..412) 2024-09-17T18:29:34.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.840+0000 7f0958ff9640 3 client.64238.objecter handle_osd_map got epochs [412,412] > 0 2024-09-17T18:29:34.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.840+0000 7f0958ff9640 3 client.64238.objecter handle_osd_map decoding full epoch 412 2024-09-17T18:29:34.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.840+0000 7f0958ff9640 20 client.64238.objecter dump_active .. 0 homeless 2024-09-17T18:29:34.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.840+0000 7f095bfff640 1 --2- 172.21.15.121:0/1179219764 >> v2:172.21.15.173:6812/2423786037 conn(0x7f09480401a0 0x7f0948042660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:34.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.841+0000 7f095bfff640 10 monclient: get_auth_request con 0x7f09480401a0 auth_method 0 2024-09-17T18:29:34.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.843+0000 7f095bfff640 1 --2- 172.21.15.121:0/1179219764 >> v2:172.21.15.173:6812/2423786037 conn(0x7f09480401a0 0x7f0948042660 secure :-1 s=READY pgs=4019 cs=0 l=1 rev1=1 crypto rx=0x7f094c000c00 tx=0x7f094c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:34.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.843+0000 7f0958ff9640 10 client.64238.objecter ms_handle_connect 0x7f09480401a0 2024-09-17T18:29:34.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.845+0000 7f0958ff9640 1 -- 172.21.15.121:0/1179219764 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f095401a060 con 0x7f095c074490 2024-09-17T18:29:34.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.845+0000 7f0958ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:29:34.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:34.845+0000 7f0958ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:29:35.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:35.085+0000 7f096287f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:29:35.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:35.085+0000 7f096287f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:29:35.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:35.085+0000 7f096287f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:29:35.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:35.085+0000 7f096287f640 1 -- 172.21.15.121:0/1179219764 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f095c122ad0 con 0x7f095c074490 2024-09-17T18:29:35.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:35.085+0000 7f0958ff9640 1 -- 172.21.15.121:0/1179219764 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f095c074c80 con 0x7f095c074490 2024-09-17T18:29:35.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:35.085+0000 7f0958ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:29:35.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:35.085+0000 7f0958ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:29:35.088 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:29:35.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:35.087+0000 7f096287f640 10 librados: watch_flush enter 2024-09-17T18:29:35.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:35.087+0000 7f096287f640 10 librados: watch_flush exit 2024-09-17T18:29:35.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:35.087+0000 7f096287f640 20 client.64238.objecter shutdown clearing up homeless session... 2024-09-17T18:29:35.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:35.087+0000 7f096287f640 10 client.64238.objecter successfully canceled tick 2024-09-17T18:29:35.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:35.087+0000 7f096287f640 1 -- 172.21.15.121:0/1179219764 >> v2:172.21.15.173:6812/2423786037 conn(0x7f09480401a0 msgr2=0x7f0948042660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:35.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:35.087+0000 7f096287f640 1 --2- 172.21.15.121:0/1179219764 >> v2:172.21.15.173:6812/2423786037 conn(0x7f09480401a0 0x7f0948042660 secure :-1 s=READY pgs=4019 cs=0 l=1 rev1=1 crypto rx=0x7f094c000c00 tx=0x7f094c02e000 comp rx=0 tx=0).stop 2024-09-17T18:29:35.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:35.087+0000 7f096287f640 10 monclient: shutdown 2024-09-17T18:29:35.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:35.087+0000 7f096287f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:29:35.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:35.087+0000 7f096287f640 1 -- 172.21.15.121:0/1179219764 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f095c074490 msgr2=0x7f095c121010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:35.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:35.087+0000 7f096287f640 1 --2- 172.21.15.121:0/1179219764 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f095c074490 0x7f095c121010 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0954003cb0 tx=0x7f0954009ef0 comp rx=0 tx=0).stop 2024-09-17T18:29:35.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:35.087+0000 7f096287f640 1 -- 172.21.15.121:0/1179219764 shutdown_connections 2024-09-17T18:29:35.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:35.087+0000 7f096287f640 1 --2- 172.21.15.121:0/1179219764 >> v2:172.21.15.173:6812/2423786037 conn(0x7f09480401a0 0x7f0948042660 unknown :-1 s=CLOSED pgs=4019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:35.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:35.087+0000 7f096287f640 1 --2- 172.21.15.121:0/1179219764 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f095c074050 0x7f095c134980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:35.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:35.087+0000 7f096287f640 1 --2- 172.21.15.121:0/1179219764 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f095c074490 0x7f095c121010 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:35.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:35.087+0000 7f096287f640 1 --2- 172.21.15.121:0/1179219764 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f095c073c70 0x7f095c134440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:35.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:35.087+0000 7f096287f640 1 -- 172.21.15.121:0/1179219764 >> 172.21.15.121:0/1179219764 conn(0x7f095c076f00 msgr2=0x7f095c0a38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:29:35.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:35.087+0000 7f096287f640 1 -- 172.21.15.121:0/1179219764 shutdown_connections 2024-09-17T18:29:35.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:35.087+0000 7f096287f640 1 -- 172.21.15.121:0/1179219764 wait complete. 2024-09-17T18:29:35.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:35.087+0000 7f096287f640 1 librados: shutdown 2024-09-17T18:29:38.101 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:29:38.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.207+0000 7f0a32a31640 10 monclient: get_monmap_and_config 2024-09-17T18:29:38.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.207+0000 7f0a32a31640 10 monclient: build_initial_monmap 2024-09-17T18:29:38.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.207+0000 7f0a32a31640 10 monclient: monmap: 2024-09-17T18:29:38.209 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:29:38.209 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:38.209 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:29:38.208216+0000 2024-09-17T18:29:38.210 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:29:38.208216+0000 2024-09-17T18:29:38.210 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:29:38.210 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:38.210 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:29:38.210 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:29:38.210 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:29:38.210 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:29:38.210 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:29:38.210 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:38.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.208+0000 7f0a32a31640 1 Processor -- start 2024-09-17T18:29:38.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.208+0000 7f0a32a31640 1 -- start start 2024-09-17T18:29:38.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.208+0000 7f0a32a31640 10 monclient: init 2024-09-17T18:29:38.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.208+0000 7f0a32a31640 10 monclient: _reopen_session rank -1 2024-09-17T18:29:38.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.208+0000 7f0a32a31640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-17T18:29:38.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.208+0000 7f0a32a31640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0a2c0a33f0 addr v1:172.21.15.173:6789/0 2024-09-17T18:29:38.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.208+0000 7f0a32a31640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0a2c09da70 0x7f0a2c09ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:38.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.208+0000 7f0a32a31640 10 monclient(hunting): picked mon.noname-a con 0x7f0a2c09da70 addr v2:172.21.15.121:3300/0 2024-09-17T18:29:38.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.208+0000 7f0a32a31640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0a2c13a850 0x7f0a2c13cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:38.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.208+0000 7f0a32a31640 10 monclient(hunting): picked mon.noname-b con 0x7f0a2c13a850 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:29:38.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.209+0000 7f0a32a31640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a2c074b40 con 0x7f0a2c0a33f0 2024-09-17T18:29:38.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.209+0000 7f0a32a31640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:38.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.209+0000 7f0a32a31640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f0a2c074550 con 0x7f0a2c09da70 2024-09-17T18:29:38.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.209+0000 7f0a32a31640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:38.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.209+0000 7f0a32a31640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0a2c0a37d0 con 0x7f0a2c13a850 2024-09-17T18:29:38.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.209+0000 7f0a32a31640 10 monclient(hunting): _renew_subs 2024-09-17T18:29:38.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.209+0000 7f0a32a31640 10 monclient(hunting): authenticate will time out at 1381.925056s 2024-09-17T18:29:38.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.209+0000 7f0a2bfff640 1 -- >> v1:172.21.15.173:6789/0 conn(0x7f0a2c0a33f0 legacy=0x7f0a2c09e550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.173:6789/0 2024-09-17T18:29:38.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.209+0000 7f0a30fce640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0a2c13a850 0x7f0a2c13cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:38.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.209+0000 7f0a2b7fe640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0a2c09da70 0x7f0a2c09ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:38.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.209+0000 7f0a2affd640 1 -- DispatchQueue::entry inject delay of 0.002000 2024-09-17T18:29:38.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.209+0000 7f0a30fce640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0a2c13a850 0x7f0a2c13cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:58480/0 (socket says 172.21.15.121:58480) 2024-09-17T18:29:38.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.209+0000 7f0a30fce640 1 -- 172.21.15.121:0/1168125404 learned_addr learned my addr 172.21.15.121:0/1168125404 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:29:38.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.209+0000 7f0a2b7fe640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0a2c09da70 0x7f0a2c09ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34396/0 (socket says 172.21.15.121:34396) 2024-09-17T18:29:38.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.209+0000 7f0a30fce640 10 monclient(hunting): get_auth_request con 0x7f0a2c13a850 auth_method 0 2024-09-17T18:29:38.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.209+0000 7f0a30fce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:38.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.209+0000 7f0a30fce640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:38.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.209+0000 7f0a30fce640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:38.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.209+0000 7f0a2b7fe640 10 monclient(hunting): get_auth_request con 0x7f0a2c09da70 auth_method 0 2024-09-17T18:29:38.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.209+0000 7f0a2b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:38.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.209+0000 7f0a2b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:38.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.209+0000 7f0a2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:38.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.209+0000 7f0a2b7fe640 1 -- 172.21.15.121:0/1168125404 >> v2:172.21.15.121:3300/0 conn(0x7f0a2c09da70 msgr2=0x7f0a2c09ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T18:29:38.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.209+0000 7f0a2b7fe640 1 -- 172.21.15.121:0/1168125404 >> v2:172.21.15.121:3300/0 conn(0x7f0a2c09da70 msgr2=0x7f0a2c09ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:29:38.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.209+0000 7f0a30fce640 1 -- 172.21.15.121:0/1168125404 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0a2c13a850 msgr2=0x7f0a2c13cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T18:29:38.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.209+0000 7f0a2b7fe640 1 --2- 172.21.15.121:0/1168125404 >> v2:172.21.15.121:3300/0 conn(0x7f0a2c09da70 0x7f0a2c09ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:29:38.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.209+0000 7f0a30fce640 1 -- 172.21.15.121:0/1168125404 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0a2c13a850 msgr2=0x7f0a2c13cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:29:38.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.209+0000 7f0a30fce640 1 --2- 172.21.15.121:0/1168125404 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0a2c13a850 0x7f0a2c13cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:29:38.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.209+0000 7f0a2b7fe640 1 --2- 172.21.15.121:0/1168125404 >> v2:172.21.15.121:3300/0 conn(0x7f0a2c09da70 0x7f0a2c09ec60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:29:38.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.209+0000 7f0a30fce640 1 --2- 172.21.15.121:0/1168125404 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0a2c13a850 0x7f0a2c13cc40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:29:38.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.410+0000 7f0a2bfff640 1 -- 172.21.15.121:0/1168125404 >> v1:172.21.15.173:6789/0 conn(0x7f0a2c0a33f0 legacy=0x7f0a2c09e550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.173:6789/0 2024-09-17T18:29:38.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.410+0000 7f0a2b7fe640 1 --2- 172.21.15.121:0/1168125404 >> v2:172.21.15.121:3300/0 conn(0x7f0a2c09da70 0x7f0a2c09ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:38.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.410+0000 7f0a30fce640 1 --2- 172.21.15.121:0/1168125404 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0a2c13a850 0x7f0a2c13cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:38.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.410+0000 7f0a2b7fe640 10 monclient(hunting): get_auth_request con 0x7f0a2c09da70 auth_method 0 2024-09-17T18:29:38.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.410+0000 7f0a2b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:38.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.410+0000 7f0a2b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:38.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.410+0000 7f0a2b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:29:38.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.411+0000 7f0a30fce640 10 monclient(hunting): get_auth_request con 0x7f0a2c13a850 auth_method 0 2024-09-17T18:29:38.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.411+0000 7f0a30fce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:38.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.411+0000 7f0a30fce640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:38.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.411+0000 7f0a30fce640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:29:38.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.411+0000 7f0a2b7fe640 1 -- 172.21.15.121:0/1168125404 >> v2:172.21.15.121:3300/0 conn(0x7f0a2c09da70 msgr2=0x7f0a2c09ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T18:29:38.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.411+0000 7f0a2b7fe640 1 -- 172.21.15.121:0/1168125404 >> v2:172.21.15.121:3300/0 conn(0x7f0a2c09da70 msgr2=0x7f0a2c09ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:29:38.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.411+0000 7f0a2b7fe640 1 --2- 172.21.15.121:0/1168125404 >> v2:172.21.15.121:3300/0 conn(0x7f0a2c09da70 0x7f0a2c09ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:29:38.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.411+0000 7f0a30fce640 1 -- 172.21.15.121:0/1168125404 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0a2c13a850 msgr2=0x7f0a2c13cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T18:29:38.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.411+0000 7f0a30fce640 1 -- 172.21.15.121:0/1168125404 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0a2c13a850 msgr2=0x7f0a2c13cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:29:38.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.411+0000 7f0a2b7fe640 1 --2- 172.21.15.121:0/1168125404 >> v2:172.21.15.121:3300/0 conn(0x7f0a2c09da70 0x7f0a2c09ec60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T18:29:38.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.411+0000 7f0a30fce640 1 --2- 172.21.15.121:0/1168125404 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0a2c13a850 0x7f0a2c13cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:29:38.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.411+0000 7f0a30fce640 1 --2- 172.21.15.121:0/1168125404 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0a2c13a850 0x7f0a2c13cc40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T18:29:38.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.811+0000 7f0a2bfff640 1 -- 172.21.15.121:0/1168125404 >> v1:172.21.15.173:6789/0 conn(0x7f0a2c0a33f0 legacy=0x7f0a2c09e550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.173:6789/0 2024-09-17T18:29:38.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.812+0000 7f0a2b7fe640 1 --2- 172.21.15.121:0/1168125404 >> v2:172.21.15.121:3300/0 conn(0x7f0a2c09da70 0x7f0a2c09ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:38.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.812+0000 7f0a2b7fe640 10 monclient(hunting): get_auth_request con 0x7f0a2c09da70 auth_method 0 2024-09-17T18:29:38.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.812+0000 7f0a2b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:38.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.812+0000 7f0a2b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:38.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.812+0000 7f0a30fce640 1 --2- 172.21.15.121:0/1168125404 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0a2c13a850 0x7f0a2c13cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:38.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.812+0000 7f0a2b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:29:38.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.812+0000 7f0a30fce640 10 monclient(hunting): get_auth_request con 0x7f0a2c13a850 auth_method 0 2024-09-17T18:29:38.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.812+0000 7f0a30fce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:38.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.812+0000 7f0a30fce640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:38.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.812+0000 7f0a30fce640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:29:38.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.813+0000 7f0a2b7fe640 1 -- 172.21.15.121:0/1168125404 >> v2:172.21.15.121:3300/0 conn(0x7f0a2c09da70 msgr2=0x7f0a2c09ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:29:38.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.813+0000 7f0a2b7fe640 1 -- 172.21.15.121:0/1168125404 >> v2:172.21.15.121:3300/0 conn(0x7f0a2c09da70 msgr2=0x7f0a2c09ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:29:38.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.813+0000 7f0a30fce640 1 -- 172.21.15.121:0/1168125404 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0a2c13a850 msgr2=0x7f0a2c13cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T18:29:38.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.813+0000 7f0a2b7fe640 1 --2- 172.21.15.121:0/1168125404 >> v2:172.21.15.121:3300/0 conn(0x7f0a2c09da70 0x7f0a2c09ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:29:38.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.813+0000 7f0a30fce640 1 -- 172.21.15.121:0/1168125404 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0a2c13a850 msgr2=0x7f0a2c13cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:29:38.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.813+0000 7f0a30fce640 1 --2- 172.21.15.121:0/1168125404 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0a2c13a850 0x7f0a2c13cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:29:38.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.813+0000 7f0a2b7fe640 1 --2- 172.21.15.121:0/1168125404 >> v2:172.21.15.121:3300/0 conn(0x7f0a2c09da70 0x7f0a2c09ec60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T18:29:38.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:38.813+0000 7f0a30fce640 1 --2- 172.21.15.121:0/1168125404 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0a2c13a850 0x7f0a2c13cc40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T18:29:39.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:39.612+0000 7f0a2bfff640 1 -- 172.21.15.121:0/1168125404 >> v1:172.21.15.173:6789/0 conn(0x7f0a2c0a33f0 legacy=0x7f0a2c09e550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.173:6789/0 2024-09-17T18:29:39.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:39.615+0000 7f0a2b7fe640 1 --2- 172.21.15.121:0/1168125404 >> v2:172.21.15.121:3300/0 conn(0x7f0a2c09da70 0x7f0a2c09ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:39.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:39.615+0000 7f0a30fce640 1 --2- 172.21.15.121:0/1168125404 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0a2c13a850 0x7f0a2c13cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:39.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:39.615+0000 7f0a2b7fe640 10 monclient(hunting): get_auth_request con 0x7f0a2c09da70 auth_method 0 2024-09-17T18:29:39.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:39.615+0000 7f0a2b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:39.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:39.615+0000 7f0a2b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:39.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:39.615+0000 7f0a2b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:29:39.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:39.615+0000 7f0a30fce640 10 monclient(hunting): get_auth_request con 0x7f0a2c13a850 auth_method 0 2024-09-17T18:29:39.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:39.615+0000 7f0a30fce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:39.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:39.615+0000 7f0a30fce640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:39.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:39.615+0000 7f0a30fce640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:29:39.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:39.615+0000 7f0a2b7fe640 1 -- 172.21.15.121:0/1168125404 >> v2:172.21.15.121:3300/0 conn(0x7f0a2c09da70 msgr2=0x7f0a2c09ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:29:39.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:39.615+0000 7f0a2b7fe640 1 -- 172.21.15.121:0/1168125404 >> v2:172.21.15.121:3300/0 conn(0x7f0a2c09da70 msgr2=0x7f0a2c09ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:29:39.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:39.615+0000 7f0a30fce640 1 -- 172.21.15.121:0/1168125404 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0a2c13a850 msgr2=0x7f0a2c13cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T18:29:39.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:39.615+0000 7f0a2b7fe640 1 --2- 172.21.15.121:0/1168125404 >> v2:172.21.15.121:3300/0 conn(0x7f0a2c09da70 0x7f0a2c09ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:29:39.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:39.615+0000 7f0a30fce640 1 -- 172.21.15.121:0/1168125404 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0a2c13a850 msgr2=0x7f0a2c13cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:29:39.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:39.615+0000 7f0a30fce640 1 --2- 172.21.15.121:0/1168125404 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0a2c13a850 0x7f0a2c13cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:29:39.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:39.615+0000 7f0a2b7fe640 1 --2- 172.21.15.121:0/1168125404 >> v2:172.21.15.121:3300/0 conn(0x7f0a2c09da70 0x7f0a2c09ec60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T18:29:39.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:39.615+0000 7f0a30fce640 1 --2- 172.21.15.121:0/1168125404 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0a2c13a850 0x7f0a2c13cc40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T18:29:41.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:41.209+0000 7f0a29ffb640 10 monclient(hunting): tick 2024-09-17T18:29:41.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:41.209+0000 7f0a29ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-17T18:29:41.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:41.209+0000 7f0a29ffb640 1 monclient(hunting): continuing hunt 2024-09-17T18:29:41.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:41.209+0000 7f0a29ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T18:29:41.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:41.209+0000 7f0a29ffb640 1 -- 172.21.15.121:0/1168125404 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0a2c13a850 msgr2=0x7f0a2c13cc40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:29:41.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:41.209+0000 7f0a29ffb640 1 --2- 172.21.15.121:0/1168125404 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0a2c13a850 0x7f0a2c13cc40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:41.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:41.209+0000 7f0a29ffb640 1 -- 172.21.15.121:0/1168125404 >> v2:172.21.15.121:3300/0 conn(0x7f0a2c09da70 msgr2=0x7f0a2c09ec60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:29:41.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:41.209+0000 7f0a29ffb640 1 --2- 172.21.15.121:0/1168125404 >> v2:172.21.15.121:3300/0 conn(0x7f0a2c09da70 0x7f0a2c09ec60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:41.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:41.209+0000 7f0a29ffb640 1 -- 172.21.15.121:0/1168125404 >> v1:172.21.15.173:6789/0 conn(0x7f0a2c0a33f0 legacy=0x7f0a2c09e550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:29:41.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:41.209+0000 7f0a29ffb640 10 monclient: _add_conns ranks=[4,0] 2024-09-17T18:29:41.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:41.209+0000 7f0a29ffb640 1 --2- 172.21.15.121:0/1168125404 >> v2:172.21.15.173:3300/0 conn(0x7f0a180036d0 0x7f0a18005b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:41.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:41.209+0000 7f0a29ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f0a180036d0 addr v2:172.21.15.173:3300/0 2024-09-17T18:29:41.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:41.209+0000 7f0a29ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0a18006220 addr v1:172.21.15.121:6789/0 2024-09-17T18:29:41.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:41.209+0000 7f0a29ffb640 1 -- 172.21.15.121:0/1168125404 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a2c0a13c0 con 0x7f0a18006220 2024-09-17T18:29:41.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:41.209+0000 7f0a29ffb640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:41.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:41.209+0000 7f0a29ffb640 1 -- 172.21.15.121:0/1168125404 --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f0a2c0a37d0 con 0x7f0a180036d0 2024-09-17T18:29:41.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:41.209+0000 7f0a29ffb640 10 monclient(hunting): _renew_subs 2024-09-17T18:29:41.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:41.209+0000 7f0a2bfff640 1 -- 172.21.15.121:0/1168125404 >> v2:172.21.15.173:3300/0 conn(0x7f0a180036d0 msgr2=0x7f0a18005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.173:3300/0 2024-09-17T18:29:41.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:41.209+0000 7f0a2bfff640 1 --2- 172.21.15.121:0/1168125404 >> v2:172.21.15.173:3300/0 conn(0x7f0a180036d0 0x7f0a18005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:29:41.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:41.211+0000 7f0a2b7fe640 1 -- 172.21.15.121:0/1168125404 >> v1:172.21.15.121:6789/0 conn(0x7f0a18006220 legacy=0x7f0a180086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:41.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:41.411+0000 7f0a2bfff640 1 -- 172.21.15.121:0/1168125404 >> v2:172.21.15.173:3300/0 conn(0x7f0a180036d0 msgr2=0x7f0a18005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.173:3300/0 2024-09-17T18:29:41.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:41.411+0000 7f0a2bfff640 1 --2- 172.21.15.121:0/1168125404 >> v2:172.21.15.173:3300/0 conn(0x7f0a180036d0 0x7f0a18005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T18:29:41.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:41.812+0000 7f0a2bfff640 1 -- 172.21.15.121:0/1168125404 >> v2:172.21.15.173:3300/0 conn(0x7f0a180036d0 msgr2=0x7f0a18005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.173:3300/0 2024-09-17T18:29:41.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:41.812+0000 7f0a2bfff640 1 --2- 172.21.15.121:0/1168125404 >> v2:172.21.15.173:3300/0 conn(0x7f0a180036d0 0x7f0a18005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T18:29:42.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:42.450+0000 7f0a2affd640 1 -- 172.21.15.121:0/1168125404 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3013740134 0 0) 0x7f0a2c0a13c0 con 0x7f0a18006220 2024-09-17T18:29:42.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:42.450+0000 7f0a2affd640 10 monclient(hunting): _negotiate 2024-09-17T18:29:42.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:42.450+0000 7f0a2affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:42.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:42.450+0000 7f0a2affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:42.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:42.450+0000 7f0a2affd640 10 monclient(hunting): my global_id is 44601 2024-09-17T18:29:42.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:42.450+0000 7f0a2affd640 1 -- 172.21.15.121:0/1168125404 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a14003520 con 0x7f0a18006220 2024-09-17T18:29:42.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:42.450+0000 7f0a2b7fe640 1 -- 172.21.15.121:0/1168125404 >> v1:172.21.15.121:6789/0 conn(0x7f0a18006220 legacy=0x7f0a180086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-17T18:29:42.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:42.450+0000 7f0a2b7fe640 1 -- 172.21.15.121:0/1168125404 >> v1:172.21.15.121:6789/0 conn(0x7f0a18006220 legacy=0x7f0a180086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:29:42.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:42.450+0000 7f0a2b7fe640 1 --1- 172.21.15.121:0/1168125404 >> v1:172.21.15.121:6789/0 conn(0x7f0a18006220 0x7f0a180086e0 :-1 s=OPENED pgs=277 cs=1 l=1).handle_message read tag failed 2024-09-17T18:29:42.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:42.450+0000 7f0a2b7fe640 1 --1- 172.21.15.121:0/1168125404 >> v1:172.21.15.121:6789/0 conn(0x7f0a18006220 0x7f0a180086e0 :-1 s=OPENED pgs=277 cs=1 l=1).fault on lossy channel, failing 2024-09-17T18:29:42.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:42.450+0000 7f0a2affd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.121:6789/0 2024-09-17T18:29:42.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:42.613+0000 7f0a2bfff640 1 -- 172.21.15.121:0/1168125404 >> v2:172.21.15.173:3300/0 conn(0x7f0a180036d0 msgr2=0x7f0a18005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.173:3300/0 2024-09-17T18:29:42.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:42.613+0000 7f0a2bfff640 1 --2- 172.21.15.121:0/1168125404 >> v2:172.21.15.173:3300/0 conn(0x7f0a180036d0 0x7f0a18005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T18:29:44.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.209+0000 7f0a29ffb640 10 monclient(hunting): tick 2024-09-17T18:29:44.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.209+0000 7f0a29ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-17T18:29:44.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.209+0000 7f0a29ffb640 1 monclient(hunting): continuing hunt 2024-09-17T18:29:44.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.209+0000 7f0a29ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T18:29:44.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.209+0000 7f0a29ffb640 1 -- 172.21.15.121:0/1168125404 >> v2:172.21.15.173:3300/0 conn(0x7f0a180036d0 msgr2=0x7f0a18005b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:29:44.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.209+0000 7f0a29ffb640 1 --2- 172.21.15.121:0/1168125404 >> v2:172.21.15.173:3300/0 conn(0x7f0a180036d0 0x7f0a18005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:44.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.209+0000 7f0a29ffb640 1 -- 172.21.15.121:0/1168125404 >> v1:172.21.15.121:6789/0 conn(0x7f0a18006220 legacy=0x7f0a180086e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:29:44.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.209+0000 7f0a29ffb640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-17T18:29:44.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.209+0000 7f0a30fce640 1 -- 172.21.15.121:0/1168125404 reap_dead start 2024-09-17T18:29:44.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.209+0000 7f0a29ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0a2c0a33f0 addr v1:172.21.15.173:6789/0 2024-09-17T18:29:44.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.209+0000 7f0a29ffb640 1 --2- 172.21.15.121:0/1168125404 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0a1800a6e0 0x7f0a18005ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:44.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.209+0000 7f0a29ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f0a1800a6e0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:29:44.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.209+0000 7f0a29ffb640 1 --2- 172.21.15.121:0/1168125404 >> v2:172.21.15.173:3300/0 conn(0x7f0a18008eb0 0x7f0a180086e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:44.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.209+0000 7f0a29ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f0a18008eb0 addr v2:172.21.15.173:3300/0 2024-09-17T18:29:44.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.209+0000 7f0a29ffb640 1 -- 172.21.15.121:0/1168125404 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a2c074b40 con 0x7f0a2c0a33f0 2024-09-17T18:29:44.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.209+0000 7f0a29ffb640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:44.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.209+0000 7f0a29ffb640 1 -- 172.21.15.121:0/1168125404 --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f0a2c0a37d0 con 0x7f0a18008eb0 2024-09-17T18:29:44.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.209+0000 7f0a29ffb640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:44.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.209+0000 7f0a29ffb640 1 -- 172.21.15.121:0/1168125404 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0a1800ac80 con 0x7f0a1800a6e0 2024-09-17T18:29:44.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.209+0000 7f0a29ffb640 10 monclient(hunting): _renew_subs 2024-09-17T18:29:44.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.209+0000 7f0a30fce640 1 -- 172.21.15.121:0/1168125404 >> v2:172.21.15.173:3300/0 conn(0x7f0a18008eb0 msgr2=0x7f0a180086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.173:3300/0 2024-09-17T18:29:44.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.209+0000 7f0a2bfff640 1 -- 172.21.15.121:0/1168125404 >> v1:172.21.15.173:6789/0 conn(0x7f0a2c0a33f0 legacy=0x7f0a18009fd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.173:6789/0 2024-09-17T18:29:44.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.209+0000 7f0a30fce640 1 --2- 172.21.15.121:0/1168125404 >> v2:172.21.15.173:3300/0 conn(0x7f0a18008eb0 0x7f0a180086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:29:44.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.209+0000 7f0a2b7fe640 1 --2- 172.21.15.121:0/1168125404 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0a1800a6e0 0x7f0a18005ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:44.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.209+0000 7f0a2b7fe640 10 monclient(hunting): get_auth_request con 0x7f0a1800a6e0 auth_method 0 2024-09-17T18:29:44.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.209+0000 7f0a2b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:44.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.209+0000 7f0a2b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:44.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.209+0000 7f0a2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:44.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.210+0000 7f0a2b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:44.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.210+0000 7f0a2b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:44.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.210+0000 7f0a2b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:44.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.210+0000 7f0a2b7fe640 10 monclient(hunting): handle_auth_done global_id 64241 payload 274 2024-09-17T18:29:44.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.210+0000 7f0a2b7fe640 1 -- 172.21.15.121:0/1168125404 >> v2:172.21.15.173:3300/0 conn(0x7f0a18008eb0 msgr2=0x7f0a180086e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:29:44.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.210+0000 7f0a2b7fe640 1 --2- 172.21.15.121:0/1168125404 >> v2:172.21.15.173:3300/0 conn(0x7f0a18008eb0 0x7f0a180086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:44.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.210+0000 7f0a2b7fe640 1 -- 172.21.15.121:0/1168125404 >> v1:172.21.15.173:6789/0 conn(0x7f0a2c0a33f0 legacy=0x7f0a18009fd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:29:44.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.210+0000 7f0a2b7fe640 10 monclient: _finish_hunting 0 2024-09-17T18:29:44.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.210+0000 7f0a2b7fe640 1 monclient: found mon.noname-b 2024-09-17T18:29:44.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.210+0000 7f0a2b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:29:44.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.210+0000 7f0a2b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:29:44.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.210+0000 7f0a2b7fe640 1 -- 172.21.15.121:0/1168125404 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a18009360 con 0x7f0a1800a6e0 2024-09-17T18:29:44.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.213+0000 7f0a2b7fe640 1 -- 172.21.15.121:0/1168125404 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0a1800a6e0 msgr2=0x7f0a18005ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:44.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.213+0000 7f0a2b7fe640 1 --2- 172.21.15.121:0/1168125404 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0a1800a6e0 0x7f0a18005ac0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0a20006fd0 tx=0x7f0a2002acc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cad6bae483c6f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:44.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.213+0000 7f0a2affd640 1 -- 172.21.15.121:0/1168125404 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a20031070 con 0x7f0a1800a6e0 2024-09-17T18:29:44.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.213+0000 7f0a2affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:44.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.213+0000 7f0a2affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:29:44.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.213+0000 7f0a2affd640 10 monclient: dump: 2024-09-17T18:29:44.220 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:44.220 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:44.220 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:44.220 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:44.220 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:44.220 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:44.220 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:44.221 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:44.221 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:44.221 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:44.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.213+0000 7f0a2affd640 10 monclient: _finish_auth 0 2024-09-17T18:29:44.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.213+0000 7f0a2affd640 10 monclient: _check_auth_tickets 2024-09-17T18:29:44.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.213+0000 7f0a2affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:29:44.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.213+0000 7f0a2affd640 1 -- 172.21.15.121:0/1168125404 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a20004030 con 0x7f0a1800a6e0 2024-09-17T18:29:44.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.213+0000 7f0a32a31640 5 monclient: authenticate success, global_id 64241 2024-09-17T18:29:44.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.213+0000 7f0a32a31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:29:44.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.213+0000 7f0a2affd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:29:44.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.213+0000 7f0a2affd640 1 -- 172.21.15.121:0/1168125404 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a20004400 con 0x7f0a1800a6e0 2024-09-17T18:29:44.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.213+0000 7f0a32a31640 10 monclient: get_monmap_and_config success 2024-09-17T18:29:44.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.213+0000 7f0a2affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:44.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.213+0000 7f0a32a31640 4 set_mon_vals no callback set 2024-09-17T18:29:44.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.213+0000 7f0a2affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:29:44.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.213+0000 7f0a2affd640 10 monclient: dump: 2024-09-17T18:29:44.222 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:44.222 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:44.222 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:44.222 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:44.222 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:44.222 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:44.222 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:44.222 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:44.222 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:44.222 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:44.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.214+0000 7f0a32a31640 10 monclient: shutdown 2024-09-17T18:29:44.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.214+0000 7f0a32a31640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:29:44.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.214+0000 7f0a32a31640 1 -- 172.21.15.121:0/1168125404 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0a1800a6e0 msgr2=0x7f0a18005ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:44.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.214+0000 7f0a32a31640 1 --2- 172.21.15.121:0/1168125404 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0a1800a6e0 0x7f0a18005ac0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0a20006fd0 tx=0x7f0a2002acc0 comp rx=0 tx=0).stop 2024-09-17T18:29:44.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.214+0000 7f0a32a31640 1 -- 172.21.15.121:0/1168125404 shutdown_connections 2024-09-17T18:29:44.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.214+0000 7f0a32a31640 1 --2- 172.21.15.121:0/1168125404 >> v2:172.21.15.173:3300/0 conn(0x7f0a18008eb0 0x7f0a180086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:44.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.214+0000 7f0a32a31640 1 --2- 172.21.15.121:0/1168125404 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0a1800a6e0 0x7f0a18005ac0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:44.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.214+0000 7f0a32a31640 1 -- 172.21.15.121:0/1168125404 >> 172.21.15.121:0/1168125404 conn(0x7f0a2c0a3b50 msgr2=0x7f0a2c0a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:29:44.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.214+0000 7f0a32a31640 1 -- 172.21.15.121:0/1168125404 shutdown_connections 2024-09-17T18:29:44.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.214+0000 7f0a32a31640 1 -- 172.21.15.121:0/1168125404 wait complete. 2024-09-17T18:29:44.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.215+0000 7f0a32a31640 10 monclient: build_initial_monmap 2024-09-17T18:29:44.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.215+0000 7f0a32a31640 10 monclient: monmap: 2024-09-17T18:29:44.223 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:29:44.223 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:29:44.223 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:29:44.223 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:29:44.223 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:29:44.223 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:44.224 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:29:44.224 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:29:44.224 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:29:44.224 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:44.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.215+0000 7f0a32a31640 1 librados: starting msgr at 2024-09-17T18:29:44.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.215+0000 7f0a32a31640 1 librados: starting objecter 2024-09-17T18:29:44.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.215+0000 7f0a32a31640 1 Processor -- start 2024-09-17T18:29:44.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.215+0000 7f0a32a31640 1 -- start start 2024-09-17T18:29:44.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.215+0000 7f0a32a31640 1 librados: setting wanted keys 2024-09-17T18:29:44.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.215+0000 7f0a32a31640 1 librados: calling monclient init 2024-09-17T18:29:44.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.215+0000 7f0a32a31640 10 monclient: init 2024-09-17T18:29:44.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.216+0000 7f0a32a31640 10 monclient: _reopen_session rank -1 2024-09-17T18:29:44.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.216+0000 7f0a32a31640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T18:29:44.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.216+0000 7f0a32a31640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a18008eb0 0x7f0a2c1c8d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:44.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.216+0000 7f0a32a31640 10 monclient(hunting): picked mon.noname-b con 0x7f0a18008eb0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:29:44.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.216+0000 7f0a32a31640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0a1800a6e0 0x7f0a2c137490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:44.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.216+0000 7f0a32a31640 10 monclient(hunting): picked mon.noname-c con 0x7f0a1800a6e0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:29:44.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.216+0000 7f0a32a31640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a2c0a33f0 0x7f0a2c1379d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:44.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.216+0000 7f0a32a31640 10 monclient(hunting): picked mon.noname-a con 0x7f0a2c0a33f0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:29:44.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.216+0000 7f0a32a31640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:44.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.216+0000 7f0a32a31640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0a2c09e6d0 con 0x7f0a2c0a33f0 2024-09-17T18:29:44.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.216+0000 7f0a32a31640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:44.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.216+0000 7f0a32a31640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0a2c09e550 con 0x7f0a18008eb0 2024-09-17T18:29:44.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.216+0000 7f0a32a31640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:44.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.216+0000 7f0a32a31640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0a2c09e850 con 0x7f0a1800a6e0 2024-09-17T18:29:44.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.216+0000 7f0a32a31640 10 monclient(hunting): _renew_subs 2024-09-17T18:29:44.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.216+0000 7f0a32a31640 10 monclient(hunting): authenticate will time out at 1387.932509s 2024-09-17T18:29:44.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.216+0000 7f0a2b7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0a1800a6e0 0x7f0a2c137490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:44.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.216+0000 7f0a2bfff640 1 -- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a18008eb0 msgr2=0x7f0a2c1c8d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.173:3300/0 2024-09-17T18:29:44.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.216+0000 7f0a30fce640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a2c0a33f0 0x7f0a2c1379d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:44.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.216+0000 7f0a2b7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0a1800a6e0 0x7f0a2c137490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:43104/0 (socket says 172.21.15.121:43104) 2024-09-17T18:29:44.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.216+0000 7f0a2b7fe640 1 -- 172.21.15.121:0/3693653468 learned_addr learned my addr 172.21.15.121:0/3693653468 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:29:44.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.216+0000 7f0a2b7fe640 10 monclient(hunting): get_auth_request con 0x7f0a1800a6e0 auth_method 0 2024-09-17T18:29:44.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.216+0000 7f0a2b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:44.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.216+0000 7f0a2b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:44.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.216+0000 7f0a2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:44.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.216+0000 7f0a2bfff640 1 --2- 172.21.15.121:0/3693653468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a18008eb0 0x7f0a2c1c8d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:29:44.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.216+0000 7f0a30fce640 10 monclient(hunting): get_auth_request con 0x7f0a2c0a33f0 auth_method 0 2024-09-17T18:29:44.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.216+0000 7f0a30fce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:44.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.216+0000 7f0a30fce640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:44.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.216+0000 7f0a30fce640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:44.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.216+0000 7f0a30fce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:44.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.216+0000 7f0a30fce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:44.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.216+0000 7f0a30fce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:44.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.216+0000 7f0a2b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:44.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.216+0000 7f0a2b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:44.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.216+0000 7f0a2b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:44.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.216+0000 7f0a30fce640 10 monclient(hunting): handle_auth_done global_id 44604 payload 824 2024-09-17T18:29:44.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.217+0000 7f0a30fce640 1 -- 172.21.15.121:0/3693653468 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0a1800a6e0 msgr2=0x7f0a2c137490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:44.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.217+0000 7f0a30fce640 1 --2- 172.21.15.121:0/3693653468 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0a1800a6e0 0x7f0a2c137490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:44.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.217+0000 7f0a30fce640 1 -- 172.21.15.121:0/3693653468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a18008eb0 msgr2=0x7f0a2c1c8d50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:29:44.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.217+0000 7f0a30fce640 1 --2- 172.21.15.121:0/3693653468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a18008eb0 0x7f0a2c1c8d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:44.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.217+0000 7f0a30fce640 10 monclient: _finish_hunting 0 2024-09-17T18:29:44.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.217+0000 7f0a30fce640 1 monclient: found mon.noname-a 2024-09-17T18:29:44.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.217+0000 7f0a30fce640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:29:44.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.217+0000 7f0a30fce640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:29:44.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.217+0000 7f0a30fce640 1 -- 172.21.15.121:0/3693653468 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a2c09e240 con 0x7f0a2c0a33f0 2024-09-17T18:29:44.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.217+0000 7f0a2b7fe640 1 --2- 172.21.15.121:0/3693653468 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0a1800a6e0 0x7f0a2c137490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:29:44.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.219+0000 7f0a30fce640 1 -- 172.21.15.121:0/3693653468 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a2c0a33f0 msgr2=0x7f0a2c1379d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:44.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.219+0000 7f0a30fce640 1 --2- 172.21.15.121:0/3693653468 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a2c0a33f0 0x7f0a2c1379d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c09f670 tx=0x7f0a24012f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:44.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.219+0000 7f0a28ff9640 10 client.?.objecter ms_handle_connect 0x7f0a2c0a33f0 2024-09-17T18:29:44.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.219+0000 7f0a28ff9640 10 client.?.objecter resend_mon_ops 2024-09-17T18:29:44.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.219+0000 7f0a28ff9640 1 -- 172.21.15.121:0/3693653468 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a2400b2d0 con 0x7f0a2c0a33f0 2024-09-17T18:29:44.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.219+0000 7f0a28ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:44.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.219+0000 7f0a28ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:29:44.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.219+0000 7f0a28ff9640 10 monclient: dump: 2024-09-17T18:29:44.231 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:44.231 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:44.231 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:44.231 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:44.231 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:44.231 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:44.231 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:44.231 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:44.231 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:44.231 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:44.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.219+0000 7f0a28ff9640 10 monclient: _finish_auth 0 2024-09-17T18:29:44.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.219+0000 7f0a28ff9640 10 monclient: _check_auth_tickets 2024-09-17T18:29:44.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.219+0000 7f0a28ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:29:44.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.219+0000 7f0a32a31640 5 monclient: authenticate success, global_id 44604 2024-09-17T18:29:44.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.219+0000 7f0a32a31640 10 monclient: _renew_subs 2024-09-17T18:29:44.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.219+0000 7f0a28ff9640 1 -- 172.21.15.121:0/3693653468 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a24017070 con 0x7f0a2c0a33f0 2024-09-17T18:29:44.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.219+0000 7f0a32a31640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:29:44.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.219+0000 7f0a32a31640 1 -- 172.21.15.121:0/3693653468 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a2c122f30 con 0x7f0a2c0a33f0 2024-09-17T18:29:44.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.219+0000 7f0a28ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T18:29:44.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.219+0000 7f0a28ff9640 1 -- 172.21.15.121:0/3693653468 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a24009700 con 0x7f0a2c0a33f0 2024-09-17T18:29:44.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.219+0000 7f0a28ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:44.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.219+0000 7f0a28ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:29:44.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.219+0000 7f0a28ff9640 10 monclient: dump: 2024-09-17T18:29:44.232 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:44.232 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:44.233 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:44.233 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:44.233 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:44.233 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:44.233 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:44.233 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:44.233 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:44.233 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:44.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.219+0000 7f0a2a7fc640 4 set_mon_vals no callback set 2024-09-17T18:29:44.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.219+0000 7f0a32a31640 10 client.44604.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:29:44.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.219+0000 7f0a32a31640 10 monclient: _renew_subs 2024-09-17T18:29:44.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.219+0000 7f0a32a31640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:29:44.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.219+0000 7f0a32a31640 1 -- 172.21.15.121:0/3693653468 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a2c120bd0 con 0x7f0a2c0a33f0 2024-09-17T18:29:44.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.219+0000 7f0a32a31640 1 librados: init done 2024-09-17T18:29:44.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.220+0000 7f0a28ff9640 1 -- 172.21.15.121:0/3693653468 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f0a24009ab0 con 0x7f0a2c0a33f0 2024-09-17T18:29:44.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.220+0000 7f0a28ff9640 1 --2- 172.21.15.121:0/3693653468 >> v2:172.21.15.173:6812/2423786037 conn(0x7f0a18045ef0 0x7f0a180483b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:44.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.220+0000 7f0a32a31640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:29:44.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.220+0000 7f0a32a31640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:29:44.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.220+0000 7f0a32a31640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:29:44.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.220+0000 7f0a28ff9640 1 -- 172.21.15.121:0/3693653468 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(413..413 src has 247..413) ==== 5358+0+0 (secure 0 0 0) 0x7f0a24054c40 con 0x7f0a2c0a33f0 2024-09-17T18:29:44.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.220+0000 7f0a28ff9640 10 client.44604.objecter ms_dispatch 0x7f0a2c000b90 osd_map(413..413 src has 247..413) 2024-09-17T18:29:44.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.221+0000 7f0a28ff9640 3 client.44604.objecter handle_osd_map got epochs [413,413] > 0 2024-09-17T18:29:44.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.221+0000 7f0a28ff9640 3 client.44604.objecter handle_osd_map decoding full epoch 413 2024-09-17T18:29:44.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.220+0000 7f0a32a31640 1 -- 172.21.15.121:0/3693653468 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a2c09ebe0 con 0x7f0a2c0a33f0 2024-09-17T18:29:44.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.221+0000 7f0a28ff9640 20 client.44604.objecter dump_active .. 0 homeless 2024-09-17T18:29:44.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.221+0000 7f0a2bfff640 1 --2- 172.21.15.121:0/3693653468 >> v2:172.21.15.173:6812/2423786037 conn(0x7f0a18045ef0 0x7f0a180483b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:44.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.221+0000 7f0a2bfff640 10 monclient: get_auth_request con 0x7f0a18045ef0 auth_method 0 2024-09-17T18:29:44.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.223+0000 7f0a28ff9640 10 client.44604.objecter ms_handle_connect 0x7f0a18045ef0 2024-09-17T18:29:44.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.223+0000 7f0a2bfff640 1 --2- 172.21.15.121:0/3693653468 >> v2:172.21.15.173:6812/2423786037 conn(0x7f0a18045ef0 0x7f0a180483b0 secure :-1 s=READY pgs=4021 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c003930 tx=0x7f0a1c00a370 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:44.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.225+0000 7f0a28ff9640 1 -- 172.21.15.121:0/3693653468 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0a24021060 con 0x7f0a2c0a33f0 2024-09-17T18:29:44.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.225+0000 7f0a28ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:29:44.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.225+0000 7f0a28ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:29:44.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.483+0000 7f0a32a31640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:29:44.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.483+0000 7f0a32a31640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:29:44.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.483+0000 7f0a32a31640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:29:44.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.483+0000 7f0a32a31640 1 -- 172.21.15.121:0/3693653468 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0a2c122750 con 0x7f0a2c0a33f0 2024-09-17T18:29:44.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.483+0000 7f0a28ff9640 1 -- 172.21.15.121:0/3693653468 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f0a2c09e240 con 0x7f0a2c0a33f0 2024-09-17T18:29:44.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.483+0000 7f0a28ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:29:44.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.483+0000 7f0a28ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:29:44.486 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":42,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:29:44.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.485+0000 7f0a32a31640 10 librados: watch_flush enter 2024-09-17T18:29:44.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.485+0000 7f0a32a31640 10 librados: watch_flush exit 2024-09-17T18:29:44.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.485+0000 7f0a32a31640 20 client.44604.objecter shutdown clearing up homeless session... 2024-09-17T18:29:44.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.485+0000 7f0a32a31640 10 client.44604.objecter successfully canceled tick 2024-09-17T18:29:44.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.486+0000 7f0a32a31640 1 -- 172.21.15.121:0/3693653468 >> v2:172.21.15.173:6812/2423786037 conn(0x7f0a18045ef0 msgr2=0x7f0a180483b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:44.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.486+0000 7f0a32a31640 1 --2- 172.21.15.121:0/3693653468 >> v2:172.21.15.173:6812/2423786037 conn(0x7f0a18045ef0 0x7f0a180483b0 secure :-1 s=READY pgs=4021 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c003930 tx=0x7f0a1c00a370 comp rx=0 tx=0).stop 2024-09-17T18:29:44.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.486+0000 7f0a32a31640 10 monclient: shutdown 2024-09-17T18:29:44.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.486+0000 7f0a32a31640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:29:44.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.486+0000 7f0a32a31640 1 -- 172.21.15.121:0/3693653468 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a2c0a33f0 msgr2=0x7f0a2c1379d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:44.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.486+0000 7f0a32a31640 1 --2- 172.21.15.121:0/3693653468 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a2c0a33f0 0x7f0a2c1379d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c09f670 tx=0x7f0a24012f40 comp rx=0 tx=0).stop 2024-09-17T18:29:44.489 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.486+0000 7f0a32a31640 1 -- 172.21.15.121:0/3693653468 shutdown_connections 2024-09-17T18:29:44.489 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.486+0000 7f0a32a31640 1 --2- 172.21.15.121:0/3693653468 >> v2:172.21.15.173:6812/2423786037 conn(0x7f0a18045ef0 0x7f0a180483b0 unknown :-1 s=CLOSED pgs=4021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:44.489 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.486+0000 7f0a32a31640 1 --2- 172.21.15.121:0/3693653468 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a2c0a33f0 0x7f0a2c1379d0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:44.489 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.486+0000 7f0a32a31640 1 --2- 172.21.15.121:0/3693653468 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0a1800a6e0 0x7f0a2c137490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:44.489 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.486+0000 7f0a32a31640 1 --2- 172.21.15.121:0/3693653468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a18008eb0 0x7f0a2c1c8d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:44.489 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.486+0000 7f0a32a31640 1 -- 172.21.15.121:0/3693653468 >> 172.21.15.121:0/3693653468 conn(0x7f0a2c0a3b50 msgr2=0x7f0a2c0a2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:29:44.489 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.486+0000 7f0a32a31640 1 -- 172.21.15.121:0/3693653468 shutdown_connections 2024-09-17T18:29:44.489 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.486+0000 7f0a32a31640 1 -- 172.21.15.121:0/3693653468 wait complete. 2024-09-17T18:29:44.489 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.486+0000 7f0a32a31640 1 librados: shutdown 2024-09-17T18:29:44.498 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-17T18:29:44.499 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-17T18:29:44.499 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T18:29:44.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.594+0000 7f5a0818a640 10 monclient: get_monmap_and_config 2024-09-17T18:29:44.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.594+0000 7f5a0818a640 10 monclient: build_initial_monmap 2024-09-17T18:29:44.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.595+0000 7f5a0818a640 10 monclient: monmap: 2024-09-17T18:29:44.597 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:29:44.597 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:44.597 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:29:44.595584+0000 2024-09-17T18:29:44.597 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:29:44.595584+0000 2024-09-17T18:29:44.597 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:29:44.597 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:44.597 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:29:44.597 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:29:44.597 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:29:44.597 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:29:44.598 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:29:44.598 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:44.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.595+0000 7f5a0818a640 1 Processor -- start 2024-09-17T18:29:44.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.595+0000 7f5a0818a640 1 -- start start 2024-09-17T18:29:44.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.595+0000 7f5a0818a640 10 monclient: init 2024-09-17T18:29:44.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a0818a640 10 monclient: _reopen_session rank -1 2024-09-17T18:29:44.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a0818a640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-17T18:29:44.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a0818a640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f5a0009e550 0x7f5a0009e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:44.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a0818a640 10 monclient(hunting): picked mon.noname-c con 0x7f5a0009e550 addr v2:172.21.15.173:3300/0 2024-09-17T18:29:44.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a0818a640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f5a0009ee70 0x7f5a00139840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:44.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a0818a640 10 monclient(hunting): picked mon.noname-a con 0x7f5a0009ee70 addr v2:172.21.15.121:3300/0 2024-09-17T18:29:44.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a0818a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a0009f2c0 0x7f5a0013bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:44.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a0818a640 10 monclient(hunting): picked mon.noname-b con 0x7f5a0009f2c0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:29:44.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a0818a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:44.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a0818a640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f5a000a4950 con 0x7f5a0009ee70 2024-09-17T18:29:44.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a0818a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:44.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a0818a640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f5a000a4460 con 0x7f5a0009e550 2024-09-17T18:29:44.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a0818a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:44.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a0818a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5a000a2b60 con 0x7f5a0009f2c0 2024-09-17T18:29:44.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a0818a640 10 monclient(hunting): _renew_subs 2024-09-17T18:29:44.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a0818a640 10 monclient(hunting): authenticate will time out at 1388.312308s 2024-09-17T18:29:44.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a05f26640 1 -- >> v2:172.21.15.173:3300/0 conn(0x7f5a0009e550 msgr2=0x7f5a0009e930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.173:3300/0 2024-09-17T18:29:44.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a06727640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a0009f2c0 0x7f5a0013bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:44.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a05725640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f5a0009ee70 0x7f5a00139840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:44.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a05f26640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f5a0009e550 0x7f5a0009e930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:29:44.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a06727640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a0009f2c0 0x7f5a0013bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:43120/0 (socket says 172.21.15.121:43120) 2024-09-17T18:29:44.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a05725640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f5a0009ee70 0x7f5a00139840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39890/0 (socket says 172.21.15.121:39890) 2024-09-17T18:29:44.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a06727640 1 -- 172.21.15.121:0/1034306432 learned_addr learned my addr 172.21.15.121:0/1034306432 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:29:44.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a06727640 10 monclient(hunting): get_auth_request con 0x7f5a0009f2c0 auth_method 0 2024-09-17T18:29:44.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a06727640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:44.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a06727640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:44.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a06727640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:44.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a05725640 10 monclient(hunting): get_auth_request con 0x7f5a0009ee70 auth_method 0 2024-09-17T18:29:44.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a05725640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:44.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a05725640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:44.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a05725640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:44.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a05725640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:44.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a05725640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:44.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a05725640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:44.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a06727640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:44.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a06727640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:44.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a06727640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:44.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a05725640 10 monclient(hunting): handle_auth_done global_id 44607 payload 274 2024-09-17T18:29:44.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a05725640 1 -- 172.21.15.121:0/1034306432 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a0009f2c0 msgr2=0x7f5a0013bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:29:44.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a05725640 1 --2- 172.21.15.121:0/1034306432 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a0009f2c0 0x7f5a0013bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:44.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a05725640 1 -- 172.21.15.121:0/1034306432 >> v2:172.21.15.173:3300/0 conn(0x7f5a0009e550 msgr2=0x7f5a0009e930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:29:44.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a05725640 1 --2- 172.21.15.121:0/1034306432 >> v2:172.21.15.173:3300/0 conn(0x7f5a0009e550 0x7f5a0009e930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:44.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a05725640 10 monclient: _finish_hunting 0 2024-09-17T18:29:44.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a05725640 1 monclient: found mon.noname-a 2024-09-17T18:29:44.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a05725640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:29:44.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a05725640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:29:44.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a05725640 1 -- 172.21.15.121:0/1034306432 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a0009f840 con 0x7f5a0009ee70 2024-09-17T18:29:44.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.596+0000 7f5a06727640 1 --2- 172.21.15.121:0/1034306432 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a0009f2c0 0x7f5a0013bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:29:44.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.599+0000 7f5a05725640 1 -- 172.21.15.121:0/1034306432 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a0009ee70 msgr2=0x7f5a00139840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:44.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.599+0000 7f5a05725640 1 --2- 172.21.15.121:0/1034306432 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a0009ee70 0x7f5a00139840 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f5a0009e0a0 tx=0x7f59f402e300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfdd92f178417404 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:44.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.599+0000 7f5a04f24640 1 -- 172.21.15.121:0/1034306432 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f400b650 con 0x7f5a0009ee70 2024-09-17T18:29:44.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.599+0000 7f5a04f24640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:44.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.599+0000 7f5a04f24640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:29:44.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.599+0000 7f5a04f24640 10 monclient: dump: 2024-09-17T18:29:44.611 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:44.611 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:44.611 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:44.611 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:44.611 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:44.611 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:44.611 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:44.611 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:44.611 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:44.612 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:44.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.599+0000 7f5a04f24640 10 monclient: _finish_auth 0 2024-09-17T18:29:44.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.599+0000 7f5a04f24640 10 monclient: _check_auth_tickets 2024-09-17T18:29:44.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.599+0000 7f5a04f24640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:29:44.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.599+0000 7f5a0818a640 5 monclient: authenticate success, global_id 44607 2024-09-17T18:29:44.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.599+0000 7f5a04f24640 1 -- 172.21.15.121:0/1034306432 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59f402ee50 con 0x7f5a0009ee70 2024-09-17T18:29:44.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.599+0000 7f5a0818a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:29:44.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.599+0000 7f5a04f24640 10 monclient: handle_config config(0 keys) 2024-09-17T18:29:44.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.599+0000 7f5a0818a640 10 monclient: get_monmap_and_config success 2024-09-17T18:29:44.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.599+0000 7f5a04f24640 1 -- 172.21.15.121:0/1034306432 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f400ba10 con 0x7f5a0009ee70 2024-09-17T18:29:44.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.599+0000 7f5a0818a640 4 set_mon_vals no callback set 2024-09-17T18:29:44.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.599+0000 7f5a04f24640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:44.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.599+0000 7f5a04f24640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:29:44.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.599+0000 7f5a04f24640 10 monclient: dump: 2024-09-17T18:29:44.612 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:44.613 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:44.613 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:44.613 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:44.613 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:44.613 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:44.613 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:44.613 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:44.613 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:44.613 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:44.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.600+0000 7f5a0818a640 10 monclient: shutdown 2024-09-17T18:29:44.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.600+0000 7f5a0818a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:29:44.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.600+0000 7f5a0818a640 1 -- 172.21.15.121:0/1034306432 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a0009ee70 msgr2=0x7f5a00139840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:44.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.600+0000 7f5a0818a640 1 --2- 172.21.15.121:0/1034306432 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a0009ee70 0x7f5a00139840 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f5a0009e0a0 tx=0x7f59f402e300 comp rx=0 tx=0).stop 2024-09-17T18:29:44.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.600+0000 7f5a0818a640 1 -- 172.21.15.121:0/1034306432 shutdown_connections 2024-09-17T18:29:44.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.600+0000 7f5a0818a640 1 --2- 172.21.15.121:0/1034306432 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a0009f2c0 0x7f5a0013bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:44.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.600+0000 7f5a0818a640 1 --2- 172.21.15.121:0/1034306432 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a0009ee70 0x7f5a00139840 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:44.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.600+0000 7f5a0818a640 1 --2- 172.21.15.121:0/1034306432 >> v2:172.21.15.173:3300/0 conn(0x7f5a0009e550 0x7f5a0009e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:44.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.600+0000 7f5a0818a640 1 -- 172.21.15.121:0/1034306432 >> 172.21.15.121:0/1034306432 conn(0x7f5a000a7830 msgr2=0x7f5a000a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:29:44.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.600+0000 7f5a0818a640 1 -- 172.21.15.121:0/1034306432 shutdown_connections 2024-09-17T18:29:44.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.600+0000 7f5a0818a640 1 -- 172.21.15.121:0/1034306432 wait complete. 2024-09-17T18:29:44.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.600+0000 7f5a0818a640 10 monclient: build_initial_monmap 2024-09-17T18:29:44.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.600+0000 7f5a0818a640 10 monclient: monmap: 2024-09-17T18:29:44.614 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:29:44.614 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:29:44.614 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:29:44.614 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:29:44.614 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:29:44.614 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:44.614 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:29:44.615 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:29:44.615 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:29:44.615 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:44.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a0818a640 1 librados: starting msgr at 2024-09-17T18:29:44.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a0818a640 1 librados: starting objecter 2024-09-17T18:29:44.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a0818a640 1 Processor -- start 2024-09-17T18:29:44.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a0818a640 1 -- start start 2024-09-17T18:29:44.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a0818a640 1 librados: setting wanted keys 2024-09-17T18:29:44.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a0818a640 1 librados: calling monclient init 2024-09-17T18:29:44.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a0818a640 10 monclient: init 2024-09-17T18:29:44.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a0818a640 10 monclient: _reopen_session rank -1 2024-09-17T18:29:44.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a0818a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T18:29:44.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a0818a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a0009e550 0x7f5a001c8f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:44.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a0818a640 10 monclient(hunting): picked mon.noname-c con 0x7f5a0009e550 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:29:44.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a0818a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a0009ee70 0x7f5a001b8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:44.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a0818a640 10 monclient(hunting): picked mon.noname-b con 0x7f5a0009ee70 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:29:44.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a0818a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a0009f2c0 0x7f5a001b8e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:44.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a0818a640 10 monclient(hunting): picked mon.noname-a con 0x7f5a0009f2c0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:29:44.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a0818a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:44.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a0818a640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0009de60 con 0x7f5a0009f2c0 2024-09-17T18:29:44.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a0818a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:44.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a0818a640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0009dce0 con 0x7f5a0009ee70 2024-09-17T18:29:44.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a0818a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:44.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a0818a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5a000747e0 con 0x7f5a0009e550 2024-09-17T18:29:44.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a0818a640 10 monclient(hunting): _renew_subs 2024-09-17T18:29:44.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a0818a640 10 monclient(hunting): authenticate will time out at 1388.317713s 2024-09-17T18:29:44.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a05f26640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a0009e550 0x7f5a001c8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:44.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a06727640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a0009f2c0 0x7f5a001b8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:44.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a06727640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a0009f2c0 0x7f5a001b8e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39892/0 (socket says 172.21.15.121:39892) 2024-09-17T18:29:44.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a05f26640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a0009e550 0x7f5a001c8f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:43122/0 (socket says 172.21.15.121:43122) 2024-09-17T18:29:44.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a06727640 1 -- 172.21.15.121:0/1330092372 learned_addr learned my addr 172.21.15.121:0/1330092372 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:29:44.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a06727640 10 monclient(hunting): get_auth_request con 0x7f5a0009f2c0 auth_method 0 2024-09-17T18:29:44.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a06727640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:44.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a06727640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:44.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a06727640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:44.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a05f26640 10 monclient(hunting): get_auth_request con 0x7f5a0009e550 auth_method 0 2024-09-17T18:29:44.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a05f26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:44.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a05f26640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:44.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a05f26640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:44.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a05725640 1 -- 172.21.15.121:0/1330092372 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a0009ee70 msgr2=0x7f5a001b8950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.173:3300/0 2024-09-17T18:29:44.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a05725640 1 --2- 172.21.15.121:0/1330092372 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a0009ee70 0x7f5a001b8950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:29:44.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a06727640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:44.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a06727640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:44.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a06727640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:44.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a05f26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:44.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a05f26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:44.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.601+0000 7f5a05f26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:44.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.602+0000 7f5a06727640 10 monclient(hunting): handle_auth_done global_id 44610 payload 824 2024-09-17T18:29:44.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.602+0000 7f5a06727640 1 -- 172.21.15.121:0/1330092372 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a0009e550 msgr2=0x7f5a001c8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:44.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.602+0000 7f5a06727640 1 --2- 172.21.15.121:0/1330092372 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a0009e550 0x7f5a001c8f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:44.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.602+0000 7f5a06727640 1 -- 172.21.15.121:0/1330092372 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a0009ee70 msgr2=0x7f5a001b8950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:29:44.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.602+0000 7f5a06727640 1 --2- 172.21.15.121:0/1330092372 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a0009ee70 0x7f5a001b8950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:44.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.602+0000 7f5a06727640 10 monclient: _finish_hunting 0 2024-09-17T18:29:44.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.602+0000 7f5a06727640 1 monclient: found mon.noname-a 2024-09-17T18:29:44.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.602+0000 7f5a06727640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:29:44.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.602+0000 7f5a06727640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:29:44.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.602+0000 7f5a06727640 1 -- 172.21.15.121:0/1330092372 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a001a7060 con 0x7f5a0009f2c0 2024-09-17T18:29:44.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.602+0000 7f5a05f26640 1 --2- 172.21.15.121:0/1330092372 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a0009e550 0x7f5a001c8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:29:44.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.604+0000 7f5a06727640 1 -- 172.21.15.121:0/1330092372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a0009f2c0 msgr2=0x7f5a001b8e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:44.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.604+0000 7f5a06727640 1 --2- 172.21.15.121:0/1330092372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a0009f2c0 0x7f5a001b8e90 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f59f800fbb0 tx=0x7f59f800d720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:44.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.604+0000 7f59fe7fc640 10 client.?.objecter ms_handle_connect 0x7f5a0009f2c0 2024-09-17T18:29:44.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.604+0000 7f59fe7fc640 10 client.?.objecter resend_mon_ops 2024-09-17T18:29:44.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.604+0000 7f59fe7fc640 1 -- 172.21.15.121:0/1330092372 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f8004220 con 0x7f5a0009f2c0 2024-09-17T18:29:44.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.604+0000 7f59fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:44.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.604+0000 7f59fe7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:29:44.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.604+0000 7f59fe7fc640 10 monclient: dump: 2024-09-17T18:29:44.619 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:44.619 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:44.619 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:44.619 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:44.619 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:44.619 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:44.620 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:44.620 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:44.620 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:44.620 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:44.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.604+0000 7f59fe7fc640 10 monclient: _finish_auth 0 2024-09-17T18:29:44.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.604+0000 7f59fe7fc640 10 monclient: _check_auth_tickets 2024-09-17T18:29:44.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.604+0000 7f59fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:29:44.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.604+0000 7f59fe7fc640 1 -- 172.21.15.121:0/1330092372 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59f80043c0 con 0x7f5a0009f2c0 2024-09-17T18:29:44.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.604+0000 7f5a0818a640 5 monclient: authenticate success, global_id 44610 2024-09-17T18:29:44.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.604+0000 7f5a0818a640 10 monclient: _renew_subs 2024-09-17T18:29:44.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.604+0000 7f5a0818a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:29:44.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.604+0000 7f5a0818a640 1 -- 172.21.15.121:0/1330092372 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a001a7340 con 0x7f5a0009f2c0 2024-09-17T18:29:44.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.604+0000 7f59fe7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:29:44.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.604+0000 7f59fe7fc640 1 -- 172.21.15.121:0/1330092372 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f8010870 con 0x7f5a0009f2c0 2024-09-17T18:29:44.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.604+0000 7f59fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:44.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.604+0000 7f59fe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:29:44.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.604+0000 7f59fe7fc640 10 monclient: dump: 2024-09-17T18:29:44.621 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:44.621 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:44.621 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:44.621 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:44.621 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:44.621 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:44.621 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:44.621 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:44.621 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:44.621 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:44.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.604+0000 7f59fffff640 4 set_mon_vals no callback set 2024-09-17T18:29:44.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.604+0000 7f5a0818a640 10 client.44610.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:29:44.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.604+0000 7f5a0818a640 10 monclient: _renew_subs 2024-09-17T18:29:44.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.604+0000 7f5a0818a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:29:44.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.604+0000 7f5a0818a640 1 -- 172.21.15.121:0/1330092372 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a0009e240 con 0x7f5a0009f2c0 2024-09-17T18:29:44.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.604+0000 7f5a0818a640 1 librados: init done 2024-09-17T18:29:44.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.605+0000 7f5a0818a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:29:44.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.605+0000 7f5a0818a640 1 --2- 172.21.15.121:0/1330092372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a001a3570 0x7f5a001a3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:44.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.605+0000 7f5a0818a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:44.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.605+0000 7f5a0818a640 1 -- 172.21.15.121:0/1330092372 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0009e930 con 0x7f5a001a3570 2024-09-17T18:29:44.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.605+0000 7f5a05f26640 1 --2- 172.21.15.121:0/1330092372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a001a3570 0x7f5a001a3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:44.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.605+0000 7f59fe7fc640 1 -- 172.21.15.121:0/1330092372 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f59f801f020 con 0x7f5a0009f2c0 2024-09-17T18:29:44.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.605+0000 7f5a05f26640 10 monclient: get_auth_request con 0x7f5a001a3570 auth_method 0 2024-09-17T18:29:44.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.605+0000 7f5a05f26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:44.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.605+0000 7f5a05f26640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:44.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.605+0000 7f5a05f26640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:44.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.605+0000 7f5a05f26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:44.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.605+0000 7f5a05f26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:44.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.605+0000 7f5a05f26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:44.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.605+0000 7f59fe7fc640 1 --2- 172.21.15.121:0/1330092372 >> v2:172.21.15.173:6812/2423786037 conn(0x7f59ec040120 0x7f59ec0425e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:44.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.605+0000 7f5a05f26640 10 monclient(hunting): handle_auth_done global_id 44613 payload 824 2024-09-17T18:29:44.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.605+0000 7f5a05f26640 1 -- 172.21.15.121:0/1330092372 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5a001a4fe0 con 0x7f5a001a3570 2024-09-17T18:29:44.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.605+0000 7f59fe7fc640 1 -- 172.21.15.121:0/1330092372 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(413..413 src has 247..413) ==== 5358+0+0 (secure 0 0 0) 0x7f59f8053c20 con 0x7f5a0009f2c0 2024-09-17T18:29:44.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.605+0000 7f59fe7fc640 10 client.44610.objecter ms_dispatch 0x7f5a00000b90 osd_map(413..413 src has 247..413) 2024-09-17T18:29:44.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.605+0000 7f59fe7fc640 3 client.44610.objecter handle_osd_map got epochs [413,413] > 0 2024-09-17T18:29:44.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.605+0000 7f59fe7fc640 3 client.44610.objecter handle_osd_map decoding full epoch 413 2024-09-17T18:29:44.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.605+0000 7f59fe7fc640 20 client.44610.objecter dump_active .. 0 homeless 2024-09-17T18:29:44.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.605+0000 7f5a05725640 1 --2- 172.21.15.121:0/1330092372 >> v2:172.21.15.173:6812/2423786037 conn(0x7f59ec040120 0x7f59ec0425e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:44.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.605+0000 7f5a05725640 10 monclient: get_auth_request con 0x7f59ec040120 auth_method 0 2024-09-17T18:29:44.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.607+0000 7f5a05f26640 1 -- 172.21.15.121:0/1330092372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a001a3570 msgr2=0x7f5a001a3950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:44.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.607+0000 7f59fe7fc640 10 client.44610.objecter ms_handle_connect 0x7f5a001a3570 2024-09-17T18:29:44.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.607+0000 7f59fe7fc640 10 client.44610.objecter resend_mon_ops 2024-09-17T18:29:44.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.607+0000 7f5a05f26640 1 --2- 172.21.15.121:0/1330092372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a001a3570 0x7f5a001a3950 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f59f0007590 tx=0x7f59f0007a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:44.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.607+0000 7f59fe7fc640 1 -- 172.21.15.121:0/1330092372 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f001a070 con 0x7f5a001a3570 2024-09-17T18:29:44.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.607+0000 7f59fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:44.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.607+0000 7f59fe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:29:44.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.607+0000 7f59fe7fc640 10 monclient: dump: 2024-09-17T18:29:44.624 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:44.624 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:44.624 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:44.624 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:44.624 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:44.624 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:44.624 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:44.625 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:44.625 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:44.625 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:44.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.608+0000 7f59fe7fc640 1 -- 172.21.15.121:0/1330092372 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f59f000aba0 con 0x7f5a001a3570 2024-09-17T18:29:44.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.608+0000 7f59fe7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:29:44.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.608+0000 7f59fe7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:29:44.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.608+0000 7f59fe7fc640 1 -- 172.21.15.121:0/1330092372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a001a3570 msgr2=0x7f5a001a3950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:44.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.608+0000 7f59fe7fc640 1 --2- 172.21.15.121:0/1330092372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a001a3570 0x7f5a001a3950 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f59f0007590 tx=0x7f59f0007a50 comp rx=0 tx=0).stop 2024-09-17T18:29:44.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.608+0000 7f59fe7fc640 1 -- 172.21.15.121:0/1330092372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a001a3570 msgr2=0x7f5a001a3950 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:29:44.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.608+0000 7f59fe7fc640 1 --2- 172.21.15.121:0/1330092372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a001a3570 0x7f5a001a3950 secure :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f59f0007590 tx=0x7f59f0007a50 comp rx=0 tx=0).stop 2024-09-17T18:29:44.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.608+0000 7f59fe7fc640 10 client.44610.objecter ms_handle_connect 0x7f59ec040120 2024-09-17T18:29:44.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.608+0000 7f5a05725640 1 --2- 172.21.15.121:0/1330092372 >> v2:172.21.15.173:6812/2423786037 conn(0x7f59ec040120 0x7f59ec0425e0 secure :-1 s=READY pgs=4022 cs=0 l=1 rev1=1 crypto rx=0x7f59f4037000 tx=0x7f59f4032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:44.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.614+0000 7f5a0818a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:29:44.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.614+0000 7f5a0818a640 1 --2- 172.21.15.121:0/1330092372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a001a3e90 0x7f5a00070900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:44.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.614+0000 7f5a0818a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:44.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.614+0000 7f5a0818a640 1 -- 172.21.15.121:0/1330092372 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0006cd40 con 0x7f5a001a3e90 2024-09-17T18:29:44.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.614+0000 7f5a05f26640 1 --2- 172.21.15.121:0/1330092372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a001a3e90 0x7f5a00070900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:44.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.614+0000 7f5a05f26640 10 monclient: get_auth_request con 0x7f5a001a3e90 auth_method 0 2024-09-17T18:29:44.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.614+0000 7f5a05f26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:44.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.614+0000 7f5a05f26640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:44.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.614+0000 7f5a05f26640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:44.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.614+0000 7f5a05f26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:44.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.614+0000 7f5a05f26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:44.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.614+0000 7f5a05f26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:44.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.614+0000 7f5a05f26640 10 monclient(hunting): handle_auth_done global_id 44616 payload 824 2024-09-17T18:29:44.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.614+0000 7f5a05f26640 1 -- 172.21.15.121:0/1330092372 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5a001a42a0 con 0x7f5a001a3e90 2024-09-17T18:29:44.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.616+0000 7f59fe7fc640 10 client.44610.objecter ms_handle_connect 0x7f5a001a3e90 2024-09-17T18:29:44.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.616+0000 7f59fe7fc640 10 client.44610.objecter resend_mon_ops 2024-09-17T18:29:44.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.616+0000 7f5a05f26640 1 -- 172.21.15.121:0/1330092372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a001a3e90 msgr2=0x7f5a00070900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:44.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.616+0000 7f5a05f26640 1 --2- 172.21.15.121:0/1330092372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a001a3e90 0x7f5a00070900 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f59f000b460 tx=0x7f59f0005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:44.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.616+0000 7f59fe7fc640 1 -- 172.21.15.121:0/1330092372 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f001a030 con 0x7f5a001a3e90 2024-09-17T18:29:44.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.616+0000 7f59fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:44.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.616+0000 7f59fe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:29:44.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.616+0000 7f59fe7fc640 10 monclient: dump: 2024-09-17T18:29:44.629 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:44.629 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:44.629 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:44.629 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:44.629 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:44.629 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:44.629 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:44.629 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:44.629 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:44.629 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:44.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.617+0000 7f59fe7fc640 1 -- 172.21.15.121:0/1330092372 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7f59f001dd50 con 0x7f5a001a3e90 2024-09-17T18:29:44.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.617+0000 7f59fe7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:29:44.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.617+0000 7f59fe7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:29:44.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.617+0000 7f59fe7fc640 1 -- 172.21.15.121:0/1330092372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a001a3e90 msgr2=0x7f5a00070900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:44.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.617+0000 7f59fe7fc640 1 --2- 172.21.15.121:0/1330092372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a001a3e90 0x7f5a00070900 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f59f000b460 tx=0x7f59f0005d50 comp rx=0 tx=0).stop 2024-09-17T18:29:44.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.617+0000 7f59fe7fc640 1 -- 172.21.15.121:0/1330092372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a001a3e90 msgr2=0x7f5a00070900 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:29:44.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.617+0000 7f59fe7fc640 1 --2- 172.21.15.121:0/1330092372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a001a3e90 0x7f5a00070900 secure :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f59f000b460 tx=0x7f59f0005d50 comp rx=0 tx=0).stop 2024-09-17T18:29:44.630 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:29:44.630 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:29:44.630 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:29:44.631 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "leader", 2024-09-17T18:29:44.631 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 42, 2024-09-17T18:29:44.631 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 410280, 2024-09-17T18:29:44.631 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:29:44.631 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:29:44.631 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:29:44.631 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:29:44.631 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 2, 2024-09-17T18:29:44.631 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:29:44.631 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:29:44.631 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:29:44.631 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:29:44.631 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:29:44.632 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:29:44.632 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:29:44.632 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:29:44.632 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:29:44.632 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:29:44.632 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:29:44.632 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:29:44.632 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:29:44.632 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:29:44.632 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:29:44.632 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:29:44.632 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:29:44.632 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:29:44.632 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:29:44.633 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:29:44.633 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:29:44.633 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:29:44.633 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:29:44.633 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:29:44.633 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:29:44.633 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:29:44.633 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:29:44.633 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:29:44.633 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:44.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:44.633 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:29:44.633 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:29:44.633 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:29:44.634 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:29:44.634 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:29:44.634 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:29:44.634 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:29:44.634 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:29:44.634 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:29:44.634 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:29:44.634 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:29:44.634 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:29:44.634 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:29:44.634 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:29:44.634 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:29:44.634 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:29:44.634 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:29:44.634 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:29:44.635 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:29:44.635 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:29:44.635 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:29:44.635 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:29:44.635 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:29:44.635 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:29:44.635 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:29:44.635 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:29:44.635 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:29:44.635 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:29:44.635 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:29:44.635 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:29:44.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:44.635 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:29:44.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:44.635 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:29:44.636 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:29:44.636 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:29:44.636 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:29:44.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:44.636 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:29:44.636 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:29:44.636 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:44.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:44.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:44.636 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:29:44.636 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:29:44.636 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:44.636 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:44.636 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:44.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:44.637 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:29:44.637 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:29:44.637 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:29:44.637 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:29:44.637 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:29:44.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:44.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:44.637 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:29:44.637 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:29:44.637 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:29:44.637 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:29:44.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:44.637 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:29:44.638 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:29:44.638 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:44.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:44.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:44.638 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:29:44.638 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:29:44.638 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:44.638 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:44.638 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:44.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:44.638 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:29:44.638 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:29:44.638 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:29:44.638 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:29:44.639 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:29:44.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:44.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:44.639 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:29:44.639 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:29:44.639 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:29:44.639 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:29:44.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:44.639 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:29:44.639 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:29:44.639 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:44.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:44.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:44.639 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:29:44.640 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:29:44.640 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:44.640 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:44.640 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:44.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:44.640 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:29:44.640 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:29:44.640 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:29:44.640 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:29:44.640 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:29:44.640 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:44.640 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:44.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:44.640 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:29:44.641 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:29:44.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:44.641 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:29:44.641 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:29:44.641 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:29:44.641 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:44.641 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:29:44.641 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:29:44.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:44.641 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:29:44.641 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:29:44.641 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 4 2024-09-17T18:29:44.641 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:44.641 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:29:44.642 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:29:44.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:44.642 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:29:44.642 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:29:44.642 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 6 2024-09-17T18:29:44.642 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:44.642 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:44.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:44.642 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:29:44.642 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:29:44.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.617+0000 7f5a0818a640 10 librados: watch_flush enter 2024-09-17T18:29:44.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.617+0000 7f5a0818a640 10 librados: watch_flush exit 2024-09-17T18:29:44.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.617+0000 7f5a0818a640 20 client.44610.objecter shutdown clearing up homeless session... 2024-09-17T18:29:44.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.617+0000 7f5a0818a640 10 client.44610.objecter successfully canceled tick 2024-09-17T18:29:44.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.617+0000 7f5a0818a640 1 -- 172.21.15.121:0/1330092372 >> v2:172.21.15.173:6812/2423786037 conn(0x7f59ec040120 msgr2=0x7f59ec0425e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:44.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.617+0000 7f5a0818a640 1 --2- 172.21.15.121:0/1330092372 >> v2:172.21.15.173:6812/2423786037 conn(0x7f59ec040120 0x7f59ec0425e0 secure :-1 s=READY pgs=4022 cs=0 l=1 rev1=1 crypto rx=0x7f59f4037000 tx=0x7f59f4032000 comp rx=0 tx=0).stop 2024-09-17T18:29:44.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.617+0000 7f5a0818a640 10 monclient: shutdown 2024-09-17T18:29:44.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.617+0000 7f5a0818a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:29:44.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.617+0000 7f5a0818a640 1 -- 172.21.15.121:0/1330092372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a0009f2c0 msgr2=0x7f5a001b8e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:44.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.617+0000 7f5a06727640 1 -- 172.21.15.121:0/1330092372 reap_dead start 2024-09-17T18:29:44.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.617+0000 7f5a0818a640 1 --2- 172.21.15.121:0/1330092372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a0009f2c0 0x7f5a001b8e90 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f59f800fbb0 tx=0x7f59f800d720 comp rx=0 tx=0).stop 2024-09-17T18:29:44.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.617+0000 7f5a0818a640 1 -- 172.21.15.121:0/1330092372 shutdown_connections 2024-09-17T18:29:44.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.617+0000 7f5a0818a640 1 --2- 172.21.15.121:0/1330092372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a0009f2c0 0x7f5a001b8e90 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:44.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.617+0000 7f5a0818a640 1 -- 172.21.15.121:0/1330092372 >> 172.21.15.121:0/1330092372 conn(0x7f5a000a7830 msgr2=0x7f5a001bb740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:29:44.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.617+0000 7f5a0818a640 1 -- 172.21.15.121:0/1330092372 shutdown_connections 2024-09-17T18:29:44.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.617+0000 7f5a0818a640 1 -- 172.21.15.121:0/1330092372 wait complete. 2024-09-17T18:29:44.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.617+0000 7f5a0818a640 1 librados: shutdown 2024-09-17T18:29:44.645 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T18:29:44.646 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-17T18:29:44.646 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T18:29:44.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.742+0000 7fbc5fe08640 10 monclient: get_monmap_and_config 2024-09-17T18:29:44.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.742+0000 7fbc5fe08640 10 monclient: build_initial_monmap 2024-09-17T18:29:44.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.742+0000 7fbc5fe08640 10 monclient: monmap: 2024-09-17T18:29:44.744 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:29:44.745 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:44.745 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:29:44.743319+0000 2024-09-17T18:29:44.745 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:29:44.743319+0000 2024-09-17T18:29:44.745 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:29:44.745 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:44.745 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:29:44.745 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:29:44.745 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:29:44.745 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:29:44.745 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:29:44.745 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:44.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.743+0000 7fbc5fe08640 1 Processor -- start 2024-09-17T18:29:44.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.743+0000 7fbc5fe08640 1 -- start start 2024-09-17T18:29:44.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.743+0000 7fbc5fe08640 10 monclient: init 2024-09-17T18:29:44.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.743+0000 7fbc5fe08640 10 monclient: _reopen_session rank -1 2024-09-17T18:29:44.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.743+0000 7fbc5fe08640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-17T18:29:44.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.743+0000 7fbc5fe08640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbc580a0a00 addr v1:172.21.15.173:6789/0 2024-09-17T18:29:44.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.743+0000 7fbc5fe08640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fbc58137c90 0x7fbc5813a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:44.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.743+0000 7fbc5fe08640 10 monclient(hunting): picked mon.noname-c con 0x7fbc58137c90 addr v2:172.21.15.173:3300/0 2024-09-17T18:29:44.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.743+0000 7fbc5fe08640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbc5813a5c0 addr v1:172.21.15.121:6789/0 2024-09-17T18:29:44.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.743+0000 7fbc5fe08640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc580a3820 con 0x7fbc5813a5c0 2024-09-17T18:29:44.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.743+0000 7fbc5fe08640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc580a2ea0 con 0x7fbc580a0a00 2024-09-17T18:29:44.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.743+0000 7fbc5fe08640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:44.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.743+0000 7fbc5fe08640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fbc5809e800 con 0x7fbc58137c90 2024-09-17T18:29:44.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.743+0000 7fbc5fe08640 10 monclient(hunting): _renew_subs 2024-09-17T18:29:44.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.743+0000 7fbc5fe08640 10 monclient(hunting): authenticate will time out at 1388.459930s 2024-09-17T18:29:44.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.744+0000 7fbc5d3a3640 1 -- >> v2:172.21.15.173:3300/0 conn(0x7fbc58137c90 msgr2=0x7fbc5813a080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.173:3300/0 2024-09-17T18:29:44.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.744+0000 7fbc5dba4640 1 -- >> v1:172.21.15.173:6789/0 conn(0x7fbc580a0a00 legacy=0x7fbc580a0de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.173:6789/0 2024-09-17T18:29:44.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.744+0000 7fbc5e3a5640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fbc5813a5c0 0x7fbc5813c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:59918/0 (socket says 172.21.15.121:59918) 2024-09-17T18:29:44.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.744+0000 7fbc5d3a3640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fbc58137c90 0x7fbc5813a080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:29:44.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.744+0000 7fbc5e3a5640 1 -- 172.21.15.121:0/3066072125 learned_addr learned my addr 172.21.15.121:0/3066072125 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:29:44.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5e3a5640 1 -- 172.21.15.121:0/3066072125 >> v1:172.21.15.121:6789/0 conn(0x7fbc5813a5c0 legacy=0x7fbc5813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:44.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5cba2640 1 -- 172.21.15.121:0/3066072125 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3814381069 0 0) 0x7fbc500017d0 con 0x7fbc5813a5c0 2024-09-17T18:29:44.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5cba2640 10 monclient(hunting): _negotiate 2024-09-17T18:29:44.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5cba2640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:44.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5cba2640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:44.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5cba2640 10 monclient(hunting): my global_id is 44619 2024-09-17T18:29:44.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5cba2640 1 -- 172.21.15.121:0/3066072125 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc44003520 con 0x7fbc5813a5c0 2024-09-17T18:29:44.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5cba2640 1 -- 172.21.15.121:0/3066072125 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2823959835 0 0) 0x7fbc44003520 con 0x7fbc5813a5c0 2024-09-17T18:29:44.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5cba2640 1 -- 172.21.15.121:0/3066072125 >> v2:172.21.15.173:3300/0 conn(0x7fbc58137c90 msgr2=0x7fbc5813a080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:29:44.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5cba2640 1 --2- 172.21.15.121:0/3066072125 >> v2:172.21.15.173:3300/0 conn(0x7fbc58137c90 0x7fbc5813a080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:44.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5cba2640 1 -- 172.21.15.121:0/3066072125 >> v1:172.21.15.173:6789/0 conn(0x7fbc580a0a00 legacy=0x7fbc580a0de0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:29:44.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5cba2640 10 monclient: _finish_hunting 0 2024-09-17T18:29:44.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5cba2640 1 monclient: found mon.noname-a-legacy 2024-09-17T18:29:44.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5cba2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:29:44.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5cba2640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.121:6789/0 2024-09-17T18:29:44.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5cba2640 1 -- 172.21.15.121:0/3066072125 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc580a2c30 con 0x7fbc5813a5c0 2024-09-17T18:29:44.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5cba2640 10 monclient: _finish_auth 0 2024-09-17T18:29:44.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5cba2640 10 monclient: _check_auth_tickets 2024-09-17T18:29:44.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5cba2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:29:44.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5cba2640 1 -- 172.21.15.121:0/3066072125 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4002019445 0 0) 0x7fbc50001ef0 con 0x7fbc5813a5c0 2024-09-17T18:29:44.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5fe08640 5 monclient: authenticate success, global_id 44619 2024-09-17T18:29:44.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5fe08640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:29:44.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5cba2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:44.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5cba2640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T18:29:44.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5cba2640 10 monclient: dump: 2024-09-17T18:29:44.754 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:44.755 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:44.755 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:44.755 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:44.755 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:44.755 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:44.755 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:44.755 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:44.755 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:44.755 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:44.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5cba2640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] but i'm connected to v1:172.21.15.121:6789/0, reconnecting 2024-09-17T18:29:44.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5cba2640 10 monclient: _reopen_session rank -1 2024-09-17T18:29:44.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5cba2640 1 -- 172.21.15.121:0/3066072125 >> v1:172.21.15.121:6789/0 conn(0x7fbc5813a5c0 legacy=0x7fbc5813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:44.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5cba2640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T18:29:44.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5cba2640 1 --2- 172.21.15.121:0/3066072125 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc44003aa0 0x7fbc44003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:44.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5cba2640 10 monclient(hunting): picked mon.a con 0x7fbc44003aa0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:29:44.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5cba2640 1 --2- 172.21.15.121:0/3066072125 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc44004800 0x7fbc44004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:44.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5cba2640 10 monclient(hunting): picked mon.c con 0x7fbc44004800 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:29:44.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5cba2640 1 --2- 172.21.15.121:0/3066072125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc440099b0 0x7fbc4400bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:44.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5cba2640 10 monclient(hunting): picked mon.b con 0x7fbc440099b0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:29:44.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5cba2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:44.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5cba2640 1 -- 172.21.15.121:0/3066072125 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fbc5809e800 con 0x7fbc44003aa0 2024-09-17T18:29:44.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5cba2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:44.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5cba2640 1 -- 172.21.15.121:0/3066072125 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4400c3f0 con 0x7fbc440099b0 2024-09-17T18:29:44.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5cba2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:44.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5cba2640 1 -- 172.21.15.121:0/3066072125 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbc4400c5f0 con 0x7fbc44004800 2024-09-17T18:29:44.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5cba2640 10 monclient(hunting): _renew_subs 2024-09-17T18:29:44.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5cba2640 10 monclient(hunting): _finish_auth 0 2024-09-17T18:29:44.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5fe08640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-17T18:29:44.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5dba4640 1 --2- 172.21.15.121:0/3066072125 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc44003aa0 0x7fbc44003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:44.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5d3a3640 1 --2- 172.21.15.121:0/3066072125 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc44004800 0x7fbc44004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:44.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5dba4640 10 monclient(hunting): get_auth_request con 0x7fbc44003aa0 auth_method 0 2024-09-17T18:29:44.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5dba4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:44.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5dba4640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:44.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5dba4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:29:44.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5d3a3640 10 monclient(hunting): get_auth_request con 0x7fbc44004800 auth_method 0 2024-09-17T18:29:44.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5d3a3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:44.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5d3a3640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:44.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5d3a3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:29:44.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5e3a5640 1 -- 172.21.15.121:0/3066072125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc440099b0 msgr2=0x7fbc4400bda0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.173:3300/0 2024-09-17T18:29:44.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5e3a5640 1 --2- 172.21.15.121:0/3066072125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc440099b0 0x7fbc4400bda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:29:44.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5dba4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:44.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5dba4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:44.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5dba4640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:29:44.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5d3a3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:44.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5d3a3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:44.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5d3a3640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:29:44.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5dba4640 10 monclient(hunting): handle_auth_done global_id 44619 payload 293 2024-09-17T18:29:44.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5dba4640 1 -- 172.21.15.121:0/3066072125 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc44004800 msgr2=0x7fbc44004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:29:44.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5dba4640 1 --2- 172.21.15.121:0/3066072125 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc44004800 0x7fbc44004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:44.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.746+0000 7fbc5dba4640 1 -- 172.21.15.121:0/3066072125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc440099b0 msgr2=0x7fbc4400bda0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:29:44.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.747+0000 7fbc5dba4640 1 --2- 172.21.15.121:0/3066072125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc440099b0 0x7fbc4400bda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:44.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.747+0000 7fbc5dba4640 10 monclient: _finish_hunting 0 2024-09-17T18:29:44.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.747+0000 7fbc5dba4640 1 monclient: found mon.a 2024-09-17T18:29:44.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.747+0000 7fbc5e3a5640 1 -- 172.21.15.121:0/3066072125 reap_dead start 2024-09-17T18:29:44.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.747+0000 7fbc5dba4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:29:44.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.747+0000 7fbc5dba4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:29:44.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.747+0000 7fbc5dba4640 1 -- 172.21.15.121:0/3066072125 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc44003520 con 0x7fbc44003aa0 2024-09-17T18:29:44.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.747+0000 7fbc5dba4640 10 monclient: _finish_auth 0 2024-09-17T18:29:44.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.747+0000 7fbc5dba4640 10 monclient: _check_auth_tickets 2024-09-17T18:29:44.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.747+0000 7fbc5dba4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:29:44.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.749+0000 7fbc5dba4640 1 -- 172.21.15.121:0/3066072125 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc44003aa0 msgr2=0x7fbc44003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:44.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.749+0000 7fbc5dba4640 1 --2- 172.21.15.121:0/3066072125 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc44003aa0 0x7fbc44003ea0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fbc54009830 tx=0x7fbc44005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ef558967f420701 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:44.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.749+0000 7fbc5cba2640 1 -- 172.21.15.121:0/3066072125 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc54009430 con 0x7fbc44003aa0 2024-09-17T18:29:44.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.749+0000 7fbc5cba2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:44.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.749+0000 7fbc5cba2640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:29:44.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.749+0000 7fbc5cba2640 10 monclient: dump: 2024-09-17T18:29:44.770 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:44.770 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:44.770 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:44.770 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:44.770 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:44.770 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:44.770 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:44.770 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:44.771 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:44.771 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:44.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.749+0000 7fbc5cba2640 1 -- 172.21.15.121:0/3066072125 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc5402e070 con 0x7fbc44003aa0 2024-09-17T18:29:44.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.749+0000 7fbc5fe08640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:29:44.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.749+0000 7fbc5cba2640 10 monclient: handle_config config(0 keys) 2024-09-17T18:29:44.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.749+0000 7fbc5fe08640 10 monclient: get_monmap_and_config success 2024-09-17T18:29:44.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.749+0000 7fbc5cba2640 1 -- 172.21.15.121:0/3066072125 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc5402a850 con 0x7fbc44003aa0 2024-09-17T18:29:44.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.749+0000 7fbc5cba2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:44.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.749+0000 7fbc5fe08640 4 set_mon_vals no callback set 2024-09-17T18:29:44.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.749+0000 7fbc5cba2640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:29:44.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.749+0000 7fbc5cba2640 10 monclient: dump: 2024-09-17T18:29:44.771 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:44.772 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:44.772 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:44.772 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:44.772 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:44.772 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:44.772 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:44.772 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:44.772 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:44.772 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:44.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.750+0000 7fbc5fe08640 10 monclient: shutdown 2024-09-17T18:29:44.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.750+0000 7fbc5fe08640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:29:44.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.750+0000 7fbc5fe08640 1 -- 172.21.15.121:0/3066072125 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc44003aa0 msgr2=0x7fbc44003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:44.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.750+0000 7fbc5fe08640 1 --2- 172.21.15.121:0/3066072125 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc44003aa0 0x7fbc44003ea0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fbc54009830 tx=0x7fbc44005830 comp rx=0 tx=0).stop 2024-09-17T18:29:44.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.750+0000 7fbc5fe08640 1 -- 172.21.15.121:0/3066072125 shutdown_connections 2024-09-17T18:29:44.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.750+0000 7fbc5fe08640 1 --2- 172.21.15.121:0/3066072125 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc44003aa0 0x7fbc44003ea0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:44.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.750+0000 7fbc5fe08640 1 -- 172.21.15.121:0/3066072125 >> 172.21.15.121:0/3066072125 conn(0x7fbc580a7830 msgr2=0x7fbc580a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:29:44.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.750+0000 7fbc5fe08640 1 -- 172.21.15.121:0/3066072125 shutdown_connections 2024-09-17T18:29:44.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.750+0000 7fbc5fe08640 1 -- 172.21.15.121:0/3066072125 wait complete. 2024-09-17T18:29:44.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.750+0000 7fbc5fe08640 10 monclient: build_initial_monmap 2024-09-17T18:29:44.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.751+0000 7fbc5fe08640 10 monclient: monmap: 2024-09-17T18:29:44.773 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:29:44.773 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:29:44.773 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:29:44.773 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:29:44.774 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:29:44.774 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:44.774 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:29:44.774 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:29:44.774 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:29:44.774 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:44.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.751+0000 7fbc5fe08640 1 librados: starting msgr at 2024-09-17T18:29:44.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.751+0000 7fbc5fe08640 1 librados: starting objecter 2024-09-17T18:29:44.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.751+0000 7fbc5fe08640 1 Processor -- start 2024-09-17T18:29:44.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.751+0000 7fbc5fe08640 1 -- start start 2024-09-17T18:29:44.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.751+0000 7fbc5fe08640 1 librados: setting wanted keys 2024-09-17T18:29:44.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.751+0000 7fbc5fe08640 1 librados: calling monclient init 2024-09-17T18:29:44.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.751+0000 7fbc5fe08640 10 monclient: init 2024-09-17T18:29:44.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.751+0000 7fbc5fe08640 10 monclient: _reopen_session rank -1 2024-09-17T18:29:44.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.751+0000 7fbc5fe08640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T18:29:44.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.751+0000 7fbc5fe08640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc44003aa0 0x7fbc58134c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:44.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.751+0000 7fbc5fe08640 10 monclient(hunting): picked mon.noname-c con 0x7fbc44003aa0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:29:44.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.751+0000 7fbc5fe08640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc58123ba0 0x7fbc58121ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:44.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.751+0000 7fbc5fe08640 10 monclient(hunting): picked mon.noname-a con 0x7fbc58123ba0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:29:44.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.751+0000 7fbc5fe08640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc58135160 0x7fbc58121ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:44.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.751+0000 7fbc5fe08640 10 monclient(hunting): picked mon.noname-b con 0x7fbc58135160 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:29:44.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.751+0000 7fbc5fe08640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:44.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.751+0000 7fbc5fe08640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fbc580a3120 con 0x7fbc58123ba0 2024-09-17T18:29:44.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.751+0000 7fbc5fe08640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:44.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.751+0000 7fbc5fe08640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbc5809ecf0 con 0x7fbc58135160 2024-09-17T18:29:44.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.751+0000 7fbc5fe08640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:44.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.751+0000 7fbc5fe08640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbc580a32a0 con 0x7fbc44003aa0 2024-09-17T18:29:44.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.751+0000 7fbc5fe08640 10 monclient(hunting): _renew_subs 2024-09-17T18:29:44.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.751+0000 7fbc5fe08640 10 monclient(hunting): authenticate will time out at 1388.467918s 2024-09-17T18:29:44.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.751+0000 7fbc5dba4640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc44003aa0 0x7fbc58134c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:44.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.751+0000 7fbc5dba4640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc44003aa0 0x7fbc58134c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:43140/0 (socket says 172.21.15.121:43140) 2024-09-17T18:29:44.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.751+0000 7fbc5dba4640 1 -- 172.21.15.121:0/3760674710 learned_addr learned my addr 172.21.15.121:0/3760674710 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:29:44.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.751+0000 7fbc5dba4640 10 monclient(hunting): get_auth_request con 0x7fbc44003aa0 auth_method 0 2024-09-17T18:29:44.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.751+0000 7fbc5dba4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:44.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.751+0000 7fbc5dba4640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:44.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.751+0000 7fbc5dba4640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:44.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.751+0000 7fbc5d3a3640 1 --2- 172.21.15.121:0/3760674710 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc58123ba0 0x7fbc58121ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:44.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.752+0000 7fbc5d3a3640 10 monclient(hunting): get_auth_request con 0x7fbc58123ba0 auth_method 0 2024-09-17T18:29:44.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.752+0000 7fbc5d3a3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:44.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.752+0000 7fbc5d3a3640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:44.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.752+0000 7fbc5d3a3640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:44.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.752+0000 7fbc5dba4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:44.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.752+0000 7fbc5dba4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:44.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.752+0000 7fbc5e3a5640 1 -- 172.21.15.121:0/3760674710 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc58135160 msgr2=0x7fbc58121ff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.173:3300/0 2024-09-17T18:29:44.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.752+0000 7fbc5dba4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:44.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.752+0000 7fbc5e3a5640 1 --2- 172.21.15.121:0/3760674710 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc58135160 0x7fbc58121ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:29:44.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.752+0000 7fbc5d3a3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:44.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.752+0000 7fbc5d3a3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:44.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.752+0000 7fbc5d3a3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:44.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.752+0000 7fbc5dba4640 10 monclient(hunting): handle_auth_done global_id 64253 payload 824 2024-09-17T18:29:44.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.752+0000 7fbc5dba4640 1 -- 172.21.15.121:0/3760674710 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc58135160 msgr2=0x7fbc58121ff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:29:44.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.752+0000 7fbc5dba4640 1 --2- 172.21.15.121:0/3760674710 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc58135160 0x7fbc58121ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:44.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.752+0000 7fbc5dba4640 1 -- 172.21.15.121:0/3760674710 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc58123ba0 msgr2=0x7fbc58121ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:44.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.752+0000 7fbc5dba4640 1 --2- 172.21.15.121:0/3760674710 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc58123ba0 0x7fbc58121ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:29:44.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.752+0000 7fbc5dba4640 10 monclient: _finish_hunting 0 2024-09-17T18:29:44.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.752+0000 7fbc5dba4640 1 monclient: found mon.noname-c 2024-09-17T18:29:44.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.752+0000 7fbc5dba4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:29:44.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.752+0000 7fbc5dba4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:29:44.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.752+0000 7fbc5dba4640 1 -- 172.21.15.121:0/3760674710 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc5809e240 con 0x7fbc44003aa0 2024-09-17T18:29:44.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.752+0000 7fbc5d3a3640 1 --2- 172.21.15.121:0/3760674710 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc58123ba0 0x7fbc58121ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:29:44.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.754+0000 7fbc5dba4640 1 -- 172.21.15.121:0/3760674710 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc44003aa0 msgr2=0x7fbc58134c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:44.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.754+0000 7fbc5dba4640 1 --2- 172.21.15.121:0/3760674710 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc44003aa0 0x7fbc58134c20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbc54009920 tx=0x7fbc54005940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:44.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.754+0000 7fbc4e7fc640 10 client.?.objecter ms_handle_connect 0x7fbc44003aa0 2024-09-17T18:29:44.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.754+0000 7fbc4e7fc640 10 client.?.objecter resend_mon_ops 2024-09-17T18:29:44.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.754+0000 7fbc4e7fc640 1 -- 172.21.15.121:0/3760674710 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc54043070 con 0x7fbc44003aa0 2024-09-17T18:29:44.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.754+0000 7fbc4e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:44.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.754+0000 7fbc4e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:29:44.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.754+0000 7fbc4e7fc640 10 monclient: dump: 2024-09-17T18:29:44.779 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:44.779 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:44.779 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:44.779 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:44.779 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:44.780 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:44.780 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:44.780 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:44.780 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:44.780 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:44.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.754+0000 7fbc4e7fc640 10 monclient: _finish_auth 0 2024-09-17T18:29:44.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.754+0000 7fbc4e7fc640 10 monclient: _check_auth_tickets 2024-09-17T18:29:44.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.754+0000 7fbc4e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:29:44.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.754+0000 7fbc4e7fc640 1 -- 172.21.15.121:0/3760674710 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc54005270 con 0x7fbc44003aa0 2024-09-17T18:29:44.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.754+0000 7fbc4e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:29:44.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.754+0000 7fbc4e7fc640 1 -- 172.21.15.121:0/3760674710 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc54005640 con 0x7fbc44003aa0 2024-09-17T18:29:44.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.754+0000 7fbc4e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:44.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.754+0000 7fbc4e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:29:44.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.754+0000 7fbc4e7fc640 10 monclient: dump: 2024-09-17T18:29:44.781 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:44.781 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:44.781 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:44.781 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:44.781 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:44.781 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:44.781 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:44.781 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:44.781 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:44.782 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:44.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.754+0000 7fbc5fe08640 5 monclient: authenticate success, global_id 64253 2024-09-17T18:29:44.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.754+0000 7fbc5fe08640 10 monclient: _renew_subs 2024-09-17T18:29:44.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.754+0000 7fbc5fe08640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:29:44.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.754+0000 7fbc5fe08640 1 -- 172.21.15.121:0/3760674710 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc58137890 con 0x7fbc44003aa0 2024-09-17T18:29:44.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.754+0000 7fbc4ffff640 4 set_mon_vals no callback set 2024-09-17T18:29:44.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.754+0000 7fbc5fe08640 10 client.64253.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:29:44.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.754+0000 7fbc5fe08640 10 monclient: _renew_subs 2024-09-17T18:29:44.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.754+0000 7fbc5fe08640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:29:44.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.754+0000 7fbc5fe08640 1 -- 172.21.15.121:0/3760674710 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc58124b00 con 0x7fbc44003aa0 2024-09-17T18:29:44.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.754+0000 7fbc5fe08640 1 librados: init done 2024-09-17T18:29:44.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.755+0000 7fbc4e7fc640 1 -- 172.21.15.121:0/3760674710 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fbc54033470 con 0x7fbc44003aa0 2024-09-17T18:29:44.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.755+0000 7fbc5fe08640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:29:44.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.755+0000 7fbc5fe08640 1 --2- 172.21.15.121:0/3760674710 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc5802d440 0x7fbc5802d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:44.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.755+0000 7fbc5fe08640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:44.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.755+0000 7fbc5fe08640 1 -- 172.21.15.121:0/3760674710 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbc581bfb70 con 0x7fbc5802d440 2024-09-17T18:29:44.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.755+0000 7fbc5d3a3640 1 --2- 172.21.15.121:0/3760674710 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc5802d440 0x7fbc5802d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:44.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.755+0000 7fbc4e7fc640 1 --2- 172.21.15.121:0/3760674710 >> v2:172.21.15.173:6812/2423786037 conn(0x7fbc3c040150 0x7fbc3c042610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:44.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.755+0000 7fbc5d3a3640 10 monclient: get_auth_request con 0x7fbc5802d440 auth_method 0 2024-09-17T18:29:44.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.755+0000 7fbc5d3a3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:44.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.755+0000 7fbc5d3a3640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:44.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.755+0000 7fbc5d3a3640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:44.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.755+0000 7fbc4e7fc640 1 -- 172.21.15.121:0/3760674710 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(413..413 src has 247..413) ==== 5358+0+0 (secure 0 0 0) 0x7fbc540731d0 con 0x7fbc44003aa0 2024-09-17T18:29:44.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.755+0000 7fbc4e7fc640 10 client.64253.objecter ms_dispatch 0x7fbc58000b90 osd_map(413..413 src has 247..413) 2024-09-17T18:29:44.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.755+0000 7fbc4e7fc640 3 client.64253.objecter handle_osd_map got epochs [413,413] > 0 2024-09-17T18:29:44.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.755+0000 7fbc4e7fc640 3 client.64253.objecter handle_osd_map decoding full epoch 413 2024-09-17T18:29:44.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.755+0000 7fbc5d3a3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:44.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.755+0000 7fbc5d3a3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:44.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.755+0000 7fbc5d3a3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:44.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.755+0000 7fbc4e7fc640 20 client.64253.objecter dump_active .. 0 homeless 2024-09-17T18:29:44.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.755+0000 7fbc5d3a3640 10 monclient(hunting): handle_auth_done global_id 64256 payload 824 2024-09-17T18:29:44.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.755+0000 7fbc5d3a3640 1 -- 172.21.15.121:0/3760674710 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbc580a4fe0 con 0x7fbc5802d440 2024-09-17T18:29:44.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.755+0000 7fbc5e3a5640 1 --2- 172.21.15.121:0/3760674710 >> v2:172.21.15.173:6812/2423786037 conn(0x7fbc3c040150 0x7fbc3c042610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:44.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.755+0000 7fbc5e3a5640 10 monclient: get_auth_request con 0x7fbc3c040150 auth_method 0 2024-09-17T18:29:44.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.758+0000 7fbc5d3a3640 1 -- 172.21.15.121:0/3760674710 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc5802d440 msgr2=0x7fbc5802d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:44.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.758+0000 7fbc4e7fc640 10 client.64253.objecter ms_handle_connect 0x7fbc5802d440 2024-09-17T18:29:44.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.758+0000 7fbc5d3a3640 1 --2- 172.21.15.121:0/3760674710 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc5802d440 0x7fbc5802d820 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbc58123b30 tx=0x7fbc4800f7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:44.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.758+0000 7fbc4e7fc640 10 client.64253.objecter resend_mon_ops 2024-09-17T18:29:44.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.758+0000 7fbc4e7fc640 1 -- 172.21.15.121:0/3760674710 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc480050d0 con 0x7fbc5802d440 2024-09-17T18:29:44.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.758+0000 7fbc4e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:44.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.758+0000 7fbc4e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:29:44.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.758+0000 7fbc4e7fc640 10 monclient: dump: 2024-09-17T18:29:44.785 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:44.785 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:44.785 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:44.785 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:44.786 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:44.786 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:44.786 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:44.786 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:44.786 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:44.786 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:44.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.758+0000 7fbc4e7fc640 1 -- 172.21.15.121:0/3760674710 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbc480050d0 con 0x7fbc5802d440 2024-09-17T18:29:44.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.758+0000 7fbc4e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:29:44.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.758+0000 7fbc4e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:29:44.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.758+0000 7fbc4e7fc640 1 -- 172.21.15.121:0/3760674710 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc5802d440 msgr2=0x7fbc5802d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:44.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.758+0000 7fbc4e7fc640 1 --2- 172.21.15.121:0/3760674710 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc5802d440 0x7fbc5802d820 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbc58123b30 tx=0x7fbc4800f7a0 comp rx=0 tx=0).stop 2024-09-17T18:29:44.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.758+0000 7fbc4e7fc640 1 -- 172.21.15.121:0/3760674710 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc5802d440 msgr2=0x7fbc5802d820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:29:44.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.758+0000 7fbc4e7fc640 1 --2- 172.21.15.121:0/3760674710 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc5802d440 0x7fbc5802d820 secure :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbc58123b30 tx=0x7fbc4800f7a0 comp rx=0 tx=0).stop 2024-09-17T18:29:44.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.758+0000 7fbc4e7fc640 10 client.64253.objecter ms_handle_connect 0x7fbc3c040150 2024-09-17T18:29:44.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.758+0000 7fbc5e3a5640 1 --2- 172.21.15.121:0/3760674710 >> v2:172.21.15.173:6812/2423786037 conn(0x7fbc3c040150 0x7fbc3c042610 secure :-1 s=READY pgs=4023 cs=0 l=1 rev1=1 crypto rx=0x7fbc5809f2c0 tx=0x7fbc50001720 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:44.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.764+0000 7fbc5fe08640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:29:44.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.764+0000 7fbc5fe08640 1 --2- 172.21.15.121:0/3760674710 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc5802dd60 0x7fbc5802e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:29:44.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.764+0000 7fbc5fe08640 10 monclient(hunting): start opening mon connection 2024-09-17T18:29:44.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.764+0000 7fbc5fe08640 1 -- 172.21.15.121:0/3760674710 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbc580a2030 con 0x7fbc5802dd60 2024-09-17T18:29:44.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.764+0000 7fbc5d3a3640 1 --2- 172.21.15.121:0/3760674710 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc5802dd60 0x7fbc5802e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:29:44.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.764+0000 7fbc5d3a3640 10 monclient: get_auth_request con 0x7fbc5802dd60 auth_method 0 2024-09-17T18:29:44.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.764+0000 7fbc5d3a3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:29:44.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.764+0000 7fbc5d3a3640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:29:44.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.764+0000 7fbc5d3a3640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:29:44.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.764+0000 7fbc5d3a3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:29:44.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.764+0000 7fbc5d3a3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:29:44.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.764+0000 7fbc5d3a3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:29:44.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.764+0000 7fbc5d3a3640 10 monclient(hunting): handle_auth_done global_id 64259 payload 824 2024-09-17T18:29:44.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.764+0000 7fbc5d3a3640 1 -- 172.21.15.121:0/3760674710 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbc580709d0 con 0x7fbc5802dd60 2024-09-17T18:29:44.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.767+0000 7fbc5d3a3640 1 -- 172.21.15.121:0/3760674710 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc5802dd60 msgr2=0x7fbc5802e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:29:44.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.767+0000 7fbc4e7fc640 10 client.64253.objecter ms_handle_connect 0x7fbc5802dd60 2024-09-17T18:29:44.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.767+0000 7fbc4e7fc640 10 client.64253.objecter resend_mon_ops 2024-09-17T18:29:44.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.767+0000 7fbc5d3a3640 1 --2- 172.21.15.121:0/3760674710 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc5802dd60 0x7fbc5802e170 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbc4801a000 tx=0x7fbc48005a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:29:44.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.767+0000 7fbc4e7fc640 1 -- 172.21.15.121:0/3760674710 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc48020a30 con 0x7fbc5802dd60 2024-09-17T18:29:44.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.767+0000 7fbc4e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:29:44.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.767+0000 7fbc4e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:29:44.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.767+0000 7fbc4e7fc640 10 monclient: dump: 2024-09-17T18:29:44.789 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:29:44.789 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:29:44.789 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:44.789 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:29:44.789 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:29:44.789 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:29:44.789 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:29:44.789 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:29:44.790 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:29:44.790 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:29:44.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.767+0000 7fbc4e7fc640 1 -- 172.21.15.121:0/3760674710 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7fbc48020a30 con 0x7fbc5802dd60 2024-09-17T18:29:44.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.767+0000 7fbc4e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:29:44.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.767+0000 7fbc4e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:29:44.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.767+0000 7fbc4e7fc640 1 -- 172.21.15.121:0/3760674710 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc5802dd60 msgr2=0x7fbc5802e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:44.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.767+0000 7fbc4e7fc640 1 --2- 172.21.15.121:0/3760674710 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc5802dd60 0x7fbc5802e170 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbc4801a000 tx=0x7fbc48005a00 comp rx=0 tx=0).stop 2024-09-17T18:29:44.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.767+0000 7fbc4e7fc640 1 -- 172.21.15.121:0/3760674710 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc5802dd60 msgr2=0x7fbc5802e170 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:29:44.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.767+0000 7fbc4e7fc640 1 --2- 172.21.15.121:0/3760674710 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc5802dd60 0x7fbc5802e170 secure :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbc4801a000 tx=0x7fbc48005a00 comp rx=0 tx=0).stop 2024-09-17T18:29:44.790 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:29:44.791 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:29:44.791 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:29:44.791 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:29:44.791 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 42, 2024-09-17T18:29:44.791 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 163499, 2024-09-17T18:29:44.791 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:29:44.791 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:29:44.791 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:29:44.791 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:29:44.791 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 2, 2024-09-17T18:29:44.791 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:29:44.791 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:29:44.792 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:29:44.792 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:29:44.792 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:29:44.792 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:29:44.792 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:29:44.792 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:29:44.792 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:29:44.792 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:29:44.792 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:29:44.792 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:29:44.792 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:29:44.793 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:29:44.793 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:29:44.793 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:29:44.793 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:29:44.793 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:29:44.793 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:29:44.793 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:29:44.793 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:29:44.793 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:29:44.793 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:29:44.793 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:29:44.794 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:29:44.794 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:29:44.794 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:29:44.794 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:29:44.794 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:44.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:44.794 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:29:44.794 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:29:44.794 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:29:44.794 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:29:44.794 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:29:44.794 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:29:44.795 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:29:44.795 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:29:44.795 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:29:44.795 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:29:44.795 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:29:44.795 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:29:44.795 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:29:44.795 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:29:44.795 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:29:44.795 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:29:44.795 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:29:44.795 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:29:44.796 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:29:44.796 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:29:44.796 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:29:44.796 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:29:44.796 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:29:44.796 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:29:44.796 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:29:44.796 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:29:44.796 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:29:44.796 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:29:44.796 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:29:44.797 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:29:44.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:44.797 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:29:44.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:44.797 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:29:44.797 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:29:44.797 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:29:44.797 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:29:44.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:44.797 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:29:44.797 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:29:44.798 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:44.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:44.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:44.798 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:29:44.798 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:29:44.798 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:44.798 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:44.798 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:44.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:44.798 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:29:44.798 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:29:44.798 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:29:44.799 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:29:44.799 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:29:44.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:44.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:44.799 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:29:44.799 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:29:44.799 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:29:44.799 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:29:44.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:44.799 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:29:44.799 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:29:44.800 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:44.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:44.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:44.800 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:29:44.800 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:29:44.800 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:44.800 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:44.800 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:44.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:44.800 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:29:44.800 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:29:44.801 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:29:44.801 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:29:44.801 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:29:44.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:44.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:44.801 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:29:44.801 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:29:44.801 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:29:44.801 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:29:44.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:44.801 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:29:44.801 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:29:44.801 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:44.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:44.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:44.802 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:29:44.802 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:29:44.802 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:29:44.802 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:44.802 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:44.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:44.802 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:29:44.802 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:29:44.802 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:29:44.802 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:29:44.802 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:29:44.802 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:44.803 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:44.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:44.803 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:29:44.803 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:29:44.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:44.803 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:29:44.803 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:29:44.803 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:29:44.803 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:44.803 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:29:44.803 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:29:44.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:44.803 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:29:44.804 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:29:44.804 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-17T18:29:44.804 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:44.804 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:29:44.804 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:29:44.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:44.804 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:29:44.804 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:29:44.804 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 3 2024-09-17T18:29:44.804 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:44.804 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:29:44.804 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": [ 2024-09-17T18:29:44.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:29:44.804 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:29:44.805 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:29:44.805 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:29:44.805 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:29:44.805 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:29:44.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:29:44.805 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:29:44.805 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:29:44.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.767+0000 7fbc5fe08640 10 librados: watch_flush enter 2024-09-17T18:29:44.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.767+0000 7fbc5fe08640 10 librados: watch_flush exit 2024-09-17T18:29:44.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.767+0000 7fbc5fe08640 20 client.64253.objecter shutdown clearing up homeless session... 2024-09-17T18:29:44.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.767+0000 7fbc5fe08640 10 client.64253.objecter successfully canceled tick 2024-09-17T18:29:44.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.767+0000 7fbc5fe08640 1 -- 172.21.15.121:0/3760674710 >> v2:172.21.15.173:6812/2423786037 conn(0x7fbc3c040150 msgr2=0x7fbc3c042610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:44.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.767+0000 7fbc5fe08640 1 --2- 172.21.15.121:0/3760674710 >> v2:172.21.15.173:6812/2423786037 conn(0x7fbc3c040150 0x7fbc3c042610 secure :-1 s=READY pgs=4023 cs=0 l=1 rev1=1 crypto rx=0x7fbc5809f2c0 tx=0x7fbc50001720 comp rx=0 tx=0).stop 2024-09-17T18:29:44.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.767+0000 7fbc5fe08640 10 monclient: shutdown 2024-09-17T18:29:44.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.767+0000 7fbc5fe08640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:29:44.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.767+0000 7fbc5fe08640 1 -- 172.21.15.121:0/3760674710 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc44003aa0 msgr2=0x7fbc58134c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:29:44.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.767+0000 7fbc5fe08640 1 --2- 172.21.15.121:0/3760674710 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc44003aa0 0x7fbc58134c20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbc54009920 tx=0x7fbc54005940 comp rx=0 tx=0).stop 2024-09-17T18:29:44.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.767+0000 7fbc5e3a5640 1 -- 172.21.15.121:0/3760674710 reap_dead start 2024-09-17T18:29:44.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.767+0000 7fbc5e3a5640 1 -- 172.21.15.121:0/3760674710 reap_dead start 2024-09-17T18:29:44.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.767+0000 7fbc5fe08640 1 -- 172.21.15.121:0/3760674710 shutdown_connections 2024-09-17T18:29:44.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.767+0000 7fbc5fe08640 1 -- 172.21.15.121:0/3760674710 >> 172.21.15.121:0/3760674710 conn(0x7fbc580a7830 msgr2=0x7fbc58136330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:29:44.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.767+0000 7fbc5fe08640 1 -- 172.21.15.121:0/3760674710 shutdown_connections 2024-09-17T18:29:44.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.767+0000 7fbc5fe08640 1 -- 172.21.15.121:0/3760674710 wait complete. 2024-09-17T18:29:44.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:29:44.767+0000 7fbc5fe08640 1 librados: shutdown 2024-09-17T18:29:44.809 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T18:29:44.809 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-17T18:30:20.446 INFO:tasks.ceph.osd.3.smithi173.stderr:2024-09-17T18:30:20.444+0000 7f297bab0640 -1 snap_mapper.add_oid found existing snaps mapped on 102:03e028f7:test-rados-api-smithi173-35399-97::foo:21, removing 2024-09-17T18:30:20.447 INFO:tasks.ceph.osd.1.smithi121.stderr:2024-09-17T18:30:20.443+0000 7f1efba5e640 -1 snap_mapper.add_oid found existing snaps mapped on 102:03e028f7:test-rados-api-smithi173-35399-97::foo:21, removing 2024-09-17T18:31:14.809 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-17T18:31:14.810 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-17T18:31:14.810 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-17T18:31:14.813 INFO:tasks.ceph.mon.b:Started 2024-09-17T18:31:14.813 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T18:31:14.814 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:31:14.914 INFO:tasks.ceph.mon.b.smithi173.stderr:2024-09-17T18:31:14.912+0000 7fceeede5c80 -1 mon.b@-1(???).log v0 log_external_backlog rewinding external_log_to from 436 -> 0 (sync_force? mon rebuild?) 2024-09-17T18:31:14.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.919+0000 7f686f4dd640 10 monclient: get_monmap_and_config 2024-09-17T18:31:14.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.919+0000 7f686f4dd640 10 monclient: build_initial_monmap 2024-09-17T18:31:14.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.919+0000 7f686f4dd640 10 monclient: monmap: 2024-09-17T18:31:14.921 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:31:14.921 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:14.922 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:31:14.920532+0000 2024-09-17T18:31:14.922 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:31:14.920532+0000 2024-09-17T18:31:14.922 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:31:14.922 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:14.922 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:31:14.922 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:31:14.922 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:31:14.922 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:31:14.922 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:31:14.922 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:14.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.920+0000 7f686f4dd640 1 Processor -- start 2024-09-17T18:31:14.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.920+0000 7f686f4dd640 1 -- start start 2024-09-17T18:31:14.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.920+0000 7f686f4dd640 10 monclient: init 2024-09-17T18:31:14.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.921+0000 7f686f4dd640 10 monclient: _reopen_session rank -1 2024-09-17T18:31:14.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.921+0000 7f686f4dd640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-17T18:31:14.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.921+0000 7f686f4dd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f68680a5c00 addr v1:172.21.15.121:6789/0 2024-09-17T18:31:14.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.921+0000 7f686f4dd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f68680a5fe0 addr v1:172.21.15.173:6789/0 2024-09-17T18:31:14.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.921+0000 7f686f4dd640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f686813ec10 0x7f6868141000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:14.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.921+0000 7f686f4dd640 10 monclient(hunting): picked mon.noname-a con 0x7f686813ec10 addr v2:172.21.15.121:3300/0 2024-09-17T18:31:14.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.921+0000 7f686f4dd640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68680a8a80 con 0x7f68680a5c00 2024-09-17T18:31:14.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.921+0000 7f686f4dd640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68680a74d0 con 0x7f68680a5fe0 2024-09-17T18:31:14.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.921+0000 7f686f4dd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:14.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.921+0000 7f686f4dd640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f68680a7110 con 0x7f686813ec10 2024-09-17T18:31:14.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.921+0000 7f686f4dd640 10 monclient(hunting): _renew_subs 2024-09-17T18:31:14.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.921+0000 7f686f4dd640 10 monclient(hunting): authenticate will time out at 1478.637297s 2024-09-17T18:31:14.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.921+0000 7f686d279640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f68680a5c00 0x7f686809d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:53050/0 (socket says 172.21.15.121:53050) 2024-09-17T18:31:14.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.921+0000 7f686d279640 1 -- 172.21.15.121:0/39082540 learned_addr learned my addr 172.21.15.121:0/39082540 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:31:14.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.921+0000 7f686da7a640 1 --2- 172.21.15.121:0/39082540 >> v2:172.21.15.121:3300/0 conn(0x7f686813ec10 0x7f6868141000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:14.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.921+0000 7f686da7a640 10 monclient(hunting): get_auth_request con 0x7f686813ec10 auth_method 0 2024-09-17T18:31:14.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.921+0000 7f686da7a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:14.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.921+0000 7f686da7a640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:14.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.921+0000 7f686da7a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:14.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.921+0000 7f686da7a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:14.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.921+0000 7f686da7a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:14.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.921+0000 7f686da7a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:14.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.921+0000 7f686da7a640 10 monclient(hunting): handle_auth_done global_id 44637 payload 274 2024-09-17T18:31:14.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.921+0000 7f686da7a640 1 -- 172.21.15.121:0/39082540 >> v1:172.21.15.173:6789/0 conn(0x7f68680a5fe0 legacy=0x7f68680a63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:31:14.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.921+0000 7f686da7a640 1 -- 172.21.15.121:0/39082540 >> v1:172.21.15.121:6789/0 conn(0x7f68680a5c00 legacy=0x7f686809d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:31:14.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.921+0000 7f686da7a640 10 monclient: _finish_hunting 0 2024-09-17T18:31:14.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.921+0000 7f686da7a640 1 monclient: found mon.noname-a 2024-09-17T18:31:14.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.921+0000 7f686da7a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:31:14.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.921+0000 7f686da7a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:31:14.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.921+0000 7f686da7a640 1 -- 172.21.15.121:0/39082540 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f686809cb90 con 0x7f686813ec10 2024-09-17T18:31:14.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.924+0000 7f686da7a640 1 -- 172.21.15.121:0/39082540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f686813ec10 msgr2=0x7f6868141000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:14.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.924+0000 7f686da7a640 1 --2- 172.21.15.121:0/39082540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f686813ec10 0x7f6868141000 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f6860009d90 tx=0x7f686002a490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ed795d11bfd9f7a server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:14.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.924+0000 7f685bfff640 1 -- 172.21.15.121:0/39082540 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6860007a30 con 0x7f686813ec10 2024-09-17T18:31:14.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.924+0000 7f685bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:14.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.924+0000 7f685bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:31:14.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.924+0000 7f685bfff640 10 monclient: dump: 2024-09-17T18:31:14.935 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:14.935 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:14.935 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:14.935 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:14.935 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:14.935 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:14.935 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:14.935 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:14.935 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:14.935 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:14.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.924+0000 7f685bfff640 10 monclient: _finish_auth 0 2024-09-17T18:31:14.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.924+0000 7f685bfff640 10 monclient: _check_auth_tickets 2024-09-17T18:31:14.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.924+0000 7f685bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:31:14.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.924+0000 7f686f4dd640 5 monclient: authenticate success, global_id 44637 2024-09-17T18:31:14.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.924+0000 7f686f4dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:31:14.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.924+0000 7f685bfff640 1 -- 172.21.15.121:0/39082540 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6860007bd0 con 0x7f686813ec10 2024-09-17T18:31:14.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.924+0000 7f685bfff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:31:14.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.924+0000 7f686f4dd640 10 monclient: get_monmap_and_config success 2024-09-17T18:31:14.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.924+0000 7f685bfff640 1 -- 172.21.15.121:0/39082540 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6860031400 con 0x7f686813ec10 2024-09-17T18:31:14.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.924+0000 7f685bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:14.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.924+0000 7f686f4dd640 4 set_mon_vals no callback set 2024-09-17T18:31:14.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.924+0000 7f685bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:31:14.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.924+0000 7f685bfff640 10 monclient: dump: 2024-09-17T18:31:14.936 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:14.936 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:14.937 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:14.937 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:14.937 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:14.937 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:14.937 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:14.937 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:14.937 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:14.937 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:14.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.925+0000 7f686f4dd640 10 monclient: shutdown 2024-09-17T18:31:14.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.925+0000 7f686f4dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:31:14.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.925+0000 7f686f4dd640 1 -- 172.21.15.121:0/39082540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f686813ec10 msgr2=0x7f6868141000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:14.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.925+0000 7f686f4dd640 1 --2- 172.21.15.121:0/39082540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f686813ec10 0x7f6868141000 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f6860009d90 tx=0x7f686002a490 comp rx=0 tx=0).stop 2024-09-17T18:31:14.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.925+0000 7f686f4dd640 1 -- 172.21.15.121:0/39082540 shutdown_connections 2024-09-17T18:31:14.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.925+0000 7f686f4dd640 1 --2- 172.21.15.121:0/39082540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f686813ec10 0x7f6868141000 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:14.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.925+0000 7f686f4dd640 1 -- 172.21.15.121:0/39082540 >> 172.21.15.121:0/39082540 conn(0x7f68680ab9b0 msgr2=0x7f68680a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:31:14.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.925+0000 7f686f4dd640 1 -- 172.21.15.121:0/39082540 shutdown_connections 2024-09-17T18:31:14.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.925+0000 7f686f4dd640 1 -- 172.21.15.121:0/39082540 wait complete. 2024-09-17T18:31:14.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.925+0000 7f686f4dd640 10 monclient: build_initial_monmap 2024-09-17T18:31:14.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.925+0000 7f686f4dd640 10 monclient: monmap: 2024-09-17T18:31:14.938 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:31:14.938 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:31:14.938 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:31:14.938 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:31:14.938 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:31:14.939 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:14.939 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:31:14.939 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:31:14.939 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:31:14.939 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:14.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.926+0000 7f686f4dd640 1 librados: starting msgr at 2024-09-17T18:31:14.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.926+0000 7f686f4dd640 1 librados: starting objecter 2024-09-17T18:31:14.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.926+0000 7f686f4dd640 1 Processor -- start 2024-09-17T18:31:14.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.926+0000 7f686f4dd640 1 -- start start 2024-09-17T18:31:14.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.926+0000 7f686f4dd640 1 librados: setting wanted keys 2024-09-17T18:31:14.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.926+0000 7f686f4dd640 1 librados: calling monclient init 2024-09-17T18:31:14.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.926+0000 7f686f4dd640 10 monclient: init 2024-09-17T18:31:14.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.926+0000 7f686f4dd640 10 monclient: _reopen_session rank -1 2024-09-17T18:31:14.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.926+0000 7f686f4dd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T18:31:14.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.926+0000 7f686f4dd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68680a5c00 0x7f68681385a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:14.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.926+0000 7f686f4dd640 10 monclient(hunting): picked mon.noname-c con 0x7f68680a5c00 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:31:14.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.926+0000 7f686f4dd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f68680a5fe0 0x7f6868138ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:14.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.926+0000 7f686f4dd640 10 monclient(hunting): picked mon.noname-b con 0x7f68680a5fe0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:31:14.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.926+0000 7f686f4dd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f686813ec10 0x7f686813b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:14.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.926+0000 7f686f4dd640 10 monclient(hunting): picked mon.noname-a con 0x7f686813ec10 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:31:14.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.926+0000 7f686f4dd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:14.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.926+0000 7f686f4dd640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6868141540 con 0x7f686813ec10 2024-09-17T18:31:14.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.926+0000 7f686f4dd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:14.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.926+0000 7f686f4dd640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f686809ced0 con 0x7f68680a5fe0 2024-09-17T18:31:14.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.926+0000 7f686f4dd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:14.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.926+0000 7f686f4dd640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f68681416c0 con 0x7f68680a5c00 2024-09-17T18:31:14.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.926+0000 7f686f4dd640 10 monclient(hunting): _renew_subs 2024-09-17T18:31:14.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.926+0000 7f686f4dd640 10 monclient(hunting): authenticate will time out at 1478.642791s 2024-09-17T18:31:14.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.926+0000 7f686d279640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68680a5c00 0x7f68681385a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:14.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.926+0000 7f686da7a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f686813ec10 0x7f686813b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:14.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.926+0000 7f686d279640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68680a5c00 0x7f68681385a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:54786/0 (socket says 172.21.15.121:54786) 2024-09-17T18:31:14.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.926+0000 7f686d279640 1 -- 172.21.15.121:0/3688710394 learned_addr learned my addr 172.21.15.121:0/3688710394 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:31:14.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.926+0000 7f686d279640 10 monclient(hunting): get_auth_request con 0x7f68680a5c00 auth_method 0 2024-09-17T18:31:14.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.926+0000 7f686d279640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:14.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.926+0000 7f686d279640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:14.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.926+0000 7f686d279640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:14.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.926+0000 7f686da7a640 10 monclient(hunting): get_auth_request con 0x7f686813ec10 auth_method 0 2024-09-17T18:31:14.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.926+0000 7f686da7a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:14.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.926+0000 7f686da7a640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:14.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.926+0000 7f686da7a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:14.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.926+0000 7f686d279640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:14.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.926+0000 7f686d279640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:14.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.926+0000 7f686d279640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:14.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.926+0000 7f686da7a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:14.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.926+0000 7f686da7a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:14.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.927+0000 7f686da7a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:14.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.927+0000 7f686ca78640 1 --2- 172.21.15.121:0/3688710394 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f68680a5fe0 0x7f6868138ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:14.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.927+0000 7f686ca78640 10 monclient(hunting): get_auth_request con 0x7f68680a5fe0 auth_method 0 2024-09-17T18:31:14.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.927+0000 7f686ca78640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:14.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.927+0000 7f686ca78640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:14.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.927+0000 7f686ca78640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:14.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.927+0000 7f686d279640 10 monclient(hunting): handle_auth_done global_id 64280 payload 824 2024-09-17T18:31:14.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.927+0000 7f686d279640 1 -- 172.21.15.121:0/3688710394 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f68680a5fe0 msgr2=0x7f6868138ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:14.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.927+0000 7f686d279640 1 --2- 172.21.15.121:0/3688710394 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f68680a5fe0 0x7f6868138ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:14.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.927+0000 7f686d279640 1 -- 172.21.15.121:0/3688710394 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f686813ec10 msgr2=0x7f686813b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:14.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.927+0000 7f686d279640 1 --2- 172.21.15.121:0/3688710394 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f686813ec10 0x7f686813b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:14.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.927+0000 7f686d279640 10 monclient: _finish_hunting 0 2024-09-17T18:31:14.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.927+0000 7f686d279640 1 monclient: found mon.noname-c 2024-09-17T18:31:14.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.927+0000 7f686d279640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:31:14.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.927+0000 7f686d279640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:31:14.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.927+0000 7f686d279640 1 -- 172.21.15.121:0/3688710394 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6868121b30 con 0x7f68680a5c00 2024-09-17T18:31:14.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.927+0000 7f686da7a640 1 --2- 172.21.15.121:0/3688710394 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f686813ec10 0x7f686813b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:31:14.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.929+0000 7f686d279640 1 -- 172.21.15.121:0/3688710394 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68680a5c00 msgr2=0x7f68681385a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:14.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.929+0000 7f686d279640 1 --2- 172.21.15.121:0/3688710394 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68680a5c00 0x7f68681385a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f686400c460 tx=0x7f686400c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:14.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.929+0000 7f6859ffb640 10 client.?.objecter ms_handle_connect 0x7f68680a5c00 2024-09-17T18:31:14.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.929+0000 7f6859ffb640 10 client.?.objecter resend_mon_ops 2024-09-17T18:31:14.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.929+0000 7f6859ffb640 1 -- 172.21.15.121:0/3688710394 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6864007070 con 0x7f68680a5c00 2024-09-17T18:31:14.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.929+0000 7f6859ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:14.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.929+0000 7f6859ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:31:14.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.929+0000 7f6859ffb640 10 monclient: dump: 2024-09-17T18:31:14.944 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:14.944 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:14.944 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:14.944 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:14.944 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:14.944 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:14.944 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:14.944 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:14.944 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:14.944 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:14.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.929+0000 7f6859ffb640 10 monclient: _finish_auth 0 2024-09-17T18:31:14.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.929+0000 7f6859ffb640 10 monclient: _check_auth_tickets 2024-09-17T18:31:14.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.929+0000 7f6859ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:31:14.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.929+0000 7f6859ffb640 1 -- 172.21.15.121:0/3688710394 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6864007230 con 0x7f68680a5c00 2024-09-17T18:31:14.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.929+0000 7f686f4dd640 5 monclient: authenticate success, global_id 64280 2024-09-17T18:31:14.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.929+0000 7f686f4dd640 10 monclient: _renew_subs 2024-09-17T18:31:14.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.929+0000 7f686f4dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:31:14.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.929+0000 7f686f4dd640 1 -- 172.21.15.121:0/3688710394 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68681411a0 con 0x7f68680a5c00 2024-09-17T18:31:14.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.929+0000 7f6859ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:31:14.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.929+0000 7f6859ffb640 1 -- 172.21.15.121:0/3688710394 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6864007620 con 0x7f68680a5c00 2024-09-17T18:31:14.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.929+0000 7f6859ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:14.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.929+0000 7f6859ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:31:14.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.929+0000 7f6859ffb640 10 monclient: dump: 2024-09-17T18:31:14.945 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:14.945 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:14.945 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:14.945 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:14.945 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:14.946 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:14.946 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:14.946 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:14.946 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:14.946 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:14.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.929+0000 7f686f4dd640 10 client.64280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:31:14.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.929+0000 7f686f4dd640 10 monclient: _renew_subs 2024-09-17T18:31:14.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.929+0000 7f685b7fe640 4 set_mon_vals no callback set 2024-09-17T18:31:14.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.929+0000 7f686f4dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:31:14.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.929+0000 7f686f4dd640 1 -- 172.21.15.121:0/3688710394 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f68681220f0 con 0x7f68680a5c00 2024-09-17T18:31:14.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.929+0000 7f686f4dd640 1 librados: init done 2024-09-17T18:31:14.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.930+0000 7f6859ffb640 1 -- 172.21.15.121:0/3688710394 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f686401b400 con 0x7f68680a5c00 2024-09-17T18:31:14.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.930+0000 7f6859ffb640 1 --2- 172.21.15.121:0/3688710394 >> v2:172.21.15.173:6812/2423786037 conn(0x7f684c040170 0x7f684c042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:14.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.930+0000 7f686f4dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:31:14.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.930+0000 7f686f4dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:31:14.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.930+0000 7f686f4dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:31:14.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.930+0000 7f6859ffb640 1 -- 172.21.15.121:0/3688710394 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(478..478 src has 247..478) ==== 6686+0+0 (secure 0 0 0) 0x7f68640538f0 con 0x7f68680a5c00 2024-09-17T18:31:14.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.930+0000 7f6859ffb640 10 client.64280.objecter ms_dispatch 0x7f6868000b90 osd_map(478..478 src has 247..478) 2024-09-17T18:31:14.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.930+0000 7f6859ffb640 3 client.64280.objecter handle_osd_map got epochs [478,478] > 0 2024-09-17T18:31:14.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.930+0000 7f6859ffb640 3 client.64280.objecter handle_osd_map decoding full epoch 478 2024-09-17T18:31:14.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.930+0000 7f686f4dd640 1 -- 172.21.15.121:0/3688710394 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f686809d8c0 con 0x7f68680a5c00 2024-09-17T18:31:14.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.930+0000 7f6859ffb640 20 client.64280.objecter dump_active .. 0 homeless 2024-09-17T18:31:14.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.931+0000 7f686ca78640 1 --2- 172.21.15.121:0/3688710394 >> v2:172.21.15.173:6812/2423786037 conn(0x7f684c040170 0x7f684c042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:14.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.931+0000 7f686ca78640 10 monclient: get_auth_request con 0x7f684c040170 auth_method 0 2024-09-17T18:31:14.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.933+0000 7f6859ffb640 10 client.64280.objecter ms_handle_connect 0x7f684c040170 2024-09-17T18:31:14.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.933+0000 7f686ca78640 1 --2- 172.21.15.121:0/3688710394 >> v2:172.21.15.173:6812/2423786037 conn(0x7f684c040170 0x7f684c042630 secure :-1 s=READY pgs=4028 cs=0 l=1 rev1=1 crypto rx=0x7f686813d6c0 tx=0x7f685c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:14.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.935+0000 7f6859ffb640 1 -- 172.21.15.121:0/3688710394 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f686401b8d0 con 0x7f68680a5c00 2024-09-17T18:31:14.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.935+0000 7f6859ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:31:14.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:14.935+0000 7f6859ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:31:15.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:15.179+0000 7f686f4dd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:31:15.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:15.179+0000 7f686f4dd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:31:15.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:15.179+0000 7f686f4dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:31:15.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:15.179+0000 7f686f4dd640 1 -- 172.21.15.121:0/3688710394 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f686806cd40 con 0x7f68680a5c00 2024-09-17T18:31:15.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:15.179+0000 7f6859ffb640 1 -- 172.21.15.121:0/3688710394 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f6864012400 con 0x7f68680a5c00 2024-09-17T18:31:15.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:15.179+0000 7f6859ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:31:15.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:15.179+0000 7f6859ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:31:15.182 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":42,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":92,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:31:15.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:15.181+0000 7f686f4dd640 10 librados: watch_flush enter 2024-09-17T18:31:15.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:15.181+0000 7f686f4dd640 10 librados: watch_flush exit 2024-09-17T18:31:15.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:15.181+0000 7f686f4dd640 20 client.64280.objecter shutdown clearing up homeless session... 2024-09-17T18:31:15.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:15.181+0000 7f686f4dd640 10 client.64280.objecter successfully canceled tick 2024-09-17T18:31:15.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:15.181+0000 7f686f4dd640 1 -- 172.21.15.121:0/3688710394 >> v2:172.21.15.173:6812/2423786037 conn(0x7f684c040170 msgr2=0x7f684c042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:15.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:15.181+0000 7f686f4dd640 1 --2- 172.21.15.121:0/3688710394 >> v2:172.21.15.173:6812/2423786037 conn(0x7f684c040170 0x7f684c042630 secure :-1 s=READY pgs=4028 cs=0 l=1 rev1=1 crypto rx=0x7f686813d6c0 tx=0x7f685c009040 comp rx=0 tx=0).stop 2024-09-17T18:31:15.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:15.181+0000 7f686f4dd640 10 monclient: shutdown 2024-09-17T18:31:15.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:15.181+0000 7f686f4dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:31:15.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:15.181+0000 7f686f4dd640 1 -- 172.21.15.121:0/3688710394 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68680a5c00 msgr2=0x7f68681385a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:15.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:15.181+0000 7f686f4dd640 1 --2- 172.21.15.121:0/3688710394 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68680a5c00 0x7f68681385a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f686400c460 tx=0x7f686400c920 comp rx=0 tx=0).stop 2024-09-17T18:31:15.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:15.181+0000 7f686f4dd640 1 -- 172.21.15.121:0/3688710394 shutdown_connections 2024-09-17T18:31:15.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:15.181+0000 7f686f4dd640 1 --2- 172.21.15.121:0/3688710394 >> v2:172.21.15.173:6812/2423786037 conn(0x7f684c040170 0x7f684c042630 unknown :-1 s=CLOSED pgs=4028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:15.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:15.181+0000 7f686f4dd640 1 --2- 172.21.15.121:0/3688710394 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f68680a5fe0 0x7f6868138ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:15.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:15.181+0000 7f686f4dd640 1 --2- 172.21.15.121:0/3688710394 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f686813ec10 0x7f686813b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:15.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:15.181+0000 7f686f4dd640 1 --2- 172.21.15.121:0/3688710394 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68680a5c00 0x7f68681385a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:15.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:15.182+0000 7f686f4dd640 1 -- 172.21.15.121:0/3688710394 >> 172.21.15.121:0/3688710394 conn(0x7f68680ab9b0 msgr2=0x7f68680a9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:31:15.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:15.182+0000 7f686f4dd640 1 -- 172.21.15.121:0/3688710394 shutdown_connections 2024-09-17T18:31:15.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:15.182+0000 7f686f4dd640 1 -- 172.21.15.121:0/3688710394 wait complete. 2024-09-17T18:31:15.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:15.182+0000 7f686f4dd640 1 librados: shutdown 2024-09-17T18:31:18.194 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:31:18.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.300+0000 7f6c7939f640 10 monclient: get_monmap_and_config 2024-09-17T18:31:18.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.300+0000 7f6c7939f640 10 monclient: build_initial_monmap 2024-09-17T18:31:18.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.300+0000 7f6c7939f640 10 monclient: monmap: 2024-09-17T18:31:18.302 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:31:18.302 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:18.302 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:31:18.301220+0000 2024-09-17T18:31:18.302 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:31:18.301220+0000 2024-09-17T18:31:18.303 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:31:18.303 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:18.303 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:31:18.303 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:31:18.303 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:31:18.303 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:31:18.303 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:31:18.303 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:18.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.301+0000 7f6c7939f640 1 Processor -- start 2024-09-17T18:31:18.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.301+0000 7f6c7939f640 1 -- start start 2024-09-17T18:31:18.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.301+0000 7f6c7939f640 10 monclient: init 2024-09-17T18:31:18.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.302+0000 7f6c7939f640 10 monclient: _reopen_session rank -1 2024-09-17T18:31:18.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.302+0000 7f6c7939f640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-17T18:31:18.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.302+0000 7f6c7939f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6c7412fa80 addr v1:172.21.15.121:6789/0 2024-09-17T18:31:18.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.302+0000 7f6c7939f640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6c7412fe60 0x7f6c7413b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:18.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.302+0000 7f6c7939f640 10 monclient(hunting): picked mon.noname-a con 0x7f6c7412fe60 addr v2:172.21.15.121:3300/0 2024-09-17T18:31:18.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.302+0000 7f6c7939f640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f6c74130270 0x7f6c7413d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:18.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.302+0000 7f6c7939f640 10 monclient(hunting): picked mon.noname-c con 0x7f6c74130270 addr v2:172.21.15.173:3300/0 2024-09-17T18:31:18.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.302+0000 7f6c7939f640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c740a0510 con 0x7f6c7412fa80 2024-09-17T18:31:18.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.302+0000 7f6c7939f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:18.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.302+0000 7f6c7939f640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f6c740a1120 con 0x7f6c7412fe60 2024-09-17T18:31:18.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.302+0000 7f6c7939f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:18.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.302+0000 7f6c7939f640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f6c740a0e40 con 0x7f6c74130270 2024-09-17T18:31:18.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.302+0000 7f6c7939f640 10 monclient(hunting): _renew_subs 2024-09-17T18:31:18.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.302+0000 7f6c7939f640 10 monclient(hunting): authenticate will time out at 1482.018174s 2024-09-17T18:31:18.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.302+0000 7f6c737fe640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6c7412fe60 0x7f6c7413b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:18.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.302+0000 7f6c73fff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f6c7412fa80 0x7f6c740a2fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:53060/0 (socket says 172.21.15.121:53060) 2024-09-17T18:31:18.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.302+0000 7f6c73fff640 1 -- 172.21.15.121:0/2776566926 learned_addr learned my addr 172.21.15.121:0/2776566926 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:31:18.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.302+0000 7f6c737fe640 10 monclient(hunting): get_auth_request con 0x7f6c7412fe60 auth_method 0 2024-09-17T18:31:18.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.302+0000 7f6c737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:18.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.302+0000 7f6c737fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:18.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.302+0000 7f6c737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:18.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.302+0000 7f6c737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:18.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.302+0000 7f6c737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:18.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.302+0000 7f6c737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:18.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.302+0000 7f6c78b9e640 1 --2- 172.21.15.121:0/2776566926 >> v2:172.21.15.173:3300/0 conn(0x7f6c74130270 0x7f6c7413d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:18.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.302+0000 7f6c78b9e640 10 monclient(hunting): get_auth_request con 0x7f6c74130270 auth_method 0 2024-09-17T18:31:18.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.302+0000 7f6c78b9e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:18.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.302+0000 7f6c78b9e640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:18.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.302+0000 7f6c78b9e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:18.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.302+0000 7f6c737fe640 10 monclient(hunting): handle_auth_done global_id 44643 payload 274 2024-09-17T18:31:18.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.302+0000 7f6c737fe640 1 -- 172.21.15.121:0/2776566926 >> v2:172.21.15.173:3300/0 conn(0x7f6c74130270 msgr2=0x7f6c7413d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:31:18.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.302+0000 7f6c737fe640 1 --2- 172.21.15.121:0/2776566926 >> v2:172.21.15.173:3300/0 conn(0x7f6c74130270 0x7f6c7413d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:18.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.302+0000 7f6c737fe640 1 -- 172.21.15.121:0/2776566926 >> v1:172.21.15.121:6789/0 conn(0x7f6c7412fa80 legacy=0x7f6c740a2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:31:18.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.302+0000 7f6c737fe640 10 monclient: _finish_hunting 0 2024-09-17T18:31:18.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.302+0000 7f6c737fe640 1 monclient: found mon.noname-a 2024-09-17T18:31:18.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.302+0000 7f6c737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:31:18.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.302+0000 7f6c737fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:31:18.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.302+0000 7f6c737fe640 1 -- 172.21.15.121:0/2776566926 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c740a29f0 con 0x7f6c7412fe60 2024-09-17T18:31:18.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.305+0000 7f6c737fe640 1 -- 172.21.15.121:0/2776566926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c7412fe60 msgr2=0x7f6c7413b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:18.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.305+0000 7f6c737fe640 1 --2- 172.21.15.121:0/2776566926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c7412fe60 0x7f6c7413b3c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6c74136040 tx=0x7f6c6802a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e01927f7a6cfe61 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:18.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.305+0000 7f6c72ffd640 1 -- 172.21.15.121:0/2776566926 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c68037070 con 0x7f6c7412fe60 2024-09-17T18:31:18.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.305+0000 7f6c72ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:18.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.305+0000 7f6c72ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:31:18.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.305+0000 7f6c72ffd640 10 monclient: dump: 2024-09-17T18:31:18.315 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:18.315 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:18.315 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:18.315 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:18.315 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:18.315 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:18.315 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:18.315 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:18.315 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:18.315 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:18.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.305+0000 7f6c72ffd640 10 monclient: _finish_auth 0 2024-09-17T18:31:18.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.305+0000 7f6c72ffd640 10 monclient: _check_auth_tickets 2024-09-17T18:31:18.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.305+0000 7f6c72ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:31:18.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.305+0000 7f6c72ffd640 1 -- 172.21.15.121:0/2776566926 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c68032440 con 0x7f6c7412fe60 2024-09-17T18:31:18.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.305+0000 7f6c7939f640 5 monclient: authenticate success, global_id 44643 2024-09-17T18:31:18.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.305+0000 7f6c7939f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:31:18.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.305+0000 7f6c72ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:31:18.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.305+0000 7f6c72ffd640 1 -- 172.21.15.121:0/2776566926 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c68032810 con 0x7f6c7412fe60 2024-09-17T18:31:18.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.305+0000 7f6c7939f640 10 monclient: get_monmap_and_config success 2024-09-17T18:31:18.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.305+0000 7f6c7939f640 4 set_mon_vals no callback set 2024-09-17T18:31:18.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.305+0000 7f6c72ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:18.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.305+0000 7f6c72ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:31:18.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.305+0000 7f6c72ffd640 10 monclient: dump: 2024-09-17T18:31:18.316 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:18.316 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:18.316 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:18.316 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:18.316 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:18.316 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:18.316 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:18.317 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:18.317 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:18.317 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:18.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.306+0000 7f6c7939f640 10 monclient: shutdown 2024-09-17T18:31:18.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.306+0000 7f6c7939f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:31:18.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.306+0000 7f6c7939f640 1 -- 172.21.15.121:0/2776566926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c7412fe60 msgr2=0x7f6c7413b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:18.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.306+0000 7f6c7939f640 1 --2- 172.21.15.121:0/2776566926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c7412fe60 0x7f6c7413b3c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6c74136040 tx=0x7f6c6802a000 comp rx=0 tx=0).stop 2024-09-17T18:31:18.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.306+0000 7f6c7939f640 1 -- 172.21.15.121:0/2776566926 shutdown_connections 2024-09-17T18:31:18.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.306+0000 7f6c7939f640 1 --2- 172.21.15.121:0/2776566926 >> v2:172.21.15.173:3300/0 conn(0x7f6c74130270 0x7f6c7413d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:18.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.306+0000 7f6c7939f640 1 --2- 172.21.15.121:0/2776566926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c7412fe60 0x7f6c7413b3c0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:18.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.306+0000 7f6c7939f640 1 -- 172.21.15.121:0/2776566926 >> 172.21.15.121:0/2776566926 conn(0x7f6c740a3890 msgr2=0x7f6c740a3ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:31:18.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.306+0000 7f6c7939f640 1 -- 172.21.15.121:0/2776566926 shutdown_connections 2024-09-17T18:31:18.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.306+0000 7f6c7939f640 1 -- 172.21.15.121:0/2776566926 wait complete. 2024-09-17T18:31:18.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.306+0000 7f6c7939f640 10 monclient: build_initial_monmap 2024-09-17T18:31:18.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.306+0000 7f6c7939f640 10 monclient: monmap: 2024-09-17T18:31:18.317 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:31:18.318 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:31:18.318 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:31:18.318 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:31:18.318 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:31:18.318 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:18.318 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:31:18.318 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:31:18.318 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:31:18.318 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:18.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.306+0000 7f6c7939f640 1 librados: starting msgr at 2024-09-17T18:31:18.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.306+0000 7f6c7939f640 1 librados: starting objecter 2024-09-17T18:31:18.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.307+0000 7f6c7939f640 1 Processor -- start 2024-09-17T18:31:18.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.307+0000 7f6c7939f640 1 -- start start 2024-09-17T18:31:18.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.307+0000 7f6c7939f640 1 librados: setting wanted keys 2024-09-17T18:31:18.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.307+0000 7f6c7939f640 1 librados: calling monclient init 2024-09-17T18:31:18.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.307+0000 7f6c7939f640 10 monclient: init 2024-09-17T18:31:18.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.307+0000 7f6c7939f640 10 monclient: _reopen_session rank -1 2024-09-17T18:31:18.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.307+0000 7f6c7939f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T18:31:18.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.307+0000 7f6c7939f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c7412fa80 0x7f6c7413d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:18.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.307+0000 7f6c7939f640 10 monclient(hunting): picked mon.noname-b con 0x7f6c7412fa80 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:31:18.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.307+0000 7f6c7939f640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c7412fe60 0x7f6c7413c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:18.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.307+0000 7f6c7939f640 10 monclient(hunting): picked mon.noname-a con 0x7f6c7412fe60 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:31:18.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.307+0000 7f6c7939f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c74130270 0x7f6c74072870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:18.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.307+0000 7f6c7939f640 10 monclient(hunting): picked mon.noname-c con 0x7f6c74130270 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:31:18.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.307+0000 7f6c7939f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:18.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.307+0000 7f6c7939f640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6c7413dfd0 con 0x7f6c7412fe60 2024-09-17T18:31:18.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.307+0000 7f6c7939f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:18.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.307+0000 7f6c7939f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6c7413de50 con 0x7f6c7412fa80 2024-09-17T18:31:18.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.307+0000 7f6c7939f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:18.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.307+0000 7f6c7939f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6c7413e150 con 0x7f6c74130270 2024-09-17T18:31:18.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.307+0000 7f6c7939f640 10 monclient(hunting): _renew_subs 2024-09-17T18:31:18.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.307+0000 7f6c7939f640 10 monclient(hunting): authenticate will time out at 1482.023797s 2024-09-17T18:31:18.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.307+0000 7f6c737fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c7412fe60 0x7f6c7413c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:18.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.307+0000 7f6c78b9e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c74130270 0x7f6c74072870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:18.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.307+0000 7f6c737fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c7412fe60 0x7f6c7413c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42788/0 (socket says 172.21.15.121:42788) 2024-09-17T18:31:18.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.307+0000 7f6c78b9e640 10 monclient(hunting): get_auth_request con 0x7f6c74130270 auth_method 0 2024-09-17T18:31:18.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.307+0000 7f6c737fe640 1 -- 172.21.15.121:0/2491617396 learned_addr learned my addr 172.21.15.121:0/2491617396 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:31:18.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.307+0000 7f6c78b9e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:18.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.307+0000 7f6c78b9e640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:18.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.307+0000 7f6c78b9e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:18.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.307+0000 7f6c737fe640 10 monclient(hunting): get_auth_request con 0x7f6c7412fe60 auth_method 0 2024-09-17T18:31:18.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.307+0000 7f6c737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:18.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.307+0000 7f6c737fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:18.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.307+0000 7f6c737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:18.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.308+0000 7f6c78b9e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:18.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.308+0000 7f6c78b9e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:18.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.308+0000 7f6c78b9e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:18.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.308+0000 7f6c737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:18.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.308+0000 7f6c737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:18.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.308+0000 7f6c73fff640 1 --2- 172.21.15.121:0/2491617396 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c7412fa80 0x7f6c7413d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:18.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.308+0000 7f6c737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:18.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.308+0000 7f6c73fff640 10 monclient(hunting): get_auth_request con 0x7f6c7412fa80 auth_method 0 2024-09-17T18:31:18.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.308+0000 7f6c73fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:18.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.308+0000 7f6c73fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:18.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.308+0000 7f6c73fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:18.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.308+0000 7f6c78b9e640 10 monclient(hunting): handle_auth_done global_id 64283 payload 824 2024-09-17T18:31:18.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.308+0000 7f6c73fff640 1 -- 172.21.15.121:0/2491617396 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c7412fa80 msgr2=0x7f6c7413d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T18:31:18.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.308+0000 7f6c73fff640 1 -- 172.21.15.121:0/2491617396 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c7412fa80 msgr2=0x7f6c7413d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:31:18.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.308+0000 7f6c73fff640 1 --2- 172.21.15.121:0/2491617396 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c7412fa80 0x7f6c7413d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:31:18.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.308+0000 7f6c78b9e640 1 -- 172.21.15.121:0/2491617396 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c7412fa80 msgr2=0x7f6c7413d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:18.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.308+0000 7f6c73fff640 1 --2- 172.21.15.121:0/2491617396 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c7412fa80 0x7f6c7413d850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:31:18.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.308+0000 7f6c78b9e640 1 --2- 172.21.15.121:0/2491617396 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c7412fa80 0x7f6c7413d850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:18.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.308+0000 7f6c78b9e640 1 -- 172.21.15.121:0/2491617396 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c7412fe60 msgr2=0x7f6c7413c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:18.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.308+0000 7f6c78b9e640 1 --2- 172.21.15.121:0/2491617396 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c7412fe60 0x7f6c7413c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:18.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.308+0000 7f6c78b9e640 10 monclient: _finish_hunting 0 2024-09-17T18:31:18.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.308+0000 7f6c78b9e640 1 monclient: found mon.noname-c 2024-09-17T18:31:18.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.308+0000 7f6c78b9e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:31:18.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.308+0000 7f6c78b9e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:31:18.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.308+0000 7f6c78b9e640 1 -- 172.21.15.121:0/2491617396 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c7409fea0 con 0x7f6c74130270 2024-09-17T18:31:18.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.308+0000 7f6c737fe640 1 --2- 172.21.15.121:0/2491617396 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c7412fe60 0x7f6c7413c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:31:18.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.310+0000 7f6c78b9e640 1 -- 172.21.15.121:0/2491617396 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c74130270 msgr2=0x7f6c74072870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:18.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.310+0000 7f6c78b9e640 1 --2- 172.21.15.121:0/2491617396 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c74130270 0x7f6c74072870 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c007d90 tx=0x7f6c6c00e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:18.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.310+0000 7f6c70ff9640 10 client.?.objecter ms_handle_connect 0x7f6c74130270 2024-09-17T18:31:18.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.310+0000 7f6c70ff9640 10 client.?.objecter resend_mon_ops 2024-09-17T18:31:18.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.310+0000 7f6c70ff9640 1 -- 172.21.15.121:0/2491617396 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c003990 con 0x7f6c74130270 2024-09-17T18:31:18.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.310+0000 7f6c70ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:18.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.310+0000 7f6c70ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:31:18.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.310+0000 7f6c70ff9640 10 monclient: dump: 2024-09-17T18:31:18.322 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:18.323 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:18.323 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:18.323 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:18.323 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:18.323 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:18.323 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:18.323 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:18.323 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:18.323 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:18.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.310+0000 7f6c70ff9640 10 monclient: _finish_auth 0 2024-09-17T18:31:18.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.310+0000 7f6c70ff9640 10 monclient: _check_auth_tickets 2024-09-17T18:31:18.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.310+0000 7f6c70ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:31:18.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.310+0000 7f6c7939f640 5 monclient: authenticate success, global_id 64283 2024-09-17T18:31:18.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.310+0000 7f6c7939f640 10 monclient: _renew_subs 2024-09-17T18:31:18.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.310+0000 7f6c70ff9640 1 -- 172.21.15.121:0/2491617396 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c6c003b50 con 0x7f6c74130270 2024-09-17T18:31:18.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.310+0000 7f6c7939f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:31:18.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.310+0000 7f6c7939f640 1 -- 172.21.15.121:0/2491617396 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c7409e700 con 0x7f6c74130270 2024-09-17T18:31:18.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.310+0000 7f6c70ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T18:31:18.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.310+0000 7f6c70ff9640 1 -- 172.21.15.121:0/2491617396 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c004020 con 0x7f6c74130270 2024-09-17T18:31:18.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.310+0000 7f6c70ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:18.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.310+0000 7f6c70ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:31:18.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.310+0000 7f6c70ff9640 10 monclient: dump: 2024-09-17T18:31:18.324 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:18.324 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:18.324 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:18.324 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:18.324 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:18.324 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:18.325 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:18.325 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:18.325 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:18.325 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:18.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.310+0000 7f6c727fc640 4 set_mon_vals no callback set 2024-09-17T18:31:18.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.310+0000 7f6c7939f640 10 client.64283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:31:18.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.310+0000 7f6c7939f640 10 monclient: _renew_subs 2024-09-17T18:31:18.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.310+0000 7f6c7939f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:31:18.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.310+0000 7f6c7939f640 1 -- 172.21.15.121:0/2491617396 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c7412f020 con 0x7f6c74130270 2024-09-17T18:31:18.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.310+0000 7f6c7939f640 1 librados: init done 2024-09-17T18:31:18.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.311+0000 7f6c7939f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:31:18.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.311+0000 7f6c7939f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:31:18.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.311+0000 7f6c7939f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:31:18.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.311+0000 7f6c70ff9640 1 -- 172.21.15.121:0/2491617396 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f6c6c01b020 con 0x7f6c74130270 2024-09-17T18:31:18.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.311+0000 7f6c7939f640 1 -- 172.21.15.121:0/2491617396 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c740a2fd0 con 0x7f6c74130270 2024-09-17T18:31:18.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.311+0000 7f6c70ff9640 1 --2- 172.21.15.121:0/2491617396 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6c600401a0 0x7f6c60042660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:18.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.311+0000 7f6c70ff9640 1 -- 172.21.15.121:0/2491617396 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(481..481 src has 247..481) ==== 6686+0+0 (secure 0 0 0) 0x7f6c6c053d80 con 0x7f6c74130270 2024-09-17T18:31:18.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.311+0000 7f6c70ff9640 10 client.64283.objecter ms_dispatch 0x7f6c74000b90 osd_map(481..481 src has 247..481) 2024-09-17T18:31:18.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.311+0000 7f6c70ff9640 3 client.64283.objecter handle_osd_map got epochs [481,481] > 0 2024-09-17T18:31:18.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.311+0000 7f6c70ff9640 3 client.64283.objecter handle_osd_map decoding full epoch 481 2024-09-17T18:31:18.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.312+0000 7f6c70ff9640 20 client.64283.objecter dump_active .. 0 homeless 2024-09-17T18:31:18.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.312+0000 7f6c73fff640 1 --2- 172.21.15.121:0/2491617396 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6c600401a0 0x7f6c60042660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:18.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.312+0000 7f6c73fff640 10 monclient: get_auth_request con 0x7f6c600401a0 auth_method 0 2024-09-17T18:31:18.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.314+0000 7f6c70ff9640 10 client.64283.objecter ms_handle_connect 0x7f6c600401a0 2024-09-17T18:31:18.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.314+0000 7f6c73fff640 1 --2- 172.21.15.121:0/2491617396 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6c600401a0 0x7f6c60042660 secure :-1 s=READY pgs=4030 cs=0 l=1 rev1=1 crypto rx=0x7f6c64007fd0 tx=0x7f6c64009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:18.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.316+0000 7f6c70ff9640 1 -- 172.21.15.121:0/2491617396 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6c6c052010 con 0x7f6c74130270 2024-09-17T18:31:18.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.316+0000 7f6c70ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:31:18.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.316+0000 7f6c70ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:31:18.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.568+0000 7f6c7939f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:31:18.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.569+0000 7f6c7939f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:31:18.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.569+0000 7f6c7939f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:31:18.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.569+0000 7f6c7939f640 1 -- 172.21.15.121:0/2491617396 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6c74074830 con 0x7f6c74130270 2024-09-17T18:31:18.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.569+0000 7f6c70ff9640 1 -- 172.21.15.121:0/2491617396 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f6c6c014420 con 0x7f6c74130270 2024-09-17T18:31:18.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.569+0000 7f6c70ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:31:18.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.569+0000 7f6c70ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:31:18.571 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":44,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:31:18.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.571+0000 7f6c7939f640 10 librados: watch_flush enter 2024-09-17T18:31:18.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.571+0000 7f6c7939f640 10 librados: watch_flush exit 2024-09-17T18:31:18.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.571+0000 7f6c7939f640 20 client.64283.objecter shutdown clearing up homeless session... 2024-09-17T18:31:18.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.571+0000 7f6c7939f640 10 client.64283.objecter successfully canceled tick 2024-09-17T18:31:18.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.571+0000 7f6c7939f640 1 -- 172.21.15.121:0/2491617396 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6c600401a0 msgr2=0x7f6c60042660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:18.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.571+0000 7f6c7939f640 1 --2- 172.21.15.121:0/2491617396 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6c600401a0 0x7f6c60042660 secure :-1 s=READY pgs=4030 cs=0 l=1 rev1=1 crypto rx=0x7f6c64007fd0 tx=0x7f6c64009040 comp rx=0 tx=0).stop 2024-09-17T18:31:18.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.571+0000 7f6c7939f640 10 monclient: shutdown 2024-09-17T18:31:18.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.571+0000 7f6c7939f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:31:18.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.571+0000 7f6c7939f640 1 -- 172.21.15.121:0/2491617396 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c74130270 msgr2=0x7f6c74072870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:18.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.571+0000 7f6c7939f640 1 --2- 172.21.15.121:0/2491617396 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c74130270 0x7f6c74072870 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c007d90 tx=0x7f6c6c00e4b0 comp rx=0 tx=0).stop 2024-09-17T18:31:18.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.571+0000 7f6c7939f640 1 -- 172.21.15.121:0/2491617396 shutdown_connections 2024-09-17T18:31:18.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.571+0000 7f6c7939f640 1 --2- 172.21.15.121:0/2491617396 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6c600401a0 0x7f6c60042660 unknown :-1 s=CLOSED pgs=4030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:18.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.571+0000 7f6c7939f640 1 --2- 172.21.15.121:0/2491617396 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c74130270 0x7f6c74072870 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:18.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.571+0000 7f6c7939f640 1 --2- 172.21.15.121:0/2491617396 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c7412fe60 0x7f6c7413c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:18.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.571+0000 7f6c7939f640 1 --2- 172.21.15.121:0/2491617396 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c7412fa80 0x7f6c7413d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:18.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.571+0000 7f6c7939f640 1 -- 172.21.15.121:0/2491617396 >> 172.21.15.121:0/2491617396 conn(0x7f6c740a3890 msgr2=0x7f6c740a4c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:31:18.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.571+0000 7f6c7939f640 1 -- 172.21.15.121:0/2491617396 shutdown_connections 2024-09-17T18:31:18.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.571+0000 7f6c7939f640 1 -- 172.21.15.121:0/2491617396 wait complete. 2024-09-17T18:31:18.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.572+0000 7f6c7939f640 1 librados: shutdown 2024-09-17T18:31:18.583 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T18:31:18.584 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T18:31:18.584 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T18:31:18.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.680+0000 7f73c5165640 10 monclient: get_monmap_and_config 2024-09-17T18:31:18.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.680+0000 7f73c5165640 10 monclient: build_initial_monmap 2024-09-17T18:31:18.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.680+0000 7f73c5165640 10 monclient: monmap: 2024-09-17T18:31:18.682 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:31:18.682 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:18.682 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:31:18.680827+0000 2024-09-17T18:31:18.682 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:31:18.680827+0000 2024-09-17T18:31:18.682 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:31:18.682 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:18.682 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:31:18.682 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:31:18.682 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:31:18.682 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:31:18.682 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:31:18.682 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:18.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.680+0000 7f73c5165640 1 Processor -- start 2024-09-17T18:31:18.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.680+0000 7f73c5165640 1 -- start start 2024-09-17T18:31:18.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.680+0000 7f73c5165640 10 monclient: init 2024-09-17T18:31:18.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73c5165640 10 monclient: _reopen_session rank -1 2024-09-17T18:31:18.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73c5165640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-17T18:31:18.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73c5165640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f73c00700f0 0x7f73c0134e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:18.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73c5165640 10 monclient(hunting): picked mon.noname-a con 0x7f73c00700f0 addr v2:172.21.15.121:3300/0 2024-09-17T18:31:18.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73c5165640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73c009e810 0x7f73c0138360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:18.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73c5165640 10 monclient(hunting): picked mon.noname-b con 0x7f73c009e810 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:31:18.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73c5165640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f73c009ec60 addr v1:172.21.15.173:6789/0 2024-09-17T18:31:18.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73c5165640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73c009ca60 con 0x7f73c009ec60 2024-09-17T18:31:18.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73c5165640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:18.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73c5165640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f73c00704d0 con 0x7f73c00700f0 2024-09-17T18:31:18.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73c5165640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:18.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73c5165640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f73c009cd20 con 0x7f73c009e810 2024-09-17T18:31:18.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73c5165640 10 monclient(hunting): _renew_subs 2024-09-17T18:31:18.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73c5165640 10 monclient(hunting): authenticate will time out at 1482.397512s 2024-09-17T18:31:18.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73b7fff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73c009e810 0x7f73c0138360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:18.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73bed9d640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f73c00700f0 0x7f73c0134e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:18.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73b7fff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73c009e810 0x7f73c0138360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:54806/0 (socket says 172.21.15.121:54806) 2024-09-17T18:31:18.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73bed9d640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f73c00700f0 0x7f73c0134e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42790/0 (socket says 172.21.15.121:42790) 2024-09-17T18:31:18.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73b7fff640 1 -- 172.21.15.121:0/2520574915 learned_addr learned my addr 172.21.15.121:0/2520574915 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:31:18.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73b7fff640 10 monclient(hunting): get_auth_request con 0x7f73c009e810 auth_method 0 2024-09-17T18:31:18.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:18.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:18.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:18.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73bed9d640 10 monclient(hunting): get_auth_request con 0x7f73c00700f0 auth_method 0 2024-09-17T18:31:18.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73bed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:18.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73bed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:18.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73bed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:18.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:18.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:18.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:18.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73bed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:18.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73bed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:18.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73bed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:18.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73b7fff640 10 monclient(hunting): handle_auth_done global_id 64286 payload 274 2024-09-17T18:31:18.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73b7fff640 1 -- 172.21.15.121:0/2520574915 >> v2:172.21.15.121:3300/0 conn(0x7f73c00700f0 msgr2=0x7f73c0134e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:31:18.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73b7fff640 1 --2- 172.21.15.121:0/2520574915 >> v2:172.21.15.121:3300/0 conn(0x7f73c00700f0 0x7f73c0134e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:18.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73b7fff640 1 -- 172.21.15.121:0/2520574915 >> v1:172.21.15.173:6789/0 conn(0x7f73c009ec60 legacy=0x7f73c01388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:31:18.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73b7fff640 10 monclient: _finish_hunting 0 2024-09-17T18:31:18.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73b7fff640 1 monclient: found mon.noname-b 2024-09-17T18:31:18.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:31:18.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73b7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:31:18.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73b7fff640 1 -- 172.21.15.121:0/2520574915 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73c009d000 con 0x7f73c009e810 2024-09-17T18:31:18.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.681+0000 7f73bed9d640 1 --2- 172.21.15.121:0/2520574915 >> v2:172.21.15.121:3300/0 conn(0x7f73c00700f0 0x7f73c0134e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:31:18.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.684+0000 7f73b7fff640 1 -- 172.21.15.121:0/2520574915 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73c009e810 msgr2=0x7f73c0138360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:18.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.684+0000 7f73b7fff640 1 --2- 172.21.15.121:0/2520574915 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73c009e810 0x7f73c0138360 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f73a8009870 tx=0x7f73a802a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27d2501fa1cbf3c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:18.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.684+0000 7f73be59c640 1 -- 172.21.15.121:0/2520574915 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a8037070 con 0x7f73c009e810 2024-09-17T18:31:18.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.684+0000 7f73be59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:18.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.684+0000 7f73be59c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:31:18.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.684+0000 7f73be59c640 10 monclient: dump: 2024-09-17T18:31:18.694 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:18.694 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:18.694 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:18.694 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:18.694 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:18.695 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:18.695 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:18.695 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:18.695 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:18.695 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:18.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.684+0000 7f73be59c640 10 monclient: _finish_auth 0 2024-09-17T18:31:18.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.684+0000 7f73be59c640 10 monclient: _check_auth_tickets 2024-09-17T18:31:18.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.684+0000 7f73be59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:31:18.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.684+0000 7f73be59c640 1 -- 172.21.15.121:0/2520574915 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73a8032440 con 0x7f73c009e810 2024-09-17T18:31:18.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.684+0000 7f73c5165640 5 monclient: authenticate success, global_id 64286 2024-09-17T18:31:18.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.684+0000 7f73c5165640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:31:18.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.684+0000 7f73be59c640 10 monclient: handle_config config(0 keys) 2024-09-17T18:31:18.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.684+0000 7f73be59c640 1 -- 172.21.15.121:0/2520574915 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a8032810 con 0x7f73c009e810 2024-09-17T18:31:18.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.684+0000 7f73be59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:18.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.684+0000 7f73be59c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:31:18.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.684+0000 7f73be59c640 10 monclient: dump: 2024-09-17T18:31:18.696 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:18.696 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:18.696 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:18.696 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:18.696 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:18.696 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:18.696 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:18.696 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:18.696 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:18.696 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:18.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.684+0000 7f73c5165640 10 monclient: get_monmap_and_config success 2024-09-17T18:31:18.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.684+0000 7f73c5165640 4 set_mon_vals no callback set 2024-09-17T18:31:18.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.685+0000 7f73c5165640 10 monclient: shutdown 2024-09-17T18:31:18.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.685+0000 7f73c5165640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:31:18.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.685+0000 7f73c5165640 1 -- 172.21.15.121:0/2520574915 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73c009e810 msgr2=0x7f73c0138360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:18.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.685+0000 7f73c5165640 1 --2- 172.21.15.121:0/2520574915 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73c009e810 0x7f73c0138360 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f73a8009870 tx=0x7f73a802a0a0 comp rx=0 tx=0).stop 2024-09-17T18:31:18.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.685+0000 7f73c5165640 1 -- 172.21.15.121:0/2520574915 shutdown_connections 2024-09-17T18:31:18.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.685+0000 7f73c5165640 1 --2- 172.21.15.121:0/2520574915 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73c009e810 0x7f73c0138360 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:18.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.685+0000 7f73c5165640 1 --2- 172.21.15.121:0/2520574915 >> v2:172.21.15.121:3300/0 conn(0x7f73c00700f0 0x7f73c0134e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:18.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.685+0000 7f73c5165640 1 -- 172.21.15.121:0/2520574915 >> 172.21.15.121:0/2520574915 conn(0x7f73c00ab830 msgr2=0x7f73c00a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:31:18.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.685+0000 7f73c5165640 1 -- 172.21.15.121:0/2520574915 shutdown_connections 2024-09-17T18:31:18.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.685+0000 7f73c5165640 1 -- 172.21.15.121:0/2520574915 wait complete. 2024-09-17T18:31:18.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.686+0000 7f73c5165640 10 monclient: build_initial_monmap 2024-09-17T18:31:18.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.686+0000 7f73c5165640 10 monclient: monmap: 2024-09-17T18:31:18.697 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:31:18.698 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:31:18.698 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:31:18.698 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:31:18.698 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:31:18.698 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:18.698 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:31:18.698 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:31:18.698 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:31:18.698 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:18.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.686+0000 7f73c5165640 1 librados: starting msgr at 2024-09-17T18:31:18.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.686+0000 7f73c5165640 1 librados: starting objecter 2024-09-17T18:31:18.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.686+0000 7f73c5165640 1 Processor -- start 2024-09-17T18:31:18.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.686+0000 7f73c5165640 1 -- start start 2024-09-17T18:31:18.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.686+0000 7f73c5165640 1 librados: setting wanted keys 2024-09-17T18:31:18.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.686+0000 7f73c5165640 1 librados: calling monclient init 2024-09-17T18:31:18.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.686+0000 7f73c5165640 10 monclient: init 2024-09-17T18:31:18.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.686+0000 7f73c5165640 10 monclient: _reopen_session rank -1 2024-09-17T18:31:18.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.686+0000 7f73c5165640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T18:31:18.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.686+0000 7f73c5165640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73c00700f0 0x7f73c00a4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:18.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.687+0000 7f73c5165640 10 monclient(hunting): picked mon.noname-c con 0x7f73c00700f0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:31:18.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.687+0000 7f73c5165640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73c009e810 0x7f73c01cd590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:18.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.687+0000 7f73c5165640 10 monclient(hunting): picked mon.noname-b con 0x7f73c009e810 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:31:18.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.687+0000 7f73c5165640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73c009ec60 0x7f73c01cdad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:18.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.687+0000 7f73c5165640 10 monclient(hunting): picked mon.noname-a con 0x7f73c009ec60 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:31:18.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.687+0000 7f73c5165640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:18.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.687+0000 7f73c5165640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f73c0135510 con 0x7f73c009ec60 2024-09-17T18:31:18.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.687+0000 7f73c5165640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:18.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.687+0000 7f73c5165640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f73c0135390 con 0x7f73c009e810 2024-09-17T18:31:18.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.687+0000 7f73c5165640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:18.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.687+0000 7f73c5165640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f73c0138fb0 con 0x7f73c00700f0 2024-09-17T18:31:18.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.687+0000 7f73c5165640 10 monclient(hunting): _renew_subs 2024-09-17T18:31:18.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.687+0000 7f73c5165640 10 monclient(hunting): authenticate will time out at 1482.403079s 2024-09-17T18:31:18.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.687+0000 7f73bed9d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73c00700f0 0x7f73c00a4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:18.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.687+0000 7f73bed9d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73c00700f0 0x7f73c00a4a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:54814/0 (socket says 172.21.15.121:54814) 2024-09-17T18:31:18.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.687+0000 7f73bed9d640 1 -- 172.21.15.121:0/497489412 learned_addr learned my addr 172.21.15.121:0/497489412 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:31:18.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.687+0000 7f73bed9d640 10 monclient(hunting): get_auth_request con 0x7f73c00700f0 auth_method 0 2024-09-17T18:31:18.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.687+0000 7f73bed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:18.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.687+0000 7f73bed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:18.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.687+0000 7f73bed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:18.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.687+0000 7f73bf59e640 1 --2- 172.21.15.121:0/497489412 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73c009ec60 0x7f73c01cdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:18.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.687+0000 7f73bed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:18.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.687+0000 7f73bed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:18.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.687+0000 7f73bed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:18.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.687+0000 7f73bf59e640 10 monclient(hunting): get_auth_request con 0x7f73c009ec60 auth_method 0 2024-09-17T18:31:18.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.687+0000 7f73bf59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:18.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.687+0000 7f73bf59e640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:18.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.687+0000 7f73bf59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:18.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.687+0000 7f73bf59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:18.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.687+0000 7f73bf59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:18.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.687+0000 7f73bf59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:18.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.687+0000 7f73bed9d640 10 monclient(hunting): handle_auth_done global_id 64289 payload 824 2024-09-17T18:31:18.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.687+0000 7f73bed9d640 1 -- 172.21.15.121:0/497489412 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73c009e810 msgr2=0x7f73c01cd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:18.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.687+0000 7f73bed9d640 1 --2- 172.21.15.121:0/497489412 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73c009e810 0x7f73c01cd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:18.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.687+0000 7f73bed9d640 1 -- 172.21.15.121:0/497489412 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73c009ec60 msgr2=0x7f73c01cdad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:18.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.687+0000 7f73bed9d640 1 --2- 172.21.15.121:0/497489412 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73c009ec60 0x7f73c01cdad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:18.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.687+0000 7f73bed9d640 10 monclient: _finish_hunting 0 2024-09-17T18:31:18.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.687+0000 7f73bed9d640 1 monclient: found mon.noname-c 2024-09-17T18:31:18.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.687+0000 7f73bed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:31:18.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.687+0000 7f73bed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:31:18.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.687+0000 7f73bed9d640 1 -- 172.21.15.121:0/497489412 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73c00a42c0 con 0x7f73c00700f0 2024-09-17T18:31:18.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.689+0000 7f73bed9d640 1 -- 172.21.15.121:0/497489412 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73c00700f0 msgr2=0x7f73c00a4a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:18.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.689+0000 7f73bed9d640 1 --2- 172.21.15.121:0/497489412 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73c00700f0 0x7f73c00a4a50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f73b000e960 tx=0x7f73b000ee20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:18.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.689+0000 7f73b77fe640 10 client.?.objecter ms_handle_connect 0x7f73c00700f0 2024-09-17T18:31:18.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.689+0000 7f73b77fe640 10 client.?.objecter resend_mon_ops 2024-09-17T18:31:18.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.689+0000 7f73b77fe640 1 -- 172.21.15.121:0/497489412 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73b0017070 con 0x7f73c00700f0 2024-09-17T18:31:18.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.689+0000 7f73b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:18.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.689+0000 7f73b77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:31:18.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.689+0000 7f73b77fe640 10 monclient: dump: 2024-09-17T18:31:18.702 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:18.702 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:18.702 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:18.702 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:18.702 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:18.703 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:18.703 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:18.703 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:18.703 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:18.703 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:18.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.689+0000 7f73b77fe640 10 monclient: _finish_auth 0 2024-09-17T18:31:18.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.689+0000 7f73b77fe640 10 monclient: _check_auth_tickets 2024-09-17T18:31:18.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.689+0000 7f73b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:31:18.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.689+0000 7f73c5165640 5 monclient: authenticate success, global_id 64289 2024-09-17T18:31:18.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.689+0000 7f73b77fe640 1 -- 172.21.15.121:0/497489412 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73b000cc10 con 0x7f73c00700f0 2024-09-17T18:31:18.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.689+0000 7f73b77fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:31:18.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.689+0000 7f73b77fe640 1 -- 172.21.15.121:0/497489412 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73b0013470 con 0x7f73c00700f0 2024-09-17T18:31:18.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.689+0000 7f73c5165640 10 monclient: _renew_subs 2024-09-17T18:31:18.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.689+0000 7f73c5165640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:31:18.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.689+0000 7f73c5165640 1 -- 172.21.15.121:0/497489412 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73c0132510 con 0x7f73c00700f0 2024-09-17T18:31:18.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.689+0000 7f73b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:18.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.689+0000 7f73b77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:31:18.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.689+0000 7f73bdd9b640 4 set_mon_vals no callback set 2024-09-17T18:31:18.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.689+0000 7f73b77fe640 10 monclient: dump: 2024-09-17T18:31:18.704 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:18.704 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:18.704 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:18.704 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:18.704 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:18.704 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:18.704 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:18.704 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:18.704 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:18.705 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:18.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.689+0000 7f73c5165640 10 client.64289.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:31:18.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.689+0000 7f73c5165640 10 monclient: _renew_subs 2024-09-17T18:31:18.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.689+0000 7f73c5165640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:31:18.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.689+0000 7f73c5165640 1 -- 172.21.15.121:0/497489412 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f73c01329b0 con 0x7f73c00700f0 2024-09-17T18:31:18.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.689+0000 7f73c5165640 1 librados: init done 2024-09-17T18:31:18.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.690+0000 7f73c5165640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:31:18.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.690+0000 7f73c5165640 1 --2- 172.21.15.121:0/497489412 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73c01ce010 0x7f73c012e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:18.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.690+0000 7f73c5165640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:18.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.690+0000 7f73c5165640 1 -- 172.21.15.121:0/497489412 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f73c01ce3f0 con 0x7f73c01ce010 2024-09-17T18:31:18.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.690+0000 7f73b7fff640 1 --2- 172.21.15.121:0/497489412 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73c01ce010 0x7f73c012e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:18.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.690+0000 7f73b7fff640 10 monclient: get_auth_request con 0x7f73c01ce010 auth_method 0 2024-09-17T18:31:18.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.690+0000 7f73b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:18.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.690+0000 7f73b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:18.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.690+0000 7f73b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:18.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.690+0000 7f73b77fe640 1 -- 172.21.15.121:0/497489412 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f73b001a070 con 0x7f73c00700f0 2024-09-17T18:31:18.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.690+0000 7f73b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:18.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.690+0000 7f73b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:18.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.690+0000 7f73b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:18.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.690+0000 7f73b7fff640 10 monclient(hunting): handle_auth_done global_id 44655 payload 824 2024-09-17T18:31:18.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.690+0000 7f73b7fff640 1 -- 172.21.15.121:0/497489412 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f73c006fe80 con 0x7f73c01ce010 2024-09-17T18:31:18.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.690+0000 7f73b77fe640 1 --2- 172.21.15.121:0/497489412 >> v2:172.21.15.173:6812/2423786037 conn(0x7f73a00401a0 0x7f73a0042660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:18.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.691+0000 7f73b77fe640 1 -- 172.21.15.121:0/497489412 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(482..482 src has 247..482) ==== 6686+0+0 (secure 0 0 0) 0x7f73b0053b60 con 0x7f73c00700f0 2024-09-17T18:31:18.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.691+0000 7f73b77fe640 10 client.64289.objecter ms_dispatch 0x7f73c0000b90 osd_map(482..482 src has 247..482) 2024-09-17T18:31:18.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.691+0000 7f73b77fe640 3 client.64289.objecter handle_osd_map got epochs [482,482] > 0 2024-09-17T18:31:18.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.691+0000 7f73b77fe640 3 client.64289.objecter handle_osd_map decoding full epoch 482 2024-09-17T18:31:18.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.691+0000 7f73b77fe640 20 client.64289.objecter dump_active .. 0 homeless 2024-09-17T18:31:18.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.691+0000 7f73bf59e640 1 --2- 172.21.15.121:0/497489412 >> v2:172.21.15.173:6812/2423786037 conn(0x7f73a00401a0 0x7f73a0042660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:18.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.691+0000 7f73bf59e640 10 monclient: get_auth_request con 0x7f73a00401a0 auth_method 0 2024-09-17T18:31:18.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.693+0000 7f73b77fe640 10 client.64289.objecter ms_handle_connect 0x7f73c01ce010 2024-09-17T18:31:18.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.693+0000 7f73b7fff640 1 -- 172.21.15.121:0/497489412 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73c01ce010 msgr2=0x7f73c012e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:18.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.693+0000 7f73b77fe640 10 client.64289.objecter resend_mon_ops 2024-09-17T18:31:18.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.693+0000 7f73b7fff640 1 --2- 172.21.15.121:0/497489412 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73c01ce010 0x7f73c012e920 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f73c00a90b0 tx=0x7f73a8032b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:18.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.693+0000 7f73b77fe640 1 -- 172.21.15.121:0/497489412 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a8041030 con 0x7f73c01ce010 2024-09-17T18:31:18.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.693+0000 7f73b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:18.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.693+0000 7f73b77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:31:18.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.693+0000 7f73b77fe640 10 monclient: dump: 2024-09-17T18:31:18.709 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:18.709 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:18.709 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:18.709 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:18.709 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:18.709 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:18.709 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:18.710 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:18.710 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:18.710 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:18.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.693+0000 7f73b77fe640 1 -- 172.21.15.121:0/497489412 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f73a803d420 con 0x7f73c01ce010 2024-09-17T18:31:18.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.693+0000 7f73b77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:31:18.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.693+0000 7f73b77fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:31:18.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.693+0000 7f73b77fe640 1 -- 172.21.15.121:0/497489412 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73c01ce010 msgr2=0x7f73c012e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:18.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.693+0000 7f73b77fe640 1 --2- 172.21.15.121:0/497489412 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73c01ce010 0x7f73c012e920 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f73c00a90b0 tx=0x7f73a8032b70 comp rx=0 tx=0).stop 2024-09-17T18:31:18.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.693+0000 7f73b77fe640 1 -- 172.21.15.121:0/497489412 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73c01ce010 msgr2=0x7f73c012e920 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:31:18.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.693+0000 7f73b77fe640 1 --2- 172.21.15.121:0/497489412 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73c01ce010 0x7f73c012e920 secure :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f73c00a90b0 tx=0x7f73a8032b70 comp rx=0 tx=0).stop 2024-09-17T18:31:18.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.693+0000 7f73bf59e640 1 --2- 172.21.15.121:0/497489412 >> v2:172.21.15.173:6812/2423786037 conn(0x7f73a00401a0 0x7f73a0042660 secure :-1 s=READY pgs=4031 cs=0 l=1 rev1=1 crypto rx=0x7f73c00a2680 tx=0x7f73b8003d10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:18.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.693+0000 7f73b77fe640 10 client.64289.objecter ms_handle_connect 0x7f73a00401a0 2024-09-17T18:31:18.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.699+0000 7f73c5165640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:31:18.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.699+0000 7f73c5165640 1 --2- 172.21.15.121:0/497489412 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73c012ee60 0x7f73c012f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:18.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.699+0000 7f73c5165640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:18.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.699+0000 7f73c5165640 1 -- 172.21.15.121:0/497489412 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f73c012f7b0 con 0x7f73c012ee60 2024-09-17T18:31:18.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.699+0000 7f73b7fff640 1 --2- 172.21.15.121:0/497489412 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73c012ee60 0x7f73c012f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:18.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.699+0000 7f73b7fff640 10 monclient: get_auth_request con 0x7f73c012ee60 auth_method 0 2024-09-17T18:31:18.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.699+0000 7f73b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:18.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.699+0000 7f73b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:18.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.699+0000 7f73b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:18.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.699+0000 7f73b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:18.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.699+0000 7f73b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:18.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.699+0000 7f73b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:18.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.699+0000 7f73b7fff640 10 monclient(hunting): handle_auth_done global_id 44658 payload 824 2024-09-17T18:31:18.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.699+0000 7f73b7fff640 1 -- 172.21.15.121:0/497489412 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f73c0134e50 con 0x7f73c012ee60 2024-09-17T18:31:18.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.702+0000 7f73b7fff640 1 -- 172.21.15.121:0/497489412 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73c012ee60 msgr2=0x7f73c012f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:18.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.702+0000 7f73b77fe640 10 client.64289.objecter ms_handle_connect 0x7f73c012ee60 2024-09-17T18:31:18.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.702+0000 7f73b77fe640 10 client.64289.objecter resend_mon_ops 2024-09-17T18:31:18.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.702+0000 7f73b7fff640 1 --2- 172.21.15.121:0/497489412 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73c012ee60 0x7f73c012f270 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f73a8009870 tx=0x7f73a8008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:18.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.702+0000 7f73b77fe640 1 -- 172.21.15.121:0/497489412 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a804d070 con 0x7f73c012ee60 2024-09-17T18:31:18.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.702+0000 7f73b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:18.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.702+0000 7f73b77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:31:18.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.702+0000 7f73b77fe640 10 monclient: dump: 2024-09-17T18:31:18.712 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:18.712 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:18.712 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:18.712 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:18.712 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:18.712 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:18.712 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:18.713 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:18.713 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:18.713 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:18.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.702+0000 7f73b77fe640 1 -- 172.21.15.121:0/497489412 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7f73a8051410 con 0x7f73c012ee60 2024-09-17T18:31:18.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.702+0000 7f73b77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:31:18.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.702+0000 7f73b77fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:31:18.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.702+0000 7f73b77fe640 1 -- 172.21.15.121:0/497489412 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73c012ee60 msgr2=0x7f73c012f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:18.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.702+0000 7f73b77fe640 1 --2- 172.21.15.121:0/497489412 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73c012ee60 0x7f73c012f270 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f73a8009870 tx=0x7f73a8008030 comp rx=0 tx=0).stop 2024-09-17T18:31:18.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.702+0000 7f73b77fe640 1 -- 172.21.15.121:0/497489412 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73c012ee60 msgr2=0x7f73c012f270 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:31:18.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.702+0000 7f73b77fe640 1 --2- 172.21.15.121:0/497489412 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73c012ee60 0x7f73c012f270 secure :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f73a8009870 tx=0x7f73a8008030 comp rx=0 tx=0).stop 2024-09-17T18:31:18.713 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:31:18.714 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:31:18.714 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:31:18.714 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "leader", 2024-09-17T18:31:18.714 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 44, 2024-09-17T18:31:18.714 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 504366, 2024-09-17T18:31:18.714 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:31:18.714 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:31:18.714 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:31:18.714 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:31:18.714 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:31:18.714 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 2, 2024-09-17T18:31:18.714 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:31:18.714 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:31:18.714 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:31:18.714 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:31:18.715 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:31:18.715 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:31:18.715 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:31:18.715 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:31:18.715 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:31:18.715 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:31:18.715 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:31:18.715 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:31:18.715 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:31:18.715 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:31:18.715 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:31:18.715 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:31:18.715 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:31:18.715 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:31:18.715 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:31:18.716 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:31:18.716 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:31:18.716 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:31:18.716 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:31:18.716 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:31:18.716 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:31:18.716 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:31:18.716 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:31:18.716 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:31:18.716 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:18.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:18.716 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:31:18.716 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:31:18.716 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:31:18.717 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:31:18.717 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:31:18.717 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:31:18.717 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:31:18.717 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:31:18.717 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:31:18.717 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:31:18.717 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:31:18.717 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:31:18.717 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:31:18.717 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:31:18.717 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:31:18.717 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:31:18.717 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:31:18.718 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:31:18.718 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:31:18.718 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:31:18.718 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:31:18.718 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:31:18.718 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:31:18.718 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:31:18.718 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:31:18.718 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:31:18.718 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:31:18.718 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:31:18.718 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:31:18.718 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:31:18.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:18.718 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:31:18.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:18.719 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:31:18.719 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:31:18.719 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:31:18.719 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:31:18.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:18.719 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:31:18.719 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:31:18.719 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:18.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:18.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:18.719 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:31:18.719 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:31:18.719 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:18.719 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:18.719 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:18.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:18.720 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:31:18.720 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:31:18.720 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:31:18.720 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:31:18.720 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:31:18.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:18.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:18.720 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:31:18.720 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:31:18.720 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:31:18.720 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:31:18.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:18.720 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:31:18.720 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:31:18.720 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:18.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:18.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:18.721 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:31:18.721 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:31:18.721 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:18.721 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:18.721 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:18.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:18.721 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:31:18.721 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:31:18.721 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:31:18.721 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:31:18.721 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:31:18.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:18.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:18.721 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:31:18.722 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:31:18.722 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:31:18.722 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:31:18.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:18.722 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:31:18.722 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:31:18.722 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:18.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:18.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:18.722 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:31:18.722 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:31:18.722 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:18.722 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:18.722 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:18.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:18.722 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:31:18.723 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:31:18.723 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:31:18.723 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:31:18.723 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:31:18.723 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:18.723 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:18.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:18.723 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:31:18.723 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:31:18.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:18.723 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:31:18.723 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:31:18.723 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:31:18.723 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:18.723 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:31:18.723 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:31:18.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:18.724 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:31:18.724 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:31:18.724 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 3 2024-09-17T18:31:18.724 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:18.724 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:31:18.724 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:31:18.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:18.724 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:31:18.724 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:31:18.724 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 4 2024-09-17T18:31:18.724 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:18.724 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:18.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:18.724 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:31:18.725 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:31:18.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.702+0000 7f73c5165640 10 librados: watch_flush enter 2024-09-17T18:31:18.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.702+0000 7f73c5165640 10 librados: watch_flush exit 2024-09-17T18:31:18.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.702+0000 7f73c5165640 20 client.64289.objecter shutdown clearing up homeless session... 2024-09-17T18:31:18.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.702+0000 7f73c5165640 10 client.64289.objecter successfully canceled tick 2024-09-17T18:31:18.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.702+0000 7f73c5165640 1 -- 172.21.15.121:0/497489412 >> v2:172.21.15.173:6812/2423786037 conn(0x7f73a00401a0 msgr2=0x7f73a0042660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:18.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.702+0000 7f73c5165640 1 --2- 172.21.15.121:0/497489412 >> v2:172.21.15.173:6812/2423786037 conn(0x7f73a00401a0 0x7f73a0042660 secure :-1 s=READY pgs=4031 cs=0 l=1 rev1=1 crypto rx=0x7f73c00a2680 tx=0x7f73b8003d10 comp rx=0 tx=0).stop 2024-09-17T18:31:18.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.702+0000 7f73c5165640 10 monclient: shutdown 2024-09-17T18:31:18.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.702+0000 7f73c5165640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:31:18.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.702+0000 7f73c5165640 1 -- 172.21.15.121:0/497489412 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73c00700f0 msgr2=0x7f73c00a4a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:18.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.702+0000 7f73c5165640 1 --2- 172.21.15.121:0/497489412 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73c00700f0 0x7f73c00a4a50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f73b000e960 tx=0x7f73b000ee20 comp rx=0 tx=0).stop 2024-09-17T18:31:18.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.702+0000 7f73bf59e640 1 -- 172.21.15.121:0/497489412 reap_dead start 2024-09-17T18:31:18.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.702+0000 7f73bf59e640 1 -- 172.21.15.121:0/497489412 reap_dead start 2024-09-17T18:31:18.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.702+0000 7f73c5165640 1 -- 172.21.15.121:0/497489412 shutdown_connections 2024-09-17T18:31:18.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.702+0000 7f73c5165640 1 -- 172.21.15.121:0/497489412 >> 172.21.15.121:0/497489412 conn(0x7f73c00ab830 msgr2=0x7f73c00a1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:31:18.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.702+0000 7f73c5165640 1 -- 172.21.15.121:0/497489412 shutdown_connections 2024-09-17T18:31:18.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.702+0000 7f73c5165640 1 -- 172.21.15.121:0/497489412 wait complete. 2024-09-17T18:31:18.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.703+0000 7f73c5165640 1 librados: shutdown 2024-09-17T18:31:18.729 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:31:18.729 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T18:31:18.730 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T18:31:18.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.825+0000 7f9a4e0bf640 10 monclient: get_monmap_and_config 2024-09-17T18:31:18.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.825+0000 7f9a4e0bf640 10 monclient: build_initial_monmap 2024-09-17T18:31:18.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.825+0000 7f9a4e0bf640 10 monclient: monmap: 2024-09-17T18:31:18.827 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:31:18.827 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:18.827 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:31:18.826180+0000 2024-09-17T18:31:18.827 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:31:18.826180+0000 2024-09-17T18:31:18.827 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:31:18.827 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:18.827 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:31:18.827 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:31:18.828 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:31:18.828 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:31:18.828 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:31:18.828 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:18.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.826+0000 7f9a4e0bf640 1 Processor -- start 2024-09-17T18:31:18.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.826+0000 7f9a4e0bf640 1 -- start start 2024-09-17T18:31:18.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.826+0000 7f9a4e0bf640 10 monclient: init 2024-09-17T18:31:18.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.826+0000 7f9a4e0bf640 10 monclient: _reopen_session rank -1 2024-09-17T18:31:18.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.826+0000 7f9a4e0bf640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-17T18:31:18.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.826+0000 7f9a4e0bf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9a4809da70 addr v1:172.21.15.121:6789/0 2024-09-17T18:31:18.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.827+0000 7f9a4e0bf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9a481368d0 addr v1:172.21.15.173:6789/0 2024-09-17T18:31:18.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.827+0000 7f9a4e0bf640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f9a48136d00 0x7f9a4813ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:18.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.827+0000 7f9a4e0bf640 10 monclient(hunting): picked mon.noname-a con 0x7f9a48136d00 addr v2:172.21.15.121:3300/0 2024-09-17T18:31:18.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.827+0000 7f9a4e0bf640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a480a2d40 con 0x7f9a4809da70 2024-09-17T18:31:18.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.827+0000 7f9a4e0bf640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a480a3020 con 0x7f9a481368d0 2024-09-17T18:31:18.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.827+0000 7f9a4e0bf640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:18.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.827+0000 7f9a4e0bf640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f9a4809de50 con 0x7f9a48136d00 2024-09-17T18:31:18.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.827+0000 7f9a4e0bf640 10 monclient(hunting): _renew_subs 2024-09-17T18:31:18.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.827+0000 7f9a4e0bf640 10 monclient(hunting): authenticate will time out at 1482.543094s 2024-09-17T18:31:18.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.827+0000 7f9a477fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f9a4809da70 0x7f9a481361c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:53062/0 (socket says 172.21.15.121:53062) 2024-09-17T18:31:18.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.827+0000 7f9a477fe640 1 -- 172.21.15.121:0/3345522542 learned_addr learned my addr 172.21.15.121:0/3345522542 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:31:18.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.827+0000 7f9a47fff640 1 --2- 172.21.15.121:0/3345522542 >> v2:172.21.15.121:3300/0 conn(0x7f9a48136d00 0x7f9a4813ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:18.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.827+0000 7f9a47fff640 10 monclient(hunting): get_auth_request con 0x7f9a48136d00 auth_method 0 2024-09-17T18:31:18.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.827+0000 7f9a47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:18.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.827+0000 7f9a47fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:18.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.827+0000 7f9a47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:18.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.827+0000 7f9a47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:18.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.827+0000 7f9a47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:18.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.827+0000 7f9a47fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:18.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.827+0000 7f9a47fff640 10 monclient(hunting): handle_auth_done global_id 44661 payload 274 2024-09-17T18:31:18.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.827+0000 7f9a47fff640 1 -- 172.21.15.121:0/3345522542 >> v1:172.21.15.173:6789/0 conn(0x7f9a481368d0 legacy=0x7f9a4813b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:31:18.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.827+0000 7f9a47fff640 1 -- 172.21.15.121:0/3345522542 >> v1:172.21.15.121:6789/0 conn(0x7f9a4809da70 legacy=0x7f9a481361c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:31:18.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.827+0000 7f9a47fff640 10 monclient: _finish_hunting 0 2024-09-17T18:31:18.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.827+0000 7f9a47fff640 1 monclient: found mon.noname-a 2024-09-17T18:31:18.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.827+0000 7f9a47fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:31:18.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.827+0000 7f9a47fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:31:18.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.827+0000 7f9a47fff640 1 -- 172.21.15.121:0/3345522542 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a480a3300 con 0x7f9a48136d00 2024-09-17T18:31:18.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.830+0000 7f9a47fff640 1 -- 172.21.15.121:0/3345522542 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a48136d00 msgr2=0x7f9a4813ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:18.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.830+0000 7f9a47fff640 1 --2- 172.21.15.121:0/3345522542 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a48136d00 0x7f9a4813ec70 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f9a40009d90 tx=0x7f9a4002a490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9d59e5387c74a72 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:18.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.830+0000 7f9a46ffd640 1 -- 172.21.15.121:0/3345522542 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a40007a30 con 0x7f9a48136d00 2024-09-17T18:31:18.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.830+0000 7f9a46ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:18.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.830+0000 7f9a46ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:31:18.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.830+0000 7f9a46ffd640 10 monclient: dump: 2024-09-17T18:31:18.836 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:18.836 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:18.836 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:18.836 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:18.836 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:18.836 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:18.836 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:18.836 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:18.836 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:18.836 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:18.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.830+0000 7f9a46ffd640 10 monclient: _finish_auth 0 2024-09-17T18:31:18.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.830+0000 7f9a46ffd640 10 monclient: _check_auth_tickets 2024-09-17T18:31:18.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.830+0000 7f9a46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:31:18.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.830+0000 7f9a46ffd640 1 -- 172.21.15.121:0/3345522542 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a40007bd0 con 0x7f9a48136d00 2024-09-17T18:31:18.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.830+0000 7f9a4e0bf640 5 monclient: authenticate success, global_id 44661 2024-09-17T18:31:18.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.830+0000 7f9a4e0bf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:31:18.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.830+0000 7f9a46ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:31:18.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.830+0000 7f9a4e0bf640 10 monclient: get_monmap_and_config success 2024-09-17T18:31:18.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.830+0000 7f9a46ffd640 1 -- 172.21.15.121:0/3345522542 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a40031400 con 0x7f9a48136d00 2024-09-17T18:31:18.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.830+0000 7f9a4e0bf640 4 set_mon_vals no callback set 2024-09-17T18:31:18.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.830+0000 7f9a46ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:18.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.830+0000 7f9a46ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:31:18.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.830+0000 7f9a46ffd640 10 monclient: dump: 2024-09-17T18:31:18.837 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:18.837 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:18.838 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:18.838 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:18.838 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:18.838 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:18.838 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:18.838 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:18.838 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:18.838 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:18.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.830+0000 7f9a4e0bf640 10 monclient: shutdown 2024-09-17T18:31:18.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.830+0000 7f9a4e0bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:31:18.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.830+0000 7f9a4e0bf640 1 -- 172.21.15.121:0/3345522542 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a48136d00 msgr2=0x7f9a4813ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:18.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.830+0000 7f9a4e0bf640 1 --2- 172.21.15.121:0/3345522542 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a48136d00 0x7f9a4813ec70 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f9a40009d90 tx=0x7f9a4002a490 comp rx=0 tx=0).stop 2024-09-17T18:31:18.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.831+0000 7f9a4e0bf640 1 -- 172.21.15.121:0/3345522542 shutdown_connections 2024-09-17T18:31:18.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.831+0000 7f9a4e0bf640 1 --2- 172.21.15.121:0/3345522542 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a48136d00 0x7f9a4813ec70 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:18.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.831+0000 7f9a4e0bf640 1 -- 172.21.15.121:0/3345522542 >> 172.21.15.121:0/3345522542 conn(0x7f9a480a7830 msgr2=0x7f9a480a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:31:18.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.831+0000 7f9a4e0bf640 1 -- 172.21.15.121:0/3345522542 shutdown_connections 2024-09-17T18:31:18.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.831+0000 7f9a4e0bf640 1 -- 172.21.15.121:0/3345522542 wait complete. 2024-09-17T18:31:18.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.831+0000 7f9a4e0bf640 10 monclient: build_initial_monmap 2024-09-17T18:31:18.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.831+0000 7f9a4e0bf640 10 monclient: monmap: 2024-09-17T18:31:18.839 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:31:18.839 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:31:18.839 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:31:18.839 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:31:18.839 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:31:18.839 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:18.839 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:31:18.839 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:31:18.839 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:31:18.839 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:18.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.831+0000 7f9a4e0bf640 1 librados: starting msgr at 2024-09-17T18:31:18.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.831+0000 7f9a4e0bf640 1 librados: starting objecter 2024-09-17T18:31:18.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.831+0000 7f9a4e0bf640 1 Processor -- start 2024-09-17T18:31:18.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a4e0bf640 1 -- start start 2024-09-17T18:31:18.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a4e0bf640 1 librados: setting wanted keys 2024-09-17T18:31:18.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a4e0bf640 1 librados: calling monclient init 2024-09-17T18:31:18.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a4e0bf640 10 monclient: init 2024-09-17T18:31:18.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a4e0bf640 10 monclient: _reopen_session rank -1 2024-09-17T18:31:18.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a4e0bf640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T18:31:18.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a4e0bf640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a4809da70 0x7f9a481cda10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:18.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a4e0bf640 10 monclient(hunting): picked mon.noname-c con 0x7f9a4809da70 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:31:18.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a4e0bf640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a481368d0 0x7f9a4813e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:18.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a4e0bf640 10 monclient(hunting): picked mon.noname-b con 0x7f9a481368d0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:31:18.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a4e0bf640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a48136d00 0x7f9a4813ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:18.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a4e0bf640 10 monclient(hunting): picked mon.noname-a con 0x7f9a48136d00 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:31:18.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a4e0bf640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:18.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a4e0bf640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9a4813f1b0 con 0x7f9a48136d00 2024-09-17T18:31:18.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a4e0bf640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:18.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a4e0bf640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9a480a3930 con 0x7f9a481368d0 2024-09-17T18:31:18.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a4e0bf640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:18.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a4e0bf640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9a4813f330 con 0x7f9a4809da70 2024-09-17T18:31:18.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a4e0bf640 10 monclient(hunting): _renew_subs 2024-09-17T18:31:18.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a4e0bf640 10 monclient(hunting): authenticate will time out at 1482.548542s 2024-09-17T18:31:18.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a477fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a4809da70 0x7f9a481cda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:18.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a47fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a48136d00 0x7f9a4813ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:18.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a477fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a4809da70 0x7f9a481cda10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:54824/0 (socket says 172.21.15.121:54824) 2024-09-17T18:31:18.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a47fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a48136d00 0x7f9a4813ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42842/0 (socket says 172.21.15.121:42842) 2024-09-17T18:31:18.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a477fe640 1 -- 172.21.15.121:0/4229295814 learned_addr learned my addr 172.21.15.121:0/4229295814 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:31:18.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a477fe640 10 monclient(hunting): get_auth_request con 0x7f9a4809da70 auth_method 0 2024-09-17T18:31:18.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:18.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a477fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:18.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:18.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a47fff640 10 monclient(hunting): get_auth_request con 0x7f9a48136d00 auth_method 0 2024-09-17T18:31:18.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:18.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a47fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:18.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:18.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:18.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:18.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:18.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:18.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:18.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a47fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:18.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a3ffff640 1 --2- 172.21.15.121:0/4229295814 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a481368d0 0x7f9a4813e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:18.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a3ffff640 10 monclient(hunting): get_auth_request con 0x7f9a481368d0 auth_method 0 2024-09-17T18:31:18.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:18.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:18.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:18.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a3ffff640 1 -- 172.21.15.121:0/4229295814 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a481368d0 msgr2=0x7f9a4813e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T18:31:18.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a477fe640 10 monclient(hunting): handle_auth_done global_id 64292 payload 824 2024-09-17T18:31:18.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a3ffff640 1 -- 172.21.15.121:0/4229295814 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a481368d0 msgr2=0x7f9a4813e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:31:18.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a3ffff640 1 --2- 172.21.15.121:0/4229295814 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a481368d0 0x7f9a4813e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:31:18.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a3ffff640 1 --2- 172.21.15.121:0/4229295814 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a481368d0 0x7f9a4813e510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:31:18.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a477fe640 1 -- 172.21.15.121:0/4229295814 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a481368d0 msgr2=0x7f9a4813e510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:31:18.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a477fe640 1 --2- 172.21.15.121:0/4229295814 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a481368d0 0x7f9a4813e510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:18.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a477fe640 1 -- 172.21.15.121:0/4229295814 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a48136d00 msgr2=0x7f9a4813ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:18.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a477fe640 1 --2- 172.21.15.121:0/4229295814 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a48136d00 0x7f9a4813ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:18.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a477fe640 10 monclient: _finish_hunting 0 2024-09-17T18:31:18.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a477fe640 1 monclient: found mon.noname-c 2024-09-17T18:31:18.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:31:18.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a477fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:31:18.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a477fe640 1 -- 172.21.15.121:0/4229295814 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a4813cfe0 con 0x7f9a4809da70 2024-09-17T18:31:18.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.832+0000 7f9a47fff640 1 --2- 172.21.15.121:0/4229295814 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a48136d00 0x7f9a4813ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:31:18.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.835+0000 7f9a477fe640 1 -- 172.21.15.121:0/4229295814 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a4809da70 msgr2=0x7f9a481cda10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:18.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.835+0000 7f9a477fe640 1 --2- 172.21.15.121:0/4229295814 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a4809da70 0x7f9a481cda10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9a3800c460 tx=0x7f9a3800c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:18.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.835+0000 7f9a44ff9640 10 client.?.objecter ms_handle_connect 0x7f9a4809da70 2024-09-17T18:31:18.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.835+0000 7f9a44ff9640 10 client.?.objecter resend_mon_ops 2024-09-17T18:31:18.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.835+0000 7f9a44ff9640 1 -- 172.21.15.121:0/4229295814 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a38007070 con 0x7f9a4809da70 2024-09-17T18:31:18.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.835+0000 7f9a44ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:18.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.835+0000 7f9a44ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:31:18.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.835+0000 7f9a44ff9640 10 monclient: dump: 2024-09-17T18:31:18.848 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:18.848 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:18.848 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:18.848 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:18.848 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:18.848 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:18.848 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:18.848 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:18.848 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:18.849 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:18.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.835+0000 7f9a44ff9640 10 monclient: _finish_auth 0 2024-09-17T18:31:18.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.835+0000 7f9a44ff9640 10 monclient: _check_auth_tickets 2024-09-17T18:31:18.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.835+0000 7f9a44ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:31:18.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.835+0000 7f9a44ff9640 1 -- 172.21.15.121:0/4229295814 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a38007230 con 0x7f9a4809da70 2024-09-17T18:31:18.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.835+0000 7f9a4e0bf640 5 monclient: authenticate success, global_id 64292 2024-09-17T18:31:18.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.835+0000 7f9a4e0bf640 10 monclient: _renew_subs 2024-09-17T18:31:18.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.835+0000 7f9a4e0bf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:31:18.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.835+0000 7f9a4e0bf640 1 -- 172.21.15.121:0/4229295814 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a4809fe40 con 0x7f9a4809da70 2024-09-17T18:31:18.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.835+0000 7f9a44ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T18:31:18.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.835+0000 7f9a44ff9640 1 -- 172.21.15.121:0/4229295814 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a38007620 con 0x7f9a4809da70 2024-09-17T18:31:18.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.835+0000 7f9a44ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:18.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.835+0000 7f9a44ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:31:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.835+0000 7f9a44ff9640 10 monclient: dump: 2024-09-17T18:31:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:18.850 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.835+0000 7f9a4e0bf640 10 client.64292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:31:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.835+0000 7f9a467fc640 4 set_mon_vals no callback set 2024-09-17T18:31:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.835+0000 7f9a4e0bf640 10 monclient: _renew_subs 2024-09-17T18:31:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.835+0000 7f9a4e0bf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:31:18.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.835+0000 7f9a4e0bf640 1 -- 172.21.15.121:0/4229295814 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a4813ca20 con 0x7f9a4809da70 2024-09-17T18:31:18.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.835+0000 7f9a4e0bf640 1 librados: init done 2024-09-17T18:31:18.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.836+0000 7f9a4e0bf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:31:18.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.836+0000 7f9a4e0bf640 1 --2- 172.21.15.121:0/4229295814 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a4809e550 0x7f9a4809e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:18.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.836+0000 7f9a4e0bf640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:18.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.836+0000 7f9a4e0bf640 1 -- 172.21.15.121:0/4229295814 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9a481361c0 con 0x7f9a4809e550 2024-09-17T18:31:18.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.836+0000 7f9a3ffff640 1 --2- 172.21.15.121:0/4229295814 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a4809e550 0x7f9a4809e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:18.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.836+0000 7f9a3ffff640 10 monclient: get_auth_request con 0x7f9a4809e550 auth_method 0 2024-09-17T18:31:18.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.836+0000 7f9a3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:18.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.836+0000 7f9a3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:18.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.836+0000 7f9a3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:18.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.836+0000 7f9a3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:18.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.836+0000 7f9a3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:18.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.836+0000 7f9a3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:18.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.836+0000 7f9a44ff9640 1 -- 172.21.15.121:0/4229295814 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f9a3801b400 con 0x7f9a4809da70 2024-09-17T18:31:18.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.836+0000 7f9a3ffff640 10 monclient(hunting): handle_auth_done global_id 64295 payload 824 2024-09-17T18:31:18.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.836+0000 7f9a3ffff640 1 -- 172.21.15.121:0/4229295814 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9a481cdf50 con 0x7f9a4809e550 2024-09-17T18:31:18.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.836+0000 7f9a44ff9640 1 --2- 172.21.15.121:0/4229295814 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9a28040170 0x7f9a28042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:18.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.836+0000 7f9a44ff9640 1 -- 172.21.15.121:0/4229295814 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(482..482 src has 247..482) ==== 6686+0+0 (secure 0 0 0) 0x7f9a380538a0 con 0x7f9a4809da70 2024-09-17T18:31:18.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.836+0000 7f9a44ff9640 10 client.64292.objecter ms_dispatch 0x7f9a48000b90 osd_map(482..482 src has 247..482) 2024-09-17T18:31:18.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.836+0000 7f9a44ff9640 3 client.64292.objecter handle_osd_map got epochs [482,482] > 0 2024-09-17T18:31:18.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.836+0000 7f9a44ff9640 3 client.64292.objecter handle_osd_map decoding full epoch 482 2024-09-17T18:31:18.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.836+0000 7f9a44ff9640 20 client.64292.objecter dump_active .. 0 homeless 2024-09-17T18:31:18.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.836+0000 7f9a47fff640 1 --2- 172.21.15.121:0/4229295814 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9a28040170 0x7f9a28042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:18.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.836+0000 7f9a47fff640 10 monclient: get_auth_request con 0x7f9a28040170 auth_method 0 2024-09-17T18:31:18.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.838+0000 7f9a44ff9640 10 client.64292.objecter ms_handle_connect 0x7f9a4809e550 2024-09-17T18:31:18.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.838+0000 7f9a3ffff640 1 -- 172.21.15.121:0/4229295814 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a4809e550 msgr2=0x7f9a4809e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:18.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.838+0000 7f9a44ff9640 10 client.64292.objecter resend_mon_ops 2024-09-17T18:31:18.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.838+0000 7f9a3ffff640 1 --2- 172.21.15.121:0/4229295814 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a4809e550 0x7f9a4809e930 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9a3000e420 tx=0x7f9a3000e8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:18.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.838+0000 7f9a44ff9640 1 -- 172.21.15.121:0/4229295814 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a30007030 con 0x7f9a4809e550 2024-09-17T18:31:18.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.838+0000 7f9a44ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:18.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.838+0000 7f9a44ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:31:18.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.838+0000 7f9a44ff9640 10 monclient: dump: 2024-09-17T18:31:18.853 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:18.853 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:18.853 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:18.853 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:18.853 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:18.853 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:18.853 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:18.853 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:18.853 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:18.853 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:18.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.839+0000 7f9a44ff9640 1 -- 172.21.15.121:0/4229295814 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9a30007030 con 0x7f9a4809e550 2024-09-17T18:31:18.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.839+0000 7f9a44ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:31:18.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.839+0000 7f9a44ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:31:18.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.839+0000 7f9a44ff9640 1 -- 172.21.15.121:0/4229295814 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a4809e550 msgr2=0x7f9a4809e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:18.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.839+0000 7f9a44ff9640 1 --2- 172.21.15.121:0/4229295814 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a4809e550 0x7f9a4809e930 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9a3000e420 tx=0x7f9a3000e8e0 comp rx=0 tx=0).stop 2024-09-17T18:31:18.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.839+0000 7f9a44ff9640 1 -- 172.21.15.121:0/4229295814 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a4809e550 msgr2=0x7f9a4809e930 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:31:18.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.839+0000 7f9a44ff9640 1 --2- 172.21.15.121:0/4229295814 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a4809e550 0x7f9a4809e930 secure :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9a3000e420 tx=0x7f9a3000e8e0 comp rx=0 tx=0).stop 2024-09-17T18:31:18.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.839+0000 7f9a44ff9640 10 client.64292.objecter ms_handle_connect 0x7f9a28040170 2024-09-17T18:31:18.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.839+0000 7f9a47fff640 1 --2- 172.21.15.121:0/4229295814 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9a28040170 0x7f9a28042630 secure :-1 s=READY pgs=4032 cs=0 l=1 rev1=1 crypto rx=0x7f9a40034000 tx=0x7f9a4002f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:18.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.845+0000 7f9a4e0bf640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:31:18.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.845+0000 7f9a4e0bf640 1 --2- 172.21.15.121:0/4229295814 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a4809ee70 0x7f9a480702d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:18.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.845+0000 7f9a4e0bf640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:18.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.845+0000 7f9a4e0bf640 1 -- 172.21.15.121:0/4229295814 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9a4813b7e0 con 0x7f9a4809ee70 2024-09-17T18:31:18.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.845+0000 7f9a3ffff640 1 --2- 172.21.15.121:0/4229295814 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a4809ee70 0x7f9a480702d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:18.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.845+0000 7f9a3ffff640 10 monclient: get_auth_request con 0x7f9a4809ee70 auth_method 0 2024-09-17T18:31:18.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.845+0000 7f9a3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:18.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.845+0000 7f9a3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:18.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.845+0000 7f9a3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:18.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.845+0000 7f9a3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:18.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.845+0000 7f9a3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:18.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.845+0000 7f9a3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:18.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.845+0000 7f9a3ffff640 10 monclient(hunting): handle_auth_done global_id 64298 payload 824 2024-09-17T18:31:18.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.845+0000 7f9a3ffff640 1 -- 172.21.15.121:0/4229295814 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9a4806cd40 con 0x7f9a4809ee70 2024-09-17T18:31:18.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.847+0000 7f9a3ffff640 1 -- 172.21.15.121:0/4229295814 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a4809ee70 msgr2=0x7f9a480702d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:18.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.847+0000 7f9a3ffff640 1 --2- 172.21.15.121:0/4229295814 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a4809ee70 0x7f9a480702d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9a3000e420 tx=0x7f9a30001020 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:18.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.847+0000 7f9a44ff9640 10 client.64292.objecter ms_handle_connect 0x7f9a4809ee70 2024-09-17T18:31:18.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.847+0000 7f9a44ff9640 10 client.64292.objecter resend_mon_ops 2024-09-17T18:31:18.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.848+0000 7f9a44ff9640 1 -- 172.21.15.121:0/4229295814 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a30006bb0 con 0x7f9a4809ee70 2024-09-17T18:31:18.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.848+0000 7f9a44ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:18.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.848+0000 7f9a44ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:31:18.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.848+0000 7f9a44ff9640 10 monclient: dump: 2024-09-17T18:31:18.856 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:18.856 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:18.856 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:18.856 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:18.857 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:18.857 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:18.857 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:18.857 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:18.857 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:18.857 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:18.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.848+0000 7f9a44ff9640 1 -- 172.21.15.121:0/4229295814 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f9a30006bb0 con 0x7f9a4809ee70 2024-09-17T18:31:18.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.848+0000 7f9a44ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:31:18.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.848+0000 7f9a44ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:31:18.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.848+0000 7f9a44ff9640 1 -- 172.21.15.121:0/4229295814 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a4809ee70 msgr2=0x7f9a480702d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:18.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.848+0000 7f9a44ff9640 1 --2- 172.21.15.121:0/4229295814 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a4809ee70 0x7f9a480702d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9a3000e420 tx=0x7f9a30001020 comp rx=0 tx=0).stop 2024-09-17T18:31:18.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.848+0000 7f9a44ff9640 1 -- 172.21.15.121:0/4229295814 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a4809ee70 msgr2=0x7f9a480702d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:31:18.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.848+0000 7f9a44ff9640 1 --2- 172.21.15.121:0/4229295814 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a4809ee70 0x7f9a480702d0 secure :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9a3000e420 tx=0x7f9a30001020 comp rx=0 tx=0).stop 2024-09-17T18:31:18.858 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:31:18.858 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:31:18.858 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:31:18.858 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:31:18.858 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 44, 2024-09-17T18:31:18.858 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 257580, 2024-09-17T18:31:18.858 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:31:18.858 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:31:18.858 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:31:18.858 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:31:18.858 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:31:18.858 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 2, 2024-09-17T18:31:18.858 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:31:18.859 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:31:18.859 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:31:18.859 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:31:18.859 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:31:18.859 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:31:18.859 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:31:18.859 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:31:18.859 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:31:18.859 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:31:18.859 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:31:18.859 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:31:18.859 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:31:18.859 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:31:18.859 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:31:18.859 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:31:18.859 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:31:18.860 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:31:18.860 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:31:18.860 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:31:18.860 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:31:18.860 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:31:18.860 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:31:18.860 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:31:18.860 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:31:18.860 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:31:18.860 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:31:18.860 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:31:18.860 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:18.860 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:18.860 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:31:18.860 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:31:18.861 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:31:18.861 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:31:18.861 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:31:18.861 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:31:18.861 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:31:18.861 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:31:18.861 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:31:18.861 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:31:18.861 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:31:18.861 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:31:18.861 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:31:18.861 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:31:18.861 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:31:18.861 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:31:18.861 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:31:18.862 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:31:18.862 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:31:18.862 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:31:18.862 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:31:18.862 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:31:18.862 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:31:18.862 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:31:18.862 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:31:18.862 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:31:18.862 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:31:18.862 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:31:18.862 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:31:18.862 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:31:18.862 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:18.862 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:31:18.862 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:18.863 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:31:18.863 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:31:18.863 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:31:18.863 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:31:18.863 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:18.863 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:31:18.863 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:31:18.863 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:18.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:18.863 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:18.863 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:31:18.863 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:31:18.863 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:18.863 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:18.863 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:18.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:18.864 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:31:18.864 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:31:18.864 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:31:18.864 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:31:18.864 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:31:18.864 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:18.864 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:18.864 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:31:18.864 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:31:18.864 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:31:18.864 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:31:18.864 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:18.864 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:31:18.864 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:31:18.864 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:18.865 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:18.865 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:18.865 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:31:18.865 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:31:18.865 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:18.865 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:18.865 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:18.865 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:18.865 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:31:18.865 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:31:18.865 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:31:18.865 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:31:18.866 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:31:18.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:18.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:18.866 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:31:18.866 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:31:18.866 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:31:18.866 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:31:18.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:18.866 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:31:18.866 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:31:18.866 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:18.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:18.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:18.867 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:31:18.867 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:31:18.867 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:18.867 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:18.867 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:18.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:18.867 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:31:18.867 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:31:18.867 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:31:18.867 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:31:18.867 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:31:18.867 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:18.867 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:18.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:18.867 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:31:18.867 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:31:18.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:18.868 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:31:18.868 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:31:18.868 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:31:18.868 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:18.868 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:31:18.868 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:31:18.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:18.868 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:31:18.868 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:31:18.868 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 3 2024-09-17T18:31:18.868 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:18.868 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:31:18.868 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:31:18.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:18.869 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:31:18.869 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:31:18.869 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 5 2024-09-17T18:31:18.869 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:18.869 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:31:18.869 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": [ 2024-09-17T18:31:18.869 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:18.869 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:31:18.869 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:31:18.869 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:31:18.869 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:18.869 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:18.869 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:18.869 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:31:18.869 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:31:18.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.848+0000 7f9a4e0bf640 10 librados: watch_flush enter 2024-09-17T18:31:18.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.848+0000 7f9a4e0bf640 10 librados: watch_flush exit 2024-09-17T18:31:18.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.848+0000 7f9a4e0bf640 20 client.64292.objecter shutdown clearing up homeless session... 2024-09-17T18:31:18.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.848+0000 7f9a4e0bf640 10 client.64292.objecter successfully canceled tick 2024-09-17T18:31:18.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.848+0000 7f9a4e0bf640 1 -- 172.21.15.121:0/4229295814 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9a28040170 msgr2=0x7f9a28042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:18.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.848+0000 7f9a4e0bf640 1 --2- 172.21.15.121:0/4229295814 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9a28040170 0x7f9a28042630 secure :-1 s=READY pgs=4032 cs=0 l=1 rev1=1 crypto rx=0x7f9a40034000 tx=0x7f9a4002f000 comp rx=0 tx=0).stop 2024-09-17T18:31:18.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.848+0000 7f9a4e0bf640 10 monclient: shutdown 2024-09-17T18:31:18.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.848+0000 7f9a4e0bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:31:18.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.848+0000 7f9a4e0bf640 1 -- 172.21.15.121:0/4229295814 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a4809da70 msgr2=0x7f9a481cda10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:18.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.848+0000 7f9a4e0bf640 1 --2- 172.21.15.121:0/4229295814 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a4809da70 0x7f9a481cda10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9a3800c460 tx=0x7f9a3800c920 comp rx=0 tx=0).stop 2024-09-17T18:31:18.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.848+0000 7f9a47fff640 1 -- 172.21.15.121:0/4229295814 reap_dead start 2024-09-17T18:31:18.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.848+0000 7f9a4e0bf640 1 -- 172.21.15.121:0/4229295814 shutdown_connections 2024-09-17T18:31:18.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.848+0000 7f9a4e0bf640 1 --2- 172.21.15.121:0/4229295814 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a4809da70 0x7f9a481cda10 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:18.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.848+0000 7f9a4e0bf640 1 -- 172.21.15.121:0/4229295814 >> 172.21.15.121:0/4229295814 conn(0x7f9a480a7830 msgr2=0x7f9a480a4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:31:18.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.848+0000 7f9a4e0bf640 1 -- 172.21.15.121:0/4229295814 shutdown_connections 2024-09-17T18:31:18.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.848+0000 7f9a4e0bf640 1 -- 172.21.15.121:0/4229295814 wait complete. 2024-09-17T18:31:18.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.848+0000 7f9a4e0bf640 1 librados: shutdown 2024-09-17T18:31:18.872 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:31:18.873 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T18:31:18.873 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T18:31:18.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.968+0000 7f88d4b90640 10 monclient: get_monmap_and_config 2024-09-17T18:31:18.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.968+0000 7f88d4b90640 10 monclient: build_initial_monmap 2024-09-17T18:31:18.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.968+0000 7f88d4b90640 10 monclient: monmap: 2024-09-17T18:31:18.970 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:31:18.970 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:18.970 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:31:18.969227+0000 2024-09-17T18:31:18.970 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:31:18.969227+0000 2024-09-17T18:31:18.971 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:31:18.971 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:18.971 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:31:18.971 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:31:18.971 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:31:18.971 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:31:18.971 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:31:18.971 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:18.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.969+0000 7f88d4b90640 1 Processor -- start 2024-09-17T18:31:18.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.969+0000 7f88d4b90640 1 -- start start 2024-09-17T18:31:18.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.969+0000 7f88d4b90640 10 monclient: init 2024-09-17T18:31:18.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.969+0000 7f88d4b90640 10 monclient: _reopen_session rank -1 2024-09-17T18:31:18.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.969+0000 7f88d4b90640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-17T18:31:18.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.969+0000 7f88d4b90640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f88d009e550 addr v1:172.21.15.173:6789/0 2024-09-17T18:31:18.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.969+0000 7f88d4b90640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f88d009f040 addr v1:172.21.15.121:6789/0 2024-09-17T18:31:18.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.969+0000 7f88d4b90640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88d013bf70 0x7f88d013e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:18.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.969+0000 7f88d4b90640 10 monclient(hunting): picked mon.noname-b con 0x7f88d013bf70 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:31:18.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.969+0000 7f88d4b90640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88d00a4460 con 0x7f88d009f040 2024-09-17T18:31:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.969+0000 7f88d4b90640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88d00a2b60 con 0x7f88d009e550 2024-09-17T18:31:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.969+0000 7f88d4b90640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.969+0000 7f88d4b90640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f88d00a4950 con 0x7f88d013bf70 2024-09-17T18:31:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.969+0000 7f88d4b90640 10 monclient(hunting): _renew_subs 2024-09-17T18:31:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.969+0000 7f88d4b90640 10 monclient(hunting): authenticate will time out at 1482.685739s 2024-09-17T18:31:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.969+0000 7f88c5d9b640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f88d009f040 0x7f88d013a850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:53066/0 (socket says 172.21.15.121:53066) 2024-09-17T18:31:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.969+0000 7f88ced9d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88d013bf70 0x7f88d013e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.969+0000 7f88c5d9b640 1 -- 172.21.15.121:0/2788976817 learned_addr learned my addr 172.21.15.121:0/2788976817 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:31:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.969+0000 7f88ced9d640 10 monclient(hunting): get_auth_request con 0x7f88d013bf70 auth_method 0 2024-09-17T18:31:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.969+0000 7f88ced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.969+0000 7f88ced9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.969+0000 7f88ced9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.969+0000 7f88ced9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:18.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.969+0000 7f88ced9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:18.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.969+0000 7f88ced9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:18.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.970+0000 7f88ced9d640 10 monclient(hunting): handle_auth_done global_id 64301 payload 274 2024-09-17T18:31:18.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.970+0000 7f88ced9d640 1 -- 172.21.15.121:0/2788976817 >> v1:172.21.15.173:6789/0 conn(0x7f88d009e550 legacy=0x7f88d009e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:31:18.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.970+0000 7f88ced9d640 1 -- 172.21.15.121:0/2788976817 >> v1:172.21.15.121:6789/0 conn(0x7f88d009f040 legacy=0x7f88d013a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:31:18.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.970+0000 7f88ced9d640 10 monclient: _finish_hunting 0 2024-09-17T18:31:18.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.970+0000 7f88ced9d640 1 monclient: found mon.noname-b 2024-09-17T18:31:18.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.970+0000 7f88ced9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:31:18.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.970+0000 7f88ced9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:31:18.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.970+0000 7f88ced9d640 1 -- 172.21.15.121:0/2788976817 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88d009f840 con 0x7f88d013bf70 2024-09-17T18:31:18.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.972+0000 7f88ced9d640 1 -- 172.21.15.121:0/2788976817 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88d013bf70 msgr2=0x7f88d013e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:18.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.972+0000 7f88ced9d640 1 --2- 172.21.15.121:0/2788976817 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88d013bf70 0x7f88d013e360 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f88c8009dc0 tx=0x7f88c802a480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2dbfba1055d8851 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:18.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.973+0000 7f88cdd9b640 1 -- 172.21.15.121:0/2788976817 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c8007a30 con 0x7f88d013bf70 2024-09-17T18:31:18.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.973+0000 7f88cdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:18.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.973+0000 7f88cdd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:31:18.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.973+0000 7f88cdd9b640 10 monclient: dump: 2024-09-17T18:31:18.982 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:18.982 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:18.982 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:18.982 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:18.982 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:18.982 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:18.982 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:18.982 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:18.982 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:18.983 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:18.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.973+0000 7f88cdd9b640 10 monclient: _finish_auth 0 2024-09-17T18:31:18.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.973+0000 7f88cdd9b640 10 monclient: _check_auth_tickets 2024-09-17T18:31:18.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.973+0000 7f88cdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:31:18.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.973+0000 7f88cdd9b640 1 -- 172.21.15.121:0/2788976817 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88c8007bd0 con 0x7f88d013bf70 2024-09-17T18:31:18.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.973+0000 7f88d4b90640 5 monclient: authenticate success, global_id 64301 2024-09-17T18:31:18.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.973+0000 7f88d4b90640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:31:18.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.973+0000 7f88cdd9b640 10 monclient: handle_config config(0 keys) 2024-09-17T18:31:18.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.973+0000 7f88cdd9b640 1 -- 172.21.15.121:0/2788976817 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c8031400 con 0x7f88d013bf70 2024-09-17T18:31:18.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.973+0000 7f88d4b90640 10 monclient: get_monmap_and_config success 2024-09-17T18:31:18.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.973+0000 7f88d4b90640 4 set_mon_vals no callback set 2024-09-17T18:31:18.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.973+0000 7f88cdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:18.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.973+0000 7f88cdd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:31:18.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.973+0000 7f88cdd9b640 10 monclient: dump: 2024-09-17T18:31:18.984 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:18.984 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:18.984 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:18.984 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:18.984 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:18.984 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:18.984 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:18.984 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:18.984 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:18.984 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:18.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.973+0000 7f88d4b90640 10 monclient: shutdown 2024-09-17T18:31:18.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.973+0000 7f88d4b90640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:31:18.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.973+0000 7f88d4b90640 1 -- 172.21.15.121:0/2788976817 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88d013bf70 msgr2=0x7f88d013e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:18.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.973+0000 7f88d4b90640 1 --2- 172.21.15.121:0/2788976817 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88d013bf70 0x7f88d013e360 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f88c8009dc0 tx=0x7f88c802a480 comp rx=0 tx=0).stop 2024-09-17T18:31:18.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.973+0000 7f88d4b90640 1 -- 172.21.15.121:0/2788976817 shutdown_connections 2024-09-17T18:31:18.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.973+0000 7f88d4b90640 1 --2- 172.21.15.121:0/2788976817 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88d013bf70 0x7f88d013e360 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:18.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.973+0000 7f88d4b90640 1 -- 172.21.15.121:0/2788976817 >> 172.21.15.121:0/2788976817 conn(0x7f88d00a7830 msgr2=0x7f88d00a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:31:18.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.973+0000 7f88d4b90640 1 -- 172.21.15.121:0/2788976817 shutdown_connections 2024-09-17T18:31:18.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.973+0000 7f88d4b90640 1 -- 172.21.15.121:0/2788976817 wait complete. 2024-09-17T18:31:18.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.974+0000 7f88d4b90640 10 monclient: build_initial_monmap 2024-09-17T18:31:18.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.974+0000 7f88d4b90640 10 monclient: monmap: 2024-09-17T18:31:18.985 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:31:18.985 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:31:18.985 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:31:18.985 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:31:18.985 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:31:18.985 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:18.986 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:31:18.986 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:31:18.986 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:31:18.986 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:18.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.974+0000 7f88d4b90640 1 librados: starting msgr at 2024-09-17T18:31:18.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.974+0000 7f88d4b90640 1 librados: starting objecter 2024-09-17T18:31:18.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.974+0000 7f88d4b90640 1 Processor -- start 2024-09-17T18:31:18.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.974+0000 7f88d4b90640 1 -- start start 2024-09-17T18:31:18.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.974+0000 7f88d4b90640 1 librados: setting wanted keys 2024-09-17T18:31:18.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.974+0000 7f88d4b90640 1 librados: calling monclient init 2024-09-17T18:31:18.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.974+0000 7f88d4b90640 10 monclient: init 2024-09-17T18:31:18.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88d4b90640 10 monclient: _reopen_session rank -1 2024-09-17T18:31:18.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88d4b90640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T18:31:18.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88d4b90640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88d009e550 0x7f88d011ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:18.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88d4b90640 10 monclient(hunting): picked mon.noname-b con 0x7f88d009e550 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:31:18.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88d4b90640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f88d009f040 0x7f88d011e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:18.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88d4b90640 10 monclient(hunting): picked mon.noname-a con 0x7f88d009f040 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:31:18.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88d4b90640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88d013bf70 0x7f88d0136de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:18.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88d4b90640 10 monclient(hunting): picked mon.noname-c con 0x7f88d013bf70 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:31:18.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88d4b90640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:18.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88d4b90640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f88d009f490 con 0x7f88d009f040 2024-09-17T18:31:18.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88d4b90640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:18.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88d4b90640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f88d009dce0 con 0x7f88d009e550 2024-09-17T18:31:18.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88d4b90640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:18.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88d4b90640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f88d011f020 con 0x7f88d013bf70 2024-09-17T18:31:18.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88d4b90640 10 monclient(hunting): _renew_subs 2024-09-17T18:31:18.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88d4b90640 10 monclient(hunting): authenticate will time out at 1482.691393s 2024-09-17T18:31:18.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88c5d9b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f88d009f040 0x7f88d011e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:18.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88ced9d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88d013bf70 0x7f88d0136de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:18.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88c5d9b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f88d009f040 0x7f88d011e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42846/0 (socket says 172.21.15.121:42846) 2024-09-17T18:31:18.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88c5d9b640 1 -- 172.21.15.121:0/2165021007 learned_addr learned my addr 172.21.15.121:0/2165021007 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:31:18.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88c5d9b640 10 monclient(hunting): get_auth_request con 0x7f88d009f040 auth_method 0 2024-09-17T18:31:18.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88ced9d640 1 --2- 172.21.15.121:0/2165021007 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88d013bf70 0x7f88d0136de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:54844/0 (socket says 172.21.15.121:54844) 2024-09-17T18:31:18.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88c5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:18.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88c5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:18.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88c5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:18.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88ced9d640 10 monclient(hunting): get_auth_request con 0x7f88d013bf70 auth_method 0 2024-09-17T18:31:18.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88ced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:18.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88ced9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:18.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88ced9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:18.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88ce59c640 1 --2- 172.21.15.121:0/2165021007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88d009e550 0x7f88d011ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:18.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88ce59c640 10 monclient(hunting): get_auth_request con 0x7f88d009e550 auth_method 0 2024-09-17T18:31:18.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88ce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:18.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88ce59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:18.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88ce59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:18.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88c5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:18.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88c5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:18.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88c5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:18.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88ced9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:18.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88ced9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:18.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88ced9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:18.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88ce59c640 1 -- 172.21.15.121:0/2165021007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88d009e550 msgr2=0x7f88d011ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T18:31:18.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88ce59c640 1 -- 172.21.15.121:0/2165021007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88d009e550 msgr2=0x7f88d011ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:31:18.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88ce59c640 1 --2- 172.21.15.121:0/2165021007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88d009e550 0x7f88d011ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:31:18.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88ce59c640 1 --2- 172.21.15.121:0/2165021007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88d009e550 0x7f88d011ff70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:31:18.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88c5d9b640 10 monclient(hunting): handle_auth_done global_id 44667 payload 824 2024-09-17T18:31:18.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88c5d9b640 1 -- 172.21.15.121:0/2165021007 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88d013bf70 msgr2=0x7f88d0136de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:18.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88c5d9b640 1 --2- 172.21.15.121:0/2165021007 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88d013bf70 0x7f88d0136de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:18.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88c5d9b640 1 -- 172.21.15.121:0/2165021007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88d009e550 msgr2=0x7f88d011ff70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:31:18.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88c5d9b640 1 --2- 172.21.15.121:0/2165021007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88d009e550 0x7f88d011ff70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:18.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88c5d9b640 10 monclient: _finish_hunting 0 2024-09-17T18:31:18.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88c5d9b640 1 monclient: found mon.noname-a 2024-09-17T18:31:18.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88c5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:31:18.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88c5d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:31:18.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.975+0000 7f88c5d9b640 1 -- 172.21.15.121:0/2165021007 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88d0120ed0 con 0x7f88d009f040 2024-09-17T18:31:18.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.978+0000 7f88c5d9b640 1 -- 172.21.15.121:0/2165021007 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f88d009f040 msgr2=0x7f88d011e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:18.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.978+0000 7f88c5d9b640 1 --2- 172.21.15.121:0/2165021007 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f88d009f040 0x7f88d011e000 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f88b800c460 tx=0x7f88b800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:18.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.978+0000 7f88c77fe640 10 client.?.objecter ms_handle_connect 0x7f88d009f040 2024-09-17T18:31:18.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.978+0000 7f88c77fe640 10 client.?.objecter resend_mon_ops 2024-09-17T18:31:18.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.978+0000 7f88c77fe640 1 -- 172.21.15.121:0/2165021007 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88b8017070 con 0x7f88d009f040 2024-09-17T18:31:18.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.978+0000 7f88c77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:18.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.978+0000 7f88c77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:31:18.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.978+0000 7f88c77fe640 10 monclient: dump: 2024-09-17T18:31:18.991 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:18.991 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:18.991 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:18.991 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:18.991 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:18.991 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:18.991 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:18.991 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:18.991 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:18.991 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:18.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.978+0000 7f88c77fe640 10 monclient: _finish_auth 0 2024-09-17T18:31:18.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.978+0000 7f88c77fe640 10 monclient: _check_auth_tickets 2024-09-17T18:31:18.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.978+0000 7f88c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:31:18.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.978+0000 7f88c77fe640 1 -- 172.21.15.121:0/2165021007 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88b80070d0 con 0x7f88d009f040 2024-09-17T18:31:18.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.978+0000 7f88d4b90640 5 monclient: authenticate success, global_id 44667 2024-09-17T18:31:18.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.978+0000 7f88c77fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:31:18.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.978+0000 7f88c77fe640 1 -- 172.21.15.121:0/2165021007 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88b80074c0 con 0x7f88d009f040 2024-09-17T18:31:18.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.978+0000 7f88d4b90640 10 monclient: _renew_subs 2024-09-17T18:31:18.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.978+0000 7f88d4b90640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:31:18.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.978+0000 7f88d4b90640 1 -- 172.21.15.121:0/2165021007 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88d009e240 con 0x7f88d009f040 2024-09-17T18:31:18.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.978+0000 7f88c77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:18.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.978+0000 7f88c77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:31:18.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.978+0000 7f88c77fe640 10 monclient: dump: 2024-09-17T18:31:18.992 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:18.993 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:18.993 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:18.993 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:18.993 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:18.993 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:18.993 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:18.993 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:18.993 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:18.993 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:18.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.978+0000 7f88cd59a640 4 set_mon_vals no callback set 2024-09-17T18:31:18.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.978+0000 7f88d4b90640 10 client.44667.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:31:18.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.978+0000 7f88d4b90640 10 monclient: _renew_subs 2024-09-17T18:31:18.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.978+0000 7f88d4b90640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:31:18.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.978+0000 7f88d4b90640 1 -- 172.21.15.121:0/2165021007 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88d011f460 con 0x7f88d009f040 2024-09-17T18:31:18.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.978+0000 7f88d4b90640 1 librados: init done 2024-09-17T18:31:18.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.979+0000 7f88c77fe640 1 -- 172.21.15.121:0/2165021007 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f88b801a070 con 0x7f88d009f040 2024-09-17T18:31:18.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.979+0000 7f88d4b90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:31:18.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.979+0000 7f88d4b90640 1 --2- 172.21.15.121:0/2165021007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88d01c9580 0x7f88d01c9960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:18.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.979+0000 7f88d4b90640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:18.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.979+0000 7f88d4b90640 1 -- 172.21.15.121:0/2165021007 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f88d01213f0 con 0x7f88d01c9580 2024-09-17T18:31:18.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.979+0000 7f88c77fe640 1 --2- 172.21.15.121:0/2165021007 >> v2:172.21.15.173:6812/2423786037 conn(0x7f88b00401c0 0x7f88b0042680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:18.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.979+0000 7f88c77fe640 1 -- 172.21.15.121:0/2165021007 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(482..482 src has 247..482) ==== 6686+0+0 (secure 0 0 0) 0x7f88b8053180 con 0x7f88d009f040 2024-09-17T18:31:18.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.979+0000 7f88c77fe640 10 client.44667.objecter ms_dispatch 0x7f88d0000b90 osd_map(482..482 src has 247..482) 2024-09-17T18:31:18.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.979+0000 7f88c77fe640 3 client.44667.objecter handle_osd_map got epochs [482,482] > 0 2024-09-17T18:31:18.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.979+0000 7f88c77fe640 3 client.44667.objecter handle_osd_map decoding full epoch 482 2024-09-17T18:31:18.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.979+0000 7f88c77fe640 20 client.44667.objecter dump_active .. 0 homeless 2024-09-17T18:31:18.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.979+0000 7f88ce59c640 1 --2- 172.21.15.121:0/2165021007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88d01c9580 0x7f88d01c9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:18.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.979+0000 7f88ce59c640 10 monclient: get_auth_request con 0x7f88d01c9580 auth_method 0 2024-09-17T18:31:18.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.979+0000 7f88ce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:18.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.979+0000 7f88ce59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:18.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.979+0000 7f88ce59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:18.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.979+0000 7f88ced9d640 1 --2- 172.21.15.121:0/2165021007 >> v2:172.21.15.173:6812/2423786037 conn(0x7f88b00401c0 0x7f88b0042680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:18.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.979+0000 7f88ced9d640 10 monclient: get_auth_request con 0x7f88b00401c0 auth_method 0 2024-09-17T18:31:18.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.979+0000 7f88ce59c640 1 -- 172.21.15.121:0/2165021007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88d01c9580 msgr2=0x7f88d01c9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T18:31:18.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.979+0000 7f88ce59c640 1 -- 172.21.15.121:0/2165021007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88d01c9580 msgr2=0x7f88d01c9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:31:18.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.979+0000 7f88ce59c640 1 --2- 172.21.15.121:0/2165021007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88d01c9580 0x7f88d01c9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:31:18.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.979+0000 7f88ce59c640 1 --2- 172.21.15.121:0/2165021007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88d01c9580 0x7f88d01c9960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:31:18.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.982+0000 7f88ced9d640 1 --2- 172.21.15.121:0/2165021007 >> v2:172.21.15.173:6812/2423786037 conn(0x7f88b00401c0 0x7f88b0042680 secure :-1 s=READY pgs=4033 cs=0 l=1 rev1=1 crypto rx=0x7f88c8034000 tx=0x7f88c802f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:18.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:18.982+0000 7f88c77fe640 10 client.44667.objecter ms_handle_connect 0x7f88b00401c0 2024-09-17T18:31:19.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:19.181+0000 7f88ce59c640 1 --2- 172.21.15.121:0/2165021007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88d01c9580 0x7f88d01c9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:19.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:19.181+0000 7f88ce59c640 10 monclient: get_auth_request con 0x7f88d01c9580 auth_method 0 2024-09-17T18:31:19.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:19.181+0000 7f88ce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:19.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:19.181+0000 7f88ce59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:19.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:19.181+0000 7f88ce59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:31:19.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:19.181+0000 7f88ce59c640 1 -- 172.21.15.121:0/2165021007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88d01c9580 msgr2=0x7f88d01c9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T18:31:19.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:19.181+0000 7f88ce59c640 1 -- 172.21.15.121:0/2165021007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88d01c9580 msgr2=0x7f88d01c9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:31:19.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:19.181+0000 7f88ce59c640 1 --2- 172.21.15.121:0/2165021007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88d01c9580 0x7f88d01c9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:31:19.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:19.181+0000 7f88ce59c640 1 --2- 172.21.15.121:0/2165021007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88d01c9580 0x7f88d01c9960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T18:31:19.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:19.582+0000 7f88ce59c640 1 --2- 172.21.15.121:0/2165021007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88d01c9580 0x7f88d01c9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:19.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:19.582+0000 7f88ce59c640 10 monclient: get_auth_request con 0x7f88d01c9580 auth_method 0 2024-09-17T18:31:19.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:19.582+0000 7f88ce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:19.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:19.582+0000 7f88ce59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:19.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:19.582+0000 7f88ce59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:31:19.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:19.582+0000 7f88ce59c640 1 -- 172.21.15.121:0/2165021007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88d01c9580 msgr2=0x7f88d01c9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T18:31:19.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:19.582+0000 7f88ce59c640 1 -- 172.21.15.121:0/2165021007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88d01c9580 msgr2=0x7f88d01c9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:31:19.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:19.582+0000 7f88ce59c640 1 --2- 172.21.15.121:0/2165021007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88d01c9580 0x7f88d01c9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:31:19.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:19.582+0000 7f88ce59c640 1 --2- 172.21.15.121:0/2165021007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88d01c9580 0x7f88d01c9960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T18:31:20.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.383+0000 7f88ce59c640 1 --2- 172.21.15.121:0/2165021007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88d01c9580 0x7f88d01c9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:20.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.384+0000 7f88ce59c640 10 monclient: get_auth_request con 0x7f88d01c9580 auth_method 0 2024-09-17T18:31:20.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.384+0000 7f88ce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:20.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.384+0000 7f88ce59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:20.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.384+0000 7f88ce59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:31:20.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.384+0000 7f88ce59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:20.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.384+0000 7f88ce59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:20.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.384+0000 7f88ce59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:20.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.384+0000 7f88ce59c640 10 monclient(hunting): handle_auth_done global_id 84100 payload 824 2024-09-17T18:31:20.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.384+0000 7f88ce59c640 1 -- 172.21.15.121:0/2165021007 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f88d0074900 con 0x7f88d01c9580 2024-09-17T18:31:20.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.386+0000 7f88ce59c640 1 -- 172.21.15.121:0/2165021007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88d01c9580 msgr2=0x7f88d01c9960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:20.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.386+0000 7f88ce59c640 1 --2- 172.21.15.121:0/2165021007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88d01c9580 0x7f88d01c9960 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f88c0013500 tx=0x7f88c00139c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:20.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.386+0000 7f88c77fe640 10 client.44667.objecter ms_handle_connect 0x7f88d01c9580 2024-09-17T18:31:20.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.386+0000 7f88c77fe640 10 client.44667.objecter resend_mon_ops 2024-09-17T18:31:20.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.387+0000 7f88c77fe640 1 -- 172.21.15.121:0/2165021007 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c001a030 con 0x7f88d01c9580 2024-09-17T18:31:20.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.387+0000 7f88c77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:20.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.387+0000 7f88c77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:31:20.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.387+0000 7f88c77fe640 10 monclient: dump: 2024-09-17T18:31:20.390 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:20.390 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:20.390 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:20.390 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:20.390 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:20.390 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:20.390 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:20.390 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:20.390 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:20.390 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:20.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.387+0000 7f88c77fe640 1 -- 172.21.15.121:0/2165021007 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f88c00158e0 con 0x7f88d01c9580 2024-09-17T18:31:20.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.387+0000 7f88c77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:31:20.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.387+0000 7f88c77fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:31:20.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.387+0000 7f88c77fe640 1 -- 172.21.15.121:0/2165021007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88d01c9580 msgr2=0x7f88d01c9960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:20.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.387+0000 7f88c77fe640 1 --2- 172.21.15.121:0/2165021007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88d01c9580 0x7f88d01c9960 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f88c0013500 tx=0x7f88c00139c0 comp rx=0 tx=0).stop 2024-09-17T18:31:20.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.387+0000 7f88c77fe640 1 -- 172.21.15.121:0/2165021007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88d01c9580 msgr2=0x7f88d01c9960 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:31:20.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.387+0000 7f88c77fe640 1 --2- 172.21.15.121:0/2165021007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88d01c9580 0x7f88d01c9960 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:20.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.393+0000 7f88d4b90640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:31:20.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.393+0000 7f88d4b90640 1 --2- 172.21.15.121:0/2165021007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88d01c9ea0 0x7f88d006f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:20.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.393+0000 7f88d4b90640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:20.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.393+0000 7f88d4b90640 1 -- 172.21.15.121:0/2165021007 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f88d009f610 con 0x7f88d01c9ea0 2024-09-17T18:31:20.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.394+0000 7f88ce59c640 1 --2- 172.21.15.121:0/2165021007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88d01c9ea0 0x7f88d006f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:20.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.394+0000 7f88ce59c640 10 monclient: get_auth_request con 0x7f88d01c9ea0 auth_method 0 2024-09-17T18:31:20.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.394+0000 7f88ce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:20.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.394+0000 7f88ce59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:20.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.394+0000 7f88ce59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:20.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.394+0000 7f88ce59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:20.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.394+0000 7f88ce59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:20.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.394+0000 7f88ce59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:20.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.394+0000 7f88ce59c640 10 monclient(hunting): handle_auth_done global_id 84103 payload 824 2024-09-17T18:31:20.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.394+0000 7f88ce59c640 1 -- 172.21.15.121:0/2165021007 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f88d006cd40 con 0x7f88d01c9ea0 2024-09-17T18:31:20.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.397+0000 7f88ce59c640 1 -- 172.21.15.121:0/2165021007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88d01c9ea0 msgr2=0x7f88d006f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:20.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.397+0000 7f88ce59c640 1 --2- 172.21.15.121:0/2165021007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88d01c9ea0 0x7f88d006f680 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f88c001d000 tx=0x7f88c0000fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:20.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.397+0000 7f88c77fe640 10 client.44667.objecter ms_handle_connect 0x7f88d01c9ea0 2024-09-17T18:31:20.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.397+0000 7f88c77fe640 10 client.44667.objecter resend_mon_ops 2024-09-17T18:31:20.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.397+0000 7f88c77fe640 1 -- 172.21.15.121:0/2165021007 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c0016e40 con 0x7f88d01c9ea0 2024-09-17T18:31:20.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.397+0000 7f88c77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:20.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.397+0000 7f88c77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:31:20.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.397+0000 7f88c77fe640 10 monclient: dump: 2024-09-17T18:31:20.399 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:20.400 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:20.400 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:20.400 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:20.400 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:20.400 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:20.400 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:20.400 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:20.400 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:20.400 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:20.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.397+0000 7f88c77fe640 1 -- 172.21.15.121:0/2165021007 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4551 (secure 0 0 0) 0x7f88c0016e40 con 0x7f88d01c9ea0 2024-09-17T18:31:20.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.397+0000 7f88c77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:31:20.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.397+0000 7f88c77fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:31:20.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.397+0000 7f88c77fe640 1 -- 172.21.15.121:0/2165021007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88d01c9ea0 msgr2=0x7f88d006f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:20.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.397+0000 7f88c77fe640 1 --2- 172.21.15.121:0/2165021007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88d01c9ea0 0x7f88d006f680 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f88c001d000 tx=0x7f88c0000fb0 comp rx=0 tx=0).stop 2024-09-17T18:31:20.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.397+0000 7f88c77fe640 1 -- 172.21.15.121:0/2165021007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88d01c9ea0 msgr2=0x7f88d006f680 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:31:20.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.397+0000 7f88c77fe640 1 --2- 172.21.15.121:0/2165021007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88d01c9ea0 0x7f88d006f680 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:20.401 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:31:20.401 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:31:20.401 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:31:20.401 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:31:20.401 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 44, 2024-09-17T18:31:20.401 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 5495, 2024-09-17T18:31:20.401 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:31:20.402 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:31:20.402 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:31:20.402 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:31:20.402 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:31:20.402 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 3, 2024-09-17T18:31:20.402 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:31:20.402 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:31:20.402 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:31:20.402 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:31:20.402 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:31:20.402 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:31:20.402 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:31:20.402 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:31:20.402 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:31:20.402 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:31:20.403 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:31:20.403 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:31:20.403 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:31:20.403 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:31:20.403 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:31:20.403 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:31:20.403 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:31:20.403 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:31:20.403 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:31:20.403 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:31:20.403 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:31:20.403 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:31:20.403 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:31:20.404 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:31:20.404 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:31:20.404 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:31:20.404 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:31:20.404 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:31:20.404 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:20.404 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:20.404 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:31:20.404 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:31:20.404 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:31:20.404 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:31:20.404 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:31:20.404 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:31:20.404 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:31:20.404 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:31:20.405 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:31:20.405 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:31:20.405 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:31:20.405 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:31:20.405 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:31:20.405 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:31:20.405 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:31:20.405 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:31:20.405 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:31:20.405 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:31:20.405 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:31:20.405 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:31:20.405 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:31:20.405 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:31:20.406 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:31:20.406 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:31:20.406 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:31:20.406 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:31:20.406 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:31:20.406 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:31:20.406 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:31:20.406 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:31:20.406 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:20.406 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:31:20.406 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:20.406 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:31:20.406 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:31:20.406 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:31:20.406 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:31:20.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:20.407 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:31:20.407 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:31:20.407 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:20.407 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:20.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:20.407 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:31:20.407 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:31:20.407 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:20.407 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:20.407 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:20.407 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:20.407 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:31:20.407 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:31:20.408 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:31:20.408 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:31:20.408 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:31:20.408 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:20.408 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:20.408 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:31:20.408 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:31:20.408 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:31:20.408 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:31:20.408 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:20.408 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:31:20.408 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:31:20.408 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:20.408 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:20.408 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:20.409 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:31:20.409 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:31:20.409 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:20.409 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:20.409 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:20.409 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:20.409 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:31:20.409 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:31:20.409 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:31:20.409 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:31:20.410 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:31:20.410 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:20.410 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:20.410 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:31:20.410 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:31:20.410 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:31:20.410 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:31:20.410 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:20.410 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:31:20.410 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:31:20.410 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:20.410 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:20.410 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:20.410 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:31:20.410 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:31:20.411 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:20.411 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:20.411 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:20.411 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:20.411 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:31:20.411 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:31:20.411 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:31:20.411 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:31:20.411 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:31:20.411 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:20.411 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:20.411 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:20.412 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:31:20.412 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:31:20.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:20.412 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:31:20.412 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:31:20.412 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:31:20.412 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:20.412 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:31:20.412 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:31:20.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:20.412 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:31:20.412 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:31:20.412 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:31:20.413 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:20.413 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:20.413 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:20.413 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:31:20.413 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:31:20.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.397+0000 7f88d4b90640 10 librados: watch_flush enter 2024-09-17T18:31:20.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.397+0000 7f88d4b90640 10 librados: watch_flush exit 2024-09-17T18:31:20.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.397+0000 7f88d4b90640 20 client.44667.objecter shutdown clearing up homeless session... 2024-09-17T18:31:20.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.397+0000 7f88d4b90640 10 client.44667.objecter successfully canceled tick 2024-09-17T18:31:20.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.397+0000 7f88d4b90640 1 -- 172.21.15.121:0/2165021007 >> v2:172.21.15.173:6812/2423786037 conn(0x7f88b00401c0 msgr2=0x7f88b0042680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:20.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.397+0000 7f88d4b90640 1 --2- 172.21.15.121:0/2165021007 >> v2:172.21.15.173:6812/2423786037 conn(0x7f88b00401c0 0x7f88b0042680 secure :-1 s=READY pgs=4033 cs=0 l=1 rev1=1 crypto rx=0x7f88c8034000 tx=0x7f88c802f000 comp rx=0 tx=0).stop 2024-09-17T18:31:20.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.397+0000 7f88d4b90640 10 monclient: shutdown 2024-09-17T18:31:20.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.397+0000 7f88d4b90640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:31:20.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.397+0000 7f88d4b90640 1 -- 172.21.15.121:0/2165021007 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f88d009f040 msgr2=0x7f88d011e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:20.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.398+0000 7f88d4b90640 1 --2- 172.21.15.121:0/2165021007 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f88d009f040 0x7f88d011e000 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f88b800c460 tx=0x7f88b800c920 comp rx=0 tx=0).stop 2024-09-17T18:31:20.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.398+0000 7f88ced9d640 1 -- 172.21.15.121:0/2165021007 reap_dead start 2024-09-17T18:31:20.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.398+0000 7f88ced9d640 1 -- 172.21.15.121:0/2165021007 reap_dead start 2024-09-17T18:31:20.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.398+0000 7f88d4b90640 1 -- 172.21.15.121:0/2165021007 shutdown_connections 2024-09-17T18:31:20.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.398+0000 7f88d4b90640 1 -- 172.21.15.121:0/2165021007 >> 172.21.15.121:0/2165021007 conn(0x7f88d00a7830 msgr2=0x7f88d00a2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:31:20.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.398+0000 7f88d4b90640 1 -- 172.21.15.121:0/2165021007 shutdown_connections 2024-09-17T18:31:20.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.398+0000 7f88d4b90640 1 -- 172.21.15.121:0/2165021007 wait complete. 2024-09-17T18:31:20.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.398+0000 7f88d4b90640 1 librados: shutdown 2024-09-17T18:31:20.416 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:31:20.416 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-17T18:31:20.416 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-17T18:31:20.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.518+0000 7f1f76b73640 10 monclient: get_monmap_and_config 2024-09-17T18:31:20.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.518+0000 7f1f76b73640 10 monclient: build_initial_monmap 2024-09-17T18:31:20.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.518+0000 7f1f76b73640 10 monclient: monmap: 2024-09-17T18:31:20.520 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:31:20.520 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:20.520 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:31:20.518914+0000 2024-09-17T18:31:20.520 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:31:20.518914+0000 2024-09-17T18:31:20.520 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:31:20.520 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:20.520 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:31:20.520 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:31:20.521 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:31:20.521 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:31:20.521 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:31:20.521 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:20.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.519+0000 7f1f76b73640 1 Processor -- start 2024-09-17T18:31:20.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.519+0000 7f1f76b73640 1 -- start start 2024-09-17T18:31:20.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.519+0000 7f1f76b73640 10 monclient: init 2024-09-17T18:31:20.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.519+0000 7f1f76b73640 10 monclient: _reopen_session rank -1 2024-09-17T18:31:20.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.519+0000 7f1f76b73640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-17T18:31:20.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.519+0000 7f1f76b73640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f1f70136800 0x7f1f70136170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:20.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.519+0000 7f1f76b73640 10 monclient(hunting): picked mon.noname-c con 0x7f1f70136800 addr v2:172.21.15.173:3300/0 2024-09-17T18:31:20.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.519+0000 7f1f76b73640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1f70136be0 0x7f1f70132c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:20.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.519+0000 7f1f76b73640 10 monclient(hunting): picked mon.noname-a con 0x7f1f70136be0 addr v2:172.21.15.121:3300/0 2024-09-17T18:31:20.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.519+0000 7f1f76b73640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f70133190 0x7f1f701335e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:20.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.519+0000 7f1f76b73640 10 monclient(hunting): picked mon.noname-b con 0x7f1f70133190 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:31:20.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.519+0000 7f1f76b73640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:20.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.519+0000 7f1f76b73640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f1f700a0c20 con 0x7f1f70136be0 2024-09-17T18:31:20.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.519+0000 7f1f76b73640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:20.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.519+0000 7f1f76b73640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f1f700a2cb0 con 0x7f1f70136800 2024-09-17T18:31:20.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.519+0000 7f1f76b73640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:20.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.519+0000 7f1f76b73640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1f700a2f90 con 0x7f1f70133190 2024-09-17T18:31:20.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.519+0000 7f1f76b73640 10 monclient(hunting): _renew_subs 2024-09-17T18:31:20.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.519+0000 7f1f76b73640 10 monclient(hunting): authenticate will time out at 1484.235634s 2024-09-17T18:31:20.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.519+0000 7f1f6ffff640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1f70136be0 0x7f1f70132c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:20.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.519+0000 7f1f75110640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f70133190 0x7f1f701335e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:20.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.519+0000 7f1f6ffff640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1f70136be0 0x7f1f70132c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60266/0 (socket says 172.21.15.121:60266) 2024-09-17T18:31:20.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.519+0000 7f1f6ffff640 1 -- 172.21.15.121:0/3487814076 learned_addr learned my addr 172.21.15.121:0/3487814076 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:31:20.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.519+0000 7f1f6ffff640 10 monclient(hunting): get_auth_request con 0x7f1f70136be0 auth_method 0 2024-09-17T18:31:20.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.519+0000 7f1f6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:20.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.519+0000 7f1f6ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:20.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.519+0000 7f1f6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:20.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.519+0000 7f1f75110640 10 monclient(hunting): get_auth_request con 0x7f1f70133190 auth_method 0 2024-09-17T18:31:20.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.519+0000 7f1f75110640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:20.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.519+0000 7f1f75110640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:20.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.519+0000 7f1f75110640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:20.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.519+0000 7f1f6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:20.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.519+0000 7f1f6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:20.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.519+0000 7f1f6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:20.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.519+0000 7f1f75110640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:20.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.519+0000 7f1f75110640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:20.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.519+0000 7f1f75110640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:20.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.520+0000 7f1f75110640 10 monclient(hunting): handle_auth_done global_id 64307 payload 274 2024-09-17T18:31:20.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.520+0000 7f1f7490f640 1 --2- 172.21.15.121:0/3487814076 >> v2:172.21.15.173:3300/0 conn(0x7f1f70136800 0x7f1f70136170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:20.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.520+0000 7f1f75110640 1 -- 172.21.15.121:0/3487814076 >> v2:172.21.15.173:3300/0 conn(0x7f1f70136800 msgr2=0x7f1f70136170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:31:20.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.520+0000 7f1f75110640 1 --2- 172.21.15.121:0/3487814076 >> v2:172.21.15.173:3300/0 conn(0x7f1f70136800 0x7f1f70136170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:20.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.520+0000 7f1f75110640 1 -- 172.21.15.121:0/3487814076 >> v2:172.21.15.121:3300/0 conn(0x7f1f70136be0 msgr2=0x7f1f70132c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:31:20.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.520+0000 7f1f75110640 1 --2- 172.21.15.121:0/3487814076 >> v2:172.21.15.121:3300/0 conn(0x7f1f70136be0 0x7f1f70132c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:20.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.520+0000 7f1f75110640 10 monclient: _finish_hunting 0 2024-09-17T18:31:20.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.520+0000 7f1f75110640 1 monclient: found mon.noname-b 2024-09-17T18:31:20.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.520+0000 7f1f75110640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:31:20.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.520+0000 7f1f75110640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:31:20.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.520+0000 7f1f75110640 1 -- 172.21.15.121:0/3487814076 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f700a3270 con 0x7f1f70133190 2024-09-17T18:31:20.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.520+0000 7f1f7490f640 10 monclient: get_auth_request con 0x7f1f70136800 auth_method 0 2024-09-17T18:31:20.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.520+0000 7f1f6ffff640 1 --2- 172.21.15.121:0/3487814076 >> v2:172.21.15.121:3300/0 conn(0x7f1f70136be0 0x7f1f70132c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:31:20.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.520+0000 7f1f7490f640 1 --2- 172.21.15.121:0/3487814076 >> v2:172.21.15.173:3300/0 conn(0x7f1f70136800 0x7f1f70136170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T18:31:20.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.522+0000 7f1f75110640 1 -- 172.21.15.121:0/3487814076 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f70133190 msgr2=0x7f1f701335e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:20.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.522+0000 7f1f75110640 1 --2- 172.21.15.121:0/3487814076 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f70133190 0x7f1f701335e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1f70076d00 tx=0x7f1f5802a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=674470f05f95278b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:20.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.522+0000 7f1f6f7fe640 1 -- 172.21.15.121:0/3487814076 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f58037070 con 0x7f1f70133190 2024-09-17T18:31:20.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.522+0000 7f1f6f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:20.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.522+0000 7f1f6f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:31:20.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.522+0000 7f1f6f7fe640 10 monclient: dump: 2024-09-17T18:31:20.533 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:20.533 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:20.533 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:20.533 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:20.533 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:20.533 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:20.533 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:20.533 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:20.533 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:20.533 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:20.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.523+0000 7f1f6f7fe640 10 monclient: _finish_auth 0 2024-09-17T18:31:20.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.523+0000 7f1f6f7fe640 10 monclient: _check_auth_tickets 2024-09-17T18:31:20.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.523+0000 7f1f6f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:31:20.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.523+0000 7f1f6f7fe640 1 -- 172.21.15.121:0/3487814076 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f58032440 con 0x7f1f70133190 2024-09-17T18:31:20.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.523+0000 7f1f76b73640 5 monclient: authenticate success, global_id 64307 2024-09-17T18:31:20.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.523+0000 7f1f76b73640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:31:20.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.523+0000 7f1f6f7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:31:20.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.523+0000 7f1f76b73640 10 monclient: get_monmap_and_config success 2024-09-17T18:31:20.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.523+0000 7f1f6f7fe640 1 -- 172.21.15.121:0/3487814076 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f58032810 con 0x7f1f70133190 2024-09-17T18:31:20.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.523+0000 7f1f76b73640 4 set_mon_vals no callback set 2024-09-17T18:31:20.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.523+0000 7f1f6f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:20.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.523+0000 7f1f6f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:31:20.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.523+0000 7f1f6f7fe640 10 monclient: dump: 2024-09-17T18:31:20.534 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:20.534 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:20.534 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:20.534 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:20.534 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:20.534 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:20.535 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:20.535 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:20.535 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:20.535 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:20.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.523+0000 7f1f76b73640 10 monclient: shutdown 2024-09-17T18:31:20.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.523+0000 7f1f76b73640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:31:20.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.523+0000 7f1f76b73640 1 -- 172.21.15.121:0/3487814076 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f70133190 msgr2=0x7f1f701335e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:20.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.523+0000 7f1f76b73640 1 --2- 172.21.15.121:0/3487814076 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f70133190 0x7f1f701335e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1f70076d00 tx=0x7f1f5802a000 comp rx=0 tx=0).stop 2024-09-17T18:31:20.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.523+0000 7f1f76b73640 1 -- 172.21.15.121:0/3487814076 shutdown_connections 2024-09-17T18:31:20.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.523+0000 7f1f76b73640 1 --2- 172.21.15.121:0/3487814076 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f70133190 0x7f1f701335e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:20.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.523+0000 7f1f76b73640 1 --2- 172.21.15.121:0/3487814076 >> v2:172.21.15.121:3300/0 conn(0x7f1f70136be0 0x7f1f70132c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:20.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.523+0000 7f1f76b73640 1 --2- 172.21.15.121:0/3487814076 >> v2:172.21.15.173:3300/0 conn(0x7f1f70136800 0x7f1f70136170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:20.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.523+0000 7f1f76b73640 1 -- 172.21.15.121:0/3487814076 >> 172.21.15.121:0/3487814076 conn(0x7f1f700a3b50 msgr2=0x7f1f700a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:31:20.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.523+0000 7f1f76b73640 1 -- 172.21.15.121:0/3487814076 shutdown_connections 2024-09-17T18:31:20.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.523+0000 7f1f76b73640 1 -- 172.21.15.121:0/3487814076 wait complete. 2024-09-17T18:31:20.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.524+0000 7f1f76b73640 10 monclient: build_initial_monmap 2024-09-17T18:31:20.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.524+0000 7f1f76b73640 10 monclient: monmap: 2024-09-17T18:31:20.536 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:31:20.536 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:31:20.536 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:31:20.536 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:31:20.536 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:31:20.536 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:20.536 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:31:20.536 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:31:20.536 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:31:20.536 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:20.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.524+0000 7f1f76b73640 1 librados: starting msgr at 2024-09-17T18:31:20.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.524+0000 7f1f76b73640 1 librados: starting objecter 2024-09-17T18:31:20.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.524+0000 7f1f76b73640 1 Processor -- start 2024-09-17T18:31:20.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.524+0000 7f1f76b73640 1 -- start start 2024-09-17T18:31:20.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.524+0000 7f1f76b73640 1 librados: setting wanted keys 2024-09-17T18:31:20.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.524+0000 7f1f76b73640 1 librados: calling monclient init 2024-09-17T18:31:20.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.524+0000 7f1f76b73640 10 monclient: init 2024-09-17T18:31:20.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f76b73640 10 monclient: _reopen_session rank -1 2024-09-17T18:31:20.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f76b73640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T18:31:20.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f76b73640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f70133190 0x7f1f7012fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:20.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f76b73640 10 monclient(hunting): picked mon.noname-b con 0x7f1f70133190 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:31:20.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f76b73640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f70136800 0x7f1f7012ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:20.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f76b73640 10 monclient(hunting): picked mon.noname-a con 0x7f1f70136800 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:31:20.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f76b73640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f70136be0 0x7f1f7011d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:20.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f76b73640 10 monclient(hunting): picked mon.noname-c con 0x7f1f70136be0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:31:20.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f76b73640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:20.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f76b73640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7009dce0 con 0x7f1f70136800 2024-09-17T18:31:20.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f76b73640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:20.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f76b73640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1f70137050 con 0x7f1f70133190 2024-09-17T18:31:20.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f76b73640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:20.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f76b73640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1f7009de60 con 0x7f1f70136be0 2024-09-17T18:31:20.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f76b73640 10 monclient(hunting): _renew_subs 2024-09-17T18:31:20.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f76b73640 10 monclient(hunting): authenticate will time out at 1484.241104s 2024-09-17T18:31:20.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f6ffff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f70136800 0x7f1f7012ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:20.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f75110640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f70136be0 0x7f1f7011d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:20.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f6ffff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f70136800 0x7f1f7012ff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60272/0 (socket says 172.21.15.121:60272) 2024-09-17T18:31:20.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f6ffff640 1 -- 172.21.15.121:0/3469602840 learned_addr learned my addr 172.21.15.121:0/3469602840 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:31:20.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f6ffff640 10 monclient(hunting): get_auth_request con 0x7f1f70136800 auth_method 0 2024-09-17T18:31:20.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:20.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f6ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:20.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:20.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f75110640 10 monclient(hunting): get_auth_request con 0x7f1f70136be0 auth_method 0 2024-09-17T18:31:20.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f75110640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:20.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f75110640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:20.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f75110640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:20.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:20.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:20.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:20.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f75110640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:20.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f75110640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:20.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f75110640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:20.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f7490f640 1 --2- 172.21.15.121:0/3469602840 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f70133190 0x7f1f7012fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:20.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f7490f640 10 monclient(hunting): get_auth_request con 0x7f1f70133190 auth_method 0 2024-09-17T18:31:20.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f7490f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:20.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f7490f640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:20.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f7490f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:20.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f6ffff640 10 monclient(hunting): handle_auth_done global_id 44673 payload 824 2024-09-17T18:31:20.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f6ffff640 1 -- 172.21.15.121:0/3469602840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f70136be0 msgr2=0x7f1f7011d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:20.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f6ffff640 1 --2- 172.21.15.121:0/3469602840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f70136be0 0x7f1f7011d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:20.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f6ffff640 1 -- 172.21.15.121:0/3469602840 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f70133190 msgr2=0x7f1f7012fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:20.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f6ffff640 1 --2- 172.21.15.121:0/3469602840 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f70133190 0x7f1f7012fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:20.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f6ffff640 10 monclient: _finish_hunting 0 2024-09-17T18:31:20.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f6ffff640 1 monclient: found mon.noname-a 2024-09-17T18:31:20.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f6ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:31:20.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f6ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:31:20.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f6ffff640 1 -- 172.21.15.121:0/3469602840 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f70121420 con 0x7f1f70136800 2024-09-17T18:31:20.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.525+0000 7f1f7490f640 1 --2- 172.21.15.121:0/3469602840 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f70133190 0x7f1f7012fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T18:31:20.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.527+0000 7f1f6ffff640 1 -- 172.21.15.121:0/3469602840 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f70136800 msgr2=0x7f1f7012ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:20.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.527+0000 7f1f6ffff640 1 --2- 172.21.15.121:0/3469602840 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f70136800 0x7f1f7012ff40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f1f6000efc0 tx=0x7f1f6000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:20.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.527+0000 7f1f6d7fa640 10 client.?.objecter ms_handle_connect 0x7f1f70136800 2024-09-17T18:31:20.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.527+0000 7f1f6d7fa640 10 client.?.objecter resend_mon_ops 2024-09-17T18:31:20.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.527+0000 7f1f6d7fa640 1 -- 172.21.15.121:0/3469602840 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f60014680 con 0x7f1f70136800 2024-09-17T18:31:20.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.527+0000 7f1f6d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:20.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.527+0000 7f1f6d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:31:20.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.527+0000 7f1f6d7fa640 10 monclient: dump: 2024-09-17T18:31:20.543 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:20.543 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:20.543 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:20.543 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:20.543 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:20.543 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:20.543 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:20.543 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:20.543 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:20.543 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:20.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.527+0000 7f1f6d7fa640 10 monclient: _finish_auth 0 2024-09-17T18:31:20.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.527+0000 7f1f6d7fa640 10 monclient: _check_auth_tickets 2024-09-17T18:31:20.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.527+0000 7f1f6d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:31:20.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.527+0000 7f1f6d7fa640 1 -- 172.21.15.121:0/3469602840 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f60019070 con 0x7f1f70136800 2024-09-17T18:31:20.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.527+0000 7f1f76b73640 5 monclient: authenticate success, global_id 44673 2024-09-17T18:31:20.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.527+0000 7f1f76b73640 10 monclient: _renew_subs 2024-09-17T18:31:20.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.527+0000 7f1f76b73640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:31:20.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.527+0000 7f1f76b73640 1 -- 172.21.15.121:0/3469602840 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f7009e240 con 0x7f1f70136800 2024-09-17T18:31:20.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.527+0000 7f1f6d7fa640 10 monclient: handle_config config(0 keys) 2024-09-17T18:31:20.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.527+0000 7f1f6d7fa640 1 -- 172.21.15.121:0/3469602840 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f60014a20 con 0x7f1f70136800 2024-09-17T18:31:20.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.527+0000 7f1f6d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:20.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.527+0000 7f1f6d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:31:20.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.527+0000 7f1f6d7fa640 10 monclient: dump: 2024-09-17T18:31:20.544 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:20.544 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:20.544 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:20.544 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:20.545 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:20.545 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:20.545 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:20.545 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:20.545 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:20.545 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:20.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.527+0000 7f1f6effd640 4 set_mon_vals no callback set 2024-09-17T18:31:20.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.527+0000 7f1f76b73640 10 client.44673.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:31:20.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.527+0000 7f1f76b73640 10 monclient: _renew_subs 2024-09-17T18:31:20.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.527+0000 7f1f76b73640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:31:20.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.527+0000 7f1f76b73640 1 -- 172.21.15.121:0/3469602840 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f70130480 con 0x7f1f70136800 2024-09-17T18:31:20.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.527+0000 7f1f76b73640 1 librados: init done 2024-09-17T18:31:20.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.528+0000 7f1f6d7fa640 1 -- 172.21.15.121:0/3469602840 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f1f60005d10 con 0x7f1f70136800 2024-09-17T18:31:20.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.528+0000 7f1f76b73640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:31:20.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.528+0000 7f1f76b73640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:31:20.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.528+0000 7f1f76b73640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:31:20.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.528+0000 7f1f76b73640 1 -- 172.21.15.121:0/3469602840 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f701335e0 con 0x7f1f70136800 2024-09-17T18:31:20.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.529+0000 7f1f6d7fa640 1 --2- 172.21.15.121:0/3469602840 >> v2:172.21.15.173:6812/2423786037 conn(0x7f1f500401a0 0x7f1f50042660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:20.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.529+0000 7f1f6d7fa640 1 -- 172.21.15.121:0/3469602840 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(484..484 src has 247..484) ==== 6678+0+0 (secure 0 0 0) 0x7f1f600541e0 con 0x7f1f70136800 2024-09-17T18:31:20.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.529+0000 7f1f6d7fa640 10 client.44673.objecter ms_dispatch 0x7f1f70000b90 osd_map(484..484 src has 247..484) 2024-09-17T18:31:20.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.529+0000 7f1f6d7fa640 3 client.44673.objecter handle_osd_map got epochs [484,484] > 0 2024-09-17T18:31:20.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.529+0000 7f1f6d7fa640 3 client.44673.objecter handle_osd_map decoding full epoch 484 2024-09-17T18:31:20.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.529+0000 7f1f6d7fa640 20 client.44673.objecter dump_active .. 0 homeless 2024-09-17T18:31:20.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.529+0000 7f1f7490f640 1 --2- 172.21.15.121:0/3469602840 >> v2:172.21.15.173:6812/2423786037 conn(0x7f1f500401a0 0x7f1f50042660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:20.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.529+0000 7f1f7490f640 10 monclient: get_auth_request con 0x7f1f500401a0 auth_method 0 2024-09-17T18:31:20.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.531+0000 7f1f7490f640 1 --2- 172.21.15.121:0/3469602840 >> v2:172.21.15.173:6812/2423786037 conn(0x7f1f500401a0 0x7f1f50042660 secure :-1 s=READY pgs=4034 cs=0 l=1 rev1=1 crypto rx=0x7f1f6800aa30 tx=0x7f1f68008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:20.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.532+0000 7f1f6d7fa640 10 client.44673.objecter ms_handle_connect 0x7f1f500401a0 2024-09-17T18:31:20.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.533+0000 7f1f6d7fa640 1 -- 172.21.15.121:0/3469602840 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f1f60010040 con 0x7f1f70136800 2024-09-17T18:31:20.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.533+0000 7f1f6d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:31:20.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.533+0000 7f1f6d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:31:20.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.772+0000 7f1f76b73640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-17T18:31:20.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.772+0000 7f1f76b73640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-17T18:31:20.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.772+0000 7f1f76b73640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:31:20.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.772+0000 7f1f76b73640 1 -- 172.21.15.121:0/3469602840 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f1f7006ce60 con 0x7f1f70136800 2024-09-17T18:31:20.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.773+0000 7f1f6d7fa640 1 -- 172.21.15.121:0/3469602840 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f1f6001c370 con 0x7f1f70136800 2024-09-17T18:31:20.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.773+0000 7f1f6d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-17T18:31:20.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.773+0000 7f1f6d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:31:20.775 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.773+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.775 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.773+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=17,config=2,health=11,kv=13,logm=57} crc {auth=62946307,config=2586702516,health=2885314881,kv=1513842021,logm=166816428}) 2024-09-17T18:31:20.775 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.773+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=17,config=2,health=11,kv=14,logm=56} crc {auth=62946307,config=2586702516,health=2452846697,kv=2790417503,logm=258859200}) 2024-09-17T18:31:20.775 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.773+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.775 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.773+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=17,config=2,health=11,kv=13,logm=57} crc {auth=62946307,config=2586702516,health=2885314881,kv=1513842021,logm=166816428}) 2024-09-17T18:31:20.775 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.773+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=17,config=2,health=11,kv=12,logm=58} crc {auth=62946307,config=2586702516,health=1925738131,kv=2296980842,logm=1605642753}) 2024-09-17T18:31:20.777 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.775+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.777 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.775+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2121158601}) 2024-09-17T18:31:20.777 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.775+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=328769498}) 2024-09-17T18:31:20.777 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.775+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.777 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.775+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2121158601}) 2024-09-17T18:31:20.777 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.775+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=54836751}) 2024-09-17T18:31:20.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.775+0000 7f1f76b73640 10 librados: watch_flush enter 2024-09-17T18:31:20.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.775+0000 7f1f76b73640 10 librados: watch_flush exit 2024-09-17T18:31:20.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.775+0000 7f1f76b73640 20 client.44673.objecter shutdown clearing up homeless session... 2024-09-17T18:31:20.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.775+0000 7f1f76b73640 10 client.44673.objecter successfully canceled tick 2024-09-17T18:31:20.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.775+0000 7f1f76b73640 1 -- 172.21.15.121:0/3469602840 >> v2:172.21.15.173:6812/2423786037 conn(0x7f1f500401a0 msgr2=0x7f1f50042660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:20.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.775+0000 7f1f76b73640 1 --2- 172.21.15.121:0/3469602840 >> v2:172.21.15.173:6812/2423786037 conn(0x7f1f500401a0 0x7f1f50042660 secure :-1 s=READY pgs=4034 cs=0 l=1 rev1=1 crypto rx=0x7f1f6800aa30 tx=0x7f1f68008040 comp rx=0 tx=0).stop 2024-09-17T18:31:20.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.775+0000 7f1f76b73640 10 monclient: shutdown 2024-09-17T18:31:20.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.775+0000 7f1f76b73640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:31:20.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.775+0000 7f1f76b73640 1 -- 172.21.15.121:0/3469602840 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f70136800 msgr2=0x7f1f7012ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:20.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.775+0000 7f1f76b73640 1 --2- 172.21.15.121:0/3469602840 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f70136800 0x7f1f7012ff40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f1f6000efc0 tx=0x7f1f6000c490 comp rx=0 tx=0).stop 2024-09-17T18:31:20.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.776+0000 7f1f76b73640 1 -- 172.21.15.121:0/3469602840 shutdown_connections 2024-09-17T18:31:20.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.776+0000 7f1f76b73640 1 --2- 172.21.15.121:0/3469602840 >> v2:172.21.15.173:6812/2423786037 conn(0x7f1f500401a0 0x7f1f50042660 unknown :-1 s=CLOSED pgs=4034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:20.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.776+0000 7f1f76b73640 1 --2- 172.21.15.121:0/3469602840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f70136be0 0x7f1f7011d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:20.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.776+0000 7f1f76b73640 1 --2- 172.21.15.121:0/3469602840 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f70136800 0x7f1f7012ff40 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:20.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.776+0000 7f1f76b73640 1 --2- 172.21.15.121:0/3469602840 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f70133190 0x7f1f7012fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:20.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.776+0000 7f1f76b73640 1 -- 172.21.15.121:0/3469602840 >> 172.21.15.121:0/3469602840 conn(0x7f1f700a3b50 msgr2=0x7f1f700a22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:31:20.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.776+0000 7f1f76b73640 1 -- 172.21.15.121:0/3469602840 shutdown_connections 2024-09-17T18:31:20.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.776+0000 7f1f76b73640 1 -- 172.21.15.121:0/3469602840 wait complete. 2024-09-17T18:31:20.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:20.776+0000 7f1f76b73640 1 librados: shutdown 2024-09-17T18:31:20.781 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.776+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.781 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.776+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3942937745}) 2024-09-17T18:31:20.781 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.777+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=314882662}) 2024-09-17T18:31:20.781 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.777+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.781 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.777+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3942937745}) 2024-09-17T18:31:20.781 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.777+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4112774482}) 2024-09-17T18:31:20.781 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.779+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.781 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.779+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3639623114}) 2024-09-17T18:31:20.782 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.779+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3068465796}) 2024-09-17T18:31:20.782 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.779+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.782 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.779+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3639623114}) 2024-09-17T18:31:20.782 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.779+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2992890354}) 2024-09-17T18:31:20.782 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.780+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.782 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.780+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1592176142}) 2024-09-17T18:31:20.783 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.780+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3907986603}) 2024-09-17T18:31:20.783 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.780+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.783 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.780+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1592176142}) 2024-09-17T18:31:20.783 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.780+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1149206619}) 2024-09-17T18:31:20.783 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.781+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.783 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.781+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3639691945}) 2024-09-17T18:31:20.784 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.781+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3257815266}) 2024-09-17T18:31:20.784 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.781+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.784 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.781+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3639691945}) 2024-09-17T18:31:20.784 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.781+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2271342540}) 2024-09-17T18:31:20.784 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.782+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.785 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.782+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3126218124}) 2024-09-17T18:31:20.785 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.782+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=524380098}) 2024-09-17T18:31:20.785 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.782+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.785 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.782+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3126218124}) 2024-09-17T18:31:20.785 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.782+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=798652782}) 2024-09-17T18:31:20.785 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.783+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.786 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.783+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1411031714}) 2024-09-17T18:31:20.786 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.783+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1776549691}) 2024-09-17T18:31:20.786 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.783+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.786 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.783+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1411031714}) 2024-09-17T18:31:20.786 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.783+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3462837748}) 2024-09-17T18:31:20.786 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.784+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.787 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.784+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3847723727}) 2024-09-17T18:31:20.787 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.784+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3897181865}) 2024-09-17T18:31:20.787 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.784+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.787 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.784+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3847723727}) 2024-09-17T18:31:20.787 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.784+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3827389736}) 2024-09-17T18:31:20.787 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.785+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.787 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.785+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=911142321}) 2024-09-17T18:31:20.788 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.785+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=623862475}) 2024-09-17T18:31:20.788 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.785+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.788 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.785+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=911142321}) 2024-09-17T18:31:20.788 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.785+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4180674779}) 2024-09-17T18:31:20.788 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.786+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.789 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.786+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4161942377}) 2024-09-17T18:31:20.789 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.786+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2166022489}) 2024-09-17T18:31:20.789 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.786+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.789 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.786+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4161942377}) 2024-09-17T18:31:20.789 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.786+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4030594965}) 2024-09-17T18:31:20.790 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.787+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.790 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.787+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3568200040}) 2024-09-17T18:31:20.790 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.787+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1212853644}) 2024-09-17T18:31:20.790 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.787+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.790 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.787+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3568200040}) 2024-09-17T18:31:20.790 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.787+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2543822096}) 2024-09-17T18:31:20.790 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.788+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.790 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.788+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2407269556}) 2024-09-17T18:31:20.790 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.788+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=81992226}) 2024-09-17T18:31:20.790 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.788+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.791 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.788+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2407269556}) 2024-09-17T18:31:20.791 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.788+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=212906409}) 2024-09-17T18:31:20.792 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.789+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.792 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.789+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3786264719}) 2024-09-17T18:31:20.792 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.789+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=376029549}) 2024-09-17T18:31:20.792 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.789+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.792 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.789+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3786264719}) 2024-09-17T18:31:20.792 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.789+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2462534872}) 2024-09-17T18:31:20.792 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.790+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.792 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.790+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3741126604}) 2024-09-17T18:31:20.792 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.790+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3383433485}) 2024-09-17T18:31:20.792 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.790+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.793 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.790+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3741126604}) 2024-09-17T18:31:20.793 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.790+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1388439203}) 2024-09-17T18:31:20.794 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.791+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.794 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.791+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2845368823}) 2024-09-17T18:31:20.794 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.791+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2950956471}) 2024-09-17T18:31:20.794 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.791+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.794 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.791+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2845368823}) 2024-09-17T18:31:20.794 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.791+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2751702631}) 2024-09-17T18:31:20.794 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.791+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.794 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.791+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=242352365}) 2024-09-17T18:31:20.794 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.791+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1532449561}) 2024-09-17T18:31:20.794 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.791+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.794 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.791+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=242352365}) 2024-09-17T18:31:20.794 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.791+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1194748065}) 2024-09-17T18:31:20.795 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-17T18:31:20.797 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.792+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.797 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.792+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4003539191}) 2024-09-17T18:31:20.797 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.792+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1964273078}) 2024-09-17T18:31:20.797 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.792+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.797 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.792+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4003539191}) 2024-09-17T18:31:20.797 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.792+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=877509960}) 2024-09-17T18:31:20.797 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.793+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.797 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.793+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2018086715}) 2024-09-17T18:31:20.797 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.793+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1703433608}) 2024-09-17T18:31:20.798 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.793+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.798 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.793+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2018086715}) 2024-09-17T18:31:20.798 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.793+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4294259158}) 2024-09-17T18:31:20.798 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.794+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.798 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.794+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=947215224}) 2024-09-17T18:31:20.798 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.794+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2419666795}) 2024-09-17T18:31:20.798 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.794+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.798 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.794+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=947215224}) 2024-09-17T18:31:20.798 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.794+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2214410843}) 2024-09-17T18:31:20.798 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.794+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.798 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.794+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3407146920}) 2024-09-17T18:31:20.798 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.794+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=766439010}) 2024-09-17T18:31:20.798 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.794+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.798 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.794+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3407146920}) 2024-09-17T18:31:20.798 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.794+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=538527794}) 2024-09-17T18:31:20.799 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.795+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.799 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.795+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=887441431}) 2024-09-17T18:31:20.799 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.795+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2752481486}) 2024-09-17T18:31:20.799 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.795+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.799 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.795+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=887441431}) 2024-09-17T18:31:20.799 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.795+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=56576013}) 2024-09-17T18:31:20.801 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.796+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.801 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.796+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1425384318}) 2024-09-17T18:31:20.801 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.796+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1298528090}) 2024-09-17T18:31:20.801 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.796+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.801 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.796+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1425384318}) 2024-09-17T18:31:20.801 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.796+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4023067461}) 2024-09-17T18:31:20.802 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.796+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.802 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.796+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=118465822}) 2024-09-17T18:31:20.802 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.796+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1570375862}) 2024-09-17T18:31:20.802 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.796+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.802 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.796+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=118465822}) 2024-09-17T18:31:20.802 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.796+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=655454255}) 2024-09-17T18:31:20.802 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.797+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.802 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.797+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3870501376}) 2024-09-17T18:31:20.802 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.797+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1718090306}) 2024-09-17T18:31:20.802 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.797+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.802 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.797+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3870501376}) 2024-09-17T18:31:20.802 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.797+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1008482665}) 2024-09-17T18:31:20.802 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.798+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.802 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.798+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3278059111}) 2024-09-17T18:31:20.802 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.798+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1997613184}) 2024-09-17T18:31:20.803 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.798+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.803 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.798+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3278059111}) 2024-09-17T18:31:20.803 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.798+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3225813867}) 2024-09-17T18:31:20.803 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.798+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.803 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.798+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2838976798}) 2024-09-17T18:31:20.803 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.798+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3204831842}) 2024-09-17T18:31:20.803 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.798+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.803 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.798+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2838976798}) 2024-09-17T18:31:20.803 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.798+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3841783219}) 2024-09-17T18:31:20.803 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.799+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.803 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.799+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3499848313}) 2024-09-17T18:31:20.803 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.799+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3110106522}) 2024-09-17T18:31:20.803 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.799+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.803 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.799+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3499848313}) 2024-09-17T18:31:20.803 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.799+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3327717758}) 2024-09-17T18:31:20.804 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.800+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.805 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.800+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3455032238}) 2024-09-17T18:31:20.805 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.800+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2096065410}) 2024-09-17T18:31:20.805 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.800+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.805 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.800+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3455032238}) 2024-09-17T18:31:20.805 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.800+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4082302507}) 2024-09-17T18:31:20.805 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.801+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.805 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.801+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4279594225}) 2024-09-17T18:31:20.805 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.801+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2425504568}) 2024-09-17T18:31:20.805 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.801+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.805 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.801+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4279594225}) 2024-09-17T18:31:20.805 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.801+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3785787620}) 2024-09-17T18:31:20.920 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.918+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.920 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.918+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3427082861}) 2024-09-17T18:31:20.920 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.918+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1117523335}) 2024-09-17T18:31:20.920 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.918+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.921 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.918+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3427082861}) 2024-09-17T18:31:20.921 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.918+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2385976268}) 2024-09-17T18:31:20.927 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.925+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.927 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.925+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1940247936}) 2024-09-17T18:31:20.927 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.925+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1617150459}) 2024-09-17T18:31:20.927 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.925+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:20.927 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.925+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1940247936}) 2024-09-17T18:31:20.927 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:20.925+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3843486882}) 2024-09-17T18:31:21.305 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.303+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.305 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.303+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4077391018}) 2024-09-17T18:31:21.305 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.303+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=58181927}) 2024-09-17T18:31:21.305 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.303+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.305 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.303+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4077391018}) 2024-09-17T18:31:21.305 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.303+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3391737483}) 2024-09-17T18:31:21.306 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.304+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.306 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.304+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=110806222}) 2024-09-17T18:31:21.306 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.304+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2689388563}) 2024-09-17T18:31:21.306 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.304+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.306 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.304+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=110806222}) 2024-09-17T18:31:21.306 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.304+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=378544381}) 2024-09-17T18:31:21.307 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.305+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.307 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.305+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1319408793}) 2024-09-17T18:31:21.308 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.305+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2605385828}) 2024-09-17T18:31:21.308 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.305+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.308 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.305+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1319408793}) 2024-09-17T18:31:21.308 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.305+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2491172029}) 2024-09-17T18:31:21.308 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.305+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.308 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.305+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=821905769}) 2024-09-17T18:31:21.308 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.305+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2230306066}) 2024-09-17T18:31:21.308 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.305+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.308 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.305+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=821905769}) 2024-09-17T18:31:21.308 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.305+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=939033897}) 2024-09-17T18:31:21.444 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.441+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.444 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.441+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2386303369}) 2024-09-17T18:31:21.444 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.441+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2061323111}) 2024-09-17T18:31:21.444 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.442+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.444 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.442+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2386303369}) 2024-09-17T18:31:21.444 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.442+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2592142384}) 2024-09-17T18:31:21.449 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.447+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.450 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.448+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3542980743}) 2024-09-17T18:31:21.450 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.448+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2376828964}) 2024-09-17T18:31:21.450 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.448+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.450 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.448+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3542980743}) 2024-09-17T18:31:21.450 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.448+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1679265841}) 2024-09-17T18:31:21.688 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.686+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.688 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.686+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1401213028}) 2024-09-17T18:31:21.688 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.686+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3579101702}) 2024-09-17T18:31:21.688 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.686+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.688 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.686+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1401213028}) 2024-09-17T18:31:21.688 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.686+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3507862040}) 2024-09-17T18:31:21.719 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.717+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.719 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.717+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=690460428}) 2024-09-17T18:31:21.719 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.717+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=118718812}) 2024-09-17T18:31:21.719 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.717+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.719 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.717+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=690460428}) 2024-09-17T18:31:21.719 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.717+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=376335282}) 2024-09-17T18:31:21.796 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T18:31:21.796 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:31:21.801 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.799+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.801 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.799+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3564575544}) 2024-09-17T18:31:21.801 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.799+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3994391148}) 2024-09-17T18:31:21.801 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.799+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.801 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.799+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3564575544}) 2024-09-17T18:31:21.801 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.799+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4016509395}) 2024-09-17T18:31:21.802 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.800+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.802 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.800+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3097921793}) 2024-09-17T18:31:21.802 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.800+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2124767070}) 2024-09-17T18:31:21.802 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.800+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.802 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.800+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3097921793}) 2024-09-17T18:31:21.802 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.800+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=371257914}) 2024-09-17T18:31:21.803 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.800+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.803 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.800+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=180008588}) 2024-09-17T18:31:21.803 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.801+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1027118836}) 2024-09-17T18:31:21.803 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.801+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.804 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.801+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=180008588}) 2024-09-17T18:31:21.804 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.801+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=166437976}) 2024-09-17T18:31:21.804 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.801+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.804 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.801+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2453188771}) 2024-09-17T18:31:21.804 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.801+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2344284612}) 2024-09-17T18:31:21.804 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.801+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.804 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.801+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2453188771}) 2024-09-17T18:31:21.804 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.801+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=127656445}) 2024-09-17T18:31:21.806 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.802+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.806 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.802+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1572479345}) 2024-09-17T18:31:21.806 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.802+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=167287299}) 2024-09-17T18:31:21.806 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.802+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.806 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.802+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1572479345}) 2024-09-17T18:31:21.806 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.802+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=666974522}) 2024-09-17T18:31:21.806 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.803+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.806 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.803+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2894106213}) 2024-09-17T18:31:21.806 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.803+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2346171265}) 2024-09-17T18:31:21.806 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.803+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.806 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.803+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2894106213}) 2024-09-17T18:31:21.806 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.803+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=50306491}) 2024-09-17T18:31:21.806 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.804+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.807 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.804+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=832122414}) 2024-09-17T18:31:21.807 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.804+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1364858034}) 2024-09-17T18:31:21.807 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.804+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.807 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.804+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=832122414}) 2024-09-17T18:31:21.807 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.804+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=435369945}) 2024-09-17T18:31:21.809 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.805+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.809 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.805+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=289612002}) 2024-09-17T18:31:21.809 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.805+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1812909311}) 2024-09-17T18:31:21.809 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.805+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.809 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.805+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=289612002}) 2024-09-17T18:31:21.809 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.805+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2938877197}) 2024-09-17T18:31:21.809 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.805+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.809 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.805+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1515309507}) 2024-09-17T18:31:21.809 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.805+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3933625769}) 2024-09-17T18:31:21.809 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.805+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.810 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.805+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1515309507}) 2024-09-17T18:31:21.810 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.805+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=350110752}) 2024-09-17T18:31:21.810 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.806+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.810 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.806+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4116801141}) 2024-09-17T18:31:21.810 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.806+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=353401781}) 2024-09-17T18:31:21.810 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.806+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.810 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.806+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4116801141}) 2024-09-17T18:31:21.810 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.806+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2569907721}) 2024-09-17T18:31:21.810 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.807+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.810 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.807+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=656576191}) 2024-09-17T18:31:21.810 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.807+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1028462845}) 2024-09-17T18:31:21.810 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.807+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.810 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.807+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=656576191}) 2024-09-17T18:31:21.810 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.807+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1850487541}) 2024-09-17T18:31:21.812 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.808+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.812 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.808+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=76,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12} crc {logm=2501381464,mdsmap=761013554,mgr=3621767920,mgr_command_descs=1957073259,mgr_metadata=402068389,mgrstat=1449420705}) 2024-09-17T18:31:21.812 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.808+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=77,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10} crc {logm=4234910827,mdsmap=761013554,mgr=3361498112,mgr_command_descs=1957073259,mgr_metadata=402068389,mgrstat=1199988723}) 2024-09-17T18:31:21.812 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.808+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.812 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.808+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=76,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12} crc {logm=2501381464,mdsmap=761013554,mgr=3621767920,mgr_command_descs=1957073259,mgr_metadata=402068389,mgrstat=1449420705}) 2024-09-17T18:31:21.813 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.808+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=77,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11} crc {logm=1303289412,mdsmap=761013554,mgr=2706156808,mgr_command_descs=1957073259,mgr_metadata=402068389,mgrstat=717438320}) 2024-09-17T18:31:21.813 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.809+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.813 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.809+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=1,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_snap=30,osdmap=40} crc {mgrstat=2186528338,mon_config_key=2290706226,monmap=2277444362,nvmeofgw=598337686,osd_metadata=121115626,osd_snap=1540469356,osdmap=2232669332}) 2024-09-17T18:31:21.813 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.809+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=1,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=31,osdmap=38} crc {mgrstat=2186528338,mon_config_key=1582663684,monmap=2277444362,nvmeofgw=598337686,osd_metadata=121115626,osd_pg_creating=3961310618,osd_snap=2932792767,osdmap=2960871463}) 2024-09-17T18:31:21.813 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.809+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.813 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.809+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=1,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_snap=30,osdmap=40} crc {mgrstat=2186528338,mon_config_key=2290706226,monmap=2277444362,nvmeofgw=598337686,osd_metadata=121115626,osd_snap=1540469356,osdmap=2232669332}) 2024-09-17T18:31:21.813 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.809+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=1,mon_config_key=17,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=30,osdmap=40} crc {mgrstat=2186528338,mon_config_key=1582663684,monmap=2277444362,nvmeofgw=1596038039,osd_metadata=121115626,osd_pg_creating=3961310618,osd_snap=634992682,osdmap=2045272387}) 2024-09-17T18:31:21.813 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.810+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.813 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.810+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1802261421}) 2024-09-17T18:31:21.813 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.810+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=404815092}) 2024-09-17T18:31:21.813 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.810+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.813 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.810+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1802261421}) 2024-09-17T18:31:21.813 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.810+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2770584714}) 2024-09-17T18:31:21.815 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.811+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.815 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.811+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3694500860}) 2024-09-17T18:31:21.815 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.811+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=868849760}) 2024-09-17T18:31:21.815 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.811+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.815 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.811+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3694500860}) 2024-09-17T18:31:21.815 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.811+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3204023181}) 2024-09-17T18:31:21.815 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.813+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.815 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.813+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4025575357}) 2024-09-17T18:31:21.815 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.813+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=489341482}) 2024-09-17T18:31:21.815 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.813+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.816 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.813+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4025575357}) 2024-09-17T18:31:21.816 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.813+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4002808017}) 2024-09-17T18:31:21.818 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.817+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:31:21.818 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.817+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=24} crc {osdmap=1068166711}) 2024-09-17T18:31:21.819 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:31:21.817+0000 7f8f04bf0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=24} crc {osdmap=2320104026}) 2024-09-17T18:31:21.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.896+0000 7f3d71c08640 10 monclient: get_monmap_and_config 2024-09-17T18:31:21.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.897+0000 7f3d71c08640 10 monclient: build_initial_monmap 2024-09-17T18:31:21.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.897+0000 7f3d71c08640 10 monclient: monmap: 2024-09-17T18:31:21.899 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:31:21.899 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:21.899 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:31:21.897612+0000 2024-09-17T18:31:21.899 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:31:21.897612+0000 2024-09-17T18:31:21.899 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:31:21.899 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:21.899 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:31:21.899 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:31:21.899 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:31:21.899 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:31:21.899 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:31:21.899 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:21.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.897+0000 7f3d71c08640 1 Processor -- start 2024-09-17T18:31:21.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.897+0000 7f3d71c08640 1 -- start start 2024-09-17T18:31:21.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.897+0000 7f3d71c08640 10 monclient: init 2024-09-17T18:31:21.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.898+0000 7f3d71c08640 10 monclient: _reopen_session rank -1 2024-09-17T18:31:21.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.898+0000 7f3d71c08640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-17T18:31:21.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.898+0000 7f3d71c08640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f3d6c0a33f0 0x7f3d6c09e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:21.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.898+0000 7f3d71c08640 10 monclient(hunting): picked mon.noname-c con 0x7f3d6c0a33f0 addr v2:172.21.15.173:3300/0 2024-09-17T18:31:21.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.898+0000 7f3d71c08640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3d6c09da70 addr v1:172.21.15.121:6789/0 2024-09-17T18:31:21.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.898+0000 7f3d71c08640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3d6c13a850 addr v1:172.21.15.173:6789/0 2024-09-17T18:31:21.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.898+0000 7f3d71c08640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d6c074b40 con 0x7f3d6c09da70 2024-09-17T18:31:21.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.898+0000 7f3d71c08640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d6c0a13c0 con 0x7f3d6c13a850 2024-09-17T18:31:21.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.898+0000 7f3d71c08640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:21.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.898+0000 7f3d71c08640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f3d6c074550 con 0x7f3d6c0a33f0 2024-09-17T18:31:21.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.898+0000 7f3d71c08640 10 monclient(hunting): _renew_subs 2024-09-17T18:31:21.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.898+0000 7f3d71c08640 10 monclient(hunting): authenticate will time out at 1485.614290s 2024-09-17T18:31:21.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.898+0000 7f3d62ffd640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f3d6c09da70 0x7f3d6c09ea90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:33624/0 (socket says 172.21.15.121:33624) 2024-09-17T18:31:21.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.898+0000 7f3d62ffd640 1 -- 172.21.15.121:0/2621109499 learned_addr learned my addr 172.21.15.121:0/2621109499 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:31:21.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.898+0000 7f3d6b7fe640 1 --2- 172.21.15.121:0/2621109499 >> v2:172.21.15.173:3300/0 conn(0x7f3d6c0a33f0 0x7f3d6c09e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:21.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.898+0000 7f3d6b7fe640 10 monclient(hunting): get_auth_request con 0x7f3d6c0a33f0 auth_method 0 2024-09-17T18:31:21.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.898+0000 7f3d6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:21.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.898+0000 7f3d6b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:21.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.898+0000 7f3d6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:21.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.898+0000 7f3d6b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:21.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.898+0000 7f3d6b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:21.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.898+0000 7f3d6b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:21.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.898+0000 7f3d6b7fe640 10 monclient(hunting): handle_auth_done global_id 84109 payload 274 2024-09-17T18:31:21.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.898+0000 7f3d6b7fe640 1 -- 172.21.15.121:0/2621109499 >> v1:172.21.15.173:6789/0 conn(0x7f3d6c13a850 legacy=0x7f3d6c13cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:31:21.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.898+0000 7f3d6b7fe640 1 -- 172.21.15.121:0/2621109499 >> v1:172.21.15.121:6789/0 conn(0x7f3d6c09da70 legacy=0x7f3d6c09ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:31:21.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.898+0000 7f3d6b7fe640 10 monclient: _finish_hunting 0 2024-09-17T18:31:21.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.898+0000 7f3d6b7fe640 1 monclient: found mon.noname-c 2024-09-17T18:31:21.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.898+0000 7f3d6b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:31:21.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.898+0000 7f3d6b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.173:3300/0 2024-09-17T18:31:21.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.898+0000 7f3d6b7fe640 1 -- 172.21.15.121:0/2621109499 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d6c077730 con 0x7f3d6c0a33f0 2024-09-17T18:31:21.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.901+0000 7f3d6b7fe640 1 -- 172.21.15.121:0/2621109499 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d6c0a33f0 msgr2=0x7f3d6c09e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:21.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.901+0000 7f3d6b7fe640 1 --2- 172.21.15.121:0/2621109499 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d6c0a33f0 0x7f3d6c09e550 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c0097a0 tx=0x7f3d5c02c1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e4d3b9b27d4e2b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:21.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.902+0000 7f3d6affd640 1 -- 172.21.15.121:0/2621109499 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d5c039070 con 0x7f3d6c0a33f0 2024-09-17T18:31:21.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.902+0000 7f3d6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:21.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.902+0000 7f3d6affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:31:21.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.902+0000 7f3d6affd640 10 monclient: dump: 2024-09-17T18:31:21.908 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:21.908 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:21.908 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:21.908 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:21.908 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:21.908 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:21.908 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:21.909 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:21.909 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:21.909 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:21.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.902+0000 7f3d6affd640 10 monclient: _finish_auth 0 2024-09-17T18:31:21.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.902+0000 7f3d6affd640 10 monclient: _check_auth_tickets 2024-09-17T18:31:21.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.902+0000 7f3d6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:31:21.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.902+0000 7f3d6affd640 1 -- 172.21.15.121:0/2621109499 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d5c034640 con 0x7f3d6c0a33f0 2024-09-17T18:31:21.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.902+0000 7f3d71c08640 5 monclient: authenticate success, global_id 84109 2024-09-17T18:31:21.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.902+0000 7f3d71c08640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:31:21.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.902+0000 7f3d6affd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:31:21.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.902+0000 7f3d71c08640 10 monclient: get_monmap_and_config success 2024-09-17T18:31:21.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.902+0000 7f3d6affd640 1 -- 172.21.15.121:0/2621109499 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d5c034a10 con 0x7f3d6c0a33f0 2024-09-17T18:31:21.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.902+0000 7f3d71c08640 4 set_mon_vals no callback set 2024-09-17T18:31:21.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.902+0000 7f3d6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:21.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.902+0000 7f3d6affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:31:21.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.902+0000 7f3d6affd640 10 monclient: dump: 2024-09-17T18:31:21.910 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:21.910 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:21.910 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:21.910 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:21.910 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:21.910 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:21.910 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:21.910 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:21.910 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:21.910 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:21.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.902+0000 7f3d71c08640 10 monclient: shutdown 2024-09-17T18:31:21.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.902+0000 7f3d71c08640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:31:21.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.902+0000 7f3d71c08640 1 -- 172.21.15.121:0/2621109499 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d6c0a33f0 msgr2=0x7f3d6c09e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:21.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.902+0000 7f3d71c08640 1 --2- 172.21.15.121:0/2621109499 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d6c0a33f0 0x7f3d6c09e550 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c0097a0 tx=0x7f3d5c02c1e0 comp rx=0 tx=0).stop 2024-09-17T18:31:21.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.902+0000 7f3d71c08640 1 -- 172.21.15.121:0/2621109499 shutdown_connections 2024-09-17T18:31:21.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.902+0000 7f3d71c08640 1 --2- 172.21.15.121:0/2621109499 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d6c0a33f0 0x7f3d6c09e550 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:21.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.902+0000 7f3d71c08640 1 -- 172.21.15.121:0/2621109499 >> 172.21.15.121:0/2621109499 conn(0x7f3d6c0a3b50 msgr2=0x7f3d6c0a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:31:21.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.902+0000 7f3d71c08640 1 -- 172.21.15.121:0/2621109499 shutdown_connections 2024-09-17T18:31:21.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.902+0000 7f3d71c08640 1 -- 172.21.15.121:0/2621109499 wait complete. 2024-09-17T18:31:21.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.903+0000 7f3d71c08640 10 monclient: build_initial_monmap 2024-09-17T18:31:21.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.903+0000 7f3d71c08640 10 monclient: monmap: 2024-09-17T18:31:21.911 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:31:21.911 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:31:21.911 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:31:21.911 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:31:21.911 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:31:21.911 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:21.912 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:31:21.912 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:31:21.912 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:31:21.912 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:21.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.903+0000 7f3d71c08640 1 librados: starting msgr at 2024-09-17T18:31:21.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.903+0000 7f3d71c08640 1 librados: starting objecter 2024-09-17T18:31:21.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.903+0000 7f3d71c08640 1 Processor -- start 2024-09-17T18:31:21.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.903+0000 7f3d71c08640 1 -- start start 2024-09-17T18:31:21.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.903+0000 7f3d71c08640 1 librados: setting wanted keys 2024-09-17T18:31:21.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.903+0000 7f3d71c08640 1 librados: calling monclient init 2024-09-17T18:31:21.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.903+0000 7f3d71c08640 10 monclient: init 2024-09-17T18:31:21.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d71c08640 10 monclient: _reopen_session rank -1 2024-09-17T18:31:21.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d71c08640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T18:31:21.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d71c08640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d6c0a33f0 0x7f3d6c133600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:21.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d71c08640 10 monclient(hunting): picked mon.noname-b con 0x7f3d6c0a33f0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:31:21.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d71c08640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d6c13a850 0x7f3d6c133b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:21.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d71c08640 10 monclient(hunting): picked mon.noname-c con 0x7f3d6c13a850 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:31:21.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d71c08640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d6c1345c0 0x7f3d6c120290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:21.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d71c08640 10 monclient(hunting): picked mon.noname-a con 0x7f3d6c1345c0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:31:21.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d71c08640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:21.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d71c08640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3d6c09f320 con 0x7f3d6c1345c0 2024-09-17T18:31:21.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d71c08640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:21.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d71c08640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3d6c09f1a0 con 0x7f3d6c0a33f0 2024-09-17T18:31:21.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d71c08640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:21.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d71c08640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3d6c0a22e0 con 0x7f3d6c13a850 2024-09-17T18:31:21.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d71c08640 10 monclient(hunting): _renew_subs 2024-09-17T18:31:21.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d71c08640 10 monclient(hunting): authenticate will time out at 1485.620292s 2024-09-17T18:31:21.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d6bfff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d6c1345c0 0x7f3d6c120290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:21.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d62ffd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d6c13a850 0x7f3d6c133b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:21.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d6bfff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d6c1345c0 0x7f3d6c120290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60288/0 (socket says 172.21.15.121:60288) 2024-09-17T18:31:21.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d6bfff640 1 -- 172.21.15.121:0/456875273 learned_addr learned my addr 172.21.15.121:0/456875273 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:31:21.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d6bfff640 10 monclient(hunting): get_auth_request con 0x7f3d6c1345c0 auth_method 0 2024-09-17T18:31:21.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:21.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d6bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:21.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:21.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d62ffd640 10 monclient(hunting): get_auth_request con 0x7f3d6c13a850 auth_method 0 2024-09-17T18:31:21.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d62ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:21.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d62ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:21.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:21.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:21.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:21.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:21.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d62ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:21.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d62ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:21.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d62ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:21.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d6b7fe640 1 --2- 172.21.15.121:0/456875273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d6c0a33f0 0x7f3d6c133600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:21.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d6b7fe640 10 monclient(hunting): get_auth_request con 0x7f3d6c0a33f0 auth_method 0 2024-09-17T18:31:21.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:21.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d6b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:21.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:21.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d62ffd640 10 monclient(hunting): handle_auth_done global_id 64313 payload 824 2024-09-17T18:31:21.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d62ffd640 1 -- 172.21.15.121:0/456875273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d6c0a33f0 msgr2=0x7f3d6c133600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:21.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d62ffd640 1 --2- 172.21.15.121:0/456875273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d6c0a33f0 0x7f3d6c133600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:21.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d62ffd640 1 -- 172.21.15.121:0/456875273 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d6c1345c0 msgr2=0x7f3d6c120290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:21.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d62ffd640 1 --2- 172.21.15.121:0/456875273 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d6c1345c0 0x7f3d6c120290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:21.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d62ffd640 10 monclient: _finish_hunting 0 2024-09-17T18:31:21.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d62ffd640 1 monclient: found mon.noname-c 2024-09-17T18:31:21.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d62ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:31:21.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d62ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:31:21.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d62ffd640 1 -- 172.21.15.121:0/456875273 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d6c121c90 con 0x7f3d6c13a850 2024-09-17T18:31:21.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d6bfff640 1 --2- 172.21.15.121:0/456875273 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d6c1345c0 0x7f3d6c120290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:31:21.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.904+0000 7f3d6b7fe640 1 --2- 172.21.15.121:0/456875273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d6c0a33f0 0x7f3d6c133600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T18:31:21.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.906+0000 7f3d62ffd640 1 -- 172.21.15.121:0/456875273 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d6c13a850 msgr2=0x7f3d6c133b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:21.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.906+0000 7f3d62ffd640 1 --2- 172.21.15.121:0/456875273 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d6c13a850 0x7f3d6c133b40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3d5400b410 tx=0x7f3d5400b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:21.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.906+0000 7f3d68ff9640 10 client.?.objecter ms_handle_connect 0x7f3d6c13a850 2024-09-17T18:31:21.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.906+0000 7f3d68ff9640 10 client.?.objecter resend_mon_ops 2024-09-17T18:31:21.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.907+0000 7f3d68ff9640 1 -- 172.21.15.121:0/456875273 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d54015020 con 0x7f3d6c13a850 2024-09-17T18:31:21.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.907+0000 7f3d68ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:21.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.907+0000 7f3d68ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:31:21.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.907+0000 7f3d68ff9640 10 monclient: dump: 2024-09-17T18:31:21.918 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:21.918 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:21.918 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:21.918 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:21.918 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:21.919 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:21.919 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:21.919 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:21.919 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:21.919 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:21.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.907+0000 7f3d68ff9640 10 monclient: _finish_auth 0 2024-09-17T18:31:21.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.907+0000 7f3d68ff9640 10 monclient: _check_auth_tickets 2024-09-17T18:31:21.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.907+0000 7f3d68ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:31:21.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.907+0000 7f3d71c08640 5 monclient: authenticate success, global_id 64313 2024-09-17T18:31:21.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.907+0000 7f3d68ff9640 1 -- 172.21.15.121:0/456875273 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d54011a60 con 0x7f3d6c13a850 2024-09-17T18:31:21.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.907+0000 7f3d71c08640 10 monclient: _renew_subs 2024-09-17T18:31:21.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.907+0000 7f3d71c08640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:31:21.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.907+0000 7f3d71c08640 1 -- 172.21.15.121:0/456875273 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d6c1227f0 con 0x7f3d6c13a850 2024-09-17T18:31:21.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.907+0000 7f3d68ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T18:31:21.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.907+0000 7f3d6a7fc640 4 set_mon_vals no callback set 2024-09-17T18:31:21.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.907+0000 7f3d71c08640 10 client.64313.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:31:21.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.907+0000 7f3d71c08640 10 monclient: _renew_subs 2024-09-17T18:31:21.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.907+0000 7f3d71c08640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:31:21.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.907+0000 7f3d71c08640 1 -- 172.21.15.121:0/456875273 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d6c123750 con 0x7f3d6c13a850 2024-09-17T18:31:21.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.907+0000 7f3d71c08640 1 librados: init done 2024-09-17T18:31:21.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.907+0000 7f3d68ff9640 1 -- 172.21.15.121:0/456875273 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d54011e50 con 0x7f3d6c13a850 2024-09-17T18:31:21.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.907+0000 7f3d68ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:21.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.907+0000 7f3d68ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:31:21.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.907+0000 7f3d68ff9640 10 monclient: dump: 2024-09-17T18:31:21.920 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:21.920 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:21.920 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:21.920 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:21.921 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:21.921 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:21.921 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:21.921 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:21.921 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:21.921 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:21.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.908+0000 7f3d68ff9640 1 -- 172.21.15.121:0/456875273 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f3d54018020 con 0x7f3d6c13a850 2024-09-17T18:31:21.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.908+0000 7f3d68ff9640 1 --2- 172.21.15.121:0/456875273 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3d4c040310 0x7f3d4c0427d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:21.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.908+0000 7f3d71c08640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:31:21.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.908+0000 7f3d71c08640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:31:21.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.908+0000 7f3d71c08640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:31:21.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.908+0000 7f3d68ff9640 1 -- 172.21.15.121:0/456875273 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(485..485 src has 247..485) ==== 6299+0+0 (secure 0 0 0) 0x7f3d54053960 con 0x7f3d6c13a850 2024-09-17T18:31:21.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.908+0000 7f3d68ff9640 10 client.64313.objecter ms_dispatch 0x7f3d6c000b90 osd_map(485..485 src has 247..485) 2024-09-17T18:31:21.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.908+0000 7f3d68ff9640 3 client.64313.objecter handle_osd_map got epochs [485,485] > 0 2024-09-17T18:31:21.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.908+0000 7f3d68ff9640 3 client.64313.objecter handle_osd_map decoding full epoch 485 2024-09-17T18:31:21.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.908+0000 7f3d71c08640 1 -- 172.21.15.121:0/456875273 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d6c09e550 con 0x7f3d6c13a850 2024-09-17T18:31:21.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.908+0000 7f3d68ff9640 20 client.64313.objecter dump_active .. 0 homeless 2024-09-17T18:31:21.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.908+0000 7f3d6b7fe640 1 --2- 172.21.15.121:0/456875273 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3d4c040310 0x7f3d4c0427d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:21.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.908+0000 7f3d6b7fe640 10 monclient: get_auth_request con 0x7f3d4c040310 auth_method 0 2024-09-17T18:31:21.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.911+0000 7f3d68ff9640 10 client.64313.objecter ms_handle_connect 0x7f3d4c040310 2024-09-17T18:31:21.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.911+0000 7f3d6b7fe640 1 --2- 172.21.15.121:0/456875273 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3d4c040310 0x7f3d4c0427d0 secure :-1 s=READY pgs=4035 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c036000 tx=0x7f3d5c030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:21.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.913+0000 7f3d68ff9640 1 -- 172.21.15.121:0/456875273 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3d5401f020 con 0x7f3d6c13a850 2024-09-17T18:31:21.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.913+0000 7f3d68ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:31:21.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:21.913+0000 7f3d68ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:31:22.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.155+0000 7f3d71c08640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:31:22.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.155+0000 7f3d71c08640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:31:22.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.155+0000 7f3d71c08640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:31:22.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.155+0000 7f3d71c08640 1 -- 172.21.15.121:0/456875273 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3d6c06ce60 con 0x7f3d6c13a850 2024-09-17T18:31:22.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.156+0000 7f3d68ff9640 1 -- 172.21.15.121:0/456875273 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f3d5400fcf0 con 0x7f3d6c13a850 2024-09-17T18:31:22.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.156+0000 7f3d68ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:31:22.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.156+0000 7f3d68ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:31:22.158 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":44,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:31:22.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.158+0000 7f3d71c08640 10 librados: watch_flush enter 2024-09-17T18:31:22.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.158+0000 7f3d71c08640 10 librados: watch_flush exit 2024-09-17T18:31:22.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.158+0000 7f3d71c08640 20 client.64313.objecter shutdown clearing up homeless session... 2024-09-17T18:31:22.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.158+0000 7f3d71c08640 10 client.64313.objecter successfully canceled tick 2024-09-17T18:31:22.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.158+0000 7f3d71c08640 1 -- 172.21.15.121:0/456875273 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3d4c040310 msgr2=0x7f3d4c0427d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:22.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.158+0000 7f3d71c08640 1 --2- 172.21.15.121:0/456875273 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3d4c040310 0x7f3d4c0427d0 secure :-1 s=READY pgs=4035 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c036000 tx=0x7f3d5c030000 comp rx=0 tx=0).stop 2024-09-17T18:31:22.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.158+0000 7f3d71c08640 10 monclient: shutdown 2024-09-17T18:31:22.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.158+0000 7f3d71c08640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:31:22.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.158+0000 7f3d71c08640 1 -- 172.21.15.121:0/456875273 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d6c13a850 msgr2=0x7f3d6c133b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:22.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.158+0000 7f3d71c08640 1 --2- 172.21.15.121:0/456875273 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d6c13a850 0x7f3d6c133b40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3d5400b410 tx=0x7f3d5400b8d0 comp rx=0 tx=0).stop 2024-09-17T18:31:22.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.158+0000 7f3d71c08640 1 -- 172.21.15.121:0/456875273 shutdown_connections 2024-09-17T18:31:22.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.158+0000 7f3d71c08640 1 --2- 172.21.15.121:0/456875273 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3d4c040310 0x7f3d4c0427d0 unknown :-1 s=CLOSED pgs=4035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:22.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.158+0000 7f3d71c08640 1 --2- 172.21.15.121:0/456875273 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d6c1345c0 0x7f3d6c120290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:22.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.158+0000 7f3d71c08640 1 --2- 172.21.15.121:0/456875273 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d6c13a850 0x7f3d6c133b40 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:22.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.158+0000 7f3d71c08640 1 --2- 172.21.15.121:0/456875273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d6c0a33f0 0x7f3d6c133600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:22.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.158+0000 7f3d71c08640 1 -- 172.21.15.121:0/456875273 >> 172.21.15.121:0/456875273 conn(0x7f3d6c0a3b50 msgr2=0x7f3d6c0a5770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:31:22.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.158+0000 7f3d71c08640 1 -- 172.21.15.121:0/456875273 shutdown_connections 2024-09-17T18:31:22.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.158+0000 7f3d71c08640 1 -- 172.21.15.121:0/456875273 wait complete. 2024-09-17T18:31:22.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.158+0000 7f3d71c08640 1 librados: shutdown 2024-09-17T18:31:22.170 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T18:31:22.171 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-17T18:31:22.171 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T18:31:22.171 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T18:31:22.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.308+0000 7f2705f9b640 10 monclient: get_monmap_and_config 2024-09-17T18:31:22.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.308+0000 7f2705f9b640 10 monclient: build_initial_monmap 2024-09-17T18:31:22.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.308+0000 7f2705f9b640 10 monclient: monmap: 2024-09-17T18:31:22.310 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:31:22.310 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:22.310 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:31:22.309265+0000 2024-09-17T18:31:22.310 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:31:22.309265+0000 2024-09-17T18:31:22.310 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:31:22.311 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:22.311 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:31:22.311 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:31:22.311 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:31:22.311 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:31:22.311 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:31:22.311 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:22.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.309+0000 7f2705f9b640 1 Processor -- start 2024-09-17T18:31:22.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.309+0000 7f2705f9b640 1 -- start start 2024-09-17T18:31:22.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.309+0000 7f2705f9b640 10 monclient: init 2024-09-17T18:31:22.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.309+0000 7f2705f9b640 10 monclient: _reopen_session rank -1 2024-09-17T18:31:22.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.309+0000 7f2705f9b640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-17T18:31:22.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.309+0000 7f2705f9b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f270009e550 addr v1:172.21.15.173:6789/0 2024-09-17T18:31:22.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.309+0000 7f2705f9b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f270009f040 addr v1:172.21.15.121:6789/0 2024-09-17T18:31:22.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.309+0000 7f2705f9b640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f270013bf70 0x7f270013e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:22.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.309+0000 7f2705f9b640 10 monclient(hunting): picked mon.noname-a con 0x7f270013bf70 addr v2:172.21.15.121:3300/0 2024-09-17T18:31:22.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.309+0000 7f2705f9b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27000a4460 con 0x7f270009f040 2024-09-17T18:31:22.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.309+0000 7f2705f9b640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27000a2b60 con 0x7f270009e550 2024-09-17T18:31:22.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.309+0000 7f2705f9b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:22.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.309+0000 7f2705f9b640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f27000a4950 con 0x7f270013bf70 2024-09-17T18:31:22.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.309+0000 7f2705f9b640 10 monclient(hunting): _renew_subs 2024-09-17T18:31:22.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.309+0000 7f2705f9b640 10 monclient(hunting): authenticate will time out at 1486.025979s 2024-09-17T18:31:22.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.310+0000 7f26fffff640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f270013bf70 0x7f270013e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:22.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.310+0000 7f26f6ffd640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f270009f040 0x7f270013a850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:33640/0 (socket says 172.21.15.121:33640) 2024-09-17T18:31:22.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.310+0000 7f26f6ffd640 1 -- 172.21.15.121:0/2590756581 learned_addr learned my addr 172.21.15.121:0/2590756581 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:31:22.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.310+0000 7f26fffff640 10 monclient(hunting): get_auth_request con 0x7f270013bf70 auth_method 0 2024-09-17T18:31:22.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.310+0000 7f26fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:22.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.310+0000 7f26fffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:22.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.310+0000 7f26fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:22.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.310+0000 7f26fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:22.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.310+0000 7f26fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:22.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.310+0000 7f26fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:22.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.310+0000 7f26fffff640 10 monclient(hunting): handle_auth_done global_id 44679 payload 274 2024-09-17T18:31:22.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.310+0000 7f26fffff640 1 -- 172.21.15.121:0/2590756581 >> v1:172.21.15.173:6789/0 conn(0x7f270009e550 legacy=0x7f270009e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:31:22.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.310+0000 7f26fffff640 1 -- 172.21.15.121:0/2590756581 >> v1:172.21.15.121:6789/0 conn(0x7f270009f040 legacy=0x7f270013a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:31:22.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.310+0000 7f26fffff640 10 monclient: _finish_hunting 0 2024-09-17T18:31:22.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.310+0000 7f26fffff640 1 monclient: found mon.noname-a 2024-09-17T18:31:22.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.310+0000 7f26fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:31:22.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.310+0000 7f26fffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:31:22.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.310+0000 7f26fffff640 1 -- 172.21.15.121:0/2590756581 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f270009f840 con 0x7f270013bf70 2024-09-17T18:31:22.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.313+0000 7f26fffff640 1 -- 172.21.15.121:0/2590756581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f270013bf70 msgr2=0x7f270013e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:22.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.313+0000 7f26fffff640 1 --2- 172.21.15.121:0/2590756581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f270013bf70 0x7f270013e360 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f270009e0a0 tx=0x7f26f802a460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9040fe3bf4e490db server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:22.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.313+0000 7f26feffd640 1 -- 172.21.15.121:0/2590756581 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f8007a30 con 0x7f270013bf70 2024-09-17T18:31:22.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.313+0000 7f26feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:22.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.313+0000 7f26feffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:31:22.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.313+0000 7f26feffd640 10 monclient: dump: 2024-09-17T18:31:22.323 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:22.323 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:22.323 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.323 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.323 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:22.323 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:22.323 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:22.323 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:22.323 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:22.323 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:22.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.313+0000 7f26feffd640 10 monclient: _finish_auth 0 2024-09-17T18:31:22.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.313+0000 7f26feffd640 10 monclient: _check_auth_tickets 2024-09-17T18:31:22.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.313+0000 7f26feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:31:22.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.313+0000 7f26feffd640 1 -- 172.21.15.121:0/2590756581 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26f8007bd0 con 0x7f270013bf70 2024-09-17T18:31:22.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.313+0000 7f2705f9b640 5 monclient: authenticate success, global_id 44679 2024-09-17T18:31:22.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.313+0000 7f2705f9b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:31:22.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.313+0000 7f26feffd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:31:22.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.313+0000 7f2705f9b640 10 monclient: get_monmap_and_config success 2024-09-17T18:31:22.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.313+0000 7f26feffd640 1 -- 172.21.15.121:0/2590756581 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f8031400 con 0x7f270013bf70 2024-09-17T18:31:22.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.313+0000 7f26feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:22.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.313+0000 7f2705f9b640 4 set_mon_vals no callback set 2024-09-17T18:31:22.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.313+0000 7f26feffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:31:22.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.313+0000 7f26feffd640 10 monclient: dump: 2024-09-17T18:31:22.324 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:22.325 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:22.325 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.325 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.325 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:22.325 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:22.325 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:22.325 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:22.325 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:22.325 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:22.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.313+0000 7f2705f9b640 10 monclient: shutdown 2024-09-17T18:31:22.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.313+0000 7f2705f9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:31:22.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.313+0000 7f2705f9b640 1 -- 172.21.15.121:0/2590756581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f270013bf70 msgr2=0x7f270013e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:22.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.313+0000 7f2705f9b640 1 --2- 172.21.15.121:0/2590756581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f270013bf70 0x7f270013e360 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f270009e0a0 tx=0x7f26f802a460 comp rx=0 tx=0).stop 2024-09-17T18:31:22.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.313+0000 7f2705f9b640 1 -- 172.21.15.121:0/2590756581 shutdown_connections 2024-09-17T18:31:22.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.313+0000 7f2705f9b640 1 --2- 172.21.15.121:0/2590756581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f270013bf70 0x7f270013e360 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:22.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.313+0000 7f2705f9b640 1 -- 172.21.15.121:0/2590756581 >> 172.21.15.121:0/2590756581 conn(0x7f27000a7830 msgr2=0x7f27000a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:31:22.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.314+0000 7f2705f9b640 1 -- 172.21.15.121:0/2590756581 shutdown_connections 2024-09-17T18:31:22.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.314+0000 7f2705f9b640 1 -- 172.21.15.121:0/2590756581 wait complete. 2024-09-17T18:31:22.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.314+0000 7f2705f9b640 10 monclient: build_initial_monmap 2024-09-17T18:31:22.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.314+0000 7f2705f9b640 10 monclient: monmap: 2024-09-17T18:31:22.326 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:31:22.326 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:31:22.326 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:31:22.326 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:31:22.326 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:31:22.326 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:22.326 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:31:22.326 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:31:22.326 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:31:22.326 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:22.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.314+0000 7f2705f9b640 1 librados: starting msgr at 2024-09-17T18:31:22.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.314+0000 7f2705f9b640 1 librados: starting objecter 2024-09-17T18:31:22.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.314+0000 7f2705f9b640 1 Processor -- start 2024-09-17T18:31:22.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f2705f9b640 1 -- start start 2024-09-17T18:31:22.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f2705f9b640 1 librados: setting wanted keys 2024-09-17T18:31:22.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f2705f9b640 1 librados: calling monclient init 2024-09-17T18:31:22.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f2705f9b640 10 monclient: init 2024-09-17T18:31:22.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f2705f9b640 10 monclient: _reopen_session rank -1 2024-09-17T18:31:22.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f2705f9b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T18:31:22.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f2705f9b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f270009e550 0x7f27000a1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:22.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f2705f9b640 10 monclient(hunting): picked mon.noname-c con 0x7f270009e550 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:31:22.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f2705f9b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f270009f040 0x7f2700137d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:22.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f2705f9b640 10 monclient(hunting): picked mon.noname-a con 0x7f270009f040 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:31:22.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f2705f9b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f270013bf70 0x7f27001cd510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:22.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f2705f9b640 10 monclient(hunting): picked mon.noname-b con 0x7f270013bf70 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:31:22.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f2705f9b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:22.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f2705f9b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f270009f490 con 0x7f270009f040 2024-09-17T18:31:22.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f2705f9b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:22.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f2705f9b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f270009dce0 con 0x7f270013bf70 2024-09-17T18:31:22.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f2705f9b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:22.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f2705f9b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f27000a62c0 con 0x7f270009e550 2024-09-17T18:31:22.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f2705f9b640 10 monclient(hunting): _renew_subs 2024-09-17T18:31:22.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f2705f9b640 10 monclient(hunting): authenticate will time out at 1486.031538s 2024-09-17T18:31:22.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f26f6ffd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f270009f040 0x7f2700137d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:22.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f26ff7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f270009e550 0x7f27000a1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:22.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f26f6ffd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f270009f040 0x7f2700137d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60312/0 (socket says 172.21.15.121:60312) 2024-09-17T18:31:22.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f26ff7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f270009e550 0x7f27000a1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:58108/0 (socket says 172.21.15.121:58108) 2024-09-17T18:31:22.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f26f6ffd640 1 -- 172.21.15.121:0/1726038177 learned_addr learned my addr 172.21.15.121:0/1726038177 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:31:22.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f26f6ffd640 10 monclient(hunting): get_auth_request con 0x7f270009f040 auth_method 0 2024-09-17T18:31:22.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f26f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:22.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f26f6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:22.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f26f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:22.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f26ff7fe640 10 monclient(hunting): get_auth_request con 0x7f270009e550 auth_method 0 2024-09-17T18:31:22.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f26ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:22.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f26ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:22.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f26ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:22.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f26ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:22.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f26ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:22.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f26ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:22.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f26f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:22.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f26f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:22.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f26fffff640 1 --2- 172.21.15.121:0/1726038177 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f270013bf70 0x7f27001cd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:22.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f26f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:22.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f26fffff640 10 monclient(hunting): get_auth_request con 0x7f270013bf70 auth_method 0 2024-09-17T18:31:22.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f26fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:22.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f26fffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:22.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f26fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:22.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f26fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:22.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f26fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:22.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f26fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:22.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f26ff7fe640 10 monclient(hunting): handle_auth_done global_id 64316 payload 824 2024-09-17T18:31:22.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f26ff7fe640 1 -- 172.21.15.121:0/1726038177 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f270013bf70 msgr2=0x7f27001cd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:22.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f26ff7fe640 1 --2- 172.21.15.121:0/1726038177 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f270013bf70 0x7f27001cd510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:22.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f26ff7fe640 1 -- 172.21.15.121:0/1726038177 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f270009f040 msgr2=0x7f2700137d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:22.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f26ff7fe640 1 --2- 172.21.15.121:0/1726038177 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f270009f040 0x7f2700137d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:22.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f26ff7fe640 10 monclient: _finish_hunting 0 2024-09-17T18:31:22.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f26ff7fe640 1 monclient: found mon.noname-c 2024-09-17T18:31:22.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f26ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:31:22.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f26ff7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:31:22.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f26ff7fe640 1 -- 172.21.15.121:0/1726038177 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2700138630 con 0x7f270009e550 2024-09-17T18:31:22.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.315+0000 7f26f6ffd640 1 --2- 172.21.15.121:0/1726038177 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f270009f040 0x7f2700137d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:31:22.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.318+0000 7f26ff7fe640 1 -- 172.21.15.121:0/1726038177 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f270009e550 msgr2=0x7f27000a1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:22.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.318+0000 7f26ff7fe640 1 --2- 172.21.15.121:0/1726038177 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f270009e550 0x7f27000a1a50 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f26f000c410 tx=0x7f26f000c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:22.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.318+0000 7f26fcff9640 10 client.?.objecter ms_handle_connect 0x7f270009e550 2024-09-17T18:31:22.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.318+0000 7f26fcff9640 10 client.?.objecter resend_mon_ops 2024-09-17T18:31:22.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.318+0000 7f26fcff9640 1 -- 172.21.15.121:0/1726038177 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f0017070 con 0x7f270009e550 2024-09-17T18:31:22.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.318+0000 7f26fcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:22.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.318+0000 7f26fcff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:31:22.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.318+0000 7f26fcff9640 10 monclient: dump: 2024-09-17T18:31:22.331 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:22.331 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:22.331 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.331 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.331 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:22.331 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:22.331 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:22.332 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:22.332 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:22.332 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:22.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.318+0000 7f26fcff9640 10 monclient: _finish_auth 0 2024-09-17T18:31:22.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.318+0000 7f26fcff9640 10 monclient: _check_auth_tickets 2024-09-17T18:31:22.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.318+0000 7f26fcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:31:22.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.318+0000 7f2705f9b640 5 monclient: authenticate success, global_id 64316 2024-09-17T18:31:22.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.318+0000 7f26fcff9640 1 -- 172.21.15.121:0/1726038177 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26f0005230 con 0x7f270009e550 2024-09-17T18:31:22.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.318+0000 7f2705f9b640 10 monclient: _renew_subs 2024-09-17T18:31:22.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.318+0000 7f2705f9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:31:22.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.318+0000 7f2705f9b640 1 -- 172.21.15.121:0/1726038177 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2700126700 con 0x7f270009e550 2024-09-17T18:31:22.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.318+0000 7f26fcff9640 10 monclient: handle_config config(0 keys) 2024-09-17T18:31:22.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.318+0000 7f26fcff9640 1 -- 172.21.15.121:0/1726038177 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f00117a0 con 0x7f270009e550 2024-09-17T18:31:22.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.318+0000 7f26fcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:22.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.318+0000 7f26fcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:31:22.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.318+0000 7f26fcff9640 10 monclient: dump: 2024-09-17T18:31:22.333 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:22.333 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:22.333 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.333 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.333 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:22.333 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:22.333 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:22.333 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:22.333 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:22.333 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:22.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.318+0000 7f26fe7fc640 4 set_mon_vals no callback set 2024-09-17T18:31:22.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.318+0000 7f2705f9b640 10 client.64316.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:31:22.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.318+0000 7f2705f9b640 10 monclient: _renew_subs 2024-09-17T18:31:22.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.318+0000 7f2705f9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:31:22.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.318+0000 7f2705f9b640 1 -- 172.21.15.121:0/1726038177 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2700126420 con 0x7f270009e550 2024-09-17T18:31:22.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.318+0000 7f2705f9b640 1 librados: init done 2024-09-17T18:31:22.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.319+0000 7f2705f9b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:31:22.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.319+0000 7f2705f9b640 1 --2- 172.21.15.121:0/1726038177 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f270002d440 0x7f270002d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:22.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.319+0000 7f2705f9b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:22.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.319+0000 7f2705f9b640 1 -- 172.21.15.121:0/1726038177 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f270009f610 con 0x7f270002d440 2024-09-17T18:31:22.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.319+0000 7f26f6ffd640 1 --2- 172.21.15.121:0/1726038177 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f270002d440 0x7f270002d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:22.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.319+0000 7f26f6ffd640 10 monclient: get_auth_request con 0x7f270002d440 auth_method 0 2024-09-17T18:31:22.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.319+0000 7f26f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:22.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.319+0000 7f26f6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:22.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.319+0000 7f26f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:22.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.319+0000 7f26fcff9640 1 -- 172.21.15.121:0/1726038177 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f26f001a070 con 0x7f270009e550 2024-09-17T18:31:22.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.319+0000 7f26f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:22.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.319+0000 7f26f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:22.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.319+0000 7f26f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:22.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.319+0000 7f26f6ffd640 10 monclient(hunting): handle_auth_done global_id 44685 payload 824 2024-09-17T18:31:22.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.319+0000 7f26f6ffd640 1 -- 172.21.15.121:0/1726038177 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f270009e240 con 0x7f270002d440 2024-09-17T18:31:22.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.319+0000 7f26fcff9640 1 --2- 172.21.15.121:0/1726038177 >> v2:172.21.15.173:6812/2423786037 conn(0x7f26e0040170 0x7f26e0042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:22.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.319+0000 7f26fcff9640 1 -- 172.21.15.121:0/1726038177 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(485..485 src has 247..485) ==== 6299+0+0 (secure 0 0 0) 0x7f26f0052d40 con 0x7f270009e550 2024-09-17T18:31:22.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.319+0000 7f26fcff9640 10 client.64316.objecter ms_dispatch 0x7f2700000b90 osd_map(485..485 src has 247..485) 2024-09-17T18:31:22.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.319+0000 7f26fcff9640 3 client.64316.objecter handle_osd_map got epochs [485,485] > 0 2024-09-17T18:31:22.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.319+0000 7f26fcff9640 3 client.64316.objecter handle_osd_map decoding full epoch 485 2024-09-17T18:31:22.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.319+0000 7f26fcff9640 20 client.64316.objecter dump_active .. 0 homeless 2024-09-17T18:31:22.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.320+0000 7f26fffff640 1 --2- 172.21.15.121:0/1726038177 >> v2:172.21.15.173:6812/2423786037 conn(0x7f26e0040170 0x7f26e0042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:22.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.320+0000 7f26fffff640 10 monclient: get_auth_request con 0x7f26e0040170 auth_method 0 2024-09-17T18:31:22.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.321+0000 7f26fcff9640 10 client.64316.objecter ms_handle_connect 0x7f270002d440 2024-09-17T18:31:22.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.321+0000 7f26f6ffd640 1 -- 172.21.15.121:0/1726038177 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f270002d440 msgr2=0x7f270002d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:22.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.321+0000 7f26fcff9640 10 client.64316.objecter resend_mon_ops 2024-09-17T18:31:22.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.321+0000 7f26f6ffd640 1 --2- 172.21.15.121:0/1726038177 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f270002d440 0x7f270002d820 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f2700121180 tx=0x7f26e8007770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:22.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.321+0000 7f26fcff9640 1 -- 172.21.15.121:0/1726038177 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e800fd30 con 0x7f270002d440 2024-09-17T18:31:22.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.321+0000 7f26fcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:22.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.321+0000 7f26fcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:31:22.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.321+0000 7f26fcff9640 10 monclient: dump: 2024-09-17T18:31:22.338 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:22.338 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:22.338 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.338 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.338 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:22.339 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:22.339 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:22.339 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:22.339 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:22.339 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:22.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.322+0000 7f26fcff9640 1 -- 172.21.15.121:0/1726038177 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f26e800fd30 con 0x7f270002d440 2024-09-17T18:31:22.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.322+0000 7f26fcff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:31:22.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.322+0000 7f26fcff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:31:22.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.322+0000 7f26fcff9640 1 -- 172.21.15.121:0/1726038177 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f270002d440 msgr2=0x7f270002d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:22.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.322+0000 7f26fcff9640 1 --2- 172.21.15.121:0/1726038177 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f270002d440 0x7f270002d820 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f2700121180 tx=0x7f26e8007770 comp rx=0 tx=0).stop 2024-09-17T18:31:22.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.322+0000 7f26fcff9640 1 -- 172.21.15.121:0/1726038177 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f270002d440 msgr2=0x7f270002d820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:31:22.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.322+0000 7f26fcff9640 1 --2- 172.21.15.121:0/1726038177 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f270002d440 0x7f270002d820 secure :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f2700121180 tx=0x7f26e8007770 comp rx=0 tx=0).stop 2024-09-17T18:31:22.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.322+0000 7f26fcff9640 10 client.64316.objecter ms_handle_connect 0x7f26e0040170 2024-09-17T18:31:22.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.322+0000 7f26fffff640 1 --2- 172.21.15.121:0/1726038177 >> v2:172.21.15.173:6812/2423786037 conn(0x7f26e0040170 0x7f26e0042630 secure :-1 s=READY pgs=4036 cs=0 l=1 rev1=1 crypto rx=0x7f2700074790 tx=0x7f26f802f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:22.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.328+0000 7f2705f9b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:31:22.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.328+0000 7f2705f9b640 1 --2- 172.21.15.121:0/1726038177 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f270002dd60 0x7f270002e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:22.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.328+0000 7f2705f9b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:22.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.328+0000 7f2705f9b640 1 -- 172.21.15.121:0/1726038177 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2700074900 con 0x7f270002dd60 2024-09-17T18:31:22.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.328+0000 7f26f6ffd640 1 --2- 172.21.15.121:0/1726038177 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f270002dd60 0x7f270002e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:22.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.328+0000 7f26f6ffd640 10 monclient: get_auth_request con 0x7f270002dd60 auth_method 0 2024-09-17T18:31:22.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.328+0000 7f26f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:22.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.328+0000 7f26f6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:22.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.328+0000 7f26f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:22.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.328+0000 7f26f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:22.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.328+0000 7f26f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:22.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.328+0000 7f26f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:22.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.328+0000 7f26f6ffd640 10 monclient(hunting): handle_auth_done global_id 44688 payload 824 2024-09-17T18:31:22.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.328+0000 7f26f6ffd640 1 -- 172.21.15.121:0/1726038177 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f270013a1f0 con 0x7f270002dd60 2024-09-17T18:31:22.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.330+0000 7f26f6ffd640 1 -- 172.21.15.121:0/1726038177 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f270002dd60 msgr2=0x7f270002e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:22.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.330+0000 7f26f6ffd640 1 --2- 172.21.15.121:0/1726038177 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f270002dd60 0x7f270002e170 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f26e8017000 tx=0x7f26e8004940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:22.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.330+0000 7f26fcff9640 10 client.64316.objecter ms_handle_connect 0x7f270002dd60 2024-09-17T18:31:22.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.330+0000 7f26fcff9640 10 client.64316.objecter resend_mon_ops 2024-09-17T18:31:22.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.330+0000 7f26fcff9640 1 -- 172.21.15.121:0/1726038177 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e8006030 con 0x7f270002dd60 2024-09-17T18:31:22.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.330+0000 7f26fcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:22.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.330+0000 7f26fcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:31:22.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.330+0000 7f26fcff9640 10 monclient: dump: 2024-09-17T18:31:22.341 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:22.341 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:22.341 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.341 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.342 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:22.342 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:22.342 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:22.342 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:22.342 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:22.342 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:22.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.331+0000 7f26fcff9640 1 -- 172.21.15.121:0/1726038177 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7f26e8006030 con 0x7f270002dd60 2024-09-17T18:31:22.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.331+0000 7f26fcff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:31:22.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.331+0000 7f26fcff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:31:22.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.331+0000 7f26fcff9640 1 -- 172.21.15.121:0/1726038177 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f270002dd60 msgr2=0x7f270002e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:22.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.331+0000 7f26fcff9640 1 --2- 172.21.15.121:0/1726038177 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f270002dd60 0x7f270002e170 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f26e8017000 tx=0x7f26e8004940 comp rx=0 tx=0).stop 2024-09-17T18:31:22.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.331+0000 7f26fcff9640 1 -- 172.21.15.121:0/1726038177 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f270002dd60 msgr2=0x7f270002e170 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:31:22.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.331+0000 7f26fcff9640 1 --2- 172.21.15.121:0/1726038177 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f270002dd60 0x7f270002e170 secure :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f26e8017000 tx=0x7f26e8004940 comp rx=0 tx=0).stop 2024-09-17T18:31:22.343 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:31:22.343 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:31:22.343 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:31:22.343 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "leader", 2024-09-17T18:31:22.343 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 44, 2024-09-17T18:31:22.343 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 507994, 2024-09-17T18:31:22.343 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:31:22.343 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:31:22.343 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:31:22.343 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:31:22.343 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:31:22.343 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 5, 2024-09-17T18:31:22.343 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:31:22.344 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:31:22.344 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:31:22.344 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:31:22.344 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:31:22.344 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:31:22.344 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:31:22.344 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:31:22.344 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:31:22.344 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:31:22.344 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:31:22.344 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:31:22.344 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:31:22.344 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:31:22.344 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:31:22.345 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:31:22.345 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:31:22.345 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:31:22.345 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:31:22.345 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:31:22.345 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:31:22.345 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:31:22.345 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:31:22.345 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:31:22.345 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:31:22.345 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:31:22.345 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:31:22.345 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:31:22.345 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:22.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:22.346 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:31:22.346 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:31:22.346 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:31:22.346 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:31:22.346 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:31:22.346 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:31:22.346 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:31:22.346 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:31:22.346 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:31:22.346 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:31:22.346 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:31:22.346 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:31:22.346 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:31:22.347 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:31:22.347 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:31:22.347 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:31:22.347 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:31:22.347 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:31:22.347 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:31:22.347 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:31:22.347 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:31:22.347 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:31:22.347 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:31:22.347 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:31:22.347 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:31:22.347 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:31:22.347 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:31:22.347 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:31:22.348 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:31:22.348 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:31:22.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:22.348 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:31:22.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:22.348 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:31:22.348 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:31:22.348 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:31:22.348 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:31:22.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:22.348 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:31:22.348 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:31:22.348 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:22.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:22.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:22.349 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:31:22.349 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:31:22.349 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:22.349 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:22.349 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:22.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:22.349 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:31:22.349 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:31:22.349 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:31:22.349 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:31:22.349 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:31:22.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:22.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:22.350 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:31:22.350 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:31:22.350 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:31:22.350 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:31:22.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:22.350 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:31:22.350 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:31:22.350 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:22.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:22.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:22.350 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:31:22.350 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:31:22.350 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:22.350 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:22.350 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:22.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:22.351 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:31:22.351 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:31:22.351 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:31:22.351 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:31:22.351 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:31:22.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:22.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:22.351 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:31:22.351 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:31:22.351 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:31:22.351 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:31:22.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:22.351 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:31:22.351 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:31:22.352 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:22.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:22.352 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:22.352 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:31:22.352 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:31:22.352 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:22.352 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:22.352 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:22.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:22.352 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:31:22.352 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:31:22.352 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:31:22.352 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:31:22.353 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:31:22.353 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:22.353 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:22.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:22.353 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:31:22.353 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:31:22.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:22.353 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:31:22.353 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:31:22.353 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:31:22.353 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:22.353 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:31:22.353 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:31:22.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:22.353 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:31:22.354 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:31:22.354 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 3 2024-09-17T18:31:22.354 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:22.354 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:31:22.354 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:31:22.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:22.354 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:31:22.354 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:31:22.354 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 4 2024-09-17T18:31:22.354 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:22.354 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:22.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:22.354 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:31:22.354 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:31:22.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.331+0000 7f2705f9b640 10 librados: watch_flush enter 2024-09-17T18:31:22.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.331+0000 7f2705f9b640 10 librados: watch_flush exit 2024-09-17T18:31:22.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.331+0000 7f2705f9b640 20 client.64316.objecter shutdown clearing up homeless session... 2024-09-17T18:31:22.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.331+0000 7f2705f9b640 10 client.64316.objecter successfully canceled tick 2024-09-17T18:31:22.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.331+0000 7f2705f9b640 1 -- 172.21.15.121:0/1726038177 >> v2:172.21.15.173:6812/2423786037 conn(0x7f26e0040170 msgr2=0x7f26e0042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:22.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.331+0000 7f2705f9b640 1 --2- 172.21.15.121:0/1726038177 >> v2:172.21.15.173:6812/2423786037 conn(0x7f26e0040170 0x7f26e0042630 secure :-1 s=READY pgs=4036 cs=0 l=1 rev1=1 crypto rx=0x7f2700074790 tx=0x7f26f802f000 comp rx=0 tx=0).stop 2024-09-17T18:31:22.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.331+0000 7f2705f9b640 10 monclient: shutdown 2024-09-17T18:31:22.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.331+0000 7f2705f9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:31:22.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.331+0000 7f2705f9b640 1 -- 172.21.15.121:0/1726038177 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f270009e550 msgr2=0x7f27000a1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:22.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.331+0000 7f2705f9b640 1 --2- 172.21.15.121:0/1726038177 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f270009e550 0x7f27000a1a50 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f26f000c410 tx=0x7f26f000c8d0 comp rx=0 tx=0).stop 2024-09-17T18:31:22.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.331+0000 7f26fffff640 1 -- 172.21.15.121:0/1726038177 reap_dead start 2024-09-17T18:31:22.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.331+0000 7f26fffff640 1 -- 172.21.15.121:0/1726038177 reap_dead start 2024-09-17T18:31:22.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.331+0000 7f2705f9b640 1 -- 172.21.15.121:0/1726038177 shutdown_connections 2024-09-17T18:31:22.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.331+0000 7f2705f9b640 1 -- 172.21.15.121:0/1726038177 >> 172.21.15.121:0/1726038177 conn(0x7f27000a7830 msgr2=0x7f27000a5b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:31:22.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.331+0000 7f2705f9b640 1 -- 172.21.15.121:0/1726038177 shutdown_connections 2024-09-17T18:31:22.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.331+0000 7f2705f9b640 1 -- 172.21.15.121:0/1726038177 wait complete. 2024-09-17T18:31:22.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.331+0000 7f2705f9b640 1 librados: shutdown 2024-09-17T18:31:22.358 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:31:22.358 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T18:31:22.358 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T18:31:22.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.455+0000 7f12a8594640 10 monclient: get_monmap_and_config 2024-09-17T18:31:22.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.455+0000 7f12a8594640 10 monclient: build_initial_monmap 2024-09-17T18:31:22.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.455+0000 7f12a8594640 10 monclient: monmap: 2024-09-17T18:31:22.457 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:31:22.457 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:22.457 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:31:22.456235+0000 2024-09-17T18:31:22.457 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:31:22.456235+0000 2024-09-17T18:31:22.457 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:31:22.458 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:22.458 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:31:22.458 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:31:22.458 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:31:22.458 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:31:22.458 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:31:22.458 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:22.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.456+0000 7f12a8594640 1 Processor -- start 2024-09-17T18:31:22.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.456+0000 7f12a8594640 1 -- start start 2024-09-17T18:31:22.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.456+0000 7f12a8594640 10 monclient: init 2024-09-17T18:31:22.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.456+0000 7f12a8594640 10 monclient: _reopen_session rank -1 2024-09-17T18:31:22.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.456+0000 7f12a8594640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-17T18:31:22.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.456+0000 7f12a8594640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f12a009e550 addr v1:172.21.15.173:6789/0 2024-09-17T18:31:22.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.456+0000 7f12a8594640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f12a009f040 0x7f12a013a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:22.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.456+0000 7f12a8594640 10 monclient(hunting): picked mon.noname-a con 0x7f12a009f040 addr v2:172.21.15.121:3300/0 2024-09-17T18:31:22.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.456+0000 7f12a8594640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f12a013ad90 0x7f12a013d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:22.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.456+0000 7f12a8594640 10 monclient(hunting): picked mon.noname-c con 0x7f12a013ad90 addr v2:172.21.15.173:3300/0 2024-09-17T18:31:22.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.456+0000 7f12a8594640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12a00a4460 con 0x7f12a009e550 2024-09-17T18:31:22.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.456+0000 7f12a8594640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:22.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.456+0000 7f12a8594640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f12a00a4950 con 0x7f12a009f040 2024-09-17T18:31:22.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.456+0000 7f12a8594640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:22.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.456+0000 7f12a8594640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f12a00a2b60 con 0x7f12a013ad90 2024-09-17T18:31:22.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.456+0000 7f12a8594640 10 monclient(hunting): _renew_subs 2024-09-17T18:31:22.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.456+0000 7f12a8594640 10 monclient(hunting): authenticate will time out at 1486.172889s 2024-09-17T18:31:22.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.456+0000 7f12a5b2f640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f12a009f040 0x7f12a013a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:22.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.457+0000 7f12a5b2f640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f12a009f040 0x7f12a013a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60326/0 (socket says 172.21.15.121:60326) 2024-09-17T18:31:22.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.457+0000 7f12a5b2f640 1 -- 172.21.15.121:0/2899742409 learned_addr learned my addr 172.21.15.121:0/2899742409 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:31:22.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.457+0000 7f12a5b2f640 10 monclient(hunting): get_auth_request con 0x7f12a009f040 auth_method 0 2024-09-17T18:31:22.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.457+0000 7f12a5b2f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:22.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.457+0000 7f12a5b2f640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:22.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.457+0000 7f12a5b2f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:22.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.457+0000 7f12a5b2f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:22.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.457+0000 7f12a5b2f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:22.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.457+0000 7f12a5b2f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:22.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.457+0000 7f12a5b2f640 10 monclient(hunting): handle_auth_done global_id 44691 payload 274 2024-09-17T18:31:22.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.457+0000 7f12a5b2f640 1 -- 172.21.15.121:0/2899742409 >> v2:172.21.15.173:3300/0 conn(0x7f12a013ad90 msgr2=0x7f12a013d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:31:22.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.457+0000 7f12a5b2f640 1 --2- 172.21.15.121:0/2899742409 >> v2:172.21.15.173:3300/0 conn(0x7f12a013ad90 0x7f12a013d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:22.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.457+0000 7f12a5b2f640 1 -- 172.21.15.121:0/2899742409 >> v1:172.21.15.173:6789/0 conn(0x7f12a009e550 legacy=0x7f12a009e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:31:22.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.457+0000 7f12a5b2f640 10 monclient: _finish_hunting 0 2024-09-17T18:31:22.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.457+0000 7f12a5b2f640 1 monclient: found mon.noname-a 2024-09-17T18:31:22.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.457+0000 7f12a5b2f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:31:22.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.457+0000 7f12a5b2f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:31:22.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.457+0000 7f12a5b2f640 1 -- 172.21.15.121:0/2899742409 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12a009f840 con 0x7f12a009f040 2024-09-17T18:31:22.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.459+0000 7f12a5b2f640 1 -- 172.21.15.121:0/2899742409 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12a009f040 msgr2=0x7f12a013a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:22.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.460+0000 7f12a5b2f640 1 --2- 172.21.15.121:0/2899742409 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12a009f040 0x7f12a013a850 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f12a0074790 tx=0x7f129402c770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7e204e98fdc55ed server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:22.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.460+0000 7f12a532e640 1 -- 172.21.15.121:0/2899742409 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1294038070 con 0x7f12a009f040 2024-09-17T18:31:22.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.460+0000 7f12a532e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:22.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.460+0000 7f12a532e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:31:22.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.460+0000 7f12a532e640 10 monclient: dump: 2024-09-17T18:31:22.469 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:22.469 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:22.469 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.470 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.470 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:22.470 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:22.470 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:22.470 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:22.470 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:22.470 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:22.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.460+0000 7f12a532e640 10 monclient: _finish_auth 0 2024-09-17T18:31:22.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.460+0000 7f12a532e640 10 monclient: _check_auth_tickets 2024-09-17T18:31:22.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.460+0000 7f12a532e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:31:22.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.460+0000 7f12a532e640 1 -- 172.21.15.121:0/2899742409 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1294031a00 con 0x7f12a009f040 2024-09-17T18:31:22.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.460+0000 7f12a8594640 5 monclient: authenticate success, global_id 44691 2024-09-17T18:31:22.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.460+0000 7f12a8594640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:31:22.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.460+0000 7f12a532e640 10 monclient: handle_config config(0 keys) 2024-09-17T18:31:22.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.460+0000 7f12a8594640 10 monclient: get_monmap_and_config success 2024-09-17T18:31:22.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.460+0000 7f12a532e640 1 -- 172.21.15.121:0/2899742409 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1294031dd0 con 0x7f12a009f040 2024-09-17T18:31:22.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.460+0000 7f12a532e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:22.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.460+0000 7f12a8594640 4 set_mon_vals no callback set 2024-09-17T18:31:22.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.460+0000 7f12a532e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:31:22.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.460+0000 7f12a532e640 10 monclient: dump: 2024-09-17T18:31:22.471 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:22.471 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:22.471 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.471 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.471 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:22.471 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:22.471 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:22.471 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:22.471 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:22.471 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:22.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.460+0000 7f12a8594640 10 monclient: shutdown 2024-09-17T18:31:22.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.460+0000 7f12a8594640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:31:22.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.460+0000 7f12a8594640 1 -- 172.21.15.121:0/2899742409 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12a009f040 msgr2=0x7f12a013a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:22.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.460+0000 7f12a8594640 1 --2- 172.21.15.121:0/2899742409 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12a009f040 0x7f12a013a850 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f12a0074790 tx=0x7f129402c770 comp rx=0 tx=0).stop 2024-09-17T18:31:22.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.460+0000 7f12a8594640 1 -- 172.21.15.121:0/2899742409 shutdown_connections 2024-09-17T18:31:22.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.460+0000 7f12a8594640 1 --2- 172.21.15.121:0/2899742409 >> v2:172.21.15.173:3300/0 conn(0x7f12a013ad90 0x7f12a013d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:22.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.460+0000 7f12a8594640 1 --2- 172.21.15.121:0/2899742409 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12a009f040 0x7f12a013a850 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:22.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.460+0000 7f12a8594640 1 -- 172.21.15.121:0/2899742409 >> 172.21.15.121:0/2899742409 conn(0x7f12a00a7830 msgr2=0x7f12a00a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:31:22.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.460+0000 7f12a8594640 1 -- 172.21.15.121:0/2899742409 shutdown_connections 2024-09-17T18:31:22.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.460+0000 7f12a8594640 1 -- 172.21.15.121:0/2899742409 wait complete. 2024-09-17T18:31:22.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.461+0000 7f12a8594640 10 monclient: build_initial_monmap 2024-09-17T18:31:22.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.461+0000 7f12a8594640 10 monclient: monmap: 2024-09-17T18:31:22.472 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:31:22.472 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:31:22.473 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:31:22.473 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:31:22.473 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:31:22.473 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:22.473 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:31:22.473 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:31:22.473 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:31:22.473 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:22.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.461+0000 7f12a8594640 1 librados: starting msgr at 2024-09-17T18:31:22.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.461+0000 7f12a8594640 1 librados: starting objecter 2024-09-17T18:31:22.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.461+0000 7f12a8594640 1 Processor -- start 2024-09-17T18:31:22.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.461+0000 7f12a8594640 1 -- start start 2024-09-17T18:31:22.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.461+0000 7f12a8594640 1 librados: setting wanted keys 2024-09-17T18:31:22.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.461+0000 7f12a8594640 1 librados: calling monclient init 2024-09-17T18:31:22.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.461+0000 7f12a8594640 10 monclient: init 2024-09-17T18:31:22.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a8594640 10 monclient: _reopen_session rank -1 2024-09-17T18:31:22.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a8594640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T18:31:22.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a8594640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f12a009e550 0x7f12a0120bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:22.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a8594640 10 monclient(hunting): picked mon.noname-b con 0x7f12a009e550 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:31:22.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a8594640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12a009f040 0x7f12a011eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:22.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a8594640 10 monclient(hunting): picked mon.noname-c con 0x7f12a009f040 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:31:22.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a8594640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12a013ad90 0x7f12a0136a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:22.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a8594640 10 monclient(hunting): picked mon.noname-a con 0x7f12a013ad90 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:31:22.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a8594640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:22.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a8594640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f12a009f490 con 0x7f12a013ad90 2024-09-17T18:31:22.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a8594640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:22.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a8594640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f12a009dce0 con 0x7f12a009e550 2024-09-17T18:31:22.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a8594640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:22.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a8594640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f12a00a6110 con 0x7f12a009f040 2024-09-17T18:31:22.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a8594640 10 monclient(hunting): _renew_subs 2024-09-17T18:31:22.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a8594640 10 monclient(hunting): authenticate will time out at 1486.178153s 2024-09-17T18:31:22.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a5b2f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12a009f040 0x7f12a011eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:22.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a6b31640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12a013ad90 0x7f12a0136a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:22.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a5b2f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12a009f040 0x7f12a011eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:58116/0 (socket says 172.21.15.121:58116) 2024-09-17T18:31:22.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a5b2f640 1 -- 172.21.15.121:0/4226743578 learned_addr learned my addr 172.21.15.121:0/4226743578 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:31:22.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a5b2f640 10 monclient(hunting): get_auth_request con 0x7f12a009f040 auth_method 0 2024-09-17T18:31:22.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a5b2f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:22.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a5b2f640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:22.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a5b2f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:22.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a6b31640 1 --2- 172.21.15.121:0/4226743578 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12a013ad90 0x7f12a0136a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60340/0 (socket says 172.21.15.121:60340) 2024-09-17T18:31:22.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a6b31640 10 monclient(hunting): get_auth_request con 0x7f12a013ad90 auth_method 0 2024-09-17T18:31:22.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a6b31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:22.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a6b31640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:22.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a6b31640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:22.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a6b31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:22.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a6b31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:22.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a6b31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:22.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a5b2f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:22.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a5b2f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:22.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a5b2f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:22.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a6330640 1 --2- 172.21.15.121:0/4226743578 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f12a009e550 0x7f12a0120bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:22.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a6330640 10 monclient(hunting): get_auth_request con 0x7f12a009e550 auth_method 0 2024-09-17T18:31:22.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a6330640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:22.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a6330640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:22.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a6330640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:22.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a6b31640 10 monclient(hunting): handle_auth_done global_id 44694 payload 824 2024-09-17T18:31:22.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a6b31640 1 -- 172.21.15.121:0/4226743578 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12a009f040 msgr2=0x7f12a011eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:22.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a6b31640 1 --2- 172.21.15.121:0/4226743578 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12a009f040 0x7f12a011eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:22.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a6b31640 1 -- 172.21.15.121:0/4226743578 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f12a009e550 msgr2=0x7f12a0120bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:22.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a6b31640 1 --2- 172.21.15.121:0/4226743578 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f12a009e550 0x7f12a0120bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:22.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a6b31640 10 monclient: _finish_hunting 0 2024-09-17T18:31:22.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a6b31640 1 monclient: found mon.noname-a 2024-09-17T18:31:22.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a6b31640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:31:22.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a6b31640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:31:22.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a6b31640 1 -- 172.21.15.121:0/4226743578 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12a00a5c00 con 0x7f12a013ad90 2024-09-17T18:31:22.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.462+0000 7f12a5b2f640 1 --2- 172.21.15.121:0/4226743578 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12a009f040 0x7f12a011eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:31:22.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.464+0000 7f12a6b31640 1 -- 172.21.15.121:0/4226743578 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12a013ad90 msgr2=0x7f12a0136a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:22.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.464+0000 7f12a6b31640 1 --2- 172.21.15.121:0/4226743578 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12a013ad90 0x7f12a0136a50 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f129800b440 tx=0x7f129800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:22.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.464+0000 7f129effd640 10 client.?.objecter ms_handle_connect 0x7f12a013ad90 2024-09-17T18:31:22.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.464+0000 7f129effd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:31:22.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.464+0000 7f129effd640 1 -- 172.21.15.121:0/4226743578 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1298015020 con 0x7f12a013ad90 2024-09-17T18:31:22.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.464+0000 7f129effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:22.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.464+0000 7f129effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:31:22.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.464+0000 7f129effd640 10 monclient: dump: 2024-09-17T18:31:22.477 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:22.478 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:22.478 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.478 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.478 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:22.478 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:22.478 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:22.478 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:22.478 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:22.478 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:22.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.464+0000 7f129effd640 10 monclient: _finish_auth 0 2024-09-17T18:31:22.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.464+0000 7f129effd640 10 monclient: _check_auth_tickets 2024-09-17T18:31:22.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.464+0000 7f129effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:31:22.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.464+0000 7f129effd640 1 -- 172.21.15.121:0/4226743578 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1298011a60 con 0x7f12a013ad90 2024-09-17T18:31:22.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.464+0000 7f12a8594640 5 monclient: authenticate success, global_id 44694 2024-09-17T18:31:22.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.464+0000 7f129effd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:31:22.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.464+0000 7f12a8594640 10 monclient: _renew_subs 2024-09-17T18:31:22.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.464+0000 7f129effd640 1 -- 172.21.15.121:0/4226743578 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1298011e50 con 0x7f12a013ad90 2024-09-17T18:31:22.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.464+0000 7f12a8594640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:31:22.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.464+0000 7f12a8594640 1 -- 172.21.15.121:0/4226743578 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12a0136f90 con 0x7f12a013ad90 2024-09-17T18:31:22.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.464+0000 7f129effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:22.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.465+0000 7f129effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:31:22.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.465+0000 7f129effd640 10 monclient: dump: 2024-09-17T18:31:22.479 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:22.479 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:22.479 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.479 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.479 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:22.479 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:22.479 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:22.479 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:22.479 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:22.479 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:22.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.464+0000 7f12a4b2d640 4 set_mon_vals no callback set 2024-09-17T18:31:22.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.465+0000 7f12a8594640 10 client.44694.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:31:22.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.465+0000 7f12a8594640 10 monclient: _renew_subs 2024-09-17T18:31:22.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.465+0000 7f12a8594640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:31:22.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.465+0000 7f12a8594640 1 -- 172.21.15.121:0/4226743578 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12a01c9080 con 0x7f12a013ad90 2024-09-17T18:31:22.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.465+0000 7f12a8594640 1 librados: init done 2024-09-17T18:31:22.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.465+0000 7f12a8594640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:31:22.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.465+0000 7f12a8594640 1 --2- 172.21.15.121:0/4226743578 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12a01c9b00 0x7f12a002d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:22.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.465+0000 7f12a8594640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:22.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.465+0000 7f12a8594640 1 -- 172.21.15.121:0/4226743578 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f12a009f610 con 0x7f12a01c9b00 2024-09-17T18:31:22.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.465+0000 7f12a6330640 1 --2- 172.21.15.121:0/4226743578 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12a01c9b00 0x7f12a002d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:22.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.465+0000 7f12a6330640 10 monclient: get_auth_request con 0x7f12a01c9b00 auth_method 0 2024-09-17T18:31:22.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.465+0000 7f12a6330640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:22.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.465+0000 7f12a6330640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:22.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.465+0000 7f12a6330640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:22.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.465+0000 7f129effd640 1 -- 172.21.15.121:0/4226743578 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f1298018020 con 0x7f12a013ad90 2024-09-17T18:31:22.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.465+0000 7f12a6330640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:22.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.465+0000 7f12a6330640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:22.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.465+0000 7f12a6330640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:22.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.466+0000 7f12a6330640 10 monclient(hunting): handle_auth_done global_id 64322 payload 824 2024-09-17T18:31:22.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.466+0000 7f12a6330640 1 -- 172.21.15.121:0/4226743578 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f12a01c9ee0 con 0x7f12a01c9b00 2024-09-17T18:31:22.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.466+0000 7f129effd640 1 --2- 172.21.15.121:0/4226743578 >> v2:172.21.15.173:6812/2423786037 conn(0x7f128c040340 0x7f128c042800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:22.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.466+0000 7f129effd640 1 -- 172.21.15.121:0/4226743578 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(485..485 src has 247..485) ==== 6299+0+0 (secure 0 0 0) 0x7f1298053970 con 0x7f12a013ad90 2024-09-17T18:31:22.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.466+0000 7f129effd640 10 client.44694.objecter ms_dispatch 0x7f12a0000b90 osd_map(485..485 src has 247..485) 2024-09-17T18:31:22.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.466+0000 7f129effd640 3 client.44694.objecter handle_osd_map got epochs [485,485] > 0 2024-09-17T18:31:22.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.466+0000 7f129effd640 3 client.44694.objecter handle_osd_map decoding full epoch 485 2024-09-17T18:31:22.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.466+0000 7f129effd640 20 client.44694.objecter dump_active .. 0 homeless 2024-09-17T18:31:22.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.466+0000 7f12a5b2f640 1 --2- 172.21.15.121:0/4226743578 >> v2:172.21.15.173:6812/2423786037 conn(0x7f128c040340 0x7f128c042800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:22.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.466+0000 7f12a5b2f640 10 monclient: get_auth_request con 0x7f128c040340 auth_method 0 2024-09-17T18:31:22.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.468+0000 7f129effd640 10 client.44694.objecter ms_handle_connect 0x7f12a01c9b00 2024-09-17T18:31:22.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.468+0000 7f129effd640 10 client.44694.objecter resend_mon_ops 2024-09-17T18:31:22.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.468+0000 7f12a6330640 1 -- 172.21.15.121:0/4226743578 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12a01c9b00 msgr2=0x7f12a002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:22.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.468+0000 7f12a6330640 1 --2- 172.21.15.121:0/4226743578 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12a01c9b00 0x7f12a002d440 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f129000e7c0 tx=0x7f129000ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:22.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.468+0000 7f129effd640 1 -- 172.21.15.121:0/4226743578 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1290003d90 con 0x7f12a01c9b00 2024-09-17T18:31:22.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.468+0000 7f129effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:22.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.468+0000 7f129effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:31:22.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.468+0000 7f129effd640 10 monclient: dump: 2024-09-17T18:31:22.484 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:22.484 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:22.484 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.484 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.485 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:22.485 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:22.485 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:22.485 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:22.485 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:22.485 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:22.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.468+0000 7f129effd640 1 -- 172.21.15.121:0/4226743578 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1290003d90 con 0x7f12a01c9b00 2024-09-17T18:31:22.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.468+0000 7f129effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:31:22.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.468+0000 7f129effd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:31:22.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.468+0000 7f129effd640 1 -- 172.21.15.121:0/4226743578 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12a01c9b00 msgr2=0x7f12a002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:22.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.468+0000 7f129effd640 1 --2- 172.21.15.121:0/4226743578 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12a01c9b00 0x7f12a002d440 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f129000e7c0 tx=0x7f129000ec80 comp rx=0 tx=0).stop 2024-09-17T18:31:22.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.469+0000 7f129effd640 1 -- 172.21.15.121:0/4226743578 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12a01c9b00 msgr2=0x7f12a002d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:31:22.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.469+0000 7f129effd640 1 --2- 172.21.15.121:0/4226743578 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12a01c9b00 0x7f12a002d440 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:22.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.469+0000 7f129effd640 10 client.44694.objecter ms_handle_connect 0x7f128c040340 2024-09-17T18:31:22.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.469+0000 7f12a5b2f640 1 --2- 172.21.15.121:0/4226743578 >> v2:172.21.15.173:6812/2423786037 conn(0x7f128c040340 0x7f128c042800 secure :-1 s=READY pgs=4037 cs=0 l=1 rev1=1 crypto rx=0x7f129402c7b0 tx=0x7f1294030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:22.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.475+0000 7f12a8594640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:31:22.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.475+0000 7f12a8594640 1 --2- 172.21.15.121:0/4226743578 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12a002d980 0x7f12a002dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:22.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.475+0000 7f12a8594640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:22.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.475+0000 7f12a8594640 1 -- 172.21.15.121:0/4226743578 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f12a0074870 con 0x7f12a002d980 2024-09-17T18:31:22.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.475+0000 7f12a6330640 1 --2- 172.21.15.121:0/4226743578 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12a002d980 0x7f12a002dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:22.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.475+0000 7f12a6330640 10 monclient: get_auth_request con 0x7f12a002d980 auth_method 0 2024-09-17T18:31:22.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.475+0000 7f12a6330640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:22.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.475+0000 7f12a6330640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:22.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.475+0000 7f12a6330640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:22.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.475+0000 7f12a6330640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:22.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.475+0000 7f12a6330640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:22.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.475+0000 7f12a6330640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:22.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.475+0000 7f12a6330640 10 monclient(hunting): handle_auth_done global_id 64325 payload 824 2024-09-17T18:31:22.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.475+0000 7f12a6330640 1 -- 172.21.15.121:0/4226743578 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f12a011fac0 con 0x7f12a002d980 2024-09-17T18:31:22.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.477+0000 7f129effd640 10 client.44694.objecter ms_handle_connect 0x7f12a002d980 2024-09-17T18:31:22.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.477+0000 7f129effd640 10 client.44694.objecter resend_mon_ops 2024-09-17T18:31:22.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.477+0000 7f12a6330640 1 -- 172.21.15.121:0/4226743578 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12a002d980 msgr2=0x7f12a002dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:22.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.477+0000 7f12a6330640 1 --2- 172.21.15.121:0/4226743578 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12a002d980 0x7f12a002dd90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1290016000 tx=0x7f12900013a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:22.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.477+0000 7f129effd640 1 -- 172.21.15.121:0/4226743578 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12900054f0 con 0x7f12a002d980 2024-09-17T18:31:22.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.477+0000 7f129effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:22.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.477+0000 7f129effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:31:22.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.477+0000 7f129effd640 10 monclient: dump: 2024-09-17T18:31:22.487 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:22.487 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:22.487 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.487 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.487 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:22.487 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:22.487 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:22.487 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:22.487 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:22.488 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:22.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.477+0000 7f129effd640 1 -- 172.21.15.121:0/4226743578 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f12900054f0 con 0x7f12a002d980 2024-09-17T18:31:22.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.477+0000 7f129effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:31:22.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.477+0000 7f129effd640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:31:22.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.478+0000 7f129effd640 1 -- 172.21.15.121:0/4226743578 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12a002d980 msgr2=0x7f12a002dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:22.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.478+0000 7f129effd640 1 --2- 172.21.15.121:0/4226743578 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12a002d980 0x7f12a002dd90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1290016000 tx=0x7f12900013a0 comp rx=0 tx=0).stop 2024-09-17T18:31:22.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.478+0000 7f129effd640 1 -- 172.21.15.121:0/4226743578 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12a002d980 msgr2=0x7f12a002dd90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:31:22.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.478+0000 7f129effd640 1 --2- 172.21.15.121:0/4226743578 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12a002d980 0x7f12a002dd90 secure :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1290016000 tx=0x7f12900013a0 comp rx=0 tx=0).stop 2024-09-17T18:31:22.488 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:31:22.488 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:31:22.488 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:31:22.488 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:31:22.489 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 44, 2024-09-17T18:31:22.489 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 261209, 2024-09-17T18:31:22.489 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:31:22.489 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:31:22.489 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:31:22.489 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:31:22.489 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:31:22.489 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 6, 2024-09-17T18:31:22.489 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:31:22.489 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:31:22.489 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:31:22.489 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:31:22.489 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:31:22.489 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:31:22.489 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:31:22.490 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:31:22.490 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:31:22.490 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:31:22.490 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:31:22.490 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:31:22.490 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:31:22.490 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:31:22.490 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:31:22.490 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:31:22.490 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:31:22.490 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:31:22.490 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:31:22.490 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:31:22.490 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:31:22.490 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:31:22.491 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:31:22.491 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:31:22.491 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:31:22.491 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:31:22.491 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:31:22.491 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:31:22.491 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:22.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:22.491 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:31:22.491 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:31:22.491 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:31:22.491 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:31:22.491 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:31:22.491 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:31:22.491 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:31:22.492 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:31:22.492 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:31:22.492 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:31:22.492 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:31:22.492 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:31:22.492 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:31:22.492 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:31:22.492 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:31:22.492 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:31:22.492 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:31:22.492 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:31:22.492 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:31:22.492 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:31:22.492 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:31:22.493 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:31:22.493 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:31:22.493 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:31:22.493 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:31:22.493 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:31:22.493 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:31:22.493 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:31:22.493 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:31:22.493 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:31:22.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:22.493 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:31:22.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:22.493 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:31:22.493 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:31:22.493 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:31:22.494 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:31:22.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:22.494 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:31:22.494 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:31:22.494 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:22.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:22.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:22.494 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:31:22.494 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:31:22.494 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:22.494 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:22.494 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:22.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:22.494 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:31:22.495 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:31:22.495 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:31:22.495 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:31:22.495 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:31:22.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:22.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:22.495 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:31:22.495 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:31:22.495 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:31:22.495 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:31:22.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:22.495 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:31:22.495 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:31:22.495 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:22.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:22.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:22.496 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:31:22.496 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:31:22.496 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:22.496 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:22.496 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:22.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:22.496 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:31:22.496 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:31:22.496 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:31:22.496 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:31:22.496 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:31:22.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:22.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:22.496 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:31:22.497 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:31:22.497 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:31:22.497 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:31:22.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:22.497 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:31:22.497 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:31:22.497 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:22.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:22.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:22.497 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:31:22.497 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:31:22.497 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:22.497 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:22.497 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:22.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:22.498 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:31:22.498 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:31:22.498 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:31:22.498 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:31:22.498 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:31:22.498 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:22.498 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:22.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:22.498 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:31:22.498 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:31:22.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:22.498 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:31:22.498 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:31:22.498 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:31:22.498 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:22.499 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:31:22.499 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:31:22.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:22.499 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:31:22.499 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:31:22.499 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 3 2024-09-17T18:31:22.499 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:22.499 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:31:22.499 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:31:22.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:22.499 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:31:22.499 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:31:22.499 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 4 2024-09-17T18:31:22.499 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:22.499 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:31:22.500 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": [ 2024-09-17T18:31:22.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:22.500 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:31:22.500 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:31:22.500 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:31:22.500 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:22.500 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:22.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:22.500 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:31:22.500 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:31:22.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.478+0000 7f12a8594640 10 librados: watch_flush enter 2024-09-17T18:31:22.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.478+0000 7f12a8594640 10 librados: watch_flush exit 2024-09-17T18:31:22.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.478+0000 7f12a8594640 20 client.44694.objecter shutdown clearing up homeless session... 2024-09-17T18:31:22.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.478+0000 7f12a8594640 10 client.44694.objecter successfully canceled tick 2024-09-17T18:31:22.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.478+0000 7f12a8594640 1 -- 172.21.15.121:0/4226743578 >> v2:172.21.15.173:6812/2423786037 conn(0x7f128c040340 msgr2=0x7f128c042800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:22.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.478+0000 7f12a8594640 1 --2- 172.21.15.121:0/4226743578 >> v2:172.21.15.173:6812/2423786037 conn(0x7f128c040340 0x7f128c042800 secure :-1 s=READY pgs=4037 cs=0 l=1 rev1=1 crypto rx=0x7f129402c7b0 tx=0x7f1294030000 comp rx=0 tx=0).stop 2024-09-17T18:31:22.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.478+0000 7f12a8594640 10 monclient: shutdown 2024-09-17T18:31:22.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.478+0000 7f12a8594640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:31:22.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.478+0000 7f12a8594640 1 -- 172.21.15.121:0/4226743578 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12a013ad90 msgr2=0x7f12a0136a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:22.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.478+0000 7f12a6b31640 1 -- 172.21.15.121:0/4226743578 reap_dead start 2024-09-17T18:31:22.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.478+0000 7f12a8594640 1 --2- 172.21.15.121:0/4226743578 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12a013ad90 0x7f12a0136a50 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f129800b440 tx=0x7f129800b900 comp rx=0 tx=0).stop 2024-09-17T18:31:22.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.478+0000 7f12a8594640 1 -- 172.21.15.121:0/4226743578 shutdown_connections 2024-09-17T18:31:22.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.478+0000 7f12a8594640 1 --2- 172.21.15.121:0/4226743578 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12a013ad90 0x7f12a0136a50 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:22.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.478+0000 7f12a8594640 1 -- 172.21.15.121:0/4226743578 >> 172.21.15.121:0/4226743578 conn(0x7f12a00a7830 msgr2=0x7f12a00a6550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:31:22.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.478+0000 7f12a8594640 1 -- 172.21.15.121:0/4226743578 shutdown_connections 2024-09-17T18:31:22.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.478+0000 7f12a8594640 1 -- 172.21.15.121:0/4226743578 wait complete. 2024-09-17T18:31:22.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.478+0000 7f12a8594640 1 librados: shutdown 2024-09-17T18:31:22.503 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:31:22.503 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T18:31:22.503 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T18:31:22.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.598+0000 7f4dd055d640 10 monclient: get_monmap_and_config 2024-09-17T18:31:22.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.598+0000 7f4dd055d640 10 monclient: build_initial_monmap 2024-09-17T18:31:22.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.598+0000 7f4dd055d640 10 monclient: monmap: 2024-09-17T18:31:22.601 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:31:22.601 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:22.601 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:31:22.599442+0000 2024-09-17T18:31:22.601 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:31:22.599442+0000 2024-09-17T18:31:22.601 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:31:22.601 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:22.601 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:31:22.601 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:31:22.601 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:31:22.601 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:31:22.601 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:31:22.601 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:22.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.599+0000 7f4dd055d640 1 Processor -- start 2024-09-17T18:31:22.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.599+0000 7f4dd055d640 1 -- start start 2024-09-17T18:31:22.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.599+0000 7f4dd055d640 10 monclient: init 2024-09-17T18:31:22.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dd055d640 10 monclient: _reopen_session rank -1 2024-09-17T18:31:22.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dd055d640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-17T18:31:22.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dd055d640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4dc8133320 0x7f4dc8074000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:22.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dd055d640 10 monclient(hunting): picked mon.noname-a con 0x7f4dc8133320 addr v2:172.21.15.121:3300/0 2024-09-17T18:31:22.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dd055d640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f4dc8077580 0x7f4dc8133700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:22.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dd055d640 10 monclient(hunting): picked mon.noname-c con 0x7f4dc8077580 addr v2:172.21.15.173:3300/0 2024-09-17T18:31:22.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dd055d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4dc80a3240 0x7f4dc80a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:22.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dd055d640 10 monclient(hunting): picked mon.noname-b con 0x7f4dc80a3240 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:31:22.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dd055d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:22.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dd055d640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f4dc8077340 con 0x7f4dc8133320 2024-09-17T18:31:22.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dd055d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:22.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dd055d640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f4dc806fee0 con 0x7f4dc8077580 2024-09-17T18:31:22.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dd055d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:22.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dd055d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4dc8073640 con 0x7f4dc80a3240 2024-09-17T18:31:22.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dd055d640 10 monclient(hunting): _renew_subs 2024-09-17T18:31:22.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dd055d640 10 monclient(hunting): authenticate will time out at 1486.316333s 2024-09-17T18:31:22.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dce2f9640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4dc8133320 0x7f4dc8074000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:22.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dce2f9640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4dc8133320 0x7f4dc8074000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60348/0 (socket says 172.21.15.121:60348) 2024-09-17T18:31:22.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dceafa640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4dc80a3240 0x7f4dc80a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:22.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dce2f9640 1 -- 172.21.15.121:0/3959884552 learned_addr learned my addr 172.21.15.121:0/3959884552 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:31:22.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dce2f9640 10 monclient(hunting): get_auth_request con 0x7f4dc8133320 auth_method 0 2024-09-17T18:31:22.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dce2f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:22.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dce2f9640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:22.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dce2f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:22.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dceafa640 10 monclient(hunting): get_auth_request con 0x7f4dc80a3240 auth_method 0 2024-09-17T18:31:22.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dceafa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:22.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dceafa640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:22.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dceafa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:22.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dce2f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:22.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dce2f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:22.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dce2f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:22.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dceafa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:22.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dceafa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:22.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dceafa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:22.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dce2f9640 10 monclient(hunting): handle_auth_done global_id 44697 payload 274 2024-09-17T18:31:22.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dce2f9640 1 -- 172.21.15.121:0/3959884552 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4dc80a3240 msgr2=0x7f4dc80a36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:31:22.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dce2f9640 1 --2- 172.21.15.121:0/3959884552 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4dc80a3240 0x7f4dc80a36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:22.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dce2f9640 1 -- 172.21.15.121:0/3959884552 >> v2:172.21.15.173:3300/0 conn(0x7f4dc8077580 msgr2=0x7f4dc8133700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:31:22.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dcdaf8640 1 --2- 172.21.15.121:0/3959884552 >> v2:172.21.15.173:3300/0 conn(0x7f4dc8077580 0x7f4dc8133700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:22.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dce2f9640 1 --2- 172.21.15.121:0/3959884552 >> v2:172.21.15.173:3300/0 conn(0x7f4dc8077580 0x7f4dc8133700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:22.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dce2f9640 10 monclient: _finish_hunting 0 2024-09-17T18:31:22.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dce2f9640 1 monclient: found mon.noname-a 2024-09-17T18:31:22.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dce2f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:31:22.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dce2f9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:31:22.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dce2f9640 1 -- 172.21.15.121:0/3959884552 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dc806d6a0 con 0x7f4dc8133320 2024-09-17T18:31:22.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dcdaf8640 10 monclient: get_auth_request con 0x7f4dc8077580 auth_method 0 2024-09-17T18:31:22.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dceafa640 1 --2- 172.21.15.121:0/3959884552 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4dc80a3240 0x7f4dc80a36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:31:22.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.600+0000 7f4dcdaf8640 1 --2- 172.21.15.121:0/3959884552 >> v2:172.21.15.173:3300/0 conn(0x7f4dc8077580 0x7f4dc8133700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T18:31:22.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.603+0000 7f4dce2f9640 1 -- 172.21.15.121:0/3959884552 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4dc8133320 msgr2=0x7f4dc8074000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:22.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.603+0000 7f4dce2f9640 1 --2- 172.21.15.121:0/3959884552 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4dc8133320 0x7f4dc8074000 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f4dc80731a0 tx=0x7f4dc402a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b070a2a7a3cd7d17 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:22.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.603+0000 7f4dcd2f7640 1 -- 172.21.15.121:0/3959884552 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc4036070 con 0x7f4dc8133320 2024-09-17T18:31:22.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.603+0000 7f4dcd2f7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:22.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.603+0000 7f4dcd2f7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:31:22.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.603+0000 7f4dcd2f7640 10 monclient: dump: 2024-09-17T18:31:22.614 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:22.614 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:22.615 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.615 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.615 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:22.615 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:22.615 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:22.615 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:22.615 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:22.615 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:22.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.603+0000 7f4dcd2f7640 10 monclient: _finish_auth 0 2024-09-17T18:31:22.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.603+0000 7f4dcd2f7640 10 monclient: _check_auth_tickets 2024-09-17T18:31:22.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.603+0000 7f4dcd2f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:31:22.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.603+0000 7f4dcd2f7640 1 -- 172.21.15.121:0/3959884552 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4dc4007800 con 0x7f4dc8133320 2024-09-17T18:31:22.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.603+0000 7f4dcd2f7640 10 monclient: handle_config config(0 keys) 2024-09-17T18:31:22.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.603+0000 7f4dcd2f7640 1 -- 172.21.15.121:0/3959884552 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc4007bd0 con 0x7f4dc8133320 2024-09-17T18:31:22.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.603+0000 7f4dd055d640 5 monclient: authenticate success, global_id 44697 2024-09-17T18:31:22.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.603+0000 7f4dd055d640 10 monclient: get_monmap_and_config success 2024-09-17T18:31:22.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.603+0000 7f4dd055d640 4 set_mon_vals no callback set 2024-09-17T18:31:22.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.603+0000 7f4dcd2f7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:22.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.603+0000 7f4dcd2f7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:31:22.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.603+0000 7f4dcd2f7640 10 monclient: dump: 2024-09-17T18:31:22.616 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:22.616 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:22.616 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.616 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.616 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:22.616 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:22.616 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:22.616 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:22.617 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:22.617 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:22.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.604+0000 7f4dd055d640 10 monclient: shutdown 2024-09-17T18:31:22.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.604+0000 7f4dd055d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:31:22.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.604+0000 7f4dd055d640 1 -- 172.21.15.121:0/3959884552 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4dc8133320 msgr2=0x7f4dc8074000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:22.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.604+0000 7f4dd055d640 1 --2- 172.21.15.121:0/3959884552 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4dc8133320 0x7f4dc8074000 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f4dc80731a0 tx=0x7f4dc402a570 comp rx=0 tx=0).stop 2024-09-17T18:31:22.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.604+0000 7f4dd055d640 1 -- 172.21.15.121:0/3959884552 shutdown_connections 2024-09-17T18:31:22.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.604+0000 7f4dd055d640 1 --2- 172.21.15.121:0/3959884552 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4dc80a3240 0x7f4dc80a36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:22.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.604+0000 7f4dd055d640 1 --2- 172.21.15.121:0/3959884552 >> v2:172.21.15.173:3300/0 conn(0x7f4dc8077580 0x7f4dc8133700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:22.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.604+0000 7f4dd055d640 1 --2- 172.21.15.121:0/3959884552 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4dc8133320 0x7f4dc8074000 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:22.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.604+0000 7f4dd055d640 1 -- 172.21.15.121:0/3959884552 >> 172.21.15.121:0/3959884552 conn(0x7f4dc80a49c0 msgr2=0x7f4dc80768d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:31:22.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.604+0000 7f4dd055d640 1 -- 172.21.15.121:0/3959884552 shutdown_connections 2024-09-17T18:31:22.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.604+0000 7f4dd055d640 1 -- 172.21.15.121:0/3959884552 wait complete. 2024-09-17T18:31:22.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.604+0000 7f4dd055d640 10 monclient: build_initial_monmap 2024-09-17T18:31:22.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.604+0000 7f4dd055d640 10 monclient: monmap: 2024-09-17T18:31:22.618 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:31:22.618 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:31:22.618 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:31:22.618 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:31:22.618 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:31:22.618 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:22.618 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:31:22.618 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:31:22.618 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:31:22.618 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:22.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.605+0000 7f4dd055d640 1 librados: starting msgr at 2024-09-17T18:31:22.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.605+0000 7f4dd055d640 1 librados: starting objecter 2024-09-17T18:31:22.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.605+0000 7f4dd055d640 1 Processor -- start 2024-09-17T18:31:22.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.605+0000 7f4dd055d640 1 -- start start 2024-09-17T18:31:22.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.605+0000 7f4dd055d640 1 librados: setting wanted keys 2024-09-17T18:31:22.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.605+0000 7f4dd055d640 1 librados: calling monclient init 2024-09-17T18:31:22.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.605+0000 7f4dd055d640 10 monclient: init 2024-09-17T18:31:22.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.605+0000 7f4dd055d640 10 monclient: _reopen_session rank -1 2024-09-17T18:31:22.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.605+0000 7f4dd055d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T18:31:22.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.605+0000 7f4dd055d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4dc8077580 0x7f4dc8131ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:22.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.605+0000 7f4dd055d640 10 monclient(hunting): picked mon.noname-c con 0x7f4dc8077580 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:31:22.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.605+0000 7f4dd055d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4dc80a3240 0x7f4dc81323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:22.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.605+0000 7f4dd055d640 10 monclient(hunting): picked mon.noname-b con 0x7f4dc80a3240 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:31:22.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.605+0000 7f4dd055d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4dc8133320 0x7f4dc81bdc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:22.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.605+0000 7f4dd055d640 10 monclient(hunting): picked mon.noname-a con 0x7f4dc8133320 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:31:22.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.605+0000 7f4dd055d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:22.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.605+0000 7f4dd055d640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc80a3d90 con 0x7f4dc8133320 2024-09-17T18:31:22.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.605+0000 7f4dd055d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:22.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.605+0000 7f4dd055d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc80a3c10 con 0x7f4dc80a3240 2024-09-17T18:31:22.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.605+0000 7f4dd055d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:22.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.605+0000 7f4dd055d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4dc80a3f10 con 0x7f4dc8077580 2024-09-17T18:31:22.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.605+0000 7f4dd055d640 10 monclient(hunting): _renew_subs 2024-09-17T18:31:22.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.605+0000 7f4dd055d640 10 monclient(hunting): authenticate will time out at 1486.321801s 2024-09-17T18:31:22.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.605+0000 7f4dce2f9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4dc8077580 0x7f4dc8131ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:22.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.605+0000 7f4dce2f9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4dc8077580 0x7f4dc8131ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:58154/0 (socket says 172.21.15.121:58154) 2024-09-17T18:31:22.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.605+0000 7f4dce2f9640 1 -- 172.21.15.121:0/3525875324 learned_addr learned my addr 172.21.15.121:0/3525875324 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:31:22.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.605+0000 7f4dce2f9640 10 monclient(hunting): get_auth_request con 0x7f4dc8077580 auth_method 0 2024-09-17T18:31:22.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.605+0000 7f4dce2f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:22.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.605+0000 7f4dce2f9640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:22.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.605+0000 7f4dce2f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:22.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.605+0000 7f4dceafa640 1 --2- 172.21.15.121:0/3525875324 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4dc8133320 0x7f4dc81bdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:22.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.605+0000 7f4dceafa640 10 monclient(hunting): get_auth_request con 0x7f4dc8133320 auth_method 0 2024-09-17T18:31:22.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.605+0000 7f4dceafa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:22.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.605+0000 7f4dceafa640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:22.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.605+0000 7f4dceafa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:22.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.605+0000 7f4dce2f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:22.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.605+0000 7f4dce2f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:22.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.605+0000 7f4dce2f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:22.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.605+0000 7f4dceafa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:22.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.605+0000 7f4dceafa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:22.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.605+0000 7f4dceafa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:22.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.606+0000 7f4dcdaf8640 1 --2- 172.21.15.121:0/3525875324 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4dc80a3240 0x7f4dc81323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:22.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.606+0000 7f4dcdaf8640 10 monclient(hunting): get_auth_request con 0x7f4dc80a3240 auth_method 0 2024-09-17T18:31:22.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.606+0000 7f4dcdaf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:22.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.606+0000 7f4dcdaf8640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:22.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.606+0000 7f4dcdaf8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:22.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.606+0000 7f4dce2f9640 10 monclient(hunting): handle_auth_done global_id 64331 payload 824 2024-09-17T18:31:22.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.606+0000 7f4dce2f9640 1 -- 172.21.15.121:0/3525875324 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4dc80a3240 msgr2=0x7f4dc81323e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:22.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.606+0000 7f4dce2f9640 1 --2- 172.21.15.121:0/3525875324 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4dc80a3240 0x7f4dc81323e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:22.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.606+0000 7f4dce2f9640 1 -- 172.21.15.121:0/3525875324 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4dc8133320 msgr2=0x7f4dc81bdc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:22.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.606+0000 7f4dce2f9640 1 --2- 172.21.15.121:0/3525875324 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4dc8133320 0x7f4dc81bdc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:22.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.606+0000 7f4dce2f9640 10 monclient: _finish_hunting 0 2024-09-17T18:31:22.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.606+0000 7f4dce2f9640 1 monclient: found mon.noname-c 2024-09-17T18:31:22.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.606+0000 7f4dce2f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:31:22.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.606+0000 7f4dce2f9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:31:22.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.606+0000 7f4dce2f9640 1 -- 172.21.15.121:0/3525875324 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dc8074c70 con 0x7f4dc8077580 2024-09-17T18:31:22.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.608+0000 7f4dce2f9640 1 -- 172.21.15.121:0/3525875324 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4dc8077580 msgr2=0x7f4dc8131ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:22.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.608+0000 7f4dce2f9640 1 --2- 172.21.15.121:0/3525875324 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4dc8077580 0x7f4dc8131ea0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4dc4034040 tx=0x7f4dc4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:22.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.608+0000 7f4dbeffd640 10 client.?.objecter ms_handle_connect 0x7f4dc8077580 2024-09-17T18:31:22.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.608+0000 7f4dbeffd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:31:22.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.608+0000 7f4dbeffd640 1 -- 172.21.15.121:0/3525875324 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc403f020 con 0x7f4dc8077580 2024-09-17T18:31:22.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.608+0000 7f4dbeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:22.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.608+0000 7f4dbeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:31:22.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.608+0000 7f4dbeffd640 10 monclient: dump: 2024-09-17T18:31:22.623 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:22.623 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:22.623 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.623 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.623 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:22.623 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:22.624 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:22.624 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:22.624 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:22.624 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:22.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.608+0000 7f4dbeffd640 10 monclient: _finish_auth 0 2024-09-17T18:31:22.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.608+0000 7f4dbeffd640 10 monclient: _check_auth_tickets 2024-09-17T18:31:22.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.608+0000 7f4dbeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:31:22.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.608+0000 7f4dd055d640 5 monclient: authenticate success, global_id 64331 2024-09-17T18:31:22.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.608+0000 7f4dbeffd640 1 -- 172.21.15.121:0/3525875324 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4dc4037040 con 0x7f4dc8077580 2024-09-17T18:31:22.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.608+0000 7f4dd055d640 10 monclient: _renew_subs 2024-09-17T18:31:22.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.608+0000 7f4dd055d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:31:22.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.608+0000 7f4dd055d640 1 -- 172.21.15.121:0/3525875324 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4dc811f9e0 con 0x7f4dc8077580 2024-09-17T18:31:22.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.608+0000 7f4dbeffd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:31:22.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.608+0000 7f4dbeffd640 1 -- 172.21.15.121:0/3525875324 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc4036070 con 0x7f4dc8077580 2024-09-17T18:31:22.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.608+0000 7f4dbeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:22.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.608+0000 7f4dbeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:31:22.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.608+0000 7f4dbeffd640 10 monclient: dump: 2024-09-17T18:31:22.625 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:22.625 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:22.625 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.625 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.625 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:22.625 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:22.625 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:22.625 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:22.625 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:22.625 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:22.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.608+0000 7f4dccaf6640 4 set_mon_vals no callback set 2024-09-17T18:31:22.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.608+0000 7f4dd055d640 10 client.64331.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:31:22.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.608+0000 7f4dd055d640 10 monclient: _renew_subs 2024-09-17T18:31:22.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.608+0000 7f4dd055d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:31:22.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.608+0000 7f4dd055d640 1 -- 172.21.15.121:0/3525875324 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4dc811f700 con 0x7f4dc8077580 2024-09-17T18:31:22.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.608+0000 7f4dd055d640 1 librados: init done 2024-09-17T18:31:22.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.609+0000 7f4dd055d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:31:22.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.609+0000 7f4dd055d640 1 --2- 172.21.15.121:0/3525875324 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4dc81be1b0 0x7f4dc81be590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:22.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.609+0000 7f4dd055d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:22.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.609+0000 7f4dd055d640 1 -- 172.21.15.121:0/3525875324 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc80a36d0 con 0x7f4dc81be1b0 2024-09-17T18:31:22.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.609+0000 7f4dbeffd640 1 -- 172.21.15.121:0/3525875324 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f4dc4042020 con 0x7f4dc8077580 2024-09-17T18:31:22.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.609+0000 7f4dcdaf8640 1 --2- 172.21.15.121:0/3525875324 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4dc81be1b0 0x7f4dc81be590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:22.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.609+0000 7f4dcdaf8640 10 monclient: get_auth_request con 0x7f4dc81be1b0 auth_method 0 2024-09-17T18:31:22.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.609+0000 7f4dcdaf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:22.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.609+0000 7f4dcdaf8640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:22.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.609+0000 7f4dcdaf8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:22.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.609+0000 7f4dbeffd640 1 --2- 172.21.15.121:0/3525875324 >> v2:172.21.15.173:6812/2423786037 conn(0x7f4dac040170 0x7f4dac042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:22.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.609+0000 7f4dcdaf8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:22.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.609+0000 7f4dcdaf8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:22.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.609+0000 7f4dcdaf8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:22.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.609+0000 7f4dbeffd640 1 -- 172.21.15.121:0/3525875324 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(485..485 src has 247..485) ==== 6299+0+0 (secure 0 0 0) 0x7f4dc4074060 con 0x7f4dc8077580 2024-09-17T18:31:22.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.609+0000 7f4dbeffd640 10 client.64331.objecter ms_dispatch 0x7f4dc8000b90 osd_map(485..485 src has 247..485) 2024-09-17T18:31:22.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.609+0000 7f4dbeffd640 3 client.64331.objecter handle_osd_map got epochs [485,485] > 0 2024-09-17T18:31:22.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.609+0000 7f4dbeffd640 3 client.64331.objecter handle_osd_map decoding full epoch 485 2024-09-17T18:31:22.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.609+0000 7f4dcdaf8640 10 monclient(hunting): handle_auth_done global_id 84124 payload 824 2024-09-17T18:31:22.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.609+0000 7f4dceafa640 1 --2- 172.21.15.121:0/3525875324 >> v2:172.21.15.173:6812/2423786037 conn(0x7f4dac040170 0x7f4dac042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:22.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.609+0000 7f4dcdaf8640 1 -- 172.21.15.121:0/3525875324 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4dc80008d0 con 0x7f4dc81be1b0 2024-09-17T18:31:22.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.609+0000 7f4dceafa640 10 monclient: get_auth_request con 0x7f4dac040170 auth_method 0 2024-09-17T18:31:22.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.609+0000 7f4dbeffd640 20 client.64331.objecter dump_active .. 0 homeless 2024-09-17T18:31:22.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.612+0000 7f4dcdaf8640 1 -- 172.21.15.121:0/3525875324 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4dc81be1b0 msgr2=0x7f4dc81be590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:22.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.612+0000 7f4dbeffd640 10 client.64331.objecter ms_handle_connect 0x7f4dc81be1b0 2024-09-17T18:31:22.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.612+0000 7f4dbeffd640 10 client.64331.objecter resend_mon_ops 2024-09-17T18:31:22.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.612+0000 7f4dcdaf8640 1 --2- 172.21.15.121:0/3525875324 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4dc81be1b0 0x7f4dc81be590 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f4dc8077990 tx=0x7f4db800f910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:22.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.612+0000 7f4dbeffd640 1 -- 172.21.15.121:0/3525875324 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db8018070 con 0x7f4dc81be1b0 2024-09-17T18:31:22.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.612+0000 7f4dbeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:22.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.612+0000 7f4dceafa640 1 --2- 172.21.15.121:0/3525875324 >> v2:172.21.15.173:6812/2423786037 conn(0x7f4dac040170 0x7f4dac042630 secure :-1 s=READY pgs=4038 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0009560 tx=0x7f4dc0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:22.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.612+0000 7f4dbeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:31:22.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.612+0000 7f4dbeffd640 10 monclient: dump: 2024-09-17T18:31:22.628 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:22.628 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:22.628 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.628 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.628 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:22.628 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:22.629 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:22.629 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:22.629 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:22.629 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:22.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.612+0000 7f4dbeffd640 10 client.64331.objecter ms_handle_connect 0x7f4dac040170 2024-09-17T18:31:22.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.612+0000 7f4dbeffd640 1 -- 172.21.15.121:0/3525875324 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4db8014b80 con 0x7f4dc81be1b0 2024-09-17T18:31:22.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.612+0000 7f4dbeffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:31:22.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.612+0000 7f4dbeffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:31:22.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.612+0000 7f4dbeffd640 1 -- 172.21.15.121:0/3525875324 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4dc81be1b0 msgr2=0x7f4dc81be590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:22.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.612+0000 7f4dbeffd640 1 --2- 172.21.15.121:0/3525875324 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4dc81be1b0 0x7f4dc81be590 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f4dc8077990 tx=0x7f4db800f910 comp rx=0 tx=0).stop 2024-09-17T18:31:22.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.612+0000 7f4dbeffd640 1 -- 172.21.15.121:0/3525875324 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4dc81be1b0 msgr2=0x7f4dc81be590 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:31:22.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.612+0000 7f4dbeffd640 1 --2- 172.21.15.121:0/3525875324 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4dc81be1b0 0x7f4dc81be590 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:22.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.618+0000 7f4dd055d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:31:22.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.618+0000 7f4dd055d640 1 --2- 172.21.15.121:0/3525875324 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4dc81bead0 0x7f4dc81b8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:22.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.618+0000 7f4dd055d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:22.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.618+0000 7f4dd055d640 1 -- 172.21.15.121:0/3525875324 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc811e1e0 con 0x7f4dc81bead0 2024-09-17T18:31:22.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.618+0000 7f4dcdaf8640 1 --2- 172.21.15.121:0/3525875324 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4dc81bead0 0x7f4dc81b8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:22.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.618+0000 7f4dcdaf8640 10 monclient: get_auth_request con 0x7f4dc81bead0 auth_method 0 2024-09-17T18:31:22.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.618+0000 7f4dcdaf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:22.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.618+0000 7f4dcdaf8640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:22.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.618+0000 7f4dcdaf8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:22.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.619+0000 7f4dcdaf8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:22.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.619+0000 7f4dcdaf8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:22.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.619+0000 7f4dcdaf8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:22.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.619+0000 7f4dcdaf8640 10 monclient(hunting): handle_auth_done global_id 84127 payload 824 2024-09-17T18:31:22.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.619+0000 7f4dcdaf8640 1 -- 172.21.15.121:0/3525875324 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4dc811e390 con 0x7f4dc81bead0 2024-09-17T18:31:22.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.621+0000 7f4dcdaf8640 1 -- 172.21.15.121:0/3525875324 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4dc81bead0 msgr2=0x7f4dc81b8c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:22.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.621+0000 7f4dcdaf8640 1 --2- 172.21.15.121:0/3525875324 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4dc81bead0 0x7f4dc81b8c30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4db801a000 tx=0x7f4db80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:22.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.621+0000 7f4dbeffd640 10 client.64331.objecter ms_handle_connect 0x7f4dc81bead0 2024-09-17T18:31:22.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.621+0000 7f4dbeffd640 10 client.64331.objecter resend_mon_ops 2024-09-17T18:31:22.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.621+0000 7f4dbeffd640 1 -- 172.21.15.121:0/3525875324 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db80135f0 con 0x7f4dc81bead0 2024-09-17T18:31:22.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.621+0000 7f4dbeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:22.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.621+0000 7f4dbeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:31:22.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.621+0000 7f4dbeffd640 10 monclient: dump: 2024-09-17T18:31:22.633 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:22.633 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:22.633 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.633 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.633 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:22.633 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:22.633 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:22.633 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:22.633 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:22.633 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:22.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.621+0000 7f4dbeffd640 1 -- 172.21.15.121:0/3525875324 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4551 (secure 0 0 0) 0x7f4db80135f0 con 0x7f4dc81bead0 2024-09-17T18:31:22.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.621+0000 7f4dbeffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:31:22.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.621+0000 7f4dbeffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:31:22.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.621+0000 7f4dbeffd640 1 -- 172.21.15.121:0/3525875324 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4dc81bead0 msgr2=0x7f4dc81b8c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:22.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.621+0000 7f4dbeffd640 1 --2- 172.21.15.121:0/3525875324 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4dc81bead0 0x7f4dc81b8c30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4db801a000 tx=0x7f4db80046c0 comp rx=0 tx=0).stop 2024-09-17T18:31:22.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.622+0000 7f4dbeffd640 1 -- 172.21.15.121:0/3525875324 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4dc81bead0 msgr2=0x7f4dc81b8c30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:31:22.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.622+0000 7f4dbeffd640 1 --2- 172.21.15.121:0/3525875324 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4dc81bead0 0x7f4dc81b8c30 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:22.634 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:31:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:31:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:31:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:31:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 44, 2024-09-17T18:31:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 7719, 2024-09-17T18:31:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:31:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:31:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:31:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:31:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:31:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 6, 2024-09-17T18:31:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:31:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:31:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:31:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:31:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:31:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:31:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:31:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:31:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:31:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:31:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:31:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:31:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:31:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:31:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:31:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:31:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:31:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:31:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:31:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:31:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:31:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:31:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:31:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:31:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:31:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:31:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:31:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:31:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:31:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:31:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:31:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:31:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:31:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:31:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:31:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:31:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:31:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:31:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:31:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:31:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:31:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:31:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:31:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:31:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:31:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:31:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:31:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:31:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:31:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:31:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:31:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:31:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:31:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:31:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:31:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:31:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:31:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:31:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:31:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:31:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:31:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:31:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:31:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:31:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:31:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:31:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:31:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:31:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:31:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:31:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:31:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:31:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:31:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:31:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:31:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:31:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:31:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:31:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:31:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:31:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:31:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:31:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:31:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:31:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:31:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:31:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:31:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:31:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:31:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:31:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:31:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:31:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:31:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:31:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:31:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:31:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:31:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:31:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:31:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:31:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:31:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:31:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:31:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:31:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:31:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:31:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:31:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:31:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:31:22.646 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:31:22.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.622+0000 7f4dd055d640 10 librados: watch_flush enter 2024-09-17T18:31:22.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.622+0000 7f4dd055d640 10 librados: watch_flush exit 2024-09-17T18:31:22.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.622+0000 7f4dd055d640 20 client.64331.objecter shutdown clearing up homeless session... 2024-09-17T18:31:22.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.622+0000 7f4dd055d640 10 client.64331.objecter successfully canceled tick 2024-09-17T18:31:22.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.622+0000 7f4dd055d640 1 -- 172.21.15.121:0/3525875324 >> v2:172.21.15.173:6812/2423786037 conn(0x7f4dac040170 msgr2=0x7f4dac042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:22.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.622+0000 7f4dd055d640 1 --2- 172.21.15.121:0/3525875324 >> v2:172.21.15.173:6812/2423786037 conn(0x7f4dac040170 0x7f4dac042630 secure :-1 s=READY pgs=4038 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0009560 tx=0x7f4dc0009190 comp rx=0 tx=0).stop 2024-09-17T18:31:22.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.622+0000 7f4dd055d640 10 monclient: shutdown 2024-09-17T18:31:22.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.622+0000 7f4dd055d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:31:22.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.622+0000 7f4dd055d640 1 -- 172.21.15.121:0/3525875324 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4dc8077580 msgr2=0x7f4dc8131ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:22.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.622+0000 7f4dd055d640 1 --2- 172.21.15.121:0/3525875324 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4dc8077580 0x7f4dc8131ea0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4dc4034040 tx=0x7f4dc4004a40 comp rx=0 tx=0).stop 2024-09-17T18:31:22.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.622+0000 7f4dceafa640 1 -- 172.21.15.121:0/3525875324 reap_dead start 2024-09-17T18:31:22.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.622+0000 7f4dceafa640 1 -- 172.21.15.121:0/3525875324 reap_dead start 2024-09-17T18:31:22.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.622+0000 7f4dd055d640 1 -- 172.21.15.121:0/3525875324 shutdown_connections 2024-09-17T18:31:22.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.622+0000 7f4dd055d640 1 -- 172.21.15.121:0/3525875324 >> 172.21.15.121:0/3525875324 conn(0x7f4dc80a49c0 msgr2=0x7f4dc809ce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:31:22.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.622+0000 7f4dd055d640 1 -- 172.21.15.121:0/3525875324 shutdown_connections 2024-09-17T18:31:22.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.622+0000 7f4dd055d640 1 -- 172.21.15.121:0/3525875324 wait complete. 2024-09-17T18:31:22.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.622+0000 7f4dd055d640 1 librados: shutdown 2024-09-17T18:31:22.649 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:31:22.649 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-17T18:31:22.649 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2024-09-17T18:31:22.649 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-17T18:31:22.650 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a store 2024-09-17T18:31:22.650 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a sync_force --yes-i-really-mean-it 2024-09-17T18:31:22.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.747+0000 7fe2b0ccd640 10 monclient: get_monmap_and_config 2024-09-17T18:31:22.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.747+0000 7fe2b0ccd640 10 monclient: build_initial_monmap 2024-09-17T18:31:22.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.747+0000 7fe2b0ccd640 10 monclient: monmap: 2024-09-17T18:31:22.749 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:31:22.749 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:22.749 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:31:22.748038+0000 2024-09-17T18:31:22.749 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:31:22.748038+0000 2024-09-17T18:31:22.749 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:31:22.749 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:22.749 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:31:22.749 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:31:22.750 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:31:22.750 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:31:22.750 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:31:22.750 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:22.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.747+0000 7fe2b0ccd640 1 Processor -- start 2024-09-17T18:31:22.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.748+0000 7fe2b0ccd640 1 -- start start 2024-09-17T18:31:22.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.748+0000 7fe2b0ccd640 10 monclient: init 2024-09-17T18:31:22.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.748+0000 7fe2b0ccd640 10 monclient: _reopen_session rank -1 2024-09-17T18:31:22.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.748+0000 7fe2b0ccd640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-17T18:31:22.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.748+0000 7fe2b0ccd640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe2ac133570 0x7fe2ac073f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:22.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.748+0000 7fe2b0ccd640 10 monclient(hunting): picked mon.noname-a con 0x7fe2ac133570 addr v2:172.21.15.121:3300/0 2024-09-17T18:31:22.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.748+0000 7fe2b0ccd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2ac133950 0x7fe2ac133d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:22.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.748+0000 7fe2b0ccd640 10 monclient(hunting): picked mon.noname-b con 0x7fe2ac133950 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:31:22.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.748+0000 7fe2b0ccd640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fe2ac0a3240 0x7fe2ac0a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:22.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.748+0000 7fe2b0ccd640 10 monclient(hunting): picked mon.noname-c con 0x7fe2ac0a3240 addr v2:172.21.15.173:3300/0 2024-09-17T18:31:22.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.748+0000 7fe2b0ccd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:22.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.748+0000 7fe2b0ccd640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fe2ac1331b0 con 0x7fe2ac133570 2024-09-17T18:31:22.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.748+0000 7fe2b0ccd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:22.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.748+0000 7fe2b0ccd640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fe2ac070180 con 0x7fe2ac0a3240 2024-09-17T18:31:22.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.748+0000 7fe2b0ccd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:22.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.748+0000 7fe2b0ccd640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe2ac0744c0 con 0x7fe2ac133950 2024-09-17T18:31:22.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.748+0000 7fe2b0ccd640 10 monclient(hunting): _renew_subs 2024-09-17T18:31:22.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.748+0000 7fe2b0ccd640 10 monclient(hunting): authenticate will time out at 1486.464775s 2024-09-17T18:31:22.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.748+0000 7fe2a9d9b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2ac133950 0x7fe2ac133d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:22.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.748+0000 7fe2aa59c640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe2ac133570 0x7fe2ac073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:22.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.748+0000 7fe2a9d9b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2ac133950 0x7fe2ac133d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:58158/0 (socket says 172.21.15.121:58158) 2024-09-17T18:31:22.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.748+0000 7fe2aa59c640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe2ac133570 0x7fe2ac073f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60358/0 (socket says 172.21.15.121:60358) 2024-09-17T18:31:22.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.748+0000 7fe2a9d9b640 1 -- 172.21.15.121:0/285331918 learned_addr learned my addr 172.21.15.121:0/285331918 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:31:22.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.748+0000 7fe2a9d9b640 10 monclient(hunting): get_auth_request con 0x7fe2ac133950 auth_method 0 2024-09-17T18:31:22.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.748+0000 7fe2a9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:22.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.748+0000 7fe2a9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:22.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.748+0000 7fe2a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:22.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.748+0000 7fe2aa59c640 10 monclient(hunting): get_auth_request con 0x7fe2ac133570 auth_method 0 2024-09-17T18:31:22.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.748+0000 7fe2aa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:22.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.748+0000 7fe2aa59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:22.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.748+0000 7fe2aa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:22.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.748+0000 7fe2a9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:22.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.748+0000 7fe2a9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:22.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.749+0000 7fe2a9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:22.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.749+0000 7fe2aa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:22.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.749+0000 7fe2aa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:22.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.749+0000 7fe2aa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:22.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.749+0000 7fe2aad9d640 1 --2- 172.21.15.121:0/285331918 >> v2:172.21.15.173:3300/0 conn(0x7fe2ac0a3240 0x7fe2ac0a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:22.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.749+0000 7fe2aad9d640 10 monclient(hunting): get_auth_request con 0x7fe2ac0a3240 auth_method 0 2024-09-17T18:31:22.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.749+0000 7fe2aad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:22.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.749+0000 7fe2aad9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:22.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.749+0000 7fe2aad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:22.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.749+0000 7fe2a9d9b640 10 monclient(hunting): handle_auth_done global_id 64334 payload 274 2024-09-17T18:31:22.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.749+0000 7fe2a9d9b640 1 -- 172.21.15.121:0/285331918 >> v2:172.21.15.173:3300/0 conn(0x7fe2ac0a3240 msgr2=0x7fe2ac0a36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:31:22.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.749+0000 7fe2a9d9b640 1 --2- 172.21.15.121:0/285331918 >> v2:172.21.15.173:3300/0 conn(0x7fe2ac0a3240 0x7fe2ac0a36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:22.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.749+0000 7fe2a9d9b640 1 -- 172.21.15.121:0/285331918 >> v2:172.21.15.121:3300/0 conn(0x7fe2ac133570 msgr2=0x7fe2ac073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:31:22.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.749+0000 7fe2a9d9b640 1 --2- 172.21.15.121:0/285331918 >> v2:172.21.15.121:3300/0 conn(0x7fe2ac133570 0x7fe2ac073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:22.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.749+0000 7fe2a9d9b640 10 monclient: _finish_hunting 0 2024-09-17T18:31:22.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.749+0000 7fe2a9d9b640 1 monclient: found mon.noname-b 2024-09-17T18:31:22.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.749+0000 7fe2a9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:31:22.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.749+0000 7fe2a9d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:31:22.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.749+0000 7fe2a9d9b640 1 -- 172.21.15.121:0/285331918 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2ac073010 con 0x7fe2ac133950 2024-09-17T18:31:22.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.749+0000 7fe2aad9d640 1 --2- 172.21.15.121:0/285331918 >> v2:172.21.15.173:3300/0 conn(0x7fe2ac0a3240 0x7fe2ac0a36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T18:31:22.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.749+0000 7fe2aa59c640 1 --2- 172.21.15.121:0/285331918 >> v2:172.21.15.121:3300/0 conn(0x7fe2ac133570 0x7fe2ac073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:31:22.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.751+0000 7fe2a9d9b640 1 -- 172.21.15.121:0/285331918 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2ac133950 msgr2=0x7fe2ac133d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:22.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.751+0000 7fe2a9d9b640 1 --2- 172.21.15.121:0/285331918 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2ac133950 0x7fe2ac133d50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac073cd0 tx=0x7fe2a002a0b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f72aadca4948abc server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:22.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.752+0000 7fe2a959a640 1 -- 172.21.15.121:0/285331918 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a0037070 con 0x7fe2ac133950 2024-09-17T18:31:22.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.752+0000 7fe2a959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:22.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.752+0000 7fe2a959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:31:22.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.752+0000 7fe2a959a640 10 monclient: dump: 2024-09-17T18:31:22.763 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:22.763 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:22.763 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.763 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.763 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:22.764 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:22.764 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:22.764 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:22.764 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:22.764 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:22.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.752+0000 7fe2a959a640 10 monclient: _finish_auth 0 2024-09-17T18:31:22.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.752+0000 7fe2a959a640 10 monclient: _check_auth_tickets 2024-09-17T18:31:22.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.752+0000 7fe2a959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:31:22.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.752+0000 7fe2a959a640 1 -- 172.21.15.121:0/285331918 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2a0032640 con 0x7fe2ac133950 2024-09-17T18:31:22.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.752+0000 7fe2b0ccd640 5 monclient: authenticate success, global_id 64334 2024-09-17T18:31:22.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.752+0000 7fe2b0ccd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:31:22.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.752+0000 7fe2a959a640 10 monclient: handle_config config(0 keys) 2024-09-17T18:31:22.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.752+0000 7fe2a959a640 1 -- 172.21.15.121:0/285331918 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a0032a10 con 0x7fe2ac133950 2024-09-17T18:31:22.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.752+0000 7fe2b0ccd640 10 monclient: get_monmap_and_config success 2024-09-17T18:31:22.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.752+0000 7fe2b0ccd640 4 set_mon_vals no callback set 2024-09-17T18:31:22.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.752+0000 7fe2a959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:22.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.752+0000 7fe2a959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:31:22.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.752+0000 7fe2a959a640 10 monclient: dump: 2024-09-17T18:31:22.765 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:22.765 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:22.765 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.765 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.765 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:22.765 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:22.765 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:22.765 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:22.765 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:22.765 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:22.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.752+0000 7fe2b0ccd640 10 monclient: shutdown 2024-09-17T18:31:22.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.752+0000 7fe2b0ccd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:31:22.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.752+0000 7fe2b0ccd640 1 -- 172.21.15.121:0/285331918 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2ac133950 msgr2=0x7fe2ac133d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:22.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.752+0000 7fe2b0ccd640 1 --2- 172.21.15.121:0/285331918 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2ac133950 0x7fe2ac133d50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac073cd0 tx=0x7fe2a002a0b0 comp rx=0 tx=0).stop 2024-09-17T18:31:22.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.752+0000 7fe2b0ccd640 1 -- 172.21.15.121:0/285331918 shutdown_connections 2024-09-17T18:31:22.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.752+0000 7fe2b0ccd640 1 --2- 172.21.15.121:0/285331918 >> v2:172.21.15.173:3300/0 conn(0x7fe2ac0a3240 0x7fe2ac0a36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:22.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.752+0000 7fe2b0ccd640 1 --2- 172.21.15.121:0/285331918 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2ac133950 0x7fe2ac133d50 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:22.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.752+0000 7fe2b0ccd640 1 --2- 172.21.15.121:0/285331918 >> v2:172.21.15.121:3300/0 conn(0x7fe2ac133570 0x7fe2ac073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:22.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.752+0000 7fe2b0ccd640 1 -- 172.21.15.121:0/285331918 >> 172.21.15.121:0/285331918 conn(0x7fe2ac0a4960 msgr2=0x7fe2ac0768d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:31:22.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.752+0000 7fe2b0ccd640 1 -- 172.21.15.121:0/285331918 shutdown_connections 2024-09-17T18:31:22.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.752+0000 7fe2b0ccd640 1 -- 172.21.15.121:0/285331918 wait complete. 2024-09-17T18:31:22.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.753+0000 7fe2b0ccd640 10 monclient: build_initial_monmap 2024-09-17T18:31:22.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.753+0000 7fe2b0ccd640 10 monclient: monmap: 2024-09-17T18:31:22.766 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:31:22.766 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:31:22.766 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:31:22.766 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:31:22.766 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:31:22.766 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:22.767 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:31:22.767 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:31:22.767 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:31:22.767 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:22.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.753+0000 7fe2b0ccd640 1 librados: starting msgr at 2024-09-17T18:31:22.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.753+0000 7fe2b0ccd640 1 librados: starting objecter 2024-09-17T18:31:22.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.753+0000 7fe2b0ccd640 1 Processor -- start 2024-09-17T18:31:22.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.753+0000 7fe2b0ccd640 1 -- start start 2024-09-17T18:31:22.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.753+0000 7fe2b0ccd640 1 librados: setting wanted keys 2024-09-17T18:31:22.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.753+0000 7fe2b0ccd640 1 librados: calling monclient init 2024-09-17T18:31:22.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.753+0000 7fe2b0ccd640 10 monclient: init 2024-09-17T18:31:22.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2b0ccd640 10 monclient: _reopen_session rank -1 2024-09-17T18:31:22.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2b0ccd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T18:31:22.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2b0ccd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2ac0a3240 0x7fe2ac0a0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:22.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2b0ccd640 10 monclient(hunting): picked mon.noname-a con 0x7fe2ac0a3240 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:31:22.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2b0ccd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2ac133570 0x7fe2ac070d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:22.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2b0ccd640 10 monclient(hunting): picked mon.noname-c con 0x7fe2ac133570 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:31:22.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2b0ccd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2ac133950 0x7fe2ac071240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:22.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2b0ccd640 10 monclient(hunting): picked mon.noname-b con 0x7fe2ac133950 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:31:22.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2b0ccd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:22.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2b0ccd640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe2ac06d700 con 0x7fe2ac0a3240 2024-09-17T18:31:22.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2b0ccd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:22.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2b0ccd640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe2ac0a3c10 con 0x7fe2ac133950 2024-09-17T18:31:22.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2b0ccd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:22.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2b0ccd640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe2ac0a3d90 con 0x7fe2ac133570 2024-09-17T18:31:22.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2b0ccd640 10 monclient(hunting): _renew_subs 2024-09-17T18:31:22.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2b0ccd640 10 monclient(hunting): authenticate will time out at 1486.470402s 2024-09-17T18:31:22.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2aa59c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2ac0a3240 0x7fe2ac0a0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:22.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2a9d9b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2ac133570 0x7fe2ac070d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:22.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2aa59c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2ac0a3240 0x7fe2ac0a0ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60362/0 (socket says 172.21.15.121:60362) 2024-09-17T18:31:22.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2aa59c640 1 -- 172.21.15.121:0/2503124908 learned_addr learned my addr 172.21.15.121:0/2503124908 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:31:22.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2aa59c640 10 monclient(hunting): get_auth_request con 0x7fe2ac0a3240 auth_method 0 2024-09-17T18:31:22.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2aa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:22.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2aa59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:22.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2aa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:22.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2a9d9b640 10 monclient(hunting): get_auth_request con 0x7fe2ac133570 auth_method 0 2024-09-17T18:31:22.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2a9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:22.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2a9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:22.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:22.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2a9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:22.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2a9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:22.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2a9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:22.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2aa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:22.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2aa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:22.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2aa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:22.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2aad9d640 1 --2- 172.21.15.121:0/2503124908 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2ac133950 0x7fe2ac071240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:22.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2a9d9b640 10 monclient(hunting): handle_auth_done global_id 64337 payload 824 2024-09-17T18:31:22.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2a9d9b640 1 -- 172.21.15.121:0/2503124908 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2ac133950 msgr2=0x7fe2ac071240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:22.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2a9d9b640 1 --2- 172.21.15.121:0/2503124908 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2ac133950 0x7fe2ac071240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:22.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2a9d9b640 1 -- 172.21.15.121:0/2503124908 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2ac0a3240 msgr2=0x7fe2ac0a0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:22.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2a9d9b640 1 --2- 172.21.15.121:0/2503124908 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2ac0a3240 0x7fe2ac0a0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:22.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2a9d9b640 10 monclient: _finish_hunting 0 2024-09-17T18:31:22.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2a9d9b640 1 monclient: found mon.noname-c 2024-09-17T18:31:22.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2a9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:31:22.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2a9d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:31:22.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2a9d9b640 1 -- 172.21.15.121:0/2503124908 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2ac09f2e0 con 0x7fe2ac133570 2024-09-17T18:31:22.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2aad9d640 10 monclient: get_auth_request con 0x7fe2ac133950 auth_method 0 2024-09-17T18:31:22.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2aad9d640 1 --2- 172.21.15.121:0/2503124908 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2ac133950 0x7fe2ac071240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T18:31:22.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.754+0000 7fe2aa59c640 1 --2- 172.21.15.121:0/2503124908 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2ac0a3240 0x7fe2ac0a0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:31:22.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.756+0000 7fe2a9d9b640 1 -- 172.21.15.121:0/2503124908 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2ac133570 msgr2=0x7fe2ac070d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:22.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.756+0000 7fe2a9d9b640 1 --2- 172.21.15.121:0/2503124908 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2ac133570 0x7fe2ac070d00 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe2a0031000 tx=0x7fe2a00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:22.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.756+0000 7fe29affd640 10 client.?.objecter ms_handle_connect 0x7fe2ac133570 2024-09-17T18:31:22.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.756+0000 7fe29affd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:31:22.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.757+0000 7fe29affd640 1 -- 172.21.15.121:0/2503124908 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a0041070 con 0x7fe2ac133570 2024-09-17T18:31:22.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.757+0000 7fe29affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:22.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.757+0000 7fe29affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:31:22.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.757+0000 7fe29affd640 10 monclient: dump: 2024-09-17T18:31:22.771 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:22.771 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:22.771 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.771 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.771 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:22.771 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:22.772 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:22.772 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:22.772 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:22.772 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:22.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.757+0000 7fe29affd640 10 monclient: _finish_auth 0 2024-09-17T18:31:22.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.757+0000 7fe29affd640 10 monclient: _check_auth_tickets 2024-09-17T18:31:22.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.757+0000 7fe29affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:31:22.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.757+0000 7fe29affd640 1 -- 172.21.15.121:0/2503124908 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2a003d830 con 0x7fe2ac133570 2024-09-17T18:31:22.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.757+0000 7fe2b0ccd640 5 monclient: authenticate success, global_id 64337 2024-09-17T18:31:22.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.757+0000 7fe2b0ccd640 10 monclient: _renew_subs 2024-09-17T18:31:22.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.757+0000 7fe2b0ccd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:31:22.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.757+0000 7fe2b0ccd640 1 -- 172.21.15.121:0/2503124908 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2ac09f5c0 con 0x7fe2ac133570 2024-09-17T18:31:22.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.757+0000 7fe29affd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:31:22.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.757+0000 7fe29affd640 1 -- 172.21.15.121:0/2503124908 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a0037040 con 0x7fe2ac133570 2024-09-17T18:31:22.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.757+0000 7fe29affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:22.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.757+0000 7fe29affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:31:22.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.757+0000 7fe29affd640 10 monclient: dump: 2024-09-17T18:31:22.773 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:22.773 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:22.773 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.773 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.773 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:22.773 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:22.773 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:22.773 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:22.773 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:22.773 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:22.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.757+0000 7fe2a8d99640 4 set_mon_vals no callback set 2024-09-17T18:31:22.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.757+0000 7fe2b0ccd640 10 client.64337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:31:22.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.757+0000 7fe2b0ccd640 10 monclient: _renew_subs 2024-09-17T18:31:22.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.757+0000 7fe2b0ccd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:31:22.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.757+0000 7fe2b0ccd640 1 -- 172.21.15.121:0/2503124908 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2ac09f000 con 0x7fe2ac133570 2024-09-17T18:31:22.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.757+0000 7fe2b0ccd640 1 librados: init done 2024-09-17T18:31:22.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.757+0000 7fe2b0ccd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:31:22.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.758+0000 7fe2b0ccd640 1 --2- 172.21.15.121:0/2503124908 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2ac0a1010 0x7fe2ac02e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:22.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.758+0000 7fe2b0ccd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:22.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.758+0000 7fe2b0ccd640 1 -- 172.21.15.121:0/2503124908 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe2ac0733a0 con 0x7fe2ac0a1010 2024-09-17T18:31:22.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.758+0000 7fe29affd640 1 -- 172.21.15.121:0/2503124908 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fe2a003dc20 con 0x7fe2ac133570 2024-09-17T18:31:22.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.758+0000 7fe2aa59c640 1 --2- 172.21.15.121:0/2503124908 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2ac0a1010 0x7fe2ac02e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:22.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.758+0000 7fe2aa59c640 10 monclient: get_auth_request con 0x7fe2ac0a1010 auth_method 0 2024-09-17T18:31:22.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.758+0000 7fe2aa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:22.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.758+0000 7fe2aa59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:22.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.758+0000 7fe2aa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:22.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.758+0000 7fe2aa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:22.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.758+0000 7fe2aa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:22.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.758+0000 7fe2aa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:22.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.758+0000 7fe2aa59c640 10 monclient(hunting): handle_auth_done global_id 44709 payload 824 2024-09-17T18:31:22.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.758+0000 7fe29affd640 1 --2- 172.21.15.121:0/2503124908 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe28c0401c0 0x7fe28c042680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:22.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.758+0000 7fe2aa59c640 1 -- 172.21.15.121:0/2503124908 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe2ac0a7230 con 0x7fe2ac0a1010 2024-09-17T18:31:22.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.758+0000 7fe29affd640 1 -- 172.21.15.121:0/2503124908 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(485..485 src has 247..485) ==== 6299+0+0 (secure 0 0 0) 0x7fe2a0073f00 con 0x7fe2ac133570 2024-09-17T18:31:22.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.758+0000 7fe29affd640 10 client.64337.objecter ms_dispatch 0x7fe2ac000b90 osd_map(485..485 src has 247..485) 2024-09-17T18:31:22.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.758+0000 7fe29affd640 3 client.64337.objecter handle_osd_map got epochs [485,485] > 0 2024-09-17T18:31:22.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.758+0000 7fe29affd640 3 client.64337.objecter handle_osd_map decoding full epoch 485 2024-09-17T18:31:22.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.758+0000 7fe29affd640 20 client.64337.objecter dump_active .. 0 homeless 2024-09-17T18:31:22.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.758+0000 7fe2aad9d640 1 --2- 172.21.15.121:0/2503124908 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe28c0401c0 0x7fe28c042680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:22.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.758+0000 7fe2aad9d640 10 monclient: get_auth_request con 0x7fe28c0401c0 auth_method 0 2024-09-17T18:31:22.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.760+0000 7fe2aa59c640 1 -- 172.21.15.121:0/2503124908 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2ac0a1010 msgr2=0x7fe2ac02e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:22.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.760+0000 7fe29affd640 10 client.64337.objecter ms_handle_connect 0x7fe2ac0a1010 2024-09-17T18:31:22.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.760+0000 7fe2aa59c640 1 --2- 172.21.15.121:0/2503124908 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2ac0a1010 0x7fe2ac02e5c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fe29c009ed0 tx=0x7fe29c00f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:22.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.760+0000 7fe29affd640 10 client.64337.objecter resend_mon_ops 2024-09-17T18:31:22.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.760+0000 7fe29affd640 1 -- 172.21.15.121:0/2503124908 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe29c014860 con 0x7fe2ac0a1010 2024-09-17T18:31:22.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.760+0000 7fe29affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:22.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.760+0000 7fe29affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:31:22.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.760+0000 7fe29affd640 10 monclient: dump: 2024-09-17T18:31:22.776 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:22.776 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:22.776 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.776 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.776 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:22.776 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:22.776 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:22.776 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:22.776 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:22.777 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:22.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.760+0000 7fe29affd640 1 -- 172.21.15.121:0/2503124908 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe29c014860 con 0x7fe2ac0a1010 2024-09-17T18:31:22.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.760+0000 7fe29affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:31:22.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.760+0000 7fe29affd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:31:22.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.760+0000 7fe29affd640 1 -- 172.21.15.121:0/2503124908 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2ac0a1010 msgr2=0x7fe2ac02e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:22.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.760+0000 7fe29affd640 1 --2- 172.21.15.121:0/2503124908 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2ac0a1010 0x7fe2ac02e5c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fe29c009ed0 tx=0x7fe29c00f610 comp rx=0 tx=0).stop 2024-09-17T18:31:22.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.760+0000 7fe29affd640 1 -- 172.21.15.121:0/2503124908 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2ac0a1010 msgr2=0x7fe2ac02e5c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:31:22.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.760+0000 7fe29affd640 1 --2- 172.21.15.121:0/2503124908 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2ac0a1010 0x7fe2ac02e5c0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:22.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.761+0000 7fe29affd640 10 client.64337.objecter ms_handle_connect 0x7fe28c0401c0 2024-09-17T18:31:22.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.761+0000 7fe2aad9d640 1 --2- 172.21.15.121:0/2503124908 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe28c0401c0 0x7fe28c042680 secure :-1 s=READY pgs=4039 cs=0 l=1 rev1=1 crypto rx=0x7fe2a40094c0 tx=0x7fe2a400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:22.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.767+0000 7fe2b0ccd640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-17T18:31:22.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.767+0000 7fe2b0ccd640 1 --2- 172.21.15.121:0/2503124908 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2ac071780 0x7fe2ac063d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:22.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.767+0000 7fe2b0ccd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:22.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.767+0000 7fe2b0ccd640 1 -- 172.21.15.121:0/2503124908 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe2ac071b60 con 0x7fe2ac071780 2024-09-17T18:31:22.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.768+0000 7fe2aa59c640 1 --2- 172.21.15.121:0/2503124908 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2ac071780 0x7fe2ac063d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:22.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.768+0000 7fe2aa59c640 10 monclient: get_auth_request con 0x7fe2ac071780 auth_method 0 2024-09-17T18:31:22.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.768+0000 7fe2aa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:22.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.768+0000 7fe2aa59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:22.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.768+0000 7fe2aa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:22.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.768+0000 7fe2aa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:22.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.768+0000 7fe2aa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:22.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.768+0000 7fe2aa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:22.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.768+0000 7fe2aa59c640 10 monclient(hunting): handle_auth_done global_id 44712 payload 824 2024-09-17T18:31:22.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.768+0000 7fe2aa59c640 1 -- 172.21.15.121:0/2503124908 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7fe2ac0a36d0 con 0x7fe2ac071780 2024-09-17T18:31:22.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.770+0000 7fe29affd640 10 client.64337.objecter ms_handle_connect 0x7fe2ac071780 2024-09-17T18:31:22.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.770+0000 7fe29affd640 10 client.64337.objecter resend_mon_ops 2024-09-17T18:31:22.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.770+0000 7fe2aa59c640 1 -- 172.21.15.121:0/2503124908 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2ac071780 msgr2=0x7fe2ac063d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:22.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.770+0000 7fe2aa59c640 1 --2- 172.21.15.121:0/2503124908 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2ac071780 0x7fe2ac063d50 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe29c01c000 tx=0x7fe29c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:22.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.770+0000 7fe29affd640 1 -- 172.21.15.121:0/2503124908 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe29c01db70 con 0x7fe2ac071780 2024-09-17T18:31:22.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.770+0000 7fe29affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:22.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.770+0000 7fe29affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:31:22.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.770+0000 7fe29affd640 10 monclient: dump: 2024-09-17T18:31:22.780 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:22.780 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:22.780 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.780 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:22.780 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:22.780 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:22.781 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:22.781 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:22.781 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:22.781 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:22.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.771+0000 7fe29affd640 1 -- 172.21.15.121:0/2503124908 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (secure 0 0 0) 0x7fe29c01db70 con 0x7fe2ac071780 2024-09-17T18:31:22.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.771+0000 7fe29affd640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-17T18:31:22.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.771+0000 7fe29affd640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:31:22.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.771+0000 7fe29affd640 1 -- 172.21.15.121:0/2503124908 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2ac071780 msgr2=0x7fe2ac063d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:22.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.771+0000 7fe29affd640 1 --2- 172.21.15.121:0/2503124908 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2ac071780 0x7fe2ac063d50 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe29c01c000 tx=0x7fe29c0046c0 comp rx=0 tx=0).stop 2024-09-17T18:31:22.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.771+0000 7fe29affd640 1 -- 172.21.15.121:0/2503124908 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2ac071780 msgr2=0x7fe2ac063d50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:31:22.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.771+0000 7fe29affd640 1 --2- 172.21.15.121:0/2503124908 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2ac071780 0x7fe2ac063d50 secure :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe29c01c000 tx=0x7fe29c0046c0 comp rx=0 tx=0).stop 2024-09-17T18:31:22.781 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:31:22.782 INFO:teuthology.orchestra.run.smithi121.stdout: "ret": 0, 2024-09-17T18:31:22.782 INFO:teuthology.orchestra.run.smithi121.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-09-17T18:31:22.782 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:31:22.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.771+0000 7fe2b0ccd640 10 librados: watch_flush enter 2024-09-17T18:31:22.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.771+0000 7fe2b0ccd640 10 librados: watch_flush exit 2024-09-17T18:31:22.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.771+0000 7fe2b0ccd640 20 client.64337.objecter shutdown clearing up homeless session... 2024-09-17T18:31:22.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.771+0000 7fe2b0ccd640 10 client.64337.objecter successfully canceled tick 2024-09-17T18:31:22.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.771+0000 7fe2b0ccd640 1 -- 172.21.15.121:0/2503124908 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe28c0401c0 msgr2=0x7fe28c042680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:22.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.771+0000 7fe2b0ccd640 1 --2- 172.21.15.121:0/2503124908 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe28c0401c0 0x7fe28c042680 secure :-1 s=READY pgs=4039 cs=0 l=1 rev1=1 crypto rx=0x7fe2a40094c0 tx=0x7fe2a400b040 comp rx=0 tx=0).stop 2024-09-17T18:31:22.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.771+0000 7fe2b0ccd640 10 monclient: shutdown 2024-09-17T18:31:22.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.771+0000 7fe2b0ccd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:31:22.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.771+0000 7fe2b0ccd640 1 -- 172.21.15.121:0/2503124908 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2ac133570 msgr2=0x7fe2ac070d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:22.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.771+0000 7fe2b0ccd640 1 --2- 172.21.15.121:0/2503124908 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2ac133570 0x7fe2ac070d00 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe2a0031000 tx=0x7fe2a00046c0 comp rx=0 tx=0).stop 2024-09-17T18:31:22.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.771+0000 7fe2aad9d640 1 -- 172.21.15.121:0/2503124908 reap_dead start 2024-09-17T18:31:22.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.771+0000 7fe2aad9d640 1 -- 172.21.15.121:0/2503124908 reap_dead start 2024-09-17T18:31:22.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.771+0000 7fe2b0ccd640 1 -- 172.21.15.121:0/2503124908 shutdown_connections 2024-09-17T18:31:22.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.771+0000 7fe2b0ccd640 1 -- 172.21.15.121:0/2503124908 >> 172.21.15.121:0/2503124908 conn(0x7fe2ac0a4960 msgr2=0x7fe2ac136680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:31:22.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.771+0000 7fe2b0ccd640 1 -- 172.21.15.121:0/2503124908 shutdown_connections 2024-09-17T18:31:22.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.771+0000 7fe2b0ccd640 1 -- 172.21.15.121:0/2503124908 wait complete. 2024-09-17T18:31:22.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:22.771+0000 7fe2b0ccd640 1 librados: shutdown 2024-09-17T18:31:22.784 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-17T18:31:22.785 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-17T18:31:22.785 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T18:31:22.800 INFO:tasks.ceph.mon.a:Stopped 2024-09-17T18:31:22.800 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-17T18:31:22.801 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-17T18:31:22.801 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-17T18:31:37.802 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-17T18:31:37.802 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-17T18:31:37.803 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-17T18:31:37.803 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:31:37.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.908+0000 7f7d8694b640 10 monclient: get_monmap_and_config 2024-09-17T18:31:37.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.908+0000 7f7d8694b640 10 monclient: build_initial_monmap 2024-09-17T18:31:37.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.908+0000 7f7d8694b640 10 monclient: monmap: 2024-09-17T18:31:37.911 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:31:37.911 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:37.911 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:31:37.909912+0000 2024-09-17T18:31:37.911 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:31:37.909912+0000 2024-09-17T18:31:37.911 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:31:37.911 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:37.911 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:31:37.911 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:31:37.912 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:31:37.912 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:31:37.912 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:31:37.912 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:37.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.909+0000 7f7d8694b640 1 Processor -- start 2024-09-17T18:31:37.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.909+0000 7f7d8694b640 1 -- start start 2024-09-17T18:31:37.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.909+0000 7f7d8694b640 10 monclient: init 2024-09-17T18:31:37.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.909+0000 7f7d8694b640 10 monclient: _reopen_session rank -1 2024-09-17T18:31:37.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.909+0000 7f7d8694b640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-17T18:31:37.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.909+0000 7f7d8694b640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f7d801348f0 0x7f7d800a0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:37.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.909+0000 7f7d8694b640 10 monclient(hunting): picked mon.noname-c con 0x7f7d801348f0 addr v2:172.21.15.173:3300/0 2024-09-17T18:31:37.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.909+0000 7f7d8694b640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7d800a0fb0 0x7f7d800a1a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:37.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.909+0000 7f7d8694b640 10 monclient(hunting): picked mon.noname-a con 0x7f7d800a0fb0 addr v2:172.21.15.121:3300/0 2024-09-17T18:31:37.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.909+0000 7f7d8694b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d800a1f90 0x7f7d800a23c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:37.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.909+0000 7f7d8694b640 10 monclient(hunting): picked mon.noname-b con 0x7f7d800a1f90 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:31:37.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.909+0000 7f7d8694b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:37.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.909+0000 7f7d8694b640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f7d8009ef80 con 0x7f7d800a0fb0 2024-09-17T18:31:37.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.909+0000 7f7d8694b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:37.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.909+0000 7f7d8694b640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f7d8009f5b0 con 0x7f7d801348f0 2024-09-17T18:31:37.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.909+0000 7f7d8694b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:37.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.909+0000 7f7d8694b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7d8009eca0 con 0x7f7d800a1f90 2024-09-17T18:31:37.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.909+0000 7f7d8694b640 10 monclient(hunting): _renew_subs 2024-09-17T18:31:37.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.909+0000 7f7d8694b640 10 monclient(hunting): authenticate will time out at 1501.626627s 2024-09-17T18:31:37.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.909+0000 7f7d7f7fe640 1 -- >> v2:172.21.15.121:3300/0 conn(0x7f7d800a0fb0 msgr2=0x7f7d800a1a50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:31:37.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.909+0000 7f7d7f7fe640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7d800a0fb0 0x7f7d800a1a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:31:37.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.909+0000 7f7d84ee8640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d800a1f90 0x7f7d800a23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:37.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.909+0000 7f7d84ee8640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d800a1f90 0x7f7d800a23c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:47576/0 (socket says 172.21.15.121:47576) 2024-09-17T18:31:37.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.909+0000 7f7d84ee8640 1 -- 172.21.15.121:0/2453085835 learned_addr learned my addr 172.21.15.121:0/2453085835 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:31:37.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.909+0000 7f7d84ee8640 10 monclient(hunting): get_auth_request con 0x7f7d800a1f90 auth_method 0 2024-09-17T18:31:37.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.909+0000 7f7d84ee8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:37.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.909+0000 7f7d84ee8640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:37.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.909+0000 7f7d84ee8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:37.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.909+0000 7f7d84ee8640 1 -- 172.21.15.121:0/2453085835 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d800a1f90 msgr2=0x7f7d800a23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T18:31:37.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.909+0000 7f7d84ee8640 1 -- 172.21.15.121:0/2453085835 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d800a1f90 msgr2=0x7f7d800a23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:31:37.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.909+0000 7f7d84ee8640 1 --2- 172.21.15.121:0/2453085835 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d800a1f90 0x7f7d800a23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:31:37.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.909+0000 7f7d84ee8640 1 --2- 172.21.15.121:0/2453085835 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d800a1f90 0x7f7d800a23c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:31:37.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.909+0000 7f7d7ffff640 1 --2- 172.21.15.121:0/2453085835 >> v2:172.21.15.173:3300/0 conn(0x7f7d801348f0 0x7f7d800a0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:37.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.909+0000 7f7d7ffff640 10 monclient(hunting): get_auth_request con 0x7f7d801348f0 auth_method 0 2024-09-17T18:31:37.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.910+0000 7f7d7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:37.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.910+0000 7f7d7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:37.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.910+0000 7f7d7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:37.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.910+0000 7f7d7ffff640 1 -- 172.21.15.121:0/2453085835 >> v2:172.21.15.173:3300/0 conn(0x7f7d801348f0 msgr2=0x7f7d800a0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:31:37.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.910+0000 7f7d7ffff640 1 -- 172.21.15.121:0/2453085835 >> v2:172.21.15.173:3300/0 conn(0x7f7d801348f0 msgr2=0x7f7d800a0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:31:37.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.910+0000 7f7d7ffff640 1 --2- 172.21.15.121:0/2453085835 >> v2:172.21.15.173:3300/0 conn(0x7f7d801348f0 0x7f7d800a0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:31:37.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:37.910+0000 7f7d7ffff640 1 --2- 172.21.15.121:0/2453085835 >> v2:172.21.15.173:3300/0 conn(0x7f7d801348f0 0x7f7d800a0a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:31:38.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:38.110+0000 7f7d7f7fe640 1 -- 172.21.15.121:0/2453085835 >> v2:172.21.15.121:3300/0 conn(0x7f7d800a0fb0 msgr2=0x7f7d800a1a50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:31:38.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:38.110+0000 7f7d7f7fe640 1 --2- 172.21.15.121:0/2453085835 >> v2:172.21.15.121:3300/0 conn(0x7f7d800a0fb0 0x7f7d800a1a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T18:31:38.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:38.110+0000 7f7d84ee8640 1 --2- 172.21.15.121:0/2453085835 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d800a1f90 0x7f7d800a23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:38.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:38.110+0000 7f7d84ee8640 10 monclient(hunting): get_auth_request con 0x7f7d800a1f90 auth_method 0 2024-09-17T18:31:38.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:38.110+0000 7f7d84ee8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:38.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:38.110+0000 7f7d84ee8640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:38.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:38.110+0000 7f7d84ee8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:31:38.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:38.110+0000 7f7d84ee8640 1 -- 172.21.15.121:0/2453085835 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d800a1f90 msgr2=0x7f7d800a23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:31:38.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:38.110+0000 7f7d84ee8640 1 -- 172.21.15.121:0/2453085835 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d800a1f90 msgr2=0x7f7d800a23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:31:38.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:38.110+0000 7f7d84ee8640 1 --2- 172.21.15.121:0/2453085835 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d800a1f90 0x7f7d800a23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:31:38.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:38.110+0000 7f7d84ee8640 1 --2- 172.21.15.121:0/2453085835 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d800a1f90 0x7f7d800a23c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T18:31:38.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:38.111+0000 7f7d7ffff640 1 --2- 172.21.15.121:0/2453085835 >> v2:172.21.15.173:3300/0 conn(0x7f7d801348f0 0x7f7d800a0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:38.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:38.111+0000 7f7d7ffff640 10 monclient(hunting): get_auth_request con 0x7f7d801348f0 auth_method 0 2024-09-17T18:31:38.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:38.111+0000 7f7d7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:38.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:38.111+0000 7f7d7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:38.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:38.111+0000 7f7d7ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:31:38.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:38.111+0000 7f7d7ffff640 1 -- 172.21.15.121:0/2453085835 >> v2:172.21.15.173:3300/0 conn(0x7f7d801348f0 msgr2=0x7f7d800a0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:31:38.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:38.111+0000 7f7d7ffff640 1 -- 172.21.15.121:0/2453085835 >> v2:172.21.15.173:3300/0 conn(0x7f7d801348f0 msgr2=0x7f7d800a0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:31:38.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:38.111+0000 7f7d7ffff640 1 --2- 172.21.15.121:0/2453085835 >> v2:172.21.15.173:3300/0 conn(0x7f7d801348f0 0x7f7d800a0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:31:38.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:38.112+0000 7f7d7ffff640 1 --2- 172.21.15.121:0/2453085835 >> v2:172.21.15.173:3300/0 conn(0x7f7d801348f0 0x7f7d800a0a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T18:31:38.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:38.511+0000 7f7d7f7fe640 1 -- 172.21.15.121:0/2453085835 >> v2:172.21.15.121:3300/0 conn(0x7f7d800a0fb0 msgr2=0x7f7d800a1a50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:31:38.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:38.511+0000 7f7d7f7fe640 1 --2- 172.21.15.121:0/2453085835 >> v2:172.21.15.121:3300/0 conn(0x7f7d800a0fb0 0x7f7d800a1a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T18:31:38.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:38.511+0000 7f7d84ee8640 1 --2- 172.21.15.121:0/2453085835 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d800a1f90 0x7f7d800a23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:38.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:38.511+0000 7f7d84ee8640 10 monclient(hunting): get_auth_request con 0x7f7d800a1f90 auth_method 0 2024-09-17T18:31:38.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:38.511+0000 7f7d84ee8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:38.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:38.511+0000 7f7d84ee8640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:38.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:38.511+0000 7f7d84ee8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:31:38.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:38.511+0000 7f7d84ee8640 1 -- 172.21.15.121:0/2453085835 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d800a1f90 msgr2=0x7f7d800a23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:31:38.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:38.511+0000 7f7d84ee8640 1 -- 172.21.15.121:0/2453085835 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d800a1f90 msgr2=0x7f7d800a23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:31:38.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:38.511+0000 7f7d84ee8640 1 --2- 172.21.15.121:0/2453085835 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d800a1f90 0x7f7d800a23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:31:38.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:38.511+0000 7f7d84ee8640 1 --2- 172.21.15.121:0/2453085835 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d800a1f90 0x7f7d800a23c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T18:31:38.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:38.514+0000 7f7d7ffff640 1 --2- 172.21.15.121:0/2453085835 >> v2:172.21.15.173:3300/0 conn(0x7f7d801348f0 0x7f7d800a0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:38.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:38.514+0000 7f7d7ffff640 10 monclient(hunting): get_auth_request con 0x7f7d801348f0 auth_method 0 2024-09-17T18:31:38.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:38.514+0000 7f7d7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:38.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:38.514+0000 7f7d7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:38.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:38.514+0000 7f7d7ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:31:38.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:38.514+0000 7f7d7ffff640 1 -- 172.21.15.121:0/2453085835 >> v2:172.21.15.173:3300/0 conn(0x7f7d801348f0 msgr2=0x7f7d800a0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:31:38.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:38.514+0000 7f7d7ffff640 1 -- 172.21.15.121:0/2453085835 >> v2:172.21.15.173:3300/0 conn(0x7f7d801348f0 msgr2=0x7f7d800a0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:31:38.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:38.514+0000 7f7d7ffff640 1 --2- 172.21.15.121:0/2453085835 >> v2:172.21.15.173:3300/0 conn(0x7f7d801348f0 0x7f7d800a0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:31:38.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:38.514+0000 7f7d7ffff640 1 --2- 172.21.15.121:0/2453085835 >> v2:172.21.15.173:3300/0 conn(0x7f7d801348f0 0x7f7d800a0a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T18:31:39.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:39.312+0000 7f7d84ee8640 1 --2- 172.21.15.121:0/2453085835 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d800a1f90 0x7f7d800a23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:39.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:39.312+0000 7f7d84ee8640 10 monclient(hunting): get_auth_request con 0x7f7d800a1f90 auth_method 0 2024-09-17T18:31:39.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:39.312+0000 7f7d84ee8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:39.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:39.312+0000 7f7d84ee8640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:39.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:39.312+0000 7f7d84ee8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:31:39.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:39.312+0000 7f7d84ee8640 1 -- 172.21.15.121:0/2453085835 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d800a1f90 msgr2=0x7f7d800a23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:31:39.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:39.312+0000 7f7d84ee8640 1 -- 172.21.15.121:0/2453085835 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d800a1f90 msgr2=0x7f7d800a23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:31:39.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:39.312+0000 7f7d84ee8640 1 --2- 172.21.15.121:0/2453085835 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d800a1f90 0x7f7d800a23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:31:39.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:39.312+0000 7f7d84ee8640 1 --2- 172.21.15.121:0/2453085835 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d800a1f90 0x7f7d800a23c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T18:31:39.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:39.312+0000 7f7d7f7fe640 1 -- 172.21.15.121:0/2453085835 >> v2:172.21.15.121:3300/0 conn(0x7f7d800a0fb0 msgr2=0x7f7d800a1a50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:31:39.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:39.312+0000 7f7d7f7fe640 1 --2- 172.21.15.121:0/2453085835 >> v2:172.21.15.121:3300/0 conn(0x7f7d800a0fb0 0x7f7d800a1a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T18:31:39.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:39.315+0000 7f7d7ffff640 1 --2- 172.21.15.121:0/2453085835 >> v2:172.21.15.173:3300/0 conn(0x7f7d801348f0 0x7f7d800a0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:39.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:39.315+0000 7f7d7ffff640 10 monclient(hunting): get_auth_request con 0x7f7d801348f0 auth_method 0 2024-09-17T18:31:39.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:39.315+0000 7f7d7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:39.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:39.315+0000 7f7d7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:39.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:39.315+0000 7f7d7ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:31:39.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:39.315+0000 7f7d7ffff640 1 -- 172.21.15.121:0/2453085835 >> v2:172.21.15.173:3300/0 conn(0x7f7d801348f0 msgr2=0x7f7d800a0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:31:39.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:39.315+0000 7f7d7ffff640 1 -- 172.21.15.121:0/2453085835 >> v2:172.21.15.173:3300/0 conn(0x7f7d801348f0 msgr2=0x7f7d800a0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:31:39.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:39.315+0000 7f7d7ffff640 1 --2- 172.21.15.121:0/2453085835 >> v2:172.21.15.173:3300/0 conn(0x7f7d801348f0 0x7f7d800a0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:31:39.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:39.315+0000 7f7d7ffff640 1 --2- 172.21.15.121:0/2453085835 >> v2:172.21.15.173:3300/0 conn(0x7f7d801348f0 0x7f7d800a0a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T18:31:40.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:40.910+0000 7f7d7dffb640 10 monclient(hunting): tick 2024-09-17T18:31:40.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:40.910+0000 7f7d7dffb640 10 monclient(hunting): _check_auth_tickets 2024-09-17T18:31:40.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:40.910+0000 7f7d7dffb640 1 monclient(hunting): continuing hunt 2024-09-17T18:31:40.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:40.910+0000 7f7d7dffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T18:31:40.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:40.910+0000 7f7d7dffb640 1 -- 172.21.15.121:0/2453085835 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d800a1f90 msgr2=0x7f7d800a23c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:31:40.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:40.910+0000 7f7d7dffb640 1 --2- 172.21.15.121:0/2453085835 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d800a1f90 0x7f7d800a23c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:40.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:40.910+0000 7f7d7dffb640 1 -- 172.21.15.121:0/2453085835 >> v2:172.21.15.173:3300/0 conn(0x7f7d801348f0 msgr2=0x7f7d800a0a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:31:40.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:40.910+0000 7f7d7dffb640 1 --2- 172.21.15.121:0/2453085835 >> v2:172.21.15.173:3300/0 conn(0x7f7d801348f0 0x7f7d800a0a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:40.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:40.910+0000 7f7d7dffb640 1 -- 172.21.15.121:0/2453085835 >> v2:172.21.15.121:3300/0 conn(0x7f7d800a0fb0 msgr2=0x7f7d800a1a50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:31:40.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:40.910+0000 7f7d7dffb640 1 --2- 172.21.15.121:0/2453085835 >> v2:172.21.15.121:3300/0 conn(0x7f7d800a0fb0 0x7f7d800a1a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:40.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:40.910+0000 7f7d7dffb640 10 monclient: _add_conns ranks=[1,0] 2024-09-17T18:31:40.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:40.910+0000 7f7d7dffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7d6c007ab0 addr v1:172.21.15.173:6789/0 2024-09-17T18:31:40.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:40.910+0000 7f7d7dffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7d6c00b800 addr v1:172.21.15.121:6789/0 2024-09-17T18:31:40.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:40.910+0000 7f7d7dffb640 1 -- 172.21.15.121:0/2453085835 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d800a38e0 con 0x7f7d6c00b800 2024-09-17T18:31:40.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:40.910+0000 7f7d7dffb640 1 -- 172.21.15.121:0/2453085835 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d6c010540 con 0x7f7d6c007ab0 2024-09-17T18:31:40.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:40.910+0000 7f7d7dffb640 10 monclient(hunting): _renew_subs 2024-09-17T18:31:40.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:40.910+0000 7f7d7f7fe640 1 -- 172.21.15.121:0/2453085835 >> v1:172.21.15.121:6789/0 conn(0x7f7d6c00b800 legacy=0x7f7d6c00dcc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.121:6789/0 2024-09-17T18:31:40.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:40.912+0000 7f7d7ffff640 1 -- 172.21.15.121:0/2453085835 >> v1:172.21.15.173:6789/0 conn(0x7f7d6c007ab0 legacy=0x7f7d6c009f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:41.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:41.111+0000 7f7d7f7fe640 1 -- 172.21.15.121:0/2453085835 >> v1:172.21.15.121:6789/0 conn(0x7f7d6c00b800 legacy=0x7f7d6c00dcc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.121:6789/0 2024-09-17T18:31:41.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:41.512+0000 7f7d7f7fe640 1 -- 172.21.15.121:0/2453085835 >> v1:172.21.15.121:6789/0 conn(0x7f7d6c00b800 legacy=0x7f7d6c00dcc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.121:6789/0 2024-09-17T18:31:42.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.313+0000 7f7d7f7fe640 1 -- 172.21.15.121:0/2453085835 >> v1:172.21.15.121:6789/0 conn(0x7f7d6c00b800 legacy=0x7f7d6c00dcc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.121:6789/0 2024-09-17T18:31:42.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.809+0000 7f7d7effd640 1 -- 172.21.15.121:0/2453085835 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2279273905 0 0) 0x7f7d6c010540 con 0x7f7d6c007ab0 2024-09-17T18:31:42.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.809+0000 7f7d7effd640 10 monclient(hunting): _negotiate 2024-09-17T18:31:42.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.809+0000 7f7d7effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:42.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.809+0000 7f7d7effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:42.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.809+0000 7f7d7effd640 10 monclient(hunting): my global_id is 84133 2024-09-17T18:31:42.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.809+0000 7f7d7effd640 1 -- 172.21.15.121:0/2453085835 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d68003520 con 0x7f7d6c007ab0 2024-09-17T18:31:42.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d7effd640 1 -- 172.21.15.121:0/2453085835 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2723264481 0 0) 0x7f7d68003520 con 0x7f7d6c007ab0 2024-09-17T18:31:42.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d7effd640 1 -- 172.21.15.121:0/2453085835 >> v1:172.21.15.121:6789/0 conn(0x7f7d6c00b800 legacy=0x7f7d6c00dcc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:31:42.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d7effd640 10 monclient: _finish_hunting 0 2024-09-17T18:31:42.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d7effd640 1 monclient: found mon.noname-c-legacy 2024-09-17T18:31:42.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d7effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:31:42.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d7effd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.173:6789/0 2024-09-17T18:31:42.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d7effd640 1 -- 172.21.15.121:0/2453085835 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d6c0117a0 con 0x7f7d6c007ab0 2024-09-17T18:31:42.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d7effd640 10 monclient: _finish_auth 0 2024-09-17T18:31:42.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d7effd640 10 monclient: _check_auth_tickets 2024-09-17T18:31:42.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:31:42.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d7effd640 1 -- 172.21.15.121:0/2453085835 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4002019445 0 0) 0x7f7d7000d680 con 0x7f7d6c007ab0 2024-09-17T18:31:42.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:42.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d7effd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-17T18:31:42.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d7effd640 10 monclient: dump: 2024-09-17T18:31:42.817 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:42.817 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:42.817 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:42.817 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:42.817 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:42.817 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:42.817 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:42.817 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:42.817 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:42.818 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:42.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d7effd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] but i'm connected to v1:172.21.15.173:6789/0, reconnecting 2024-09-17T18:31:42.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d7effd640 10 monclient: _reopen_session rank -1 2024-09-17T18:31:42.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d7effd640 1 -- 172.21.15.121:0/2453085835 >> v1:172.21.15.173:6789/0 conn(0x7f7d6c007ab0 legacy=0x7f7d6c009f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:42.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d84ee8640 1 -- 172.21.15.121:0/2453085835 reap_dead start 2024-09-17T18:31:42.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d7effd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T18:31:42.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d7effd640 1 --2- 172.21.15.121:0/2453085835 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d680039b0 0x7f7d68003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:42.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d7effd640 10 monclient(hunting): picked mon.c con 0x7f7d680039b0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:31:42.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d7effd640 1 --2- 172.21.15.121:0/2453085835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d68004800 0x7f7d68004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:42.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d7effd640 10 monclient(hunting): picked mon.a con 0x7f7d68004800 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:31:42.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d7effd640 1 --2- 172.21.15.121:0/2453085835 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d680053c0 0x7f7d6800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:42.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d7effd640 10 monclient(hunting): picked mon.b con 0x7f7d680053c0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:31:42.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d7effd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:42.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d7effd640 1 -- 172.21.15.121:0/2453085835 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6800c220 con 0x7f7d68004800 2024-09-17T18:31:42.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d7effd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:42.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d7effd640 1 -- 172.21.15.121:0/2453085835 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6800c3d0 con 0x7f7d680053c0 2024-09-17T18:31:42.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d7effd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:42.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d7effd640 1 -- 172.21.15.121:0/2453085835 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7d6800c5d0 con 0x7f7d680039b0 2024-09-17T18:31:42.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d7effd640 10 monclient(hunting): _renew_subs 2024-09-17T18:31:42.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d7effd640 10 monclient(hunting): _finish_auth 0 2024-09-17T18:31:42.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d7f7fe640 1 -- 172.21.15.121:0/2453085835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d68004800 msgr2=0x7f7d68004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:31:42.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d7f7fe640 1 --2- 172.21.15.121:0/2453085835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d68004800 0x7f7d68004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:31:42.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d7ffff640 1 --2- 172.21.15.121:0/2453085835 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d680039b0 0x7f7d68003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:42.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d7ffff640 10 monclient(hunting): get_auth_request con 0x7f7d680039b0 auth_method 0 2024-09-17T18:31:42.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:42.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:42.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d7ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:31:42.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d7ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:42.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d7ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:42.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.810+0000 7f7d7ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:31:42.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.811+0000 7f7d7ffff640 10 monclient(hunting): handle_auth_done global_id 84133 payload 293 2024-09-17T18:31:42.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.811+0000 7f7d7ffff640 1 -- 172.21.15.121:0/2453085835 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d680053c0 msgr2=0x7f7d6800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:31:42.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.811+0000 7f7d7ffff640 1 --2- 172.21.15.121:0/2453085835 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d680053c0 0x7f7d6800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:42.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.811+0000 7f7d7ffff640 1 -- 172.21.15.121:0/2453085835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d68004800 msgr2=0x7f7d68004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:31:42.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.811+0000 7f7d7ffff640 1 --2- 172.21.15.121:0/2453085835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d68004800 0x7f7d68004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:42.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.811+0000 7f7d7ffff640 10 monclient: _finish_hunting 0 2024-09-17T18:31:42.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.811+0000 7f7d7ffff640 1 monclient: found mon.c 2024-09-17T18:31:42.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.811+0000 7f7d7ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:31:42.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.811+0000 7f7d7ffff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:31:42.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.811+0000 7f7d7ffff640 1 -- 172.21.15.121:0/2453085835 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d68003520 con 0x7f7d680039b0 2024-09-17T18:31:42.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.811+0000 7f7d7ffff640 10 monclient: _finish_auth 0 2024-09-17T18:31:42.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.811+0000 7f7d7ffff640 10 monclient: _check_auth_tickets 2024-09-17T18:31:42.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.811+0000 7f7d7ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:31:42.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.811+0000 7f7d8694b640 5 monclient: authenticate success, global_id 84133 2024-09-17T18:31:42.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.811+0000 7f7d8694b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:31:42.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.814+0000 7f7d7ffff640 1 -- 172.21.15.121:0/2453085835 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d680039b0 msgr2=0x7f7d68003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:42.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.814+0000 7f7d7ffff640 1 --2- 172.21.15.121:0/2453085835 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d680039b0 0x7f7d68003d90 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7d70006fd0 tx=0x7f7d68005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbe2685c2adc4e27 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:42.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.814+0000 7f7d7effd640 1 -- 172.21.15.121:0/2453085835 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d70031070 con 0x7f7d680039b0 2024-09-17T18:31:42.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.814+0000 7f7d7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:42.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.814+0000 7f7d7effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:31:42.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.814+0000 7f7d7effd640 10 monclient: dump: 2024-09-17T18:31:42.827 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:42.827 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:42.827 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:42.827 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:42.827 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:42.827 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:42.827 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:42.827 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:42.827 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:42.827 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:42.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.814+0000 7f7d7effd640 1 -- 172.21.15.121:0/2453085835 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d7002bbd0 con 0x7f7d680039b0 2024-09-17T18:31:42.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.814+0000 7f7d8694b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:31:42.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.814+0000 7f7d7effd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:31:42.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.814+0000 7f7d7effd640 1 -- 172.21.15.121:0/2453085835 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d7000ddc0 con 0x7f7d680039b0 2024-09-17T18:31:42.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.814+0000 7f7d8694b640 10 monclient: get_monmap_and_config success 2024-09-17T18:31:42.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.814+0000 7f7d7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:42.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.814+0000 7f7d8694b640 4 set_mon_vals no callback set 2024-09-17T18:31:42.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.814+0000 7f7d7effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:31:42.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.814+0000 7f7d7effd640 10 monclient: dump: 2024-09-17T18:31:42.828 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:42.828 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:42.828 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:42.828 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:42.828 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:42.828 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:42.828 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:42.829 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:42.829 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:42.829 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:42.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.815+0000 7f7d8694b640 10 monclient: shutdown 2024-09-17T18:31:42.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.815+0000 7f7d8694b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:31:42.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.815+0000 7f7d8694b640 1 -- 172.21.15.121:0/2453085835 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d680039b0 msgr2=0x7f7d68003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:42.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.815+0000 7f7d8694b640 1 --2- 172.21.15.121:0/2453085835 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d680039b0 0x7f7d68003d90 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7d70006fd0 tx=0x7f7d68005830 comp rx=0 tx=0).stop 2024-09-17T18:31:42.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.815+0000 7f7d8694b640 1 -- 172.21.15.121:0/2453085835 shutdown_connections 2024-09-17T18:31:42.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.815+0000 7f7d8694b640 1 --2- 172.21.15.121:0/2453085835 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d680053c0 0x7f7d6800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:42.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.815+0000 7f7d8694b640 1 --2- 172.21.15.121:0/2453085835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d68004800 0x7f7d68004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:42.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.815+0000 7f7d8694b640 1 --2- 172.21.15.121:0/2453085835 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d680039b0 0x7f7d68003d90 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:42.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.815+0000 7f7d8694b640 1 -- 172.21.15.121:0/2453085835 >> 172.21.15.121:0/2453085835 conn(0x7f7d800a3b50 msgr2=0x7f7d800a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:31:42.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.815+0000 7f7d8694b640 1 -- 172.21.15.121:0/2453085835 shutdown_connections 2024-09-17T18:31:42.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.815+0000 7f7d8694b640 1 -- 172.21.15.121:0/2453085835 wait complete. 2024-09-17T18:31:42.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.816+0000 7f7d8694b640 10 monclient: build_initial_monmap 2024-09-17T18:31:42.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.816+0000 7f7d8694b640 10 monclient: monmap: 2024-09-17T18:31:42.830 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:31:42.830 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:31:42.830 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:31:42.830 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:31:42.830 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:31:42.830 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:42.830 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:31:42.830 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:31:42.830 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:31:42.830 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:42.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.816+0000 7f7d8694b640 1 librados: starting msgr at 2024-09-17T18:31:42.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.816+0000 7f7d8694b640 1 librados: starting objecter 2024-09-17T18:31:42.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.816+0000 7f7d8694b640 1 Processor -- start 2024-09-17T18:31:42.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.817+0000 7f7d8694b640 1 -- start start 2024-09-17T18:31:42.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.817+0000 7f7d8694b640 1 librados: setting wanted keys 2024-09-17T18:31:42.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.817+0000 7f7d8694b640 1 librados: calling monclient init 2024-09-17T18:31:42.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.817+0000 7f7d8694b640 10 monclient: init 2024-09-17T18:31:42.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.817+0000 7f7d8694b640 10 monclient: _reopen_session rank -1 2024-09-17T18:31:42.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.817+0000 7f7d8694b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T18:31:42.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.817+0000 7f7d8694b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d680039b0 0x7f7d801c8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:42.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.817+0000 7f7d8694b640 10 monclient(hunting): picked mon.noname-c con 0x7f7d680039b0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:31:42.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.817+0000 7f7d8694b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d68004800 0x7f7d801338d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:42.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.817+0000 7f7d8694b640 10 monclient(hunting): picked mon.noname-b con 0x7f7d68004800 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:31:42.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.817+0000 7f7d8694b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d680053c0 0x7f7d80133e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:42.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.817+0000 7f7d8694b640 10 monclient(hunting): picked mon.noname-a con 0x7f7d680053c0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:31:42.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.817+0000 7f7d8694b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:42.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.817+0000 7f7d8694b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7d800a1540 con 0x7f7d680053c0 2024-09-17T18:31:42.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.817+0000 7f7d8694b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:42.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.817+0000 7f7d8694b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7d800a13c0 con 0x7f7d68004800 2024-09-17T18:31:42.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.817+0000 7f7d8694b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:42.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.817+0000 7f7d8694b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7d8009dce0 con 0x7f7d680039b0 2024-09-17T18:31:42.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.817+0000 7f7d8694b640 10 monclient(hunting): _renew_subs 2024-09-17T18:31:42.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.817+0000 7f7d8694b640 10 monclient(hunting): authenticate will time out at 1506.534872s 2024-09-17T18:31:42.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.817+0000 7f7d84ee8640 1 -- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d680053c0 msgr2=0x7f7d80133e10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:31:42.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.817+0000 7f7d84ee8640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d680053c0 0x7f7d80133e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:31:42.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.818+0000 7f7d7ffff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d680039b0 0x7f7d801c8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:42.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.818+0000 7f7d7ffff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d680039b0 0x7f7d801c8aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:56044/0 (socket says 172.21.15.121:56044) 2024-09-17T18:31:42.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.818+0000 7f7d7ffff640 1 -- 172.21.15.121:0/3088102664 learned_addr learned my addr 172.21.15.121:0/3088102664 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:31:42.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.818+0000 7f7d7ffff640 10 monclient(hunting): get_auth_request con 0x7f7d680039b0 auth_method 0 2024-09-17T18:31:42.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.818+0000 7f7d7f7fe640 1 --2- 172.21.15.121:0/3088102664 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d68004800 0x7f7d801338d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:42.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.818+0000 7f7d7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:42.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.818+0000 7f7d7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:42.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.818+0000 7f7d7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:42.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.818+0000 7f7d7f7fe640 10 monclient(hunting): get_auth_request con 0x7f7d68004800 auth_method 0 2024-09-17T18:31:42.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.818+0000 7f7d7f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:42.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.818+0000 7f7d7f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:42.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.818+0000 7f7d7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:42.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.818+0000 7f7d7ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:42.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.818+0000 7f7d7ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:42.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.818+0000 7f7d7ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:42.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.818+0000 7f7d7f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:42.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.818+0000 7f7d7f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:42.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.818+0000 7f7d7f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:42.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.818+0000 7f7d7ffff640 10 monclient(hunting): handle_auth_done global_id 64340 payload 824 2024-09-17T18:31:42.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.818+0000 7f7d7ffff640 1 -- 172.21.15.121:0/3088102664 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d68004800 msgr2=0x7f7d801338d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:42.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.818+0000 7f7d7ffff640 1 --2- 172.21.15.121:0/3088102664 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d68004800 0x7f7d801338d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:42.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.818+0000 7f7d7ffff640 1 -- 172.21.15.121:0/3088102664 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d680053c0 msgr2=0x7f7d80133e10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:31:42.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.818+0000 7f7d7ffff640 1 --2- 172.21.15.121:0/3088102664 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d680053c0 0x7f7d80133e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:42.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.818+0000 7f7d7ffff640 10 monclient: _finish_hunting 0 2024-09-17T18:31:42.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.818+0000 7f7d7ffff640 1 monclient: found mon.noname-c 2024-09-17T18:31:42.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.818+0000 7f7d7ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:31:42.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.818+0000 7f7d7ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:31:42.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.818+0000 7f7d7ffff640 1 -- 172.21.15.121:0/3088102664 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d8011ef30 con 0x7f7d680039b0 2024-09-17T18:31:42.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.818+0000 7f7d7f7fe640 1 --2- 172.21.15.121:0/3088102664 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d68004800 0x7f7d801338d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:31:42.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.820+0000 7f7d7ffff640 1 -- 172.21.15.121:0/3088102664 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d680039b0 msgr2=0x7f7d801c8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:42.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.820+0000 7f7d7ffff640 1 --2- 172.21.15.121:0/3088102664 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d680039b0 0x7f7d801c8aa0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7d7000d310 tx=0x7f7d700046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:42.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.820+0000 7f7d7cff9640 10 client.?.objecter ms_handle_connect 0x7f7d680039b0 2024-09-17T18:31:42.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.820+0000 7f7d7cff9640 10 client.?.objecter resend_mon_ops 2024-09-17T18:31:42.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.820+0000 7f7d7cff9640 1 -- 172.21.15.121:0/3088102664 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d7000ddc0 con 0x7f7d680039b0 2024-09-17T18:31:42.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.820+0000 7f7d7cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:42.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.820+0000 7f7d7cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:31:42.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.820+0000 7f7d7cff9640 10 monclient: dump: 2024-09-17T18:31:42.834 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:42.835 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:42.835 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:42.835 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:42.835 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:42.835 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:42.835 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:42.835 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:42.835 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:42.835 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:42.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.820+0000 7f7d7cff9640 10 monclient: _finish_auth 0 2024-09-17T18:31:42.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.820+0000 7f7d7cff9640 10 monclient: _check_auth_tickets 2024-09-17T18:31:42.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.820+0000 7f7d7cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:31:42.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.820+0000 7f7d8694b640 5 monclient: authenticate success, global_id 64340 2024-09-17T18:31:42.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.820+0000 7f7d7cff9640 1 -- 172.21.15.121:0/3088102664 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d70003820 con 0x7f7d680039b0 2024-09-17T18:31:42.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.820+0000 7f7d8694b640 10 monclient: _renew_subs 2024-09-17T18:31:42.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.820+0000 7f7d8694b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:31:42.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.820+0000 7f7d8694b640 1 -- 172.21.15.121:0/3088102664 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d801209f0 con 0x7f7d680039b0 2024-09-17T18:31:42.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.821+0000 7f7d7cff9640 10 monclient: handle_config config(0 keys) 2024-09-17T18:31:42.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.821+0000 7f7d7cff9640 1 -- 172.21.15.121:0/3088102664 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d70003c10 con 0x7f7d680039b0 2024-09-17T18:31:42.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.821+0000 7f7d7cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:42.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.821+0000 7f7d7cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:31:42.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.821+0000 7f7d7cff9640 10 monclient: dump: 2024-09-17T18:31:42.836 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:42.836 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:42.836 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:42.836 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:42.836 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:42.836 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:42.836 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:42.836 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:42.836 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:42.836 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:42.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.821+0000 7f7d8694b640 10 client.64340.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:31:42.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.821+0000 7f7d8694b640 10 monclient: _renew_subs 2024-09-17T18:31:42.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.821+0000 7f7d8694b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:31:42.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.821+0000 7f7d8694b640 1 -- 172.21.15.121:0/3088102664 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d801c8fe0 con 0x7f7d680039b0 2024-09-17T18:31:42.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.821+0000 7f7d8694b640 1 librados: init done 2024-09-17T18:31:42.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.821+0000 7f7d7e7fc640 4 set_mon_vals no callback set 2024-09-17T18:31:42.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.821+0000 7f7d7cff9640 1 -- 172.21.15.121:0/3088102664 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f7d70035410 con 0x7f7d680039b0 2024-09-17T18:31:42.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.822+0000 7f7d8694b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:31:42.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.822+0000 7f7d8694b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:31:42.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.822+0000 7f7d8694b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:31:42.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.822+0000 7f7d7cff9640 1 --2- 172.21.15.121:0/3088102664 >> v2:172.21.15.173:6812/2423786037 conn(0x7f7d6c044ee0 0x7f7d6c0473a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:42.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.822+0000 7f7d8694b640 1 -- 172.21.15.121:0/3088102664 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d800a1a50 con 0x7f7d680039b0 2024-09-17T18:31:42.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.822+0000 7f7d7cff9640 1 -- 172.21.15.121:0/3088102664 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(486..486 src has 247..486) ==== 5880+0+0 (secure 0 0 0) 0x7f7d70073de0 con 0x7f7d680039b0 2024-09-17T18:31:42.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.822+0000 7f7d7cff9640 10 client.64340.objecter ms_dispatch 0x7f7d80000b90 osd_map(486..486 src has 247..486) 2024-09-17T18:31:42.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.822+0000 7f7d7cff9640 3 client.64340.objecter handle_osd_map got epochs [486,486] > 0 2024-09-17T18:31:42.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.822+0000 7f7d7cff9640 3 client.64340.objecter handle_osd_map decoding full epoch 486 2024-09-17T18:31:42.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.822+0000 7f7d7f7fe640 1 --2- 172.21.15.121:0/3088102664 >> v2:172.21.15.173:6812/2423786037 conn(0x7f7d6c044ee0 0x7f7d6c0473a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:42.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.822+0000 7f7d7f7fe640 10 monclient: get_auth_request con 0x7f7d6c044ee0 auth_method 0 2024-09-17T18:31:42.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.822+0000 7f7d7cff9640 20 client.64340.objecter dump_active .. 0 homeless 2024-09-17T18:31:42.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.825+0000 7f7d7f7fe640 1 --2- 172.21.15.121:0/3088102664 >> v2:172.21.15.173:6812/2423786037 conn(0x7f7d6c044ee0 0x7f7d6c0473a0 secure :-1 s=READY pgs=4040 cs=0 l=1 rev1=1 crypto rx=0x7f7d74003cb0 tx=0x7f7d74009300 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:42.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.825+0000 7f7d7cff9640 10 client.64340.objecter ms_handle_connect 0x7f7d6c044ee0 2024-09-17T18:31:42.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.827+0000 7f7d7cff9640 1 -- 172.21.15.121:0/3088102664 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7d70072390 con 0x7f7d680039b0 2024-09-17T18:31:42.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.827+0000 7f7d7cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:31:42.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:42.827+0000 7f7d7cff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:31:43.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.070+0000 7f7d8694b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:31:43.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.070+0000 7f7d8694b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:31:43.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.070+0000 7f7d8694b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:31:43.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.070+0000 7f7d8694b640 1 -- 172.21.15.121:0/3088102664 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7d8009e1c0 con 0x7f7d680039b0 2024-09-17T18:31:43.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.071+0000 7f7d7cff9640 1 -- 172.21.15.121:0/3088102664 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f7d70035dc0 con 0x7f7d680039b0 2024-09-17T18:31:43.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.071+0000 7f7d7cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:31:43.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.071+0000 7f7d7cff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:31:43.074 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":46,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:31:43.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.073+0000 7f7d8694b640 10 librados: watch_flush enter 2024-09-17T18:31:43.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.073+0000 7f7d8694b640 10 librados: watch_flush exit 2024-09-17T18:31:43.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.073+0000 7f7d8694b640 20 client.64340.objecter shutdown clearing up homeless session... 2024-09-17T18:31:43.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.073+0000 7f7d8694b640 10 client.64340.objecter successfully canceled tick 2024-09-17T18:31:43.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.073+0000 7f7d8694b640 1 -- 172.21.15.121:0/3088102664 >> v2:172.21.15.173:6812/2423786037 conn(0x7f7d6c044ee0 msgr2=0x7f7d6c0473a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:43.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.073+0000 7f7d8694b640 1 --2- 172.21.15.121:0/3088102664 >> v2:172.21.15.173:6812/2423786037 conn(0x7f7d6c044ee0 0x7f7d6c0473a0 secure :-1 s=READY pgs=4040 cs=0 l=1 rev1=1 crypto rx=0x7f7d74003cb0 tx=0x7f7d74009300 comp rx=0 tx=0).stop 2024-09-17T18:31:43.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.073+0000 7f7d8694b640 10 monclient: shutdown 2024-09-17T18:31:43.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.073+0000 7f7d8694b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:31:43.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.073+0000 7f7d8694b640 1 -- 172.21.15.121:0/3088102664 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d680039b0 msgr2=0x7f7d801c8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:43.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.073+0000 7f7d8694b640 1 --2- 172.21.15.121:0/3088102664 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d680039b0 0x7f7d801c8aa0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7d7000d310 tx=0x7f7d700046c0 comp rx=0 tx=0).stop 2024-09-17T18:31:43.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.073+0000 7f7d8694b640 1 -- 172.21.15.121:0/3088102664 shutdown_connections 2024-09-17T18:31:43.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.073+0000 7f7d8694b640 1 --2- 172.21.15.121:0/3088102664 >> v2:172.21.15.173:6812/2423786037 conn(0x7f7d6c044ee0 0x7f7d6c0473a0 unknown :-1 s=CLOSED pgs=4040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:43.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.073+0000 7f7d8694b640 1 --2- 172.21.15.121:0/3088102664 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d68004800 0x7f7d801338d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:43.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.073+0000 7f7d8694b640 1 --2- 172.21.15.121:0/3088102664 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d680053c0 0x7f7d80133e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:43.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.073+0000 7f7d8694b640 1 --2- 172.21.15.121:0/3088102664 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d680039b0 0x7f7d801c8aa0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:43.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.073+0000 7f7d8694b640 1 -- 172.21.15.121:0/3088102664 >> 172.21.15.121:0/3088102664 conn(0x7f7d800a3b50 msgr2=0x7f7d801bc6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:31:43.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.073+0000 7f7d8694b640 1 -- 172.21.15.121:0/3088102664 shutdown_connections 2024-09-17T18:31:43.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.073+0000 7f7d8694b640 1 -- 172.21.15.121:0/3088102664 wait complete. 2024-09-17T18:31:43.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.073+0000 7f7d8694b640 1 librados: shutdown 2024-09-17T18:31:43.087 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-17T18:31:43.087 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-17T18:31:43.087 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T18:31:43.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.182+0000 7f65814f6640 10 monclient: get_monmap_and_config 2024-09-17T18:31:43.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.182+0000 7f65814f6640 10 monclient: build_initial_monmap 2024-09-17T18:31:43.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.182+0000 7f65814f6640 10 monclient: monmap: 2024-09-17T18:31:43.185 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:31:43.185 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:43.185 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:31:43.184413+0000 2024-09-17T18:31:43.186 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:31:43.184413+0000 2024-09-17T18:31:43.186 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:31:43.186 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:43.186 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:31:43.186 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:31:43.186 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:31:43.186 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:31:43.186 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:31:43.186 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:43.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.183+0000 7f65814f6640 1 Processor -- start 2024-09-17T18:31:43.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.183+0000 7f65814f6640 1 -- start start 2024-09-17T18:31:43.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.183+0000 7f65814f6640 10 monclient: init 2024-09-17T18:31:43.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.183+0000 7f65814f6640 10 monclient: _reopen_session rank -1 2024-09-17T18:31:43.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.183+0000 7f65814f6640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-17T18:31:43.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.183+0000 7f65814f6640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f657c0a5b90 0x7f657c0a5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:43.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.183+0000 7f65814f6640 10 monclient(hunting): picked mon.noname-b con 0x7f657c0a5b90 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:31:43.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.183+0000 7f65814f6640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f657c09d830 0x7f657c0a64b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:43.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.183+0000 7f65814f6640 10 monclient(hunting): picked mon.noname-a con 0x7f657c09d830 addr v2:172.21.15.121:3300/0 2024-09-17T18:31:43.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.183+0000 7f65814f6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f657c0a7080 addr v1:172.21.15.121:6789/0 2024-09-17T18:31:43.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.183+0000 7f65814f6640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f657c0a8690 con 0x7f657c0a7080 2024-09-17T18:31:43.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.183+0000 7f65814f6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:43.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.183+0000 7f65814f6640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f657c0a69f0 con 0x7f657c09d830 2024-09-17T18:31:43.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.183+0000 7f65814f6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:43.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.183+0000 7f65814f6640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f657c09cde0 con 0x7f657c0a5b90 2024-09-17T18:31:43.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.183+0000 7f65814f6640 10 monclient(hunting): _renew_subs 2024-09-17T18:31:43.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.183+0000 7f65814f6640 10 monclient(hunting): authenticate will time out at 1506.900945s 2024-09-17T18:31:43.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.183+0000 7f657a7fc640 1 -- >> v2:172.21.15.121:3300/0 conn(0x7f657c09d830 msgr2=0x7f657c0a64b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:31:43.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.183+0000 7f657b7fe640 1 -- >> v1:172.21.15.121:6789/0 conn(0x7f657c0a7080 legacy=0x7f657c138950 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.121:6789/0 2024-09-17T18:31:43.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.183+0000 7f657a7fc640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f657c09d830 0x7f657c0a64b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:31:43.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.184+0000 7f657affd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f657c0a5b90 0x7f657c0a5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:43.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.184+0000 7f657affd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f657c0a5b90 0x7f657c0a5f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:56056/0 (socket says 172.21.15.121:56056) 2024-09-17T18:31:43.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.184+0000 7f657affd640 1 -- 172.21.15.121:0/2374617207 learned_addr learned my addr 172.21.15.121:0/2374617207 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:31:43.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.184+0000 7f657affd640 10 monclient(hunting): get_auth_request con 0x7f657c0a5b90 auth_method 0 2024-09-17T18:31:43.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.184+0000 7f657affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:43.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.184+0000 7f657affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:43.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.184+0000 7f657affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:43.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.184+0000 7f657affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:43.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.184+0000 7f657affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:43.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.184+0000 7f657affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:43.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.184+0000 7f657affd640 10 monclient(hunting): handle_auth_done global_id 64343 payload 274 2024-09-17T18:31:43.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.184+0000 7f657affd640 1 -- 172.21.15.121:0/2374617207 >> v2:172.21.15.121:3300/0 conn(0x7f657c09d830 msgr2=0x7f657c0a64b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:31:43.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.184+0000 7f657affd640 1 --2- 172.21.15.121:0/2374617207 >> v2:172.21.15.121:3300/0 conn(0x7f657c09d830 0x7f657c0a64b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:43.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.184+0000 7f657affd640 1 -- 172.21.15.121:0/2374617207 >> v1:172.21.15.121:6789/0 conn(0x7f657c0a7080 legacy=0x7f657c138950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:31:43.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.184+0000 7f657affd640 10 monclient: _finish_hunting 0 2024-09-17T18:31:43.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.184+0000 7f657affd640 1 monclient: found mon.noname-b 2024-09-17T18:31:43.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.184+0000 7f657affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:31:43.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.184+0000 7f657affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:31:43.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.184+0000 7f657affd640 1 -- 172.21.15.121:0/2374617207 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f657c09cb00 con 0x7f657c0a5b90 2024-09-17T18:31:43.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.187+0000 7f657affd640 1 -- 172.21.15.121:0/2374617207 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f657c0a5b90 msgr2=0x7f657c0a5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:43.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.187+0000 7f657affd640 1 --2- 172.21.15.121:0/2374617207 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f657c0a5b90 0x7f657c0a5f70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6574009e50 tx=0x7f657402a570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f16aec357fe599e server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:43.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.187+0000 7f6579ffb640 1 -- 172.21.15.121:0/2374617207 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6574036070 con 0x7f657c0a5b90 2024-09-17T18:31:43.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.187+0000 7f6579ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:43.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.187+0000 7f6579ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:31:43.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.187+0000 7f6579ffb640 10 monclient: dump: 2024-09-17T18:31:43.200 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:43.200 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:43.200 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:43.200 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:43.200 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:43.200 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:43.200 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:43.200 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:43.200 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:43.200 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:43.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.187+0000 7f6579ffb640 10 monclient: _finish_auth 0 2024-09-17T18:31:43.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.187+0000 7f6579ffb640 10 monclient: _check_auth_tickets 2024-09-17T18:31:43.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.187+0000 7f6579ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:31:43.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.187+0000 7f6579ffb640 1 -- 172.21.15.121:0/2374617207 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6574007800 con 0x7f657c0a5b90 2024-09-17T18:31:43.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.187+0000 7f65814f6640 5 monclient: authenticate success, global_id 64343 2024-09-17T18:31:43.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.187+0000 7f65814f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:31:43.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.187+0000 7f6579ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:31:43.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.187+0000 7f6579ffb640 1 -- 172.21.15.121:0/2374617207 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6574007bd0 con 0x7f657c0a5b90 2024-09-17T18:31:43.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.187+0000 7f65814f6640 10 monclient: get_monmap_and_config success 2024-09-17T18:31:43.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.187+0000 7f65814f6640 4 set_mon_vals no callback set 2024-09-17T18:31:43.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.187+0000 7f6579ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:43.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.187+0000 7f6579ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:31:43.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.187+0000 7f6579ffb640 10 monclient: dump: 2024-09-17T18:31:43.201 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:43.201 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:43.201 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:43.201 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:43.201 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:43.202 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:43.202 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:43.202 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:43.202 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:43.202 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:43.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.187+0000 7f65814f6640 10 monclient: shutdown 2024-09-17T18:31:43.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.187+0000 7f65814f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:31:43.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.187+0000 7f65814f6640 1 -- 172.21.15.121:0/2374617207 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f657c0a5b90 msgr2=0x7f657c0a5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:43.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.187+0000 7f65814f6640 1 --2- 172.21.15.121:0/2374617207 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f657c0a5b90 0x7f657c0a5f70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6574009e50 tx=0x7f657402a570 comp rx=0 tx=0).stop 2024-09-17T18:31:43.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.187+0000 7f65814f6640 1 -- 172.21.15.121:0/2374617207 shutdown_connections 2024-09-17T18:31:43.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.187+0000 7f65814f6640 1 --2- 172.21.15.121:0/2374617207 >> v2:172.21.15.121:3300/0 conn(0x7f657c09d830 0x7f657c0a64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:43.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.187+0000 7f65814f6640 1 --2- 172.21.15.121:0/2374617207 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f657c0a5b90 0x7f657c0a5f70 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:43.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.187+0000 7f65814f6640 1 -- 172.21.15.121:0/2374617207 >> 172.21.15.121:0/2374617207 conn(0x7f657c0ab880 msgr2=0x7f657c0a7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:31:43.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.187+0000 7f65814f6640 1 -- 172.21.15.121:0/2374617207 shutdown_connections 2024-09-17T18:31:43.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.187+0000 7f65814f6640 1 -- 172.21.15.121:0/2374617207 wait complete. 2024-09-17T18:31:43.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.188+0000 7f65814f6640 10 monclient: build_initial_monmap 2024-09-17T18:31:43.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.188+0000 7f65814f6640 10 monclient: monmap: 2024-09-17T18:31:43.203 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:31:43.203 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:31:43.203 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:31:43.203 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:31:43.203 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:31:43.203 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:43.203 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:31:43.203 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:31:43.203 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:31:43.203 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:43.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.188+0000 7f65814f6640 1 librados: starting msgr at 2024-09-17T18:31:43.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.188+0000 7f65814f6640 1 librados: starting objecter 2024-09-17T18:31:43.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.188+0000 7f65814f6640 1 Processor -- start 2024-09-17T18:31:43.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.188+0000 7f65814f6640 1 -- start start 2024-09-17T18:31:43.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.188+0000 7f65814f6640 1 librados: setting wanted keys 2024-09-17T18:31:43.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.188+0000 7f65814f6640 1 librados: calling monclient init 2024-09-17T18:31:43.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.188+0000 7f65814f6640 10 monclient: init 2024-09-17T18:31:43.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f65814f6640 10 monclient: _reopen_session rank -1 2024-09-17T18:31:43.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f65814f6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T18:31:43.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f65814f6640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f657c09d830 0x7f657c137ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:43.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f65814f6640 10 monclient(hunting): picked mon.noname-c con 0x7f657c09d830 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:31:43.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f65814f6640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f657c0a5b90 0x7f657c138400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:43.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f65814f6640 10 monclient(hunting): picked mon.noname-a con 0x7f657c0a5b90 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:31:43.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f65814f6640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f657c0a7080 0x7f657c125aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:43.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f65814f6640 10 monclient(hunting): picked mon.noname-b con 0x7f657c0a7080 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:31:43.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f65814f6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:43.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f65814f6640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f657c139060 con 0x7f657c0a5b90 2024-09-17T18:31:43.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f65814f6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:43.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f65814f6640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f657c09d140 con 0x7f657c0a7080 2024-09-17T18:31:43.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f65814f6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:43.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f65814f6640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f657c1391e0 con 0x7f657c09d830 2024-09-17T18:31:43.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f65814f6640 10 monclient(hunting): _renew_subs 2024-09-17T18:31:43.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f65814f6640 10 monclient(hunting): authenticate will time out at 1506.906385s 2024-09-17T18:31:43.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f657a7fc640 1 -- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f657c0a5b90 msgr2=0x7f657c138400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:31:43.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f657a7fc640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f657c0a5b90 0x7f657c138400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:31:43.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f657affd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f657c09d830 0x7f657c137ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:43.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f657affd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f657c09d830 0x7f657c137ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:56060/0 (socket says 172.21.15.121:56060) 2024-09-17T18:31:43.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f657affd640 1 -- 172.21.15.121:0/3116052637 learned_addr learned my addr 172.21.15.121:0/3116052637 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:31:43.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f657affd640 10 monclient(hunting): get_auth_request con 0x7f657c09d830 auth_method 0 2024-09-17T18:31:43.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f657affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:43.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f657affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:43.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f657affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:43.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f657affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:43.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f657affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:43.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f657affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:43.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f657b7fe640 1 --2- 172.21.15.121:0/3116052637 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f657c0a7080 0x7f657c125aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:43.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f657b7fe640 10 monclient(hunting): get_auth_request con 0x7f657c0a7080 auth_method 0 2024-09-17T18:31:43.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f657b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:43.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f657b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:43.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f657b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:43.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f657affd640 10 monclient(hunting): handle_auth_done global_id 64346 payload 824 2024-09-17T18:31:43.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f657affd640 1 -- 172.21.15.121:0/3116052637 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f657c0a7080 msgr2=0x7f657c125aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:43.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f657affd640 1 --2- 172.21.15.121:0/3116052637 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f657c0a7080 0x7f657c125aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:43.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f657affd640 1 -- 172.21.15.121:0/3116052637 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f657c0a5b90 msgr2=0x7f657c138400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:31:43.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f657affd640 1 --2- 172.21.15.121:0/3116052637 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f657c0a5b90 0x7f657c138400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:43.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f657affd640 10 monclient: _finish_hunting 0 2024-09-17T18:31:43.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f657affd640 1 monclient: found mon.noname-c 2024-09-17T18:31:43.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f657affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:31:43.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f657affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:31:43.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.189+0000 7f657affd640 1 -- 172.21.15.121:0/3116052637 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f657c0a2380 con 0x7f657c09d830 2024-09-17T18:31:43.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.191+0000 7f657affd640 1 -- 172.21.15.121:0/3116052637 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f657c09d830 msgr2=0x7f657c137ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:43.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.191+0000 7f657affd640 1 --2- 172.21.15.121:0/3116052637 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f657c09d830 0x7f657c137ec0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f65740093e0 tx=0x7f6574004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:43.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.191+0000 7f65737fe640 10 client.?.objecter ms_handle_connect 0x7f657c09d830 2024-09-17T18:31:43.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.191+0000 7f65737fe640 10 client.?.objecter resend_mon_ops 2024-09-17T18:31:43.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.192+0000 7f65737fe640 1 -- 172.21.15.121:0/3116052637 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6574041020 con 0x7f657c09d830 2024-09-17T18:31:43.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.192+0000 7f65737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:43.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.192+0000 7f65737fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:31:43.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.192+0000 7f65737fe640 10 monclient: dump: 2024-09-17T18:31:43.208 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:43.208 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:43.208 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:43.208 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:43.208 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:43.208 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:43.208 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:43.208 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:43.208 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:43.208 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:43.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.192+0000 7f65737fe640 10 monclient: _finish_auth 0 2024-09-17T18:31:43.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.192+0000 7f65737fe640 10 monclient: _check_auth_tickets 2024-09-17T18:31:43.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.192+0000 7f65737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:31:43.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.192+0000 7f65737fe640 1 -- 172.21.15.121:0/3116052637 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65740319c0 con 0x7f657c09d830 2024-09-17T18:31:43.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.192+0000 7f65814f6640 5 monclient: authenticate success, global_id 64346 2024-09-17T18:31:43.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.192+0000 7f65814f6640 10 monclient: _renew_subs 2024-09-17T18:31:43.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.192+0000 7f65814f6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:31:43.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.192+0000 7f65814f6640 1 -- 172.21.15.121:0/3116052637 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f657c138940 con 0x7f657c09d830 2024-09-17T18:31:43.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.192+0000 7f65737fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:31:43.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.192+0000 7f65737fe640 1 -- 172.21.15.121:0/3116052637 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6574036040 con 0x7f657c09d830 2024-09-17T18:31:43.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.192+0000 7f65737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:43.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.192+0000 7f65737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:31:43.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.192+0000 7f65737fe640 10 monclient: dump: 2024-09-17T18:31:43.209 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:43.209 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:43.209 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:43.209 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:43.209 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:43.209 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:43.210 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:43.210 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:43.210 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:43.210 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:43.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.192+0000 7f65814f6640 10 client.64346.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:31:43.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.192+0000 7f65814f6640 10 monclient: _renew_subs 2024-09-17T18:31:43.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.192+0000 7f65814f6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:31:43.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.192+0000 7f65814f6640 1 -- 172.21.15.121:0/3116052637 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f657c129300 con 0x7f657c09d830 2024-09-17T18:31:43.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.192+0000 7f65797fa640 4 set_mon_vals no callback set 2024-09-17T18:31:43.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.192+0000 7f65814f6640 1 librados: init done 2024-09-17T18:31:43.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.192+0000 7f65737fe640 1 -- 172.21.15.121:0/3116052637 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f6574044020 con 0x7f657c09d830 2024-09-17T18:31:43.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.193+0000 7f65814f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:31:43.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.193+0000 7f65814f6640 1 --2- 172.21.15.121:0/3116052637 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f657c1280c0 0x7f657c125fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:43.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.193+0000 7f65814f6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:43.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.193+0000 7f65814f6640 1 -- 172.21.15.121:0/3116052637 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f657c129540 con 0x7f657c1280c0 2024-09-17T18:31:43.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.193+0000 7f65737fe640 1 --2- 172.21.15.121:0/3116052637 >> v2:172.21.15.173:6812/2423786037 conn(0x7f655c040120 0x7f655c0425e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:43.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.193+0000 7f657a7fc640 1 --2- 172.21.15.121:0/3116052637 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f657c1280c0 0x7f657c125fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:43.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.193+0000 7f657a7fc640 10 monclient: get_auth_request con 0x7f657c1280c0 auth_method 0 2024-09-17T18:31:43.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.193+0000 7f657a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:43.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.193+0000 7f657a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:43.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.193+0000 7f657a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:43.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.193+0000 7f65737fe640 1 -- 172.21.15.121:0/3116052637 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(486..486 src has 247..486) ==== 5880+0+0 (secure 0 0 0) 0x7f6574074800 con 0x7f657c09d830 2024-09-17T18:31:43.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.193+0000 7f65737fe640 10 client.64346.objecter ms_dispatch 0x7f657c000b90 osd_map(486..486 src has 247..486) 2024-09-17T18:31:43.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.193+0000 7f65737fe640 3 client.64346.objecter handle_osd_map got epochs [486,486] > 0 2024-09-17T18:31:43.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.193+0000 7f65737fe640 3 client.64346.objecter handle_osd_map decoding full epoch 486 2024-09-17T18:31:43.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.193+0000 7f657a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:43.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.193+0000 7f657a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:43.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.193+0000 7f657a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:43.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.193+0000 7f65737fe640 20 client.64346.objecter dump_active .. 0 homeless 2024-09-17T18:31:43.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.193+0000 7f657b7fe640 1 --2- 172.21.15.121:0/3116052637 >> v2:172.21.15.173:6812/2423786037 conn(0x7f655c040120 0x7f655c0425e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:43.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.193+0000 7f657b7fe640 10 monclient: get_auth_request con 0x7f655c040120 auth_method 0 2024-09-17T18:31:43.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.193+0000 7f657a7fc640 10 monclient(hunting): handle_auth_done global_id 64349 payload 824 2024-09-17T18:31:43.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.193+0000 7f657a7fc640 1 -- 172.21.15.121:0/3116052637 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f657c0a3be0 con 0x7f657c1280c0 2024-09-17T18:31:43.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.195+0000 7f65737fe640 10 client.64346.objecter ms_handle_connect 0x7f657c1280c0 2024-09-17T18:31:43.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.195+0000 7f65737fe640 10 client.64346.objecter resend_mon_ops 2024-09-17T18:31:43.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.195+0000 7f657a7fc640 1 -- 172.21.15.121:0/3116052637 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f657c1280c0 msgr2=0x7f657c125fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:43.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.195+0000 7f657a7fc640 1 --2- 172.21.15.121:0/3116052637 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f657c1280c0 0x7f657c125fe0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f657c128050 tx=0x7f656400b8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:43.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.195+0000 7f65737fe640 1 -- 172.21.15.121:0/3116052637 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6564016070 con 0x7f657c1280c0 2024-09-17T18:31:43.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.195+0000 7f65737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:43.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.195+0000 7f65737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:31:43.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.195+0000 7f65737fe640 10 monclient: dump: 2024-09-17T18:31:43.212 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:43.212 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:43.212 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:43.212 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:43.213 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:43.213 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:43.213 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:43.213 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:43.213 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:43.213 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:43.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.195+0000 7f65737fe640 10 client.64346.objecter ms_handle_connect 0x7f655c040120 2024-09-17T18:31:43.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.195+0000 7f657b7fe640 1 --2- 172.21.15.121:0/3116052637 >> v2:172.21.15.173:6812/2423786037 conn(0x7f655c040120 0x7f655c0425e0 secure :-1 s=READY pgs=4041 cs=0 l=1 rev1=1 crypto rx=0x7f656c006fd0 tx=0x7f656c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:43.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.196+0000 7f65737fe640 1 -- 172.21.15.121:0/3116052637 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6564005270 con 0x7f657c1280c0 2024-09-17T18:31:43.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.196+0000 7f65737fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:31:43.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.196+0000 7f65737fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:31:43.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.196+0000 7f65737fe640 1 -- 172.21.15.121:0/3116052637 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f657c1280c0 msgr2=0x7f657c125fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:43.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.196+0000 7f65737fe640 1 --2- 172.21.15.121:0/3116052637 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f657c1280c0 0x7f657c125fe0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f657c128050 tx=0x7f656400b8e0 comp rx=0 tx=0).stop 2024-09-17T18:31:43.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.196+0000 7f65737fe640 1 -- 172.21.15.121:0/3116052637 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f657c1280c0 msgr2=0x7f657c125fe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:31:43.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.196+0000 7f65737fe640 1 --2- 172.21.15.121:0/3116052637 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f657c1280c0 0x7f657c125fe0 secure :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f657c128050 tx=0x7f656400b8e0 comp rx=0 tx=0).stop 2024-09-17T18:31:43.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.202+0000 7f65814f6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:31:43.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.202+0000 7f65814f6640 1 --2- 172.21.15.121:0/3116052637 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f657c1cdc50 0x7f657c126520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:43.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.202+0000 7f65814f6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:43.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.202+0000 7f65814f6640 1 -- 172.21.15.121:0/3116052637 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f657c139680 con 0x7f657c1cdc50 2024-09-17T18:31:43.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.202+0000 7f657a7fc640 1 --2- 172.21.15.121:0/3116052637 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f657c1cdc50 0x7f657c126520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:43.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.202+0000 7f657a7fc640 10 monclient: get_auth_request con 0x7f657c1cdc50 auth_method 0 2024-09-17T18:31:43.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.202+0000 7f657a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:43.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.202+0000 7f657a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:43.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.202+0000 7f657a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:43.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.202+0000 7f657a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:43.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.202+0000 7f657a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:43.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.202+0000 7f657a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:43.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.202+0000 7f657a7fc640 10 monclient(hunting): handle_auth_done global_id 64352 payload 824 2024-09-17T18:31:43.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.202+0000 7f657a7fc640 1 -- 172.21.15.121:0/3116052637 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f657c06cd40 con 0x7f657c1cdc50 2024-09-17T18:31:43.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.204+0000 7f65737fe640 10 client.64346.objecter ms_handle_connect 0x7f657c1cdc50 2024-09-17T18:31:43.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.204+0000 7f657a7fc640 1 -- 172.21.15.121:0/3116052637 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f657c1cdc50 msgr2=0x7f657c126520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:43.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.204+0000 7f65737fe640 10 client.64346.objecter resend_mon_ops 2024-09-17T18:31:43.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.204+0000 7f657a7fc640 1 --2- 172.21.15.121:0/3116052637 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f657c1cdc50 0x7f657c126520 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f656400bfd0 tx=0x7f65640059d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:43.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.204+0000 7f65737fe640 1 -- 172.21.15.121:0/3116052637 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6564016050 con 0x7f657c1cdc50 2024-09-17T18:31:43.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.204+0000 7f65737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:43.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.204+0000 7f65737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:31:43.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.204+0000 7f65737fe640 10 monclient: dump: 2024-09-17T18:31:43.216 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:43.216 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:43.216 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:43.217 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:43.217 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:43.217 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:43.217 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:43.217 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:43.217 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:43.217 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:43.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.204+0000 7f65737fe640 1 -- 172.21.15.121:0/3116052637 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f6564017040 con 0x7f657c1cdc50 2024-09-17T18:31:43.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.204+0000 7f65737fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:31:43.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.204+0000 7f65737fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:31:43.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.204+0000 7f65737fe640 1 -- 172.21.15.121:0/3116052637 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f657c1cdc50 msgr2=0x7f657c126520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:43.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.204+0000 7f65737fe640 1 --2- 172.21.15.121:0/3116052637 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f657c1cdc50 0x7f657c126520 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f656400bfd0 tx=0x7f65640059d0 comp rx=0 tx=0).stop 2024-09-17T18:31:43.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.204+0000 7f65737fe640 1 -- 172.21.15.121:0/3116052637 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f657c1cdc50 msgr2=0x7f657c126520 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:31:43.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.204+0000 7f65737fe640 1 --2- 172.21.15.121:0/3116052637 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f657c1cdc50 0x7f657c126520 secure :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f656400bfd0 tx=0x7f65640059d0 comp rx=0 tx=0).stop 2024-09-17T18:31:43.218 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:31:43.218 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:31:43.218 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:31:43.218 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:31:43.218 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 46, 2024-09-17T18:31:43.218 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 281937, 2024-09-17T18:31:43.218 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:31:43.218 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:31:43.218 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:31:43.218 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:31:43.218 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 0, 2024-09-17T18:31:43.218 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:31:43.218 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:31:43.219 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:31:43.219 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:31:43.219 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:31:43.219 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:31:43.219 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:31:43.219 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:31:43.219 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:31:43.219 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:31:43.219 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:31:43.219 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:31:43.219 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:31:43.219 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:31:43.219 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:31:43.219 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:31:43.219 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:31:43.219 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:31:43.220 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:31:43.220 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:31:43.220 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:31:43.220 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:31:43.220 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:31:43.220 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:31:43.220 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:31:43.220 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:31:43.220 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:31:43.220 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:31:43.220 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:43.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:43.220 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:31:43.220 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:31:43.220 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:31:43.221 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:31:43.221 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:31:43.221 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:31:43.221 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:31:43.221 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:31:43.221 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:31:43.221 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:31:43.221 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:31:43.221 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:31:43.221 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:31:43.221 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:31:43.221 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:31:43.221 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:31:43.221 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:31:43.221 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:31:43.222 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:31:43.222 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:31:43.222 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:31:43.222 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:31:43.222 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:31:43.222 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:31:43.222 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:31:43.222 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:31:43.222 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:31:43.222 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:31:43.222 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:31:43.222 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:31:43.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:43.222 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:31:43.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:43.222 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:31:43.223 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:31:43.223 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:31:43.223 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:31:43.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:43.223 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:31:43.223 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:31:43.223 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:43.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:43.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:43.223 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:31:43.223 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:31:43.223 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:43.223 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:43.223 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:43.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:43.223 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:31:43.224 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:31:43.224 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:31:43.224 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:31:43.224 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:31:43.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:43.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:43.224 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:31:43.224 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:31:43.224 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:31:43.224 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:31:43.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:43.224 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:31:43.224 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:31:43.224 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:43.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:43.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:43.225 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:31:43.225 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:31:43.225 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:43.225 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:43.225 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:43.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:43.225 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:31:43.225 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:31:43.225 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:31:43.225 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:31:43.225 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:31:43.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:43.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:43.226 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:31:43.226 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:31:43.226 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:31:43.226 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:31:43.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:43.226 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:31:43.226 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:31:43.226 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:43.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:43.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:43.226 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:31:43.226 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:31:43.226 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:43.226 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:43.226 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:43.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:43.227 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:31:43.227 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:31:43.227 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:31:43.227 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:31:43.227 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:31:43.227 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:43.227 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:43.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:43.227 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:31:43.227 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:31:43.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:43.227 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:31:43.227 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:31:43.228 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:31:43.228 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:43.228 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:31:43.228 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:31:43.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:43.228 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:31:43.228 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:31:43.228 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 6 2024-09-17T18:31:43.228 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:43.228 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:31:43.228 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:31:43.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:43.228 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:31:43.229 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:31:43.229 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 6 2024-09-17T18:31:43.229 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:43.229 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:31:43.229 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": [ 2024-09-17T18:31:43.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:43.229 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:31:43.229 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:31:43.229 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:31:43.229 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:43.229 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:43.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:43.229 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:31:43.229 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:31:43.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.205+0000 7f65814f6640 10 librados: watch_flush enter 2024-09-17T18:31:43.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.205+0000 7f65814f6640 10 librados: watch_flush exit 2024-09-17T18:31:43.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.205+0000 7f65814f6640 20 client.64346.objecter shutdown clearing up homeless session... 2024-09-17T18:31:43.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.205+0000 7f65814f6640 10 client.64346.objecter successfully canceled tick 2024-09-17T18:31:43.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.205+0000 7f65814f6640 1 -- 172.21.15.121:0/3116052637 >> v2:172.21.15.173:6812/2423786037 conn(0x7f655c040120 msgr2=0x7f655c0425e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:43.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.205+0000 7f65814f6640 1 --2- 172.21.15.121:0/3116052637 >> v2:172.21.15.173:6812/2423786037 conn(0x7f655c040120 0x7f655c0425e0 secure :-1 s=READY pgs=4041 cs=0 l=1 rev1=1 crypto rx=0x7f656c006fd0 tx=0x7f656c008040 comp rx=0 tx=0).stop 2024-09-17T18:31:43.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.205+0000 7f65814f6640 10 monclient: shutdown 2024-09-17T18:31:43.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.205+0000 7f65814f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:31:43.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.205+0000 7f65814f6640 1 -- 172.21.15.121:0/3116052637 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f657c09d830 msgr2=0x7f657c137ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:43.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.205+0000 7f65814f6640 1 --2- 172.21.15.121:0/3116052637 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f657c09d830 0x7f657c137ec0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f65740093e0 tx=0x7f6574004a40 comp rx=0 tx=0).stop 2024-09-17T18:31:43.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.205+0000 7f657b7fe640 1 -- 172.21.15.121:0/3116052637 reap_dead start 2024-09-17T18:31:43.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.205+0000 7f65814f6640 1 -- 172.21.15.121:0/3116052637 shutdown_connections 2024-09-17T18:31:43.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.205+0000 7f65814f6640 1 --2- 172.21.15.121:0/3116052637 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f657c09d830 0x7f657c137ec0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:43.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.205+0000 7f65814f6640 1 -- 172.21.15.121:0/3116052637 >> 172.21.15.121:0/3116052637 conn(0x7f657c0ab880 msgr2=0x7f657c0a4000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:31:43.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.205+0000 7f65814f6640 1 -- 172.21.15.121:0/3116052637 shutdown_connections 2024-09-17T18:31:43.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.205+0000 7f65814f6640 1 -- 172.21.15.121:0/3116052637 wait complete. 2024-09-17T18:31:43.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.205+0000 7f65814f6640 1 librados: shutdown 2024-09-17T18:31:43.232 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T18:31:43.233 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-17T18:31:43.233 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T18:31:43.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.327+0000 7f46e3cc6640 10 monclient: get_monmap_and_config 2024-09-17T18:31:43.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.327+0000 7f46e3cc6640 10 monclient: build_initial_monmap 2024-09-17T18:31:43.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.327+0000 7f46e3cc6640 10 monclient: monmap: 2024-09-17T18:31:43.330 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:31:43.330 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:43.330 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:31:43.329128+0000 2024-09-17T18:31:43.330 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:31:43.329128+0000 2024-09-17T18:31:43.330 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:31:43.330 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:43.331 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:31:43.331 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:31:43.331 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:31:43.331 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:31:43.331 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:31:43.331 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:43.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.328+0000 7f46e3cc6640 1 Processor -- start 2024-09-17T18:31:43.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.328+0000 7f46e3cc6640 1 -- start start 2024-09-17T18:31:43.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.328+0000 7f46e3cc6640 10 monclient: init 2024-09-17T18:31:43.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.328+0000 7f46e3cc6640 10 monclient: _reopen_session rank -1 2024-09-17T18:31:43.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.328+0000 7f46e3cc6640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-17T18:31:43.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.328+0000 7f46e3cc6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f46dc09e550 addr v1:172.21.15.173:6789/0 2024-09-17T18:31:43.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.328+0000 7f46e3cc6640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f46dc09f040 0x7f46dc13a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:43.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.328+0000 7f46e3cc6640 10 monclient(hunting): picked mon.noname-a con 0x7f46dc09f040 addr v2:172.21.15.121:3300/0 2024-09-17T18:31:43.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.328+0000 7f46e3cc6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f46dc13ad90 addr v1:172.21.15.121:6789/0 2024-09-17T18:31:43.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.328+0000 7f46e3cc6640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46dc0a4460 con 0x7f46dc13ad90 2024-09-17T18:31:43.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.328+0000 7f46e3cc6640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46dc0a2b60 con 0x7f46dc09e550 2024-09-17T18:31:43.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.328+0000 7f46e3cc6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:43.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.328+0000 7f46e3cc6640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f46dc0a4950 con 0x7f46dc09f040 2024-09-17T18:31:43.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.328+0000 7f46e1261640 1 -- >> v2:172.21.15.121:3300/0 conn(0x7f46dc09f040 msgr2=0x7f46dc13a850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:31:43.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.328+0000 7f46e3cc6640 10 monclient(hunting): _renew_subs 2024-09-17T18:31:43.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.328+0000 7f46e2263640 1 -- >> v1:172.21.15.121:6789/0 conn(0x7f46dc13ad90 legacy=0x7f46dc13d180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.121:6789/0 2024-09-17T18:31:43.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.328+0000 7f46e3cc6640 10 monclient(hunting): authenticate will time out at 1507.045957s 2024-09-17T18:31:43.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.328+0000 7f46e1261640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f46dc09f040 0x7f46dc13a850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:31:43.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.329+0000 7f46e1a62640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f46dc09e550 0x7f46dc09e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.121:48504/0 (socket says 172.21.15.121:48504) 2024-09-17T18:31:43.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.329+0000 7f46e1a62640 1 -- 172.21.15.121:0/3411486034 learned_addr learned my addr 172.21.15.121:0/3411486034 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:31:43.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.331+0000 7f46e1a62640 1 -- 172.21.15.121:0/3411486034 >> v1:172.21.15.173:6789/0 conn(0x7f46dc09e550 legacy=0x7f46dc09e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:43.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.331+0000 7f46e0a60640 1 -- 172.21.15.121:0/3411486034 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1642433899 0 0) 0x7f46dc0a2b60 con 0x7f46dc09e550 2024-09-17T18:31:43.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.331+0000 7f46e0a60640 10 monclient(hunting): _negotiate 2024-09-17T18:31:43.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.331+0000 7f46e0a60640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:43.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.331+0000 7f46e0a60640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:43.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.331+0000 7f46e0a60640 10 monclient(hunting): my global_id is 84142 2024-09-17T18:31:43.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.331+0000 7f46e0a60640 1 -- 172.21.15.121:0/3411486034 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46c8003520 con 0x7f46dc09e550 2024-09-17T18:31:43.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.331+0000 7f46e0a60640 1 -- 172.21.15.121:0/3411486034 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4257350616 0 0) 0x7f46c8003520 con 0x7f46dc09e550 2024-09-17T18:31:43.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.331+0000 7f46e0a60640 1 -- 172.21.15.121:0/3411486034 >> v2:172.21.15.121:3300/0 conn(0x7f46dc09f040 msgr2=0x7f46dc13a850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:31:43.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.331+0000 7f46e0a60640 1 --2- 172.21.15.121:0/3411486034 >> v2:172.21.15.121:3300/0 conn(0x7f46dc09f040 0x7f46dc13a850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:43.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.331+0000 7f46e0a60640 1 -- 172.21.15.121:0/3411486034 >> v1:172.21.15.121:6789/0 conn(0x7f46dc13ad90 legacy=0x7f46dc13d180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:31:43.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.331+0000 7f46e0a60640 10 monclient: _finish_hunting 0 2024-09-17T18:31:43.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.331+0000 7f46e0a60640 1 monclient: found mon.noname-c-legacy 2024-09-17T18:31:43.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.331+0000 7f46e0a60640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:31:43.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e0a60640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.173:6789/0 2024-09-17T18:31:43.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e0a60640 1 -- 172.21.15.121:0/3411486034 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46dc09f840 con 0x7f46dc09e550 2024-09-17T18:31:43.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e0a60640 10 monclient: _finish_auth 0 2024-09-17T18:31:43.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e0a60640 10 monclient: _check_auth_tickets 2024-09-17T18:31:43.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e0a60640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:31:43.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e0a60640 1 -- 172.21.15.121:0/3411486034 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4002019445 0 0) 0x7f46d8001a60 con 0x7f46dc09e550 2024-09-17T18:31:43.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e0a60640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:43.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e0a60640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-17T18:31:43.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e0a60640 10 monclient: dump: 2024-09-17T18:31:43.339 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:43.339 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:43.339 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:43.339 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:43.339 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:43.339 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:43.339 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:43.339 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:43.339 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:43.340 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:43.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e0a60640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] but i'm connected to v1:172.21.15.173:6789/0, reconnecting 2024-09-17T18:31:43.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e0a60640 10 monclient: _reopen_session rank -1 2024-09-17T18:31:43.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e0a60640 1 -- 172.21.15.121:0/3411486034 >> v1:172.21.15.173:6789/0 conn(0x7f46dc09e550 legacy=0x7f46dc09e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:43.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e0a60640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T18:31:43.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e0a60640 1 --2- 172.21.15.121:0/3411486034 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46c8003a20 0x7f46c8003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:43.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e0a60640 10 monclient(hunting): picked mon.c con 0x7f46c8003a20 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:31:43.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e0a60640 1 --2- 172.21.15.121:0/3411486034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46c8004800 0x7f46c8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:43.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e0a60640 10 monclient(hunting): picked mon.a con 0x7f46c8004800 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:31:43.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e0a60640 1 --2- 172.21.15.121:0/3411486034 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46c8005410 0x7f46c800ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:43.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e0a60640 10 monclient(hunting): picked mon.b con 0x7f46c8005410 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:31:43.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e0a60640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:43.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e0a60640 1 -- 172.21.15.121:0/3411486034 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f46c800c320 con 0x7f46c8004800 2024-09-17T18:31:43.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e0a60640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:43.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e0a60640 1 -- 172.21.15.121:0/3411486034 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f46c800c4d0 con 0x7f46c8005410 2024-09-17T18:31:43.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e0a60640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:43.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e0a60640 1 -- 172.21.15.121:0/3411486034 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f46c800c6d0 con 0x7f46c8003a20 2024-09-17T18:31:43.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e0a60640 10 monclient(hunting): _renew_subs 2024-09-17T18:31:43.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e0a60640 10 monclient(hunting): _finish_auth 0 2024-09-17T18:31:43.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e1261640 1 -- 172.21.15.121:0/3411486034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46c8004800 msgr2=0x7f46c8004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:31:43.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e1261640 1 --2- 172.21.15.121:0/3411486034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46c8004800 0x7f46c8004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:31:43.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e1a62640 1 --2- 172.21.15.121:0/3411486034 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46c8003a20 0x7f46c8003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:43.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e1a62640 10 monclient(hunting): get_auth_request con 0x7f46c8003a20 auth_method 0 2024-09-17T18:31:43.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e1a62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:43.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e1a62640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:43.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e1a62640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:31:43.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e2263640 1 --2- 172.21.15.121:0/3411486034 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46c8005410 0x7f46c800ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:43.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e1a62640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:43.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e1a62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:43.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e1a62640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:31:43.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e2263640 10 monclient(hunting): get_auth_request con 0x7f46c8005410 auth_method 0 2024-09-17T18:31:43.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e2263640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:43.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e2263640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:43.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e2263640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:31:43.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e2263640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:43.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e2263640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:43.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e2263640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:31:43.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e1a62640 10 monclient(hunting): handle_auth_done global_id 84142 payload 293 2024-09-17T18:31:43.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e1a62640 1 -- 172.21.15.121:0/3411486034 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46c8005410 msgr2=0x7f46c800ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:31:43.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e1a62640 1 --2- 172.21.15.121:0/3411486034 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46c8005410 0x7f46c800ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:43.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e1a62640 1 -- 172.21.15.121:0/3411486034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46c8004800 msgr2=0x7f46c8004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:31:43.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e1a62640 1 --2- 172.21.15.121:0/3411486034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46c8004800 0x7f46c8004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:43.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e1a62640 10 monclient: _finish_hunting 0 2024-09-17T18:31:43.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e1a62640 1 monclient: found mon.c 2024-09-17T18:31:43.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e1a62640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:31:43.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e2263640 1 -- 172.21.15.121:0/3411486034 reap_dead start 2024-09-17T18:31:43.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e1a62640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:31:43.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e1a62640 1 -- 172.21.15.121:0/3411486034 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46c8003520 con 0x7f46c8003a20 2024-09-17T18:31:43.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e1a62640 10 monclient: _finish_auth 0 2024-09-17T18:31:43.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e1a62640 10 monclient: _check_auth_tickets 2024-09-17T18:31:43.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e1a62640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:31:43.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e3cc6640 5 monclient: authenticate success, global_id 84142 2024-09-17T18:31:43.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.332+0000 7f46e3cc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:31:43.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.335+0000 7f46e1a62640 1 -- 172.21.15.121:0/3411486034 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46c8003a20 msgr2=0x7f46c8003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:43.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.335+0000 7f46e1a62640 1 --2- 172.21.15.121:0/3411486034 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46c8003a20 0x7f46c8003e40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f46d80014d0 tx=0x7f46c8005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7916a7cb98a2cbe server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:43.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.335+0000 7f46e0a60640 1 -- 172.21.15.121:0/3411486034 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d8036070 con 0x7f46c8003a20 2024-09-17T18:31:43.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.335+0000 7f46e0a60640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:43.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.335+0000 7f46e0a60640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:31:43.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.335+0000 7f46e0a60640 10 monclient: dump: 2024-09-17T18:31:43.351 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:43.351 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:43.351 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:43.351 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:43.351 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:43.351 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:43.351 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:43.351 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:43.351 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:43.351 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:43.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.335+0000 7f46e3cc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:31:43.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.335+0000 7f46e0a60640 1 -- 172.21.15.121:0/3411486034 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46d8005af0 con 0x7f46c8003a20 2024-09-17T18:31:43.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.335+0000 7f46e0a60640 10 monclient: handle_config config(0 keys) 2024-09-17T18:31:43.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.335+0000 7f46e3cc6640 10 monclient: get_monmap_and_config success 2024-09-17T18:31:43.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.335+0000 7f46e3cc6640 4 set_mon_vals no callback set 2024-09-17T18:31:43.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.335+0000 7f46e0a60640 1 -- 172.21.15.121:0/3411486034 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d800c0e0 con 0x7f46c8003a20 2024-09-17T18:31:43.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.335+0000 7f46e0a60640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:43.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.335+0000 7f46e0a60640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:31:43.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.335+0000 7f46e0a60640 10 monclient: dump: 2024-09-17T18:31:43.352 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:43.352 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:43.352 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:43.352 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:43.352 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:43.352 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:43.352 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:43.353 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:43.353 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:43.353 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:43.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.336+0000 7f46e3cc6640 10 monclient: shutdown 2024-09-17T18:31:43.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.336+0000 7f46e3cc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:31:43.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.336+0000 7f46e3cc6640 1 -- 172.21.15.121:0/3411486034 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46c8003a20 msgr2=0x7f46c8003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:43.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.336+0000 7f46e3cc6640 1 --2- 172.21.15.121:0/3411486034 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46c8003a20 0x7f46c8003e40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f46d80014d0 tx=0x7f46c8005830 comp rx=0 tx=0).stop 2024-09-17T18:31:43.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.336+0000 7f46e3cc6640 1 -- 172.21.15.121:0/3411486034 shutdown_connections 2024-09-17T18:31:43.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.336+0000 7f46e3cc6640 1 --2- 172.21.15.121:0/3411486034 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46c8003a20 0x7f46c8003e40 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:43.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.336+0000 7f46e3cc6640 1 -- 172.21.15.121:0/3411486034 >> 172.21.15.121:0/3411486034 conn(0x7f46dc0a7830 msgr2=0x7f46dc0a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:31:43.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.336+0000 7f46e3cc6640 1 -- 172.21.15.121:0/3411486034 shutdown_connections 2024-09-17T18:31:43.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.336+0000 7f46e3cc6640 1 -- 172.21.15.121:0/3411486034 wait complete. 2024-09-17T18:31:43.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.336+0000 7f46e3cc6640 10 monclient: build_initial_monmap 2024-09-17T18:31:43.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.336+0000 7f46e3cc6640 10 monclient: monmap: 2024-09-17T18:31:43.354 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:31:43.354 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:31:43.354 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:31:43.354 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:31:43.354 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:31:43.354 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:43.354 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:31:43.354 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:31:43.354 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:31:43.354 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:43.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.337+0000 7f46e3cc6640 1 librados: starting msgr at 2024-09-17T18:31:43.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.337+0000 7f46e3cc6640 1 librados: starting objecter 2024-09-17T18:31:43.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.337+0000 7f46e3cc6640 1 Processor -- start 2024-09-17T18:31:43.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.337+0000 7f46e3cc6640 1 -- start start 2024-09-17T18:31:43.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.337+0000 7f46e3cc6640 1 librados: setting wanted keys 2024-09-17T18:31:43.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.337+0000 7f46e3cc6640 1 librados: calling monclient init 2024-09-17T18:31:43.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.337+0000 7f46e3cc6640 10 monclient: init 2024-09-17T18:31:43.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.337+0000 7f46e3cc6640 10 monclient: _reopen_session rank -1 2024-09-17T18:31:43.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.337+0000 7f46e3cc6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:31:43.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.337+0000 7f46e3cc6640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46c8003a20 0x7f46dc0a1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:43.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.337+0000 7f46e3cc6640 10 monclient(hunting): picked mon.noname-a con 0x7f46c8003a20 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:31:43.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.337+0000 7f46e3cc6640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46dc136c60 0x7f46dc1218e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:43.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.337+0000 7f46e3cc6640 10 monclient(hunting): picked mon.noname-b con 0x7f46dc136c60 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:31:43.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.337+0000 7f46e3cc6640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46dc1370e0 0x7f46dc121e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:43.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.337+0000 7f46e1a62640 1 -- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46c8003a20 msgr2=0x7f46dc0a1a80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:31:43.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.337+0000 7f46e3cc6640 10 monclient(hunting): picked mon.noname-c con 0x7f46dc1370e0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:31:43.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.337+0000 7f46e3cc6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:43.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.337+0000 7f46e3cc6640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f46dc09f490 con 0x7f46c8003a20 2024-09-17T18:31:43.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.337+0000 7f46d27fc640 1 -- DispatchQueue::entry inject delay of 0.002000 2024-09-17T18:31:43.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.337+0000 7f46e1a62640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46c8003a20 0x7f46dc0a1a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:31:43.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.337+0000 7f46e3cc6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:43.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.337+0000 7f46e3cc6640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f46dc09dce0 con 0x7f46dc136c60 2024-09-17T18:31:43.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.337+0000 7f46e3cc6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:43.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.337+0000 7f46e3cc6640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f46dc123d80 con 0x7f46dc1370e0 2024-09-17T18:31:43.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.337+0000 7f46e3cc6640 10 monclient(hunting): _renew_subs 2024-09-17T18:31:43.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.337+0000 7f46e3cc6640 10 monclient(hunting): authenticate will time out at 1507.054720s 2024-09-17T18:31:43.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.337+0000 7f46e2263640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46dc1370e0 0x7f46dc121e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:43.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.337+0000 7f46e2263640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46dc1370e0 0x7f46dc121e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:56108/0 (socket says 172.21.15.121:56108) 2024-09-17T18:31:43.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.337+0000 7f46e2263640 1 -- 172.21.15.121:0/3425641915 learned_addr learned my addr 172.21.15.121:0/3425641915 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:31:43.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.337+0000 7f46e2263640 10 monclient(hunting): get_auth_request con 0x7f46dc1370e0 auth_method 0 2024-09-17T18:31:43.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.337+0000 7f46e2263640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:43.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.337+0000 7f46e2263640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:43.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.337+0000 7f46e2263640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:43.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.337+0000 7f46e2263640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:43.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.337+0000 7f46e2263640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:43.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.337+0000 7f46e2263640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:43.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.338+0000 7f46e2263640 10 monclient(hunting): handle_auth_done global_id 64355 payload 824 2024-09-17T18:31:43.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.338+0000 7f46e1261640 1 --2- 172.21.15.121:0/3425641915 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46dc136c60 0x7f46dc1218e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:43.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.338+0000 7f46e2263640 1 -- 172.21.15.121:0/3425641915 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46dc136c60 msgr2=0x7f46dc1218e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:43.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.338+0000 7f46e2263640 1 --2- 172.21.15.121:0/3425641915 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46dc136c60 0x7f46dc1218e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:43.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.338+0000 7f46e2263640 1 -- 172.21.15.121:0/3425641915 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46c8003a20 msgr2=0x7f46dc0a1a80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:31:43.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.338+0000 7f46e2263640 1 --2- 172.21.15.121:0/3425641915 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46c8003a20 0x7f46dc0a1a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:43.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.338+0000 7f46e2263640 10 monclient: _finish_hunting 0 2024-09-17T18:31:43.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.338+0000 7f46e2263640 1 monclient: found mon.noname-c 2024-09-17T18:31:43.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.338+0000 7f46e2263640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:31:43.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.338+0000 7f46e2263640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:31:43.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.338+0000 7f46e2263640 1 -- 172.21.15.121:0/3425641915 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46dc124340 con 0x7f46dc1370e0 2024-09-17T18:31:43.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.340+0000 7f46e2263640 1 -- 172.21.15.121:0/3425641915 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46dc1370e0 msgr2=0x7f46dc121e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:43.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.340+0000 7f46d27fc640 10 client.?.objecter ms_handle_connect 0x7f46dc1370e0 2024-09-17T18:31:43.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.340+0000 7f46e2263640 1 --2- 172.21.15.121:0/3425641915 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46dc1370e0 0x7f46dc121e20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f46dc074b70 tx=0x7f46d400eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:43.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.340+0000 7f46d27fc640 10 client.?.objecter resend_mon_ops 2024-09-17T18:31:43.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.340+0000 7f46d27fc640 1 -- 172.21.15.121:0/3425641915 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d4017440 con 0x7f46dc1370e0 2024-09-17T18:31:43.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.340+0000 7f46d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:43.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.340+0000 7f46d27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:31:43.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.340+0000 7f46d27fc640 10 monclient: dump: 2024-09-17T18:31:43.358 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:43.358 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:43.358 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:43.358 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:43.358 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:43.358 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:43.358 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:43.358 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:43.358 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:43.358 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:43.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.340+0000 7f46d27fc640 10 monclient: _finish_auth 0 2024-09-17T18:31:43.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.340+0000 7f46d27fc640 10 monclient: _check_auth_tickets 2024-09-17T18:31:43.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.340+0000 7f46d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:31:43.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.340+0000 7f46e3cc6640 5 monclient: authenticate success, global_id 64355 2024-09-17T18:31:43.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.340+0000 7f46d27fc640 1 -- 172.21.15.121:0/3425641915 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46d40175e0 con 0x7f46dc1370e0 2024-09-17T18:31:43.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.340+0000 7f46d27fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:31:43.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.340+0000 7f46d27fc640 1 -- 172.21.15.121:0/3425641915 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d40179d0 con 0x7f46dc1370e0 2024-09-17T18:31:43.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.340+0000 7f46e3cc6640 10 monclient: _renew_subs 2024-09-17T18:31:43.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.340+0000 7f46e3cc6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:31:43.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.340+0000 7f46e3cc6640 1 -- 172.21.15.121:0/3425641915 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46dc124620 con 0x7f46dc1370e0 2024-09-17T18:31:43.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.340+0000 7f46d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:43.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.340+0000 7f46d27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:31:43.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.340+0000 7f46d3fff640 4 set_mon_vals no callback set 2024-09-17T18:31:43.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.340+0000 7f46d27fc640 10 monclient: dump: 2024-09-17T18:31:43.359 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:43.359 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:43.360 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:43.360 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:43.360 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:43.360 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:43.360 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:43.360 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:43.360 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:43.360 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:43.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.340+0000 7f46e3cc6640 10 client.64355.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:31:43.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.340+0000 7f46e3cc6640 10 monclient: _renew_subs 2024-09-17T18:31:43.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.340+0000 7f46e3cc6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:31:43.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.340+0000 7f46e3cc6640 1 -- 172.21.15.121:0/3425641915 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f46dc1375d0 con 0x7f46dc1370e0 2024-09-17T18:31:43.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.340+0000 7f46e3cc6640 1 librados: init done 2024-09-17T18:31:43.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.341+0000 7f46e3cc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:31:43.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.341+0000 7f46e3cc6640 1 --2- 172.21.15.121:0/3425641915 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46dc122360 0x7f46dc071560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:43.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.341+0000 7f46e3cc6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:43.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.341+0000 7f46e3cc6640 1 -- 172.21.15.121:0/3425641915 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f46dc09f610 con 0x7f46dc122360 2024-09-17T18:31:43.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.341+0000 7f46d27fc640 1 -- 172.21.15.121:0/3425641915 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f46d4020440 con 0x7f46dc1370e0 2024-09-17T18:31:43.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.341+0000 7f46e1a62640 1 --2- 172.21.15.121:0/3425641915 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46dc122360 0x7f46dc071560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:43.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.341+0000 7f46e1a62640 10 monclient: get_auth_request con 0x7f46dc122360 auth_method 0 2024-09-17T18:31:43.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.341+0000 7f46e1a62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:43.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.341+0000 7f46e1a62640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:43.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.341+0000 7f46e1a62640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:43.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.341+0000 7f46e1a62640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:43.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.341+0000 7f46e1a62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:43.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.341+0000 7f46e1a62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:43.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.341+0000 7f46d27fc640 1 --2- 172.21.15.121:0/3425641915 >> v2:172.21.15.173:6812/2423786037 conn(0x7f46c0040170 0x7f46c0042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:43.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.341+0000 7f46d27fc640 1 -- 172.21.15.121:0/3425641915 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(486..486 src has 247..486) ==== 5880+0+0 (secure 0 0 0) 0x7f46d40542a0 con 0x7f46dc1370e0 2024-09-17T18:31:43.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.341+0000 7f46d27fc640 10 client.64355.objecter ms_dispatch 0x7f46dc000b90 osd_map(486..486 src has 247..486) 2024-09-17T18:31:43.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.341+0000 7f46d27fc640 3 client.64355.objecter handle_osd_map got epochs [486,486] > 0 2024-09-17T18:31:43.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.341+0000 7f46d27fc640 3 client.64355.objecter handle_osd_map decoding full epoch 486 2024-09-17T18:31:43.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.341+0000 7f46e1a62640 10 monclient(hunting): handle_auth_done global_id 84145 payload 824 2024-09-17T18:31:43.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.341+0000 7f46e1a62640 1 -- 172.21.15.121:0/3425641915 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f46dc122740 con 0x7f46dc122360 2024-09-17T18:31:43.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.341+0000 7f46d27fc640 20 client.64355.objecter dump_active .. 0 homeless 2024-09-17T18:31:43.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.341+0000 7f46e1261640 1 --2- 172.21.15.121:0/3425641915 >> v2:172.21.15.173:6812/2423786037 conn(0x7f46c0040170 0x7f46c0042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:43.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.342+0000 7f46e1261640 10 monclient: get_auth_request con 0x7f46c0040170 auth_method 0 2024-09-17T18:31:43.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.344+0000 7f46e1a62640 1 -- 172.21.15.121:0/3425641915 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46dc122360 msgr2=0x7f46dc071560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:43.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.344+0000 7f46d27fc640 10 client.64355.objecter ms_handle_connect 0x7f46dc122360 2024-09-17T18:31:43.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.344+0000 7f46d27fc640 10 client.64355.objecter resend_mon_ops 2024-09-17T18:31:43.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.344+0000 7f46e1a62640 1 --2- 172.21.15.121:0/3425641915 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46dc122360 0x7f46dc071560 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f46d8033000 tx=0x7f46dc09fa10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:43.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.344+0000 7f46d27fc640 1 -- 172.21.15.121:0/3425641915 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d8036070 con 0x7f46dc122360 2024-09-17T18:31:43.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.344+0000 7f46d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:43.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.344+0000 7f46d27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:31:43.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.344+0000 7f46d27fc640 10 monclient: dump: 2024-09-17T18:31:43.362 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:43.362 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:43.363 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:43.363 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:43.363 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:43.363 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:43.363 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:43.363 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:43.363 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:43.363 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:43.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.344+0000 7f46d27fc640 10 client.64355.objecter ms_handle_connect 0x7f46c0040170 2024-09-17T18:31:43.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.344+0000 7f46e1261640 1 --2- 172.21.15.121:0/3425641915 >> v2:172.21.15.173:6812/2423786037 conn(0x7f46c0040170 0x7f46c0042630 secure :-1 s=READY pgs=4042 cs=0 l=1 rev1=1 crypto rx=0x7f46dc1cc320 tx=0x7f46cc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:43.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.344+0000 7f46d27fc640 1 -- 172.21.15.121:0/3425641915 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f46d8037040 con 0x7f46dc122360 2024-09-17T18:31:43.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.344+0000 7f46d27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:31:43.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.344+0000 7f46d27fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:31:43.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.344+0000 7f46d27fc640 1 -- 172.21.15.121:0/3425641915 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46dc122360 msgr2=0x7f46dc071560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:43.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.344+0000 7f46d27fc640 1 --2- 172.21.15.121:0/3425641915 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46dc122360 0x7f46dc071560 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f46d8033000 tx=0x7f46dc09fa10 comp rx=0 tx=0).stop 2024-09-17T18:31:43.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.344+0000 7f46d27fc640 1 -- 172.21.15.121:0/3425641915 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46dc122360 msgr2=0x7f46dc071560 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:31:43.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.344+0000 7f46d27fc640 1 --2- 172.21.15.121:0/3425641915 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46dc122360 0x7f46dc071560 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:31:43.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.350+0000 7f46e3cc6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:31:43.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.350+0000 7f46e3cc6640 1 --2- 172.21.15.121:0/3425641915 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46dc071aa0 0x7f46dc071eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:31:43.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.350+0000 7f46e3cc6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:31:43.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.350+0000 7f46e3cc6640 1 -- 172.21.15.121:0/3425641915 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f46dc123f30 con 0x7f46dc071aa0 2024-09-17T18:31:43.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.351+0000 7f46e1a62640 1 --2- 172.21.15.121:0/3425641915 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46dc071aa0 0x7f46dc071eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:31:43.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.351+0000 7f46e1a62640 10 monclient: get_auth_request con 0x7f46dc071aa0 auth_method 0 2024-09-17T18:31:43.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.351+0000 7f46e1a62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:31:43.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.351+0000 7f46e1a62640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:31:43.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.351+0000 7f46e1a62640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:31:43.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.351+0000 7f46e1a62640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:31:43.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.351+0000 7f46e1a62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:31:43.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.351+0000 7f46e1a62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:31:43.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.351+0000 7f46e1a62640 10 monclient(hunting): handle_auth_done global_id 84148 payload 824 2024-09-17T18:31:43.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.351+0000 7f46e1a62640 1 -- 172.21.15.121:0/3425641915 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f46dc09e240 con 0x7f46dc071aa0 2024-09-17T18:31:43.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.353+0000 7f46e1a62640 1 -- 172.21.15.121:0/3425641915 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46dc071aa0 msgr2=0x7f46dc071eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:31:43.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.353+0000 7f46e1a62640 1 --2- 172.21.15.121:0/3425641915 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46dc071aa0 0x7f46dc071eb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f46d80014d0 tx=0x7f46d8007b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:31:43.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.353+0000 7f46d27fc640 10 client.64355.objecter ms_handle_connect 0x7f46dc071aa0 2024-09-17T18:31:43.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.353+0000 7f46d27fc640 10 client.64355.objecter resend_mon_ops 2024-09-17T18:31:43.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.353+0000 7f46d27fc640 1 -- 172.21.15.121:0/3425641915 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d8031650 con 0x7f46dc071aa0 2024-09-17T18:31:43.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.353+0000 7f46d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:31:43.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.353+0000 7f46d27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:31:43.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.353+0000 7f46d27fc640 10 monclient: dump: 2024-09-17T18:31:43.367 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:31:43.367 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:31:43.367 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:43.367 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:31:43.367 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:31:43.367 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:31:43.367 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:31:43.367 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:31:43.367 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:31:43.367 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:31:43.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.354+0000 7f46d27fc640 1 -- 172.21.15.121:0/3425641915 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4543 (secure 0 0 0) 0x7f46d8031650 con 0x7f46dc071aa0 2024-09-17T18:31:43.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.354+0000 7f46d27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:31:43.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.354+0000 7f46d27fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:31:43.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.354+0000 7f46d27fc640 1 -- 172.21.15.121:0/3425641915 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46dc071aa0 msgr2=0x7f46dc071eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:43.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.354+0000 7f46d27fc640 1 --2- 172.21.15.121:0/3425641915 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46dc071aa0 0x7f46dc071eb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f46d80014d0 tx=0x7f46d8007b80 comp rx=0 tx=0).stop 2024-09-17T18:31:43.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.354+0000 7f46d27fc640 1 -- 172.21.15.121:0/3425641915 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46dc071aa0 msgr2=0x7f46dc071eb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:31:43.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.354+0000 7f46d27fc640 1 --2- 172.21.15.121:0/3425641915 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46dc071aa0 0x7f46dc071eb0 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f46d80014d0 tx=0x7f46d8007b80 comp rx=0 tx=0).stop 2024-09-17T18:31:43.368 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:31:43.368 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:31:43.368 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:31:43.368 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "leader", 2024-09-17T18:31:43.368 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 46, 2024-09-17T18:31:43.368 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 28453, 2024-09-17T18:31:43.368 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:31:43.368 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:31:43.368 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:31:43.369 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:31:43.369 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 0, 2024-09-17T18:31:43.369 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:31:43.369 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:31:43.369 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:31:43.369 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:31:43.369 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:31:43.369 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:31:43.369 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:31:43.369 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:31:43.369 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:31:43.369 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:31:43.369 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:31:43.369 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:31:43.369 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:31:43.369 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:31:43.369 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:31:43.370 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:31:43.370 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:31:43.370 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:31:43.370 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:31:43.370 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:31:43.370 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:31:43.370 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:31:43.370 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:31:43.370 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:31:43.370 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:31:43.370 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:31:43.370 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:31:43.370 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:31:43.370 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:43.370 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:43.370 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:31:43.371 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:31:43.371 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:31:43.371 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:31:43.371 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:31:43.371 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:31:43.371 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:31:43.371 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:31:43.371 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:31:43.371 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:31:43.371 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:31:43.371 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:31:43.371 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:31:43.371 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:31:43.371 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:31:43.371 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:31:43.371 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:31:43.372 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:31:43.372 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:31:43.372 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:31:43.372 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:31:43.372 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:31:43.372 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:31:43.372 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:31:43.372 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:31:43.372 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:31:43.372 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:31:43.372 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:31:43.372 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:31:43.372 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:31:43.372 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:43.372 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:31:43.373 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:43.373 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:31:43.373 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:31:43.373 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:31:43.373 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:31:43.373 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:43.373 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:31:43.373 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:31:43.373 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:43.373 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:43.373 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:43.373 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:31:43.373 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:31:43.373 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:43.373 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:43.374 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:43.374 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:43.374 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:31:43.374 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:31:43.374 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:31:43.374 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:31:43.374 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:31:43.374 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:43.374 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:43.374 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:31:43.374 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:31:43.374 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:31:43.374 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:31:43.374 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:43.374 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:31:43.374 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:31:43.374 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:43.375 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:43.375 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:43.375 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:31:43.375 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:31:43.375 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:43.375 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:43.375 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:43.375 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:43.375 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:31:43.375 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:31:43.375 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:31:43.375 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:31:43.375 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:31:43.375 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:43.375 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:43.375 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:31:43.376 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:31:43.376 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:31:43.376 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:31:43.376 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:43.376 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:31:43.376 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:31:43.376 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:43.376 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:43.376 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:43.376 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:31:43.376 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:31:43.376 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:31:43.376 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:43.376 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:43.376 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:43.376 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:31:43.377 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:31:43.377 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:31:43.377 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:31:43.377 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:31:43.377 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:43.377 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:43.377 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:43.377 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:31:43.377 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:31:43.377 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:43.377 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:31:43.377 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:31:43.377 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:31:43.377 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:43.377 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:31:43.378 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:31:43.378 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:31:43.378 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:31:43.378 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:31:43.378 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 3 2024-09-17T18:31:43.378 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:31:43.378 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:31:43.378 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:31:43.378 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:31:43.378 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:31:43.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.354+0000 7f46e3cc6640 10 librados: watch_flush enter 2024-09-17T18:31:43.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.354+0000 7f46e3cc6640 10 librados: watch_flush exit 2024-09-17T18:31:43.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.354+0000 7f46e3cc6640 20 client.64355.objecter shutdown clearing up homeless session... 2024-09-17T18:31:43.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.354+0000 7f46e3cc6640 10 client.64355.objecter successfully canceled tick 2024-09-17T18:31:43.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.354+0000 7f46e3cc6640 1 -- 172.21.15.121:0/3425641915 >> v2:172.21.15.173:6812/2423786037 conn(0x7f46c0040170 msgr2=0x7f46c0042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:43.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.354+0000 7f46e3cc6640 1 --2- 172.21.15.121:0/3425641915 >> v2:172.21.15.173:6812/2423786037 conn(0x7f46c0040170 0x7f46c0042630 secure :-1 s=READY pgs=4042 cs=0 l=1 rev1=1 crypto rx=0x7f46dc1cc320 tx=0x7f46cc008040 comp rx=0 tx=0).stop 2024-09-17T18:31:43.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.354+0000 7f46e3cc6640 10 monclient: shutdown 2024-09-17T18:31:43.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.354+0000 7f46e3cc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:31:43.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.354+0000 7f46e3cc6640 1 -- 172.21.15.121:0/3425641915 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46dc1370e0 msgr2=0x7f46dc121e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:31:43.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.354+0000 7f46e3cc6640 1 --2- 172.21.15.121:0/3425641915 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46dc1370e0 0x7f46dc121e20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f46dc074b70 tx=0x7f46d400eea0 comp rx=0 tx=0).stop 2024-09-17T18:31:43.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.354+0000 7f46e2263640 1 -- 172.21.15.121:0/3425641915 reap_dead start 2024-09-17T18:31:43.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.354+0000 7f46e2263640 1 -- 172.21.15.121:0/3425641915 reap_dead start 2024-09-17T18:31:43.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.354+0000 7f46e3cc6640 1 -- 172.21.15.121:0/3425641915 shutdown_connections 2024-09-17T18:31:43.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.354+0000 7f46e3cc6640 1 -- 172.21.15.121:0/3425641915 >> 172.21.15.121:0/3425641915 conn(0x7f46dc0a7830 msgr2=0x7f46dc0a4f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:31:43.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.354+0000 7f46e3cc6640 1 -- 172.21.15.121:0/3425641915 shutdown_connections 2024-09-17T18:31:43.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.354+0000 7f46e3cc6640 1 -- 172.21.15.121:0/3425641915 wait complete. 2024-09-17T18:31:43.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:31:43.354+0000 7f46e3cc6640 1 librados: shutdown 2024-09-17T18:31:43.381 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T18:31:43.381 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-17T18:33:13.382 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-17T18:33:13.382 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-17T18:33:13.382 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-17T18:33:13.386 INFO:tasks.ceph.mon.a:Started 2024-09-17T18:33:13.386 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-17T18:33:13.386 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-17T18:33:13.386 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-17T18:33:13.487 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:13.484+0000 7fefbed22c80 -1 mon.a@-1(???).log v0 log_external_backlog rewinding external_log_to from 513 -> 0 (sync_force? mon rebuild?) 2024-09-17T18:33:28.387 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-17T18:33:28.388 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-17T18:33:28.388 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T18:33:28.388 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:33:28.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.494+0000 7f80c86a9640 10 monclient: get_monmap_and_config 2024-09-17T18:33:28.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.494+0000 7f80c86a9640 10 monclient: build_initial_monmap 2024-09-17T18:33:28.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.494+0000 7f80c86a9640 10 monclient: monmap: 2024-09-17T18:33:28.496 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:33:28.497 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:28.497 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:33:28.495360+0000 2024-09-17T18:33:28.497 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:33:28.495360+0000 2024-09-17T18:33:28.497 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:33:28.497 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:28.497 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:33:28.497 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:33:28.497 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:33:28.497 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:33:28.497 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:33:28.497 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:28.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.495+0000 7f80c86a9640 1 Processor -- start 2024-09-17T18:33:28.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.495+0000 7f80c86a9640 1 -- start start 2024-09-17T18:33:28.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.495+0000 7f80c86a9640 10 monclient: init 2024-09-17T18:33:28.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.496+0000 7f80c86a9640 10 monclient: _reopen_session rank -1 2024-09-17T18:33:28.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.496+0000 7f80c86a9640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-17T18:33:28.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.496+0000 7f80c86a9640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f80c00a0a70 0x7f80c01348f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:28.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.496+0000 7f80c86a9640 10 monclient(hunting): picked mon.noname-c con 0x7f80c00a0a70 addr v2:172.21.15.173:3300/0 2024-09-17T18:33:28.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.496+0000 7f80c86a9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f80c00a0e50 addr v1:172.21.15.173:6789/0 2024-09-17T18:33:28.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.496+0000 7f80c86a9640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f80c00a1260 0x7f80c00a29f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:28.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.496+0000 7f80c86a9640 10 monclient(hunting): picked mon.noname-a con 0x7f80c00a1260 addr v2:172.21.15.121:3300/0 2024-09-17T18:33:28.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.496+0000 7f80c86a9640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80c009f5b0 con 0x7f80c00a0e50 2024-09-17T18:33:28.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.496+0000 7f80c86a9640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:28.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.496+0000 7f80c86a9640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f80c00a48b0 con 0x7f80c00a1260 2024-09-17T18:33:28.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.496+0000 7f80c86a9640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:28.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.496+0000 7f80c86a9640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f80c009ef80 con 0x7f80c00a0a70 2024-09-17T18:33:28.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.496+0000 7f80c86a9640 10 monclient(hunting): _renew_subs 2024-09-17T18:33:28.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.496+0000 7f80c86a9640 10 monclient(hunting): authenticate will time out at 1612.212129s 2024-09-17T18:33:28.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.496+0000 7f80c6c46640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f80c00a1260 0x7f80c00a29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:28.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.496+0000 7f80c6c46640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f80c00a1260 0x7f80c00a29f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59882/0 (socket says 172.21.15.121:59882) 2024-09-17T18:33:28.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.496+0000 7f80c6c46640 1 -- 172.21.15.121:0/4192326357 learned_addr learned my addr 172.21.15.121:0/4192326357 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:33:28.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.496+0000 7f80c6c46640 10 monclient(hunting): get_auth_request con 0x7f80c00a1260 auth_method 0 2024-09-17T18:33:28.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.496+0000 7f80c6c46640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:28.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.496+0000 7f80c6c46640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:28.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.496+0000 7f80c6c46640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:28.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.496+0000 7f80c6c46640 1 -- 172.21.15.121:0/4192326357 >> v2:172.21.15.121:3300/0 conn(0x7f80c00a1260 msgr2=0x7f80c00a29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T18:33:28.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.496+0000 7f80c6c46640 1 -- 172.21.15.121:0/4192326357 >> v2:172.21.15.121:3300/0 conn(0x7f80c00a1260 msgr2=0x7f80c00a29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:33:28.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.496+0000 7f80c6c46640 1 --2- 172.21.15.121:0/4192326357 >> v2:172.21.15.121:3300/0 conn(0x7f80c00a1260 0x7f80c00a29f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:33:28.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.496+0000 7f80c6c46640 1 --2- 172.21.15.121:0/4192326357 >> v2:172.21.15.121:3300/0 conn(0x7f80c00a1260 0x7f80c00a29f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:33:28.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.496+0000 7f80c6445640 1 --2- 172.21.15.121:0/4192326357 >> v2:172.21.15.173:3300/0 conn(0x7f80c00a0a70 0x7f80c01348f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:28.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.496+0000 7f80c6445640 10 monclient(hunting): get_auth_request con 0x7f80c00a0a70 auth_method 0 2024-09-17T18:33:28.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.496+0000 7f80c6445640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:28.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.496+0000 7f80c6445640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:28.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.496+0000 7f80c6445640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:28.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.496+0000 7f80c6445640 1 -- 172.21.15.121:0/4192326357 >> v2:172.21.15.173:3300/0 conn(0x7f80c00a0a70 msgr2=0x7f80c01348f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:33:28.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.496+0000 7f80c6445640 1 -- 172.21.15.121:0/4192326357 >> v2:172.21.15.173:3300/0 conn(0x7f80c00a0a70 msgr2=0x7f80c01348f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:33:28.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.496+0000 7f80c6445640 1 --2- 172.21.15.121:0/4192326357 >> v2:172.21.15.173:3300/0 conn(0x7f80c00a0a70 0x7f80c01348f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:33:28.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.496+0000 7f80c6445640 1 --2- 172.21.15.121:0/4192326357 >> v2:172.21.15.173:3300/0 conn(0x7f80c00a0a70 0x7f80c01348f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:33:28.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.498+0000 7f80c5c44640 1 -- 172.21.15.121:0/4192326357 >> v1:172.21.15.173:6789/0 conn(0x7f80c00a0e50 legacy=0x7f80c00a22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:28.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.698+0000 7f80c6c46640 1 --2- 172.21.15.121:0/4192326357 >> v2:172.21.15.121:3300/0 conn(0x7f80c00a1260 0x7f80c00a29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:28.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.698+0000 7f80c6c46640 10 monclient(hunting): get_auth_request con 0x7f80c00a1260 auth_method 0 2024-09-17T18:33:28.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.698+0000 7f80c6c46640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:28.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.698+0000 7f80c6c46640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:28.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.698+0000 7f80c6c46640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:33:28.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.698+0000 7f80c6445640 1 --2- 172.21.15.121:0/4192326357 >> v2:172.21.15.173:3300/0 conn(0x7f80c00a0a70 0x7f80c01348f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:28.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.698+0000 7f80c6c46640 1 -- 172.21.15.121:0/4192326357 >> v2:172.21.15.121:3300/0 conn(0x7f80c00a1260 msgr2=0x7f80c00a29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:33:28.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.698+0000 7f80c6c46640 1 -- 172.21.15.121:0/4192326357 >> v2:172.21.15.121:3300/0 conn(0x7f80c00a1260 msgr2=0x7f80c00a29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:33:28.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.698+0000 7f80c6c46640 1 --2- 172.21.15.121:0/4192326357 >> v2:172.21.15.121:3300/0 conn(0x7f80c00a1260 0x7f80c00a29f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:33:28.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.698+0000 7f80c6445640 10 monclient(hunting): get_auth_request con 0x7f80c00a0a70 auth_method 0 2024-09-17T18:33:28.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.698+0000 7f80c6445640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:28.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.698+0000 7f80c6445640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:28.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.698+0000 7f80c6445640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:33:28.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.698+0000 7f80c6c46640 1 --2- 172.21.15.121:0/4192326357 >> v2:172.21.15.121:3300/0 conn(0x7f80c00a1260 0x7f80c00a29f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T18:33:28.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.698+0000 7f80c6445640 1 -- 172.21.15.121:0/4192326357 >> v2:172.21.15.173:3300/0 conn(0x7f80c00a0a70 msgr2=0x7f80c01348f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T18:33:28.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.698+0000 7f80c6445640 1 -- 172.21.15.121:0/4192326357 >> v2:172.21.15.173:3300/0 conn(0x7f80c00a0a70 msgr2=0x7f80c01348f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:33:28.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.698+0000 7f80c6445640 1 --2- 172.21.15.121:0/4192326357 >> v2:172.21.15.173:3300/0 conn(0x7f80c00a0a70 0x7f80c01348f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:33:28.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.698+0000 7f80c6445640 1 --2- 172.21.15.121:0/4192326357 >> v2:172.21.15.173:3300/0 conn(0x7f80c00a0a70 0x7f80c01348f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T18:33:28.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.812+0000 7f80c5443640 1 -- 172.21.15.121:0/4192326357 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1104571799 0 0) 0x7f80c009f5b0 con 0x7f80c00a0e50 2024-09-17T18:33:28.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.812+0000 7f80c5443640 10 monclient(hunting): _negotiate 2024-09-17T18:33:28.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.812+0000 7f80c5443640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:28.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.812+0000 7f80c5443640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:28.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.812+0000 7f80c5443640 10 monclient(hunting): my global_id is 84160 2024-09-17T18:33:28.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.812+0000 7f80c5443640 1 -- 172.21.15.121:0/4192326357 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f80ac003520 con 0x7f80c00a0e50 2024-09-17T18:33:28.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5443640 1 -- 172.21.15.121:0/4192326357 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1228347664 0 0) 0x7f80ac003520 con 0x7f80c00a0e50 2024-09-17T18:33:28.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5443640 1 -- 172.21.15.121:0/4192326357 >> v2:172.21.15.173:3300/0 conn(0x7f80c00a0a70 msgr2=0x7f80c01348f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:33:28.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5443640 1 --2- 172.21.15.121:0/4192326357 >> v2:172.21.15.173:3300/0 conn(0x7f80c00a0a70 0x7f80c01348f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:28.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5443640 1 -- 172.21.15.121:0/4192326357 >> v2:172.21.15.121:3300/0 conn(0x7f80c00a1260 msgr2=0x7f80c00a29f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:33:28.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5443640 1 --2- 172.21.15.121:0/4192326357 >> v2:172.21.15.121:3300/0 conn(0x7f80c00a1260 0x7f80c00a29f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:28.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5443640 10 monclient: _finish_hunting 0 2024-09-17T18:33:28.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5443640 1 monclient: found mon.noname-c-legacy 2024-09-17T18:33:28.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5443640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:33:28.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5443640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.173:6789/0 2024-09-17T18:33:28.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5443640 1 -- 172.21.15.121:0/4192326357 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80c009eca0 con 0x7f80c00a0e50 2024-09-17T18:33:28.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5443640 10 monclient: _finish_auth 0 2024-09-17T18:33:28.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5443640 10 monclient: _check_auth_tickets 2024-09-17T18:33:28.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5443640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:33:28.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5443640 1 -- 172.21.15.121:0/4192326357 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4002019445 0 0) 0x7f80bc0019f0 con 0x7f80c00a0e50 2024-09-17T18:33:28.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5443640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:28.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5443640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-17T18:33:28.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5443640 10 monclient: dump: 2024-09-17T18:33:28.828 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:28.828 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:28.828 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:28.828 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:28.828 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:28.828 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:28.828 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:28.828 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:28.828 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:28.828 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:28.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5443640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] but i'm connected to v1:172.21.15.173:6789/0, reconnecting 2024-09-17T18:33:28.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5443640 10 monclient: _reopen_session rank -1 2024-09-17T18:33:28.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5443640 1 -- 172.21.15.121:0/4192326357 >> v1:172.21.15.173:6789/0 conn(0x7f80c00a0e50 legacy=0x7f80c00a22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:28.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5443640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T18:33:28.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5443640 1 --2- 172.21.15.121:0/4192326357 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f80ac003b50 0x7f80ac003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:28.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5443640 10 monclient(hunting): picked mon.c con 0x7f80ac003b50 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:33:28.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5443640 1 --2- 172.21.15.121:0/4192326357 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80ac004800 0x7f80ac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:28.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5443640 10 monclient(hunting): picked mon.a con 0x7f80ac004800 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:33:28.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5443640 1 --2- 172.21.15.121:0/4192326357 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80ac00bbc0 0x7f80ac00dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:28.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5443640 10 monclient(hunting): picked mon.b con 0x7f80ac00bbc0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:33:28.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5443640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:28.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5443640 1 -- 172.21.15.121:0/4192326357 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f80c00a48b0 con 0x7f80ac004800 2024-09-17T18:33:28.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5443640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:28.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5443640 1 -- 172.21.15.121:0/4192326357 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f80c009ef80 con 0x7f80ac00bbc0 2024-09-17T18:33:28.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5443640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:28.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5443640 1 -- 172.21.15.121:0/4192326357 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f80ac00e5a0 con 0x7f80ac003b50 2024-09-17T18:33:28.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5443640 10 monclient(hunting): _renew_subs 2024-09-17T18:33:28.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5443640 10 monclient(hunting): _finish_auth 0 2024-09-17T18:33:28.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c6445640 1 --2- 172.21.15.121:0/4192326357 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f80ac003b50 0x7f80ac003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:28.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5c44640 1 --2- 172.21.15.121:0/4192326357 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80ac004800 0x7f80ac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:28.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c6445640 10 monclient(hunting): get_auth_request con 0x7f80ac003b50 auth_method 0 2024-09-17T18:33:28.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c6445640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:28.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c6445640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:28.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c6445640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:33:28.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5c44640 10 monclient(hunting): get_auth_request con 0x7f80ac004800 auth_method 0 2024-09-17T18:33:28.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5c44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:28.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5c44640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:28.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5c44640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:33:28.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c6c46640 1 --2- 172.21.15.121:0/4192326357 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80ac00bbc0 0x7f80ac00dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:28.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c6445640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:28.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c6445640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:28.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c6445640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:33:28.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c6c46640 10 monclient(hunting): get_auth_request con 0x7f80ac00bbc0 auth_method 0 2024-09-17T18:33:28.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c6c46640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:28.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c6c46640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:28.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c6c46640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:33:28.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5c44640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:28.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5c44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:28.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5c44640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:33:28.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c6c46640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:28.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c6c46640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:28.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c6c46640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:33:28.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c6445640 10 monclient(hunting): handle_auth_done global_id 84160 payload 293 2024-09-17T18:33:28.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c6445640 1 -- 172.21.15.121:0/4192326357 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80ac00bbc0 msgr2=0x7f80ac00dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:33:28.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c6445640 1 --2- 172.21.15.121:0/4192326357 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80ac00bbc0 0x7f80ac00dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:28.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c6445640 1 -- 172.21.15.121:0/4192326357 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80ac004800 msgr2=0x7f80ac004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:33:28.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c6445640 1 --2- 172.21.15.121:0/4192326357 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80ac004800 0x7f80ac004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:28.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c6445640 10 monclient: _finish_hunting 0 2024-09-17T18:33:28.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c6445640 1 monclient: found mon.c 2024-09-17T18:33:28.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c6445640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:33:28.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c6445640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:33:28.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c6445640 1 -- 172.21.15.121:0/4192326357 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80ac003520 con 0x7f80ac003b50 2024-09-17T18:33:28.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c6445640 10 monclient: _finish_auth 0 2024-09-17T18:33:28.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c6445640 10 monclient: _check_auth_tickets 2024-09-17T18:33:28.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c6445640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:33:28.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c6c46640 1 -- 172.21.15.121:0/4192326357 reap_dead start 2024-09-17T18:33:28.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c86a9640 5 monclient: authenticate success, global_id 84160 2024-09-17T18:33:28.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c86a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:33:28.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.823+0000 7f80c5c44640 1 --2- 172.21.15.121:0/4192326357 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80ac004800 0x7f80ac004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:33:28.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.826+0000 7f80c6445640 1 -- 172.21.15.121:0/4192326357 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f80ac003b50 msgr2=0x7f80ac003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:28.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.826+0000 7f80c6445640 1 --2- 172.21.15.121:0/4192326357 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f80ac003b50 0x7f80ac003f70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f80b000daf0 tx=0x7f80ac005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6382b356a5ba611 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:28.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.826+0000 7f80c5443640 1 -- 172.21.15.121:0/4192326357 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b0031070 con 0x7f80ac003b50 2024-09-17T18:33:28.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.826+0000 7f80c5443640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:28.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.826+0000 7f80c5443640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:33:28.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.826+0000 7f80c5443640 10 monclient: dump: 2024-09-17T18:33:28.840 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:28.840 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:28.840 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:28.840 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:28.841 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:28.841 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:28.841 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:28.841 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:28.841 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:28.841 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:28.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.826+0000 7f80c86a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:33:28.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.826+0000 7f80c5443640 1 -- 172.21.15.121:0/4192326357 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80b002b720 con 0x7f80ac003b50 2024-09-17T18:33:28.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.826+0000 7f80c5443640 10 monclient: handle_config config(0 keys) 2024-09-17T18:33:28.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.826+0000 7f80c5443640 1 -- 172.21.15.121:0/4192326357 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b000b040 con 0x7f80ac003b50 2024-09-17T18:33:28.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.826+0000 7f80c86a9640 10 monclient: get_monmap_and_config success 2024-09-17T18:33:28.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.826+0000 7f80c86a9640 4 set_mon_vals no callback set 2024-09-17T18:33:28.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.826+0000 7f80c5443640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:28.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.826+0000 7f80c5443640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:33:28.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.826+0000 7f80c5443640 10 monclient: dump: 2024-09-17T18:33:28.842 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:28.842 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:28.842 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:28.842 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:28.842 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:28.842 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:28.842 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:28.842 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:28.842 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:28.842 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:28.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.827+0000 7f80c86a9640 10 monclient: shutdown 2024-09-17T18:33:28.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.827+0000 7f80c86a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:33:28.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.827+0000 7f80c86a9640 1 -- 172.21.15.121:0/4192326357 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f80ac003b50 msgr2=0x7f80ac003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:28.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.827+0000 7f80c86a9640 1 --2- 172.21.15.121:0/4192326357 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f80ac003b50 0x7f80ac003f70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f80b000daf0 tx=0x7f80ac005830 comp rx=0 tx=0).stop 2024-09-17T18:33:28.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.827+0000 7f80c86a9640 1 -- 172.21.15.121:0/4192326357 shutdown_connections 2024-09-17T18:33:28.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.827+0000 7f80c86a9640 1 --2- 172.21.15.121:0/4192326357 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f80ac003b50 0x7f80ac003f70 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:28.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.827+0000 7f80c86a9640 1 -- 172.21.15.121:0/4192326357 >> 172.21.15.121:0/4192326357 conn(0x7f80c00a3b50 msgr2=0x7f80c00a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:33:28.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.827+0000 7f80c86a9640 1 -- 172.21.15.121:0/4192326357 shutdown_connections 2024-09-17T18:33:28.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.827+0000 7f80c86a9640 1 -- 172.21.15.121:0/4192326357 wait complete. 2024-09-17T18:33:28.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.828+0000 7f80c86a9640 10 monclient: build_initial_monmap 2024-09-17T18:33:28.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.828+0000 7f80c86a9640 10 monclient: monmap: 2024-09-17T18:33:28.843 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:33:28.843 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:33:28.843 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:33:28.843 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:33:28.844 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:33:28.844 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:28.844 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:33:28.844 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:33:28.844 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:33:28.844 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:28.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.828+0000 7f80c86a9640 1 librados: starting msgr at 2024-09-17T18:33:28.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.828+0000 7f80c86a9640 1 librados: starting objecter 2024-09-17T18:33:28.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.828+0000 7f80c86a9640 1 Processor -- start 2024-09-17T18:33:28.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.828+0000 7f80c86a9640 1 -- start start 2024-09-17T18:33:28.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.828+0000 7f80c86a9640 1 librados: setting wanted keys 2024-09-17T18:33:28.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.828+0000 7f80c86a9640 1 librados: calling monclient init 2024-09-17T18:33:28.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.828+0000 7f80c86a9640 10 monclient: init 2024-09-17T18:33:28.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c86a9640 10 monclient: _reopen_session rank -1 2024-09-17T18:33:28.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c86a9640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T18:33:28.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c86a9640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80ac003b50 0x7f80c011ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:28.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c86a9640 10 monclient(hunting): picked mon.noname-b con 0x7f80ac003b50 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:33:28.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c86a9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f80c011f160 0x7f80c011f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:28.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c86a9640 10 monclient(hunting): picked mon.noname-c con 0x7f80c011f160 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:33:28.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c86a9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80c01cd7e0 0x7f80c01cfbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:28.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c86a9640 10 monclient(hunting): picked mon.noname-a con 0x7f80c01cd7e0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:33:28.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c86a9640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:28.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c86a9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f80c009de60 con 0x7f80c01cd7e0 2024-09-17T18:33:28.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c86a9640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:28.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c86a9640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f80c009dce0 con 0x7f80ac003b50 2024-09-17T18:33:28.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c86a9640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:28.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c86a9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f80c01348f0 con 0x7f80c011f160 2024-09-17T18:33:28.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c86a9640 10 monclient(hunting): _renew_subs 2024-09-17T18:33:28.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c86a9640 10 monclient(hunting): authenticate will time out at 1612.545315s 2024-09-17T18:33:28.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c5c44640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f80c011f160 0x7f80c011f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:28.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c6c46640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80c01cd7e0 0x7f80c01cfbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:28.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c5c44640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f80c011f160 0x7f80c011f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:42832/0 (socket says 172.21.15.121:42832) 2024-09-17T18:33:28.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c5c44640 1 -- 172.21.15.121:0/3040957856 learned_addr learned my addr 172.21.15.121:0/3040957856 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:33:28.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c6c46640 10 monclient(hunting): get_auth_request con 0x7f80c01cd7e0 auth_method 0 2024-09-17T18:33:28.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c6c46640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:28.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c6c46640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:28.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c6c46640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:28.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c5c44640 10 monclient(hunting): get_auth_request con 0x7f80c011f160 auth_method 0 2024-09-17T18:33:28.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c5c44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:28.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c5c44640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:28.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c5c44640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:28.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c6445640 1 --2- 172.21.15.121:0/3040957856 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80ac003b50 0x7f80c011ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:28.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c6445640 10 monclient(hunting): get_auth_request con 0x7f80ac003b50 auth_method 0 2024-09-17T18:33:28.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c6445640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:28.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c6445640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:28.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c6445640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:28.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c6c46640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:28.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c6c46640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:28.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c6c46640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:28.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c5c44640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:28.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c5c44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:28.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c5c44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:28.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c6445640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:28.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c6445640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:28.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c6445640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:28.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c6c46640 10 monclient(hunting): handle_auth_done global_id 94098 payload 824 2024-09-17T18:33:28.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c6c46640 1 -- 172.21.15.121:0/3040957856 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f80c011f160 msgr2=0x7f80c011f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:28.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c6c46640 1 --2- 172.21.15.121:0/3040957856 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f80c011f160 0x7f80c011f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:28.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c6c46640 1 -- 172.21.15.121:0/3040957856 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80ac003b50 msgr2=0x7f80c011ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:28.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c6c46640 1 --2- 172.21.15.121:0/3040957856 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80ac003b50 0x7f80c011ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:28.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c6c46640 10 monclient: _finish_hunting 0 2024-09-17T18:33:28.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c6c46640 1 monclient: found mon.noname-a 2024-09-17T18:33:28.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c6c46640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:33:28.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c6c46640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:33:28.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c6c46640 1 -- 172.21.15.121:0/3040957856 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80c0131750 con 0x7f80c01cd7e0 2024-09-17T18:33:28.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.829+0000 7f80c5c44640 1 --2- 172.21.15.121:0/3040957856 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f80c011f160 0x7f80c011f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:33:28.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.831+0000 7f80c6c46640 1 -- 172.21.15.121:0/3040957856 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80c01cd7e0 msgr2=0x7f80c01cfbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:28.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.831+0000 7f80c6c46640 1 --2- 172.21.15.121:0/3040957856 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80c01cd7e0 0x7f80c01cfbd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f80c00a1640 tx=0x7f80b8007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:28.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.831+0000 7f80b6ffd640 10 client.?.objecter ms_handle_connect 0x7f80c01cd7e0 2024-09-17T18:33:28.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.832+0000 7f80b6ffd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:33:28.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.832+0000 7f80b6ffd640 1 -- 172.21.15.121:0/3040957856 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b8011d60 con 0x7f80c01cd7e0 2024-09-17T18:33:28.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.832+0000 7f80b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:28.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.832+0000 7f80b6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:33:28.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.832+0000 7f80b6ffd640 10 monclient: dump: 2024-09-17T18:33:28.850 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:28.850 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:28.850 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:28.850 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:28.850 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:28.850 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:28.850 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:28.850 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:28.850 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:28.850 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:28.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.832+0000 7f80b6ffd640 10 monclient: _finish_auth 0 2024-09-17T18:33:28.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.832+0000 7f80b6ffd640 10 monclient: _check_auth_tickets 2024-09-17T18:33:28.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.832+0000 7f80b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:33:28.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.832+0000 7f80b6ffd640 1 -- 172.21.15.121:0/3040957856 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80b8016450 con 0x7f80c01cd7e0 2024-09-17T18:33:28.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.832+0000 7f80c86a9640 5 monclient: authenticate success, global_id 94098 2024-09-17T18:33:28.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.832+0000 7f80c86a9640 10 monclient: _renew_subs 2024-09-17T18:33:28.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.832+0000 7f80c86a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:33:28.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.832+0000 7f80c86a9640 1 -- 172.21.15.121:0/3040957856 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80c0121bd0 con 0x7f80c01cd7e0 2024-09-17T18:33:28.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.832+0000 7f80b6ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:33:28.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.832+0000 7f80b6ffd640 1 -- 172.21.15.121:0/3040957856 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b8016840 con 0x7f80c01cd7e0 2024-09-17T18:33:28.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.832+0000 7f80b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:28.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.832+0000 7f80b6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:33:28.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.832+0000 7f80b6ffd640 10 monclient: dump: 2024-09-17T18:33:28.851 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:28.851 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:28.851 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:28.852 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:28.852 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:28.852 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:28.852 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:28.852 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:28.852 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:28.852 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:28.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.832+0000 7f80c86a9640 10 client.94098.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:33:28.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.832+0000 7f80c86a9640 10 monclient: _renew_subs 2024-09-17T18:33:28.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.832+0000 7f80c86a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:33:28.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.832+0000 7f80c86a9640 1 -- 172.21.15.121:0/3040957856 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80c009e240 con 0x7f80c01cd7e0 2024-09-17T18:33:28.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.832+0000 7f80c86a9640 1 librados: init done 2024-09-17T18:33:28.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.832+0000 7f80c4c42640 4 set_mon_vals no callback set 2024-09-17T18:33:28.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.833+0000 7f80b6ffd640 1 -- 172.21.15.121:0/3040957856 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f80b8005d10 con 0x7f80c01cd7e0 2024-09-17T18:33:28.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.833+0000 7f80c86a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:33:28.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.833+0000 7f80c86a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:33:28.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.833+0000 7f80c86a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:33:28.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.833+0000 7f80c86a9640 1 -- 172.21.15.121:0/3040957856 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80c00a1a50 con 0x7f80c01cd7e0 2024-09-17T18:33:28.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.833+0000 7f80b6ffd640 1 --2- 172.21.15.121:0/3040957856 >> v2:172.21.15.173:6812/2423786037 conn(0x7f80a40401a0 0x7f80a4042660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:28.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.833+0000 7f80b6ffd640 1 -- 172.21.15.121:0/3040957856 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(561..561 src has 247..561) ==== 5925+0+0 (secure 0 0 0) 0x7f80b8053e90 con 0x7f80c01cd7e0 2024-09-17T18:33:28.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.833+0000 7f80b6ffd640 10 client.94098.objecter ms_dispatch 0x7f80c0000b90 osd_map(561..561 src has 247..561) 2024-09-17T18:33:28.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.833+0000 7f80b6ffd640 3 client.94098.objecter handle_osd_map got epochs [561,561] > 0 2024-09-17T18:33:28.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.833+0000 7f80b6ffd640 3 client.94098.objecter handle_osd_map decoding full epoch 561 2024-09-17T18:33:28.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.833+0000 7f80b6ffd640 20 client.94098.objecter dump_active .. 0 homeless 2024-09-17T18:33:28.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.833+0000 7f80c6445640 1 --2- 172.21.15.121:0/3040957856 >> v2:172.21.15.173:6812/2423786037 conn(0x7f80a40401a0 0x7f80a4042660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:28.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.833+0000 7f80c6445640 1 -- 172.21.15.121:0/3040957856 >> v2:172.21.15.173:6812/2423786037 conn(0x7f80a40401a0 msgr2=0x7f80a4042660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T18:33:28.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.833+0000 7f80c6445640 1 -- 172.21.15.121:0/3040957856 >> v2:172.21.15.173:6812/2423786037 conn(0x7f80a40401a0 msgr2=0x7f80a4042660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:33:28.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.833+0000 7f80c6445640 1 --2- 172.21.15.121:0/3040957856 >> v2:172.21.15.173:6812/2423786037 conn(0x7f80a40401a0 0x7f80a4042660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:33:28.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.833+0000 7f80c6445640 1 --2- 172.21.15.121:0/3040957856 >> v2:172.21.15.173:6812/2423786037 conn(0x7f80a40401a0 0x7f80a4042660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:33:28.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.838+0000 7f80b6ffd640 1 -- 172.21.15.121:0/3040957856 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f80b8024020 con 0x7f80c01cd7e0 2024-09-17T18:33:28.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.838+0000 7f80b6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:33:28.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:28.838+0000 7f80b6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:33:29.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.035+0000 7f80c6445640 1 --2- 172.21.15.121:0/3040957856 >> v2:172.21.15.173:6812/2423786037 conn(0x7f80a40401a0 0x7f80a4042660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:29.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.035+0000 7f80c6445640 10 monclient: get_auth_request con 0x7f80a40401a0 auth_method 0 2024-09-17T18:33:29.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.038+0000 7f80c6445640 1 --2- 172.21.15.121:0/3040957856 >> v2:172.21.15.173:6812/2423786037 conn(0x7f80a40401a0 0x7f80a4042660 secure :-1 s=READY pgs=4048 cs=0 l=1 rev1=1 crypto rx=0x7f80b0036000 tx=0x7f80b0031000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:29.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.038+0000 7f80b6ffd640 10 client.94098.objecter ms_handle_connect 0x7f80a40401a0 2024-09-17T18:33:29.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.085+0000 7f80c86a9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:33:29.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.085+0000 7f80c86a9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:33:29.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.085+0000 7f80c86a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:33:29.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.085+0000 7f80c86a9640 1 -- 172.21.15.121:0/3040957856 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f80c01214b0 con 0x7f80c01cd7e0 2024-09-17T18:33:29.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.085+0000 7f80b6ffd640 1 -- 172.21.15.121:0/3040957856 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f80b801ee20 con 0x7f80c01cd7e0 2024-09-17T18:33:29.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.085+0000 7f80b6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:33:29.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.085+0000 7f80b6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:33:29.088 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":54,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:33:29.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.087+0000 7f80c86a9640 10 librados: watch_flush enter 2024-09-17T18:33:29.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.087+0000 7f80c86a9640 10 librados: watch_flush exit 2024-09-17T18:33:29.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.087+0000 7f80c86a9640 20 client.94098.objecter shutdown clearing up homeless session... 2024-09-17T18:33:29.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.087+0000 7f80c86a9640 10 client.94098.objecter successfully canceled tick 2024-09-17T18:33:29.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.088+0000 7f80c86a9640 1 -- 172.21.15.121:0/3040957856 >> v2:172.21.15.173:6812/2423786037 conn(0x7f80a40401a0 msgr2=0x7f80a4042660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:29.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.088+0000 7f80c86a9640 1 --2- 172.21.15.121:0/3040957856 >> v2:172.21.15.173:6812/2423786037 conn(0x7f80a40401a0 0x7f80a4042660 secure :-1 s=READY pgs=4048 cs=0 l=1 rev1=1 crypto rx=0x7f80b0036000 tx=0x7f80b0031000 comp rx=0 tx=0).stop 2024-09-17T18:33:29.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.088+0000 7f80c86a9640 10 monclient: shutdown 2024-09-17T18:33:29.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.088+0000 7f80c86a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:33:29.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.088+0000 7f80c86a9640 1 -- 172.21.15.121:0/3040957856 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80c01cd7e0 msgr2=0x7f80c01cfbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:29.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.088+0000 7f80c86a9640 1 --2- 172.21.15.121:0/3040957856 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80c01cd7e0 0x7f80c01cfbd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f80c00a1640 tx=0x7f80b8007f60 comp rx=0 tx=0).stop 2024-09-17T18:33:29.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.088+0000 7f80c86a9640 1 -- 172.21.15.121:0/3040957856 shutdown_connections 2024-09-17T18:33:29.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.088+0000 7f80c86a9640 1 --2- 172.21.15.121:0/3040957856 >> v2:172.21.15.173:6812/2423786037 conn(0x7f80a40401a0 0x7f80a4042660 unknown :-1 s=CLOSED pgs=4048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:29.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.088+0000 7f80c86a9640 1 --2- 172.21.15.121:0/3040957856 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80c01cd7e0 0x7f80c01cfbd0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:29.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.088+0000 7f80c86a9640 1 --2- 172.21.15.121:0/3040957856 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f80c011f160 0x7f80c011f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:29.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.088+0000 7f80c86a9640 1 --2- 172.21.15.121:0/3040957856 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80ac003b50 0x7f80c011ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:29.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.088+0000 7f80c86a9640 1 -- 172.21.15.121:0/3040957856 >> 172.21.15.121:0/3040957856 conn(0x7f80c00a3b50 msgr2=0x7f80c01330e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:33:29.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.088+0000 7f80c86a9640 1 -- 172.21.15.121:0/3040957856 shutdown_connections 2024-09-17T18:33:29.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.088+0000 7f80c86a9640 1 -- 172.21.15.121:0/3040957856 wait complete. 2024-09-17T18:33:29.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.088+0000 7f80c86a9640 1 librados: shutdown 2024-09-17T18:33:29.100 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T18:33:29.100 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T18:33:29.100 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T18:33:29.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.196+0000 7f49fa633640 10 monclient: get_monmap_and_config 2024-09-17T18:33:29.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.196+0000 7f49fa633640 10 monclient: build_initial_monmap 2024-09-17T18:33:29.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.196+0000 7f49fa633640 10 monclient: monmap: 2024-09-17T18:33:29.198 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:33:29.198 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:29.198 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:33:29.197232+0000 2024-09-17T18:33:29.198 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:33:29.197232+0000 2024-09-17T18:33:29.199 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:33:29.199 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:29.199 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:33:29.199 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:33:29.199 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:33:29.199 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:33:29.199 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:33:29.199 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:29.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.197+0000 7f49fa633640 1 Processor -- start 2024-09-17T18:33:29.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.197+0000 7f49fa633640 1 -- start start 2024-09-17T18:33:29.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.197+0000 7f49fa633640 10 monclient: init 2024-09-17T18:33:29.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.197+0000 7f49fa633640 10 monclient: _reopen_session rank -1 2024-09-17T18:33:29.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.197+0000 7f49fa633640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-17T18:33:29.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.197+0000 7f49fa633640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49f409e550 0x7f49f409e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:29.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.197+0000 7f49fa633640 10 monclient(hunting): picked mon.noname-b con 0x7f49f409e550 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:33:29.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.197+0000 7f49fa633640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f49f409ee70 0x7f49f4139840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:29.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.197+0000 7f49fa633640 10 monclient(hunting): picked mon.noname-a con 0x7f49f409ee70 addr v2:172.21.15.121:3300/0 2024-09-17T18:33:29.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.197+0000 7f49fa633640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f49f409f290 addr v1:172.21.15.173:6789/0 2024-09-17T18:33:29.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.197+0000 7f49fa633640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49f40a4460 con 0x7f49f409f290 2024-09-17T18:33:29.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.197+0000 7f49fa633640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:29.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.197+0000 7f49fa633640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f49f40a4950 con 0x7f49f409ee70 2024-09-17T18:33:29.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.197+0000 7f49fa633640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:29.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.197+0000 7f49fa633640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f49f40a2b60 con 0x7f49f409e550 2024-09-17T18:33:29.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.197+0000 7f49fa633640 10 monclient(hunting): _renew_subs 2024-09-17T18:33:29.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.197+0000 7f49fa633640 10 monclient(hunting): authenticate will time out at 1612.914000s 2024-09-17T18:33:29.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.198+0000 7f49ebfff640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f49f409ee70 0x7f49f4139840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:29.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.198+0000 7f49f3fff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49f409e550 0x7f49f409e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:29.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.198+0000 7f49ebfff640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f49f409ee70 0x7f49f4139840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60358/0 (socket says 172.21.15.121:60358) 2024-09-17T18:33:29.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.198+0000 7f49ebfff640 1 -- 172.21.15.121:0/309943084 learned_addr learned my addr 172.21.15.121:0/309943084 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:33:29.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.198+0000 7f49ebfff640 10 monclient(hunting): get_auth_request con 0x7f49f409ee70 auth_method 0 2024-09-17T18:33:29.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.198+0000 7f49ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:29.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.198+0000 7f49ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:29.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.198+0000 7f49ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:29.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.198+0000 7f49f3fff640 10 monclient(hunting): get_auth_request con 0x7f49f409e550 auth_method 0 2024-09-17T18:33:29.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.198+0000 7f49f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:29.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.198+0000 7f49f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:29.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.198+0000 7f49f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:29.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.198+0000 7f49ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:29.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.198+0000 7f49ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:29.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.198+0000 7f49ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:29.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.198+0000 7f49f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:29.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.198+0000 7f49f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:29.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.198+0000 7f49f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:29.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.198+0000 7f49ebfff640 10 monclient(hunting): handle_auth_done global_id 94101 payload 274 2024-09-17T18:33:29.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.198+0000 7f49ebfff640 1 -- 172.21.15.121:0/309943084 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49f409e550 msgr2=0x7f49f409e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:33:29.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.198+0000 7f49ebfff640 1 --2- 172.21.15.121:0/309943084 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49f409e550 0x7f49f409e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:29.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.198+0000 7f49ebfff640 1 -- 172.21.15.121:0/309943084 >> v1:172.21.15.173:6789/0 conn(0x7f49f409f290 legacy=0x7f49f413bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:33:29.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.198+0000 7f49ebfff640 10 monclient: _finish_hunting 0 2024-09-17T18:33:29.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.198+0000 7f49ebfff640 1 monclient: found mon.noname-a 2024-09-17T18:33:29.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.198+0000 7f49ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:33:29.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.198+0000 7f49ebfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:33:29.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.198+0000 7f49ebfff640 1 -- 172.21.15.121:0/309943084 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49f409f840 con 0x7f49f409ee70 2024-09-17T18:33:29.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.198+0000 7f49f3fff640 1 --2- 172.21.15.121:0/309943084 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49f409e550 0x7f49f409e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:33:29.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.201+0000 7f49ebfff640 1 -- 172.21.15.121:0/309943084 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49f409ee70 msgr2=0x7f49f4139840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:29.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.201+0000 7f49ebfff640 1 --2- 172.21.15.121:0/309943084 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49f409ee70 0x7f49f4139840 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f49f409e1a0 tx=0x7f49dc02a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe431970d56e049c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:29.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.201+0000 7f49f37fe640 1 -- 172.21.15.121:0/309943084 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49dc037070 con 0x7f49f409ee70 2024-09-17T18:33:29.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.201+0000 7f49f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:29.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.201+0000 7f49f37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:33:29.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.201+0000 7f49f37fe640 10 monclient: dump: 2024-09-17T18:33:29.213 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:29.213 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:29.213 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.213 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.213 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:29.213 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:29.213 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:29.213 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:29.213 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:29.213 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:29.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.201+0000 7f49f37fe640 10 monclient: _finish_auth 0 2024-09-17T18:33:29.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.201+0000 7f49f37fe640 10 monclient: _check_auth_tickets 2024-09-17T18:33:29.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.201+0000 7f49f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:33:29.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.201+0000 7f49f37fe640 1 -- 172.21.15.121:0/309943084 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49dc032440 con 0x7f49f409ee70 2024-09-17T18:33:29.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.201+0000 7f49fa633640 5 monclient: authenticate success, global_id 94101 2024-09-17T18:33:29.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.201+0000 7f49fa633640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:33:29.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.201+0000 7f49f37fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:33:29.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.201+0000 7f49fa633640 10 monclient: get_monmap_and_config success 2024-09-17T18:33:29.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.201+0000 7f49f37fe640 1 -- 172.21.15.121:0/309943084 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49dc032810 con 0x7f49f409ee70 2024-09-17T18:33:29.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.201+0000 7f49f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:29.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.201+0000 7f49fa633640 4 set_mon_vals no callback set 2024-09-17T18:33:29.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.201+0000 7f49f37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:33:29.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.201+0000 7f49f37fe640 10 monclient: dump: 2024-09-17T18:33:29.214 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:29.214 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:29.214 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.214 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.215 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:29.215 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:29.215 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:29.215 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:29.215 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:29.215 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:29.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.201+0000 7f49fa633640 10 monclient: shutdown 2024-09-17T18:33:29.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.201+0000 7f49fa633640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:33:29.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.201+0000 7f49fa633640 1 -- 172.21.15.121:0/309943084 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49f409ee70 msgr2=0x7f49f4139840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:29.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.201+0000 7f49fa633640 1 --2- 172.21.15.121:0/309943084 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49f409ee70 0x7f49f4139840 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f49f409e1a0 tx=0x7f49dc02a060 comp rx=0 tx=0).stop 2024-09-17T18:33:29.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.201+0000 7f49fa633640 1 -- 172.21.15.121:0/309943084 shutdown_connections 2024-09-17T18:33:29.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.201+0000 7f49fa633640 1 --2- 172.21.15.121:0/309943084 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49f409ee70 0x7f49f4139840 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:29.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.201+0000 7f49fa633640 1 --2- 172.21.15.121:0/309943084 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49f409e550 0x7f49f409e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:29.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.201+0000 7f49fa633640 1 -- 172.21.15.121:0/309943084 >> 172.21.15.121:0/309943084 conn(0x7f49f40a7830 msgr2=0x7f49f40a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:33:29.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.201+0000 7f49fa633640 1 -- 172.21.15.121:0/309943084 shutdown_connections 2024-09-17T18:33:29.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.201+0000 7f49fa633640 1 -- 172.21.15.121:0/309943084 wait complete. 2024-09-17T18:33:29.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.202+0000 7f49fa633640 10 monclient: build_initial_monmap 2024-09-17T18:33:29.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.202+0000 7f49fa633640 10 monclient: monmap: 2024-09-17T18:33:29.216 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:33:29.216 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:33:29.216 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:33:29.216 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:33:29.216 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:33:29.216 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:29.216 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:33:29.216 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:33:29.216 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:33:29.216 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:29.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.202+0000 7f49fa633640 1 librados: starting msgr at 2024-09-17T18:33:29.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.202+0000 7f49fa633640 1 librados: starting objecter 2024-09-17T18:33:29.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.202+0000 7f49fa633640 1 Processor -- start 2024-09-17T18:33:29.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.202+0000 7f49fa633640 1 -- start start 2024-09-17T18:33:29.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.202+0000 7f49fa633640 1 librados: setting wanted keys 2024-09-17T18:33:29.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.202+0000 7f49fa633640 1 librados: calling monclient init 2024-09-17T18:33:29.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.202+0000 7f49fa633640 10 monclient: init 2024-09-17T18:33:29.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49fa633640 10 monclient: _reopen_session rank -1 2024-09-17T18:33:29.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49fa633640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T18:33:29.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49fa633640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49f409e550 0x7f49f4133b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:29.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49fa633640 10 monclient(hunting): picked mon.noname-b con 0x7f49f409e550 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:33:29.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49fa633640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49f409ee70 0x7f49f41340d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:29.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49fa633640 10 monclient(hunting): picked mon.noname-a con 0x7f49f409ee70 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:33:29.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49fa633640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49f409f290 0x7f49f4138b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:29.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49fa633640 10 monclient(hunting): picked mon.noname-c con 0x7f49f409f290 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:33:29.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49fa633640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:29.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49fa633640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f49f409de60 con 0x7f49f409ee70 2024-09-17T18:33:29.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49fa633640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:29.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49fa633640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f49f409dce0 con 0x7f49f409e550 2024-09-17T18:33:29.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49fa633640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:29.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49fa633640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f49f40747e0 con 0x7f49f409f290 2024-09-17T18:33:29.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49fa633640 10 monclient(hunting): _renew_subs 2024-09-17T18:33:29.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49fa633640 10 monclient(hunting): authenticate will time out at 1612.919342s 2024-09-17T18:33:29.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49ebfff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49f409ee70 0x7f49f41340d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:29.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49ebfff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49f409ee70 0x7f49f41340d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60364/0 (socket says 172.21.15.121:60364) 2024-09-17T18:33:29.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49ebfff640 1 -- 172.21.15.121:0/2329849403 learned_addr learned my addr 172.21.15.121:0/2329849403 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:33:29.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49ebfff640 10 monclient(hunting): get_auth_request con 0x7f49f409ee70 auth_method 0 2024-09-17T18:33:29.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:29.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:29.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:29.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49f8bd0640 1 --2- 172.21.15.121:0/2329849403 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49f409f290 0x7f49f4138b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:29.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49f8bd0640 10 monclient(hunting): get_auth_request con 0x7f49f409f290 auth_method 0 2024-09-17T18:33:29.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49f8bd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:29.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49f8bd0640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:29.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49f8bd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:29.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49f3fff640 1 --2- 172.21.15.121:0/2329849403 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49f409e550 0x7f49f4133b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:29.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:29.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:29.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:29.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49f3fff640 10 monclient(hunting): get_auth_request con 0x7f49f409e550 auth_method 0 2024-09-17T18:33:29.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:29.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:29.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:29.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49f8bd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:29.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49f8bd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:29.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49f8bd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:29.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:29.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:29.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:29.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49ebfff640 10 monclient(hunting): handle_auth_done global_id 94104 payload 824 2024-09-17T18:33:29.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49ebfff640 1 -- 172.21.15.121:0/2329849403 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49f409f290 msgr2=0x7f49f4138b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:29.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49ebfff640 1 --2- 172.21.15.121:0/2329849403 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49f409f290 0x7f49f4138b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:29.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49ebfff640 1 -- 172.21.15.121:0/2329849403 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49f409e550 msgr2=0x7f49f4133b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:29.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49ebfff640 1 --2- 172.21.15.121:0/2329849403 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49f409e550 0x7f49f4133b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:29.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49ebfff640 10 monclient: _finish_hunting 0 2024-09-17T18:33:29.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49ebfff640 1 monclient: found mon.noname-a 2024-09-17T18:33:29.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:33:29.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49ebfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:33:29.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.203+0000 7f49ebfff640 1 -- 172.21.15.121:0/2329849403 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49f413d290 con 0x7f49f409ee70 2024-09-17T18:33:29.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.205+0000 7f49ebfff640 1 -- 172.21.15.121:0/2329849403 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49f409ee70 msgr2=0x7f49f41340d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:29.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.205+0000 7f49ebfff640 1 --2- 172.21.15.121:0/2329849403 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49f409ee70 0x7f49f41340d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f49dc0090b0 tx=0x7f49dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:29.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.205+0000 7f49f17fa640 10 client.?.objecter ms_handle_connect 0x7f49f409ee70 2024-09-17T18:33:29.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.205+0000 7f49f17fa640 10 client.?.objecter resend_mon_ops 2024-09-17T18:33:29.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.206+0000 7f49f17fa640 1 -- 172.21.15.121:0/2329849403 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49dc041020 con 0x7f49f409ee70 2024-09-17T18:33:29.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.206+0000 7f49f17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:29.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.206+0000 7f49f17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:33:29.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.206+0000 7f49f17fa640 10 monclient: dump: 2024-09-17T18:33:29.221 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:29.221 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:29.221 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.221 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.222 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:29.222 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:29.222 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:29.222 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:29.222 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:29.222 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:29.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.206+0000 7f49f17fa640 10 monclient: _finish_auth 0 2024-09-17T18:33:29.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.206+0000 7f49f17fa640 10 monclient: _check_auth_tickets 2024-09-17T18:33:29.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.206+0000 7f49f17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:33:29.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.206+0000 7f49fa633640 5 monclient: authenticate success, global_id 94104 2024-09-17T18:33:29.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.206+0000 7f49fa633640 10 monclient: _renew_subs 2024-09-17T18:33:29.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.206+0000 7f49f17fa640 1 -- 172.21.15.121:0/2329849403 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49dc03d830 con 0x7f49f409ee70 2024-09-17T18:33:29.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.206+0000 7f49fa633640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:33:29.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.206+0000 7f49fa633640 1 -- 172.21.15.121:0/2329849403 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49f413c8b0 con 0x7f49f409ee70 2024-09-17T18:33:29.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.206+0000 7f49f17fa640 10 monclient: handle_config config(0 keys) 2024-09-17T18:33:29.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.206+0000 7f49f17fa640 1 -- 172.21.15.121:0/2329849403 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49dc037040 con 0x7f49f409ee70 2024-09-17T18:33:29.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.206+0000 7f49f17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:29.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.206+0000 7f49f17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:33:29.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.206+0000 7f49f17fa640 10 monclient: dump: 2024-09-17T18:33:29.223 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:29.223 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:29.223 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.223 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.223 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:29.223 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:29.223 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:29.223 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:29.224 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:29.224 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:29.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.206+0000 7f49fa633640 10 client.94104.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:33:29.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.206+0000 7f49f2ffd640 4 set_mon_vals no callback set 2024-09-17T18:33:29.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.206+0000 7f49fa633640 10 monclient: _renew_subs 2024-09-17T18:33:29.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.206+0000 7f49fa633640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:33:29.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.206+0000 7f49fa633640 1 -- 172.21.15.121:0/2329849403 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49f413c010 con 0x7f49f409ee70 2024-09-17T18:33:29.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.206+0000 7f49fa633640 1 librados: init done 2024-09-17T18:33:29.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.206+0000 7f49fa633640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:33:29.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.207+0000 7f49fa633640 1 --2- 172.21.15.121:0/2329849403 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49f4139830 0x7f49f402d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:29.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.207+0000 7f49fa633640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:29.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.207+0000 7f49fa633640 1 -- 172.21.15.121:0/2329849403 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f49f413c510 con 0x7f49f4139830 2024-09-17T18:33:29.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.207+0000 7f49f3fff640 1 --2- 172.21.15.121:0/2329849403 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49f4139830 0x7f49f402d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:29.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.207+0000 7f49f3fff640 10 monclient: get_auth_request con 0x7f49f4139830 auth_method 0 2024-09-17T18:33:29.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.207+0000 7f49f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:29.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.207+0000 7f49f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:29.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.207+0000 7f49f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:29.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.207+0000 7f49ebfff640 1 --2- 172.21.15.121:0/2329849403 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49f409ee70 0x7f49f41340d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f49dc0090b0 tx=0x7f49dc0046c0 comp rx=0 tx=0).queue_received will delay after 1.7266e+09 on 0x7f49dc044020 mgrmap(e 5) 2024-09-17T18:33:29.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.207+0000 7f49f17fa640 1 -- 172.21.15.121:0/2329849403 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f49dc044020 con 0x7f49f409ee70 2024-09-17T18:33:29.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.207+0000 7f49f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:29.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.207+0000 7f49f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:29.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.207+0000 7f49f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:29.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.207+0000 7f49f3fff640 10 monclient(hunting): handle_auth_done global_id 94107 payload 824 2024-09-17T18:33:29.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.207+0000 7f49f3fff640 1 -- 172.21.15.121:0/2329849403 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f49f409e930 con 0x7f49f4139830 2024-09-17T18:33:29.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.207+0000 7f49f17fa640 1 --2- 172.21.15.121:0/2329849403 >> v2:172.21.15.173:6812/2423786037 conn(0x7f49d40401d0 0x7f49d4042690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:29.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.207+0000 7f49f8bd0640 1 --2- 172.21.15.121:0/2329849403 >> v2:172.21.15.173:6812/2423786037 conn(0x7f49d40401d0 0x7f49d4042690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:29.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.207+0000 7f49f8bd0640 10 monclient: get_auth_request con 0x7f49d40401d0 auth_method 0 2024-09-17T18:33:29.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.209+0000 7f49f3fff640 1 -- 172.21.15.121:0/2329849403 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49f4139830 msgr2=0x7f49f402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:29.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.209+0000 7f49f17fa640 10 client.94104.objecter ms_handle_connect 0x7f49f4139830 2024-09-17T18:33:29.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.209+0000 7f49f17fa640 10 client.94104.objecter resend_mon_ops 2024-09-17T18:33:29.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.209+0000 7f49f3fff640 1 --2- 172.21.15.121:0/2329849403 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49f4139830 0x7f49f402d440 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f49e40118e0 tx=0x7f49e4011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:29.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.209+0000 7f49f17fa640 1 -- 172.21.15.121:0/2329849403 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e4014440 con 0x7f49f4139830 2024-09-17T18:33:29.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.209+0000 7f49f17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:29.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.209+0000 7f49f17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:33:29.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.209+0000 7f49f17fa640 10 monclient: dump: 2024-09-17T18:33:29.226 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:29.226 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:29.226 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.226 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.226 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:29.227 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:29.227 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:29.227 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:29.227 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:29.227 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:29.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.209+0000 7f49f17fa640 1 -- 172.21.15.121:0/2329849403 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f49e4014440 con 0x7f49f4139830 2024-09-17T18:33:29.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.209+0000 7f49f17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:33:29.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.209+0000 7f49f17fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:33:29.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.209+0000 7f49f17fa640 1 -- 172.21.15.121:0/2329849403 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49f4139830 msgr2=0x7f49f402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:29.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.209+0000 7f49f17fa640 1 --2- 172.21.15.121:0/2329849403 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49f4139830 0x7f49f402d440 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f49e40118e0 tx=0x7f49e4011da0 comp rx=0 tx=0).stop 2024-09-17T18:33:29.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.209+0000 7f49f17fa640 1 -- 172.21.15.121:0/2329849403 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49f4139830 msgr2=0x7f49f402d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:33:29.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.209+0000 7f49f17fa640 1 --2- 172.21.15.121:0/2329849403 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49f4139830 0x7f49f402d440 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f49e40118e0 tx=0x7f49e4011da0 comp rx=0 tx=0).stop 2024-09-17T18:33:29.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.210+0000 7f49f17fa640 10 client.94104.objecter ms_handle_connect 0x7f49d40401d0 2024-09-17T18:33:29.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.210+0000 7f49f8bd0640 1 --2- 172.21.15.121:0/2329849403 >> v2:172.21.15.173:6812/2423786037 conn(0x7f49d40401d0 0x7f49d4042690 secure :-1 s=READY pgs=4049 cs=0 l=1 rev1=1 crypto rx=0x7f49f40a65e0 tx=0x7f49ec003d10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:29.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.215+0000 7f49fa633640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:33:29.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.215+0000 7f49fa633640 1 --2- 172.21.15.121:0/2329849403 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49f4139c10 0x7f49f402d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:29.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.216+0000 7f49fa633640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:29.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.216+0000 7f49fa633640 1 -- 172.21.15.121:0/2329849403 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f49f413bd00 con 0x7f49f4139c10 2024-09-17T18:33:29.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.216+0000 7f49f3fff640 1 --2- 172.21.15.121:0/2329849403 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49f4139c10 0x7f49f402d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:29.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.216+0000 7f49f3fff640 10 monclient: get_auth_request con 0x7f49f4139c10 auth_method 0 2024-09-17T18:33:29.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.216+0000 7f49f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:29.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.216+0000 7f49f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:29.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.216+0000 7f49f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:29.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.216+0000 7f49f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:29.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.216+0000 7f49f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:29.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.216+0000 7f49f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:29.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.216+0000 7f49f3fff640 10 monclient(hunting): handle_auth_done global_id 94110 payload 824 2024-09-17T18:33:29.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.216+0000 7f49f3fff640 1 -- 172.21.15.121:0/2329849403 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f49f406cd40 con 0x7f49f4139c10 2024-09-17T18:33:29.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.218+0000 7f49f3fff640 1 -- 172.21.15.121:0/2329849403 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49f4139c10 msgr2=0x7f49f402d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:29.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.218+0000 7f49f17fa640 10 client.94104.objecter ms_handle_connect 0x7f49f4139c10 2024-09-17T18:33:29.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.218+0000 7f49f17fa640 10 client.94104.objecter resend_mon_ops 2024-09-17T18:33:29.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.218+0000 7f49f3fff640 1 --2- 172.21.15.121:0/2329849403 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49f4139c10 0x7f49f402d980 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f49e4000c00 tx=0x7f49e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:29.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.218+0000 7f49f17fa640 1 -- 172.21.15.121:0/2329849403 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e40229a0 con 0x7f49f4139c10 2024-09-17T18:33:29.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.218+0000 7f49f17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:29.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.218+0000 7f49f17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:33:29.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.218+0000 7f49f17fa640 10 monclient: dump: 2024-09-17T18:33:29.231 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:29.231 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:29.231 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.231 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.231 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:29.231 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:29.232 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:29.232 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:29.232 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:29.232 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:29.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.218+0000 7f49f17fa640 1 -- 172.21.15.121:0/2329849403 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4554 (secure 0 0 0) 0x7f49e402b470 con 0x7f49f4139c10 2024-09-17T18:33:29.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.218+0000 7f49f17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:33:29.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.218+0000 7f49f17fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:33:29.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.218+0000 7f49f17fa640 1 -- 172.21.15.121:0/2329849403 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49f4139c10 msgr2=0x7f49f402d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:29.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.218+0000 7f49f17fa640 1 --2- 172.21.15.121:0/2329849403 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49f4139c10 0x7f49f402d980 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f49e4000c00 tx=0x7f49e40046c0 comp rx=0 tx=0).stop 2024-09-17T18:33:29.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.218+0000 7f49f17fa640 1 -- 172.21.15.121:0/2329849403 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49f4139c10 msgr2=0x7f49f402d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:33:29.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.218+0000 7f49f17fa640 1 --2- 172.21.15.121:0/2329849403 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49f4139c10 0x7f49f402d980 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f49e4000c00 tx=0x7f49e40046c0 comp rx=0 tx=0).stop 2024-09-17T18:33:29.232 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:33:29.233 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:33:29.233 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:33:29.233 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "leader", 2024-09-17T18:33:29.233 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 54, 2024-09-17T18:33:29.233 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 15744, 2024-09-17T18:33:29.233 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:33:29.233 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:33:29.233 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:33:29.233 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:33:29.233 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:33:29.233 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 0, 2024-09-17T18:33:29.233 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:33:29.233 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:33:29.234 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:33:29.234 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:33:29.234 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:33:29.234 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:33:29.234 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:33:29.234 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:33:29.234 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:33:29.234 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:33:29.234 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:33:29.234 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:33:29.234 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:33:29.234 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:33:29.234 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:33:29.234 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:33:29.235 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:33:29.235 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:33:29.235 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:33:29.235 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:33:29.235 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:33:29.235 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:33:29.235 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:33:29.235 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:33:29.235 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:33:29.235 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:33:29.235 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:33:29.235 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:33:29.235 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:29.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:29.236 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:33:29.236 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:33:29.236 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:33:29.236 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:33:29.236 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:33:29.236 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:33:29.236 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:33:29.236 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:33:29.236 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:33:29.236 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:33:29.236 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:33:29.237 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:33:29.237 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:33:29.237 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:33:29.237 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:33:29.237 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:33:29.237 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:33:29.237 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:33:29.237 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:33:29.237 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:33:29.237 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:33:29.237 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:33:29.237 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:33:29.237 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:33:29.237 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:33:29.238 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:33:29.238 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:33:29.238 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:33:29.238 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:33:29.238 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:33:29.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:29.238 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:33:29.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:29.238 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:33:29.238 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:33:29.238 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:33:29.238 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:33:29.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:29.238 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:33:29.238 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:33:29.239 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:29.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:29.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:29.239 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:33:29.239 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:33:29.239 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:29.239 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:29.239 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:29.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:29.239 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:33:29.239 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:33:29.239 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:33:29.239 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:33:29.239 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:33:29.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:29.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:29.240 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:33:29.240 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:33:29.240 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:33:29.240 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:33:29.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:29.240 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:33:29.240 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:33:29.240 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:29.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:29.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:29.240 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:33:29.241 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:33:29.241 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:29.241 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:29.241 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:29.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:29.241 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:33:29.241 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:33:29.241 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:33:29.241 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:33:29.241 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:33:29.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:29.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:29.242 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:33:29.242 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:33:29.242 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:33:29.242 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:33:29.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:29.242 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:33:29.242 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:33:29.242 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:29.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:29.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:29.242 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:33:29.242 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:33:29.242 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:29.242 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:29.243 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:29.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:29.243 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:33:29.243 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:33:29.243 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:33:29.243 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:33:29.243 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:33:29.243 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:29.243 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:29.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:29.243 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:33:29.243 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:33:29.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:29.243 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:33:29.244 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:33:29.244 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:33:29.244 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:29.244 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:33:29.244 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:33:29.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:29.244 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:33:29.244 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:33:29.244 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-17T18:33:29.244 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:29.244 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:29.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:29.244 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:33:29.244 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:33:29.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.219+0000 7f49fa633640 10 librados: watch_flush enter 2024-09-17T18:33:29.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.219+0000 7f49fa633640 10 librados: watch_flush exit 2024-09-17T18:33:29.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.219+0000 7f49fa633640 20 client.94104.objecter shutdown clearing up homeless session... 2024-09-17T18:33:29.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.219+0000 7f49fa633640 10 client.94104.objecter successfully canceled tick 2024-09-17T18:33:29.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.219+0000 7f49fa633640 1 -- 172.21.15.121:0/2329849403 >> v2:172.21.15.173:6812/2423786037 conn(0x7f49d40401d0 msgr2=0x7f49d4042690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:29.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.219+0000 7f49fa633640 1 --2- 172.21.15.121:0/2329849403 >> v2:172.21.15.173:6812/2423786037 conn(0x7f49d40401d0 0x7f49d4042690 secure :-1 s=READY pgs=4049 cs=0 l=1 rev1=1 crypto rx=0x7f49f40a65e0 tx=0x7f49ec003d10 comp rx=0 tx=0).stop 2024-09-17T18:33:29.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.219+0000 7f49fa633640 10 monclient: shutdown 2024-09-17T18:33:29.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.219+0000 7f49fa633640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:33:29.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.219+0000 7f49fa633640 1 -- 172.21.15.121:0/2329849403 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49f409ee70 msgr2=0x7f49f41340d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:29.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.219+0000 7f49fa633640 1 --2- 172.21.15.121:0/2329849403 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49f409ee70 0x7f49f41340d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f49dc0090b0 tx=0x7f49dc0046c0 comp rx=0 tx=0).stop 2024-09-17T18:33:29.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.219+0000 7f49f8bd0640 1 -- 172.21.15.121:0/2329849403 reap_dead start 2024-09-17T18:33:29.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.219+0000 7f49f17fa640 1 -- 172.21.15.121:0/2329849403 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(561..561 src has 247..561) ==== 5925+0+0 (unknown 0 0 0) 0x7f49dc073be0 con 0x7f49f409ee70 2024-09-17T18:33:29.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.219+0000 7f49f8bd0640 1 -- 172.21.15.121:0/2329849403 reap_dead start 2024-09-17T18:33:29.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.219+0000 7f49f17fa640 10 client.94104.objecter ms_dispatch 0x7f49f4000b90 osd_map(561..561 src has 247..561) 2024-09-17T18:33:29.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.219+0000 7f49fa633640 1 -- 172.21.15.121:0/2329849403 shutdown_connections 2024-09-17T18:33:29.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.219+0000 7f49f17fa640 10 librados: disconnected, discarding osd_map(561..561 src has 247..561) 2024-09-17T18:33:29.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.219+0000 7f49fa633640 1 -- 172.21.15.121:0/2329849403 >> 172.21.15.121:0/2329849403 conn(0x7f49f40a7830 msgr2=0x7f49f40a4f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:33:29.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.219+0000 7f49fa633640 1 -- 172.21.15.121:0/2329849403 shutdown_connections 2024-09-17T18:33:29.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.219+0000 7f49fa633640 1 -- 172.21.15.121:0/2329849403 wait complete. 2024-09-17T18:33:29.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.219+0000 7f49fa633640 1 librados: shutdown 2024-09-17T18:33:29.248 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:33:29.248 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T18:33:29.248 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T18:33:29.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.345+0000 7fe7874bf640 10 monclient: get_monmap_and_config 2024-09-17T18:33:29.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.345+0000 7fe7874bf640 10 monclient: build_initial_monmap 2024-09-17T18:33:29.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.345+0000 7fe7874bf640 10 monclient: monmap: 2024-09-17T18:33:29.347 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:33:29.347 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:29.347 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:33:29.346129+0000 2024-09-17T18:33:29.347 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:33:29.346129+0000 2024-09-17T18:33:29.347 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:33:29.347 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:29.348 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:33:29.348 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:33:29.348 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:33:29.348 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:33:29.348 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:33:29.348 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:29.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.346+0000 7fe7874bf640 1 Processor -- start 2024-09-17T18:33:29.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.346+0000 7fe7874bf640 1 -- start start 2024-09-17T18:33:29.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.346+0000 7fe7874bf640 10 monclient: init 2024-09-17T18:33:29.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.346+0000 7fe7874bf640 10 monclient: _reopen_session rank -1 2024-09-17T18:33:29.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.346+0000 7fe7874bf640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-17T18:33:29.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.346+0000 7fe7874bf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe7800a0a00 addr v1:172.21.15.121:6789/0 2024-09-17T18:33:29.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.346+0000 7fe7874bf640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe780137c90 0x7fe78013a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:29.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.346+0000 7fe7874bf640 10 monclient(hunting): picked mon.noname-a con 0x7fe780137c90 addr v2:172.21.15.121:3300/0 2024-09-17T18:33:29.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.346+0000 7fe7874bf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe78013a5c0 addr v1:172.21.15.173:6789/0 2024-09-17T18:33:29.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.346+0000 7fe7874bf640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7800a3820 con 0x7fe7800a0a00 2024-09-17T18:33:29.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.346+0000 7fe7874bf640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7800a2ea0 con 0x7fe78013a5c0 2024-09-17T18:33:29.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.346+0000 7fe7874bf640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:29.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.346+0000 7fe7874bf640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fe78009e800 con 0x7fe780137c90 2024-09-17T18:33:29.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.346+0000 7fe7874bf640 10 monclient(hunting): _renew_subs 2024-09-17T18:33:29.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.346+0000 7fe7874bf640 10 monclient(hunting): authenticate will time out at 1613.062773s 2024-09-17T18:33:29.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.346+0000 7fe784a5a640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe780137c90 0x7fe78013a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:29.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.346+0000 7fe78525b640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fe7800a0a00 0x7fe7800a0de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:57624/0 (socket says 172.21.15.121:57624) 2024-09-17T18:33:29.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.346+0000 7fe784a5a640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe780137c90 0x7fe78013a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60398/0 (socket says 172.21.15.121:60398) 2024-09-17T18:33:29.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.346+0000 7fe78525b640 1 -- 172.21.15.121:0/673353839 learned_addr learned my addr 172.21.15.121:0/673353839 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:33:29.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.346+0000 7fe784a5a640 10 monclient(hunting): get_auth_request con 0x7fe780137c90 auth_method 0 2024-09-17T18:33:29.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.346+0000 7fe784a5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:29.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.346+0000 7fe784a5a640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:29.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.346+0000 7fe784a5a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:29.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.346+0000 7fe784a5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:29.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.346+0000 7fe784a5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:29.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.347+0000 7fe784a5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:29.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.347+0000 7fe784a5a640 10 monclient(hunting): handle_auth_done global_id 94113 payload 274 2024-09-17T18:33:29.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.347+0000 7fe784a5a640 1 -- 172.21.15.121:0/673353839 >> v1:172.21.15.173:6789/0 conn(0x7fe78013a5c0 legacy=0x7fe78013c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:33:29.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.347+0000 7fe784a5a640 1 -- 172.21.15.121:0/673353839 >> v1:172.21.15.121:6789/0 conn(0x7fe7800a0a00 legacy=0x7fe7800a0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:33:29.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.347+0000 7fe784a5a640 10 monclient: _finish_hunting 0 2024-09-17T18:33:29.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.347+0000 7fe784a5a640 1 monclient: found mon.noname-a 2024-09-17T18:33:29.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.347+0000 7fe784a5a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:33:29.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.347+0000 7fe784a5a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:33:29.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.347+0000 7fe784a5a640 1 -- 172.21.15.121:0/673353839 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7800a2c30 con 0x7fe780137c90 2024-09-17T18:33:29.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.349+0000 7fe784a5a640 1 -- 172.21.15.121:0/673353839 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe780137c90 msgr2=0x7fe78013a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:29.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.349+0000 7fe784a5a640 1 --2- 172.21.15.121:0/673353839 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe780137c90 0x7fe78013a080 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe77400aed0 tx=0x7fe77402b620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a05cc8ee55a6907a server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:29.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.350+0000 7fe77ffff640 1 -- 172.21.15.121:0/673353839 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe774037070 con 0x7fe780137c90 2024-09-17T18:33:29.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.350+0000 7fe77ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:29.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.350+0000 7fe77ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:33:29.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.350+0000 7fe77ffff640 10 monclient: dump: 2024-09-17T18:33:29.359 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:29.359 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:29.359 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.359 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.359 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:29.359 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:29.360 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:29.360 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:29.360 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:29.360 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:29.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.350+0000 7fe77ffff640 10 monclient: _finish_auth 0 2024-09-17T18:33:29.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.350+0000 7fe77ffff640 10 monclient: _check_auth_tickets 2024-09-17T18:33:29.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.350+0000 7fe77ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:33:29.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.350+0000 7fe77ffff640 1 -- 172.21.15.121:0/673353839 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe774008650 con 0x7fe780137c90 2024-09-17T18:33:29.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.350+0000 7fe7874bf640 5 monclient: authenticate success, global_id 94113 2024-09-17T18:33:29.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.350+0000 7fe7874bf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:33:29.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.350+0000 7fe77ffff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:33:29.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.350+0000 7fe7874bf640 10 monclient: get_monmap_and_config success 2024-09-17T18:33:29.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.350+0000 7fe77ffff640 1 -- 172.21.15.121:0/673353839 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe774008a20 con 0x7fe780137c90 2024-09-17T18:33:29.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.350+0000 7fe77ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:29.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.350+0000 7fe7874bf640 4 set_mon_vals no callback set 2024-09-17T18:33:29.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.350+0000 7fe77ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:33:29.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.350+0000 7fe77ffff640 10 monclient: dump: 2024-09-17T18:33:29.361 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:29.361 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:29.361 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.361 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.361 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:29.361 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:29.361 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:29.361 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:29.361 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:29.361 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:29.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.350+0000 7fe7874bf640 10 monclient: shutdown 2024-09-17T18:33:29.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.350+0000 7fe7874bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:33:29.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.350+0000 7fe7874bf640 1 -- 172.21.15.121:0/673353839 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe780137c90 msgr2=0x7fe78013a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:29.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.350+0000 7fe7874bf640 1 --2- 172.21.15.121:0/673353839 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe780137c90 0x7fe78013a080 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe77400aed0 tx=0x7fe77402b620 comp rx=0 tx=0).stop 2024-09-17T18:33:29.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.350+0000 7fe7874bf640 1 -- 172.21.15.121:0/673353839 shutdown_connections 2024-09-17T18:33:29.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.350+0000 7fe7874bf640 1 --2- 172.21.15.121:0/673353839 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe780137c90 0x7fe78013a080 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:29.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.350+0000 7fe7874bf640 1 -- 172.21.15.121:0/673353839 >> 172.21.15.121:0/673353839 conn(0x7fe7800a7830 msgr2=0x7fe7800a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:33:29.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.350+0000 7fe7874bf640 1 -- 172.21.15.121:0/673353839 shutdown_connections 2024-09-17T18:33:29.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.350+0000 7fe7874bf640 1 -- 172.21.15.121:0/673353839 wait complete. 2024-09-17T18:33:29.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.351+0000 7fe7874bf640 10 monclient: build_initial_monmap 2024-09-17T18:33:29.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.351+0000 7fe7874bf640 10 monclient: monmap: 2024-09-17T18:33:29.362 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:33:29.362 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:33:29.362 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:33:29.362 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:33:29.362 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:33:29.363 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:29.363 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:33:29.363 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:33:29.363 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:33:29.363 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:29.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.351+0000 7fe7874bf640 1 librados: starting msgr at 2024-09-17T18:33:29.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.351+0000 7fe7874bf640 1 librados: starting objecter 2024-09-17T18:33:29.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.351+0000 7fe7874bf640 1 Processor -- start 2024-09-17T18:33:29.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.351+0000 7fe7874bf640 1 -- start start 2024-09-17T18:33:29.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.351+0000 7fe7874bf640 1 librados: setting wanted keys 2024-09-17T18:33:29.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.351+0000 7fe7874bf640 1 librados: calling monclient init 2024-09-17T18:33:29.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.351+0000 7fe7874bf640 10 monclient: init 2024-09-17T18:33:29.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe7874bf640 10 monclient: _reopen_session rank -1 2024-09-17T18:33:29.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe7874bf640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T18:33:29.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe7874bf640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7800a0a00 0x7fe780134930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:29.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe7874bf640 10 monclient(hunting): picked mon.noname-b con 0x7fe7800a0a00 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:33:29.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe7874bf640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe780137c90 0x7fe780121ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:29.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe7874bf640 10 monclient(hunting): picked mon.noname-a con 0x7fe780137c90 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:33:29.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe7874bf640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe78013a5c0 0x7fe780121ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:29.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe7874bf640 10 monclient(hunting): picked mon.noname-c con 0x7fe78013a5c0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:33:29.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe7874bf640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:29.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe7874bf640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe78009de60 con 0x7fe780137c90 2024-09-17T18:33:29.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe7874bf640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:29.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe7874bf640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe78009dce0 con 0x7fe7800a0a00 2024-09-17T18:33:29.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe7874bf640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:29.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe7874bf640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe7800a3120 con 0x7fe78013a5c0 2024-09-17T18:33:29.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe7874bf640 10 monclient(hunting): _renew_subs 2024-09-17T18:33:29.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe7874bf640 10 monclient(hunting): authenticate will time out at 1613.068087s 2024-09-17T18:33:29.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe784a5a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe780137c90 0x7fe780121ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:29.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe784a5a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe780137c90 0x7fe780121ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60404/0 (socket says 172.21.15.121:60404) 2024-09-17T18:33:29.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe784a5a640 1 -- 172.21.15.121:0/1960232340 learned_addr learned my addr 172.21.15.121:0/1960232340 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:33:29.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe784a5a640 10 monclient(hunting): get_auth_request con 0x7fe780137c90 auth_method 0 2024-09-17T18:33:29.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe784a5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:29.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe784a5a640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:29.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe784a5a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:29.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe785a5c640 1 --2- 172.21.15.121:0/1960232340 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe78013a5c0 0x7fe780121ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:29.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe785a5c640 10 monclient(hunting): get_auth_request con 0x7fe78013a5c0 auth_method 0 2024-09-17T18:33:29.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe785a5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:29.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe785a5c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:29.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe785a5c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:29.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe784a5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:29.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe784a5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:29.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe784a5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:29.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe785a5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:29.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe785a5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:29.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe785a5c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:29.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe78525b640 1 --2- 172.21.15.121:0/1960232340 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7800a0a00 0x7fe780134930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:29.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe78525b640 10 monclient(hunting): get_auth_request con 0x7fe7800a0a00 auth_method 0 2024-09-17T18:33:29.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe78525b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:29.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe78525b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:29.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe78525b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:29.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe784a5a640 10 monclient(hunting): handle_auth_done global_id 94116 payload 824 2024-09-17T18:33:29.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe784a5a640 1 -- 172.21.15.121:0/1960232340 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe78013a5c0 msgr2=0x7fe780121ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:29.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe784a5a640 1 --2- 172.21.15.121:0/1960232340 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe78013a5c0 0x7fe780121ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:29.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe784a5a640 1 -- 172.21.15.121:0/1960232340 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7800a0a00 msgr2=0x7fe780134930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:29.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe784a5a640 1 --2- 172.21.15.121:0/1960232340 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7800a0a00 0x7fe780134930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:29.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe784a5a640 10 monclient: _finish_hunting 0 2024-09-17T18:33:29.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe784a5a640 1 monclient: found mon.noname-a 2024-09-17T18:33:29.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe784a5a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:33:29.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe784a5a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:33:29.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe784a5a640 1 -- 172.21.15.121:0/1960232340 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe780125080 con 0x7fe780137c90 2024-09-17T18:33:29.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.352+0000 7fe78525b640 1 --2- 172.21.15.121:0/1960232340 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7800a0a00 0x7fe780134930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T18:33:29.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.354+0000 7fe784a5a640 1 -- 172.21.15.121:0/1960232340 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe780137c90 msgr2=0x7fe780121ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:29.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.354+0000 7fe784a5a640 1 --2- 172.21.15.121:0/1960232340 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe780137c90 0x7fe780121ab0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe774000f80 tx=0x7fe774004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:29.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.354+0000 7fe77dffb640 10 client.?.objecter ms_handle_connect 0x7fe780137c90 2024-09-17T18:33:29.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.354+0000 7fe77dffb640 10 client.?.objecter resend_mon_ops 2024-09-17T18:33:29.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.354+0000 7fe77dffb640 1 -- 172.21.15.121:0/1960232340 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe774037050 con 0x7fe780137c90 2024-09-17T18:33:29.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.354+0000 7fe77dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:29.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.354+0000 7fe77dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:33:29.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.354+0000 7fe77dffb640 10 monclient: dump: 2024-09-17T18:33:29.367 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:29.367 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:29.368 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.368 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.368 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:29.368 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:29.368 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:29.368 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:29.368 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:29.368 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:29.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.354+0000 7fe77dffb640 10 monclient: _finish_auth 0 2024-09-17T18:33:29.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.354+0000 7fe77dffb640 10 monclient: _check_auth_tickets 2024-09-17T18:33:29.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.354+0000 7fe77dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:33:29.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.354+0000 7fe77dffb640 1 -- 172.21.15.121:0/1960232340 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe774038040 con 0x7fe780137c90 2024-09-17T18:33:29.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.354+0000 7fe77dffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:33:29.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.354+0000 7fe77dffb640 1 -- 172.21.15.121:0/1960232340 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe774003df0 con 0x7fe780137c90 2024-09-17T18:33:29.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.354+0000 7fe7874bf640 5 monclient: authenticate success, global_id 94116 2024-09-17T18:33:29.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.354+0000 7fe77dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:29.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.354+0000 7fe77dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:33:29.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.354+0000 7fe77dffb640 10 monclient: dump: 2024-09-17T18:33:29.369 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:29.369 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:29.369 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.369 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.369 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:29.369 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:29.369 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:29.369 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:29.369 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:29.369 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:29.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.354+0000 7fe7874bf640 10 monclient: _renew_subs 2024-09-17T18:33:29.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.354+0000 7fe77f7fe640 4 set_mon_vals no callback set 2024-09-17T18:33:29.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.354+0000 7fe7874bf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:33:29.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.354+0000 7fe7874bf640 1 -- 172.21.15.121:0/1960232340 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe780137580 con 0x7fe780137c90 2024-09-17T18:33:29.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.355+0000 7fe7874bf640 10 client.94116.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:33:29.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.355+0000 7fe7874bf640 10 monclient: _renew_subs 2024-09-17T18:33:29.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.355+0000 7fe7874bf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:33:29.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.355+0000 7fe7874bf640 1 -- 172.21.15.121:0/1960232340 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe780124da0 con 0x7fe780137c90 2024-09-17T18:33:29.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.355+0000 7fe7874bf640 1 librados: init done 2024-09-17T18:33:29.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.355+0000 7fe7874bf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:33:29.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.355+0000 7fe7874bf640 1 --2- 172.21.15.121:0/1960232340 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe780134e70 0x7fe78002d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:29.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.355+0000 7fe7874bf640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:29.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.355+0000 7fe7874bf640 1 -- 172.21.15.121:0/1960232340 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe7800a56e0 con 0x7fe780134e70 2024-09-17T18:33:29.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.355+0000 7fe77dffb640 1 -- 172.21.15.121:0/1960232340 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fe77403c820 con 0x7fe780137c90 2024-09-17T18:33:29.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.355+0000 7fe78525b640 1 --2- 172.21.15.121:0/1960232340 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe780134e70 0x7fe78002d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:29.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.356+0000 7fe78525b640 10 monclient: get_auth_request con 0x7fe780134e70 auth_method 0 2024-09-17T18:33:29.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.356+0000 7fe78525b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:29.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.356+0000 7fe78525b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:29.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.356+0000 7fe78525b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:29.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.356+0000 7fe78525b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:29.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.356+0000 7fe78525b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:29.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.356+0000 7fe78525b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:29.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.356+0000 7fe77dffb640 1 --2- 172.21.15.121:0/1960232340 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe76c040170 0x7fe76c042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:29.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.356+0000 7fe77dffb640 1 -- 172.21.15.121:0/1960232340 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(561..561 src has 247..561) ==== 5925+0+0 (secure 0 0 0) 0x7fe7740747b0 con 0x7fe780137c90 2024-09-17T18:33:29.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.356+0000 7fe77dffb640 10 client.94116.objecter ms_dispatch 0x7fe780000b90 osd_map(561..561 src has 247..561) 2024-09-17T18:33:29.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.356+0000 7fe77dffb640 3 client.94116.objecter handle_osd_map got epochs [561,561] > 0 2024-09-17T18:33:29.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.356+0000 7fe77dffb640 3 client.94116.objecter handle_osd_map decoding full epoch 561 2024-09-17T18:33:29.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.356+0000 7fe78525b640 10 monclient(hunting): handle_auth_done global_id 64373 payload 824 2024-09-17T18:33:29.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.356+0000 7fe77dffb640 20 client.94116.objecter dump_active .. 0 homeless 2024-09-17T18:33:29.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.356+0000 7fe78525b640 1 -- 172.21.15.121:0/1960232340 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe7800a1fa0 con 0x7fe780134e70 2024-09-17T18:33:29.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.356+0000 7fe785a5c640 1 --2- 172.21.15.121:0/1960232340 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe76c040170 0x7fe76c042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:29.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.356+0000 7fe785a5c640 10 monclient: get_auth_request con 0x7fe76c040170 auth_method 0 2024-09-17T18:33:29.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.358+0000 7fe78525b640 1 -- 172.21.15.121:0/1960232340 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe780134e70 msgr2=0x7fe78002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:29.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.358+0000 7fe78525b640 1 --2- 172.21.15.121:0/1960232340 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe780134e70 0x7fe78002d440 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe77800e520 tx=0x7fe77800e9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:29.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.358+0000 7fe77dffb640 10 client.94116.objecter ms_handle_connect 0x7fe780134e70 2024-09-17T18:33:29.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.358+0000 7fe77dffb640 10 client.94116.objecter resend_mon_ops 2024-09-17T18:33:29.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.358+0000 7fe77dffb640 1 -- 172.21.15.121:0/1960232340 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe778005cf0 con 0x7fe780134e70 2024-09-17T18:33:29.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.358+0000 7fe77dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:29.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.358+0000 7fe77dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:33:29.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.358+0000 7fe77dffb640 10 monclient: dump: 2024-09-17T18:33:29.374 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:29.374 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:29.375 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.375 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.375 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:29.375 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:29.375 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:29.375 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:29.375 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:29.375 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:29.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.358+0000 7fe77dffb640 1 -- 172.21.15.121:0/1960232340 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe778018070 con 0x7fe780134e70 2024-09-17T18:33:29.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.358+0000 7fe77dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:33:29.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.358+0000 7fe77dffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:33:29.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.358+0000 7fe77dffb640 1 -- 172.21.15.121:0/1960232340 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe780134e70 msgr2=0x7fe78002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:29.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.358+0000 7fe77dffb640 1 --2- 172.21.15.121:0/1960232340 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe780134e70 0x7fe78002d440 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe77800e520 tx=0x7fe77800e9e0 comp rx=0 tx=0).stop 2024-09-17T18:33:29.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.358+0000 7fe77dffb640 1 -- 172.21.15.121:0/1960232340 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe780134e70 msgr2=0x7fe78002d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:33:29.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.358+0000 7fe77dffb640 1 --2- 172.21.15.121:0/1960232340 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe780134e70 0x7fe78002d440 secure :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe77800e520 tx=0x7fe77800e9e0 comp rx=0 tx=0).stop 2024-09-17T18:33:29.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.358+0000 7fe77dffb640 10 client.94116.objecter ms_handle_connect 0x7fe76c040170 2024-09-17T18:33:29.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.358+0000 7fe785a5c640 1 --2- 172.21.15.121:0/1960232340 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe76c040170 0x7fe76c042630 secure :-1 s=READY pgs=4050 cs=0 l=1 rev1=1 crypto rx=0x7fe780135990 tx=0x7fe770007db0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:29.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.364+0000 7fe7874bf640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:33:29.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.364+0000 7fe7874bf640 1 --2- 172.21.15.121:0/1960232340 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe78002d980 0x7fe78002dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:29.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.364+0000 7fe7874bf640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:29.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.364+0000 7fe7874bf640 1 -- 172.21.15.121:0/1960232340 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe7800709d0 con 0x7fe78002d980 2024-09-17T18:33:29.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.365+0000 7fe78525b640 1 --2- 172.21.15.121:0/1960232340 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe78002d980 0x7fe78002dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:29.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.365+0000 7fe78525b640 10 monclient: get_auth_request con 0x7fe78002d980 auth_method 0 2024-09-17T18:33:29.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.365+0000 7fe78525b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:29.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.365+0000 7fe78525b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:29.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.365+0000 7fe78525b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:29.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.365+0000 7fe78525b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:29.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.365+0000 7fe78525b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:29.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.365+0000 7fe78525b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:29.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.365+0000 7fe78525b640 10 monclient(hunting): handle_auth_done global_id 64376 payload 824 2024-09-17T18:33:29.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.365+0000 7fe78525b640 1 -- 172.21.15.121:0/1960232340 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe7801236e0 con 0x7fe78002d980 2024-09-17T18:33:29.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.367+0000 7fe78525b640 1 -- 172.21.15.121:0/1960232340 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe78002d980 msgr2=0x7fe78002dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:29.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.367+0000 7fe78525b640 1 --2- 172.21.15.121:0/1960232340 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe78002d980 0x7fe78002dd90 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe778017000 tx=0x7fe778004940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:29.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.367+0000 7fe77dffb640 10 client.94116.objecter ms_handle_connect 0x7fe78002d980 2024-09-17T18:33:29.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.367+0000 7fe77dffb640 10 client.94116.objecter resend_mon_ops 2024-09-17T18:33:29.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.367+0000 7fe77dffb640 1 -- 172.21.15.121:0/1960232340 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7780072e0 con 0x7fe78002d980 2024-09-17T18:33:29.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.367+0000 7fe77dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:29.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.367+0000 7fe77dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:33:29.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.367+0000 7fe77dffb640 10 monclient: dump: 2024-09-17T18:33:29.377 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:29.377 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:29.377 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.377 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.377 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:29.377 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:29.378 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:29.378 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:29.378 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:29.378 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:29.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.367+0000 7fe77dffb640 1 -- 172.21.15.121:0/1960232340 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7fe7780072e0 con 0x7fe78002d980 2024-09-17T18:33:29.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.367+0000 7fe77dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:33:29.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.367+0000 7fe77dffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:33:29.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.367+0000 7fe77dffb640 1 -- 172.21.15.121:0/1960232340 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe78002d980 msgr2=0x7fe78002dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:29.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.367+0000 7fe77dffb640 1 --2- 172.21.15.121:0/1960232340 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe78002d980 0x7fe78002dd90 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe778017000 tx=0x7fe778004940 comp rx=0 tx=0).stop 2024-09-17T18:33:29.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.367+0000 7fe77dffb640 1 -- 172.21.15.121:0/1960232340 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe78002d980 msgr2=0x7fe78002dd90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:33:29.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.367+0000 7fe77dffb640 1 --2- 172.21.15.121:0/1960232340 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe78002d980 0x7fe78002dd90 secure :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe778017000 tx=0x7fe778004940 comp rx=0 tx=0).stop 2024-09-17T18:33:29.379 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:33:29.379 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:33:29.379 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:33:29.379 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:33:29.379 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 54, 2024-09-17T18:33:29.379 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 388100, 2024-09-17T18:33:29.379 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:33:29.379 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:33:29.379 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:33:29.379 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:33:29.379 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:33:29.379 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 0, 2024-09-17T18:33:29.379 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:33:29.379 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:33:29.380 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:33:29.380 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:33:29.380 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:33:29.380 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:33:29.380 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:33:29.380 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:33:29.380 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:33:29.380 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:33:29.380 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:33:29.380 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:33:29.380 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:33:29.380 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:33:29.380 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:33:29.380 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:33:29.380 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:33:29.381 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:33:29.381 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:33:29.381 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:33:29.381 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:33:29.381 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:33:29.381 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:33:29.381 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:33:29.381 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:33:29.381 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:33:29.381 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:33:29.381 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:33:29.381 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:29.382 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:29.382 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:33:29.382 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:33:29.382 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:33:29.382 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:33:29.382 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:33:29.382 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:33:29.382 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:33:29.382 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:33:29.382 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:33:29.382 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:33:29.382 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:33:29.382 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:33:29.382 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:33:29.382 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:33:29.383 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:33:29.383 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:33:29.383 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:33:29.383 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:33:29.383 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:33:29.383 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:33:29.383 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:33:29.383 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:33:29.383 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:33:29.383 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:33:29.383 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:33:29.383 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:33:29.383 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:33:29.384 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:33:29.384 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:33:29.384 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:33:29.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:29.384 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:33:29.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:29.384 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:33:29.384 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:33:29.384 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:33:29.384 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:33:29.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:29.384 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:33:29.384 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:33:29.385 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:29.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:29.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:29.385 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:33:29.385 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:33:29.385 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:29.385 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:29.385 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:29.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:29.385 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:33:29.385 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:33:29.385 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:33:29.385 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:33:29.385 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:33:29.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:29.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:29.386 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:33:29.386 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:33:29.386 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:33:29.386 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:33:29.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:29.386 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:33:29.386 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:33:29.386 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:29.386 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:29.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:29.386 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:33:29.386 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:33:29.386 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:29.386 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:29.386 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:29.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:29.387 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:33:29.387 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:33:29.387 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:33:29.387 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:33:29.387 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:33:29.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:29.387 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:29.387 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:33:29.387 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:33:29.387 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:33:29.387 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:33:29.387 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:29.387 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:33:29.387 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:33:29.388 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:29.388 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:29.388 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:29.388 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:33:29.388 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:33:29.388 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:29.388 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:29.388 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:29.388 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:29.388 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:33:29.388 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:33:29.388 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:33:29.388 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:33:29.388 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:33:29.388 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:29.389 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:29.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:29.389 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:33:29.389 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:33:29.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:29.389 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:33:29.389 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:33:29.389 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:33:29.389 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:29.389 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:33:29.389 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:33:29.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:29.390 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:33:29.390 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:33:29.390 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 6 2024-09-17T18:33:29.390 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:29.390 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:33:29.390 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:33:29.390 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:29.390 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:33:29.390 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:33:29.390 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 3 2024-09-17T18:33:29.390 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:29.390 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:29.390 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:29.390 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:33:29.390 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:33:29.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.368+0000 7fe7874bf640 10 librados: watch_flush enter 2024-09-17T18:33:29.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.368+0000 7fe7874bf640 10 librados: watch_flush exit 2024-09-17T18:33:29.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.368+0000 7fe7874bf640 20 client.94116.objecter shutdown clearing up homeless session... 2024-09-17T18:33:29.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.368+0000 7fe7874bf640 10 client.94116.objecter successfully canceled tick 2024-09-17T18:33:29.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.368+0000 7fe7874bf640 1 -- 172.21.15.121:0/1960232340 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe76c040170 msgr2=0x7fe76c042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:29.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.368+0000 7fe7874bf640 1 --2- 172.21.15.121:0/1960232340 >> v2:172.21.15.173:6812/2423786037 conn(0x7fe76c040170 0x7fe76c042630 secure :-1 s=READY pgs=4050 cs=0 l=1 rev1=1 crypto rx=0x7fe780135990 tx=0x7fe770007db0 comp rx=0 tx=0).stop 2024-09-17T18:33:29.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.368+0000 7fe7874bf640 10 monclient: shutdown 2024-09-17T18:33:29.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.368+0000 7fe7874bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:33:29.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.368+0000 7fe7874bf640 1 -- 172.21.15.121:0/1960232340 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe780137c90 msgr2=0x7fe780121ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:29.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.368+0000 7fe7874bf640 1 --2- 172.21.15.121:0/1960232340 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe780137c90 0x7fe780121ab0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe774000f80 tx=0x7fe774004a40 comp rx=0 tx=0).stop 2024-09-17T18:33:29.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.368+0000 7fe785a5c640 1 -- 172.21.15.121:0/1960232340 reap_dead start 2024-09-17T18:33:29.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.368+0000 7fe785a5c640 1 -- 172.21.15.121:0/1960232340 reap_dead start 2024-09-17T18:33:29.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.368+0000 7fe7874bf640 1 -- 172.21.15.121:0/1960232340 shutdown_connections 2024-09-17T18:33:29.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.368+0000 7fe7874bf640 1 -- 172.21.15.121:0/1960232340 >> 172.21.15.121:0/1960232340 conn(0x7fe7800a7830 msgr2=0x7fe780136400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:33:29.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.368+0000 7fe7874bf640 1 -- 172.21.15.121:0/1960232340 shutdown_connections 2024-09-17T18:33:29.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.368+0000 7fe7874bf640 1 -- 172.21.15.121:0/1960232340 wait complete. 2024-09-17T18:33:29.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.368+0000 7fe7874bf640 1 librados: shutdown 2024-09-17T18:33:29.393 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:33:29.393 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T18:33:29.394 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T18:33:29.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.489+0000 7f4a233c6640 10 monclient: get_monmap_and_config 2024-09-17T18:33:29.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.489+0000 7f4a233c6640 10 monclient: build_initial_monmap 2024-09-17T18:33:29.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.489+0000 7f4a233c6640 10 monclient: monmap: 2024-09-17T18:33:29.491 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:33:29.491 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:29.491 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:33:29.489646+0000 2024-09-17T18:33:29.491 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:33:29.489646+0000 2024-09-17T18:33:29.491 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:33:29.491 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:29.491 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:33:29.491 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:33:29.491 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:33:29.491 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:33:29.491 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:33:29.491 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:29.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.489+0000 7f4a233c6640 1 Processor -- start 2024-09-17T18:33:29.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.489+0000 7f4a233c6640 1 -- start start 2024-09-17T18:33:29.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.489+0000 7f4a233c6640 10 monclient: init 2024-09-17T18:33:29.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.490+0000 7f4a233c6640 10 monclient: _reopen_session rank -1 2024-09-17T18:33:29.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.490+0000 7f4a233c6640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-17T18:33:29.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.490+0000 7f4a233c6640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4a1c0700f0 0x7f4a1c134e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:29.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.490+0000 7f4a233c6640 10 monclient(hunting): picked mon.noname-a con 0x7f4a1c0700f0 addr v2:172.21.15.121:3300/0 2024-09-17T18:33:29.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.490+0000 7f4a233c6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4a1c09e810 addr v1:172.21.15.173:6789/0 2024-09-17T18:33:29.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.490+0000 7f4a233c6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4a1c09ec60 addr v1:172.21.15.121:6789/0 2024-09-17T18:33:29.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.490+0000 7f4a233c6640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a1c139180 con 0x7f4a1c09ec60 2024-09-17T18:33:29.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.490+0000 7f4a233c6640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a1c09ca60 con 0x7f4a1c09e810 2024-09-17T18:33:29.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.490+0000 7f4a233c6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:29.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.490+0000 7f4a233c6640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f4a1c0704d0 con 0x7f4a1c0700f0 2024-09-17T18:33:29.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.490+0000 7f4a233c6640 10 monclient(hunting): _renew_subs 2024-09-17T18:33:29.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.490+0000 7f4a233c6640 10 monclient(hunting): authenticate will time out at 1613.206395s 2024-09-17T18:33:29.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.490+0000 7f4a21162640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4a1c0700f0 0x7f4a1c134e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:29.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.490+0000 7f4a21963640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f4a1c09ec60 0x7f4a1c138a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:57626/0 (socket says 172.21.15.121:57626) 2024-09-17T18:33:29.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.490+0000 7f4a21963640 1 -- 172.21.15.121:0/3966703617 learned_addr learned my addr 172.21.15.121:0/3966703617 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:33:29.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.490+0000 7f4a21162640 10 monclient(hunting): get_auth_request con 0x7f4a1c0700f0 auth_method 0 2024-09-17T18:33:29.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.490+0000 7f4a21162640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:29.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.490+0000 7f4a21162640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:29.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.490+0000 7f4a21162640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:29.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.490+0000 7f4a21162640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:29.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.490+0000 7f4a21162640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:29.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.490+0000 7f4a21162640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:29.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.490+0000 7f4a21162640 10 monclient(hunting): handle_auth_done global_id 94119 payload 274 2024-09-17T18:33:29.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.490+0000 7f4a21162640 1 -- 172.21.15.121:0/3966703617 >> v1:172.21.15.173:6789/0 conn(0x7f4a1c09e810 legacy=0x7f4a1c138360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:33:29.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.490+0000 7f4a21162640 1 -- 172.21.15.121:0/3966703617 >> v1:172.21.15.121:6789/0 conn(0x7f4a1c09ec60 legacy=0x7f4a1c138a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:33:29.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.490+0000 7f4a21162640 10 monclient: _finish_hunting 0 2024-09-17T18:33:29.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.490+0000 7f4a21162640 1 monclient: found mon.noname-a 2024-09-17T18:33:29.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.490+0000 7f4a21162640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:33:29.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.490+0000 7f4a21162640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:33:29.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.490+0000 7f4a21162640 1 -- 172.21.15.121:0/3966703617 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a1c09cd20 con 0x7f4a1c0700f0 2024-09-17T18:33:29.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.493+0000 7f4a21162640 1 -- 172.21.15.121:0/3966703617 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a1c0700f0 msgr2=0x7f4a1c134e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:29.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.493+0000 7f4a21162640 1 --2- 172.21.15.121:0/3966703617 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a1c0700f0 0x7f4a1c134e50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4a140097a0 tx=0x7f4a14029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80861a2cc62aa5bd server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:29.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.493+0000 7f4a1bfff640 1 -- 172.21.15.121:0/3966703617 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a14037070 con 0x7f4a1c0700f0 2024-09-17T18:33:29.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.493+0000 7f4a1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:29.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.493+0000 7f4a1bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:33:29.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.493+0000 7f4a1bfff640 10 monclient: dump: 2024-09-17T18:33:29.501 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:29.501 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:29.501 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.501 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.501 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:29.501 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:29.501 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:29.501 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:29.501 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:29.502 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:29.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.493+0000 7f4a1bfff640 10 monclient: _finish_auth 0 2024-09-17T18:33:29.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.493+0000 7f4a1bfff640 10 monclient: _check_auth_tickets 2024-09-17T18:33:29.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.493+0000 7f4a1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:33:29.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.493+0000 7f4a1bfff640 1 -- 172.21.15.121:0/3966703617 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a14032640 con 0x7f4a1c0700f0 2024-09-17T18:33:29.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.493+0000 7f4a233c6640 5 monclient: authenticate success, global_id 94119 2024-09-17T18:33:29.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.493+0000 7f4a233c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:33:29.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.493+0000 7f4a1bfff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:33:29.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.493+0000 7f4a233c6640 10 monclient: get_monmap_and_config success 2024-09-17T18:33:29.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.493+0000 7f4a1bfff640 1 -- 172.21.15.121:0/3966703617 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a14032a10 con 0x7f4a1c0700f0 2024-09-17T18:33:29.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.493+0000 7f4a233c6640 4 set_mon_vals no callback set 2024-09-17T18:33:29.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.493+0000 7f4a1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:29.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.493+0000 7f4a1bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:33:29.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.493+0000 7f4a1bfff640 10 monclient: dump: 2024-09-17T18:33:29.502 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:29.503 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:29.503 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.503 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.503 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:29.503 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:29.503 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:29.503 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:29.503 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:29.503 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:29.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.494+0000 7f4a233c6640 10 monclient: shutdown 2024-09-17T18:33:29.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.494+0000 7f4a233c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:33:29.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.494+0000 7f4a233c6640 1 -- 172.21.15.121:0/3966703617 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a1c0700f0 msgr2=0x7f4a1c134e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:29.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.494+0000 7f4a233c6640 1 --2- 172.21.15.121:0/3966703617 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a1c0700f0 0x7f4a1c134e50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4a140097a0 tx=0x7f4a14029fd0 comp rx=0 tx=0).stop 2024-09-17T18:33:29.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.494+0000 7f4a233c6640 1 -- 172.21.15.121:0/3966703617 shutdown_connections 2024-09-17T18:33:29.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.494+0000 7f4a233c6640 1 --2- 172.21.15.121:0/3966703617 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a1c0700f0 0x7f4a1c134e50 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:29.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.494+0000 7f4a233c6640 1 -- 172.21.15.121:0/3966703617 >> 172.21.15.121:0/3966703617 conn(0x7f4a1c0ab830 msgr2=0x7f4a1c0a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:33:29.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.494+0000 7f4a233c6640 1 -- 172.21.15.121:0/3966703617 shutdown_connections 2024-09-17T18:33:29.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.494+0000 7f4a233c6640 1 -- 172.21.15.121:0/3966703617 wait complete. 2024-09-17T18:33:29.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.494+0000 7f4a233c6640 10 monclient: build_initial_monmap 2024-09-17T18:33:29.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.494+0000 7f4a233c6640 10 monclient: monmap: 2024-09-17T18:33:29.504 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:33:29.504 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:33:29.504 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:33:29.504 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:33:29.504 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:33:29.504 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:29.504 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:33:29.504 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:33:29.504 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:33:29.504 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:29.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.495+0000 7f4a233c6640 1 librados: starting msgr at 2024-09-17T18:33:29.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.495+0000 7f4a233c6640 1 librados: starting objecter 2024-09-17T18:33:29.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.495+0000 7f4a233c6640 1 Processor -- start 2024-09-17T18:33:29.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.495+0000 7f4a233c6640 1 -- start start 2024-09-17T18:33:29.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.495+0000 7f4a233c6640 1 librados: setting wanted keys 2024-09-17T18:33:29.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.495+0000 7f4a233c6640 1 librados: calling monclient init 2024-09-17T18:33:29.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.495+0000 7f4a233c6640 10 monclient: init 2024-09-17T18:33:29.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.495+0000 7f4a233c6640 10 monclient: _reopen_session rank -1 2024-09-17T18:33:29.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.495+0000 7f4a233c6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T18:33:29.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.495+0000 7f4a233c6640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a1c0700f0 0x7f4a1c131c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:29.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.495+0000 7f4a233c6640 10 monclient(hunting): picked mon.noname-b con 0x7f4a1c0700f0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:33:29.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.495+0000 7f4a233c6640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a1c09e810 0x7f4a1c132170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:29.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.495+0000 7f4a233c6640 10 monclient(hunting): picked mon.noname-a con 0x7f4a1c09e810 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:33:29.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.495+0000 7f4a233c6640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a1c09ec60 0x7f4a1c136fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:29.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.495+0000 7f4a233c6640 10 monclient(hunting): picked mon.noname-c con 0x7f4a1c09ec60 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:33:29.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.495+0000 7f4a233c6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:29.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.495+0000 7f4a233c6640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4a1c135390 con 0x7f4a1c09e810 2024-09-17T18:33:29.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.495+0000 7f4a233c6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:29.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.495+0000 7f4a233c6640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4a1c09d2e0 con 0x7f4a1c0700f0 2024-09-17T18:33:29.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.495+0000 7f4a233c6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:29.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.495+0000 7f4a233c6640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4a1c135510 con 0x7f4a1c09ec60 2024-09-17T18:33:29.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.495+0000 7f4a233c6640 10 monclient(hunting): _renew_subs 2024-09-17T18:33:29.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.495+0000 7f4a233c6640 10 monclient(hunting): authenticate will time out at 1613.211753s 2024-09-17T18:33:29.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.495+0000 7f4a20961640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a1c09e810 0x7f4a1c132170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:29.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.495+0000 7f4a20961640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a1c09e810 0x7f4a1c132170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60430/0 (socket says 172.21.15.121:60430) 2024-09-17T18:33:29.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.495+0000 7f4a21963640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a1c09ec60 0x7f4a1c136fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:29.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.495+0000 7f4a20961640 1 -- 172.21.15.121:0/2762203911 learned_addr learned my addr 172.21.15.121:0/2762203911 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:33:29.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.495+0000 7f4a20961640 10 monclient(hunting): get_auth_request con 0x7f4a1c09e810 auth_method 0 2024-09-17T18:33:29.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.495+0000 7f4a20961640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:29.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.495+0000 7f4a20961640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:29.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.495+0000 7f4a20961640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:29.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.495+0000 7f4a21963640 10 monclient(hunting): get_auth_request con 0x7f4a1c09ec60 auth_method 0 2024-09-17T18:33:29.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.495+0000 7f4a21963640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:29.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.495+0000 7f4a21963640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:29.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.495+0000 7f4a21963640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:29.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.495+0000 7f4a20961640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:29.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.495+0000 7f4a20961640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:29.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.495+0000 7f4a20961640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:29.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.495+0000 7f4a21963640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:29.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.495+0000 7f4a21963640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:29.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.495+0000 7f4a21963640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:29.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.496+0000 7f4a21162640 1 --2- 172.21.15.121:0/2762203911 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a1c0700f0 0x7f4a1c131c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:29.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.496+0000 7f4a20961640 10 monclient(hunting): handle_auth_done global_id 94122 payload 824 2024-09-17T18:33:29.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.496+0000 7f4a20961640 1 -- 172.21.15.121:0/2762203911 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a1c09ec60 msgr2=0x7f4a1c136fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:29.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.496+0000 7f4a20961640 1 --2- 172.21.15.121:0/2762203911 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a1c09ec60 0x7f4a1c136fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:29.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.496+0000 7f4a20961640 1 -- 172.21.15.121:0/2762203911 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a1c0700f0 msgr2=0x7f4a1c131c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:29.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.496+0000 7f4a20961640 1 --2- 172.21.15.121:0/2762203911 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a1c0700f0 0x7f4a1c131c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:29.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.496+0000 7f4a20961640 10 monclient: _finish_hunting 0 2024-09-17T18:33:29.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.496+0000 7f4a20961640 1 monclient: found mon.noname-a 2024-09-17T18:33:29.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.496+0000 7f4a20961640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:33:29.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.496+0000 7f4a20961640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:33:29.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.496+0000 7f4a20961640 1 -- 172.21.15.121:0/2762203911 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a1c13ae10 con 0x7f4a1c09e810 2024-09-17T18:33:29.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.496+0000 7f4a21162640 10 monclient: get_auth_request con 0x7f4a1c0700f0 auth_method 0 2024-09-17T18:33:29.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.496+0000 7f4a21162640 1 --2- 172.21.15.121:0/2762203911 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a1c0700f0 0x7f4a1c131c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T18:33:29.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.498+0000 7f4a20961640 1 -- 172.21.15.121:0/2762203911 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a1c09e810 msgr2=0x7f4a1c132170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:29.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.498+0000 7f4a20961640 1 --2- 172.21.15.121:0/2762203911 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a1c09e810 0x7f4a1c132170 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4a1000c7c0 tx=0x7f4a1000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:29.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.498+0000 7f4a19ffb640 10 client.?.objecter ms_handle_connect 0x7f4a1c09e810 2024-09-17T18:33:29.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.498+0000 7f4a19ffb640 10 client.?.objecter resend_mon_ops 2024-09-17T18:33:29.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.498+0000 7f4a19ffb640 1 -- 172.21.15.121:0/2762203911 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a10016020 con 0x7f4a1c09e810 2024-09-17T18:33:29.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.498+0000 7f4a19ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:29.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.498+0000 7f4a19ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:33:29.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.498+0000 7f4a19ffb640 10 monclient: dump: 2024-09-17T18:33:29.509 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:29.509 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:29.509 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.509 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.509 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:29.509 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:29.509 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:29.509 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:29.509 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:29.509 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:29.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.498+0000 7f4a19ffb640 10 monclient: _finish_auth 0 2024-09-17T18:33:29.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.498+0000 7f4a19ffb640 10 monclient: _check_auth_tickets 2024-09-17T18:33:29.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.498+0000 7f4a19ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:33:29.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.498+0000 7f4a19ffb640 1 -- 172.21.15.121:0/2762203911 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a1000ac00 con 0x7f4a1c09e810 2024-09-17T18:33:29.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.498+0000 7f4a19ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:33:29.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.498+0000 7f4a19ffb640 1 -- 172.21.15.121:0/2762203911 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a10012420 con 0x7f4a1c09e810 2024-09-17T18:33:29.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.498+0000 7f4a233c6640 5 monclient: authenticate success, global_id 94122 2024-09-17T18:33:29.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.498+0000 7f4a19ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:29.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.498+0000 7f4a19ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:33:29.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.498+0000 7f4a1b7fe640 4 set_mon_vals no callback set 2024-09-17T18:33:29.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.498+0000 7f4a19ffb640 10 monclient: dump: 2024-09-17T18:33:29.510 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:29.510 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:29.510 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.510 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.510 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:29.510 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:29.510 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:29.510 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:29.511 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:29.511 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:29.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.498+0000 7f4a233c6640 10 monclient: _renew_subs 2024-09-17T18:33:29.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.498+0000 7f4a233c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:33:29.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.498+0000 7f4a233c6640 1 -- 172.21.15.121:0/2762203911 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a1c13b0f0 con 0x7f4a1c09e810 2024-09-17T18:33:29.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.498+0000 7f4a233c6640 10 client.94122.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:33:29.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.498+0000 7f4a233c6640 10 monclient: _renew_subs 2024-09-17T18:33:29.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.498+0000 7f4a233c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:33:29.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.498+0000 7f4a233c6640 1 -- 172.21.15.121:0/2762203911 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a1c13ab30 con 0x7f4a1c09e810 2024-09-17T18:33:29.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.498+0000 7f4a233c6640 1 librados: init done 2024-09-17T18:33:29.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.499+0000 7f4a19ffb640 1 -- 172.21.15.121:0/2762203911 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f4a10019020 con 0x7f4a1c09e810 2024-09-17T18:33:29.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.499+0000 7f4a233c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:33:29.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.499+0000 7f4a233c6640 1 --2- 172.21.15.121:0/2762203911 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a1c137520 0x7f4a1c137900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:29.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.499+0000 7f4a233c6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:29.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.499+0000 7f4a233c6640 1 -- 172.21.15.121:0/2762203911 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4a1c134e50 con 0x7f4a1c137520 2024-09-17T18:33:29.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.499+0000 7f4a19ffb640 1 --2- 172.21.15.121:0/2762203911 >> v2:172.21.15.173:6812/2423786037 conn(0x7f4a080401a0 0x7f4a08042660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:29.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.499+0000 7f4a19ffb640 1 -- 172.21.15.121:0/2762203911 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(561..561 src has 247..561) ==== 5925+0+0 (secure 0 0 0) 0x7f4a10053660 con 0x7f4a1c09e810 2024-09-17T18:33:29.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.499+0000 7f4a19ffb640 10 client.94122.objecter ms_dispatch 0x7f4a1c000b90 osd_map(561..561 src has 247..561) 2024-09-17T18:33:29.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.500+0000 7f4a19ffb640 3 client.94122.objecter handle_osd_map got epochs [561,561] > 0 2024-09-17T18:33:29.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.500+0000 7f4a19ffb640 3 client.94122.objecter handle_osd_map decoding full epoch 561 2024-09-17T18:33:29.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.500+0000 7f4a21162640 1 --2- 172.21.15.121:0/2762203911 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a1c137520 0x7f4a1c137900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:29.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.500+0000 7f4a19ffb640 20 client.94122.objecter dump_active .. 0 homeless 2024-09-17T18:33:29.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.500+0000 7f4a21162640 10 monclient: get_auth_request con 0x7f4a1c137520 auth_method 0 2024-09-17T18:33:29.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.500+0000 7f4a21162640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:29.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.500+0000 7f4a21162640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:29.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.500+0000 7f4a21162640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:29.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.500+0000 7f4a21963640 1 --2- 172.21.15.121:0/2762203911 >> v2:172.21.15.173:6812/2423786037 conn(0x7f4a080401a0 0x7f4a08042660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:29.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.500+0000 7f4a21963640 10 monclient: get_auth_request con 0x7f4a080401a0 auth_method 0 2024-09-17T18:33:29.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.500+0000 7f4a21162640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:29.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.500+0000 7f4a21162640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:29.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.500+0000 7f4a21162640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:29.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.500+0000 7f4a21162640 10 monclient(hunting): handle_auth_done global_id 84172 payload 824 2024-09-17T18:33:29.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.500+0000 7f4a21162640 1 -- 172.21.15.121:0/2762203911 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4a1c1410b0 con 0x7f4a1c137520 2024-09-17T18:33:29.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.502+0000 7f4a21963640 1 --2- 172.21.15.121:0/2762203911 >> v2:172.21.15.173:6812/2423786037 conn(0x7f4a080401a0 0x7f4a08042660 secure :-1 s=READY pgs=4051 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c007c90 tx=0x7f4a0c007be0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:29.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.502+0000 7f4a21162640 1 -- 172.21.15.121:0/2762203911 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a1c137520 msgr2=0x7f4a1c137900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:29.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.502+0000 7f4a19ffb640 10 client.94122.objecter ms_handle_connect 0x7f4a080401a0 2024-09-17T18:33:29.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.502+0000 7f4a21162640 1 --2- 172.21.15.121:0/2762203911 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a1c137520 0x7f4a1c137900 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c070930 tx=0x7f4a140046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:29.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.502+0000 7f4a19ffb640 10 client.94122.objecter ms_handle_connect 0x7f4a1c137520 2024-09-17T18:33:29.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.502+0000 7f4a19ffb640 10 client.94122.objecter resend_mon_ops 2024-09-17T18:33:29.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.502+0000 7f4a19ffb640 1 -- 172.21.15.121:0/2762203911 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a14042070 con 0x7f4a1c137520 2024-09-17T18:33:29.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.502+0000 7f4a19ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:29.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.503+0000 7f4a19ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:33:29.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.503+0000 7f4a19ffb640 10 monclient: dump: 2024-09-17T18:33:29.517 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:29.517 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:29.518 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.518 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.518 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:29.518 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:29.518 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:29.518 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:29.518 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:29.518 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:29.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.503+0000 7f4a19ffb640 1 -- 172.21.15.121:0/2762203911 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4a1403e870 con 0x7f4a1c137520 2024-09-17T18:33:29.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.503+0000 7f4a19ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:33:29.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.503+0000 7f4a19ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:33:29.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.503+0000 7f4a19ffb640 1 -- 172.21.15.121:0/2762203911 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a1c137520 msgr2=0x7f4a1c137900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:29.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.503+0000 7f4a19ffb640 1 --2- 172.21.15.121:0/2762203911 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a1c137520 0x7f4a1c137900 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c070930 tx=0x7f4a140046c0 comp rx=0 tx=0).stop 2024-09-17T18:33:29.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.503+0000 7f4a19ffb640 1 -- 172.21.15.121:0/2762203911 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a1c137520 msgr2=0x7f4a1c137900 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:33:29.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.503+0000 7f4a19ffb640 1 --2- 172.21.15.121:0/2762203911 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a1c137520 0x7f4a1c137900 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c070930 tx=0x7f4a140046c0 comp rx=0 tx=0).stop 2024-09-17T18:33:29.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.509+0000 7f4a233c6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:33:29.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.509+0000 7f4a233c6640 1 --2- 172.21.15.121:0/2762203911 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a1c137e40 0x7f4a1c09f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:29.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.509+0000 7f4a233c6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:29.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.509+0000 7f4a233c6640 1 -- 172.21.15.121:0/2762203911 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4a1c139650 con 0x7f4a1c137e40 2024-09-17T18:33:29.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.509+0000 7f4a21162640 1 --2- 172.21.15.121:0/2762203911 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a1c137e40 0x7f4a1c09f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:29.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.509+0000 7f4a21162640 10 monclient: get_auth_request con 0x7f4a1c137e40 auth_method 0 2024-09-17T18:33:29.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.509+0000 7f4a21162640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:29.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.509+0000 7f4a21162640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:29.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.509+0000 7f4a21162640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:29.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.509+0000 7f4a21162640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:29.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.509+0000 7f4a21162640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:29.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.509+0000 7f4a21162640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:29.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.509+0000 7f4a21162640 10 monclient(hunting): handle_auth_done global_id 84175 payload 824 2024-09-17T18:33:29.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.509+0000 7f4a21162640 1 -- 172.21.15.121:0/2762203911 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4a1c06a660 con 0x7f4a1c137e40 2024-09-17T18:33:29.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.512+0000 7f4a21162640 1 -- 172.21.15.121:0/2762203911 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a1c137e40 msgr2=0x7f4a1c09f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:29.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.512+0000 7f4a21162640 1 --2- 172.21.15.121:0/2762203911 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a1c137e40 0x7f4a1c09f0a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4a1402a010 tx=0x7f4a14009190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:29.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.512+0000 7f4a19ffb640 10 client.94122.objecter ms_handle_connect 0x7f4a1c137e40 2024-09-17T18:33:29.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.512+0000 7f4a19ffb640 10 client.94122.objecter resend_mon_ops 2024-09-17T18:33:29.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.512+0000 7f4a19ffb640 1 -- 172.21.15.121:0/2762203911 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a14037040 con 0x7f4a1c137e40 2024-09-17T18:33:29.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.512+0000 7f4a19ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:29.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.512+0000 7f4a19ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:33:29.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.512+0000 7f4a19ffb640 10 monclient: dump: 2024-09-17T18:33:29.520 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:29.520 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:29.520 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.520 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.520 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:29.520 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:29.520 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:29.520 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:29.521 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:29.521 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:29.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.513+0000 7f4a19ffb640 1 -- 172.21.15.121:0/2762203911 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7f4a1403cc80 con 0x7f4a1c137e40 2024-09-17T18:33:29.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.513+0000 7f4a19ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:33:29.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.513+0000 7f4a19ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:33:29.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.513+0000 7f4a19ffb640 1 -- 172.21.15.121:0/2762203911 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a1c137e40 msgr2=0x7f4a1c09f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:29.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.513+0000 7f4a19ffb640 1 --2- 172.21.15.121:0/2762203911 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a1c137e40 0x7f4a1c09f0a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4a1402a010 tx=0x7f4a14009190 comp rx=0 tx=0).stop 2024-09-17T18:33:29.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.513+0000 7f4a19ffb640 1 -- 172.21.15.121:0/2762203911 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a1c137e40 msgr2=0x7f4a1c09f0a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:33:29.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.513+0000 7f4a19ffb640 1 --2- 172.21.15.121:0/2762203911 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a1c137e40 0x7f4a1c09f0a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:29.521 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:33:29.521 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:33:29.521 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:33:29.521 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:33:29.522 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 54, 2024-09-17T18:33:29.522 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 134610, 2024-09-17T18:33:29.522 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:33:29.522 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:33:29.522 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:33:29.522 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:33:29.522 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:33:29.522 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 0, 2024-09-17T18:33:29.522 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:33:29.522 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:33:29.522 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:33:29.522 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:33:29.522 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:33:29.522 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:33:29.523 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:33:29.523 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:33:29.523 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:33:29.523 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:33:29.523 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:33:29.523 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:33:29.523 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:33:29.523 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:33:29.523 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:33:29.523 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:33:29.523 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:33:29.523 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:33:29.523 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:33:29.523 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:33:29.524 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:33:29.524 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:33:29.524 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:33:29.524 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:33:29.524 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:33:29.524 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:33:29.524 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:33:29.524 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:33:29.524 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:29.524 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:29.524 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:33:29.524 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:33:29.524 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:33:29.525 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:33:29.525 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:33:29.525 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:33:29.525 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:33:29.525 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:33:29.525 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:33:29.525 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:33:29.525 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:33:29.525 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:33:29.525 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:33:29.525 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:33:29.525 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:33:29.525 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:33:29.525 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:33:29.525 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:33:29.526 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:33:29.526 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:33:29.526 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:33:29.526 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:33:29.526 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:33:29.526 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:33:29.526 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:33:29.526 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:33:29.526 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:33:29.526 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:33:29.526 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:33:29.526 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:33:29.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:29.526 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:33:29.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:29.526 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:33:29.527 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:33:29.527 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:33:29.527 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:33:29.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:29.527 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:33:29.527 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:33:29.527 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:29.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:29.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:29.527 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:33:29.527 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:33:29.527 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:29.527 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:29.527 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:29.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:29.527 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:33:29.528 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:33:29.528 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:33:29.528 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:33:29.528 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:33:29.528 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:29.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:29.528 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:33:29.528 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:33:29.528 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:33:29.528 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:33:29.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:29.529 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:33:29.529 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:33:29.529 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:29.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:29.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:29.529 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:33:29.529 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:33:29.529 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:29.529 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:29.529 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:29.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:29.529 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:33:29.529 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:33:29.529 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:33:29.529 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:33:29.530 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:33:29.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:29.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:29.530 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:33:29.530 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:33:29.530 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:33:29.530 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:33:29.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:29.530 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:33:29.530 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:33:29.530 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:29.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:29.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:29.530 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:33:29.530 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:33:29.530 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:29.531 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:29.531 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:29.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:29.531 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:33:29.531 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:33:29.531 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:33:29.531 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:33:29.531 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:33:29.531 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:29.531 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:29.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:29.531 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:33:29.531 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:33:29.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:29.531 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:33:29.532 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:33:29.532 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:33:29.532 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:29.532 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:33:29.532 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:33:29.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:29.532 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:33:29.532 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:33:29.532 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 4 2024-09-17T18:33:29.532 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:29.533 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:33:29.533 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": [ 2024-09-17T18:33:29.533 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:29.533 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:33:29.533 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:33:29.533 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:33:29.533 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:29.533 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:29.533 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:29.533 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:33:29.533 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:33:29.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.513+0000 7f4a233c6640 10 librados: watch_flush enter 2024-09-17T18:33:29.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.513+0000 7f4a233c6640 10 librados: watch_flush exit 2024-09-17T18:33:29.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.513+0000 7f4a233c6640 20 client.94122.objecter shutdown clearing up homeless session... 2024-09-17T18:33:29.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.513+0000 7f4a233c6640 10 client.94122.objecter successfully canceled tick 2024-09-17T18:33:29.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.513+0000 7f4a233c6640 1 -- 172.21.15.121:0/2762203911 >> v2:172.21.15.173:6812/2423786037 conn(0x7f4a080401a0 msgr2=0x7f4a08042660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:29.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.513+0000 7f4a233c6640 1 --2- 172.21.15.121:0/2762203911 >> v2:172.21.15.173:6812/2423786037 conn(0x7f4a080401a0 0x7f4a08042660 secure :-1 s=READY pgs=4051 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c007c90 tx=0x7f4a0c007be0 comp rx=0 tx=0).stop 2024-09-17T18:33:29.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.513+0000 7f4a233c6640 10 monclient: shutdown 2024-09-17T18:33:29.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.513+0000 7f4a233c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:33:29.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.513+0000 7f4a233c6640 1 -- 172.21.15.121:0/2762203911 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a1c09e810 msgr2=0x7f4a1c132170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:29.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.513+0000 7f4a233c6640 1 --2- 172.21.15.121:0/2762203911 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a1c09e810 0x7f4a1c132170 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4a1000c7c0 tx=0x7f4a1000cc80 comp rx=0 tx=0).stop 2024-09-17T18:33:29.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.513+0000 7f4a21963640 1 -- 172.21.15.121:0/2762203911 reap_dead start 2024-09-17T18:33:29.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.513+0000 7f4a21963640 1 -- 172.21.15.121:0/2762203911 reap_dead start 2024-09-17T18:33:29.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.513+0000 7f4a233c6640 1 -- 172.21.15.121:0/2762203911 shutdown_connections 2024-09-17T18:33:29.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.513+0000 7f4a233c6640 1 -- 172.21.15.121:0/2762203911 >> 172.21.15.121:0/2762203911 conn(0x7f4a1c0ab830 msgr2=0x7f4a1c132fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:33:29.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.513+0000 7f4a233c6640 1 -- 172.21.15.121:0/2762203911 shutdown_connections 2024-09-17T18:33:29.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.513+0000 7f4a233c6640 1 -- 172.21.15.121:0/2762203911 wait complete. 2024-09-17T18:33:29.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.513+0000 7f4a233c6640 1 librados: shutdown 2024-09-17T18:33:29.536 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:33:29.536 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-17T18:33:29.536 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-17T18:33:29.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.631+0000 7f61847cd640 10 monclient: get_monmap_and_config 2024-09-17T18:33:29.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.631+0000 7f61847cd640 10 monclient: build_initial_monmap 2024-09-17T18:33:29.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.631+0000 7f61847cd640 10 monclient: monmap: 2024-09-17T18:33:29.633 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:33:29.633 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:29.633 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:33:29.632202+0000 2024-09-17T18:33:29.633 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:33:29.632202+0000 2024-09-17T18:33:29.633 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:33:29.634 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:29.634 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:33:29.634 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:33:29.634 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:33:29.634 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:33:29.634 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:33:29.634 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:29.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.632+0000 7f61847cd640 1 Processor -- start 2024-09-17T18:33:29.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.632+0000 7f61847cd640 1 -- start start 2024-09-17T18:33:29.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.632+0000 7f61847cd640 10 monclient: init 2024-09-17T18:33:29.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.632+0000 7f61847cd640 10 monclient: _reopen_session rank -1 2024-09-17T18:33:29.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.632+0000 7f61847cd640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-17T18:33:29.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.632+0000 7f61847cd640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f617c136800 0x7f617c136170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:29.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.632+0000 7f61847cd640 10 monclient(hunting): picked mon.noname-c con 0x7f617c136800 addr v2:172.21.15.173:3300/0 2024-09-17T18:33:29.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.632+0000 7f61847cd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f617c136be0 addr v1:172.21.15.173:6789/0 2024-09-17T18:33:29.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.632+0000 7f61847cd640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f617c133360 0x7f617c13e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:29.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.632+0000 7f61847cd640 10 monclient(hunting): picked mon.noname-a con 0x7f617c133360 addr v2:172.21.15.121:3300/0 2024-09-17T18:33:29.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.632+0000 7f61847cd640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f617c0a0c20 con 0x7f617c136be0 2024-09-17T18:33:29.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.632+0000 7f61847cd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:29.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.632+0000 7f61847cd640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f617c0a2cb0 con 0x7f617c133360 2024-09-17T18:33:29.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.632+0000 7f61847cd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:29.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.632+0000 7f61847cd640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f617c0a2f90 con 0x7f617c136800 2024-09-17T18:33:29.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.632+0000 7f61847cd640 10 monclient(hunting): _renew_subs 2024-09-17T18:33:29.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.632+0000 7f61847cd640 10 monclient(hunting): authenticate will time out at 1613.348813s 2024-09-17T18:33:29.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.632+0000 7f6182d6a640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f617c133360 0x7f617c13e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:29.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.632+0000 7f6182d6a640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f617c133360 0x7f617c13e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60440/0 (socket says 172.21.15.121:60440) 2024-09-17T18:33:29.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.632+0000 7f6182d6a640 1 -- 172.21.15.121:0/3635914633 learned_addr learned my addr 172.21.15.121:0/3635914633 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:33:29.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.632+0000 7f6182d6a640 10 monclient(hunting): get_auth_request con 0x7f617c133360 auth_method 0 2024-09-17T18:33:29.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.632+0000 7f6182d6a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:29.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.632+0000 7f6182d6a640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:29.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.632+0000 7f6182d6a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:29.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.633+0000 7f6182d6a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:29.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.633+0000 7f6182d6a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:29.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.633+0000 7f6182d6a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:29.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.633+0000 7f6182d6a640 10 monclient(hunting): handle_auth_done global_id 94125 payload 274 2024-09-17T18:33:29.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.633+0000 7f6182d6a640 1 -- 172.21.15.121:0/3635914633 >> v2:172.21.15.173:3300/0 conn(0x7f617c136800 msgr2=0x7f617c136170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:33:29.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.633+0000 7f6182569640 1 --2- 172.21.15.121:0/3635914633 >> v2:172.21.15.173:3300/0 conn(0x7f617c136800 0x7f617c136170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:29.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.633+0000 7f6182d6a640 1 --2- 172.21.15.121:0/3635914633 >> v2:172.21.15.173:3300/0 conn(0x7f617c136800 0x7f617c136170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:29.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.633+0000 7f6182d6a640 1 -- 172.21.15.121:0/3635914633 >> v1:172.21.15.173:6789/0 conn(0x7f617c136be0 legacy=0x7f617c132c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:33:29.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.633+0000 7f6182d6a640 10 monclient: _finish_hunting 0 2024-09-17T18:33:29.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.633+0000 7f6182d6a640 1 monclient: found mon.noname-a 2024-09-17T18:33:29.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.633+0000 7f6182d6a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:33:29.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.633+0000 7f6182d6a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:33:29.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.633+0000 7f6182d6a640 1 -- 172.21.15.121:0/3635914633 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f617c0a3270 con 0x7f617c133360 2024-09-17T18:33:29.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.633+0000 7f6182569640 10 monclient: get_auth_request con 0x7f617c136800 auth_method 0 2024-09-17T18:33:29.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.633+0000 7f6182569640 1 --2- 172.21.15.121:0/3635914633 >> v2:172.21.15.173:3300/0 conn(0x7f617c136800 0x7f617c136170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T18:33:29.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.635+0000 7f6182d6a640 1 -- 172.21.15.121:0/3635914633 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f617c133360 msgr2=0x7f617c13e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:29.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.635+0000 7f6182d6a640 1 --2- 172.21.15.121:0/3635914633 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f617c133360 0x7f617c13e510 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f617c076d00 tx=0x7f617402a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1014d6f150040875 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:29.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.636+0000 7f6181567640 1 -- 172.21.15.121:0/3635914633 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6174036070 con 0x7f617c133360 2024-09-17T18:33:29.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.636+0000 7f6181567640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:29.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.636+0000 7f6181567640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:33:29.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.636+0000 7f6181567640 10 monclient: dump: 2024-09-17T18:33:29.645 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:29.645 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:29.645 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.645 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.646 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:29.646 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:29.646 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:29.646 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:29.646 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:29.646 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:29.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.636+0000 7f6181567640 10 monclient: _finish_auth 0 2024-09-17T18:33:29.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.636+0000 7f6181567640 10 monclient: _check_auth_tickets 2024-09-17T18:33:29.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.636+0000 7f6181567640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:33:29.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.636+0000 7f6181567640 1 -- 172.21.15.121:0/3635914633 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6174007800 con 0x7f617c133360 2024-09-17T18:33:29.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.636+0000 7f61847cd640 5 monclient: authenticate success, global_id 94125 2024-09-17T18:33:29.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.636+0000 7f61847cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:33:29.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.636+0000 7f6181567640 10 monclient: handle_config config(0 keys) 2024-09-17T18:33:29.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.636+0000 7f61847cd640 10 monclient: get_monmap_and_config success 2024-09-17T18:33:29.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.636+0000 7f6181567640 1 -- 172.21.15.121:0/3635914633 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6174007bd0 con 0x7f617c133360 2024-09-17T18:33:29.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.636+0000 7f6181567640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:29.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.636+0000 7f61847cd640 4 set_mon_vals no callback set 2024-09-17T18:33:29.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.636+0000 7f6181567640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:33:29.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.636+0000 7f6181567640 10 monclient: dump: 2024-09-17T18:33:29.647 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:29.647 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:29.647 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.647 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.647 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:29.647 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:29.647 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:29.647 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:29.648 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:29.648 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:29.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.636+0000 7f61847cd640 10 monclient: shutdown 2024-09-17T18:33:29.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.636+0000 7f61847cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:33:29.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.636+0000 7f61847cd640 1 -- 172.21.15.121:0/3635914633 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f617c133360 msgr2=0x7f617c13e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:29.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.636+0000 7f61847cd640 1 --2- 172.21.15.121:0/3635914633 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f617c133360 0x7f617c13e510 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f617c076d00 tx=0x7f617402a570 comp rx=0 tx=0).stop 2024-09-17T18:33:29.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.636+0000 7f61847cd640 1 -- 172.21.15.121:0/3635914633 shutdown_connections 2024-09-17T18:33:29.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.636+0000 7f61847cd640 1 --2- 172.21.15.121:0/3635914633 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f617c133360 0x7f617c13e510 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:29.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.636+0000 7f61847cd640 1 --2- 172.21.15.121:0/3635914633 >> v2:172.21.15.173:3300/0 conn(0x7f617c136800 0x7f617c136170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:29.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.636+0000 7f61847cd640 1 -- 172.21.15.121:0/3635914633 >> 172.21.15.121:0/3635914633 conn(0x7f617c0a3b50 msgr2=0x7f617c0a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:33:29.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.636+0000 7f61847cd640 1 -- 172.21.15.121:0/3635914633 shutdown_connections 2024-09-17T18:33:29.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.636+0000 7f61847cd640 1 -- 172.21.15.121:0/3635914633 wait complete. 2024-09-17T18:33:29.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.637+0000 7f61847cd640 10 monclient: build_initial_monmap 2024-09-17T18:33:29.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.637+0000 7f61847cd640 10 monclient: monmap: 2024-09-17T18:33:29.649 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:33:29.649 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:33:29.649 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:33:29.649 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:33:29.649 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:33:29.649 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:29.649 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:33:29.649 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:33:29.649 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:33:29.649 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:29.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.637+0000 7f61847cd640 1 librados: starting msgr at 2024-09-17T18:33:29.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.637+0000 7f61847cd640 1 librados: starting objecter 2024-09-17T18:33:29.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.637+0000 7f61847cd640 1 Processor -- start 2024-09-17T18:33:29.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.637+0000 7f61847cd640 1 -- start start 2024-09-17T18:33:29.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.637+0000 7f61847cd640 1 librados: setting wanted keys 2024-09-17T18:33:29.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.637+0000 7f61847cd640 1 librados: calling monclient init 2024-09-17T18:33:29.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.637+0000 7f61847cd640 10 monclient: init 2024-09-17T18:33:29.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f61847cd640 10 monclient: _reopen_session rank -1 2024-09-17T18:33:29.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f61847cd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T18:33:29.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f61847cd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f617c133360 0x7f617c1cb320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:29.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f61847cd640 10 monclient(hunting): picked mon.noname-c con 0x7f617c133360 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:33:29.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f61847cd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f617c136800 0x7f617c12f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:29.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f61847cd640 10 monclient(hunting): picked mon.noname-b con 0x7f617c136800 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:33:29.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f61847cd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f617c136be0 0x7f617c12fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:29.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f61847cd640 10 monclient(hunting): picked mon.noname-a con 0x7f617c136be0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:33:29.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f61847cd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:29.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f61847cd640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f617c09dce0 con 0x7f617c136be0 2024-09-17T18:33:29.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f61847cd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:29.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f61847cd640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f617c137050 con 0x7f617c136800 2024-09-17T18:33:29.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f61847cd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:29.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f61847cd640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f617c09de60 con 0x7f617c133360 2024-09-17T18:33:29.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f61847cd640 10 monclient(hunting): _renew_subs 2024-09-17T18:33:29.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f61847cd640 10 monclient(hunting): authenticate will time out at 1613.354109s 2024-09-17T18:33:29.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f6182569640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f617c133360 0x7f617c1cb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:29.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f6182d6a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f617c136be0 0x7f617c12fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:29.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f6182569640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f617c133360 0x7f617c1cb320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:46770/0 (socket says 172.21.15.121:46770) 2024-09-17T18:33:29.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f6182569640 1 -- 172.21.15.121:0/875145023 learned_addr learned my addr 172.21.15.121:0/875145023 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:33:29.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f6182569640 10 monclient(hunting): get_auth_request con 0x7f617c133360 auth_method 0 2024-09-17T18:33:29.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f6182569640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:29.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f6182569640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:29.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f6182569640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:29.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f6182d6a640 10 monclient(hunting): get_auth_request con 0x7f617c136be0 auth_method 0 2024-09-17T18:33:29.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f6182d6a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:29.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f6182d6a640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:29.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f6182d6a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:29.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f6182569640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:29.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f6182569640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:29.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f6182569640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:29.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f6182d6a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:29.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f6182d6a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:29.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f6182d6a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:29.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f6181d68640 1 --2- 172.21.15.121:0/875145023 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f617c136800 0x7f617c12f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:29.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f6181d68640 10 monclient(hunting): get_auth_request con 0x7f617c136800 auth_method 0 2024-09-17T18:33:29.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f6181d68640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:29.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f6181d68640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:29.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f6181d68640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:29.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f6181d68640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:29.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f6181d68640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:29.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f6181d68640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:29.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f6182569640 10 monclient(hunting): handle_auth_done global_id 64382 payload 824 2024-09-17T18:33:29.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f6182569640 1 -- 172.21.15.121:0/875145023 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f617c136800 msgr2=0x7f617c12f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:29.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f6182569640 1 --2- 172.21.15.121:0/875145023 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f617c136800 0x7f617c12f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:29.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f6182569640 1 -- 172.21.15.121:0/875145023 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f617c136be0 msgr2=0x7f617c12fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:29.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f6182569640 1 --2- 172.21.15.121:0/875145023 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f617c136be0 0x7f617c12fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:29.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f6182569640 10 monclient: _finish_hunting 0 2024-09-17T18:33:29.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f6182569640 1 monclient: found mon.noname-c 2024-09-17T18:33:29.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f6182569640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:33:29.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f6182569640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:33:29.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f6182569640 1 -- 172.21.15.121:0/875145023 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f617c0a6850 con 0x7f617c133360 2024-09-17T18:33:29.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.638+0000 7f6182d6a640 1 --2- 172.21.15.121:0/875145023 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f617c136be0 0x7f617c12fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:33:29.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.640+0000 7f6182569640 1 -- 172.21.15.121:0/875145023 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f617c133360 msgr2=0x7f617c1cb320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:29.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.640+0000 7f6182569640 1 --2- 172.21.15.121:0/875145023 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f617c133360 0x7f617c1cb320 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f617800c7f0 tx=0x7f617800cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:29.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.640+0000 7f6172ffd640 10 client.?.objecter ms_handle_connect 0x7f617c133360 2024-09-17T18:33:29.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.640+0000 7f6172ffd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:33:29.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.640+0000 7f6172ffd640 1 -- 172.21.15.121:0/875145023 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6178015070 con 0x7f617c133360 2024-09-17T18:33:29.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.640+0000 7f6172ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:29.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.640+0000 7f6172ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:33:29.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.640+0000 7f6172ffd640 10 monclient: dump: 2024-09-17T18:33:29.654 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:29.654 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:29.655 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.655 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.655 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:29.655 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:29.655 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:29.655 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:29.655 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:29.655 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:29.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.640+0000 7f6172ffd640 10 monclient: _finish_auth 0 2024-09-17T18:33:29.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.640+0000 7f6172ffd640 10 monclient: _check_auth_tickets 2024-09-17T18:33:29.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.640+0000 7f6172ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:33:29.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.640+0000 7f6172ffd640 1 -- 172.21.15.121:0/875145023 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6178007c40 con 0x7f617c133360 2024-09-17T18:33:29.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.640+0000 7f61847cd640 5 monclient: authenticate success, global_id 64382 2024-09-17T18:33:29.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.640+0000 7f61847cd640 10 monclient: _renew_subs 2024-09-17T18:33:29.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.640+0000 7f61847cd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:33:29.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.640+0000 7f61847cd640 1 -- 172.21.15.121:0/875145023 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f617c073710 con 0x7f617c133360 2024-09-17T18:33:29.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.640+0000 7f6172ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:33:29.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.640+0000 7f6172ffd640 1 -- 172.21.15.121:0/875145023 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f617800ce40 con 0x7f617c133360 2024-09-17T18:33:29.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.640+0000 7f6172ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:29.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.640+0000 7f6172ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:33:29.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.640+0000 7f6172ffd640 10 monclient: dump: 2024-09-17T18:33:29.656 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:29.656 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:29.656 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.656 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:29.656 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:29.656 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:29.656 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:29.656 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:29.657 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:29.657 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:29.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.640+0000 7f6180d66640 4 set_mon_vals no callback set 2024-09-17T18:33:29.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.640+0000 7f61847cd640 10 client.64382.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:33:29.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.640+0000 7f61847cd640 10 monclient: _renew_subs 2024-09-17T18:33:29.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.640+0000 7f61847cd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:33:29.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.640+0000 7f61847cd640 1 -- 172.21.15.121:0/875145023 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f617c070450 con 0x7f617c133360 2024-09-17T18:33:29.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.640+0000 7f61847cd640 1 librados: init done 2024-09-17T18:33:29.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.641+0000 7f61847cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:33:29.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.642+0000 7f61847cd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:33:29.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.642+0000 7f61847cd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:33:29.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.642+0000 7f6172ffd640 1 -- 172.21.15.121:0/875145023 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f6178018070 con 0x7f617c133360 2024-09-17T18:33:29.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.642+0000 7f61847cd640 1 -- 172.21.15.121:0/875145023 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f617c1337b0 con 0x7f617c133360 2024-09-17T18:33:29.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.642+0000 7f6172ffd640 1 --2- 172.21.15.121:0/875145023 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6160040170 0x7f6160042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:29.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.642+0000 7f6172ffd640 1 -- 172.21.15.121:0/875145023 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(561..561 src has 247..561) ==== 5925+0+0 (secure 0 0 0) 0x7f61780539e0 con 0x7f617c133360 2024-09-17T18:33:29.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.642+0000 7f6172ffd640 10 client.64382.objecter ms_dispatch 0x7f617c000b90 osd_map(561..561 src has 247..561) 2024-09-17T18:33:29.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.642+0000 7f6172ffd640 3 client.64382.objecter handle_osd_map got epochs [561,561] > 0 2024-09-17T18:33:29.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.642+0000 7f6172ffd640 3 client.64382.objecter handle_osd_map decoding full epoch 561 2024-09-17T18:33:29.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.642+0000 7f6172ffd640 20 client.64382.objecter dump_active .. 0 homeless 2024-09-17T18:33:29.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.642+0000 7f6181d68640 1 --2- 172.21.15.121:0/875145023 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6160040170 0x7f6160042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:29.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.642+0000 7f6181d68640 10 monclient: get_auth_request con 0x7f6160040170 auth_method 0 2024-09-17T18:33:29.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.645+0000 7f6181d68640 1 --2- 172.21.15.121:0/875145023 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6160040170 0x7f6160042630 secure :-1 s=READY pgs=4052 cs=0 l=1 rev1=1 crypto rx=0x7f616c004f60 tx=0x7f616c007e30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:29.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.645+0000 7f6172ffd640 10 client.64382.objecter ms_handle_connect 0x7f6160040170 2024-09-17T18:33:29.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.646+0000 7f6172ffd640 1 -- 172.21.15.121:0/875145023 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6178051f60 con 0x7f617c133360 2024-09-17T18:33:29.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.646+0000 7f6172ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:33:29.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.646+0000 7f6172ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:33:29.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.888+0000 7f61847cd640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-17T18:33:29.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.888+0000 7f61847cd640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-17T18:33:29.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.888+0000 7f61847cd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:33:29.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.888+0000 7f61847cd640 1 -- 172.21.15.121:0/875145023 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f617c09e1c0 con 0x7f617c133360 2024-09-17T18:33:29.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.892+0000 7f6172ffd640 1 -- 172.21.15.121:0/875145023 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f6178011bf0 con 0x7f617c133360 2024-09-17T18:33:29.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.892+0000 7f6172ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-17T18:33:29.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.892+0000 7f6172ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:33:29.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.894+0000 7f61847cd640 10 librados: watch_flush enter 2024-09-17T18:33:29.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.894+0000 7f61847cd640 10 librados: watch_flush exit 2024-09-17T18:33:29.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.894+0000 7f61847cd640 20 client.64382.objecter shutdown clearing up homeless session... 2024-09-17T18:33:29.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.894+0000 7f61847cd640 10 client.64382.objecter successfully canceled tick 2024-09-17T18:33:29.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.894+0000 7f61847cd640 1 -- 172.21.15.121:0/875145023 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6160040170 msgr2=0x7f6160042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:29.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.894+0000 7f61847cd640 1 --2- 172.21.15.121:0/875145023 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6160040170 0x7f6160042630 secure :-1 s=READY pgs=4052 cs=0 l=1 rev1=1 crypto rx=0x7f616c004f60 tx=0x7f616c007e30 comp rx=0 tx=0).stop 2024-09-17T18:33:29.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.894+0000 7f61847cd640 10 monclient: shutdown 2024-09-17T18:33:29.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.894+0000 7f61847cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:33:29.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.894+0000 7f61847cd640 1 -- 172.21.15.121:0/875145023 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f617c133360 msgr2=0x7f617c1cb320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:29.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.894+0000 7f61847cd640 1 --2- 172.21.15.121:0/875145023 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f617c133360 0x7f617c1cb320 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f617800c7f0 tx=0x7f617800cbf0 comp rx=0 tx=0).stop 2024-09-17T18:33:29.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.894+0000 7f61847cd640 1 -- 172.21.15.121:0/875145023 shutdown_connections 2024-09-17T18:33:29.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.894+0000 7f61847cd640 1 --2- 172.21.15.121:0/875145023 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6160040170 0x7f6160042630 unknown :-1 s=CLOSED pgs=4052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:29.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.894+0000 7f61847cd640 1 --2- 172.21.15.121:0/875145023 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f617c136800 0x7f617c12f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:29.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.894+0000 7f61847cd640 1 --2- 172.21.15.121:0/875145023 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f617c136be0 0x7f617c12fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:29.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.894+0000 7f61847cd640 1 --2- 172.21.15.121:0/875145023 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f617c133360 0x7f617c1cb320 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:29.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.894+0000 7f61847cd640 1 -- 172.21.15.121:0/875145023 >> 172.21.15.121:0/875145023 conn(0x7f617c0a3b50 msgr2=0x7f617c0a5990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:33:29.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.894+0000 7f61847cd640 1 -- 172.21.15.121:0/875145023 shutdown_connections 2024-09-17T18:33:29.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.894+0000 7f61847cd640 1 -- 172.21.15.121:0/875145023 wait complete. 2024-09-17T18:33:29.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:29.894+0000 7f61847cd640 1 librados: shutdown 2024-09-17T18:33:29.902 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.900+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:29.902 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.900+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=17,config=2,health=12,kv=12,logm=57} crc {auth=3547624052,config=2586702516,health=4061656232,kv=834464557,logm=531347352}) 2024-09-17T18:33:29.902 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.900+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=18,config=2,health=11,kv=14,logm=55} crc {auth=777880952,config=2586702516,health=3078763577,kv=2790417503,logm=1420488260}) 2024-09-17T18:33:29.902 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.900+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:29.902 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.900+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=17,config=2,health=12,kv=12,logm=57} crc {auth=3547624052,config=2586702516,health=4061656232,kv=834464557,logm=531347352}) 2024-09-17T18:33:29.902 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.900+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=18,config=2,health=12,kv=14,logm=54} crc {auth=777880952,config=2586702516,health=1503098812,kv=2790417503,logm=3886047921}) 2024-09-17T18:33:29.903 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.901+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:29.903 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.901+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1757207405}) 2024-09-17T18:33:29.903 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.901+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=767761866}) 2024-09-17T18:33:29.903 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.901+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:29.903 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.901+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1757207405}) 2024-09-17T18:33:29.903 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.901+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1946565215}) 2024-09-17T18:33:29.904 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.902+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:29.904 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.902+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=715886093}) 2024-09-17T18:33:29.904 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.902+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4193191121}) 2024-09-17T18:33:29.905 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.902+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:29.905 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.902+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=715886093}) 2024-09-17T18:33:29.905 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.902+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1349247376}) 2024-09-17T18:33:29.905 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.904+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:29.906 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.904+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=105043127}) 2024-09-17T18:33:29.906 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.904+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1686707328}) 2024-09-17T18:33:29.906 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.904+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:29.906 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.904+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=105043127}) 2024-09-17T18:33:29.906 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.904+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3836994241}) 2024-09-17T18:33:29.907 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-17T18:33:29.907 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.905+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:29.907 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.905+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2967948298}) 2024-09-17T18:33:29.908 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.905+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1597435453}) 2024-09-17T18:33:29.908 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.905+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:29.908 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.905+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2967948298}) 2024-09-17T18:33:29.908 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.905+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3064389183}) 2024-09-17T18:33:29.908 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.906+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:29.908 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.906+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2050812248}) 2024-09-17T18:33:29.908 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.906+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=167363242}) 2024-09-17T18:33:29.908 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.906+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:29.908 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.906+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2050812248}) 2024-09-17T18:33:29.908 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.906+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3964395434}) 2024-09-17T18:33:29.940 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.939+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:29.941 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.939+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4117056900}) 2024-09-17T18:33:29.941 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.939+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1753659495}) 2024-09-17T18:33:29.941 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.939+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:29.941 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.939+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4117056900}) 2024-09-17T18:33:29.941 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.939+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4124655531}) 2024-09-17T18:33:29.943 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.941+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:29.943 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.941+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3296252839}) 2024-09-17T18:33:29.943 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.941+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=219815541}) 2024-09-17T18:33:29.943 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.941+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:29.943 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.941+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3296252839}) 2024-09-17T18:33:29.943 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.941+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1032098182}) 2024-09-17T18:33:29.948 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.946+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:29.948 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.946+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2069810060}) 2024-09-17T18:33:29.948 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.946+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2077077677}) 2024-09-17T18:33:29.948 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.946+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:29.948 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.946+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2069810060}) 2024-09-17T18:33:29.949 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:29.946+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4034682766}) 2024-09-17T18:33:30.801 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.800+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.802 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.800+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=383237494}) 2024-09-17T18:33:30.802 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.800+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2008724167}) 2024-09-17T18:33:30.802 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.800+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.802 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.800+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=383237494}) 2024-09-17T18:33:30.802 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.800+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=909155822}) 2024-09-17T18:33:30.851 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.850+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.852 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.850+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=841343715}) 2024-09-17T18:33:30.852 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.850+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1532090060}) 2024-09-17T18:33:30.852 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.850+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.852 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.850+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=841343715}) 2024-09-17T18:33:30.852 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.850+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2522851799}) 2024-09-17T18:33:30.853 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.851+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.853 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.851+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=373271414}) 2024-09-17T18:33:30.853 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.851+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2522174717}) 2024-09-17T18:33:30.853 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.851+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.853 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.851+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=373271414}) 2024-09-17T18:33:30.853 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.851+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2982313881}) 2024-09-17T18:33:30.854 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.852+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.855 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.852+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3029818434}) 2024-09-17T18:33:30.855 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.852+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=138506050}) 2024-09-17T18:33:30.855 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.852+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.855 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.852+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3029818434}) 2024-09-17T18:33:30.855 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.852+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3653481593}) 2024-09-17T18:33:30.855 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.853+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.855 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.853+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3343809974}) 2024-09-17T18:33:30.855 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.853+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3695971430}) 2024-09-17T18:33:30.855 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.853+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.855 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.853+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3343809974}) 2024-09-17T18:33:30.855 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.853+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1809092705}) 2024-09-17T18:33:30.857 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.854+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.857 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.854+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1656849243}) 2024-09-17T18:33:30.857 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.854+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3168669061}) 2024-09-17T18:33:30.857 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.854+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.857 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.854+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1656849243}) 2024-09-17T18:33:30.857 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.854+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3951435658}) 2024-09-17T18:33:30.857 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.855+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.857 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.855+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2236580829}) 2024-09-17T18:33:30.857 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.855+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1360169831}) 2024-09-17T18:33:30.857 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.855+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.858 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.855+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2236580829}) 2024-09-17T18:33:30.858 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.855+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4284851305}) 2024-09-17T18:33:30.859 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.856+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.859 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.856+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=368789789}) 2024-09-17T18:33:30.859 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.856+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3816428456}) 2024-09-17T18:33:30.859 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.856+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.859 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.856+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=368789789}) 2024-09-17T18:33:30.859 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.856+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1885063462}) 2024-09-17T18:33:30.859 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.857+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.859 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.857+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=89182551}) 2024-09-17T18:33:30.859 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.857+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3380544100}) 2024-09-17T18:33:30.860 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.857+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.860 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.857+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=89182551}) 2024-09-17T18:33:30.860 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.857+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2988535647}) 2024-09-17T18:33:30.861 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.858+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.861 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.858+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3282549758}) 2024-09-17T18:33:30.861 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.858+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4008199922}) 2024-09-17T18:33:30.861 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.858+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.861 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.858+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3282549758}) 2024-09-17T18:33:30.861 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.858+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1416194589}) 2024-09-17T18:33:30.861 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.859+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.861 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.859+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4258402221}) 2024-09-17T18:33:30.862 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.859+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=264008611}) 2024-09-17T18:33:30.862 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.859+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.862 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.859+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4258402221}) 2024-09-17T18:33:30.862 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.859+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1054673772}) 2024-09-17T18:33:30.863 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.860+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.864 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.860+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3985394680}) 2024-09-17T18:33:30.864 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.860+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1768351946}) 2024-09-17T18:33:30.864 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.860+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.864 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.860+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3985394680}) 2024-09-17T18:33:30.864 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.860+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2057485290}) 2024-09-17T18:33:30.864 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.861+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.864 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.861+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1921158848}) 2024-09-17T18:33:30.864 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.861+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3148633457}) 2024-09-17T18:33:30.864 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.861+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.864 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.861+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1921158848}) 2024-09-17T18:33:30.864 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.861+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2849646837}) 2024-09-17T18:33:30.865 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.862+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.865 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.862+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1611093850}) 2024-09-17T18:33:30.865 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.862+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2441069367}) 2024-09-17T18:33:30.865 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.862+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.865 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.862+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1611093850}) 2024-09-17T18:33:30.865 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.862+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3288649024}) 2024-09-17T18:33:30.867 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.863+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.867 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.863+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1733238234}) 2024-09-17T18:33:30.867 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.863+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=762035501}) 2024-09-17T18:33:30.867 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.863+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.867 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.863+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1733238234}) 2024-09-17T18:33:30.867 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.863+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=926179579}) 2024-09-17T18:33:30.867 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.864+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.867 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.864+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=919504845}) 2024-09-17T18:33:30.867 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.864+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=801880719}) 2024-09-17T18:33:30.867 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.864+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.867 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.864+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=919504845}) 2024-09-17T18:33:30.867 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.864+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1014688259}) 2024-09-17T18:33:30.867 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.865+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.868 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.865+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=144996752}) 2024-09-17T18:33:30.868 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.865+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1618600993}) 2024-09-17T18:33:30.868 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.865+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.868 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.865+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=144996752}) 2024-09-17T18:33:30.868 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.865+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1215460432}) 2024-09-17T18:33:30.870 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.866+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.870 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.866+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=214446128}) 2024-09-17T18:33:30.870 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.866+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2281175740}) 2024-09-17T18:33:30.870 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.866+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.870 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.866+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=214446128}) 2024-09-17T18:33:30.870 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.866+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3789289239}) 2024-09-17T18:33:30.870 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.867+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.870 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.867+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2178954093}) 2024-09-17T18:33:30.870 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.867+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3954705267}) 2024-09-17T18:33:30.871 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.867+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.871 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.867+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2178954093}) 2024-09-17T18:33:30.871 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.867+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2531476033}) 2024-09-17T18:33:30.871 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.868+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.871 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.868+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3367397273}) 2024-09-17T18:33:30.871 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.868+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=731222038}) 2024-09-17T18:33:30.871 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.868+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.871 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.868+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3367397273}) 2024-09-17T18:33:30.871 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.868+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3181408501}) 2024-09-17T18:33:30.872 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.869+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.872 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.869+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1067207824}) 2024-09-17T18:33:30.873 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.869+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1625264929}) 2024-09-17T18:33:30.873 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.869+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.873 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.869+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1067207824}) 2024-09-17T18:33:30.873 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.869+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1875673788}) 2024-09-17T18:33:30.873 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.870+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.873 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.870+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1972406482}) 2024-09-17T18:33:30.873 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.870+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2385898189}) 2024-09-17T18:33:30.873 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.870+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.873 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.870+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1972406482}) 2024-09-17T18:33:30.873 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.870+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2964097194}) 2024-09-17T18:33:30.892 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.890+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.892 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.890+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1565489392}) 2024-09-17T18:33:30.892 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.890+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4291149019}) 2024-09-17T18:33:30.892 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.890+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.892 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.890+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1565489392}) 2024-09-17T18:33:30.892 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.890+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3698962148}) 2024-09-17T18:33:30.901 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.899+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.901 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.899+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4180553113}) 2024-09-17T18:33:30.901 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.899+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=376731258}) 2024-09-17T18:33:30.902 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.899+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.902 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.899+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4180553113}) 2024-09-17T18:33:30.902 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.899+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=833190246}) 2024-09-17T18:33:30.902 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: require_osd_release = squid 2024-09-17T18:33:30.902 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (8989 ms) 2024-09-17T18:33:30.902 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-09-17T18:33:30.902 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: require_osd_release = squid 2024-09-17T18:33:30.902 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (6004 ms) 2024-09-17T18:33:30.902 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-09-17T18:33:30.902 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: require_osd_release = squid 2024-09-17T18:33:30.903 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (17994 ms) 2024-09-17T18:33:30.903 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-09-17T18:33:30.903 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: require_osd_release = squid 2024-09-17T18:33:30.903 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (41403 ms) 2024-09-17T18:33:30.903 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-09-17T18:33:30.903 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: require_osd_release = squid 2024-09-17T18:33:30.903 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (13498 ms) 2024-09-17T18:33:30.903 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-09-17T18:33:30.903 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (11118 ms) 2024-09-17T18:33:30.903 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-09-17T18:33:30.903 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-09-17T18:33:30.903 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: 00000010 69 73 |is| 2024-09-17T18:33:30.903 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: 00000012 2024-09-17T18:33:30.903 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12007 ms) 2024-09-17T18:33:30.903 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-09-17T18:33:30.904 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: require_osd_release = squid 2024-09-17T18:33:30.904 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (5989 ms) 2024-09-17T18:33:30.904 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (634392 ms total) 2024-09-17T18:33:30.904 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: 2024-09-17T18:33:30.904 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-09-17T18:33:30.904 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-09-17T18:33:30.904 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1028 ms) 2024-09-17T18:33:30.904 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-09-17T18:33:30.904 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (10981 ms) 2024-09-17T18:33:30.904 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-09-17T18:33:30.904 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (38364 ms) 2024-09-17T18:33:30.904 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-09-17T18:33:30.904 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (7 ms) 2024-09-17T18:33:30.904 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (50380 ms total) 2024-09-17T18:33:30.904 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: 2024-09-17T18:33:30.905 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-09-17T18:33:30.905 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-09-17T18:33:30.905 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (6957 ms) 2024-09-17T18:33:30.905 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-09-17T18:33:30.905 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (8428 ms) 2024-09-17T18:33:30.905 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-09-17T18:33:30.905 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: waiting for scrub... 2024-09-17T18:33:30.905 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: done waiting 2024-09-17T18:33:30.905 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (24647 ms) 2024-09-17T18:33:30.905 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-09-17T18:33:30.905 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (9981 ms) 2024-09-17T18:33:30.905 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-09-17T18:33:30.905 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (7953 ms) 2024-09-17T18:33:30.906 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-09-17T18:33:30.906 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (8638 ms) 2024-09-17T18:33:30.906 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-09-17T18:33:30.906 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (9377 ms) 2024-09-17T18:33:30.906 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-09-17T18:33:30.907 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T18:33:30.908 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:33:30.947 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.946+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.947 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.946+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2710096845}) 2024-09-17T18:33:30.948 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.946+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3528336090}) 2024-09-17T18:33:30.948 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.946+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.948 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.946+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2710096845}) 2024-09-17T18:33:30.948 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.946+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3769029087}) 2024-09-17T18:33:30.954 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.952+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.954 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.952+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1853549216}) 2024-09-17T18:33:30.954 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.952+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=257612925}) 2024-09-17T18:33:30.954 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.952+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:30.954 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.952+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1853549216}) 2024-09-17T18:33:30.954 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:30.952+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2798274790}) 2024-09-17T18:33:31.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.006+0000 7f375a3ad640 10 monclient: get_monmap_and_config 2024-09-17T18:33:31.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.006+0000 7f375a3ad640 10 monclient: build_initial_monmap 2024-09-17T18:33:31.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.006+0000 7f375a3ad640 10 monclient: monmap: 2024-09-17T18:33:31.008 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:33:31.008 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:31.008 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:33:31.006881+0000 2024-09-17T18:33:31.008 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:33:31.006881+0000 2024-09-17T18:33:31.008 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:33:31.008 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:31.008 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:33:31.008 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:33:31.009 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:33:31.009 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:33:31.009 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:33:31.009 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:31.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.006+0000 7f375a3ad640 1 Processor -- start 2024-09-17T18:33:31.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.007+0000 7f375a3ad640 1 -- start start 2024-09-17T18:33:31.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.007+0000 7f375a3ad640 10 monclient: init 2024-09-17T18:33:31.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.007+0000 7f375a3ad640 10 monclient: _reopen_session rank -1 2024-09-17T18:33:31.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.007+0000 7f375a3ad640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-17T18:33:31.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.007+0000 7f375a3ad640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f37540a33f0 0x7f375409e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:31.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.007+0000 7f375a3ad640 10 monclient(hunting): picked mon.noname-a con 0x7f37540a33f0 addr v2:172.21.15.121:3300/0 2024-09-17T18:33:31.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.007+0000 7f375a3ad640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f375409da70 addr v1:172.21.15.173:6789/0 2024-09-17T18:33:31.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.007+0000 7f375a3ad640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f375413a850 0x7f375413cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:31.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.007+0000 7f375a3ad640 10 monclient(hunting): picked mon.noname-c con 0x7f375413a850 addr v2:172.21.15.173:3300/0 2024-09-17T18:33:31.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.007+0000 7f375a3ad640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3754074b40 con 0x7f375409da70 2024-09-17T18:33:31.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.007+0000 7f375a3ad640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:31.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.007+0000 7f375a3ad640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f3754074550 con 0x7f37540a33f0 2024-09-17T18:33:31.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.007+0000 7f375a3ad640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:31.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.007+0000 7f375a3ad640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f37540a37d0 con 0x7f375413a850 2024-09-17T18:33:31.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.007+0000 7f375a3ad640 10 monclient(hunting): _renew_subs 2024-09-17T18:33:31.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.007+0000 7f375a3ad640 10 monclient(hunting): authenticate will time out at 1614.723549s 2024-09-17T18:33:31.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.007+0000 7f3753fff640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f37540a33f0 0x7f375409e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:31.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.007+0000 7f3753fff640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f37540a33f0 0x7f375409e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60448/0 (socket says 172.21.15.121:60448) 2024-09-17T18:33:31.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.007+0000 7f3753fff640 1 -- 172.21.15.121:0/838126253 learned_addr learned my addr 172.21.15.121:0/838126253 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:33:31.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.007+0000 7f3753fff640 10 monclient(hunting): get_auth_request con 0x7f37540a33f0 auth_method 0 2024-09-17T18:33:31.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.007+0000 7f3753fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:31.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.007+0000 7f3753fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:31.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.007+0000 7f3753fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:31.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.007+0000 7f3753fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:31.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.007+0000 7f3753fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:31.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.007+0000 7f3753fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:31.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.007+0000 7f375894a640 1 --2- 172.21.15.121:0/838126253 >> v2:172.21.15.173:3300/0 conn(0x7f375413a850 0x7f375413cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:31.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.007+0000 7f375894a640 10 monclient(hunting): get_auth_request con 0x7f375413a850 auth_method 0 2024-09-17T18:33:31.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.007+0000 7f375894a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:31.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.007+0000 7f375894a640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:31.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.007+0000 7f375894a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:31.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.007+0000 7f3753fff640 10 monclient(hunting): handle_auth_done global_id 94131 payload 274 2024-09-17T18:33:31.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.007+0000 7f3753fff640 1 -- 172.21.15.121:0/838126253 >> v2:172.21.15.173:3300/0 conn(0x7f375413a850 msgr2=0x7f375413cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:33:31.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.007+0000 7f3753fff640 1 --2- 172.21.15.121:0/838126253 >> v2:172.21.15.173:3300/0 conn(0x7f375413a850 0x7f375413cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:31.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.007+0000 7f3753fff640 1 -- 172.21.15.121:0/838126253 >> v1:172.21.15.173:6789/0 conn(0x7f375409da70 legacy=0x7f375409ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:33:31.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.007+0000 7f3753fff640 10 monclient: _finish_hunting 0 2024-09-17T18:33:31.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.007+0000 7f3753fff640 1 monclient: found mon.noname-a 2024-09-17T18:33:31.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.007+0000 7f3753fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:33:31.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.007+0000 7f3753fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:33:31.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.007+0000 7f3753fff640 1 -- 172.21.15.121:0/838126253 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37540a13c0 con 0x7f37540a33f0 2024-09-17T18:33:31.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.007+0000 7f375894a640 1 --2- 172.21.15.121:0/838126253 >> v2:172.21.15.173:3300/0 conn(0x7f375413a850 0x7f375413cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T18:33:31.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.010+0000 7f3753fff640 1 -- 172.21.15.121:0/838126253 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37540a33f0 msgr2=0x7f375409e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:31.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.010+0000 7f3753fff640 1 --2- 172.21.15.121:0/838126253 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37540a33f0 0x7f375409e550 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f37540a0190 tx=0x7f374402a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5601823a2214b195 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:31.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.010+0000 7f3752ffd640 1 -- 172.21.15.121:0/838126253 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3744037070 con 0x7f37540a33f0 2024-09-17T18:33:31.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.010+0000 7f3752ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:31.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.010+0000 7f3752ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:33:31.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.010+0000 7f3752ffd640 10 monclient: dump: 2024-09-17T18:33:31.021 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:31.021 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:31.021 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.021 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.021 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:31.021 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:31.021 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:31.021 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:31.021 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:31.021 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:31.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.010+0000 7f3752ffd640 10 monclient: _finish_auth 0 2024-09-17T18:33:31.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.010+0000 7f3752ffd640 10 monclient: _check_auth_tickets 2024-09-17T18:33:31.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.010+0000 7f3752ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:33:31.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.010+0000 7f375a3ad640 5 monclient: authenticate success, global_id 94131 2024-09-17T18:33:31.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.010+0000 7f375a3ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:33:31.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.010+0000 7f3752ffd640 1 -- 172.21.15.121:0/838126253 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3744032640 con 0x7f37540a33f0 2024-09-17T18:33:31.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.010+0000 7f3752ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:33:31.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.010+0000 7f375a3ad640 10 monclient: get_monmap_and_config success 2024-09-17T18:33:31.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.010+0000 7f375a3ad640 4 set_mon_vals no callback set 2024-09-17T18:33:31.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.010+0000 7f3752ffd640 1 -- 172.21.15.121:0/838126253 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3744032a10 con 0x7f37540a33f0 2024-09-17T18:33:31.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.010+0000 7f3752ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:31.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.010+0000 7f3752ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:33:31.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.010+0000 7f3752ffd640 10 monclient: dump: 2024-09-17T18:33:31.022 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:31.022 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:31.022 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.022 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.022 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:31.022 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:31.023 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:31.023 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:31.023 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:31.023 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:31.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.011+0000 7f375a3ad640 10 monclient: shutdown 2024-09-17T18:33:31.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.011+0000 7f375a3ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:33:31.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.011+0000 7f375a3ad640 1 -- 172.21.15.121:0/838126253 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37540a33f0 msgr2=0x7f375409e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:31.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.011+0000 7f375a3ad640 1 --2- 172.21.15.121:0/838126253 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37540a33f0 0x7f375409e550 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f37540a0190 tx=0x7f374402a100 comp rx=0 tx=0).stop 2024-09-17T18:33:31.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.011+0000 7f375a3ad640 1 -- 172.21.15.121:0/838126253 shutdown_connections 2024-09-17T18:33:31.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.011+0000 7f375a3ad640 1 --2- 172.21.15.121:0/838126253 >> v2:172.21.15.173:3300/0 conn(0x7f375413a850 0x7f375413cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:31.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.011+0000 7f375a3ad640 1 --2- 172.21.15.121:0/838126253 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37540a33f0 0x7f375409e550 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:31.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.011+0000 7f375a3ad640 1 -- 172.21.15.121:0/838126253 >> 172.21.15.121:0/838126253 conn(0x7f37540a3b50 msgr2=0x7f37540a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:33:31.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.011+0000 7f375a3ad640 1 -- 172.21.15.121:0/838126253 shutdown_connections 2024-09-17T18:33:31.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.011+0000 7f375a3ad640 1 -- 172.21.15.121:0/838126253 wait complete. 2024-09-17T18:33:31.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.012+0000 7f375a3ad640 10 monclient: build_initial_monmap 2024-09-17T18:33:31.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.012+0000 7f375a3ad640 10 monclient: monmap: 2024-09-17T18:33:31.024 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:33:31.024 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:33:31.024 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:33:31.024 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:33:31.024 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:33:31.024 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:31.024 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:33:31.024 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:33:31.024 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:33:31.024 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:31.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.012+0000 7f375a3ad640 1 librados: starting msgr at 2024-09-17T18:33:31.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.012+0000 7f375a3ad640 1 librados: starting objecter 2024-09-17T18:33:31.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.012+0000 7f375a3ad640 1 Processor -- start 2024-09-17T18:33:31.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.012+0000 7f375a3ad640 1 -- start start 2024-09-17T18:33:31.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.012+0000 7f375a3ad640 1 librados: setting wanted keys 2024-09-17T18:33:31.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.012+0000 7f375a3ad640 1 librados: calling monclient init 2024-09-17T18:33:31.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.012+0000 7f375a3ad640 10 monclient: init 2024-09-17T18:33:31.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f375a3ad640 10 monclient: _reopen_session rank -1 2024-09-17T18:33:31.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f375a3ad640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T18:33:31.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f375a3ad640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37540a33f0 0x7f3754133020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:31.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f375a3ad640 10 monclient(hunting): picked mon.noname-c con 0x7f37540a33f0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:33:31.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f375a3ad640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f375413a850 0x7f3754133560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:31.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f375a3ad640 10 monclient(hunting): picked mon.noname-a con 0x7f375413a850 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:33:31.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f375a3ad640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3754072a20 0x7f3754072eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:31.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f375a3ad640 10 monclient(hunting): picked mon.noname-b con 0x7f3754072a20 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:33:31.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f375a3ad640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:31.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f375a3ad640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f375409f320 con 0x7f375413a850 2024-09-17T18:33:31.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f375a3ad640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:31.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f375a3ad640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f375409f1a0 con 0x7f3754072a20 2024-09-17T18:33:31.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f375a3ad640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:31.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f375a3ad640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f37540a22e0 con 0x7f37540a33f0 2024-09-17T18:33:31.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f375a3ad640 10 monclient(hunting): _renew_subs 2024-09-17T18:33:31.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f375a3ad640 10 monclient(hunting): authenticate will time out at 1614.729126s 2024-09-17T18:33:31.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f3753fff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37540a33f0 0x7f3754133020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:31.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f37537fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f375413a850 0x7f3754133560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:31.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f3753fff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37540a33f0 0x7f3754133020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:46772/0 (socket says 172.21.15.121:46772) 2024-09-17T18:33:31.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f37537fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f375413a850 0x7f3754133560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60450/0 (socket says 172.21.15.121:60450) 2024-09-17T18:33:31.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f3753fff640 1 -- 172.21.15.121:0/1894354843 learned_addr learned my addr 172.21.15.121:0/1894354843 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:33:31.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f3753fff640 10 monclient(hunting): get_auth_request con 0x7f37540a33f0 auth_method 0 2024-09-17T18:33:31.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f3753fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:31.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f3753fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:31.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f3753fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:31.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f37537fe640 10 monclient(hunting): get_auth_request con 0x7f375413a850 auth_method 0 2024-09-17T18:33:31.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f37537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:31.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f37537fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:31.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f37537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:31.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f37537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:31.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f37537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:31.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f37537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:31.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f3753fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:31.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f3753fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:31.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f3753fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:31.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f375894a640 1 --2- 172.21.15.121:0/1894354843 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3754072a20 0x7f3754072eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:31.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f375894a640 10 monclient(hunting): get_auth_request con 0x7f3754072a20 auth_method 0 2024-09-17T18:33:31.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f375894a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:31.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f375894a640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:31.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f375894a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:31.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f37537fe640 10 monclient(hunting): handle_auth_done global_id 94134 payload 824 2024-09-17T18:33:31.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f37537fe640 1 -- 172.21.15.121:0/1894354843 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37540a33f0 msgr2=0x7f3754133020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:31.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f37537fe640 1 --2- 172.21.15.121:0/1894354843 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37540a33f0 0x7f3754133020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:31.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f37537fe640 1 -- 172.21.15.121:0/1894354843 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3754072a20 msgr2=0x7f3754072eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:31.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f37537fe640 1 --2- 172.21.15.121:0/1894354843 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3754072a20 0x7f3754072eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:31.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f37537fe640 10 monclient: _finish_hunting 0 2024-09-17T18:33:31.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f37537fe640 1 monclient: found mon.noname-a 2024-09-17T18:33:31.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f37537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:33:31.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f37537fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:33:31.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.013+0000 7f37537fe640 1 -- 172.21.15.121:0/1894354843 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3754123ee0 con 0x7f375413a850 2024-09-17T18:33:31.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.015+0000 7f37537fe640 1 -- 172.21.15.121:0/1894354843 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f375413a850 msgr2=0x7f3754133560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:31.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.015+0000 7f37537fe640 1 --2- 172.21.15.121:0/1894354843 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f375413a850 0x7f3754133560 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f37540a2060 tx=0x7f374800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:31.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.015+0000 7f3750ff9640 10 client.?.objecter ms_handle_connect 0x7f375413a850 2024-09-17T18:33:31.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.015+0000 7f3750ff9640 10 client.?.objecter resend_mon_ops 2024-09-17T18:33:31.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.015+0000 7f3750ff9640 1 -- 172.21.15.121:0/1894354843 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3748015030 con 0x7f375413a850 2024-09-17T18:33:31.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.015+0000 7f3750ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:31.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.015+0000 7f3750ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:33:31.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.015+0000 7f3750ff9640 10 monclient: dump: 2024-09-17T18:33:31.029 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:31.029 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:31.029 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.029 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.029 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:31.030 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:31.030 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:31.030 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:31.030 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:31.030 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:31.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.015+0000 7f3750ff9640 10 monclient: _finish_auth 0 2024-09-17T18:33:31.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.015+0000 7f3750ff9640 10 monclient: _check_auth_tickets 2024-09-17T18:33:31.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.015+0000 7f3750ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:33:31.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.015+0000 7f375a3ad640 5 monclient: authenticate success, global_id 94134 2024-09-17T18:33:31.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.015+0000 7f3750ff9640 1 -- 172.21.15.121:0/1894354843 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3748004020 con 0x7f375413a850 2024-09-17T18:33:31.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.015+0000 7f375a3ad640 10 monclient: _renew_subs 2024-09-17T18:33:31.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.015+0000 7f375a3ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:33:31.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.015+0000 7f375a3ad640 1 -- 172.21.15.121:0/1894354843 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37541347d0 con 0x7f375413a850 2024-09-17T18:33:31.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.015+0000 7f3750ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T18:33:31.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.015+0000 7f3750ff9640 1 -- 172.21.15.121:0/1894354843 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3748004450 con 0x7f375413a850 2024-09-17T18:33:31.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.015+0000 7f3750ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:31.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.015+0000 7f3750ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:33:31.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.015+0000 7f3750ff9640 10 monclient: dump: 2024-09-17T18:33:31.031 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:31.031 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:31.031 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.031 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.031 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:31.031 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:31.031 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:31.031 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:31.031 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:31.031 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:31.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.015+0000 7f375a3ad640 10 client.94134.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:33:31.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.015+0000 7f375a3ad640 10 monclient: _renew_subs 2024-09-17T18:33:31.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.015+0000 7f375a3ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:33:31.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.015+0000 7f375a3ad640 1 -- 172.21.15.121:0/1894354843 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3754123920 con 0x7f375413a850 2024-09-17T18:33:31.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.015+0000 7f37527fc640 4 set_mon_vals no callback set 2024-09-17T18:33:31.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.015+0000 7f375a3ad640 1 librados: init done 2024-09-17T18:33:31.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.016+0000 7f375a3ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:33:31.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.016+0000 7f375a3ad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:33:31.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.016+0000 7f375a3ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:33:31.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.016+0000 7f3750ff9640 1 -- 172.21.15.121:0/1894354843 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f3748018070 con 0x7f375413a850 2024-09-17T18:33:31.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.016+0000 7f375a3ad640 1 -- 172.21.15.121:0/1894354843 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f375409e550 con 0x7f375413a850 2024-09-17T18:33:31.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.017+0000 7f3750ff9640 1 --2- 172.21.15.121:0/1894354843 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3740040170 0x7f3740042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:31.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.017+0000 7f3750ff9640 1 -- 172.21.15.121:0/1894354843 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(563..563 src has 247..563) ==== 5516+0+0 (secure 0 0 0) 0x7f3748053230 con 0x7f375413a850 2024-09-17T18:33:31.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.017+0000 7f3750ff9640 10 client.94134.objecter ms_dispatch 0x7f3754000b90 osd_map(563..563 src has 247..563) 2024-09-17T18:33:31.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.017+0000 7f3750ff9640 3 client.94134.objecter handle_osd_map got epochs [563,563] > 0 2024-09-17T18:33:31.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.017+0000 7f3750ff9640 3 client.94134.objecter handle_osd_map decoding full epoch 563 2024-09-17T18:33:31.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.017+0000 7f3750ff9640 20 client.94134.objecter dump_active .. 0 homeless 2024-09-17T18:33:31.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.017+0000 7f3753fff640 1 --2- 172.21.15.121:0/1894354843 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3740040170 0x7f3740042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:31.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.017+0000 7f3753fff640 10 monclient: get_auth_request con 0x7f3740040170 auth_method 0 2024-09-17T18:33:31.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.019+0000 7f3750ff9640 10 client.94134.objecter ms_handle_connect 0x7f3740040170 2024-09-17T18:33:31.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.019+0000 7f3753fff640 1 --2- 172.21.15.121:0/1894354843 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3740040170 0x7f3740042630 secure :-1 s=READY pgs=4053 cs=0 l=1 rev1=1 crypto rx=0x7f3744000c00 tx=0x7f374402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:31.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.021+0000 7f3750ff9640 1 -- 172.21.15.121:0/1894354843 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3748051950 con 0x7f375413a850 2024-09-17T18:33:31.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.021+0000 7f3750ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:33:31.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.021+0000 7f3750ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:33:31.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.263+0000 7f375a3ad640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:33:31.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.263+0000 7f375a3ad640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:33:31.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.263+0000 7f375a3ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:33:31.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.263+0000 7f375a3ad640 1 -- 172.21.15.121:0/1894354843 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3754123310 con 0x7f375413a850 2024-09-17T18:33:31.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.263+0000 7f3750ff9640 1 -- 172.21.15.121:0/1894354843 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f3748003db0 con 0x7f375413a850 2024-09-17T18:33:31.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.263+0000 7f3750ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:33:31.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.263+0000 7f3750ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:33:31.265 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":54,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:33:31.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.265+0000 7f375a3ad640 10 librados: watch_flush enter 2024-09-17T18:33:31.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.265+0000 7f375a3ad640 10 librados: watch_flush exit 2024-09-17T18:33:31.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.265+0000 7f375a3ad640 20 client.94134.objecter shutdown clearing up homeless session... 2024-09-17T18:33:31.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.265+0000 7f375a3ad640 10 client.94134.objecter successfully canceled tick 2024-09-17T18:33:31.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.265+0000 7f375a3ad640 1 -- 172.21.15.121:0/1894354843 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3740040170 msgr2=0x7f3740042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:31.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.265+0000 7f375a3ad640 1 --2- 172.21.15.121:0/1894354843 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3740040170 0x7f3740042630 secure :-1 s=READY pgs=4053 cs=0 l=1 rev1=1 crypto rx=0x7f3744000c00 tx=0x7f374402e000 comp rx=0 tx=0).stop 2024-09-17T18:33:31.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.265+0000 7f375a3ad640 10 monclient: shutdown 2024-09-17T18:33:31.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.265+0000 7f375a3ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:33:31.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.265+0000 7f375a3ad640 1 -- 172.21.15.121:0/1894354843 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f375413a850 msgr2=0x7f3754133560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:31.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.265+0000 7f375a3ad640 1 --2- 172.21.15.121:0/1894354843 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f375413a850 0x7f3754133560 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f37540a2060 tx=0x7f374800b870 comp rx=0 tx=0).stop 2024-09-17T18:33:31.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.265+0000 7f375a3ad640 1 -- 172.21.15.121:0/1894354843 shutdown_connections 2024-09-17T18:33:31.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.265+0000 7f375a3ad640 1 --2- 172.21.15.121:0/1894354843 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3740040170 0x7f3740042630 unknown :-1 s=CLOSED pgs=4053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:31.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.265+0000 7f375a3ad640 1 --2- 172.21.15.121:0/1894354843 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3754072a20 0x7f3754072eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:31.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.265+0000 7f375a3ad640 1 --2- 172.21.15.121:0/1894354843 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f375413a850 0x7f3754133560 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:31.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.265+0000 7f375a3ad640 1 --2- 172.21.15.121:0/1894354843 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37540a33f0 0x7f3754133020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:31.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.265+0000 7f375a3ad640 1 -- 172.21.15.121:0/1894354843 >> 172.21.15.121:0/1894354843 conn(0x7f37540a3b50 msgr2=0x7f3754134a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:33:31.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.265+0000 7f375a3ad640 1 -- 172.21.15.121:0/1894354843 shutdown_connections 2024-09-17T18:33:31.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.265+0000 7f375a3ad640 1 -- 172.21.15.121:0/1894354843 wait complete. 2024-09-17T18:33:31.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.265+0000 7f375a3ad640 1 librados: shutdown 2024-09-17T18:33:31.277 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T18:33:31.277 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-17T18:33:31.277 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T18:33:31.278 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T18:33:31.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.414+0000 7f1f74873640 10 monclient: get_monmap_and_config 2024-09-17T18:33:31.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.414+0000 7f1f74873640 10 monclient: build_initial_monmap 2024-09-17T18:33:31.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.414+0000 7f1f74873640 10 monclient: monmap: 2024-09-17T18:33:31.416 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:33:31.416 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:31.416 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:33:31.415163+0000 2024-09-17T18:33:31.416 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:33:31.415163+0000 2024-09-17T18:33:31.417 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:33:31.417 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:31.417 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:33:31.417 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:33:31.417 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:33:31.417 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:33:31.417 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:33:31.417 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:31.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.415+0000 7f1f74873640 1 Processor -- start 2024-09-17T18:33:31.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.415+0000 7f1f74873640 1 -- start start 2024-09-17T18:33:31.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.415+0000 7f1f74873640 10 monclient: init 2024-09-17T18:33:31.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.415+0000 7f1f74873640 10 monclient: _reopen_session rank -1 2024-09-17T18:33:31.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.415+0000 7f1f74873640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-17T18:33:31.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.415+0000 7f1f74873640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1f7009e550 addr v1:172.21.15.173:6789/0 2024-09-17T18:33:31.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.415+0000 7f1f74873640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f1f7009f040 0x7f1f7013a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:31.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.415+0000 7f1f74873640 10 monclient(hunting): picked mon.noname-c con 0x7f1f7009f040 addr v2:172.21.15.173:3300/0 2024-09-17T18:33:31.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.415+0000 7f1f74873640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1f7013ad90 addr v1:172.21.15.121:6789/0 2024-09-17T18:33:31.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.415+0000 7f1f74873640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f700a4460 con 0x7f1f7013ad90 2024-09-17T18:33:31.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.415+0000 7f1f74873640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f700a2b60 con 0x7f1f7009e550 2024-09-17T18:33:31.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.415+0000 7f1f74873640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:31.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.415+0000 7f1f74873640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f1f700a4950 con 0x7f1f7009f040 2024-09-17T18:33:31.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.415+0000 7f1f74873640 10 monclient(hunting): _renew_subs 2024-09-17T18:33:31.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.415+0000 7f1f74873640 10 monclient(hunting): authenticate will time out at 1615.131684s 2024-09-17T18:33:31.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.415+0000 7f1f6ed9d640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f1f7013ad90 0x7f1f7013d180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:57630/0 (socket says 172.21.15.121:57630) 2024-09-17T18:33:31.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.415+0000 7f1f6ed9d640 1 -- 172.21.15.121:0/117284311 learned_addr learned my addr 172.21.15.121:0/117284311 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:33:31.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.416+0000 7f1f65d9b640 1 --2- 172.21.15.121:0/117284311 >> v2:172.21.15.173:3300/0 conn(0x7f1f7009f040 0x7f1f7013a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:31.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.416+0000 7f1f65d9b640 10 monclient(hunting): get_auth_request con 0x7f1f7009f040 auth_method 0 2024-09-17T18:33:31.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.416+0000 7f1f65d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:31.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.416+0000 7f1f65d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:31.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.416+0000 7f1f65d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:31.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.416+0000 7f1f65d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:31.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.416+0000 7f1f65d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:31.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.416+0000 7f1f65d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:31.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.416+0000 7f1f65d9b640 10 monclient(hunting): handle_auth_done global_id 84187 payload 274 2024-09-17T18:33:31.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.416+0000 7f1f65d9b640 1 -- 172.21.15.121:0/117284311 >> v1:172.21.15.173:6789/0 conn(0x7f1f7009e550 legacy=0x7f1f7009e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:33:31.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.416+0000 7f1f65d9b640 1 -- 172.21.15.121:0/117284311 >> v1:172.21.15.121:6789/0 conn(0x7f1f7013ad90 legacy=0x7f1f7013d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:33:31.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.416+0000 7f1f65d9b640 10 monclient: _finish_hunting 0 2024-09-17T18:33:31.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.416+0000 7f1f65d9b640 1 monclient: found mon.noname-c 2024-09-17T18:33:31.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.416+0000 7f1f65d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:33:31.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.416+0000 7f1f65d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.173:3300/0 2024-09-17T18:33:31.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.416+0000 7f1f65d9b640 1 -- 172.21.15.121:0/117284311 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f7009f840 con 0x7f1f7009f040 2024-09-17T18:33:31.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.419+0000 7f1f65d9b640 1 -- 172.21.15.121:0/117284311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f7009f040 msgr2=0x7f1f7013a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:31.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.419+0000 7f1f65d9b640 1 --2- 172.21.15.121:0/117284311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f7009f040 0x7f1f7013a850 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1f7009e0a0 tx=0x7f1f58029fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46513075df457130 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:31.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.419+0000 7f1f6dd9b640 1 -- 172.21.15.121:0/117284311 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f58037070 con 0x7f1f7009f040 2024-09-17T18:33:31.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.419+0000 7f1f6dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:31.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.419+0000 7f1f6dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:33:31.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.419+0000 7f1f6dd9b640 10 monclient: dump: 2024-09-17T18:33:31.429 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:31.429 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:31.429 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.429 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.429 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:31.429 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:31.429 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:31.430 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:31.430 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:31.430 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:31.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.419+0000 7f1f6dd9b640 10 monclient: _finish_auth 0 2024-09-17T18:33:31.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.419+0000 7f1f6dd9b640 10 monclient: _check_auth_tickets 2024-09-17T18:33:31.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.419+0000 7f1f6dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:33:31.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.419+0000 7f1f6dd9b640 1 -- 172.21.15.121:0/117284311 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f58032440 con 0x7f1f7009f040 2024-09-17T18:33:31.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.419+0000 7f1f74873640 5 monclient: authenticate success, global_id 84187 2024-09-17T18:33:31.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.419+0000 7f1f74873640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:33:31.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.419+0000 7f1f6dd9b640 10 monclient: handle_config config(0 keys) 2024-09-17T18:33:31.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.419+0000 7f1f74873640 10 monclient: get_monmap_and_config success 2024-09-17T18:33:31.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.419+0000 7f1f6dd9b640 1 -- 172.21.15.121:0/117284311 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f58032810 con 0x7f1f7009f040 2024-09-17T18:33:31.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.419+0000 7f1f6dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:31.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.419+0000 7f1f74873640 4 set_mon_vals no callback set 2024-09-17T18:33:31.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.419+0000 7f1f6dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:33:31.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.419+0000 7f1f6dd9b640 10 monclient: dump: 2024-09-17T18:33:31.431 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:31.431 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:31.431 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.431 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.431 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:31.431 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:31.431 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:31.431 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:31.431 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:31.432 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:31.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.420+0000 7f1f74873640 10 monclient: shutdown 2024-09-17T18:33:31.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.420+0000 7f1f74873640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:33:31.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.420+0000 7f1f74873640 1 -- 172.21.15.121:0/117284311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f7009f040 msgr2=0x7f1f7013a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:31.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.420+0000 7f1f74873640 1 --2- 172.21.15.121:0/117284311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f7009f040 0x7f1f7013a850 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1f7009e0a0 tx=0x7f1f58029fb0 comp rx=0 tx=0).stop 2024-09-17T18:33:31.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.420+0000 7f1f74873640 1 -- 172.21.15.121:0/117284311 shutdown_connections 2024-09-17T18:33:31.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.420+0000 7f1f74873640 1 --2- 172.21.15.121:0/117284311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f7009f040 0x7f1f7013a850 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:31.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.420+0000 7f1f74873640 1 -- 172.21.15.121:0/117284311 >> 172.21.15.121:0/117284311 conn(0x7f1f700a7830 msgr2=0x7f1f700a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:33:31.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.420+0000 7f1f74873640 1 -- 172.21.15.121:0/117284311 shutdown_connections 2024-09-17T18:33:31.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.420+0000 7f1f74873640 1 -- 172.21.15.121:0/117284311 wait complete. 2024-09-17T18:33:31.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.420+0000 7f1f74873640 10 monclient: build_initial_monmap 2024-09-17T18:33:31.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.420+0000 7f1f74873640 10 monclient: monmap: 2024-09-17T18:33:31.432 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:33:31.432 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:33:31.432 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:33:31.433 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:33:31.433 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:33:31.433 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:31.433 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:33:31.433 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:33:31.433 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:33:31.433 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:31.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.421+0000 7f1f74873640 1 librados: starting msgr at 2024-09-17T18:33:31.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.421+0000 7f1f74873640 1 librados: starting objecter 2024-09-17T18:33:31.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.421+0000 7f1f74873640 1 Processor -- start 2024-09-17T18:33:31.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.421+0000 7f1f74873640 1 -- start start 2024-09-17T18:33:31.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.421+0000 7f1f74873640 1 librados: setting wanted keys 2024-09-17T18:33:31.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.421+0000 7f1f74873640 1 librados: calling monclient init 2024-09-17T18:33:31.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.421+0000 7f1f74873640 10 monclient: init 2024-09-17T18:33:31.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.421+0000 7f1f74873640 10 monclient: _reopen_session rank -1 2024-09-17T18:33:31.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.421+0000 7f1f74873640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T18:33:31.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.421+0000 7f1f74873640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f7009e550 0x7f1f700a1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:31.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.421+0000 7f1f74873640 10 monclient(hunting): picked mon.noname-b con 0x7f1f7009e550 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:33:31.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.421+0000 7f1f74873640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f7009f040 0x7f1f70137760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:31.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.421+0000 7f1f74873640 10 monclient(hunting): picked mon.noname-c con 0x7f1f7009f040 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:33:31.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.421+0000 7f1f74873640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f7013ad90 0x7f1f70137ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:31.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.421+0000 7f1f74873640 10 monclient(hunting): picked mon.noname-a con 0x7f1f7013ad90 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:33:31.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.421+0000 7f1f74873640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:31.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.421+0000 7f1f74873640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7009f490 con 0x7f1f7013ad90 2024-09-17T18:33:31.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.421+0000 7f1f74873640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:31.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.421+0000 7f1f74873640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7009dce0 con 0x7f1f7009e550 2024-09-17T18:33:31.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.421+0000 7f1f74873640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:31.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.421+0000 7f1f74873640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1f7006f2e0 con 0x7f1f7009f040 2024-09-17T18:33:31.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.421+0000 7f1f74873640 10 monclient(hunting): _renew_subs 2024-09-17T18:33:31.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.421+0000 7f1f74873640 10 monclient(hunting): authenticate will time out at 1615.137847s 2024-09-17T18:33:31.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.421+0000 7f1f6ed9d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f7013ad90 0x7f1f70137ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:31.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.421+0000 7f1f65d9b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f7009f040 0x7f1f70137760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:31.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.421+0000 7f1f6ed9d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f7013ad90 0x7f1f70137ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60462/0 (socket says 172.21.15.121:60462) 2024-09-17T18:33:31.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.421+0000 7f1f6ed9d640 1 -- 172.21.15.121:0/1768562025 learned_addr learned my addr 172.21.15.121:0/1768562025 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:33:31.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.421+0000 7f1f6ed9d640 10 monclient(hunting): get_auth_request con 0x7f1f7013ad90 auth_method 0 2024-09-17T18:33:31.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.421+0000 7f1f6ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:31.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.421+0000 7f1f6ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:31.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.421+0000 7f1f6ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:31.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.421+0000 7f1f65d9b640 10 monclient(hunting): get_auth_request con 0x7f1f7009f040 auth_method 0 2024-09-17T18:33:31.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.421+0000 7f1f65d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:31.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.422+0000 7f1f65d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:31.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.422+0000 7f1f65d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:31.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.422+0000 7f1f6ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:31.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.422+0000 7f1f6ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:31.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.422+0000 7f1f6ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:31.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.422+0000 7f1f65d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:31.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.422+0000 7f1f65d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:31.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.422+0000 7f1f65d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:31.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.422+0000 7f1f6e59c640 1 --2- 172.21.15.121:0/1768562025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f7009e550 0x7f1f700a1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:31.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.422+0000 7f1f6e59c640 10 monclient(hunting): get_auth_request con 0x7f1f7009e550 auth_method 0 2024-09-17T18:33:31.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.422+0000 7f1f6e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:31.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.422+0000 7f1f6e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:31.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.422+0000 7f1f6e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:31.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.422+0000 7f1f6e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:31.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.422+0000 7f1f6e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:31.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.422+0000 7f1f6e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:31.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.422+0000 7f1f65d9b640 10 monclient(hunting): handle_auth_done global_id 64388 payload 824 2024-09-17T18:33:31.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.422+0000 7f1f65d9b640 1 -- 172.21.15.121:0/1768562025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f7009e550 msgr2=0x7f1f700a1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:31.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.422+0000 7f1f65d9b640 1 --2- 172.21.15.121:0/1768562025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f7009e550 0x7f1f700a1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:31.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.422+0000 7f1f65d9b640 1 -- 172.21.15.121:0/1768562025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f7013ad90 msgr2=0x7f1f70137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:31.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.422+0000 7f1f65d9b640 1 --2- 172.21.15.121:0/1768562025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f7013ad90 0x7f1f70137ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:31.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.422+0000 7f1f65d9b640 10 monclient: _finish_hunting 0 2024-09-17T18:33:31.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.422+0000 7f1f65d9b640 1 monclient: found mon.noname-c 2024-09-17T18:33:31.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.422+0000 7f1f65d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:33:31.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.422+0000 7f1f65d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:33:31.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.422+0000 7f1f65d9b640 1 -- 172.21.15.121:0/1768562025 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f70124880 con 0x7f1f7009f040 2024-09-17T18:33:31.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.422+0000 7f1f6ed9d640 1 --2- 172.21.15.121:0/1768562025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f7013ad90 0x7f1f70137ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:33:31.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.424+0000 7f1f65d9b640 1 -- 172.21.15.121:0/1768562025 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f7009f040 msgr2=0x7f1f70137760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:31.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.424+0000 7f1f65d9b640 1 --2- 172.21.15.121:0/1768562025 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f7009f040 0x7f1f70137760 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1f58034000 tx=0x7f1f580046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:31.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.424+0000 7f1f677fe640 10 client.?.objecter ms_handle_connect 0x7f1f7009f040 2024-09-17T18:33:31.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.424+0000 7f1f677fe640 10 client.?.objecter resend_mon_ops 2024-09-17T18:33:31.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.424+0000 7f1f677fe640 1 -- 172.21.15.121:0/1768562025 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f58042020 con 0x7f1f7009f040 2024-09-17T18:33:31.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.424+0000 7f1f677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:31.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.424+0000 7f1f677fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:33:31.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.424+0000 7f1f677fe640 10 monclient: dump: 2024-09-17T18:33:31.438 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:31.439 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:31.439 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.439 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.439 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:31.439 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:31.439 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:31.439 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:31.439 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:31.439 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:31.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.424+0000 7f1f677fe640 10 monclient: _finish_auth 0 2024-09-17T18:33:31.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.424+0000 7f1f677fe640 10 monclient: _check_auth_tickets 2024-09-17T18:33:31.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.424+0000 7f1f677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:33:31.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.424+0000 7f1f74873640 5 monclient: authenticate success, global_id 64388 2024-09-17T18:33:31.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.424+0000 7f1f677fe640 1 -- 172.21.15.121:0/1768562025 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f5803d8a0 con 0x7f1f7009f040 2024-09-17T18:33:31.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.424+0000 7f1f677fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:33:31.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.424+0000 7f1f677fe640 1 -- 172.21.15.121:0/1768562025 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f58037040 con 0x7f1f7009f040 2024-09-17T18:33:31.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.424+0000 7f1f74873640 10 monclient: _renew_subs 2024-09-17T18:33:31.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.424+0000 7f1f74873640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:33:31.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.424+0000 7f1f74873640 1 -- 172.21.15.121:0/1768562025 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f700a2530 con 0x7f1f7009f040 2024-09-17T18:33:31.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.424+0000 7f1f6d59a640 4 set_mon_vals no callback set 2024-09-17T18:33:31.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.424+0000 7f1f677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:31.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.424+0000 7f1f677fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:33:31.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.424+0000 7f1f677fe640 10 monclient: dump: 2024-09-17T18:33:31.440 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:31.440 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:31.440 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.441 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.441 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:31.441 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:31.441 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:31.441 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:31.441 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:31.441 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:31.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.424+0000 7f1f74873640 10 client.64388.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:33:31.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.424+0000 7f1f74873640 10 monclient: _renew_subs 2024-09-17T18:33:31.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.424+0000 7f1f74873640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:33:31.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.424+0000 7f1f74873640 1 -- 172.21.15.121:0/1768562025 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f70125120 con 0x7f1f7009f040 2024-09-17T18:33:31.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.424+0000 7f1f74873640 1 librados: init done 2024-09-17T18:33:31.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.425+0000 7f1f74873640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:33:31.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.425+0000 7f1f74873640 1 --2- 172.21.15.121:0/1768562025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f701223e0 0x7f1f701227c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:31.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.425+0000 7f1f74873640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:31.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.425+0000 7f1f74873640 1 -- 172.21.15.121:0/1768562025 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7009f610 con 0x7f1f701223e0 2024-09-17T18:33:31.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.425+0000 7f1f6e59c640 1 --2- 172.21.15.121:0/1768562025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f701223e0 0x7f1f701227c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:31.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.425+0000 7f1f677fe640 1 -- 172.21.15.121:0/1768562025 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f1f58045020 con 0x7f1f7009f040 2024-09-17T18:33:31.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.425+0000 7f1f6e59c640 10 monclient: get_auth_request con 0x7f1f701223e0 auth_method 0 2024-09-17T18:33:31.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.425+0000 7f1f6e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:31.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.425+0000 7f1f6e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:31.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.425+0000 7f1f6e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:31.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.425+0000 7f1f6e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:31.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.425+0000 7f1f6e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:31.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.425+0000 7f1f6e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:31.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.425+0000 7f1f6e59c640 10 monclient(hunting): handle_auth_done global_id 94140 payload 824 2024-09-17T18:33:31.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.425+0000 7f1f677fe640 1 --2- 172.21.15.121:0/1768562025 >> v2:172.21.15.173:6812/2423786037 conn(0x7f1f50040170 0x7f1f50042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:31.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.425+0000 7f1f6e59c640 1 -- 172.21.15.121:0/1768562025 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1f70074900 con 0x7f1f701223e0 2024-09-17T18:33:31.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.425+0000 7f1f677fe640 1 -- 172.21.15.121:0/1768562025 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(563..563 src has 247..563) ==== 5516+0+0 (secure 0 0 0) 0x7f1f58073f30 con 0x7f1f7009f040 2024-09-17T18:33:31.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.425+0000 7f1f677fe640 10 client.64388.objecter ms_dispatch 0x7f1f70000b90 osd_map(563..563 src has 247..563) 2024-09-17T18:33:31.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.425+0000 7f1f677fe640 3 client.64388.objecter handle_osd_map got epochs [563,563] > 0 2024-09-17T18:33:31.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.425+0000 7f1f677fe640 3 client.64388.objecter handle_osd_map decoding full epoch 563 2024-09-17T18:33:31.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.426+0000 7f1f677fe640 20 client.64388.objecter dump_active .. 0 homeless 2024-09-17T18:33:31.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.426+0000 7f1f6ed9d640 1 --2- 172.21.15.121:0/1768562025 >> v2:172.21.15.173:6812/2423786037 conn(0x7f1f50040170 0x7f1f50042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:31.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.426+0000 7f1f6ed9d640 10 monclient: get_auth_request con 0x7f1f50040170 auth_method 0 2024-09-17T18:33:31.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.428+0000 7f1f677fe640 10 client.64388.objecter ms_handle_connect 0x7f1f701223e0 2024-09-17T18:33:31.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.428+0000 7f1f677fe640 10 client.64388.objecter resend_mon_ops 2024-09-17T18:33:31.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.428+0000 7f1f6e59c640 1 -- 172.21.15.121:0/1768562025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f701223e0 msgr2=0x7f1f701227c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:31.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.428+0000 7f1f6e59c640 1 --2- 172.21.15.121:0/1768562025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f701223e0 0x7f1f701227c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1f6000f770 tx=0x7f1f6000fc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:31.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.428+0000 7f1f677fe640 1 -- 172.21.15.121:0/1768562025 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f60018030 con 0x7f1f701223e0 2024-09-17T18:33:31.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.428+0000 7f1f677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:31.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.428+0000 7f1f677fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:33:31.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.428+0000 7f1f677fe640 10 monclient: dump: 2024-09-17T18:33:31.446 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:31.446 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:31.446 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.446 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.447 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:31.447 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:31.447 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:31.447 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:31.447 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:31.447 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:31.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.428+0000 7f1f677fe640 10 client.64388.objecter ms_handle_connect 0x7f1f50040170 2024-09-17T18:33:31.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.428+0000 7f1f6ed9d640 1 --2- 172.21.15.121:0/1768562025 >> v2:172.21.15.173:6812/2423786037 conn(0x7f1f50040170 0x7f1f50042630 secure :-1 s=READY pgs=4054 cs=0 l=1 rev1=1 crypto rx=0x7f1f6800d5e0 tx=0x7f1f68009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:31.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.428+0000 7f1f677fe640 1 -- 172.21.15.121:0/1768562025 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1f6000fe40 con 0x7f1f701223e0 2024-09-17T18:33:31.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.428+0000 7f1f677fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:33:31.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.428+0000 7f1f677fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:33:31.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.428+0000 7f1f677fe640 1 -- 172.21.15.121:0/1768562025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f701223e0 msgr2=0x7f1f701227c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:31.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.428+0000 7f1f677fe640 1 --2- 172.21.15.121:0/1768562025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f701223e0 0x7f1f701227c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1f6000f770 tx=0x7f1f6000fc30 comp rx=0 tx=0).stop 2024-09-17T18:33:31.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.428+0000 7f1f677fe640 1 -- 172.21.15.121:0/1768562025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f701223e0 msgr2=0x7f1f701227c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:33:31.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.428+0000 7f1f677fe640 1 --2- 172.21.15.121:0/1768562025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f701223e0 0x7f1f701227c0 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1f6000f770 tx=0x7f1f6000fc30 comp rx=0 tx=0).stop 2024-09-17T18:33:31.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.434+0000 7f1f74873640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:33:31.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.434+0000 7f1f74873640 1 --2- 172.21.15.121:0/1768562025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f70122d00 0x7f1f700717f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:31.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.434+0000 7f1f74873640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:31.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.434+0000 7f1f74873640 1 -- 172.21.15.121:0/1768562025 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1f70123980 con 0x7f1f70122d00 2024-09-17T18:33:31.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.434+0000 7f1f6e59c640 1 --2- 172.21.15.121:0/1768562025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f70122d00 0x7f1f700717f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:31.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.435+0000 7f1f6e59c640 10 monclient: get_auth_request con 0x7f1f70122d00 auth_method 0 2024-09-17T18:33:31.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.435+0000 7f1f6e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:31.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.435+0000 7f1f6e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:31.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.435+0000 7f1f6e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:31.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.435+0000 7f1f6e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:31.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.435+0000 7f1f6e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:31.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.435+0000 7f1f6e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:31.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.435+0000 7f1f6e59c640 10 monclient(hunting): handle_auth_done global_id 94143 payload 824 2024-09-17T18:33:31.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.435+0000 7f1f6e59c640 1 -- 172.21.15.121:0/1768562025 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1f70124c40 con 0x7f1f70122d00 2024-09-17T18:33:31.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.437+0000 7f1f6e59c640 1 -- 172.21.15.121:0/1768562025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f70122d00 msgr2=0x7f1f700717f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:31.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.437+0000 7f1f6e59c640 1 --2- 172.21.15.121:0/1768562025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f70122d00 0x7f1f700717f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1f6000f770 tx=0x7f1f60003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:31.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.437+0000 7f1f677fe640 10 client.64388.objecter ms_handle_connect 0x7f1f70122d00 2024-09-17T18:33:31.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.437+0000 7f1f677fe640 10 client.64388.objecter resend_mon_ops 2024-09-17T18:33:31.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.437+0000 7f1f677fe640 1 -- 172.21.15.121:0/1768562025 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f60007440 con 0x7f1f70122d00 2024-09-17T18:33:31.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.437+0000 7f1f677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:31.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.437+0000 7f1f677fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:33:31.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.437+0000 7f1f677fe640 10 monclient: dump: 2024-09-17T18:33:31.449 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:31.449 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:31.449 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.449 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.450 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:31.450 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:31.450 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:31.450 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:31.450 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:31.450 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:31.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.437+0000 7f1f677fe640 1 -- 172.21.15.121:0/1768562025 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4554 (secure 0 0 0) 0x7f1f60007440 con 0x7f1f70122d00 2024-09-17T18:33:31.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.437+0000 7f1f677fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:33:31.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.437+0000 7f1f677fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:33:31.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.437+0000 7f1f677fe640 1 -- 172.21.15.121:0/1768562025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f70122d00 msgr2=0x7f1f700717f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:31.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.437+0000 7f1f677fe640 1 --2- 172.21.15.121:0/1768562025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f70122d00 0x7f1f700717f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1f6000f770 tx=0x7f1f60003760 comp rx=0 tx=0).stop 2024-09-17T18:33:31.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.437+0000 7f1f677fe640 1 -- 172.21.15.121:0/1768562025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f70122d00 msgr2=0x7f1f700717f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:33:31.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.437+0000 7f1f677fe640 1 --2- 172.21.15.121:0/1768562025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f70122d00 0x7f1f700717f0 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1f6000f770 tx=0x7f1f60003760 comp rx=0 tx=0).stop 2024-09-17T18:33:31.451 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:33:31.451 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:33:31.451 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:33:31.451 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "leader", 2024-09-17T18:33:31.451 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 54, 2024-09-17T18:33:31.451 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 17963, 2024-09-17T18:33:31.451 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:33:31.451 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:33:31.451 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:33:31.451 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:33:31.451 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:33:31.452 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 2, 2024-09-17T18:33:31.452 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:33:31.452 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:33:31.452 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:33:31.452 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:33:31.452 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:33:31.452 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:33:31.452 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:33:31.452 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:33:31.452 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:33:31.452 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:33:31.452 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:33:31.452 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:33:31.453 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:33:31.453 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:33:31.453 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:33:31.453 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:33:31.453 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:33:31.453 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:33:31.453 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:33:31.453 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:33:31.453 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:33:31.453 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:33:31.453 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:33:31.453 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:33:31.453 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:33:31.454 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:33:31.454 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:33:31.454 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:33:31.454 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:31.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:31.454 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:33:31.454 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:33:31.454 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:33:31.454 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:33:31.454 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:33:31.454 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:33:31.454 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:33:31.454 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:33:31.455 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:33:31.455 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:33:31.455 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:33:31.455 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:33:31.455 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:33:31.455 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:33:31.455 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:33:31.455 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:33:31.455 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:33:31.455 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:33:31.455 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:33:31.456 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:33:31.456 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:33:31.456 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:33:31.456 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:33:31.456 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:33:31.456 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:33:31.456 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:33:31.456 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:33:31.456 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:33:31.456 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:33:31.456 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:33:31.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:31.456 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:33:31.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:31.457 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:33:31.457 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:33:31.457 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:33:31.457 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:33:31.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:31.457 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:33:31.457 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:33:31.457 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:31.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:31.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:31.457 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:33:31.457 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:33:31.457 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:31.457 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:31.458 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:31.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:31.458 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:33:31.458 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:33:31.458 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:33:31.458 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:33:31.458 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:33:31.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:31.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:31.458 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:33:31.458 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:33:31.458 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:33:31.458 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:33:31.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:31.459 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:33:31.459 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:33:31.459 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:31.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:31.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:31.459 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:33:31.459 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:33:31.459 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:31.459 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:31.459 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:31.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:31.459 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:33:31.460 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:33:31.460 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:33:31.460 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:33:31.460 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:33:31.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:31.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:31.460 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:33:31.460 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:33:31.460 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:33:31.460 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:33:31.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:31.460 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:33:31.460 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:33:31.460 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:31.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:31.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:31.461 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:33:31.461 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:33:31.461 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:31.461 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:31.461 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:31.461 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:31.461 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:33:31.461 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:33:31.461 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:33:31.461 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:33:31.461 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:33:31.461 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:31.461 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:31.461 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:31.462 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:33:31.462 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:33:31.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:31.462 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:33:31.462 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:33:31.462 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:33:31.462 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:31.462 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:33:31.462 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:33:31.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:31.462 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:33:31.462 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:33:31.462 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:33:31.462 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:31.463 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:31.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:31.463 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:33:31.463 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:33:31.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.437+0000 7f1f74873640 10 librados: watch_flush enter 2024-09-17T18:33:31.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.437+0000 7f1f74873640 10 librados: watch_flush exit 2024-09-17T18:33:31.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.438+0000 7f1f74873640 20 client.64388.objecter shutdown clearing up homeless session... 2024-09-17T18:33:31.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.438+0000 7f1f74873640 10 client.64388.objecter successfully canceled tick 2024-09-17T18:33:31.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.438+0000 7f1f74873640 1 -- 172.21.15.121:0/1768562025 >> v2:172.21.15.173:6812/2423786037 conn(0x7f1f50040170 msgr2=0x7f1f50042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:31.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.438+0000 7f1f74873640 1 --2- 172.21.15.121:0/1768562025 >> v2:172.21.15.173:6812/2423786037 conn(0x7f1f50040170 0x7f1f50042630 secure :-1 s=READY pgs=4054 cs=0 l=1 rev1=1 crypto rx=0x7f1f6800d5e0 tx=0x7f1f68009040 comp rx=0 tx=0).stop 2024-09-17T18:33:31.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.438+0000 7f1f74873640 10 monclient: shutdown 2024-09-17T18:33:31.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.438+0000 7f1f74873640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:33:31.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.438+0000 7f1f74873640 1 -- 172.21.15.121:0/1768562025 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f7009f040 msgr2=0x7f1f70137760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:31.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.438+0000 7f1f74873640 1 --2- 172.21.15.121:0/1768562025 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f7009f040 0x7f1f70137760 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1f58034000 tx=0x7f1f580046c0 comp rx=0 tx=0).stop 2024-09-17T18:33:31.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.438+0000 7f1f6ed9d640 1 -- 172.21.15.121:0/1768562025 reap_dead start 2024-09-17T18:33:31.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.438+0000 7f1f6ed9d640 1 -- 172.21.15.121:0/1768562025 reap_dead start 2024-09-17T18:33:31.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.438+0000 7f1f74873640 1 -- 172.21.15.121:0/1768562025 shutdown_connections 2024-09-17T18:33:31.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.438+0000 7f1f74873640 1 -- 172.21.15.121:0/1768562025 >> 172.21.15.121:0/1768562025 conn(0x7f1f700a7830 msgr2=0x7f1f700a4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:33:31.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.438+0000 7f1f74873640 1 -- 172.21.15.121:0/1768562025 shutdown_connections 2024-09-17T18:33:31.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.438+0000 7f1f74873640 1 -- 172.21.15.121:0/1768562025 wait complete. 2024-09-17T18:33:31.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.438+0000 7f1f74873640 1 librados: shutdown 2024-09-17T18:33:31.466 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:33:31.466 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T18:33:31.466 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T18:33:31.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.562+0000 7fb71cc85640 10 monclient: get_monmap_and_config 2024-09-17T18:33:31.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.562+0000 7fb71cc85640 10 monclient: build_initial_monmap 2024-09-17T18:33:31.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.562+0000 7fb71cc85640 10 monclient: monmap: 2024-09-17T18:33:31.564 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:33:31.564 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:31.564 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:33:31.562931+0000 2024-09-17T18:33:31.564 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:33:31.562931+0000 2024-09-17T18:33:31.564 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:33:31.564 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:31.565 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:33:31.565 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:33:31.565 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:33:31.565 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:33:31.565 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:33:31.565 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:31.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.563+0000 7fb71cc85640 1 Processor -- start 2024-09-17T18:33:31.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.563+0000 7fb71cc85640 1 -- start start 2024-09-17T18:33:31.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.563+0000 7fb71cc85640 10 monclient: init 2024-09-17T18:33:31.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.563+0000 7fb71cc85640 10 monclient: _reopen_session rank -1 2024-09-17T18:33:31.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.563+0000 7fb71cc85640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-17T18:33:31.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.563+0000 7fb71cc85640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb7180a0a00 0x7fb7180a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:31.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.563+0000 7fb71cc85640 10 monclient(hunting): picked mon.noname-a con 0x7fb7180a0a00 addr v2:172.21.15.121:3300/0 2024-09-17T18:33:31.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.563+0000 7fb71cc85640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb7180a1320 addr v1:172.21.15.173:6789/0 2024-09-17T18:33:31.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.563+0000 7fb71cc85640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb718135090 0x7fb71813ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:31.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.563+0000 7fb71cc85640 10 monclient(hunting): picked mon.noname-b con 0x7fb718135090 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:33:31.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.563+0000 7fb71cc85640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7180a3820 con 0x7fb7180a1320 2024-09-17T18:33:31.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.563+0000 7fb71cc85640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:31.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.563+0000 7fb71cc85640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fb7180a2ea0 con 0x7fb7180a0a00 2024-09-17T18:33:31.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.563+0000 7fb71cc85640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:31.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.563+0000 7fb71cc85640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb71809e800 con 0x7fb718135090 2024-09-17T18:33:31.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.563+0000 7fb71cc85640 10 monclient(hunting): _renew_subs 2024-09-17T18:33:31.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.563+0000 7fb71cc85640 10 monclient(hunting): authenticate will time out at 1615.279589s 2024-09-17T18:33:31.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.563+0000 7fb71659c640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb7180a0a00 0x7fb7180a0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:31.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.563+0000 7fb716d9d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb718135090 0x7fb71813ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:31.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.563+0000 7fb71659c640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb7180a0a00 0x7fb7180a0de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60492/0 (socket says 172.21.15.121:60492) 2024-09-17T18:33:31.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.563+0000 7fb71659c640 1 -- 172.21.15.121:0/4180866912 learned_addr learned my addr 172.21.15.121:0/4180866912 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:33:31.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.563+0000 7fb71659c640 10 monclient(hunting): get_auth_request con 0x7fb7180a0a00 auth_method 0 2024-09-17T18:33:31.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.563+0000 7fb71659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:31.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.563+0000 7fb71659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:31.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.563+0000 7fb71659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:31.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.563+0000 7fb716d9d640 10 monclient(hunting): get_auth_request con 0x7fb718135090 auth_method 0 2024-09-17T18:33:31.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.563+0000 7fb716d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:31.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.563+0000 7fb716d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:31.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.563+0000 7fb716d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:31.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.563+0000 7fb71659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:31.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.563+0000 7fb71659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:31.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.563+0000 7fb71659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:31.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.563+0000 7fb716d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:31.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.563+0000 7fb716d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:31.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.563+0000 7fb716d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:31.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.563+0000 7fb716d9d640 10 monclient(hunting): handle_auth_done global_id 64391 payload 274 2024-09-17T18:33:31.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.564+0000 7fb716d9d640 1 -- 172.21.15.121:0/4180866912 >> v2:172.21.15.121:3300/0 conn(0x7fb7180a0a00 msgr2=0x7fb7180a0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:33:31.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.564+0000 7fb716d9d640 1 --2- 172.21.15.121:0/4180866912 >> v2:172.21.15.121:3300/0 conn(0x7fb7180a0a00 0x7fb7180a0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:31.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.564+0000 7fb716d9d640 1 -- 172.21.15.121:0/4180866912 >> v1:172.21.15.173:6789/0 conn(0x7fb7180a1320 legacy=0x7fb718134980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:33:31.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.564+0000 7fb716d9d640 10 monclient: _finish_hunting 0 2024-09-17T18:33:31.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.564+0000 7fb716d9d640 1 monclient: found mon.noname-b 2024-09-17T18:33:31.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.564+0000 7fb716d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:33:31.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.564+0000 7fb716d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:33:31.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.564+0000 7fb716d9d640 1 -- 172.21.15.121:0/4180866912 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7180a2c30 con 0x7fb718135090 2024-09-17T18:33:31.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.564+0000 7fb71659c640 1 --2- 172.21.15.121:0/4180866912 >> v2:172.21.15.121:3300/0 conn(0x7fb7180a0a00 0x7fb7180a0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:33:31.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.566+0000 7fb716d9d640 1 -- 172.21.15.121:0/4180866912 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb718135090 msgr2=0x7fb71813ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:31.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.566+0000 7fb716d9d640 1 --2- 172.21.15.121:0/4180866912 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb718135090 0x7fb71813ccc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb71000aea0 tx=0x7fb71002b5e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea7f258958f9f8eb server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:31.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.566+0000 7fb715d9b640 1 -- 172.21.15.121:0/4180866912 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb710037070 con 0x7fb718135090 2024-09-17T18:33:31.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.566+0000 7fb715d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:31.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.566+0000 7fb715d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:33:31.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.566+0000 7fb715d9b640 10 monclient: dump: 2024-09-17T18:33:31.578 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:31.578 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:31.578 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.578 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.578 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:31.578 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:31.578 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:31.578 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:31.578 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:31.578 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:31.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.566+0000 7fb715d9b640 10 monclient: _finish_auth 0 2024-09-17T18:33:31.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.566+0000 7fb715d9b640 10 monclient: _check_auth_tickets 2024-09-17T18:33:31.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.566+0000 7fb715d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:33:31.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.566+0000 7fb715d9b640 1 -- 172.21.15.121:0/4180866912 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb710008850 con 0x7fb718135090 2024-09-17T18:33:31.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.566+0000 7fb71cc85640 5 monclient: authenticate success, global_id 64391 2024-09-17T18:33:31.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.566+0000 7fb71cc85640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:33:31.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.566+0000 7fb715d9b640 10 monclient: handle_config config(0 keys) 2024-09-17T18:33:31.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.566+0000 7fb71cc85640 10 monclient: get_monmap_and_config success 2024-09-17T18:33:31.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.566+0000 7fb715d9b640 1 -- 172.21.15.121:0/4180866912 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb710008c20 con 0x7fb718135090 2024-09-17T18:33:31.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.566+0000 7fb715d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:31.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.566+0000 7fb71cc85640 4 set_mon_vals no callback set 2024-09-17T18:33:31.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.566+0000 7fb715d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:33:31.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.566+0000 7fb715d9b640 10 monclient: dump: 2024-09-17T18:33:31.579 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:31.579 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:31.579 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.580 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.580 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:31.580 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:31.580 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:31.580 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:31.580 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:31.580 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:31.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.567+0000 7fb71cc85640 10 monclient: shutdown 2024-09-17T18:33:31.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.567+0000 7fb71cc85640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:33:31.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.567+0000 7fb71cc85640 1 -- 172.21.15.121:0/4180866912 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb718135090 msgr2=0x7fb71813ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:31.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.567+0000 7fb71cc85640 1 --2- 172.21.15.121:0/4180866912 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb718135090 0x7fb71813ccc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb71000aea0 tx=0x7fb71002b5e0 comp rx=0 tx=0).stop 2024-09-17T18:33:31.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.567+0000 7fb71cc85640 1 -- 172.21.15.121:0/4180866912 shutdown_connections 2024-09-17T18:33:31.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.567+0000 7fb71cc85640 1 --2- 172.21.15.121:0/4180866912 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb718135090 0x7fb71813ccc0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:31.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.567+0000 7fb71cc85640 1 --2- 172.21.15.121:0/4180866912 >> v2:172.21.15.121:3300/0 conn(0x7fb7180a0a00 0x7fb7180a0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:31.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.567+0000 7fb71cc85640 1 -- 172.21.15.121:0/4180866912 >> 172.21.15.121:0/4180866912 conn(0x7fb7180a7830 msgr2=0x7fb7180a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:33:31.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.567+0000 7fb71cc85640 1 -- 172.21.15.121:0/4180866912 shutdown_connections 2024-09-17T18:33:31.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.567+0000 7fb71cc85640 1 -- 172.21.15.121:0/4180866912 wait complete. 2024-09-17T18:33:31.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.568+0000 7fb71cc85640 10 monclient: build_initial_monmap 2024-09-17T18:33:31.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.568+0000 7fb71cc85640 10 monclient: monmap: 2024-09-17T18:33:31.581 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:33:31.581 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:33:31.581 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:33:31.581 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:33:31.581 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:33:31.581 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:31.581 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:33:31.581 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:33:31.581 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:33:31.581 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:31.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.568+0000 7fb71cc85640 1 librados: starting msgr at 2024-09-17T18:33:31.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.568+0000 7fb71cc85640 1 librados: starting objecter 2024-09-17T18:33:31.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.568+0000 7fb71cc85640 1 Processor -- start 2024-09-17T18:33:31.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.568+0000 7fb71cc85640 1 -- start start 2024-09-17T18:33:31.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.568+0000 7fb71cc85640 1 librados: setting wanted keys 2024-09-17T18:33:31.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.568+0000 7fb71cc85640 1 librados: calling monclient init 2024-09-17T18:33:31.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.568+0000 7fb71cc85640 10 monclient: init 2024-09-17T18:33:31.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb71cc85640 10 monclient: _reopen_session rank -1 2024-09-17T18:33:31.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb71cc85640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:33:31.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb71cc85640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7180a0a00 0x7fb7181c9430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:31.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb71cc85640 10 monclient(hunting): picked mon.noname-a con 0x7fb7180a0a00 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:33:31.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb71cc85640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7180a1320 0x7fb718133ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:31.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb71cc85640 10 monclient(hunting): picked mon.noname-b con 0x7fb7180a1320 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:33:31.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb71cc85640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb718135090 0x7fb7181341e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:31.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb71cc85640 10 monclient(hunting): picked mon.noname-c con 0x7fb718135090 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:33:31.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb71cc85640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:31.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb71cc85640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb7180a3120 con 0x7fb7180a0a00 2024-09-17T18:33:31.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb71cc85640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:31.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb71cc85640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb7180a32a0 con 0x7fb7180a1320 2024-09-17T18:33:31.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb71cc85640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:31.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb71cc85640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb7180a1f10 con 0x7fb718135090 2024-09-17T18:33:31.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb71cc85640 10 monclient(hunting): _renew_subs 2024-09-17T18:33:31.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb71cc85640 10 monclient(hunting): authenticate will time out at 1615.285090s 2024-09-17T18:33:31.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb71659c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7180a0a00 0x7fb7181c9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:31.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb71659c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7180a0a00 0x7fb7181c9430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60496/0 (socket says 172.21.15.121:60496) 2024-09-17T18:33:31.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb71659c640 1 -- 172.21.15.121:0/110533258 learned_addr learned my addr 172.21.15.121:0/110533258 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:33:31.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb71659c640 10 monclient(hunting): get_auth_request con 0x7fb7180a0a00 auth_method 0 2024-09-17T18:33:31.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb71659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:31.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb71659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:31.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb716d9d640 1 --2- 172.21.15.121:0/110533258 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb718135090 0x7fb7181341e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:31.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb71659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:31.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb716d9d640 10 monclient(hunting): get_auth_request con 0x7fb718135090 auth_method 0 2024-09-17T18:33:31.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb716d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:31.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb716d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:31.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb716d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:31.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb716d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:31.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb716d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:31.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb716d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:31.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb71659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:31.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb71659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:31.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb71659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:31.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb70ffff640 1 --2- 172.21.15.121:0/110533258 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7180a1320 0x7fb718133ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:31.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb70ffff640 10 monclient(hunting): get_auth_request con 0x7fb7180a1320 auth_method 0 2024-09-17T18:33:31.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb70ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:31.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb70ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:31.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb70ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:31.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb716d9d640 10 monclient(hunting): handle_auth_done global_id 64394 payload 824 2024-09-17T18:33:31.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb716d9d640 1 -- 172.21.15.121:0/110533258 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7180a1320 msgr2=0x7fb718133ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:31.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb716d9d640 1 --2- 172.21.15.121:0/110533258 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7180a1320 0x7fb718133ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:31.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb716d9d640 1 -- 172.21.15.121:0/110533258 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7180a0a00 msgr2=0x7fb7181c9430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:31.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb716d9d640 1 --2- 172.21.15.121:0/110533258 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7180a0a00 0x7fb7181c9430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:31.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb716d9d640 10 monclient: _finish_hunting 0 2024-09-17T18:33:31.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb716d9d640 1 monclient: found mon.noname-c 2024-09-17T18:33:31.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb716d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:33:31.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb716d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:33:31.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb716d9d640 1 -- 172.21.15.121:0/110533258 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb718074050 con 0x7fb718135090 2024-09-17T18:33:31.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.569+0000 7fb71659c640 1 --2- 172.21.15.121:0/110533258 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7180a0a00 0x7fb7181c9430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:33:31.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.571+0000 7fb716d9d640 1 -- 172.21.15.121:0/110533258 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb718135090 msgr2=0x7fb7181341e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:31.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.571+0000 7fb716d9d640 1 --2- 172.21.15.121:0/110533258 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb718135090 0x7fb7181341e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb710031000 tx=0x7fb710004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:31.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.571+0000 7fb70effd640 10 client.?.objecter ms_handle_connect 0x7fb718135090 2024-09-17T18:33:31.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.571+0000 7fb70effd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:33:31.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.571+0000 7fb70effd640 1 -- 172.21.15.121:0/110533258 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb710042020 con 0x7fb718135090 2024-09-17T18:33:31.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.571+0000 7fb70effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:31.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.571+0000 7fb70effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:33:31.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.571+0000 7fb70effd640 10 monclient: dump: 2024-09-17T18:33:31.586 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:31.586 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:31.586 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.587 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.587 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:31.587 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:31.587 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:31.587 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:31.587 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:31.587 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:31.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.571+0000 7fb70effd640 10 monclient: _finish_auth 0 2024-09-17T18:33:31.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.571+0000 7fb70effd640 10 monclient: _check_auth_tickets 2024-09-17T18:33:31.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.571+0000 7fb70effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:33:31.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.571+0000 7fb71cc85640 5 monclient: authenticate success, global_id 64394 2024-09-17T18:33:31.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.571+0000 7fb70effd640 1 -- 172.21.15.121:0/110533258 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7100328c0 con 0x7fb718135090 2024-09-17T18:33:31.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.571+0000 7fb71cc85640 10 monclient: _renew_subs 2024-09-17T18:33:31.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.571+0000 7fb71cc85640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:33:31.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.571+0000 7fb71cc85640 1 -- 172.21.15.121:0/110533258 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb71811f300 con 0x7fb718135090 2024-09-17T18:33:31.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.571+0000 7fb70effd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:33:31.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.571+0000 7fb70effd640 1 -- 172.21.15.121:0/110533258 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb710037040 con 0x7fb718135090 2024-09-17T18:33:31.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.571+0000 7fb70effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:31.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.571+0000 7fb70effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:33:31.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.571+0000 7fb70effd640 10 monclient: dump: 2024-09-17T18:33:31.588 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:31.588 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:31.588 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.588 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.588 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:31.588 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:31.588 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:31.589 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:31.589 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:31.589 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:31.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.571+0000 7fb71559a640 4 set_mon_vals no callback set 2024-09-17T18:33:31.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.571+0000 7fb71cc85640 10 client.64394.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:33:31.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.571+0000 7fb71cc85640 10 monclient: _renew_subs 2024-09-17T18:33:31.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.571+0000 7fb71cc85640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:33:31.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.571+0000 7fb71cc85640 1 -- 172.21.15.121:0/110533258 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7181c9970 con 0x7fb718135090 2024-09-17T18:33:31.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.571+0000 7fb71cc85640 1 librados: init done 2024-09-17T18:33:31.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.572+0000 7fb70effd640 1 -- 172.21.15.121:0/110533258 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fb710045020 con 0x7fb718135090 2024-09-17T18:33:31.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.572+0000 7fb71cc85640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:33:31.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.572+0000 7fb71cc85640 1 --2- 172.21.15.121:0/110533258 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb71811d940 0x7fb71811dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:31.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.572+0000 7fb71cc85640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:31.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.572+0000 7fb71cc85640 1 -- 172.21.15.121:0/110533258 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb71809ece0 con 0x7fb71811d940 2024-09-17T18:33:31.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.572+0000 7fb71659c640 1 --2- 172.21.15.121:0/110533258 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb71811d940 0x7fb71811dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:31.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.572+0000 7fb71659c640 10 monclient: get_auth_request con 0x7fb71811d940 auth_method 0 2024-09-17T18:33:31.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.572+0000 7fb71659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:31.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.572+0000 7fb71659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:31.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.572+0000 7fb71659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:31.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.572+0000 7fb70effd640 1 --2- 172.21.15.121:0/110533258 >> v2:172.21.15.173:6812/2423786037 conn(0x7fb6f8040170 0x7fb6f8042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:31.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.573+0000 7fb71659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:31.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.573+0000 7fb71659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:31.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.573+0000 7fb70effd640 1 -- 172.21.15.121:0/110533258 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(563..563 src has 247..563) ==== 5516+0+0 (secure 0 0 0) 0x7fb710075520 con 0x7fb718135090 2024-09-17T18:33:31.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.573+0000 7fb70effd640 10 client.64394.objecter ms_dispatch 0x7fb718000b90 osd_map(563..563 src has 247..563) 2024-09-17T18:33:31.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.573+0000 7fb71659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:31.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.573+0000 7fb70effd640 3 client.64394.objecter handle_osd_map got epochs [563,563] > 0 2024-09-17T18:33:31.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.573+0000 7fb70effd640 3 client.64394.objecter handle_osd_map decoding full epoch 563 2024-09-17T18:33:31.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.573+0000 7fb70effd640 20 client.64394.objecter dump_active .. 0 homeless 2024-09-17T18:33:31.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.573+0000 7fb71659c640 10 monclient(hunting): handle_auth_done global_id 64397 payload 824 2024-09-17T18:33:31.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.573+0000 7fb71659c640 1 -- 172.21.15.121:0/110533258 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb71809e290 con 0x7fb71811d940 2024-09-17T18:33:31.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.573+0000 7fb70ffff640 1 --2- 172.21.15.121:0/110533258 >> v2:172.21.15.173:6812/2423786037 conn(0x7fb6f8040170 0x7fb6f8042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:31.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.573+0000 7fb70ffff640 10 monclient: get_auth_request con 0x7fb6f8040170 auth_method 0 2024-09-17T18:33:31.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.575+0000 7fb70effd640 10 client.64394.objecter ms_handle_connect 0x7fb71811d940 2024-09-17T18:33:31.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.575+0000 7fb70effd640 10 client.64394.objecter resend_mon_ops 2024-09-17T18:33:31.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.575+0000 7fb71659c640 1 -- 172.21.15.121:0/110533258 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb71811d940 msgr2=0x7fb71811dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:31.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.575+0000 7fb71659c640 1 --2- 172.21.15.121:0/110533258 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb71811d940 0x7fb71811dd20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb708009f20 tx=0x7fb70800f610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:31.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.575+0000 7fb70effd640 1 -- 172.21.15.121:0/110533258 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb708004030 con 0x7fb71811d940 2024-09-17T18:33:31.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.575+0000 7fb70effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:31.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.575+0000 7fb70effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:33:31.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.575+0000 7fb70effd640 10 monclient: dump: 2024-09-17T18:33:31.591 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:31.591 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:31.591 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.592 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.592 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:31.592 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:31.592 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:31.592 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:31.592 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:31.592 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:31.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.575+0000 7fb70effd640 10 client.64394.objecter ms_handle_connect 0x7fb6f8040170 2024-09-17T18:33:31.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.575+0000 7fb70ffff640 1 --2- 172.21.15.121:0/110533258 >> v2:172.21.15.173:6812/2423786037 conn(0x7fb6f8040170 0x7fb6f8042630 secure :-1 s=READY pgs=4055 cs=0 l=1 rev1=1 crypto rx=0x7fb71811fdf0 tx=0x7fb700009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:31.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.575+0000 7fb70effd640 1 -- 172.21.15.121:0/110533258 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb708004030 con 0x7fb71811d940 2024-09-17T18:33:31.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.575+0000 7fb70effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:33:31.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.575+0000 7fb70effd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:33:31.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.575+0000 7fb70effd640 1 -- 172.21.15.121:0/110533258 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb71811d940 msgr2=0x7fb71811dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:31.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.575+0000 7fb70effd640 1 --2- 172.21.15.121:0/110533258 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb71811d940 0x7fb71811dd20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb708009f20 tx=0x7fb70800f610 comp rx=0 tx=0).stop 2024-09-17T18:33:31.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.575+0000 7fb70effd640 1 -- 172.21.15.121:0/110533258 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb71811d940 msgr2=0x7fb71811dd20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:33:31.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.575+0000 7fb70effd640 1 --2- 172.21.15.121:0/110533258 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb71811d940 0x7fb71811dd20 secure :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb708009f20 tx=0x7fb70800f610 comp rx=0 tx=0).stop 2024-09-17T18:33:31.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.582+0000 7fb71cc85640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:33:31.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.582+0000 7fb71cc85640 1 --2- 172.21.15.121:0/110533258 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb71811e260 0x7fb71806f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:31.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.582+0000 7fb71cc85640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:31.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.582+0000 7fb71cc85640 1 -- 172.21.15.121:0/110533258 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb7180a6600 con 0x7fb71811e260 2024-09-17T18:33:31.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.582+0000 7fb71659c640 1 --2- 172.21.15.121:0/110533258 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb71811e260 0x7fb71806f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:31.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.582+0000 7fb71659c640 10 monclient: get_auth_request con 0x7fb71811e260 auth_method 0 2024-09-17T18:33:31.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.582+0000 7fb71659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:31.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.582+0000 7fb71659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:31.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.582+0000 7fb71659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:31.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.582+0000 7fb71659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:31.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.582+0000 7fb71659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:31.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.582+0000 7fb71659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:31.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.582+0000 7fb71659c640 10 monclient(hunting): handle_auth_done global_id 64400 payload 824 2024-09-17T18:33:31.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.582+0000 7fb71659c640 1 -- 172.21.15.121:0/110533258 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb718135500 con 0x7fb71811e260 2024-09-17T18:33:31.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.584+0000 7fb71659c640 1 -- 172.21.15.121:0/110533258 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb71811e260 msgr2=0x7fb71806f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:31.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.584+0000 7fb70effd640 10 client.64394.objecter ms_handle_connect 0x7fb71811e260 2024-09-17T18:33:31.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.584+0000 7fb70effd640 10 client.64394.objecter resend_mon_ops 2024-09-17T18:33:31.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.584+0000 7fb71659c640 1 --2- 172.21.15.121:0/110533258 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb71811e260 0x7fb71806f2c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb708009f20 tx=0x7fb7080046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:31.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.584+0000 7fb70effd640 1 -- 172.21.15.121:0/110533258 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb708003c30 con 0x7fb71811e260 2024-09-17T18:33:31.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.584+0000 7fb70effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:31.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.584+0000 7fb70effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:33:31.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.584+0000 7fb70effd640 10 monclient: dump: 2024-09-17T18:33:31.596 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:31.596 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:31.596 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.596 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.596 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:31.596 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:31.596 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:31.596 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:31.596 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:31.597 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:31.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.584+0000 7fb70effd640 1 -- 172.21.15.121:0/110533258 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7fb708003c30 con 0x7fb71811e260 2024-09-17T18:33:31.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.584+0000 7fb70effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:33:31.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.584+0000 7fb70effd640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:33:31.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.584+0000 7fb70effd640 1 -- 172.21.15.121:0/110533258 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb71811e260 msgr2=0x7fb71806f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:31.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.584+0000 7fb70effd640 1 --2- 172.21.15.121:0/110533258 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb71811e260 0x7fb71806f2c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb708009f20 tx=0x7fb7080046c0 comp rx=0 tx=0).stop 2024-09-17T18:33:31.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.584+0000 7fb70effd640 1 -- 172.21.15.121:0/110533258 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb71811e260 msgr2=0x7fb71806f2c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:33:31.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.584+0000 7fb70effd640 1 --2- 172.21.15.121:0/110533258 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb71811e260 0x7fb71806f2c0 secure :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb708009f20 tx=0x7fb7080046c0 comp rx=0 tx=0).stop 2024-09-17T18:33:31.597 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:33:31.597 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:33:31.597 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:33:31.597 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:33:31.598 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 54, 2024-09-17T18:33:31.598 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 390317, 2024-09-17T18:33:31.598 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:33:31.598 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:33:31.598 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:33:31.598 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:33:31.598 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:33:31.598 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 2, 2024-09-17T18:33:31.598 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:33:31.598 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:33:31.598 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:33:31.598 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:33:31.598 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:33:31.598 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:33:31.598 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:33:31.599 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:33:31.599 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:33:31.599 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:33:31.599 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:33:31.599 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:33:31.599 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:33:31.599 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:33:31.599 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:33:31.599 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:33:31.599 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:33:31.599 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:33:31.599 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:33:31.599 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:33:31.599 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:33:31.599 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:33:31.600 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:33:31.600 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:33:31.600 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:33:31.600 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:33:31.600 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:33:31.600 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:33:31.600 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:31.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:31.600 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:33:31.600 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:33:31.600 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:33:31.600 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:33:31.600 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:33:31.600 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:33:31.600 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:33:31.601 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:33:31.601 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:33:31.601 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:33:31.601 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:33:31.601 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:33:31.601 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:33:31.601 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:33:31.601 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:33:31.601 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:33:31.601 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:33:31.601 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:33:31.601 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:33:31.601 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:33:31.602 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:33:31.602 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:33:31.602 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:33:31.602 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:33:31.602 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:33:31.602 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:33:31.602 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:33:31.602 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:33:31.602 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:33:31.602 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:33:31.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:31.602 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:33:31.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:31.602 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:33:31.603 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:33:31.603 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:33:31.603 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:33:31.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:31.603 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:33:31.603 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:33:31.603 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:31.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:31.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:31.603 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:33:31.603 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:33:31.603 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:31.603 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:31.603 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:31.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:31.604 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:33:31.604 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:33:31.604 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:33:31.604 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:33:31.604 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:33:31.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:31.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:31.604 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:33:31.604 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:33:31.604 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:33:31.604 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:33:31.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:31.604 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:33:31.604 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:33:31.604 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:31.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:31.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:31.605 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:33:31.605 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:33:31.605 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:31.605 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:31.605 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:31.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:31.605 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:33:31.605 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:33:31.605 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:33:31.605 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:33:31.605 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:33:31.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:31.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:31.605 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:33:31.606 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:33:31.606 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:33:31.606 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:33:31.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:31.606 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:33:31.606 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:33:31.606 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:31.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:31.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:31.606 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:33:31.606 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:33:31.606 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:31.606 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:31.606 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:31.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:31.606 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:33:31.607 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:33:31.607 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:33:31.607 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:33:31.607 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:33:31.607 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:31.607 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:31.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:31.607 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:33:31.607 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:33:31.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:31.607 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:33:31.607 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:33:31.607 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:33:31.607 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:31.607 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:33:31.607 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:33:31.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:31.608 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:33:31.608 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:33:31.608 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 6 2024-09-17T18:33:31.608 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:31.608 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:33:31.608 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:33:31.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:31.608 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:33:31.608 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:33:31.608 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 4 2024-09-17T18:33:31.608 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:31.608 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:31.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:31.609 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:33:31.609 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:33:31.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.585+0000 7fb71cc85640 10 librados: watch_flush enter 2024-09-17T18:33:31.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.585+0000 7fb71cc85640 10 librados: watch_flush exit 2024-09-17T18:33:31.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.585+0000 7fb71cc85640 20 client.64394.objecter shutdown clearing up homeless session... 2024-09-17T18:33:31.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.585+0000 7fb71cc85640 10 client.64394.objecter successfully canceled tick 2024-09-17T18:33:31.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.585+0000 7fb71cc85640 1 -- 172.21.15.121:0/110533258 >> v2:172.21.15.173:6812/2423786037 conn(0x7fb6f8040170 msgr2=0x7fb6f8042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:31.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.585+0000 7fb71cc85640 1 --2- 172.21.15.121:0/110533258 >> v2:172.21.15.173:6812/2423786037 conn(0x7fb6f8040170 0x7fb6f8042630 secure :-1 s=READY pgs=4055 cs=0 l=1 rev1=1 crypto rx=0x7fb71811fdf0 tx=0x7fb700009040 comp rx=0 tx=0).stop 2024-09-17T18:33:31.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.585+0000 7fb71cc85640 10 monclient: shutdown 2024-09-17T18:33:31.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.585+0000 7fb71cc85640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:33:31.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.585+0000 7fb71cc85640 1 -- 172.21.15.121:0/110533258 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb718135090 msgr2=0x7fb7181341e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:31.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.585+0000 7fb716d9d640 1 -- 172.21.15.121:0/110533258 reap_dead start 2024-09-17T18:33:31.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.585+0000 7fb71cc85640 1 --2- 172.21.15.121:0/110533258 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb718135090 0x7fb7181341e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb710031000 tx=0x7fb710004a40 comp rx=0 tx=0).stop 2024-09-17T18:33:31.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.585+0000 7fb71cc85640 1 -- 172.21.15.121:0/110533258 shutdown_connections 2024-09-17T18:33:31.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.585+0000 7fb71cc85640 1 --2- 172.21.15.121:0/110533258 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb718135090 0x7fb7181341e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:31.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.585+0000 7fb71cc85640 1 -- 172.21.15.121:0/110533258 >> 172.21.15.121:0/110533258 conn(0x7fb7180a7830 msgr2=0x7fb7181bcc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:33:31.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.585+0000 7fb71cc85640 1 -- 172.21.15.121:0/110533258 shutdown_connections 2024-09-17T18:33:31.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.585+0000 7fb71cc85640 1 -- 172.21.15.121:0/110533258 wait complete. 2024-09-17T18:33:31.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.585+0000 7fb71cc85640 1 librados: shutdown 2024-09-17T18:33:31.612 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:33:31.612 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T18:33:31.612 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T18:33:31.616 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.614+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.616 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.614+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3975728151}) 2024-09-17T18:33:31.616 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.614+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1997879311}) 2024-09-17T18:33:31.616 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.614+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.616 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.614+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3975728151}) 2024-09-17T18:33:31.616 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.614+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2528996000}) 2024-09-17T18:33:31.617 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.615+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.617 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.615+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=547861032}) 2024-09-17T18:33:31.617 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.615+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=259541560}) 2024-09-17T18:33:31.617 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.615+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.617 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.615+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=547861032}) 2024-09-17T18:33:31.617 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.615+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1969914345}) 2024-09-17T18:33:31.618 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.616+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.618 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.616+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4061558495}) 2024-09-17T18:33:31.618 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.616+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2803351007}) 2024-09-17T18:33:31.618 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.616+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.618 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.616+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4061558495}) 2024-09-17T18:33:31.618 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.616+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=234915031}) 2024-09-17T18:33:31.620 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.617+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.620 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.617+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3340291952}) 2024-09-17T18:33:31.620 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.617+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2170862967}) 2024-09-17T18:33:31.621 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.617+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.621 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.617+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3340291952}) 2024-09-17T18:33:31.621 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.617+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1256133694}) 2024-09-17T18:33:31.621 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.618+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.621 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.618+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2246306204}) 2024-09-17T18:33:31.621 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.618+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2759264084}) 2024-09-17T18:33:31.621 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.618+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.621 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.618+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2246306204}) 2024-09-17T18:33:31.621 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.618+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1992500005}) 2024-09-17T18:33:31.621 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.618+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.621 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.618+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=967186982}) 2024-09-17T18:33:31.621 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.618+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2415544196}) 2024-09-17T18:33:31.621 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.618+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.622 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.618+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=967186982}) 2024-09-17T18:33:31.622 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.618+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3339588501}) 2024-09-17T18:33:31.623 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.619+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.623 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.619+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2557085472}) 2024-09-17T18:33:31.624 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.619+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3902837635}) 2024-09-17T18:33:31.624 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.619+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.624 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.619+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2557085472}) 2024-09-17T18:33:31.624 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.619+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3822940233}) 2024-09-17T18:33:31.624 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.620+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.624 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.620+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=476445521}) 2024-09-17T18:33:31.624 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.620+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3632874477}) 2024-09-17T18:33:31.624 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.620+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.624 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.620+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=476445521}) 2024-09-17T18:33:31.624 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.620+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=169358932}) 2024-09-17T18:33:31.624 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.621+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.624 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.621+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3450908691}) 2024-09-17T18:33:31.624 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.621+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3188480222}) 2024-09-17T18:33:31.624 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.621+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.625 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.621+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3450908691}) 2024-09-17T18:33:31.625 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.621+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2533754076}) 2024-09-17T18:33:31.625 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.621+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.625 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.621+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3863130219}) 2024-09-17T18:33:31.625 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.621+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=171220741}) 2024-09-17T18:33:31.625 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.622+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.625 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.622+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3863130219}) 2024-09-17T18:33:31.625 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.622+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4132528109}) 2024-09-17T18:33:31.627 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.622+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.627 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.622+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1123483333}) 2024-09-17T18:33:31.627 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.622+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3293821580}) 2024-09-17T18:33:31.627 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.622+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.627 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.622+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1123483333}) 2024-09-17T18:33:31.627 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.622+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2945337427}) 2024-09-17T18:33:31.627 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.623+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.627 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.623+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2811160165}) 2024-09-17T18:33:31.627 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.623+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=321528511}) 2024-09-17T18:33:31.628 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.623+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.628 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.623+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2811160165}) 2024-09-17T18:33:31.628 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.623+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4242947740}) 2024-09-17T18:33:31.628 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.624+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.628 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.624+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=874281470}) 2024-09-17T18:33:31.628 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.624+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2841544654}) 2024-09-17T18:33:31.628 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.624+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.628 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.624+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=874281470}) 2024-09-17T18:33:31.628 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.624+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1471534555}) 2024-09-17T18:33:31.628 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.625+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.628 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.625+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2370935840}) 2024-09-17T18:33:31.628 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.625+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2352387515}) 2024-09-17T18:33:31.628 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.625+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.628 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.625+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2370935840}) 2024-09-17T18:33:31.629 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.625+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3367302406}) 2024-09-17T18:33:31.631 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.626+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.631 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.626+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=67225045}) 2024-09-17T18:33:31.631 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.626+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=913525067}) 2024-09-17T18:33:31.631 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.626+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.631 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.626+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=67225045}) 2024-09-17T18:33:31.632 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.626+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=159644718}) 2024-09-17T18:33:31.632 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.626+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.632 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.626+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=451944127}) 2024-09-17T18:33:31.632 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.627+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1953891381}) 2024-09-17T18:33:31.632 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.627+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.632 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.627+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=451944127}) 2024-09-17T18:33:31.632 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.627+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1678501526}) 2024-09-17T18:33:31.632 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.627+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.632 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.627+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1125084235}) 2024-09-17T18:33:31.632 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.627+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1679728627}) 2024-09-17T18:33:31.632 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.627+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.632 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.627+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1125084235}) 2024-09-17T18:33:31.632 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.627+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1344326883}) 2024-09-17T18:33:31.632 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.628+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.632 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.628+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3156081913}) 2024-09-17T18:33:31.633 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.628+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1118560113}) 2024-09-17T18:33:31.633 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.628+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.633 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.628+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3156081913}) 2024-09-17T18:33:31.633 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.628+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=857547508}) 2024-09-17T18:33:31.633 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.629+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.633 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.629+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3732441289}) 2024-09-17T18:33:31.633 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.629+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4082240608}) 2024-09-17T18:33:31.633 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.629+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.633 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.629+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3732441289}) 2024-09-17T18:33:31.633 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.629+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2632009180}) 2024-09-17T18:33:31.638 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.630+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.638 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.630+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2080531700}) 2024-09-17T18:33:31.638 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.630+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3621393722}) 2024-09-17T18:33:31.638 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.630+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.638 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.630+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2080531700}) 2024-09-17T18:33:31.638 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.630+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1259894965}) 2024-09-17T18:33:31.638 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.631+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.638 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.631+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2974571976}) 2024-09-17T18:33:31.638 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.631+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=354422836}) 2024-09-17T18:33:31.638 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.631+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.639 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.631+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2974571976}) 2024-09-17T18:33:31.639 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.631+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=587683862}) 2024-09-17T18:33:31.639 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.632+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.639 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.632+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3230119731}) 2024-09-17T18:33:31.639 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.632+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2848030792}) 2024-09-17T18:33:31.639 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.632+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.639 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.632+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3230119731}) 2024-09-17T18:33:31.639 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.632+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=935700939}) 2024-09-17T18:33:31.639 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.632+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.639 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.632+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3456002795}) 2024-09-17T18:33:31.639 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.632+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1381999592}) 2024-09-17T18:33:31.639 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.632+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.639 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.632+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3456002795}) 2024-09-17T18:33:31.639 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.632+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=334105930}) 2024-09-17T18:33:31.640 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.633+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.640 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.633+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=63,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=15} crc {logm=2093972014,mdsmap=761013554,mgr=2893552865,mgr_command_descs=1957073259,mgr_metadata=402068389,mgrstat=3421887247,mon_config_key=965173948}) 2024-09-17T18:33:31.640 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.633+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=65,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=12} crc {logm=2360285439,mdsmap=761013554,mgr=3361498112,mgr_command_descs=1957073259,mgr_metadata=402068389,mgrstat=3421887247,mon_config_key=4141692420}) 2024-09-17T18:33:31.640 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.633+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.640 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.633+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=63,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=15} crc {logm=2093972014,mdsmap=761013554,mgr=2893552865,mgr_command_descs=1957073259,mgr_metadata=402068389,mgrstat=3421887247,mon_config_key=965173948}) 2024-09-17T18:33:31.640 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.633+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=70,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=7} crc {logm=3481883456,mdsmap=761013554,mgr=3361498112,mgr_command_descs=1957073259,mgr_metadata=402068389,mgrstat=3421887247,mon_config_key=168581242}) 2024-09-17T18:33:31.640 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.634+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.640 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.634+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=2,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=35,osdmap=51} crc {mon_config_key=3829049047,monmap=344458555,nvmeofgw=598337686,osd_metadata=121115626,osd_pg_creating=679042580,osd_snap=4263905449,osdmap=3749964874}) 2024-09-17T18:33:31.640 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.634+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=2,monmap=3,nvmeofgw=3,osd_metadata=6,osd_snap=34,osdmap=52} crc {mon_config_key=3829049047,monmap=2277444362,nvmeofgw=598337686,osd_metadata=1345550747,osd_snap=3815863125,osdmap=1932033554}) 2024-09-17T18:33:31.640 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.634+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.640 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.634+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=2,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=35,osdmap=51} crc {mon_config_key=3829049047,monmap=344458555,nvmeofgw=598337686,osd_metadata=121115626,osd_pg_creating=679042580,osd_snap=4263905449,osdmap=3749964874}) 2024-09-17T18:33:31.640 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.634+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=2,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=34,osdmap=51} crc {mon_config_key=3829049047,monmap=2277444362,nvmeofgw=598337686,osd_metadata=121115626,osd_pg_creating=679042580,osd_snap=3599683538,osdmap=3749964874}) 2024-09-17T18:33:31.640 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.635+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.640 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.635+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3769753356}) 2024-09-17T18:33:31.640 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.635+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1267926120}) 2024-09-17T18:33:31.641 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.635+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.641 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.635+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3769753356}) 2024-09-17T18:33:31.641 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.635+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1251547013}) 2024-09-17T18:33:31.641 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.636+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.641 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.636+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4026182652}) 2024-09-17T18:33:31.641 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.636+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=140444837}) 2024-09-17T18:33:31.641 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.636+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.641 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.636+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4026182652}) 2024-09-17T18:33:31.641 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.636+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1555248216}) 2024-09-17T18:33:31.642 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.637+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.642 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.637+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3055636536}) 2024-09-17T18:33:31.642 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.637+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3590997122}) 2024-09-17T18:33:31.642 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.637+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.642 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.637+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3055636536}) 2024-09-17T18:33:31.643 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.637+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2475107464}) 2024-09-17T18:33:31.643 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.639+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.643 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.639+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4250644613}) 2024-09-17T18:33:31.643 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.639+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1367348998}) 2024-09-17T18:33:31.643 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.639+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.643 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.639+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4250644613}) 2024-09-17T18:33:31.643 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.639+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=783561130}) 2024-09-17T18:33:31.645 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.643+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.645 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.643+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=17} crc {osdmap=2126186032}) 2024-09-17T18:33:31.645 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.643+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=15} crc {osdmap=153547325}) 2024-09-17T18:33:31.645 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.643+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:33:31.645 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.643+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=17} crc {osdmap=2126186032}) 2024-09-17T18:33:31.646 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:33:31.643+0000 7fefb72bc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=16} crc {osdmap=2729187942}) 2024-09-17T18:33:31.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.711+0000 7fd92b050640 10 monclient: get_monmap_and_config 2024-09-17T18:33:31.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.711+0000 7fd92b050640 10 monclient: build_initial_monmap 2024-09-17T18:33:31.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.711+0000 7fd92b050640 10 monclient: monmap: 2024-09-17T18:33:31.713 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:33:31.713 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:31.713 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:33:31.712010+0000 2024-09-17T18:33:31.713 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:33:31.712010+0000 2024-09-17T18:33:31.713 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:33:31.713 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:31.713 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:33:31.713 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:33:31.713 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:33:31.714 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:33:31.714 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:33:31.714 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:31.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.712+0000 7fd92b050640 1 Processor -- start 2024-09-17T18:33:31.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.712+0000 7fd92b050640 1 -- start start 2024-09-17T18:33:31.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.712+0000 7fd92b050640 10 monclient: init 2024-09-17T18:33:31.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.712+0000 7fd92b050640 10 monclient: _reopen_session rank -1 2024-09-17T18:33:31.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.712+0000 7fd92b050640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-17T18:33:31.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.712+0000 7fd92b050640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9240700f0 0x7fd924134e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:31.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.712+0000 7fd92b050640 10 monclient(hunting): picked mon.noname-b con 0x7fd9240700f0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:33:31.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.712+0000 7fd92b050640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fd92409e810 0x7fd924138360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:31.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.712+0000 7fd92b050640 10 monclient(hunting): picked mon.noname-c con 0x7fd92409e810 addr v2:172.21.15.173:3300/0 2024-09-17T18:33:31.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.712+0000 7fd92b050640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd92409ec30 0x7fd9241388a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:31.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.712+0000 7fd92b050640 10 monclient(hunting): picked mon.noname-a con 0x7fd92409ec30 addr v2:172.21.15.121:3300/0 2024-09-17T18:33:31.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.712+0000 7fd92b050640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:31.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.712+0000 7fd92b050640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fd9240704d0 con 0x7fd92409ec30 2024-09-17T18:33:31.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.712+0000 7fd92b050640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:31.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.712+0000 7fd92b050640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fd92409ca60 con 0x7fd92409e810 2024-09-17T18:33:31.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.712+0000 7fd92b050640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:31.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.712+0000 7fd92b050640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd92409cd20 con 0x7fd9240700f0 2024-09-17T18:33:31.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.712+0000 7fd92b050640 10 monclient(hunting): _renew_subs 2024-09-17T18:33:31.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.712+0000 7fd92b050640 10 monclient(hunting): authenticate will time out at 1615.428746s 2024-09-17T18:33:31.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.712+0000 7fd928dec640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9240700f0 0x7fd924134e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:31.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.712+0000 7fd9295ed640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd92409ec30 0x7fd9241388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:31.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.712+0000 7fd9295ed640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd92409ec30 0x7fd9241388a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60498/0 (socket says 172.21.15.121:60498) 2024-09-17T18:33:31.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.712+0000 7fd9295ed640 1 -- 172.21.15.121:0/3937496311 learned_addr learned my addr 172.21.15.121:0/3937496311 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:33:31.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.712+0000 7fd9295ed640 10 monclient(hunting): get_auth_request con 0x7fd92409ec30 auth_method 0 2024-09-17T18:33:31.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.712+0000 7fd9295ed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:31.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.712+0000 7fd9295ed640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:31.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.712+0000 7fd9295ed640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:31.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.712+0000 7fd928dec640 1 --2- 172.21.15.121:0/3937496311 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9240700f0 0x7fd924134e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:46826/0 (socket says 172.21.15.121:46826) 2024-09-17T18:33:31.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.712+0000 7fd928dec640 10 monclient(hunting): get_auth_request con 0x7fd9240700f0 auth_method 0 2024-09-17T18:33:31.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.712+0000 7fd928dec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:31.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.712+0000 7fd928dec640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:31.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.712+0000 7fd928dec640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:31.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.712+0000 7fd9295ed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:31.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.712+0000 7fd9295ed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:31.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.713+0000 7fd9295ed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:31.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.713+0000 7fd928dec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:31.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.713+0000 7fd928dec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:31.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.713+0000 7fd928dec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:31.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.713+0000 7fd923fff640 1 --2- 172.21.15.121:0/3937496311 >> v2:172.21.15.173:3300/0 conn(0x7fd92409e810 0x7fd924138360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:31.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.713+0000 7fd9295ed640 10 monclient(hunting): handle_auth_done global_id 94152 payload 274 2024-09-17T18:33:31.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.713+0000 7fd9295ed640 1 -- 172.21.15.121:0/3937496311 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9240700f0 msgr2=0x7fd924134e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:33:31.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.713+0000 7fd9295ed640 1 --2- 172.21.15.121:0/3937496311 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9240700f0 0x7fd924134e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:31.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.713+0000 7fd9295ed640 1 -- 172.21.15.121:0/3937496311 >> v2:172.21.15.173:3300/0 conn(0x7fd92409e810 msgr2=0x7fd924138360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:33:31.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.713+0000 7fd9295ed640 1 --2- 172.21.15.121:0/3937496311 >> v2:172.21.15.173:3300/0 conn(0x7fd92409e810 0x7fd924138360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:31.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.713+0000 7fd9295ed640 10 monclient: _finish_hunting 0 2024-09-17T18:33:31.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.713+0000 7fd9295ed640 1 monclient: found mon.noname-a 2024-09-17T18:33:31.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.713+0000 7fd9295ed640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:33:31.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.713+0000 7fd9295ed640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:33:31.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.713+0000 7fd9295ed640 1 -- 172.21.15.121:0/3937496311 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd92409d000 con 0x7fd92409ec30 2024-09-17T18:33:31.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.713+0000 7fd923fff640 10 monclient: get_auth_request con 0x7fd92409e810 auth_method 0 2024-09-17T18:33:31.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.713+0000 7fd923fff640 1 --2- 172.21.15.121:0/3937496311 >> v2:172.21.15.173:3300/0 conn(0x7fd92409e810 0x7fd924138360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T18:33:31.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.713+0000 7fd928dec640 1 --2- 172.21.15.121:0/3937496311 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9240700f0 0x7fd924134e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:33:31.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.715+0000 7fd9295ed640 1 -- 172.21.15.121:0/3937496311 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd92409ec30 msgr2=0x7fd9241388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:31.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.715+0000 7fd9295ed640 1 --2- 172.21.15.121:0/3937496311 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd92409ec30 0x7fd9241388a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd924070720 tx=0x7fd91c02a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f27d110017ddb4de server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:31.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.716+0000 7fd9237fe640 1 -- 172.21.15.121:0/3937496311 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd91c036070 con 0x7fd92409ec30 2024-09-17T18:33:31.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.716+0000 7fd9237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:31.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.716+0000 7fd9237fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:33:31.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.716+0000 7fd9237fe640 10 monclient: dump: 2024-09-17T18:33:31.728 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:31.728 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:31.728 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.728 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.728 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:31.728 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:31.728 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:31.728 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:31.728 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:31.728 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:31.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.716+0000 7fd9237fe640 10 monclient: _finish_auth 0 2024-09-17T18:33:31.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.716+0000 7fd9237fe640 10 monclient: _check_auth_tickets 2024-09-17T18:33:31.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.716+0000 7fd9237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:33:31.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.716+0000 7fd9237fe640 1 -- 172.21.15.121:0/3937496311 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd91c007800 con 0x7fd92409ec30 2024-09-17T18:33:31.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.716+0000 7fd92b050640 5 monclient: authenticate success, global_id 94152 2024-09-17T18:33:31.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.716+0000 7fd92b050640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:33:31.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.716+0000 7fd9237fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:33:31.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.716+0000 7fd9237fe640 1 -- 172.21.15.121:0/3937496311 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd91c007bd0 con 0x7fd92409ec30 2024-09-17T18:33:31.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.716+0000 7fd92b050640 10 monclient: get_monmap_and_config success 2024-09-17T18:33:31.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.716+0000 7fd92b050640 4 set_mon_vals no callback set 2024-09-17T18:33:31.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.716+0000 7fd9237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:31.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.716+0000 7fd9237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:33:31.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.716+0000 7fd9237fe640 10 monclient: dump: 2024-09-17T18:33:31.729 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:31.729 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:31.730 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.730 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.730 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:31.730 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:31.730 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:31.730 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:31.730 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:31.730 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:31.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.716+0000 7fd92b050640 10 monclient: shutdown 2024-09-17T18:33:31.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.716+0000 7fd92b050640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:33:31.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.716+0000 7fd92b050640 1 -- 172.21.15.121:0/3937496311 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd92409ec30 msgr2=0x7fd9241388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:31.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.716+0000 7fd92b050640 1 --2- 172.21.15.121:0/3937496311 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd92409ec30 0x7fd9241388a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd924070720 tx=0x7fd91c02a610 comp rx=0 tx=0).stop 2024-09-17T18:33:31.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.716+0000 7fd92b050640 1 -- 172.21.15.121:0/3937496311 shutdown_connections 2024-09-17T18:33:31.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.716+0000 7fd92b050640 1 --2- 172.21.15.121:0/3937496311 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd92409ec30 0x7fd9241388a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:31.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.716+0000 7fd92b050640 1 --2- 172.21.15.121:0/3937496311 >> v2:172.21.15.173:3300/0 conn(0x7fd92409e810 0x7fd924138360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:31.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.716+0000 7fd92b050640 1 --2- 172.21.15.121:0/3937496311 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9240700f0 0x7fd924134e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:31.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.716+0000 7fd92b050640 1 -- 172.21.15.121:0/3937496311 >> 172.21.15.121:0/3937496311 conn(0x7fd9240ab830 msgr2=0x7fd9240a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:33:31.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.716+0000 7fd92b050640 1 -- 172.21.15.121:0/3937496311 shutdown_connections 2024-09-17T18:33:31.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.716+0000 7fd92b050640 1 -- 172.21.15.121:0/3937496311 wait complete. 2024-09-17T18:33:31.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.717+0000 7fd92b050640 10 monclient: build_initial_monmap 2024-09-17T18:33:31.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.717+0000 7fd92b050640 10 monclient: monmap: 2024-09-17T18:33:31.731 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:33:31.731 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:33:31.731 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:33:31.731 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:33:31.731 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:33:31.731 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:31.731 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:33:31.731 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:33:31.732 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:33:31.732 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:31.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.717+0000 7fd92b050640 1 librados: starting msgr at 2024-09-17T18:33:31.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.717+0000 7fd92b050640 1 librados: starting objecter 2024-09-17T18:33:31.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.717+0000 7fd92b050640 1 Processor -- start 2024-09-17T18:33:31.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.717+0000 7fd92b050640 1 -- start start 2024-09-17T18:33:31.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.717+0000 7fd92b050640 1 librados: setting wanted keys 2024-09-17T18:33:31.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.717+0000 7fd92b050640 1 librados: calling monclient init 2024-09-17T18:33:31.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.717+0000 7fd92b050640 10 monclient: init 2024-09-17T18:33:31.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd92b050640 10 monclient: _reopen_session rank -1 2024-09-17T18:33:31.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd92b050640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:33:31.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd92b050640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9240700f0 0x7fd9241cd550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:31.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd92b050640 10 monclient(hunting): picked mon.noname-a con 0x7fd9240700f0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:33:31.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd92b050640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd92409e810 0x7fd924131a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:31.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd92b050640 10 monclient(hunting): picked mon.noname-b con 0x7fd92409e810 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:33:31.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd92b050640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd92409ec30 0x7fd924131f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:31.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd92b050640 10 monclient(hunting): picked mon.noname-c con 0x7fd92409ec30 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:33:31.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd92b050640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:31.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd92b050640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd924135510 con 0x7fd9240700f0 2024-09-17T18:33:31.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd92b050640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:31.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd92b050640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd924135390 con 0x7fd92409e810 2024-09-17T18:33:31.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd92b050640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:31.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd92b050640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd924138de0 con 0x7fd92409ec30 2024-09-17T18:33:31.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd92b050640 10 monclient(hunting): _renew_subs 2024-09-17T18:33:31.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd92b050640 10 monclient(hunting): authenticate will time out at 1615.434343s 2024-09-17T18:33:31.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd928dec640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9240700f0 0x7fd9241cd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:31.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd928dec640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9240700f0 0x7fd9241cd550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60504/0 (socket says 172.21.15.121:60504) 2024-09-17T18:33:31.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd928dec640 1 -- 172.21.15.121:0/4167340058 learned_addr learned my addr 172.21.15.121:0/4167340058 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:33:31.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd928dec640 10 monclient(hunting): get_auth_request con 0x7fd9240700f0 auth_method 0 2024-09-17T18:33:31.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd928dec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:31.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd928dec640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:31.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd928dec640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:31.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd9295ed640 1 --2- 172.21.15.121:0/4167340058 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd92409ec30 0x7fd924131f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:31.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd9295ed640 10 monclient(hunting): get_auth_request con 0x7fd92409ec30 auth_method 0 2024-09-17T18:33:31.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd9295ed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:31.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd9295ed640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:31.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd9295ed640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:31.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd928dec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:31.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd928dec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:31.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd928dec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:31.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd9295ed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:31.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd9295ed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:31.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd923fff640 1 --2- 172.21.15.121:0/4167340058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd92409e810 0x7fd924131a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:31.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd9295ed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:31.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd923fff640 10 monclient(hunting): get_auth_request con 0x7fd92409e810 auth_method 0 2024-09-17T18:33:31.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd923fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:31.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd923fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:31.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd923fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:31.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd923fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:31.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd923fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:31.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd923fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:31.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd928dec640 10 monclient(hunting): handle_auth_done global_id 94155 payload 824 2024-09-17T18:33:31.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd928dec640 1 -- 172.21.15.121:0/4167340058 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd92409ec30 msgr2=0x7fd924131f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:31.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd928dec640 1 --2- 172.21.15.121:0/4167340058 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd92409ec30 0x7fd924131f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:31.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd928dec640 1 -- 172.21.15.121:0/4167340058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd92409e810 msgr2=0x7fd924131a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:31.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd928dec640 1 --2- 172.21.15.121:0/4167340058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd92409e810 0x7fd924131a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:31.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd928dec640 10 monclient: _finish_hunting 0 2024-09-17T18:33:31.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd928dec640 1 monclient: found mon.noname-a 2024-09-17T18:33:31.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd928dec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:33:31.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd928dec640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:33:31.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd928dec640 1 -- 172.21.15.121:0/4167340058 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd924123140 con 0x7fd9240700f0 2024-09-17T18:33:31.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.718+0000 7fd9295ed640 1 --2- 172.21.15.121:0/4167340058 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd92409ec30 0x7fd924131f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:33:31.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.720+0000 7fd928dec640 1 -- 172.21.15.121:0/4167340058 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9240700f0 msgr2=0x7fd9241cd550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:31.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.721+0000 7fd928dec640 1 --2- 172.21.15.121:0/4167340058 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9240700f0 0x7fd9241cd550 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd91800ebe0 tx=0x7fd91800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:31.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.721+0000 7fd9217fa640 10 client.?.objecter ms_handle_connect 0x7fd9240700f0 2024-09-17T18:33:31.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.721+0000 7fd9217fa640 10 client.?.objecter resend_mon_ops 2024-09-17T18:33:31.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.721+0000 7fd9217fa640 1 -- 172.21.15.121:0/4167340058 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd918017070 con 0x7fd9240700f0 2024-09-17T18:33:31.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.721+0000 7fd9217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:31.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.721+0000 7fd9217fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:33:31.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.721+0000 7fd9217fa640 10 monclient: dump: 2024-09-17T18:33:31.737 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:31.737 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:31.737 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.737 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.737 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:31.737 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:31.737 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:31.737 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:31.737 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:31.737 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:31.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.721+0000 7fd9217fa640 10 monclient: _finish_auth 0 2024-09-17T18:33:31.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.721+0000 7fd9217fa640 10 monclient: _check_auth_tickets 2024-09-17T18:33:31.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.721+0000 7fd9217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:33:31.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.721+0000 7fd92b050640 5 monclient: authenticate success, global_id 94155 2024-09-17T18:33:31.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.721+0000 7fd9217fa640 1 -- 172.21.15.121:0/4167340058 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd91800cde0 con 0x7fd9240700f0 2024-09-17T18:33:31.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.721+0000 7fd92b050640 10 monclient: _renew_subs 2024-09-17T18:33:31.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.721+0000 7fd92b050640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:33:31.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.721+0000 7fd92b050640 1 -- 172.21.15.121:0/4167340058 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd924123420 con 0x7fd9240700f0 2024-09-17T18:33:31.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.721+0000 7fd9217fa640 10 monclient: handle_config config(0 keys) 2024-09-17T18:33:31.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.721+0000 7fd9217fa640 1 -- 172.21.15.121:0/4167340058 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd918013690 con 0x7fd9240700f0 2024-09-17T18:33:31.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.721+0000 7fd9217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:31.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.721+0000 7fd9217fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:33:31.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.721+0000 7fd9217fa640 10 monclient: dump: 2024-09-17T18:33:31.738 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:31.738 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:31.739 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.739 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.739 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:31.739 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:31.739 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:31.739 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:31.739 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:31.739 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:31.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.721+0000 7fd92b050640 10 client.94155.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:33:31.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.721+0000 7fd92b050640 10 monclient: _renew_subs 2024-09-17T18:33:31.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.721+0000 7fd922ffd640 4 set_mon_vals no callback set 2024-09-17T18:33:31.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.721+0000 7fd92b050640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:33:31.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.721+0000 7fd92b050640 1 -- 172.21.15.121:0/4167340058 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9241324a0 con 0x7fd9240700f0 2024-09-17T18:33:31.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.721+0000 7fd92b050640 1 librados: init done 2024-09-17T18:33:31.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.722+0000 7fd9217fa640 1 -- 172.21.15.121:0/4167340058 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fd91801a070 con 0x7fd9240700f0 2024-09-17T18:33:31.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.722+0000 7fd92b050640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:33:31.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.722+0000 7fd92b050640 1 --2- 172.21.15.121:0/4167340058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9240a3570 0x7fd9240a3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:31.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.722+0000 7fd92b050640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:31.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.722+0000 7fd92b050640 1 -- 172.21.15.121:0/4167340058 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd9240a4f60 con 0x7fd9240a3570 2024-09-17T18:33:31.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.722+0000 7fd9217fa640 1 --2- 172.21.15.121:0/4167340058 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd9080401c0 0x7fd908042680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:31.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.722+0000 7fd9217fa640 1 -- 172.21.15.121:0/4167340058 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(563..563 src has 247..563) ==== 5516+0+0 (secure 0 0 0) 0x7fd918053730 con 0x7fd9240700f0 2024-09-17T18:33:31.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.722+0000 7fd9217fa640 10 client.94155.objecter ms_dispatch 0x7fd924000b90 osd_map(563..563 src has 247..563) 2024-09-17T18:33:31.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.722+0000 7fd9217fa640 3 client.94155.objecter handle_osd_map got epochs [563,563] > 0 2024-09-17T18:33:31.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.722+0000 7fd9217fa640 3 client.94155.objecter handle_osd_map decoding full epoch 563 2024-09-17T18:33:31.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.722+0000 7fd9217fa640 20 client.94155.objecter dump_active .. 0 homeless 2024-09-17T18:33:31.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.722+0000 7fd923fff640 1 --2- 172.21.15.121:0/4167340058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9240a3570 0x7fd9240a3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:31.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.722+0000 7fd923fff640 10 monclient: get_auth_request con 0x7fd9240a3570 auth_method 0 2024-09-17T18:33:31.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.722+0000 7fd923fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:31.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.722+0000 7fd923fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:31.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.722+0000 7fd923fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:31.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.722+0000 7fd9295ed640 1 --2- 172.21.15.121:0/4167340058 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd9080401c0 0x7fd908042680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:31.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.722+0000 7fd9295ed640 10 monclient: get_auth_request con 0x7fd9080401c0 auth_method 0 2024-09-17T18:33:31.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.722+0000 7fd923fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:31.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.722+0000 7fd923fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:31.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.722+0000 7fd923fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:31.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.722+0000 7fd923fff640 10 monclient(hunting): handle_auth_done global_id 84199 payload 824 2024-09-17T18:33:31.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.722+0000 7fd923fff640 1 -- 172.21.15.121:0/4167340058 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd9240008d0 con 0x7fd9240a3570 2024-09-17T18:33:31.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.725+0000 7fd9217fa640 10 client.94155.objecter ms_handle_connect 0x7fd9240a3570 2024-09-17T18:33:31.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.725+0000 7fd9217fa640 10 client.94155.objecter resend_mon_ops 2024-09-17T18:33:31.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.725+0000 7fd923fff640 1 -- 172.21.15.121:0/4167340058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9240a3570 msgr2=0x7fd9240a3950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:31.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.725+0000 7fd923fff640 1 --2- 172.21.15.121:0/4167340058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9240a3570 0x7fd9240a3950 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd91000f810 tx=0x7fd91000fcd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:31.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.725+0000 7fd9217fa640 10 client.94155.objecter ms_handle_connect 0x7fd9080401c0 2024-09-17T18:33:31.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.725+0000 7fd9295ed640 1 --2- 172.21.15.121:0/4167340058 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd9080401c0 0x7fd908042680 secure :-1 s=READY pgs=4056 cs=0 l=1 rev1=1 crypto rx=0x7fd91c033000 tx=0x7fd91c009000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:31.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.725+0000 7fd9217fa640 1 -- 172.21.15.121:0/4167340058 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd91000de30 con 0x7fd9240a3570 2024-09-17T18:33:31.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.725+0000 7fd9217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:31.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.725+0000 7fd9217fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:33:31.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.725+0000 7fd9217fa640 10 monclient: dump: 2024-09-17T18:33:31.742 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:31.742 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:31.742 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.742 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.742 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:31.742 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:31.742 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:31.742 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:31.742 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:31.742 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:31.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.725+0000 7fd9217fa640 1 -- 172.21.15.121:0/4167340058 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd91000de30 con 0x7fd9240a3570 2024-09-17T18:33:31.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.725+0000 7fd9217fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:33:31.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.725+0000 7fd9217fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:33:31.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.725+0000 7fd9217fa640 1 -- 172.21.15.121:0/4167340058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9240a3570 msgr2=0x7fd9240a3950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:31.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.725+0000 7fd9217fa640 1 --2- 172.21.15.121:0/4167340058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9240a3570 0x7fd9240a3950 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd91000f810 tx=0x7fd91000fcd0 comp rx=0 tx=0).stop 2024-09-17T18:33:31.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.725+0000 7fd9217fa640 1 -- 172.21.15.121:0/4167340058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9240a3570 msgr2=0x7fd9240a3950 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:33:31.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.725+0000 7fd9217fa640 1 --2- 172.21.15.121:0/4167340058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9240a3570 0x7fd9240a3950 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:31.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.731+0000 7fd92b050640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:33:31.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.731+0000 7fd92b050640 1 --2- 172.21.15.121:0/4167340058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9240a3e90 0x7fd92409f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:31.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.731+0000 7fd92b050640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:31.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.731+0000 7fd92b050640 1 -- 172.21.15.121:0/4167340058 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd92406a660 con 0x7fd9240a3e90 2024-09-17T18:33:31.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.732+0000 7fd923fff640 1 --2- 172.21.15.121:0/4167340058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9240a3e90 0x7fd92409f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:31.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.732+0000 7fd923fff640 10 monclient: get_auth_request con 0x7fd9240a3e90 auth_method 0 2024-09-17T18:33:31.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.732+0000 7fd923fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:31.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.732+0000 7fd923fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:31.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.732+0000 7fd923fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:31.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.732+0000 7fd923fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:31.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.732+0000 7fd923fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:31.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.732+0000 7fd923fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:31.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.732+0000 7fd923fff640 10 monclient(hunting): handle_auth_done global_id 84202 payload 824 2024-09-17T18:33:31.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.732+0000 7fd923fff640 1 -- 172.21.15.121:0/4167340058 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd9240a42a0 con 0x7fd9240a3e90 2024-09-17T18:33:31.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.734+0000 7fd923fff640 1 -- 172.21.15.121:0/4167340058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9240a3e90 msgr2=0x7fd92409f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:31.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.734+0000 7fd923fff640 1 --2- 172.21.15.121:0/4167340058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9240a3e90 0x7fd92409f100 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd91000f810 tx=0x7fd9100046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:31.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.734+0000 7fd9217fa640 10 client.94155.objecter ms_handle_connect 0x7fd9240a3e90 2024-09-17T18:33:31.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.734+0000 7fd9217fa640 10 client.94155.objecter resend_mon_ops 2024-09-17T18:33:31.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.735+0000 7fd9217fa640 1 -- 172.21.15.121:0/4167340058 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd910006700 con 0x7fd9240a3e90 2024-09-17T18:33:31.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.735+0000 7fd9217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:31.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.735+0000 7fd9217fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:33:31.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.735+0000 7fd9217fa640 10 monclient: dump: 2024-09-17T18:33:31.746 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:31.746 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:31.746 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.746 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.746 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:31.746 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:31.746 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:31.746 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:31.746 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:31.746 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:31.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.735+0000 7fd9217fa640 1 -- 172.21.15.121:0/4167340058 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7fd910006700 con 0x7fd9240a3e90 2024-09-17T18:33:31.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.735+0000 7fd9217fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:33:31.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.735+0000 7fd9217fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:33:31.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.735+0000 7fd9217fa640 1 -- 172.21.15.121:0/4167340058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9240a3e90 msgr2=0x7fd92409f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:31.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.735+0000 7fd9217fa640 1 --2- 172.21.15.121:0/4167340058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9240a3e90 0x7fd92409f100 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd91000f810 tx=0x7fd9100046c0 comp rx=0 tx=0).stop 2024-09-17T18:33:31.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.735+0000 7fd9217fa640 1 -- 172.21.15.121:0/4167340058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9240a3e90 msgr2=0x7fd92409f100 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:33:31.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.735+0000 7fd9217fa640 1 --2- 172.21.15.121:0/4167340058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9240a3e90 0x7fd92409f100 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd91000f810 tx=0x7fd9100046c0 comp rx=0 tx=0).stop 2024-09-17T18:33:31.747 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:33:31.747 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:33:31.747 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:33:31.747 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:33:31.747 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 54, 2024-09-17T18:33:31.747 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 136833, 2024-09-17T18:33:31.747 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:33:31.748 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:33:31.748 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:33:31.748 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:33:31.748 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:33:31.748 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 2, 2024-09-17T18:33:31.748 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:33:31.748 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:33:31.748 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:33:31.748 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:33:31.748 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:33:31.748 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:33:31.748 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:33:31.748 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:33:31.748 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:33:31.749 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:33:31.749 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:33:31.749 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:33:31.749 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:33:31.749 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:33:31.749 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:33:31.749 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:33:31.749 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:33:31.749 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:33:31.749 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:33:31.749 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:33:31.749 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:33:31.749 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:33:31.749 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:33:31.750 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:33:31.750 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:33:31.750 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:33:31.750 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:33:31.750 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:33:31.750 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:31.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:31.750 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:33:31.750 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:33:31.750 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:33:31.750 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:33:31.750 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:33:31.750 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:33:31.750 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:33:31.750 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:33:31.750 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:33:31.751 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:33:31.751 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:33:31.751 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:33:31.751 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:33:31.751 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:33:31.751 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:33:31.751 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:33:31.751 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:33:31.751 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:33:31.751 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:33:31.751 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:33:31.751 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:33:31.751 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:33:31.751 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:33:31.751 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:33:31.752 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:33:31.752 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:33:31.752 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:33:31.752 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:33:31.752 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:33:31.752 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:33:31.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:31.752 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:33:31.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:31.752 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:33:31.752 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:33:31.752 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:33:31.752 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:33:31.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:31.753 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:33:31.753 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:33:31.753 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:31.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:31.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:31.753 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:33:31.753 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:33:31.753 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:31.753 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:31.753 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:31.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:31.753 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:33:31.753 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:33:31.753 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:33:31.753 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:33:31.754 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:33:31.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:31.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:31.754 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:33:31.754 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:33:31.754 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:33:31.754 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:33:31.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:31.754 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:33:31.754 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:33:31.754 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:31.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:31.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:31.754 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:33:31.754 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:33:31.755 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:31.755 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:31.755 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:31.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:31.755 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:33:31.755 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:33:31.755 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:33:31.755 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:33:31.755 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:33:31.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:31.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:31.755 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:33:31.755 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:33:31.755 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:33:31.755 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:33:31.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:31.756 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:33:31.756 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:33:31.756 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:31.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:31.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:31.756 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:33:31.756 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:33:31.756 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:31.756 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:31.756 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:31.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:31.756 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:33:31.756 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:33:31.756 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:33:31.756 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:33:31.757 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:33:31.757 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:31.757 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:31.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:31.757 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:33:31.757 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:33:31.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:31.757 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:33:31.757 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:33:31.757 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:33:31.757 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:31.757 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:33:31.757 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:33:31.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:31.757 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:33:31.758 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:33:31.758 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 4 2024-09-17T18:33:31.758 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:31.758 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:33:31.758 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": [ 2024-09-17T18:33:31.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:31.758 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:33:31.758 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:33:31.758 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:33:31.758 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:31.758 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:31.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:31.758 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:33:31.758 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:33:31.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.736+0000 7fd92b050640 10 librados: watch_flush enter 2024-09-17T18:33:31.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.736+0000 7fd92b050640 10 librados: watch_flush exit 2024-09-17T18:33:31.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.736+0000 7fd92b050640 20 client.94155.objecter shutdown clearing up homeless session... 2024-09-17T18:33:31.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.736+0000 7fd92b050640 10 client.94155.objecter successfully canceled tick 2024-09-17T18:33:31.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.736+0000 7fd92b050640 1 -- 172.21.15.121:0/4167340058 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd9080401c0 msgr2=0x7fd908042680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:31.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.736+0000 7fd92b050640 1 --2- 172.21.15.121:0/4167340058 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd9080401c0 0x7fd908042680 secure :-1 s=READY pgs=4056 cs=0 l=1 rev1=1 crypto rx=0x7fd91c033000 tx=0x7fd91c009000 comp rx=0 tx=0).stop 2024-09-17T18:33:31.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.736+0000 7fd92b050640 10 monclient: shutdown 2024-09-17T18:33:31.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.736+0000 7fd92b050640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:33:31.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.736+0000 7fd92b050640 1 -- 172.21.15.121:0/4167340058 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9240700f0 msgr2=0x7fd9241cd550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:31.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.736+0000 7fd92b050640 1 --2- 172.21.15.121:0/4167340058 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9240700f0 0x7fd9241cd550 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd91800ebe0 tx=0x7fd91800c520 comp rx=0 tx=0).stop 2024-09-17T18:33:31.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.736+0000 7fd9295ed640 1 -- 172.21.15.121:0/4167340058 reap_dead start 2024-09-17T18:33:31.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.736+0000 7fd9295ed640 1 -- 172.21.15.121:0/4167340058 reap_dead start 2024-09-17T18:33:31.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.736+0000 7fd92b050640 1 -- 172.21.15.121:0/4167340058 shutdown_connections 2024-09-17T18:33:31.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.736+0000 7fd92b050640 1 -- 172.21.15.121:0/4167340058 >> 172.21.15.121:0/4167340058 conn(0x7fd9240ab830 msgr2=0x7fd924133410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:33:31.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.736+0000 7fd92b050640 1 -- 172.21.15.121:0/4167340058 shutdown_connections 2024-09-17T18:33:31.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.736+0000 7fd92b050640 1 -- 172.21.15.121:0/4167340058 wait complete. 2024-09-17T18:33:31.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.736+0000 7fd92b050640 1 librados: shutdown 2024-09-17T18:33:31.761 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:33:31.761 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-17T18:33:31.762 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-17T18:33:31.762 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-17T18:33:31.762 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-17T18:33:31.762 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-17T18:33:31.762 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T18:33:31.811 INFO:tasks.ceph.mon.a:Stopped 2024-09-17T18:33:31.811 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-17T18:33:31.811 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:33:31.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.945+0000 7f85bdb17640 10 monclient: get_monmap_and_config 2024-09-17T18:33:31.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.945+0000 7f85bdb17640 10 monclient: build_initial_monmap 2024-09-17T18:33:31.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.945+0000 7f85bdb17640 10 monclient: monmap: 2024-09-17T18:33:31.947 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:33:31.948 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:31.948 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:33:31.946482+0000 2024-09-17T18:33:31.948 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:33:31.946482+0000 2024-09-17T18:33:31.948 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:33:31.948 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:31.948 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:33:31.948 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:33:31.948 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:33:31.948 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:33:31.948 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:33:31.948 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:31.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.946+0000 7f85bdb17640 1 Processor -- start 2024-09-17T18:33:31.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.946+0000 7f85bdb17640 1 -- start start 2024-09-17T18:33:31.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.946+0000 7f85bdb17640 10 monclient: init 2024-09-17T18:33:31.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.947+0000 7f85bdb17640 10 monclient: _reopen_session rank -1 2024-09-17T18:33:31.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.947+0000 7f85bdb17640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-17T18:33:31.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.947+0000 7f85bdb17640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85b80a33f0 0x7f85b809e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:31.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.947+0000 7f85bdb17640 10 monclient(hunting): picked mon.noname-b con 0x7f85b80a33f0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:33:31.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.947+0000 7f85bdb17640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f85b809da70 0x7f85b809ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:31.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.947+0000 7f85bdb17640 10 monclient(hunting): picked mon.noname-a con 0x7f85b809da70 addr v2:172.21.15.121:3300/0 2024-09-17T18:33:31.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.947+0000 7f85bdb17640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f85b809efd0 addr v1:172.21.15.173:6789/0 2024-09-17T18:33:31.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.947+0000 7f85bdb17640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85b8074b40 con 0x7f85b809efd0 2024-09-17T18:33:31.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.947+0000 7f85bdb17640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:31.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.947+0000 7f85bdb17640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f85b8074550 con 0x7f85b809da70 2024-09-17T18:33:31.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.947+0000 7f85bdb17640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:31.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.947+0000 7f85bdb17640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f85b80a37d0 con 0x7f85b80a33f0 2024-09-17T18:33:31.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.947+0000 7f85bdb17640 10 monclient(hunting): _renew_subs 2024-09-17T18:33:31.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.947+0000 7f85bdb17640 10 monclient(hunting): authenticate will time out at 1615.663221s 2024-09-17T18:33:31.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.947+0000 7f85aeffd640 1 -- >> v2:172.21.15.121:3300/0 conn(0x7f85b809da70 msgr2=0x7f85b809ea90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:33:31.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.947+0000 7f85aeffd640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f85b809da70 0x7f85b809ea90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:33:31.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.947+0000 7f85b77fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85b80a33f0 0x7f85b809e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:31.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.947+0000 7f85b77fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85b80a33f0 0x7f85b809e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:46838/0 (socket says 172.21.15.121:46838) 2024-09-17T18:33:31.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.947+0000 7f85b77fe640 1 -- 172.21.15.121:0/1966186322 learned_addr learned my addr 172.21.15.121:0/1966186322 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:33:31.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.947+0000 7f85b77fe640 10 monclient(hunting): get_auth_request con 0x7f85b80a33f0 auth_method 0 2024-09-17T18:33:31.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.947+0000 7f85b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:31.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.947+0000 7f85b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:31.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.947+0000 7f85b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:31.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.947+0000 7f85b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:31.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.947+0000 7f85b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:31.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.947+0000 7f85b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:31.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.947+0000 7f85b77fe640 10 monclient(hunting): handle_auth_done global_id 64409 payload 274 2024-09-17T18:33:31.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.947+0000 7f85b77fe640 1 -- 172.21.15.121:0/1966186322 >> v2:172.21.15.121:3300/0 conn(0x7f85b809da70 msgr2=0x7f85b809ea90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:33:31.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.947+0000 7f85b77fe640 1 --2- 172.21.15.121:0/1966186322 >> v2:172.21.15.121:3300/0 conn(0x7f85b809da70 0x7f85b809ea90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:31.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.947+0000 7f85b77fe640 1 -- 172.21.15.121:0/1966186322 >> v1:172.21.15.173:6789/0 conn(0x7f85b809efd0 legacy=0x7f85b813b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:33:31.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.947+0000 7f85b77fe640 10 monclient: _finish_hunting 0 2024-09-17T18:33:31.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.947+0000 7f85b77fe640 1 monclient: found mon.noname-b 2024-09-17T18:33:31.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.947+0000 7f85b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:33:31.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.947+0000 7f85b77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:33:31.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.947+0000 7f85b77fe640 1 -- 172.21.15.121:0/1966186322 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85b80a13c0 con 0x7f85b80a33f0 2024-09-17T18:33:31.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.950+0000 7f85b77fe640 1 -- 172.21.15.121:0/1966186322 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85b80a33f0 msgr2=0x7f85b809e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:31.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.950+0000 7f85b77fe640 1 --2- 172.21.15.121:0/1966186322 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85b80a33f0 0x7f85b809e550 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f85a8009870 tx=0x7f85a802a0d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c0b0b8d621b6fe1 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:31.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.950+0000 7f85b6ffd640 1 -- 172.21.15.121:0/1966186322 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85a8037070 con 0x7f85b80a33f0 2024-09-17T18:33:31.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.950+0000 7f85b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:31.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.950+0000 7f85b6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:33:31.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.950+0000 7f85b6ffd640 10 monclient: dump: 2024-09-17T18:33:31.960 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:31.960 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:31.960 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.960 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.960 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:31.960 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:31.960 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:31.960 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:31.960 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:31.960 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:31.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.950+0000 7f85b6ffd640 10 monclient: _finish_auth 0 2024-09-17T18:33:31.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.950+0000 7f85b6ffd640 10 monclient: _check_auth_tickets 2024-09-17T18:33:31.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.950+0000 7f85b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:33:31.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.950+0000 7f85bdb17640 5 monclient: authenticate success, global_id 64409 2024-09-17T18:33:31.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.950+0000 7f85bdb17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:33:31.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.950+0000 7f85b6ffd640 1 -- 172.21.15.121:0/1966186322 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85a8032640 con 0x7f85b80a33f0 2024-09-17T18:33:31.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.950+0000 7f85b6ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:33:31.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.950+0000 7f85bdb17640 10 monclient: get_monmap_and_config success 2024-09-17T18:33:31.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.950+0000 7f85b6ffd640 1 -- 172.21.15.121:0/1966186322 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85a8032a10 con 0x7f85b80a33f0 2024-09-17T18:33:31.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.950+0000 7f85bdb17640 4 set_mon_vals no callback set 2024-09-17T18:33:31.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.950+0000 7f85b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:31.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.950+0000 7f85b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:33:31.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.950+0000 7f85b6ffd640 10 monclient: dump: 2024-09-17T18:33:31.961 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:31.961 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:31.961 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.961 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.961 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:31.962 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:31.962 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:31.962 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:31.962 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:31.962 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:31.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.950+0000 7f85bdb17640 10 monclient: shutdown 2024-09-17T18:33:31.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.950+0000 7f85bdb17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:33:31.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.950+0000 7f85bdb17640 1 -- 172.21.15.121:0/1966186322 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85b80a33f0 msgr2=0x7f85b809e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:31.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.950+0000 7f85bdb17640 1 --2- 172.21.15.121:0/1966186322 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85b80a33f0 0x7f85b809e550 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f85a8009870 tx=0x7f85a802a0d0 comp rx=0 tx=0).stop 2024-09-17T18:33:31.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.951+0000 7f85bdb17640 1 -- 172.21.15.121:0/1966186322 shutdown_connections 2024-09-17T18:33:31.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.951+0000 7f85bdb17640 1 --2- 172.21.15.121:0/1966186322 >> v2:172.21.15.121:3300/0 conn(0x7f85b809da70 0x7f85b809ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:31.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.951+0000 7f85bdb17640 1 --2- 172.21.15.121:0/1966186322 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85b80a33f0 0x7f85b809e550 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:31.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.951+0000 7f85bdb17640 1 -- 172.21.15.121:0/1966186322 >> 172.21.15.121:0/1966186322 conn(0x7f85b80a3b50 msgr2=0x7f85b80a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:33:31.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.951+0000 7f85bdb17640 1 -- 172.21.15.121:0/1966186322 shutdown_connections 2024-09-17T18:33:31.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.951+0000 7f85bdb17640 1 -- 172.21.15.121:0/1966186322 wait complete. 2024-09-17T18:33:31.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.951+0000 7f85bdb17640 10 monclient: build_initial_monmap 2024-09-17T18:33:31.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.951+0000 7f85bdb17640 10 monclient: monmap: 2024-09-17T18:33:31.963 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:33:31.963 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:33:31.963 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:33:31.963 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:33:31.963 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:33:31.963 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:31.963 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:33:31.963 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:33:31.963 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:33:31.963 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:31.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.951+0000 7f85bdb17640 1 librados: starting msgr at 2024-09-17T18:33:31.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.951+0000 7f85bdb17640 1 librados: starting objecter 2024-09-17T18:33:31.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.951+0000 7f85bdb17640 1 Processor -- start 2024-09-17T18:33:31.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85bdb17640 1 -- start start 2024-09-17T18:33:31.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85bdb17640 1 librados: setting wanted keys 2024-09-17T18:33:31.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85bdb17640 1 librados: calling monclient init 2024-09-17T18:33:31.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85bdb17640 10 monclient: init 2024-09-17T18:33:31.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85bdb17640 10 monclient: _reopen_session rank -1 2024-09-17T18:33:31.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85bdb17640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T18:33:31.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85bdb17640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85b809efd0 0x7f85b81cb280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:31.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85bdb17640 10 monclient(hunting): picked mon.noname-c con 0x7f85b809efd0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:33:31.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85bdb17640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f85b80a33f0 0x7f85b8135910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:31.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85bdb17640 10 monclient(hunting): picked mon.noname-a con 0x7f85b80a33f0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:33:31.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85bdb17640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85b8135e50 0x7f85b811d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:31.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85bdb17640 10 monclient(hunting): picked mon.noname-b con 0x7f85b8135e50 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:33:31.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85bdb17640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:31.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85bdb17640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f85b813cf70 con 0x7f85b80a33f0 2024-09-17T18:33:31.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85bdb17640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:31.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85bdb17640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f85b809f7b0 con 0x7f85b8135e50 2024-09-17T18:33:31.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85bdb17640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:31.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85bdb17640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f85b813d0f0 con 0x7f85b809efd0 2024-09-17T18:33:31.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85bdb17640 10 monclient(hunting): _renew_subs 2024-09-17T18:33:31.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85aeffd640 1 -- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f85b80a33f0 msgr2=0x7f85b8135910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:33:31.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85bdb17640 10 monclient(hunting): authenticate will time out at 1615.668673s 2024-09-17T18:33:31.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85aeffd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f85b80a33f0 0x7f85b8135910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:33:31.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85b77fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85b809efd0 0x7f85b81cb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:31.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85b77fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85b809efd0 0x7f85b81cb280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:46842/0 (socket says 172.21.15.121:46842) 2024-09-17T18:33:31.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85b77fe640 1 -- 172.21.15.121:0/3381582428 learned_addr learned my addr 172.21.15.121:0/3381582428 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:33:31.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85b77fe640 10 monclient(hunting): get_auth_request con 0x7f85b809efd0 auth_method 0 2024-09-17T18:33:31.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:31.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:31.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:31.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:31.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:31.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:31.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85b7fff640 1 --2- 172.21.15.121:0/3381582428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85b8135e50 0x7f85b811d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:31.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85b77fe640 10 monclient(hunting): handle_auth_done global_id 64412 payload 824 2024-09-17T18:33:31.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85b77fe640 1 -- 172.21.15.121:0/3381582428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85b8135e50 msgr2=0x7f85b811d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:31.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85b77fe640 1 --2- 172.21.15.121:0/3381582428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85b8135e50 0x7f85b811d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:31.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85b77fe640 1 -- 172.21.15.121:0/3381582428 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f85b80a33f0 msgr2=0x7f85b8135910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:33:31.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85b77fe640 1 --2- 172.21.15.121:0/3381582428 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f85b80a33f0 0x7f85b8135910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:31.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85b77fe640 10 monclient: _finish_hunting 0 2024-09-17T18:33:31.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85b77fe640 1 monclient: found mon.noname-c 2024-09-17T18:33:31.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:33:31.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85b77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:33:31.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85b77fe640 1 -- 172.21.15.121:0/3381582428 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85b8120ff0 con 0x7f85b809efd0 2024-09-17T18:33:31.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85b7fff640 10 monclient: get_auth_request con 0x7f85b8135e50 auth_method 0 2024-09-17T18:33:31.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.952+0000 7f85b7fff640 1 --2- 172.21.15.121:0/3381582428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85b8135e50 0x7f85b811d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T18:33:31.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.955+0000 7f85b77fe640 1 -- 172.21.15.121:0/3381582428 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85b809efd0 msgr2=0x7f85b81cb280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:31.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.955+0000 7f85b77fe640 1 --2- 172.21.15.121:0/3381582428 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85b809efd0 0x7f85b81cb280 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f85a8000c00 tx=0x7f85a80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:31.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.955+0000 7f85b4ff9640 10 client.?.objecter ms_handle_connect 0x7f85b809efd0 2024-09-17T18:33:31.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.955+0000 7f85b4ff9640 10 client.?.objecter resend_mon_ops 2024-09-17T18:33:31.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.955+0000 7f85b4ff9640 1 -- 172.21.15.121:0/3381582428 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85a8041070 con 0x7f85b809efd0 2024-09-17T18:33:31.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.955+0000 7f85b4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:31.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.955+0000 7f85b77fe640 1 --2- 172.21.15.121:0/3381582428 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85b809efd0 0x7f85b81cb280 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f85a8000c00 tx=0x7f85a80046c0 comp rx=0 tx=0).queue_received will delay after 1.7266e+09 on 0x7f85a803caa0 config(0 keys) 2024-09-17T18:33:31.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.955+0000 7f85b4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:33:31.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.955+0000 7f85b4ff9640 10 monclient: dump: 2024-09-17T18:33:31.967 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:31.967 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:31.967 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.968 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.968 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:31.968 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:31.968 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:31.968 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:31.968 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:31.968 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:31.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.955+0000 7f85b4ff9640 10 monclient: _finish_auth 0 2024-09-17T18:33:31.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.955+0000 7f85b4ff9640 10 monclient: _check_auth_tickets 2024-09-17T18:33:31.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.955+0000 7f85b4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:33:31.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.955+0000 7f85b4ff9640 1 -- 172.21.15.121:0/3381582428 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85a803caa0 con 0x7f85b809efd0 2024-09-17T18:33:31.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.955+0000 7f85bdb17640 5 monclient: authenticate success, global_id 64412 2024-09-17T18:33:31.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.955+0000 7f85bdb17640 10 monclient: _renew_subs 2024-09-17T18:33:31.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.955+0000 7f85bdb17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:33:31.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.955+0000 7f85bdb17640 1 -- 172.21.15.121:0/3381582428 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85b81212d0 con 0x7f85b809efd0 2024-09-17T18:33:31.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.955+0000 7f85b4ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T18:33:31.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.955+0000 7f85b67fc640 4 set_mon_vals no callback set 2024-09-17T18:33:31.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.955+0000 7f85bdb17640 10 client.64412.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:33:31.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.955+0000 7f85bdb17640 10 monclient: _renew_subs 2024-09-17T18:33:31.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.955+0000 7f85bdb17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:33:31.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.955+0000 7f85bdb17640 1 -- 172.21.15.121:0/3381582428 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f85b811fad0 con 0x7f85b809efd0 2024-09-17T18:33:31.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.955+0000 7f85bdb17640 1 librados: init done 2024-09-17T18:33:31.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.956+0000 7f85b4ff9640 1 -- 172.21.15.121:0/3381582428 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85a8037040 con 0x7f85b809efd0 2024-09-17T18:33:31.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.956+0000 7f85b4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:31.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.956+0000 7f85bdb17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:33:31.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.956+0000 7f85b4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:33:31.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.956+0000 7f85b4ff9640 10 monclient: dump: 2024-09-17T18:33:31.969 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:31.970 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:31.970 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.970 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:31.970 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:31.970 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:31.970 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:31.970 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:31.970 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:31.970 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:31.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.956+0000 7f85b4ff9640 1 -- 172.21.15.121:0/3381582428 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f85a8044070 con 0x7f85b809efd0 2024-09-17T18:33:31.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.956+0000 7f85bdb17640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:33:31.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.956+0000 7f85bdb17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:33:31.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.956+0000 7f85bdb17640 1 -- 172.21.15.121:0/3381582428 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85b809e550 con 0x7f85b809efd0 2024-09-17T18:33:31.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.956+0000 7f85b4ff9640 1 --2- 172.21.15.121:0/3381582428 >> v2:172.21.15.173:6812/2423786037 conn(0x7f8598040120 0x7f85980425e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:31.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.956+0000 7f85aeffd640 1 --2- 172.21.15.121:0/3381582428 >> v2:172.21.15.173:6812/2423786037 conn(0x7f8598040120 0x7f85980425e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:31.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.957+0000 7f85aeffd640 10 monclient: get_auth_request con 0x7f8598040120 auth_method 0 2024-09-17T18:33:31.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.959+0000 7f85aeffd640 1 --2- 172.21.15.121:0/3381582428 >> v2:172.21.15.173:6812/2423786037 conn(0x7f8598040120 0x7f85980425e0 secure :-1 s=READY pgs=4057 cs=0 l=1 rev1=1 crypto rx=0x7f85a0006fd0 tx=0x7f85a0006ce0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:31.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.959+0000 7f85b4ff9640 10 client.64412.objecter ms_handle_connect 0x7f8598040120 2024-09-17T18:33:31.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:31.961+0000 7f85b77fe640 1 --2- 172.21.15.121:0/3381582428 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85b809efd0 0x7f85b81cb280 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f85a8000c00 tx=0x7f85a80046c0 comp rx=0 tx=0).queue_received will delay after 1.7266e+09 on 0x7f85a803be10 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) 2024-09-17T18:33:32.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:32.579+0000 7f85b4ff9640 1 -- 172.21.15.121:0/3381582428 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(563..563 src has 247..563) ==== 5516+0+0 (secure 0 0 0) 0x7f85a8074180 con 0x7f85b809efd0 2024-09-17T18:33:32.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:32.579+0000 7f85b4ff9640 10 client.64412.objecter ms_dispatch 0x7f85b8000b90 osd_map(563..563 src has 247..563) 2024-09-17T18:33:32.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:32.579+0000 7f85b4ff9640 3 client.64412.objecter handle_osd_map got epochs [563,563] > 0 2024-09-17T18:33:32.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:32.579+0000 7f85b4ff9640 3 client.64412.objecter handle_osd_map decoding full epoch 563 2024-09-17T18:33:32.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:32.579+0000 7f85b4ff9640 20 client.64412.objecter dump_active .. 0 homeless 2024-09-17T18:33:32.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:32.789+0000 7f85b4ff9640 1 -- 172.21.15.121:0/3381582428 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f85a803be10 con 0x7f85b809efd0 2024-09-17T18:33:32.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:32.789+0000 7f85b4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:33:32.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:32.789+0000 7f85b4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:33:33.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:33.037+0000 7f85bdb17640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:33:33.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:33.037+0000 7f85bdb17640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:33:33.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:33.037+0000 7f85bdb17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:33:33.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:33.037+0000 7f85bdb17640 1 -- 172.21.15.121:0/3381582428 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f85b811f850 con 0x7f85b809efd0 2024-09-17T18:33:33.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:33.037+0000 7f85b4ff9640 1 -- 172.21.15.121:0/3381582428 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f85b811f850 con 0x7f85b809efd0 2024-09-17T18:33:33.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:33.037+0000 7f85b4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:33:33.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:33.037+0000 7f85b4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:33:33.040 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":54,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:33:33.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:33.039+0000 7f85bdb17640 10 librados: watch_flush enter 2024-09-17T18:33:33.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:33.039+0000 7f85bdb17640 10 librados: watch_flush exit 2024-09-17T18:33:33.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:33.039+0000 7f85bdb17640 20 client.64412.objecter shutdown clearing up homeless session... 2024-09-17T18:33:33.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:33.039+0000 7f85bdb17640 10 client.64412.objecter successfully canceled tick 2024-09-17T18:33:33.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:33.039+0000 7f85bdb17640 1 -- 172.21.15.121:0/3381582428 >> v2:172.21.15.173:6812/2423786037 conn(0x7f8598040120 msgr2=0x7f85980425e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:33.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:33.039+0000 7f85bdb17640 1 --2- 172.21.15.121:0/3381582428 >> v2:172.21.15.173:6812/2423786037 conn(0x7f8598040120 0x7f85980425e0 secure :-1 s=READY pgs=4057 cs=0 l=1 rev1=1 crypto rx=0x7f85a0006fd0 tx=0x7f85a0006ce0 comp rx=0 tx=0).stop 2024-09-17T18:33:33.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:33.040+0000 7f85bdb17640 10 monclient: shutdown 2024-09-17T18:33:33.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:33.040+0000 7f85bdb17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:33:33.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:33.040+0000 7f85bdb17640 1 -- 172.21.15.121:0/3381582428 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85b809efd0 msgr2=0x7f85b81cb280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:33.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:33.040+0000 7f85bdb17640 1 --2- 172.21.15.121:0/3381582428 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85b809efd0 0x7f85b81cb280 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f85a8000c00 tx=0x7f85a80046c0 comp rx=0 tx=0).stop 2024-09-17T18:33:33.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:33.040+0000 7f85bdb17640 1 -- 172.21.15.121:0/3381582428 shutdown_connections 2024-09-17T18:33:33.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:33.040+0000 7f85bdb17640 1 --2- 172.21.15.121:0/3381582428 >> v2:172.21.15.173:6812/2423786037 conn(0x7f8598040120 0x7f85980425e0 unknown :-1 s=CLOSED pgs=4057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:33.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:33.040+0000 7f85bdb17640 1 --2- 172.21.15.121:0/3381582428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85b8135e50 0x7f85b811d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:33.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:33.040+0000 7f85bdb17640 1 --2- 172.21.15.121:0/3381582428 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f85b80a33f0 0x7f85b8135910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:33.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:33.040+0000 7f85bdb17640 1 --2- 172.21.15.121:0/3381582428 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85b809efd0 0x7f85b81cb280 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:33.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:33.040+0000 7f85bdb17640 1 -- 172.21.15.121:0/3381582428 >> 172.21.15.121:0/3381582428 conn(0x7f85b80a3b50 msgr2=0x7f85b80738e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:33:33.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:33.040+0000 7f85bdb17640 1 -- 172.21.15.121:0/3381582428 shutdown_connections 2024-09-17T18:33:33.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:33.040+0000 7f85bdb17640 1 -- 172.21.15.121:0/3381582428 wait complete. 2024-09-17T18:33:33.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:33.040+0000 7f85bdb17640 1 librados: shutdown 2024-09-17T18:33:36.053 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:33:36.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.159+0000 7f64d1472640 10 monclient: get_monmap_and_config 2024-09-17T18:33:36.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.159+0000 7f64d1472640 10 monclient: build_initial_monmap 2024-09-17T18:33:36.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.159+0000 7f64d1472640 10 monclient: monmap: 2024-09-17T18:33:36.162 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:33:36.162 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:36.162 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:33:36.160445+0000 2024-09-17T18:33:36.162 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:33:36.160445+0000 2024-09-17T18:33:36.162 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:33:36.162 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:36.162 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:33:36.162 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:33:36.162 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:33:36.162 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:33:36.163 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:33:36.163 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:36.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.160+0000 7f64d1472640 1 Processor -- start 2024-09-17T18:33:36.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.160+0000 7f64d1472640 1 -- start start 2024-09-17T18:33:36.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.160+0000 7f64d1472640 10 monclient: init 2024-09-17T18:33:36.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.160+0000 7f64d1472640 10 monclient: _reopen_session rank -1 2024-09-17T18:33:36.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.160+0000 7f64d1472640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-17T18:33:36.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.160+0000 7f64d1472640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f64cc0a5ca0 0x7f64cc0a6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:36.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.160+0000 7f64d1472640 10 monclient(hunting): picked mon.noname-a con 0x7f64cc0a5ca0 addr v2:172.21.15.121:3300/0 2024-09-17T18:33:36.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.160+0000 7f64d1472640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f64cc0a65c0 addr v1:172.21.15.173:6789/0 2024-09-17T18:33:36.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.161+0000 7f64d1472640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f64cc139170 0x7f64cc140cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:36.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.161+0000 7f64d1472640 10 monclient(hunting): picked mon.noname-c con 0x7f64cc139170 addr v2:172.21.15.173:3300/0 2024-09-17T18:33:36.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.161+0000 7f64d1472640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64cc0a1f70 con 0x7f64cc0a65c0 2024-09-17T18:33:36.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.161+0000 7f64d1472640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:36.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.161+0000 7f64d1472640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f64cc0a6e40 con 0x7f64cc0a5ca0 2024-09-17T18:33:36.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.161+0000 7f64d1472640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:36.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.161+0000 7f64d1472640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f64cc0a54f0 con 0x7f64cc139170 2024-09-17T18:33:36.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.161+0000 7f64d1472640 10 monclient(hunting): _renew_subs 2024-09-17T18:33:36.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.161+0000 7f64d1472640 10 monclient(hunting): authenticate will time out at 1619.877049s 2024-09-17T18:33:36.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.161+0000 7f64caffd640 1 -- >> v2:172.21.15.121:3300/0 conn(0x7f64cc0a5ca0 msgr2=0x7f64cc0a6080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:33:36.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.161+0000 7f64caffd640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f64cc0a5ca0 0x7f64cc0a6080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:33:36.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.161+0000 7f64cb7fe640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f64cc139170 0x7f64cc140cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:36.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.161+0000 7f64ca7fc640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f64cc0a65c0 0x7f64cc138a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.121:44930/0 (socket says 172.21.15.121:44930) 2024-09-17T18:33:36.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.161+0000 7f64ca7fc640 1 -- 172.21.15.121:0/3619465950 learned_addr learned my addr 172.21.15.121:0/3619465950 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:33:36.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.161+0000 7f64cb7fe640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f64cc139170 0x7f64cc140cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.121:57402/0 (socket says 172.21.15.121:57402) 2024-09-17T18:33:36.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.161+0000 7f64cb7fe640 10 monclient(hunting): get_auth_request con 0x7f64cc139170 auth_method 0 2024-09-17T18:33:36.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.161+0000 7f64cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:36.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.161+0000 7f64cb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:36.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.161+0000 7f64cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:36.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.161+0000 7f64cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:36.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.161+0000 7f64cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:36.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.161+0000 7f64cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:36.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.161+0000 7f64cb7fe640 10 monclient(hunting): handle_auth_done global_id 84205 payload 274 2024-09-17T18:33:36.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.161+0000 7f64cb7fe640 1 -- 172.21.15.121:0/3619465950 >> v2:172.21.15.121:3300/0 conn(0x7f64cc0a5ca0 msgr2=0x7f64cc0a6080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:33:36.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.161+0000 7f64cb7fe640 1 --2- 172.21.15.121:0/3619465950 >> v2:172.21.15.121:3300/0 conn(0x7f64cc0a5ca0 0x7f64cc0a6080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:36.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.161+0000 7f64cb7fe640 1 -- 172.21.15.121:0/3619465950 >> v1:172.21.15.173:6789/0 conn(0x7f64cc0a65c0 legacy=0x7f64cc138a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:33:36.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.161+0000 7f64cb7fe640 10 monclient: _finish_hunting 0 2024-09-17T18:33:36.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.161+0000 7f64cb7fe640 1 monclient: found mon.noname-c 2024-09-17T18:33:36.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.161+0000 7f64cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:33:36.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.161+0000 7f64cb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.173:3300/0 2024-09-17T18:33:36.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.161+0000 7f64cb7fe640 1 -- 172.21.15.121:0/3619465950 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64cc0a3250 con 0x7f64cc139170 2024-09-17T18:33:36.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.164+0000 7f64cb7fe640 1 -- 172.21.15.121:0/3619465950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64cc139170 msgr2=0x7f64cc140cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:36.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.164+0000 7f64cb7fe640 1 --2- 172.21.15.121:0/3619465950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64cc139170 0x7f64cc140cc0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f64cc0a3110 tx=0x7f64bc02a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f063043aa7ccf11f server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:36.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.164+0000 7f64c9ffb640 1 -- 172.21.15.121:0/3619465950 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64bc037070 con 0x7f64cc139170 2024-09-17T18:33:36.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.164+0000 7f64c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:36.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.164+0000 7f64c9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:33:36.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.164+0000 7f64c9ffb640 10 monclient: dump: 2024-09-17T18:33:36.175 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:36.175 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:36.175 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:36.175 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:36.175 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:36.175 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:36.175 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:36.175 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:36.175 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:36.175 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:36.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.164+0000 7f64c9ffb640 10 monclient: _finish_auth 0 2024-09-17T18:33:36.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.164+0000 7f64c9ffb640 10 monclient: _check_auth_tickets 2024-09-17T18:33:36.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.164+0000 7f64c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:33:36.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.164+0000 7f64c9ffb640 1 -- 172.21.15.121:0/3619465950 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64bc032640 con 0x7f64cc139170 2024-09-17T18:33:36.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.164+0000 7f64d1472640 5 monclient: authenticate success, global_id 84205 2024-09-17T18:33:36.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.164+0000 7f64d1472640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:33:36.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.164+0000 7f64c9ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:33:36.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.164+0000 7f64d1472640 10 monclient: get_monmap_and_config success 2024-09-17T18:33:36.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.164+0000 7f64d1472640 4 set_mon_vals no callback set 2024-09-17T18:33:36.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.164+0000 7f64c9ffb640 1 -- 172.21.15.121:0/3619465950 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64bc032a10 con 0x7f64cc139170 2024-09-17T18:33:36.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.164+0000 7f64c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:36.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.164+0000 7f64c9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:33:36.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.164+0000 7f64c9ffb640 10 monclient: dump: 2024-09-17T18:33:36.176 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:36.176 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:36.176 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:36.176 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:36.177 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:36.177 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:36.177 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:36.177 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:36.177 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:36.177 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:36.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.165+0000 7f64d1472640 10 monclient: shutdown 2024-09-17T18:33:36.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.165+0000 7f64d1472640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:33:36.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.165+0000 7f64d1472640 1 -- 172.21.15.121:0/3619465950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64cc139170 msgr2=0x7f64cc140cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:36.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.165+0000 7f64d1472640 1 --2- 172.21.15.121:0/3619465950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64cc139170 0x7f64cc140cc0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f64cc0a3110 tx=0x7f64bc02a000 comp rx=0 tx=0).stop 2024-09-17T18:33:36.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.165+0000 7f64d1472640 1 -- 172.21.15.121:0/3619465950 shutdown_connections 2024-09-17T18:33:36.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.165+0000 7f64d1472640 1 --2- 172.21.15.121:0/3619465950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64cc139170 0x7f64cc140cc0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:36.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.165+0000 7f64d1472640 1 --2- 172.21.15.121:0/3619465950 >> v2:172.21.15.121:3300/0 conn(0x7f64cc0a5ca0 0x7f64cc0a6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:36.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.165+0000 7f64d1472640 1 -- 172.21.15.121:0/3619465950 >> 172.21.15.121:0/3619465950 conn(0x7f64cc06d850 msgr2=0x7f64cc06dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:33:36.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.165+0000 7f64d1472640 1 -- 172.21.15.121:0/3619465950 shutdown_connections 2024-09-17T18:33:36.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.165+0000 7f64d1472640 1 -- 172.21.15.121:0/3619465950 wait complete. 2024-09-17T18:33:36.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.166+0000 7f64d1472640 10 monclient: build_initial_monmap 2024-09-17T18:33:36.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.166+0000 7f64d1472640 10 monclient: monmap: 2024-09-17T18:33:36.178 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:33:36.178 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:33:36.178 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:33:36.178 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:33:36.178 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:33:36.178 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:36.178 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:33:36.178 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:33:36.178 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:33:36.178 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:36.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.166+0000 7f64d1472640 1 librados: starting msgr at 2024-09-17T18:33:36.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.166+0000 7f64d1472640 1 librados: starting objecter 2024-09-17T18:33:36.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.166+0000 7f64d1472640 1 Processor -- start 2024-09-17T18:33:36.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.166+0000 7f64d1472640 1 -- start start 2024-09-17T18:33:36.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.166+0000 7f64d1472640 1 librados: setting wanted keys 2024-09-17T18:33:36.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.166+0000 7f64d1472640 1 librados: calling monclient init 2024-09-17T18:33:36.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.166+0000 7f64d1472640 10 monclient: init 2024-09-17T18:33:36.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64d1472640 10 monclient: _reopen_session rank -1 2024-09-17T18:33:36.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64d1472640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T18:33:36.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64d1472640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64cc0a5ca0 0x7f64cc1cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:36.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64d1472640 10 monclient(hunting): picked mon.noname-c con 0x7f64cc0a5ca0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:33:36.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64d1472640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64cc0a65c0 0x7f64cc1ce230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:36.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64d1472640 10 monclient(hunting): picked mon.noname-b con 0x7f64cc0a65c0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:33:36.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64d1472640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64cc139170 0x7f64cc125da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:36.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64d1472640 10 monclient(hunting): picked mon.noname-a con 0x7f64cc139170 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:33:36.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64d1472640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:36.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64d1472640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f64cc0a69d0 con 0x7f64cc139170 2024-09-17T18:33:36.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64d1472640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:36.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64d1472640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f64cc0a56d0 con 0x7f64cc0a65c0 2024-09-17T18:33:36.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64d1472640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:36.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64d1472640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f64cc070000 con 0x7f64cc0a5ca0 2024-09-17T18:33:36.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64d1472640 10 monclient(hunting): _renew_subs 2024-09-17T18:33:36.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64d1472640 10 monclient(hunting): authenticate will time out at 1619.883233s 2024-09-17T18:33:36.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64cb7fe640 1 -- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64cc139170 msgr2=0x7f64cc125da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:33:36.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64cb7fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64cc139170 0x7f64cc125da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:33:36.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64caffd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64cc0a5ca0 0x7f64cc1cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:36.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64caffd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64cc0a5ca0 0x7f64cc1cdcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:46844/0 (socket says 172.21.15.121:46844) 2024-09-17T18:33:36.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64caffd640 1 -- 172.21.15.121:0/122161886 learned_addr learned my addr 172.21.15.121:0/122161886 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:33:36.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64caffd640 10 monclient(hunting): get_auth_request con 0x7f64cc0a5ca0 auth_method 0 2024-09-17T18:33:36.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:36.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64caffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:36.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:36.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64caffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:36.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64caffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:36.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64caffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:36.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64ca7fc640 1 --2- 172.21.15.121:0/122161886 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64cc0a65c0 0x7f64cc1ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:36.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64ca7fc640 10 monclient(hunting): get_auth_request con 0x7f64cc0a65c0 auth_method 0 2024-09-17T18:33:36.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64ca7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:36.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64ca7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:36.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:36.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64caffd640 10 monclient(hunting): handle_auth_done global_id 64415 payload 824 2024-09-17T18:33:36.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64caffd640 1 -- 172.21.15.121:0/122161886 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64cc0a65c0 msgr2=0x7f64cc1ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:36.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64caffd640 1 --2- 172.21.15.121:0/122161886 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64cc0a65c0 0x7f64cc1ce230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:36.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64caffd640 1 -- 172.21.15.121:0/122161886 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64cc139170 msgr2=0x7f64cc125da0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:33:36.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64caffd640 1 --2- 172.21.15.121:0/122161886 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64cc139170 0x7f64cc125da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:36.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64caffd640 10 monclient: _finish_hunting 0 2024-09-17T18:33:36.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64caffd640 1 monclient: found mon.noname-c 2024-09-17T18:33:36.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:33:36.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64caffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:33:36.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64caffd640 1 -- 172.21.15.121:0/122161886 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64cc0a2490 con 0x7f64cc0a5ca0 2024-09-17T18:33:36.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.167+0000 7f64ca7fc640 1 --2- 172.21.15.121:0/122161886 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64cc0a65c0 0x7f64cc1ce230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T18:33:36.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.169+0000 7f64caffd640 1 -- 172.21.15.121:0/122161886 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64cc0a5ca0 msgr2=0x7f64cc1cdcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:36.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.169+0000 7f64caffd640 1 --2- 172.21.15.121:0/122161886 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64cc0a5ca0 0x7f64cc1cdcf0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f64c400ba10 tx=0x7f64c400bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:36.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.169+0000 7f64c37fe640 10 client.?.objecter ms_handle_connect 0x7f64cc0a5ca0 2024-09-17T18:33:36.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.169+0000 7f64c37fe640 10 client.?.objecter resend_mon_ops 2024-09-17T18:33:36.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.170+0000 7f64c37fe640 1 -- 172.21.15.121:0/122161886 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c400cde0 con 0x7f64cc0a5ca0 2024-09-17T18:33:36.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.170+0000 7f64c37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:36.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.170+0000 7f64c37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:33:36.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.170+0000 7f64c37fe640 10 monclient: dump: 2024-09-17T18:33:36.183 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:36.183 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:36.183 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:36.183 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:36.183 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:36.183 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:36.183 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:36.183 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:36.183 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:36.183 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:36.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.170+0000 7f64c37fe640 10 monclient: _finish_auth 0 2024-09-17T18:33:36.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.170+0000 7f64c37fe640 10 monclient: _check_auth_tickets 2024-09-17T18:33:36.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.170+0000 7f64c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:33:36.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.170+0000 7f64d1472640 5 monclient: authenticate success, global_id 64415 2024-09-17T18:33:36.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.170+0000 7f64d1472640 10 monclient: _renew_subs 2024-09-17T18:33:36.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.170+0000 7f64c37fe640 1 -- 172.21.15.121:0/122161886 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64c4010070 con 0x7f64cc0a5ca0 2024-09-17T18:33:36.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.170+0000 7f64d1472640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:33:36.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.170+0000 7f64d1472640 1 -- 172.21.15.121:0/122161886 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64cc0a8840 con 0x7f64cc0a5ca0 2024-09-17T18:33:36.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.170+0000 7f64c37fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:33:36.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.170+0000 7f64c37fe640 1 -- 172.21.15.121:0/122161886 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c40156b0 con 0x7f64cc0a5ca0 2024-09-17T18:33:36.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.170+0000 7f64c37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:36.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.170+0000 7f64c37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:33:36.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.170+0000 7f64c37fe640 10 monclient: dump: 2024-09-17T18:33:36.185 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:36.185 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:36.185 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:36.185 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:36.185 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:36.185 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:36.185 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:36.185 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:36.185 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:36.185 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:36.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.170+0000 7f64c97fa640 4 set_mon_vals no callback set 2024-09-17T18:33:36.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.170+0000 7f64d1472640 10 client.64415.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:33:36.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.170+0000 7f64d1472640 10 monclient: _renew_subs 2024-09-17T18:33:36.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.170+0000 7f64d1472640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:33:36.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.170+0000 7f64d1472640 1 -- 172.21.15.121:0/122161886 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f64cc1ce770 con 0x7f64cc0a5ca0 2024-09-17T18:33:36.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.170+0000 7f64d1472640 1 librados: init done 2024-09-17T18:33:36.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.171+0000 7f64c37fe640 1 -- 172.21.15.121:0/122161886 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f64c401e3f0 con 0x7f64cc0a5ca0 2024-09-17T18:33:36.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.171+0000 7f64d1472640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:33:36.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.171+0000 7f64c37fe640 1 --2- 172.21.15.121:0/122161886 >> v2:172.21.15.173:6812/2423786037 conn(0x7f64ac040150 0x7f64ac042610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:36.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.171+0000 7f64d1472640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:33:36.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.171+0000 7f64d1472640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:33:36.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.171+0000 7f64d1472640 1 -- 172.21.15.121:0/122161886 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64cc0a3c50 con 0x7f64cc0a5ca0 2024-09-17T18:33:36.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.171+0000 7f64c37fe640 1 -- 172.21.15.121:0/122161886 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(563..563 src has 247..563) ==== 5516+0+0 (secure 0 0 0) 0x7f64c40539e0 con 0x7f64cc0a5ca0 2024-09-17T18:33:36.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.171+0000 7f64c37fe640 10 client.64415.objecter ms_dispatch 0x7f64cc000b90 osd_map(563..563 src has 247..563) 2024-09-17T18:33:36.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.171+0000 7f64c37fe640 3 client.64415.objecter handle_osd_map got epochs [563,563] > 0 2024-09-17T18:33:36.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.171+0000 7f64c37fe640 3 client.64415.objecter handle_osd_map decoding full epoch 563 2024-09-17T18:33:36.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.171+0000 7f64c37fe640 20 client.64415.objecter dump_active .. 0 homeless 2024-09-17T18:33:36.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.171+0000 7f64ca7fc640 1 --2- 172.21.15.121:0/122161886 >> v2:172.21.15.173:6812/2423786037 conn(0x7f64ac040150 0x7f64ac042610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:36.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.171+0000 7f64ca7fc640 10 monclient: get_auth_request con 0x7f64ac040150 auth_method 0 2024-09-17T18:33:36.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.174+0000 7f64c37fe640 1 -- 172.21.15.121:0/122161886 DispatchQueue::entry inject delay of 0.002000 2024-09-17T18:33:36.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.174+0000 7f64ca7fc640 1 --2- 172.21.15.121:0/122161886 >> v2:172.21.15.173:6812/2423786037 conn(0x7f64ac040150 0x7f64ac042610 secure :-1 s=READY pgs=4058 cs=0 l=1 rev1=1 crypto rx=0x7f64cc128ce0 tx=0x7f64b4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:36.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.176+0000 7f64c37fe640 10 client.64415.objecter ms_handle_connect 0x7f64ac040150 2024-09-17T18:33:36.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.176+0000 7f64c37fe640 1 -- 172.21.15.121:0/122161886 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f64c401d3a0 con 0x7f64cc0a5ca0 2024-09-17T18:33:36.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.176+0000 7f64c37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:33:36.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.176+0000 7f64c37fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:33:36.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.419+0000 7f64d1472640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:33:36.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.419+0000 7f64d1472640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:33:36.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.419+0000 7f64d1472640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:33:36.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.419+0000 7f64d1472640 1 -- 172.21.15.121:0/122161886 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f64cc127810 con 0x7f64cc0a5ca0 2024-09-17T18:33:36.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.420+0000 7f64c37fe640 1 -- 172.21.15.121:0/122161886 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f64c401eb10 con 0x7f64cc0a5ca0 2024-09-17T18:33:36.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.420+0000 7f64c37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:33:36.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.420+0000 7f64c37fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:33:36.422 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":54,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:33:36.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.422+0000 7f64d1472640 10 librados: watch_flush enter 2024-09-17T18:33:36.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.422+0000 7f64d1472640 10 librados: watch_flush exit 2024-09-17T18:33:36.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.422+0000 7f64d1472640 20 client.64415.objecter shutdown clearing up homeless session... 2024-09-17T18:33:36.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.422+0000 7f64d1472640 10 client.64415.objecter successfully canceled tick 2024-09-17T18:33:36.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.422+0000 7f64d1472640 1 -- 172.21.15.121:0/122161886 >> v2:172.21.15.173:6812/2423786037 conn(0x7f64ac040150 msgr2=0x7f64ac042610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:36.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.422+0000 7f64d1472640 1 --2- 172.21.15.121:0/122161886 >> v2:172.21.15.173:6812/2423786037 conn(0x7f64ac040150 0x7f64ac042610 secure :-1 s=READY pgs=4058 cs=0 l=1 rev1=1 crypto rx=0x7f64cc128ce0 tx=0x7f64b4009040 comp rx=0 tx=0).stop 2024-09-17T18:33:36.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.422+0000 7f64d1472640 10 monclient: shutdown 2024-09-17T18:33:36.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.422+0000 7f64d1472640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:33:36.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.422+0000 7f64d1472640 1 -- 172.21.15.121:0/122161886 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64cc0a5ca0 msgr2=0x7f64cc1cdcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:36.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.422+0000 7f64d1472640 1 --2- 172.21.15.121:0/122161886 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64cc0a5ca0 0x7f64cc1cdcf0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f64c400ba10 tx=0x7f64c400bed0 comp rx=0 tx=0).stop 2024-09-17T18:33:36.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.422+0000 7f64d1472640 1 -- 172.21.15.121:0/122161886 shutdown_connections 2024-09-17T18:33:36.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.422+0000 7f64d1472640 1 --2- 172.21.15.121:0/122161886 >> v2:172.21.15.173:6812/2423786037 conn(0x7f64ac040150 0x7f64ac042610 unknown :-1 s=CLOSED pgs=4058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:36.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.422+0000 7f64d1472640 1 --2- 172.21.15.121:0/122161886 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64cc0a65c0 0x7f64cc1ce230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:36.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.422+0000 7f64d1472640 1 --2- 172.21.15.121:0/122161886 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64cc139170 0x7f64cc125da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:36.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.422+0000 7f64d1472640 1 --2- 172.21.15.121:0/122161886 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64cc0a5ca0 0x7f64cc1cdcf0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:36.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.422+0000 7f64d1472640 1 -- 172.21.15.121:0/122161886 >> 172.21.15.121:0/122161886 conn(0x7f64cc06d850 msgr2=0x7f64cc09ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:33:36.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.422+0000 7f64d1472640 1 -- 172.21.15.121:0/122161886 shutdown_connections 2024-09-17T18:33:36.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.422+0000 7f64d1472640 1 -- 172.21.15.121:0/122161886 wait complete. 2024-09-17T18:33:36.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:36.422+0000 7f64d1472640 1 librados: shutdown 2024-09-17T18:33:39.435 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:33:39.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.542+0000 7f8415dd5640 10 monclient: get_monmap_and_config 2024-09-17T18:33:39.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.542+0000 7f8415dd5640 10 monclient: build_initial_monmap 2024-09-17T18:33:39.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.542+0000 7f8415dd5640 10 monclient: monmap: 2024-09-17T18:33:39.544 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:33:39.544 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:39.544 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:33:39.543013+0000 2024-09-17T18:33:39.544 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:33:39.543013+0000 2024-09-17T18:33:39.544 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:33:39.544 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:39.545 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:33:39.545 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:33:39.545 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:33:39.545 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:33:39.545 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:33:39.545 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:39.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.543+0000 7f8415dd5640 1 Processor -- start 2024-09-17T18:33:39.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.543+0000 7f8415dd5640 1 -- start start 2024-09-17T18:33:39.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.543+0000 7f8415dd5640 10 monclient: init 2024-09-17T18:33:39.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.543+0000 7f8415dd5640 10 monclient: _reopen_session rank -1 2024-09-17T18:33:39.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.543+0000 7f8415dd5640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-17T18:33:39.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.543+0000 7f8415dd5640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f84100738a0 0x7f8410138a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:39.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.543+0000 7f8415dd5640 10 monclient(hunting): picked mon.noname-c con 0x7f84100738a0 addr v2:172.21.15.173:3300/0 2024-09-17T18:33:39.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.543+0000 7f8415dd5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8410073c80 0x7f84100740c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:39.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.543+0000 7f8415dd5640 10 monclient(hunting): picked mon.noname-b con 0x7f8410073c80 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:33:39.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.543+0000 7f8415dd5640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f84100a0e70 0x7f84100a12d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:39.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.543+0000 7f8415dd5640 10 monclient(hunting): picked mon.noname-a con 0x7f84100a0e70 addr v2:172.21.15.121:3300/0 2024-09-17T18:33:39.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.543+0000 7f8415dd5640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:39.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.543+0000 7f8415dd5640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f8410070350 con 0x7f84100a0e70 2024-09-17T18:33:39.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.543+0000 7f8415dd5640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:39.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.543+0000 7f8415dd5640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f841006fd90 con 0x7f84100738a0 2024-09-17T18:33:39.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.543+0000 7f8415dd5640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:39.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.543+0000 7f8415dd5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8410070070 con 0x7f8410073c80 2024-09-17T18:33:39.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.543+0000 7f8415dd5640 10 monclient(hunting): _renew_subs 2024-09-17T18:33:39.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.543+0000 7f8415dd5640 10 monclient(hunting): authenticate will time out at 1623.259869s 2024-09-17T18:33:39.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.543+0000 7f840ffff640 1 -- >> v2:172.21.15.121:3300/0 conn(0x7f84100a0e70 msgr2=0x7f84100a12d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:33:39.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.543+0000 7f840ffff640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f84100a0e70 0x7f84100a12d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:33:39.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.543+0000 7f840effd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8410073c80 0x7f84100740c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:39.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.544+0000 7f840effd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8410073c80 0x7f84100740c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:57280/0 (socket says 172.21.15.121:57280) 2024-09-17T18:33:39.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.544+0000 7f840effd640 1 -- 172.21.15.121:0/1858194936 learned_addr learned my addr 172.21.15.121:0/1858194936 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:33:39.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.544+0000 7f840effd640 10 monclient(hunting): get_auth_request con 0x7f8410073c80 auth_method 0 2024-09-17T18:33:39.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.544+0000 7f840effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:39.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.544+0000 7f840effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:39.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.544+0000 7f840effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:39.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.544+0000 7f840effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:39.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.544+0000 7f840effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:39.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.544+0000 7f840effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:39.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.544+0000 7f840f7fe640 1 --2- 172.21.15.121:0/1858194936 >> v2:172.21.15.173:3300/0 conn(0x7f84100738a0 0x7f8410138a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:39.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.544+0000 7f840f7fe640 10 monclient(hunting): get_auth_request con 0x7f84100738a0 auth_method 0 2024-09-17T18:33:39.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.544+0000 7f840f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:39.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.544+0000 7f840f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:39.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.544+0000 7f840f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:39.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.544+0000 7f840effd640 10 monclient(hunting): handle_auth_done global_id 64418 payload 274 2024-09-17T18:33:39.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.544+0000 7f840effd640 1 -- 172.21.15.121:0/1858194936 >> v2:172.21.15.173:3300/0 conn(0x7f84100738a0 msgr2=0x7f8410138a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:33:39.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.544+0000 7f840effd640 1 --2- 172.21.15.121:0/1858194936 >> v2:172.21.15.173:3300/0 conn(0x7f84100738a0 0x7f8410138a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:39.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.544+0000 7f840effd640 1 -- 172.21.15.121:0/1858194936 >> v2:172.21.15.121:3300/0 conn(0x7f84100a0e70 msgr2=0x7f84100a12d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:33:39.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.544+0000 7f840effd640 1 --2- 172.21.15.121:0/1858194936 >> v2:172.21.15.121:3300/0 conn(0x7f84100a0e70 0x7f84100a12d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:39.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.544+0000 7f840effd640 10 monclient: _finish_hunting 0 2024-09-17T18:33:39.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.544+0000 7f840effd640 1 monclient: found mon.noname-b 2024-09-17T18:33:39.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.544+0000 7f840effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:33:39.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.544+0000 7f840effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:33:39.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.544+0000 7f840effd640 1 -- 172.21.15.121:0/1858194936 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84100a4020 con 0x7f8410073c80 2024-09-17T18:33:39.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.547+0000 7f840effd640 1 -- 172.21.15.121:0/1858194936 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8410073c80 msgr2=0x7f84100740c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:39.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.547+0000 7f840effd640 1 --2- 172.21.15.121:0/1858194936 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8410073c80 0x7f84100740c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f84100a77a0 tx=0x7f840402a0d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be7f016c0e73bf75 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:39.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.547+0000 7f840e7fc640 1 -- 172.21.15.121:0/1858194936 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8404037070 con 0x7f8410073c80 2024-09-17T18:33:39.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.547+0000 7f840e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:39.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.547+0000 7f840e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:33:39.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.547+0000 7f840e7fc640 10 monclient: dump: 2024-09-17T18:33:39.556 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:39.557 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:39.557 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:39.557 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:39.557 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:39.557 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:39.557 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:39.557 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:39.557 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:39.557 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:39.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.547+0000 7f840e7fc640 10 monclient: _finish_auth 0 2024-09-17T18:33:39.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.547+0000 7f840e7fc640 10 monclient: _check_auth_tickets 2024-09-17T18:33:39.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.547+0000 7f840e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:33:39.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.547+0000 7f8415dd5640 5 monclient: authenticate success, global_id 64418 2024-09-17T18:33:39.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.547+0000 7f8415dd5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:33:39.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.547+0000 7f840e7fc640 1 -- 172.21.15.121:0/1858194936 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8404032640 con 0x7f8410073c80 2024-09-17T18:33:39.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.547+0000 7f840e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:33:39.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.547+0000 7f8415dd5640 10 monclient: get_monmap_and_config success 2024-09-17T18:33:39.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.547+0000 7f8415dd5640 4 set_mon_vals no callback set 2024-09-17T18:33:39.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.547+0000 7f840e7fc640 1 -- 172.21.15.121:0/1858194936 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8404032a10 con 0x7f8410073c80 2024-09-17T18:33:39.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.547+0000 7f840e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:39.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.547+0000 7f840e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:33:39.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.547+0000 7f840e7fc640 10 monclient: dump: 2024-09-17T18:33:39.558 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:39.558 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:39.558 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:39.558 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:39.558 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:39.558 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:39.558 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:39.558 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:39.558 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:39.559 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:39.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.547+0000 7f8415dd5640 10 monclient: shutdown 2024-09-17T18:33:39.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.547+0000 7f8415dd5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:33:39.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.547+0000 7f8415dd5640 1 -- 172.21.15.121:0/1858194936 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8410073c80 msgr2=0x7f84100740c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:39.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.547+0000 7f8415dd5640 1 --2- 172.21.15.121:0/1858194936 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8410073c80 0x7f84100740c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f84100a77a0 tx=0x7f840402a0d0 comp rx=0 tx=0).stop 2024-09-17T18:33:39.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.547+0000 7f8415dd5640 1 -- 172.21.15.121:0/1858194936 shutdown_connections 2024-09-17T18:33:39.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.547+0000 7f8415dd5640 1 --2- 172.21.15.121:0/1858194936 >> v2:172.21.15.121:3300/0 conn(0x7f84100a0e70 0x7f84100a12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:39.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.547+0000 7f8415dd5640 1 --2- 172.21.15.121:0/1858194936 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8410073c80 0x7f84100740c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:39.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.547+0000 7f8415dd5640 1 --2- 172.21.15.121:0/1858194936 >> v2:172.21.15.173:3300/0 conn(0x7f84100738a0 0x7f8410138a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:39.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.547+0000 7f8415dd5640 1 -- 172.21.15.121:0/1858194936 >> 172.21.15.121:0/1858194936 conn(0x7f8410076f00 msgr2=0x7f8410077310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:33:39.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.547+0000 7f8415dd5640 1 -- 172.21.15.121:0/1858194936 shutdown_connections 2024-09-17T18:33:39.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.547+0000 7f8415dd5640 1 -- 172.21.15.121:0/1858194936 wait complete. 2024-09-17T18:33:39.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.548+0000 7f8415dd5640 10 monclient: build_initial_monmap 2024-09-17T18:33:39.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.548+0000 7f8415dd5640 10 monclient: monmap: 2024-09-17T18:33:39.559 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:33:39.560 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:33:39.560 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:33:39.560 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:33:39.560 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:33:39.560 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:39.560 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:33:39.560 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:33:39.560 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:33:39.560 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:39.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.548+0000 7f8415dd5640 1 librados: starting msgr at 2024-09-17T18:33:39.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.548+0000 7f8415dd5640 1 librados: starting objecter 2024-09-17T18:33:39.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.548+0000 7f8415dd5640 1 Processor -- start 2024-09-17T18:33:39.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f8415dd5640 1 -- start start 2024-09-17T18:33:39.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f8415dd5640 1 librados: setting wanted keys 2024-09-17T18:33:39.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f8415dd5640 1 librados: calling monclient init 2024-09-17T18:33:39.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f8415dd5640 10 monclient: init 2024-09-17T18:33:39.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f8415dd5640 10 monclient: _reopen_session rank -1 2024-09-17T18:33:39.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f8415dd5640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T18:33:39.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f8415dd5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f84100738a0 0x7f84101351c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:39.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f8415dd5640 10 monclient(hunting): picked mon.noname-c con 0x7f84100738a0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:33:39.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f8415dd5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8410073c80 0x7f8410135700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:39.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f8415dd5640 10 monclient(hunting): picked mon.noname-a con 0x7f8410073c80 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:33:39.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f8415dd5640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f84100a0e70 0x7f841011ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:39.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f8415dd5640 10 monclient(hunting): picked mon.noname-b con 0x7f84100a0e70 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:33:39.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f8415dd5640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:39.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f8415dd5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f841006d820 con 0x7f8410073c80 2024-09-17T18:33:39.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f8415dd5640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:39.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f8415dd5640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f841006d6a0 con 0x7f84100a0e70 2024-09-17T18:33:39.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f8415dd5640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:39.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f8415dd5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f84100a1810 con 0x7f84100738a0 2024-09-17T18:33:39.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f8415dd5640 10 monclient(hunting): _renew_subs 2024-09-17T18:33:39.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f8415dd5640 10 monclient(hunting): authenticate will time out at 1623.265580s 2024-09-17T18:33:39.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f840effd640 1 -- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8410073c80 msgr2=0x7f8410135700 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:33:39.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f840effd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8410073c80 0x7f8410135700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:33:39.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f840f7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f84100738a0 0x7f84101351c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:39.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f840f7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f84100738a0 0x7f84101351c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:57286/0 (socket says 172.21.15.121:57286) 2024-09-17T18:33:39.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f840f7fe640 1 -- 172.21.15.121:0/1683256138 learned_addr learned my addr 172.21.15.121:0/1683256138 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:33:39.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f840f7fe640 10 monclient(hunting): get_auth_request con 0x7f84100738a0 auth_method 0 2024-09-17T18:33:39.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f840f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:39.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f840f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:39.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f840f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:39.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f840f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:39.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f840f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:39.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f840f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:39.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f840ffff640 1 --2- 172.21.15.121:0/1683256138 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f84100a0e70 0x7f841011ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:39.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f840ffff640 10 monclient(hunting): get_auth_request con 0x7f84100a0e70 auth_method 0 2024-09-17T18:33:39.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f840ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:39.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f840ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:39.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f840ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:39.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f840f7fe640 10 monclient(hunting): handle_auth_done global_id 64421 payload 824 2024-09-17T18:33:39.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f840f7fe640 1 -- 172.21.15.121:0/1683256138 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f84100a0e70 msgr2=0x7f841011ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:39.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f840f7fe640 1 --2- 172.21.15.121:0/1683256138 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f84100a0e70 0x7f841011ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:39.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f840f7fe640 1 -- 172.21.15.121:0/1683256138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8410073c80 msgr2=0x7f8410135700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:33:39.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f840f7fe640 1 --2- 172.21.15.121:0/1683256138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8410073c80 0x7f8410135700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:39.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f840f7fe640 10 monclient: _finish_hunting 0 2024-09-17T18:33:39.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f840f7fe640 1 monclient: found mon.noname-c 2024-09-17T18:33:39.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f840f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:33:39.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f840f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:33:39.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.549+0000 7f840f7fe640 1 -- 172.21.15.121:0/1683256138 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8410074c80 con 0x7f84100738a0 2024-09-17T18:33:39.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.552+0000 7f840f7fe640 1 -- 172.21.15.121:0/1683256138 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f84100738a0 msgr2=0x7f84101351c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:39.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.552+0000 7f840f7fe640 1 --2- 172.21.15.121:0/1683256138 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f84100738a0 0x7f84101351c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f840000d800 tx=0x7f840000dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:39.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.552+0000 7f83f7fff640 10 client.?.objecter ms_handle_connect 0x7f84100738a0 2024-09-17T18:33:39.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.552+0000 7f83f7fff640 10 client.?.objecter resend_mon_ops 2024-09-17T18:33:39.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.552+0000 7f83f7fff640 1 -- 172.21.15.121:0/1683256138 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f840000b8d0 con 0x7f84100738a0 2024-09-17T18:33:39.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.552+0000 7f83f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:39.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.552+0000 7f83f7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:33:39.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.552+0000 7f83f7fff640 10 monclient: dump: 2024-09-17T18:33:39.564 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:39.564 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:39.565 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:39.565 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:39.565 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:39.565 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:39.565 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:39.565 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:39.565 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:39.565 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:39.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.552+0000 7f83f7fff640 10 monclient: _finish_auth 0 2024-09-17T18:33:39.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.552+0000 7f83f7fff640 10 monclient: _check_auth_tickets 2024-09-17T18:33:39.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.552+0000 7f83f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:33:39.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.552+0000 7f8415dd5640 5 monclient: authenticate success, global_id 64421 2024-09-17T18:33:39.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.552+0000 7f83f7fff640 1 -- 172.21.15.121:0/1683256138 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8400019070 con 0x7f84100738a0 2024-09-17T18:33:39.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.552+0000 7f83f7fff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:33:39.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.552+0000 7f83f7fff640 1 -- 172.21.15.121:0/1683256138 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f840000bc90 con 0x7f84100738a0 2024-09-17T18:33:39.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.552+0000 7f8415dd5640 10 monclient: _renew_subs 2024-09-17T18:33:39.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.552+0000 7f8415dd5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:33:39.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.552+0000 7f8415dd5640 1 -- 172.21.15.121:0/1683256138 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8410120820 con 0x7f84100738a0 2024-09-17T18:33:39.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.552+0000 7f83f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:39.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.552+0000 7f83f7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:33:39.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.552+0000 7f83f7fff640 10 monclient: dump: 2024-09-17T18:33:39.566 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:39.566 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:39.566 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:39.566 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:39.566 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:39.566 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:39.566 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:39.567 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:39.567 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:39.567 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:39.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.552+0000 7f840d7fa640 4 set_mon_vals no callback set 2024-09-17T18:33:39.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.552+0000 7f8415dd5640 10 client.64421.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:33:39.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.552+0000 7f8415dd5640 10 monclient: _renew_subs 2024-09-17T18:33:39.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.552+0000 7f8415dd5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:33:39.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.552+0000 7f8415dd5640 1 -- 172.21.15.121:0/1683256138 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f84101cd9b0 con 0x7f84100738a0 2024-09-17T18:33:39.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.552+0000 7f8415dd5640 1 librados: init done 2024-09-17T18:33:39.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.553+0000 7f83f7fff640 1 -- 172.21.15.121:0/1683256138 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f840001e460 con 0x7f84100738a0 2024-09-17T18:33:39.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.553+0000 7f8415dd5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:33:39.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.553+0000 7f8415dd5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:33:39.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.553+0000 7f8415dd5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:33:39.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.553+0000 7f8415dd5640 1 -- 172.21.15.121:0/1683256138 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8410122520 con 0x7f84100738a0 2024-09-17T18:33:39.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.553+0000 7f83f7fff640 1 --2- 172.21.15.121:0/1683256138 >> v2:172.21.15.173:6812/2423786037 conn(0x7f83fc040120 0x7f83fc0425e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:39.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.553+0000 7f83f7fff640 1 -- 172.21.15.121:0/1683256138 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(563..563 src has 247..563) ==== 5516+0+0 (secure 0 0 0) 0x7f8400053cd0 con 0x7f84100738a0 2024-09-17T18:33:39.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.553+0000 7f83f7fff640 10 client.64421.objecter ms_dispatch 0x7f8410000b90 osd_map(563..563 src has 247..563) 2024-09-17T18:33:39.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.553+0000 7f83f7fff640 3 client.64421.objecter handle_osd_map got epochs [563,563] > 0 2024-09-17T18:33:39.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.553+0000 7f83f7fff640 3 client.64421.objecter handle_osd_map decoding full epoch 563 2024-09-17T18:33:39.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.553+0000 7f83f7fff640 20 client.64421.objecter dump_active .. 0 homeless 2024-09-17T18:33:39.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.553+0000 7f840effd640 1 --2- 172.21.15.121:0/1683256138 >> v2:172.21.15.173:6812/2423786037 conn(0x7f83fc040120 0x7f83fc0425e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:39.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.553+0000 7f840effd640 10 monclient: get_auth_request con 0x7f83fc040120 auth_method 0 2024-09-17T18:33:39.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.556+0000 7f840effd640 1 --2- 172.21.15.121:0/1683256138 >> v2:172.21.15.173:6812/2423786037 conn(0x7f83fc040120 0x7f83fc0425e0 secure :-1 s=READY pgs=4059 cs=0 l=1 rev1=1 crypto rx=0x7f8404034000 tx=0x7f840402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:39.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.556+0000 7f83f7fff640 10 client.64421.objecter ms_handle_connect 0x7f83fc040120 2024-09-17T18:33:39.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.558+0000 7f83f7fff640 1 -- 172.21.15.121:0/1683256138 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f84000523f0 con 0x7f84100738a0 2024-09-17T18:33:39.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.558+0000 7f83f7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:33:39.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.558+0000 7f83f7fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:33:39.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.798+0000 7f8415dd5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:33:39.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.798+0000 7f8415dd5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:33:39.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.798+0000 7f8415dd5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:33:39.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.798+0000 7f8415dd5640 1 -- 172.21.15.121:0/1683256138 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8410122ae0 con 0x7f84100738a0 2024-09-17T18:33:39.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.799+0000 7f83f7fff640 1 -- 172.21.15.121:0/1683256138 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f8400014430 con 0x7f84100738a0 2024-09-17T18:33:39.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.799+0000 7f83f7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:33:39.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.799+0000 7f83f7fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:33:39.801 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":54,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:33:39.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.800+0000 7f8415dd5640 10 librados: watch_flush enter 2024-09-17T18:33:39.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.800+0000 7f8415dd5640 10 librados: watch_flush exit 2024-09-17T18:33:39.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.800+0000 7f8415dd5640 20 client.64421.objecter shutdown clearing up homeless session... 2024-09-17T18:33:39.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.800+0000 7f8415dd5640 10 client.64421.objecter successfully canceled tick 2024-09-17T18:33:39.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.801+0000 7f8415dd5640 1 -- 172.21.15.121:0/1683256138 >> v2:172.21.15.173:6812/2423786037 conn(0x7f83fc040120 msgr2=0x7f83fc0425e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:39.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.801+0000 7f8415dd5640 1 --2- 172.21.15.121:0/1683256138 >> v2:172.21.15.173:6812/2423786037 conn(0x7f83fc040120 0x7f83fc0425e0 secure :-1 s=READY pgs=4059 cs=0 l=1 rev1=1 crypto rx=0x7f8404034000 tx=0x7f840402e000 comp rx=0 tx=0).stop 2024-09-17T18:33:39.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.801+0000 7f8415dd5640 10 monclient: shutdown 2024-09-17T18:33:39.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.801+0000 7f8415dd5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:33:39.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.801+0000 7f8415dd5640 1 -- 172.21.15.121:0/1683256138 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f84100738a0 msgr2=0x7f84101351c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:39.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.801+0000 7f8415dd5640 1 --2- 172.21.15.121:0/1683256138 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f84100738a0 0x7f84101351c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f840000d800 tx=0x7f840000dcc0 comp rx=0 tx=0).stop 2024-09-17T18:33:39.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.801+0000 7f8415dd5640 1 -- 172.21.15.121:0/1683256138 shutdown_connections 2024-09-17T18:33:39.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.801+0000 7f8415dd5640 1 --2- 172.21.15.121:0/1683256138 >> v2:172.21.15.173:6812/2423786037 conn(0x7f83fc040120 0x7f83fc0425e0 unknown :-1 s=CLOSED pgs=4059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:39.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.801+0000 7f8415dd5640 1 --2- 172.21.15.121:0/1683256138 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f84100a0e70 0x7f841011ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:39.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.801+0000 7f8415dd5640 1 --2- 172.21.15.121:0/1683256138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8410073c80 0x7f8410135700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:39.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.801+0000 7f8415dd5640 1 --2- 172.21.15.121:0/1683256138 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f84100738a0 0x7f84101351c0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:39.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.801+0000 7f8415dd5640 1 -- 172.21.15.121:0/1683256138 >> 172.21.15.121:0/1683256138 conn(0x7f8410076f00 msgr2=0x7f84100728a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:33:39.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.801+0000 7f8415dd5640 1 -- 172.21.15.121:0/1683256138 shutdown_connections 2024-09-17T18:33:39.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.801+0000 7f8415dd5640 1 -- 172.21.15.121:0/1683256138 wait complete. 2024-09-17T18:33:39.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:39.801+0000 7f8415dd5640 1 librados: shutdown 2024-09-17T18:33:42.815 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:33:42.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:42.922+0000 7f334aeed640 10 monclient: get_monmap_and_config 2024-09-17T18:33:42.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:42.922+0000 7f334aeed640 10 monclient: build_initial_monmap 2024-09-17T18:33:42.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:42.922+0000 7f334aeed640 10 monclient: monmap: 2024-09-17T18:33:42.924 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:33:42.924 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:42.924 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:33:42.923052+0000 2024-09-17T18:33:42.924 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:33:42.923052+0000 2024-09-17T18:33:42.924 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:33:42.924 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:42.925 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:33:42.925 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:33:42.925 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:33:42.925 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:33:42.925 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:33:42.925 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:42.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:42.923+0000 7f334aeed640 1 Processor -- start 2024-09-17T18:33:42.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:42.923+0000 7f334aeed640 1 -- start start 2024-09-17T18:33:42.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:42.923+0000 7f334aeed640 10 monclient: init 2024-09-17T18:33:42.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:42.923+0000 7f334aeed640 10 monclient: _reopen_session rank -1 2024-09-17T18:33:42.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:42.923+0000 7f334aeed640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-17T18:33:42.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:42.923+0000 7f334aeed640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f33440a0a70 addr v1:172.21.15.173:6789/0 2024-09-17T18:33:42.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:42.923+0000 7f334aeed640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f33441348f0 0x7f3344139c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:42.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:42.923+0000 7f334aeed640 10 monclient(hunting): picked mon.noname-c con 0x7f33441348f0 addr v2:172.21.15.173:3300/0 2024-09-17T18:33:42.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:42.923+0000 7f334aeed640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f334413a150 0x7f334413c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:42.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:42.923+0000 7f334aeed640 10 monclient(hunting): picked mon.noname-b con 0x7f334413a150 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:33:42.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:42.923+0000 7f334aeed640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f334409f5b0 con 0x7f33440a0a70 2024-09-17T18:33:42.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:42.923+0000 7f334aeed640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:42.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:42.923+0000 7f334aeed640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f334409ef80 con 0x7f33441348f0 2024-09-17T18:33:42.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:42.923+0000 7f334aeed640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:42.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:42.923+0000 7f334aeed640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f33440a1560 con 0x7f334413a150 2024-09-17T18:33:42.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:42.923+0000 7f334aeed640 10 monclient(hunting): _renew_subs 2024-09-17T18:33:42.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:42.923+0000 7f334aeed640 10 monclient(hunting): authenticate will time out at 1626.639805s 2024-09-17T18:33:42.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:42.923+0000 7f334948a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f334413a150 0x7f334413c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:42.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:42.923+0000 7f334948a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f334413a150 0x7f334413c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:57292/0 (socket says 172.21.15.121:57292) 2024-09-17T18:33:42.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:42.923+0000 7f334948a640 1 -- 172.21.15.121:0/446496336 learned_addr learned my addr 172.21.15.121:0/446496336 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:33:42.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:42.923+0000 7f334948a640 10 monclient(hunting): get_auth_request con 0x7f334413a150 auth_method 0 2024-09-17T18:33:42.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:42.923+0000 7f334948a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:42.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:42.924+0000 7f334948a640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:42.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:42.924+0000 7f334948a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:42.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:42.924+0000 7f334948a640 1 -- 172.21.15.121:0/446496336 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f334413a150 msgr2=0x7f334413c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T18:33:42.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:42.924+0000 7f334948a640 1 -- 172.21.15.121:0/446496336 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f334413a150 msgr2=0x7f334413c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:33:42.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:42.924+0000 7f334948a640 1 --2- 172.21.15.121:0/446496336 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f334413a150 0x7f334413c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:33:42.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:42.924+0000 7f334948a640 1 --2- 172.21.15.121:0/446496336 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f334413a150 0x7f334413c540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:33:42.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:42.924+0000 7f333bfff640 1 --2- 172.21.15.121:0/446496336 >> v2:172.21.15.173:3300/0 conn(0x7f33441348f0 0x7f3344139c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:42.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:42.924+0000 7f333bfff640 10 monclient(hunting): get_auth_request con 0x7f33441348f0 auth_method 0 2024-09-17T18:33:42.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:42.924+0000 7f333bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:42.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:42.924+0000 7f333bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:42.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:42.924+0000 7f333bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:42.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:42.924+0000 7f333bfff640 1 -- 172.21.15.121:0/446496336 >> v2:172.21.15.173:3300/0 conn(0x7f33441348f0 msgr2=0x7f3344139c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T18:33:42.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:42.924+0000 7f333bfff640 1 -- 172.21.15.121:0/446496336 >> v2:172.21.15.173:3300/0 conn(0x7f33441348f0 msgr2=0x7f3344139c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:33:42.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:42.924+0000 7f333bfff640 1 --2- 172.21.15.121:0/446496336 >> v2:172.21.15.173:3300/0 conn(0x7f33441348f0 0x7f3344139c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:33:42.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:42.924+0000 7f333bfff640 1 --2- 172.21.15.121:0/446496336 >> v2:172.21.15.173:3300/0 conn(0x7f33441348f0 0x7f3344139c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:33:42.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:42.926+0000 7f3348c89640 1 -- 172.21.15.121:0/446496336 >> v1:172.21.15.173:6789/0 conn(0x7f33440a0a70 legacy=0x7f33440a0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:43.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:43.125+0000 7f334948a640 1 --2- 172.21.15.121:0/446496336 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f334413a150 0x7f334413c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:43.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:43.125+0000 7f334948a640 10 monclient(hunting): get_auth_request con 0x7f334413a150 auth_method 0 2024-09-17T18:33:43.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:43.125+0000 7f334948a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:43.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:43.125+0000 7f334948a640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:43.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:43.125+0000 7f334948a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:33:43.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:43.125+0000 7f334948a640 1 -- 172.21.15.121:0/446496336 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f334413a150 msgr2=0x7f334413c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T18:33:43.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:43.125+0000 7f334948a640 1 -- 172.21.15.121:0/446496336 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f334413a150 msgr2=0x7f334413c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:33:43.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:43.125+0000 7f334948a640 1 --2- 172.21.15.121:0/446496336 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f334413a150 0x7f334413c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:33:43.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:43.125+0000 7f334948a640 1 --2- 172.21.15.121:0/446496336 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f334413a150 0x7f334413c540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T18:33:43.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:43.125+0000 7f333bfff640 1 --2- 172.21.15.121:0/446496336 >> v2:172.21.15.173:3300/0 conn(0x7f33441348f0 0x7f3344139c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:43.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:43.125+0000 7f333bfff640 10 monclient(hunting): get_auth_request con 0x7f33441348f0 auth_method 0 2024-09-17T18:33:43.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:43.125+0000 7f333bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:43.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:43.125+0000 7f333bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:43.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:43.125+0000 7f333bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:33:43.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:43.126+0000 7f333bfff640 1 -- 172.21.15.121:0/446496336 >> v2:172.21.15.173:3300/0 conn(0x7f33441348f0 msgr2=0x7f3344139c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T18:33:43.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:43.126+0000 7f333bfff640 1 -- 172.21.15.121:0/446496336 >> v2:172.21.15.173:3300/0 conn(0x7f33441348f0 msgr2=0x7f3344139c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:33:43.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:43.126+0000 7f333bfff640 1 --2- 172.21.15.121:0/446496336 >> v2:172.21.15.173:3300/0 conn(0x7f33441348f0 0x7f3344139c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:33:43.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:43.126+0000 7f333bfff640 1 --2- 172.21.15.121:0/446496336 >> v2:172.21.15.173:3300/0 conn(0x7f33441348f0 0x7f3344139c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T18:33:43.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:43.526+0000 7f334948a640 1 --2- 172.21.15.121:0/446496336 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f334413a150 0x7f334413c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:43.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:43.526+0000 7f334948a640 10 monclient(hunting): get_auth_request con 0x7f334413a150 auth_method 0 2024-09-17T18:33:43.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:43.526+0000 7f334948a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:43.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:43.526+0000 7f334948a640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:43.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:43.526+0000 7f334948a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:33:43.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:43.526+0000 7f334948a640 1 -- 172.21.15.121:0/446496336 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f334413a150 msgr2=0x7f334413c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T18:33:43.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:43.526+0000 7f334948a640 1 -- 172.21.15.121:0/446496336 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f334413a150 msgr2=0x7f334413c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:33:43.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:43.526+0000 7f334948a640 1 --2- 172.21.15.121:0/446496336 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f334413a150 0x7f334413c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:33:43.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:43.526+0000 7f334948a640 1 --2- 172.21.15.121:0/446496336 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f334413a150 0x7f334413c540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T18:33:43.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:43.527+0000 7f333bfff640 1 --2- 172.21.15.121:0/446496336 >> v2:172.21.15.173:3300/0 conn(0x7f33441348f0 0x7f3344139c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:43.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:43.527+0000 7f333bfff640 10 monclient(hunting): get_auth_request con 0x7f33441348f0 auth_method 0 2024-09-17T18:33:43.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:43.527+0000 7f333bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:43.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:43.527+0000 7f333bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:43.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:43.527+0000 7f333bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:33:43.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:43.528+0000 7f333bfff640 1 -- 172.21.15.121:0/446496336 >> v2:172.21.15.173:3300/0 conn(0x7f33441348f0 msgr2=0x7f3344139c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T18:33:43.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:43.528+0000 7f333bfff640 1 -- 172.21.15.121:0/446496336 >> v2:172.21.15.173:3300/0 conn(0x7f33441348f0 msgr2=0x7f3344139c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:33:43.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:43.528+0000 7f333bfff640 1 --2- 172.21.15.121:0/446496336 >> v2:172.21.15.173:3300/0 conn(0x7f33441348f0 0x7f3344139c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:33:43.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:43.528+0000 7f333bfff640 1 --2- 172.21.15.121:0/446496336 >> v2:172.21.15.173:3300/0 conn(0x7f33441348f0 0x7f3344139c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T18:33:44.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:44.327+0000 7f334948a640 1 --2- 172.21.15.121:0/446496336 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f334413a150 0x7f334413c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:44.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:44.327+0000 7f334948a640 10 monclient(hunting): get_auth_request con 0x7f334413a150 auth_method 0 2024-09-17T18:33:44.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:44.327+0000 7f334948a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:44.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:44.327+0000 7f334948a640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:44.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:44.327+0000 7f334948a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:33:44.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:44.327+0000 7f334948a640 1 -- 172.21.15.121:0/446496336 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f334413a150 msgr2=0x7f334413c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T18:33:44.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:44.327+0000 7f334948a640 1 -- 172.21.15.121:0/446496336 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f334413a150 msgr2=0x7f334413c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:33:44.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:44.327+0000 7f334948a640 1 --2- 172.21.15.121:0/446496336 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f334413a150 0x7f334413c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:33:44.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:44.327+0000 7f334948a640 1 --2- 172.21.15.121:0/446496336 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f334413a150 0x7f334413c540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T18:33:44.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:44.330+0000 7f333bfff640 1 --2- 172.21.15.121:0/446496336 >> v2:172.21.15.173:3300/0 conn(0x7f33441348f0 0x7f3344139c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:44.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:44.330+0000 7f333bfff640 10 monclient(hunting): get_auth_request con 0x7f33441348f0 auth_method 0 2024-09-17T18:33:44.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:44.330+0000 7f333bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:44.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:44.330+0000 7f333bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:44.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:44.330+0000 7f333bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:33:44.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:44.330+0000 7f333bfff640 1 -- 172.21.15.121:0/446496336 >> v2:172.21.15.173:3300/0 conn(0x7f33441348f0 msgr2=0x7f3344139c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T18:33:44.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:44.330+0000 7f333bfff640 1 -- 172.21.15.121:0/446496336 >> v2:172.21.15.173:3300/0 conn(0x7f33441348f0 msgr2=0x7f3344139c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:33:44.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:44.330+0000 7f333bfff640 1 --2- 172.21.15.121:0/446496336 >> v2:172.21.15.173:3300/0 conn(0x7f33441348f0 0x7f3344139c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:33:44.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:44.330+0000 7f333bfff640 1 --2- 172.21.15.121:0/446496336 >> v2:172.21.15.173:3300/0 conn(0x7f33441348f0 0x7f3344139c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T18:33:45.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:45.924+0000 7f333a7fc640 10 monclient(hunting): tick 2024-09-17T18:33:45.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:45.924+0000 7f333a7fc640 10 monclient(hunting): _check_auth_tickets 2024-09-17T18:33:45.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:45.924+0000 7f333a7fc640 1 monclient(hunting): continuing hunt 2024-09-17T18:33:45.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:45.924+0000 7f333a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T18:33:45.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:45.924+0000 7f333a7fc640 1 -- 172.21.15.121:0/446496336 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f334413a150 msgr2=0x7f334413c540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:33:45.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:45.924+0000 7f333a7fc640 1 --2- 172.21.15.121:0/446496336 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f334413a150 0x7f334413c540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:45.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:45.924+0000 7f333a7fc640 1 -- 172.21.15.121:0/446496336 >> v2:172.21.15.173:3300/0 conn(0x7f33441348f0 msgr2=0x7f3344139c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:33:45.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:45.924+0000 7f333a7fc640 1 --2- 172.21.15.121:0/446496336 >> v2:172.21.15.173:3300/0 conn(0x7f33441348f0 0x7f3344139c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:45.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:45.924+0000 7f333a7fc640 1 -- 172.21.15.121:0/446496336 >> v1:172.21.15.173:6789/0 conn(0x7f33440a0a70 legacy=0x7f33440a0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:45.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:45.924+0000 7f333a7fc640 10 monclient: _add_conns ranks=[3,0] 2024-09-17T18:33:45.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:45.924+0000 7f333a7fc640 1 --2- 172.21.15.121:0/446496336 >> v2:172.21.15.121:3300/0 conn(0x7f3330003790 0x7f3330005bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:45.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:45.924+0000 7f333a7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f3330003790 addr v2:172.21.15.121:3300/0 2024-09-17T18:33:45.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:45.924+0000 7f333a7fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f33300062c0 addr v1:172.21.15.121:6789/0 2024-09-17T18:33:45.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:45.924+0000 7f333a7fc640 1 -- 172.21.15.121:0/446496336 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f334409eca0 con 0x7f33300062c0 2024-09-17T18:33:45.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:45.924+0000 7f333a7fc640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:45.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:45.924+0000 7f333a7fc640 1 -- 172.21.15.121:0/446496336 --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f334409ef80 con 0x7f3330003790 2024-09-17T18:33:45.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:45.924+0000 7f333a7fc640 10 monclient(hunting): _renew_subs 2024-09-17T18:33:45.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:45.924+0000 7f333bfff640 1 -- 172.21.15.121:0/446496336 >> v1:172.21.15.121:6789/0 conn(0x7f33300062c0 legacy=0x7f3330008760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.121:6789/0 2024-09-17T18:33:45.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:45.924+0000 7f3348c89640 1 -- 172.21.15.121:0/446496336 >> v2:172.21.15.121:3300/0 conn(0x7f3330003790 msgr2=0x7f3330005bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:33:45.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:45.924+0000 7f3348c89640 1 --2- 172.21.15.121:0/446496336 >> v2:172.21.15.121:3300/0 conn(0x7f3330003790 0x7f3330005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:33:46.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:46.125+0000 7f333bfff640 1 -- 172.21.15.121:0/446496336 >> v1:172.21.15.121:6789/0 conn(0x7f33300062c0 legacy=0x7f3330008760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.121:6789/0 2024-09-17T18:33:46.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:46.125+0000 7f3348c89640 1 -- 172.21.15.121:0/446496336 >> v2:172.21.15.121:3300/0 conn(0x7f3330003790 msgr2=0x7f3330005bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:33:46.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:46.125+0000 7f3348c89640 1 --2- 172.21.15.121:0/446496336 >> v2:172.21.15.121:3300/0 conn(0x7f3330003790 0x7f3330005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T18:33:46.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:46.526+0000 7f333bfff640 1 -- 172.21.15.121:0/446496336 >> v1:172.21.15.121:6789/0 conn(0x7f33300062c0 legacy=0x7f3330008760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.121:6789/0 2024-09-17T18:33:46.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:46.526+0000 7f3348c89640 1 -- 172.21.15.121:0/446496336 >> v2:172.21.15.121:3300/0 conn(0x7f3330003790 msgr2=0x7f3330005bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:33:46.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:46.526+0000 7f3348c89640 1 --2- 172.21.15.121:0/446496336 >> v2:172.21.15.121:3300/0 conn(0x7f3330003790 0x7f3330005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T18:33:47.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:47.327+0000 7f333bfff640 1 -- 172.21.15.121:0/446496336 >> v1:172.21.15.121:6789/0 conn(0x7f33300062c0 legacy=0x7f3330008760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.121:6789/0 2024-09-17T18:33:47.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:47.327+0000 7f3348c89640 1 -- 172.21.15.121:0/446496336 >> v2:172.21.15.121:3300/0 conn(0x7f3330003790 msgr2=0x7f3330005bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:33:47.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:47.327+0000 7f3348c89640 1 --2- 172.21.15.121:0/446496336 >> v2:172.21.15.121:3300/0 conn(0x7f3330003790 0x7f3330005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T18:33:48.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.924+0000 7f333a7fc640 10 monclient(hunting): tick 2024-09-17T18:33:48.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.924+0000 7f333a7fc640 10 monclient(hunting): _check_auth_tickets 2024-09-17T18:33:48.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.924+0000 7f333a7fc640 1 monclient(hunting): continuing hunt 2024-09-17T18:33:48.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.924+0000 7f333a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T18:33:48.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.924+0000 7f333a7fc640 1 -- 172.21.15.121:0/446496336 >> v2:172.21.15.121:3300/0 conn(0x7f3330003790 msgr2=0x7f3330005bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:33:48.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.924+0000 7f333a7fc640 1 --2- 172.21.15.121:0/446496336 >> v2:172.21.15.121:3300/0 conn(0x7f3330003790 0x7f3330005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:48.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.924+0000 7f333a7fc640 1 -- 172.21.15.121:0/446496336 >> v1:172.21.15.121:6789/0 conn(0x7f33300062c0 legacy=0x7f3330008760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:33:48.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.924+0000 7f333a7fc640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-17T18:33:48.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.924+0000 7f334948a640 1 -- 172.21.15.121:0/446496336 reap_dead start 2024-09-17T18:33:48.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.924+0000 7f333a7fc640 1 --2- 172.21.15.121:0/446496336 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f334413a150 0x7f3330009ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:48.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.924+0000 7f333a7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f334413a150 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:33:48.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.924+0000 7f333a7fc640 1 --2- 172.21.15.121:0/446496336 >> v2:172.21.15.173:3300/0 conn(0x7f333000a530 0x7f3330005bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:48.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.924+0000 7f333a7fc640 10 monclient(hunting): picked mon.noname-c con 0x7f333000a530 addr v2:172.21.15.173:3300/0 2024-09-17T18:33:48.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.924+0000 7f333a7fc640 1 --2- 172.21.15.121:0/446496336 >> v2:172.21.15.121:3300/0 conn(0x7f333000aa40 0x7f33300086b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:48.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.924+0000 7f333a7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f333000aa40 addr v2:172.21.15.121:3300/0 2024-09-17T18:33:48.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.924+0000 7f333a7fc640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:48.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.924+0000 7f333a7fc640 1 -- 172.21.15.121:0/446496336 --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f334409ef80 con 0x7f333000aa40 2024-09-17T18:33:48.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.924+0000 7f333a7fc640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:48.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.924+0000 7f333a7fc640 1 -- 172.21.15.121:0/446496336 --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f33440a1560 con 0x7f333000a530 2024-09-17T18:33:48.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.924+0000 7f333a7fc640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:48.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.924+0000 7f333a7fc640 1 -- 172.21.15.121:0/446496336 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3330008c20 con 0x7f334413a150 2024-09-17T18:33:48.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.924+0000 7f333a7fc640 10 monclient(hunting): _renew_subs 2024-09-17T18:33:48.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.924+0000 7f334948a640 1 -- 172.21.15.121:0/446496336 >> v2:172.21.15.121:3300/0 conn(0x7f333000aa40 msgr2=0x7f33300086b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:33:48.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.924+0000 7f334948a640 1 --2- 172.21.15.121:0/446496336 >> v2:172.21.15.121:3300/0 conn(0x7f333000aa40 0x7f33300086b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:33:48.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.924+0000 7f3348c89640 1 --2- 172.21.15.121:0/446496336 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f334413a150 0x7f3330009ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:48.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.924+0000 7f3348c89640 10 monclient(hunting): get_auth_request con 0x7f334413a150 auth_method 0 2024-09-17T18:33:48.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.924+0000 7f3348c89640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:48.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.924+0000 7f3348c89640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:48.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.924+0000 7f3348c89640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:48.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.924+0000 7f3348c89640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:48.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.924+0000 7f3348c89640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:48.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.924+0000 7f3348c89640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:48.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.924+0000 7f333bfff640 1 --2- 172.21.15.121:0/446496336 >> v2:172.21.15.173:3300/0 conn(0x7f333000a530 0x7f3330005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:48.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.924+0000 7f333bfff640 10 monclient(hunting): get_auth_request con 0x7f333000a530 auth_method 0 2024-09-17T18:33:48.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.924+0000 7f333bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:48.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.924+0000 7f333bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:48.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.924+0000 7f333bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:48.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.925+0000 7f3348c89640 10 monclient(hunting): handle_auth_done global_id 64424 payload 274 2024-09-17T18:33:48.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.925+0000 7f3348c89640 1 -- 172.21.15.121:0/446496336 >> v2:172.21.15.173:3300/0 conn(0x7f333000a530 msgr2=0x7f3330005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:33:48.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.925+0000 7f3348c89640 1 --2- 172.21.15.121:0/446496336 >> v2:172.21.15.173:3300/0 conn(0x7f333000a530 0x7f3330005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:48.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.925+0000 7f3348c89640 1 -- 172.21.15.121:0/446496336 >> v2:172.21.15.121:3300/0 conn(0x7f333000aa40 msgr2=0x7f33300086b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:33:48.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.925+0000 7f3348c89640 1 --2- 172.21.15.121:0/446496336 >> v2:172.21.15.121:3300/0 conn(0x7f333000aa40 0x7f33300086b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:48.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.925+0000 7f3348c89640 10 monclient: _finish_hunting 0 2024-09-17T18:33:48.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.925+0000 7f3348c89640 1 monclient: found mon.noname-b 2024-09-17T18:33:48.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.925+0000 7f3348c89640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:33:48.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.925+0000 7f3348c89640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:33:48.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.925+0000 7f3348c89640 1 -- 172.21.15.121:0/446496336 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f333000b0c0 con 0x7f334413a150 2024-09-17T18:33:48.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.927+0000 7f3348c89640 1 -- 172.21.15.121:0/446496336 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f334413a150 msgr2=0x7f3330009ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:48.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.927+0000 7f3348c89640 1 --2- 172.21.15.121:0/446496336 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f334413a150 0x7f3330009ff0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f33440a4940 tx=0x7f334002a260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=236ae6ee7113326c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:48.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.928+0000 7f333b7fe640 1 -- 172.21.15.121:0/446496336 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3340007070 con 0x7f334413a150 2024-09-17T18:33:48.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.928+0000 7f333b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:48.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.928+0000 7f333b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:33:48.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.928+0000 7f333b7fe640 10 monclient: dump: 2024-09-17T18:33:48.937 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:48.937 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:48.937 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:48.937 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:48.937 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:48.937 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:48.937 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:48.937 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:48.937 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:48.938 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:48.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.928+0000 7f333b7fe640 10 monclient: _finish_auth 0 2024-09-17T18:33:48.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.928+0000 7f333b7fe640 10 monclient: _check_auth_tickets 2024-09-17T18:33:48.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.928+0000 7f333b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:33:48.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.928+0000 7f333b7fe640 1 -- 172.21.15.121:0/446496336 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f334002e070 con 0x7f334413a150 2024-09-17T18:33:48.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.928+0000 7f333b7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:33:48.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.928+0000 7f333b7fe640 1 -- 172.21.15.121:0/446496336 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3340007470 con 0x7f334413a150 2024-09-17T18:33:48.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.928+0000 7f334aeed640 5 monclient: authenticate success, global_id 64424 2024-09-17T18:33:48.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.928+0000 7f334aeed640 10 monclient: get_monmap_and_config success 2024-09-17T18:33:48.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.928+0000 7f334aeed640 4 set_mon_vals no callback set 2024-09-17T18:33:48.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.928+0000 7f333b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:48.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.928+0000 7f333b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:33:48.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.928+0000 7f333b7fe640 10 monclient: dump: 2024-09-17T18:33:48.938 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:48.939 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:48.939 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:48.939 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:48.939 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:48.939 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:48.939 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:48.939 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:48.939 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:48.939 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:48.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.929+0000 7f334aeed640 10 monclient: shutdown 2024-09-17T18:33:48.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.929+0000 7f334aeed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:33:48.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.929+0000 7f334aeed640 1 -- 172.21.15.121:0/446496336 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f334413a150 msgr2=0x7f3330009ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:48.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.929+0000 7f334aeed640 1 --2- 172.21.15.121:0/446496336 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f334413a150 0x7f3330009ff0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f33440a4940 tx=0x7f334002a260 comp rx=0 tx=0).stop 2024-09-17T18:33:48.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.929+0000 7f334aeed640 1 -- 172.21.15.121:0/446496336 shutdown_connections 2024-09-17T18:33:48.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.929+0000 7f334aeed640 1 --2- 172.21.15.121:0/446496336 >> v2:172.21.15.121:3300/0 conn(0x7f333000aa40 0x7f33300086b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:48.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.929+0000 7f334aeed640 1 --2- 172.21.15.121:0/446496336 >> v2:172.21.15.173:3300/0 conn(0x7f333000a530 0x7f3330005bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:48.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.929+0000 7f334aeed640 1 --2- 172.21.15.121:0/446496336 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f334413a150 0x7f3330009ff0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:48.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.929+0000 7f334aeed640 1 -- 172.21.15.121:0/446496336 >> 172.21.15.121:0/446496336 conn(0x7f33440a3b50 msgr2=0x7f33440a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:33:48.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.929+0000 7f334aeed640 1 -- 172.21.15.121:0/446496336 shutdown_connections 2024-09-17T18:33:48.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.929+0000 7f334aeed640 1 -- 172.21.15.121:0/446496336 wait complete. 2024-09-17T18:33:48.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.930+0000 7f334aeed640 10 monclient: build_initial_monmap 2024-09-17T18:33:48.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.930+0000 7f334aeed640 10 monclient: monmap: 2024-09-17T18:33:48.940 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:33:48.940 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:33:48.940 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:33:48.940 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:33:48.940 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:33:48.940 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:48.940 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:33:48.941 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:33:48.941 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:33:48.941 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:48.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.930+0000 7f334aeed640 1 librados: starting msgr at 2024-09-17T18:33:48.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.930+0000 7f334aeed640 1 librados: starting objecter 2024-09-17T18:33:48.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.930+0000 7f334aeed640 1 Processor -- start 2024-09-17T18:33:48.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.930+0000 7f334aeed640 1 -- start start 2024-09-17T18:33:48.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.930+0000 7f334aeed640 1 librados: setting wanted keys 2024-09-17T18:33:48.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.930+0000 7f334aeed640 1 librados: calling monclient init 2024-09-17T18:33:48.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.930+0000 7f334aeed640 10 monclient: init 2024-09-17T18:33:48.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f334aeed640 10 monclient: _reopen_session rank -1 2024-09-17T18:33:48.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f334aeed640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T18:33:48.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f334aeed640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f333000a530 0x7f33441336b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:48.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f334aeed640 10 monclient(hunting): picked mon.noname-a con 0x7f333000a530 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:33:48.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f334aeed640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f333000aa40 0x7f3344133bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:48.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f334aeed640 10 monclient(hunting): picked mon.noname-c con 0x7f333000aa40 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:33:48.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f334aeed640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f334413a150 0x7f33441cd490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:48.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f334aeed640 10 monclient(hunting): picked mon.noname-b con 0x7f334413a150 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:33:48.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f334aeed640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:48.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f334aeed640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f334409dce0 con 0x7f333000a530 2024-09-17T18:33:48.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f334aeed640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:48.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f334aeed640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3344134d80 con 0x7f334413a150 2024-09-17T18:33:48.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f334aeed640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:48.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f334aeed640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f334409de60 con 0x7f333000aa40 2024-09-17T18:33:48.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f334aeed640 10 monclient(hunting): _renew_subs 2024-09-17T18:33:48.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f334aeed640 10 monclient(hunting): authenticate will time out at 1632.647416s 2024-09-17T18:33:48.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f3348c89640 1 -- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f333000a530 msgr2=0x7f33441336b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:33:48.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f3348c89640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f333000a530 0x7f33441336b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:33:48.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f333bfff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f333000aa40 0x7f3344133bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:48.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f333bfff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f333000aa40 0x7f3344133bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:57342/0 (socket says 172.21.15.121:57342) 2024-09-17T18:33:48.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f333bfff640 1 -- 172.21.15.121:0/2194156873 learned_addr learned my addr 172.21.15.121:0/2194156873 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:33:48.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f333bfff640 10 monclient(hunting): get_auth_request con 0x7f333000aa40 auth_method 0 2024-09-17T18:33:48.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f333bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:48.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f333bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:48.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f333bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:48.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f333bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:48.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f333bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:48.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f333bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:48.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f334948a640 1 --2- 172.21.15.121:0/2194156873 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f334413a150 0x7f33441cd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:48.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f334948a640 10 monclient(hunting): get_auth_request con 0x7f334413a150 auth_method 0 2024-09-17T18:33:48.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f334948a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:48.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f334948a640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:48.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f334948a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:48.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f333bfff640 10 monclient(hunting): handle_auth_done global_id 64427 payload 824 2024-09-17T18:33:48.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f333bfff640 1 -- 172.21.15.121:0/2194156873 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f334413a150 msgr2=0x7f33441cd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:48.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f333bfff640 1 --2- 172.21.15.121:0/2194156873 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f334413a150 0x7f33441cd490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:48.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f333bfff640 1 -- 172.21.15.121:0/2194156873 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f333000a530 msgr2=0x7f33441336b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:33:48.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f333bfff640 1 --2- 172.21.15.121:0/2194156873 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f333000a530 0x7f33441336b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:48.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f333bfff640 10 monclient: _finish_hunting 0 2024-09-17T18:33:48.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f333bfff640 1 monclient: found mon.noname-c 2024-09-17T18:33:48.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f333bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:33:48.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f333bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:33:48.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f333bfff640 1 -- 172.21.15.121:0/2194156873 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3344124170 con 0x7f333000aa40 2024-09-17T18:33:48.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.931+0000 7f334948a640 1 --2- 172.21.15.121:0/2194156873 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f334413a150 0x7f33441cd490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T18:33:48.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.934+0000 7f333bfff640 1 -- 172.21.15.121:0/2194156873 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f333000aa40 msgr2=0x7f3344133bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:48.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.934+0000 7f333bfff640 1 --2- 172.21.15.121:0/2194156873 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f333000aa40 0x7f3344133bf0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f333400f4c0 tx=0x7f333400f980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:48.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.934+0000 7f33397fa640 10 client.?.objecter ms_handle_connect 0x7f333000aa40 2024-09-17T18:33:48.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.934+0000 7f33397fa640 10 client.?.objecter resend_mon_ops 2024-09-17T18:33:48.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.934+0000 7f33397fa640 1 -- 172.21.15.121:0/2194156873 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f333401b070 con 0x7f333000aa40 2024-09-17T18:33:48.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.934+0000 7f33397fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:48.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.934+0000 7f33397fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:33:48.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.934+0000 7f33397fa640 10 monclient: dump: 2024-09-17T18:33:48.945 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:48.945 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:48.945 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:48.945 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:48.945 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:48.945 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:48.945 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:48.945 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:48.945 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:48.945 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:48.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.934+0000 7f33397fa640 10 monclient: _finish_auth 0 2024-09-17T18:33:48.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.934+0000 7f33397fa640 10 monclient: _check_auth_tickets 2024-09-17T18:33:48.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.934+0000 7f33397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:33:48.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.934+0000 7f33397fa640 1 -- 172.21.15.121:0/2194156873 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3334015b80 con 0x7f333000aa40 2024-09-17T18:33:48.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.934+0000 7f334aeed640 5 monclient: authenticate success, global_id 64427 2024-09-17T18:33:48.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.934+0000 7f334aeed640 10 monclient: _renew_subs 2024-09-17T18:33:48.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.934+0000 7f334aeed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:33:48.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.934+0000 7f334aeed640 1 -- 172.21.15.121:0/2194156873 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3344139ee0 con 0x7f333000aa40 2024-09-17T18:33:48.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.934+0000 7f33397fa640 10 monclient: handle_config config(0 keys) 2024-09-17T18:33:48.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.934+0000 7f33397fa640 1 -- 172.21.15.121:0/2194156873 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3334020460 con 0x7f333000aa40 2024-09-17T18:33:48.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.934+0000 7f33397fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:48.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.934+0000 7f33397fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:33:48.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.934+0000 7f33397fa640 10 monclient: dump: 2024-09-17T18:33:48.946 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:48.947 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:48.947 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:48.947 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:48.947 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:48.947 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:48.947 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:48.947 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:48.947 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:48.947 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:48.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.934+0000 7f334aeed640 10 client.64427.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:33:48.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.934+0000 7f334aeed640 10 monclient: _renew_subs 2024-09-17T18:33:48.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.934+0000 7f334aeed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:33:48.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.934+0000 7f334aeed640 1 -- 172.21.15.121:0/2194156873 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f334409e240 con 0x7f333000aa40 2024-09-17T18:33:48.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.934+0000 7f334aeed640 1 librados: init done 2024-09-17T18:33:48.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.934+0000 7f333affd640 4 set_mon_vals no callback set 2024-09-17T18:33:48.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.935+0000 7f334aeed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:33:48.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.935+0000 7f334aeed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:33:48.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.935+0000 7f334aeed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:33:48.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.935+0000 7f33397fa640 1 -- 172.21.15.121:0/2194156873 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f3334020640 con 0x7f333000aa40 2024-09-17T18:33:48.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.935+0000 7f334aeed640 1 -- 172.21.15.121:0/2194156873 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3344134130 con 0x7f333000aa40 2024-09-17T18:33:48.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.935+0000 7f33397fa640 1 --2- 172.21.15.121:0/2194156873 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3330041a70 0x7f3330043f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:48.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.935+0000 7f33397fa640 1 -- 172.21.15.121:0/2194156873 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(567..567 src has 247..567) ==== 5897+0+0 (secure 0 0 0) 0x7f3334054b50 con 0x7f333000aa40 2024-09-17T18:33:48.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.935+0000 7f33397fa640 10 client.64427.objecter ms_dispatch 0x7f3344000b90 osd_map(567..567 src has 247..567) 2024-09-17T18:33:48.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.935+0000 7f33397fa640 3 client.64427.objecter handle_osd_map got epochs [567,567] > 0 2024-09-17T18:33:48.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.935+0000 7f33397fa640 3 client.64427.objecter handle_osd_map decoding full epoch 567 2024-09-17T18:33:48.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.936+0000 7f33397fa640 20 client.64427.objecter dump_active .. 0 homeless 2024-09-17T18:33:48.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.936+0000 7f3348c89640 1 --2- 172.21.15.121:0/2194156873 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3330041a70 0x7f3330043f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:48.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.936+0000 7f3348c89640 10 monclient: get_auth_request con 0x7f3330041a70 auth_method 0 2024-09-17T18:33:48.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.939+0000 7f3348c89640 1 --2- 172.21.15.121:0/2194156873 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3330041a70 0x7f3330043f30 secure :-1 s=READY pgs=4060 cs=0 l=1 rev1=1 crypto rx=0x7f333000aa10 tx=0x7f3340007000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:48.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.939+0000 7f33397fa640 10 client.64427.objecter ms_handle_connect 0x7f3330041a70 2024-09-17T18:33:48.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.940+0000 7f33397fa640 1 -- 172.21.15.121:0/2194156873 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f333401e360 con 0x7f333000aa40 2024-09-17T18:33:48.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.940+0000 7f33397fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:33:48.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:48.940+0000 7f33397fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:33:49.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.187+0000 7f334aeed640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:33:49.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.187+0000 7f334aeed640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:33:49.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.187+0000 7f334aeed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:33:49.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.187+0000 7f334aeed640 1 -- 172.21.15.121:0/2194156873 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f33440a2820 con 0x7f333000aa40 2024-09-17T18:33:49.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.187+0000 7f33397fa640 1 -- 172.21.15.121:0/2194156873 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f33340530f0 con 0x7f333000aa40 2024-09-17T18:33:49.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.187+0000 7f33397fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:33:49.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.187+0000 7f33397fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:33:49.190 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":56,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:33:49.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.189+0000 7f334aeed640 10 librados: watch_flush enter 2024-09-17T18:33:49.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.189+0000 7f334aeed640 10 librados: watch_flush exit 2024-09-17T18:33:49.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.189+0000 7f334aeed640 20 client.64427.objecter shutdown clearing up homeless session... 2024-09-17T18:33:49.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.189+0000 7f334aeed640 10 client.64427.objecter successfully canceled tick 2024-09-17T18:33:49.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.190+0000 7f334aeed640 1 -- 172.21.15.121:0/2194156873 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3330041a70 msgr2=0x7f3330043f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:49.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.190+0000 7f334aeed640 1 --2- 172.21.15.121:0/2194156873 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3330041a70 0x7f3330043f30 secure :-1 s=READY pgs=4060 cs=0 l=1 rev1=1 crypto rx=0x7f333000aa10 tx=0x7f3340007000 comp rx=0 tx=0).stop 2024-09-17T18:33:49.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.190+0000 7f334aeed640 10 monclient: shutdown 2024-09-17T18:33:49.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.190+0000 7f334aeed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:33:49.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.190+0000 7f334aeed640 1 -- 172.21.15.121:0/2194156873 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f333000aa40 msgr2=0x7f3344133bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:49.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.190+0000 7f334aeed640 1 --2- 172.21.15.121:0/2194156873 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f333000aa40 0x7f3344133bf0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f333400f4c0 tx=0x7f333400f980 comp rx=0 tx=0).stop 2024-09-17T18:33:49.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.190+0000 7f334aeed640 1 -- 172.21.15.121:0/2194156873 shutdown_connections 2024-09-17T18:33:49.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.190+0000 7f334aeed640 1 --2- 172.21.15.121:0/2194156873 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3330041a70 0x7f3330043f30 unknown :-1 s=CLOSED pgs=4060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:49.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.190+0000 7f334aeed640 1 --2- 172.21.15.121:0/2194156873 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f334413a150 0x7f33441cd490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:49.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.190+0000 7f334aeed640 1 --2- 172.21.15.121:0/2194156873 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f333000aa40 0x7f3344133bf0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:49.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.190+0000 7f334aeed640 1 --2- 172.21.15.121:0/2194156873 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f333000a530 0x7f33441336b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:49.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.190+0000 7f334aeed640 1 -- 172.21.15.121:0/2194156873 >> 172.21.15.121:0/2194156873 conn(0x7f33440a3b50 msgr2=0x7f33440a1950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:33:49.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.190+0000 7f334aeed640 1 -- 172.21.15.121:0/2194156873 shutdown_connections 2024-09-17T18:33:49.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.190+0000 7f334aeed640 1 -- 172.21.15.121:0/2194156873 wait complete. 2024-09-17T18:33:49.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.190+0000 7f334aeed640 1 librados: shutdown 2024-09-17T18:33:49.202 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-17T18:33:49.203 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-17T18:33:49.203 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T18:33:49.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.298+0000 7fd63307f640 10 monclient: get_monmap_and_config 2024-09-17T18:33:49.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.298+0000 7fd63307f640 10 monclient: build_initial_monmap 2024-09-17T18:33:49.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.298+0000 7fd63307f640 10 monclient: monmap: 2024-09-17T18:33:49.300 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:33:49.300 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:49.301 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:33:49.299089+0000 2024-09-17T18:33:49.301 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:33:49.299089+0000 2024-09-17T18:33:49.301 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:33:49.301 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:49.301 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:33:49.301 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:33:49.301 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:33:49.301 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:33:49.301 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:33:49.301 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:49.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.299+0000 7fd63307f640 1 Processor -- start 2024-09-17T18:33:49.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.299+0000 7fd63307f640 1 -- start start 2024-09-17T18:33:49.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.299+0000 7fd63307f640 10 monclient: init 2024-09-17T18:33:49.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.299+0000 7fd63307f640 10 monclient: _reopen_session rank -1 2024-09-17T18:33:49.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.299+0000 7fd63307f640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-17T18:33:49.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.299+0000 7fd63307f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd62c0a0a00 addr v1:172.21.15.121:6789/0 2024-09-17T18:33:49.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.299+0000 7fd63307f640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fd62c137c90 0x7fd62c13a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:49.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.299+0000 7fd63307f640 10 monclient(hunting): picked mon.noname-c con 0x7fd62c137c90 addr v2:172.21.15.173:3300/0 2024-09-17T18:33:49.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.299+0000 7fd63307f640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd62c13a5c0 0x7fd62c13c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:49.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.299+0000 7fd63307f640 10 monclient(hunting): picked mon.noname-a con 0x7fd62c13a5c0 addr v2:172.21.15.121:3300/0 2024-09-17T18:33:49.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.299+0000 7fd63307f640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd62c0a3820 con 0x7fd62c0a0a00 2024-09-17T18:33:49.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.299+0000 7fd63307f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:49.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.299+0000 7fd63307f640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fd62c0a2ea0 con 0x7fd62c13a5c0 2024-09-17T18:33:49.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.299+0000 7fd63307f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:49.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.299+0000 7fd63307f640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fd62c09e800 con 0x7fd62c137c90 2024-09-17T18:33:49.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.299+0000 7fd63307f640 10 monclient(hunting): _renew_subs 2024-09-17T18:33:49.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.299+0000 7fd63307f640 10 monclient(hunting): authenticate will time out at 1633.015797s 2024-09-17T18:33:49.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.299+0000 7fd630e1b640 1 -- >> v1:172.21.15.121:6789/0 conn(0x7fd62c0a0a00 legacy=0x7fd62c0a0de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.121:6789/0 2024-09-17T18:33:49.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.299+0000 7fd63161c640 1 -- >> v2:172.21.15.121:3300/0 conn(0x7fd62c13a5c0 msgr2=0x7fd62c13c9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:33:49.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.299+0000 7fd63161c640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd62c13a5c0 0x7fd62c13c9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:33:49.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.300+0000 7fd62bfff640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fd62c137c90 0x7fd62c13a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:49.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.300+0000 7fd62bfff640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fd62c137c90 0x7fd62c13a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.121:36038/0 (socket says 172.21.15.121:36038) 2024-09-17T18:33:49.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.300+0000 7fd62bfff640 1 -- 172.21.15.121:0/1722361211 learned_addr learned my addr 172.21.15.121:0/1722361211 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:33:49.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.300+0000 7fd62bfff640 10 monclient(hunting): get_auth_request con 0x7fd62c137c90 auth_method 0 2024-09-17T18:33:49.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.300+0000 7fd62bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:49.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.300+0000 7fd62bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:49.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.300+0000 7fd62bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:49.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.300+0000 7fd62bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:49.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.300+0000 7fd62bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:49.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.300+0000 7fd62bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:49.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.300+0000 7fd62bfff640 10 monclient(hunting): handle_auth_done global_id 84223 payload 274 2024-09-17T18:33:49.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.300+0000 7fd62bfff640 1 -- 172.21.15.121:0/1722361211 >> v2:172.21.15.121:3300/0 conn(0x7fd62c13a5c0 msgr2=0x7fd62c13c9b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:33:49.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.300+0000 7fd62bfff640 1 --2- 172.21.15.121:0/1722361211 >> v2:172.21.15.121:3300/0 conn(0x7fd62c13a5c0 0x7fd62c13c9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:49.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.300+0000 7fd62bfff640 1 -- 172.21.15.121:0/1722361211 >> v1:172.21.15.121:6789/0 conn(0x7fd62c0a0a00 legacy=0x7fd62c0a0de0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:33:49.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.300+0000 7fd62bfff640 10 monclient: _finish_hunting 0 2024-09-17T18:33:49.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.300+0000 7fd62bfff640 1 monclient: found mon.noname-c 2024-09-17T18:33:49.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.300+0000 7fd62bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:33:49.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.300+0000 7fd62bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.173:3300/0 2024-09-17T18:33:49.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.300+0000 7fd62bfff640 1 -- 172.21.15.121:0/1722361211 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd62c0a2c30 con 0x7fd62c137c90 2024-09-17T18:33:49.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.303+0000 7fd62bfff640 1 -- 172.21.15.121:0/1722361211 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd62c137c90 msgr2=0x7fd62c13a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:49.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.303+0000 7fd62bfff640 1 --2- 172.21.15.121:0/1722361211 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd62c137c90 0x7fd62c13a080 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd62c0a3080 tx=0x7fd61402a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2343defbece7b5c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:49.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.303+0000 7fd62b7fe640 1 -- 172.21.15.121:0/1722361211 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd614037070 con 0x7fd62c137c90 2024-09-17T18:33:49.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.303+0000 7fd62b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:49.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.303+0000 7fd62b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:33:49.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.303+0000 7fd62b7fe640 10 monclient: dump: 2024-09-17T18:33:49.315 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:49.315 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:49.315 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:49.315 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:49.315 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:49.315 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:49.315 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:49.316 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:49.316 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:49.316 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:49.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.303+0000 7fd62b7fe640 10 monclient: _finish_auth 0 2024-09-17T18:33:49.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.303+0000 7fd62b7fe640 10 monclient: _check_auth_tickets 2024-09-17T18:33:49.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.303+0000 7fd62b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:33:49.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.303+0000 7fd63307f640 5 monclient: authenticate success, global_id 84223 2024-09-17T18:33:49.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.303+0000 7fd63307f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:33:49.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.303+0000 7fd62b7fe640 1 -- 172.21.15.121:0/1722361211 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd614032640 con 0x7fd62c137c90 2024-09-17T18:33:49.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.303+0000 7fd62b7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:33:49.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.303+0000 7fd63307f640 10 monclient: get_monmap_and_config success 2024-09-17T18:33:49.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.303+0000 7fd63307f640 4 set_mon_vals no callback set 2024-09-17T18:33:49.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.303+0000 7fd62b7fe640 1 -- 172.21.15.121:0/1722361211 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd614032a10 con 0x7fd62c137c90 2024-09-17T18:33:49.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.303+0000 7fd62b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:49.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.303+0000 7fd62b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:33:49.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.303+0000 7fd62b7fe640 10 monclient: dump: 2024-09-17T18:33:49.317 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:49.317 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:49.317 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:49.317 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:49.317 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:49.317 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:49.317 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:49.317 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:49.318 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:49.318 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:49.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.304+0000 7fd63307f640 10 monclient: shutdown 2024-09-17T18:33:49.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.304+0000 7fd63307f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:33:49.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.304+0000 7fd63307f640 1 -- 172.21.15.121:0/1722361211 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd62c137c90 msgr2=0x7fd62c13a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:49.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.304+0000 7fd63307f640 1 --2- 172.21.15.121:0/1722361211 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd62c137c90 0x7fd62c13a080 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd62c0a3080 tx=0x7fd61402a100 comp rx=0 tx=0).stop 2024-09-17T18:33:49.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.304+0000 7fd63307f640 1 -- 172.21.15.121:0/1722361211 shutdown_connections 2024-09-17T18:33:49.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.304+0000 7fd63307f640 1 --2- 172.21.15.121:0/1722361211 >> v2:172.21.15.121:3300/0 conn(0x7fd62c13a5c0 0x7fd62c13c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:49.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.304+0000 7fd63307f640 1 --2- 172.21.15.121:0/1722361211 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd62c137c90 0x7fd62c13a080 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:49.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.304+0000 7fd63307f640 1 -- 172.21.15.121:0/1722361211 >> 172.21.15.121:0/1722361211 conn(0x7fd62c0a7830 msgr2=0x7fd62c0a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:33:49.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.304+0000 7fd63307f640 1 -- 172.21.15.121:0/1722361211 shutdown_connections 2024-09-17T18:33:49.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.304+0000 7fd63307f640 1 -- 172.21.15.121:0/1722361211 wait complete. 2024-09-17T18:33:49.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.304+0000 7fd63307f640 10 monclient: build_initial_monmap 2024-09-17T18:33:49.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.304+0000 7fd63307f640 10 monclient: monmap: 2024-09-17T18:33:49.319 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:33:49.319 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:33:49.319 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:33:49.319 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:33:49.319 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:33:49.319 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:49.319 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:33:49.319 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:33:49.319 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:33:49.320 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:49.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.304+0000 7fd63307f640 1 librados: starting msgr at 2024-09-17T18:33:49.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.304+0000 7fd63307f640 1 librados: starting objecter 2024-09-17T18:33:49.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.304+0000 7fd63307f640 1 Processor -- start 2024-09-17T18:33:49.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.305+0000 7fd63307f640 1 -- start start 2024-09-17T18:33:49.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.305+0000 7fd63307f640 1 librados: setting wanted keys 2024-09-17T18:33:49.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.305+0000 7fd63307f640 1 librados: calling monclient init 2024-09-17T18:33:49.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.305+0000 7fd63307f640 10 monclient: init 2024-09-17T18:33:49.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.305+0000 7fd63307f640 10 monclient: _reopen_session rank -1 2024-09-17T18:33:49.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.305+0000 7fd63307f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T18:33:49.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.305+0000 7fd63307f640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd62c0a0a00 0x7fd62c134570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:49.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.305+0000 7fd63307f640 10 monclient(hunting): picked mon.noname-a con 0x7fd62c0a0a00 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:33:49.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.305+0000 7fd63307f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd62c137c90 0x7fd62c134ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:49.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.305+0000 7fd63307f640 10 monclient(hunting): picked mon.noname-c con 0x7fd62c137c90 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:33:49.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.305+0000 7fd63307f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd62c13a5c0 0x7fd62c11d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:49.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.305+0000 7fd63307f640 10 monclient(hunting): picked mon.noname-b con 0x7fd62c13a5c0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:33:49.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.305+0000 7fd63307f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:49.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.305+0000 7fd63307f640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd62c09de60 con 0x7fd62c0a0a00 2024-09-17T18:33:49.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.305+0000 7fd63307f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:49.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.305+0000 7fd63307f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd62c09dce0 con 0x7fd62c13a5c0 2024-09-17T18:33:49.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.305+0000 7fd63307f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:49.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.305+0000 7fd63307f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd62c0a3120 con 0x7fd62c137c90 2024-09-17T18:33:49.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.305+0000 7fd63307f640 10 monclient(hunting): _renew_subs 2024-09-17T18:33:49.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.305+0000 7fd63307f640 10 monclient(hunting): authenticate will time out at 1633.021662s 2024-09-17T18:33:49.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.305+0000 7fd630e1b640 1 -- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd62c0a0a00 msgr2=0x7fd62c134570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:33:49.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.305+0000 7fd630e1b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd62c0a0a00 0x7fd62c134570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:33:49.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.305+0000 7fd62bfff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd62c137c90 0x7fd62c134ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:49.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.305+0000 7fd62bfff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd62c137c90 0x7fd62c134ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:60924/0 (socket says 172.21.15.121:60924) 2024-09-17T18:33:49.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.305+0000 7fd62bfff640 1 -- 172.21.15.121:0/4189769423 learned_addr learned my addr 172.21.15.121:0/4189769423 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:33:49.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.305+0000 7fd62bfff640 10 monclient(hunting): get_auth_request con 0x7fd62c137c90 auth_method 0 2024-09-17T18:33:49.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.305+0000 7fd62bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:49.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.305+0000 7fd62bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:49.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.305+0000 7fd62bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:49.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.305+0000 7fd62bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:49.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.305+0000 7fd62bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:49.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.305+0000 7fd62bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:49.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.305+0000 7fd63161c640 1 --2- 172.21.15.121:0/4189769423 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd62c13a5c0 0x7fd62c11d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:49.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.306+0000 7fd62bfff640 10 monclient(hunting): handle_auth_done global_id 64430 payload 824 2024-09-17T18:33:49.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.306+0000 7fd62bfff640 1 -- 172.21.15.121:0/4189769423 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd62c13a5c0 msgr2=0x7fd62c11d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:49.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.306+0000 7fd62bfff640 1 --2- 172.21.15.121:0/4189769423 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd62c13a5c0 0x7fd62c11d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:49.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.306+0000 7fd62bfff640 1 -- 172.21.15.121:0/4189769423 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd62c0a0a00 msgr2=0x7fd62c134570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:33:49.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.306+0000 7fd62bfff640 1 --2- 172.21.15.121:0/4189769423 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd62c0a0a00 0x7fd62c134570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:49.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.306+0000 7fd62bfff640 10 monclient: _finish_hunting 0 2024-09-17T18:33:49.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.306+0000 7fd62bfff640 1 monclient: found mon.noname-c 2024-09-17T18:33:49.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.306+0000 7fd62bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:33:49.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.306+0000 7fd62bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:33:49.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.306+0000 7fd62bfff640 1 -- 172.21.15.121:0/4189769423 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd62c124d30 con 0x7fd62c137c90 2024-09-17T18:33:49.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.306+0000 7fd63161c640 10 monclient: get_auth_request con 0x7fd62c13a5c0 auth_method 0 2024-09-17T18:33:49.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.306+0000 7fd63161c640 1 --2- 172.21.15.121:0/4189769423 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd62c13a5c0 0x7fd62c11d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T18:33:49.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.308+0000 7fd62bfff640 1 -- 172.21.15.121:0/4189769423 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd62c137c90 msgr2=0x7fd62c134ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:49.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.308+0000 7fd62bfff640 1 --2- 172.21.15.121:0/4189769423 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd62c137c90 0x7fd62c134ab0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd6140090b0 tx=0x7fd6140046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:49.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.308+0000 7fd6297fa640 10 client.?.objecter ms_handle_connect 0x7fd62c137c90 2024-09-17T18:33:49.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.308+0000 7fd6297fa640 10 client.?.objecter resend_mon_ops 2024-09-17T18:33:49.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.308+0000 7fd6297fa640 1 -- 172.21.15.121:0/4189769423 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd614041070 con 0x7fd62c137c90 2024-09-17T18:33:49.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.308+0000 7fd6297fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:49.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.308+0000 7fd6297fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:33:49.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.308+0000 7fd6297fa640 10 monclient: dump: 2024-09-17T18:33:49.324 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:49.324 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:49.325 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:49.325 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:49.325 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:49.325 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:49.325 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:49.325 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:49.325 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:49.325 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:49.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.308+0000 7fd6297fa640 10 monclient: _finish_auth 0 2024-09-17T18:33:49.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.308+0000 7fd6297fa640 10 monclient: _check_auth_tickets 2024-09-17T18:33:49.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.308+0000 7fd6297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:33:49.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.308+0000 7fd63307f640 5 monclient: authenticate success, global_id 64430 2024-09-17T18:33:49.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.308+0000 7fd63307f640 10 monclient: _renew_subs 2024-09-17T18:33:49.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.308+0000 7fd63307f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:33:49.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.308+0000 7fd63307f640 1 -- 172.21.15.121:0/4189769423 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd62c11ede0 con 0x7fd62c137c90 2024-09-17T18:33:49.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.308+0000 7fd6297fa640 1 -- 172.21.15.121:0/4189769423 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd61403ca80 con 0x7fd62c137c90 2024-09-17T18:33:49.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.308+0000 7fd6297fa640 10 monclient: handle_config config(0 keys) 2024-09-17T18:33:49.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.308+0000 7fd6297fa640 1 -- 172.21.15.121:0/4189769423 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd614037040 con 0x7fd62c137c90 2024-09-17T18:33:49.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.308+0000 7fd6297fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:49.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.308+0000 7fd6297fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:33:49.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.308+0000 7fd6297fa640 10 monclient: dump: 2024-09-17T18:33:49.326 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:49.326 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:49.326 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:49.326 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:49.326 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:49.326 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:49.327 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:49.327 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:49.327 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:49.327 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:49.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.308+0000 7fd63307f640 10 client.64430.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:33:49.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.308+0000 7fd63307f640 10 monclient: _renew_subs 2024-09-17T18:33:49.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.308+0000 7fd63307f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:33:49.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.308+0000 7fd63307f640 1 -- 172.21.15.121:0/4189769423 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd62c124a50 con 0x7fd62c137c90 2024-09-17T18:33:49.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.308+0000 7fd63307f640 1 librados: init done 2024-09-17T18:33:49.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.308+0000 7fd62affd640 4 set_mon_vals no callback set 2024-09-17T18:33:49.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.309+0000 7fd63307f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:33:49.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.309+0000 7fd63307f640 1 --2- 172.21.15.121:0/4189769423 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd62c11dbe0 0x7fd62c11dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:49.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.309+0000 7fd63307f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:49.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.309+0000 7fd63307f640 1 -- 172.21.15.121:0/4189769423 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd62c0a1fa0 con 0x7fd62c11dbe0 2024-09-17T18:33:49.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.309+0000 7fd630e1b640 1 --2- 172.21.15.121:0/4189769423 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd62c11dbe0 0x7fd62c11dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:49.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.309+0000 7fd630e1b640 10 monclient: get_auth_request con 0x7fd62c11dbe0 auth_method 0 2024-09-17T18:33:49.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.309+0000 7fd630e1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:49.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.309+0000 7fd630e1b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:49.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.309+0000 7fd630e1b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:49.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.309+0000 7fd6297fa640 1 -- 172.21.15.121:0/4189769423 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fd614044070 con 0x7fd62c137c90 2024-09-17T18:33:49.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.309+0000 7fd630e1b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:49.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.309+0000 7fd630e1b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:49.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.309+0000 7fd630e1b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:49.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.309+0000 7fd630e1b640 10 monclient(hunting): handle_auth_done global_id 64433 payload 824 2024-09-17T18:33:49.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.309+0000 7fd630e1b640 1 -- 172.21.15.121:0/4189769423 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd62c11f430 con 0x7fd62c11dbe0 2024-09-17T18:33:49.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.309+0000 7fd6297fa640 1 --2- 172.21.15.121:0/4189769423 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd60c040120 0x7fd60c0425e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:49.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.309+0000 7fd6297fa640 1 -- 172.21.15.121:0/4189769423 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(567..567 src has 247..567) ==== 5897+0+0 (secure 0 0 0) 0x7fd614074250 con 0x7fd62c137c90 2024-09-17T18:33:49.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.309+0000 7fd6297fa640 10 client.64430.objecter ms_dispatch 0x7fd62c000b90 osd_map(567..567 src has 247..567) 2024-09-17T18:33:49.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.309+0000 7fd6297fa640 3 client.64430.objecter handle_osd_map got epochs [567,567] > 0 2024-09-17T18:33:49.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.309+0000 7fd6297fa640 3 client.64430.objecter handle_osd_map decoding full epoch 567 2024-09-17T18:33:49.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.309+0000 7fd6297fa640 20 client.64430.objecter dump_active .. 0 homeless 2024-09-17T18:33:49.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.310+0000 7fd63161c640 1 --2- 172.21.15.121:0/4189769423 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd60c040120 0x7fd60c0425e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:49.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.310+0000 7fd63161c640 10 monclient: get_auth_request con 0x7fd60c040120 auth_method 0 2024-09-17T18:33:49.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.311+0000 7fd630e1b640 1 -- 172.21.15.121:0/4189769423 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd62c11dbe0 msgr2=0x7fd62c11dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:49.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.311+0000 7fd6297fa640 10 client.64430.objecter ms_handle_connect 0x7fd62c11dbe0 2024-09-17T18:33:49.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.311+0000 7fd6297fa640 10 client.64430.objecter resend_mon_ops 2024-09-17T18:33:49.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.311+0000 7fd630e1b640 1 --2- 172.21.15.121:0/4189769423 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd62c11dbe0 0x7fd62c11dfc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd624003980 tx=0x7fd62400b8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:49.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.312+0000 7fd6297fa640 1 -- 172.21.15.121:0/4189769423 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd624005270 con 0x7fd62c11dbe0 2024-09-17T18:33:49.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.312+0000 7fd6297fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:49.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.312+0000 7fd6297fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:33:49.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.312+0000 7fd6297fa640 10 monclient: dump: 2024-09-17T18:33:49.330 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:49.330 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:49.330 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:49.330 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:49.330 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:49.330 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:49.330 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:49.330 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:49.330 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:49.330 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:49.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.312+0000 7fd6297fa640 1 -- 172.21.15.121:0/4189769423 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd624015070 con 0x7fd62c11dbe0 2024-09-17T18:33:49.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.312+0000 7fd6297fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:33:49.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.312+0000 7fd6297fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:33:49.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.312+0000 7fd6297fa640 1 -- 172.21.15.121:0/4189769423 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd62c11dbe0 msgr2=0x7fd62c11dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:49.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.312+0000 7fd6297fa640 1 --2- 172.21.15.121:0/4189769423 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd62c11dbe0 0x7fd62c11dfc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd624003980 tx=0x7fd62400b8c0 comp rx=0 tx=0).stop 2024-09-17T18:33:49.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.312+0000 7fd6297fa640 1 -- 172.21.15.121:0/4189769423 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd62c11dbe0 msgr2=0x7fd62c11dfc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:33:49.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.312+0000 7fd6297fa640 1 --2- 172.21.15.121:0/4189769423 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd62c11dbe0 0x7fd62c11dfc0 secure :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd624003980 tx=0x7fd62400b8c0 comp rx=0 tx=0).stop 2024-09-17T18:33:49.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.312+0000 7fd63161c640 1 --2- 172.21.15.121:0/4189769423 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd60c040120 0x7fd60c0425e0 secure :-1 s=READY pgs=4061 cs=0 l=1 rev1=1 crypto rx=0x7fd62c11f060 tx=0x7fd61c007340 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:49.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.312+0000 7fd6297fa640 10 client.64430.objecter ms_handle_connect 0x7fd60c040120 2024-09-17T18:33:49.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.318+0000 7fd63307f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:33:49.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.318+0000 7fd63307f640 1 --2- 172.21.15.121:0/4189769423 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd62c11e500 0x7fd62c06f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:49.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.318+0000 7fd63307f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:49.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.318+0000 7fd63307f640 1 -- 172.21.15.121:0/4189769423 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd62c125250 con 0x7fd62c11e500 2024-09-17T18:33:49.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.318+0000 7fd630e1b640 1 --2- 172.21.15.121:0/4189769423 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd62c11e500 0x7fd62c06f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:49.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.318+0000 7fd630e1b640 10 monclient: get_auth_request con 0x7fd62c11e500 auth_method 0 2024-09-17T18:33:49.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.318+0000 7fd630e1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:49.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.318+0000 7fd630e1b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:49.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.318+0000 7fd630e1b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:49.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.318+0000 7fd630e1b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:49.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.318+0000 7fd630e1b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:49.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.318+0000 7fd630e1b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:49.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.318+0000 7fd630e1b640 10 monclient(hunting): handle_auth_done global_id 64436 payload 824 2024-09-17T18:33:49.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.318+0000 7fd630e1b640 1 -- 172.21.15.121:0/4189769423 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd62c06cd40 con 0x7fd62c11e500 2024-09-17T18:33:49.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.320+0000 7fd630e1b640 1 -- 172.21.15.121:0/4189769423 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd62c11e500 msgr2=0x7fd62c06f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:49.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.320+0000 7fd630e1b640 1 --2- 172.21.15.121:0/4189769423 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd62c11e500 0x7fd62c06f2c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd624012040 tx=0x7fd624005a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:49.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.320+0000 7fd6297fa640 10 client.64430.objecter ms_handle_connect 0x7fd62c11e500 2024-09-17T18:33:49.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.320+0000 7fd6297fa640 10 client.64430.objecter resend_mon_ops 2024-09-17T18:33:49.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.321+0000 7fd6297fa640 1 -- 172.21.15.121:0/4189769423 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd62401ae60 con 0x7fd62c11e500 2024-09-17T18:33:49.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.321+0000 7fd6297fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:49.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.321+0000 7fd6297fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:33:49.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.321+0000 7fd6297fa640 10 monclient: dump: 2024-09-17T18:33:49.335 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:49.335 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:49.335 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:49.335 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:49.335 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:49.335 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:49.335 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:49.335 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:49.335 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:49.335 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:49.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.321+0000 7fd6297fa640 1 -- 172.21.15.121:0/4189769423 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4709 (secure 0 0 0) 0x7fd624010400 con 0x7fd62c11e500 2024-09-17T18:33:49.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.321+0000 7fd6297fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:33:49.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.321+0000 7fd6297fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:33:49.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.321+0000 7fd6297fa640 1 -- 172.21.15.121:0/4189769423 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd62c11e500 msgr2=0x7fd62c06f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:49.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.321+0000 7fd6297fa640 1 --2- 172.21.15.121:0/4189769423 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd62c11e500 0x7fd62c06f2c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd624012040 tx=0x7fd624005a20 comp rx=0 tx=0).stop 2024-09-17T18:33:49.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.321+0000 7fd6297fa640 1 -- 172.21.15.121:0/4189769423 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd62c11e500 msgr2=0x7fd62c06f2c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:33:49.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.321+0000 7fd6297fa640 1 --2- 172.21.15.121:0/4189769423 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd62c11e500 0x7fd62c06f2c0 secure :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd624012040 tx=0x7fd624005a20 comp rx=0 tx=0).stop 2024-09-17T18:33:49.336 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:33:49.336 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:33:49.336 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:33:49.336 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:33:49.337 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 56, 2024-09-17T18:33:49.337 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 408053, 2024-09-17T18:33:49.337 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:33:49.337 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:33:49.337 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:33:49.337 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:33:49.337 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 3, 2024-09-17T18:33:49.337 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:33:49.337 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:33:49.337 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:33:49.337 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:33:49.337 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:33:49.338 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:33:49.338 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:33:49.338 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:33:49.338 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:33:49.338 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:33:49.338 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:33:49.338 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:33:49.338 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:33:49.338 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:33:49.338 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:33:49.338 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:33:49.338 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:33:49.338 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:33:49.338 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:33:49.338 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:33:49.339 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:33:49.339 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:33:49.339 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:33:49.339 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:33:49.339 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:33:49.339 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:33:49.339 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:33:49.339 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:33:49.339 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:49.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:49.339 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:33:49.339 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:33:49.340 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:33:49.340 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:33:49.340 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:33:49.340 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:33:49.340 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:33:49.340 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:33:49.340 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:33:49.340 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:33:49.340 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:33:49.340 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:33:49.340 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:33:49.340 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:33:49.341 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:33:49.341 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:33:49.341 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:33:49.341 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:33:49.341 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:33:49.341 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:33:49.341 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:33:49.341 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:33:49.341 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:33:49.341 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:33:49.341 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:33:49.341 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:33:49.341 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:33:49.342 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:33:49.342 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:33:49.342 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:33:49.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:49.342 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:33:49.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:49.342 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:33:49.342 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:33:49.342 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:33:49.342 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:33:49.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:49.342 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:33:49.343 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:33:49.343 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:49.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:49.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:49.343 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:33:49.343 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:33:49.343 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:49.343 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:49.343 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:49.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:49.343 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:33:49.343 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:33:49.343 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:33:49.343 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:33:49.344 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:33:49.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:49.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:49.344 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:33:49.344 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:33:49.344 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:33:49.344 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:33:49.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:49.344 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:33:49.344 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:33:49.344 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:49.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:49.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:49.345 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:33:49.345 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:33:49.345 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:49.345 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:49.345 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:49.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:49.345 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:33:49.345 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:33:49.345 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:33:49.345 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:33:49.345 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:33:49.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:49.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:49.345 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:33:49.346 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:33:49.346 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:33:49.346 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:33:49.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:49.346 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:33:49.346 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:33:49.346 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:49.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:49.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:49.346 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:33:49.346 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:33:49.346 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:49.346 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:49.347 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:49.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:49.347 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:33:49.347 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:33:49.347 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:33:49.347 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:33:49.347 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:33:49.347 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:49.347 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:49.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:49.347 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:33:49.347 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:33:49.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:49.348 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:33:49.348 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:33:49.348 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:33:49.348 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:49.348 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:33:49.348 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:33:49.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:49.348 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:33:49.348 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:33:49.348 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 6 2024-09-17T18:33:49.348 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:49.348 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:33:49.349 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:33:49.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:49.349 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:33:49.349 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:33:49.349 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 4 2024-09-17T18:33:49.349 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:49.349 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:49.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:49.349 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:33:49.349 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:33:49.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.321+0000 7fd63307f640 10 librados: watch_flush enter 2024-09-17T18:33:49.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.321+0000 7fd63307f640 10 librados: watch_flush exit 2024-09-17T18:33:49.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.321+0000 7fd63307f640 20 client.64430.objecter shutdown clearing up homeless session... 2024-09-17T18:33:49.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.321+0000 7fd63307f640 10 client.64430.objecter successfully canceled tick 2024-09-17T18:33:49.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.321+0000 7fd63307f640 1 -- 172.21.15.121:0/4189769423 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd60c040120 msgr2=0x7fd60c0425e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:49.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.321+0000 7fd63307f640 1 --2- 172.21.15.121:0/4189769423 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd60c040120 0x7fd60c0425e0 secure :-1 s=READY pgs=4061 cs=0 l=1 rev1=1 crypto rx=0x7fd62c11f060 tx=0x7fd61c007340 comp rx=0 tx=0).stop 2024-09-17T18:33:49.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.321+0000 7fd63307f640 10 monclient: shutdown 2024-09-17T18:33:49.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.321+0000 7fd63307f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:33:49.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.321+0000 7fd63307f640 1 -- 172.21.15.121:0/4189769423 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd62c137c90 msgr2=0x7fd62c134ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:49.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.321+0000 7fd63307f640 1 --2- 172.21.15.121:0/4189769423 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd62c137c90 0x7fd62c134ab0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd6140090b0 tx=0x7fd6140046c0 comp rx=0 tx=0).stop 2024-09-17T18:33:49.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.321+0000 7fd63161c640 1 -- 172.21.15.121:0/4189769423 reap_dead start 2024-09-17T18:33:49.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.321+0000 7fd63161c640 1 -- 172.21.15.121:0/4189769423 reap_dead start 2024-09-17T18:33:49.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.321+0000 7fd63307f640 1 -- 172.21.15.121:0/4189769423 shutdown_connections 2024-09-17T18:33:49.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.321+0000 7fd63307f640 1 -- 172.21.15.121:0/4189769423 >> 172.21.15.121:0/4189769423 conn(0x7fd62c0a7830 msgr2=0x7fd62c135a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:33:49.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.321+0000 7fd63307f640 1 -- 172.21.15.121:0/4189769423 shutdown_connections 2024-09-17T18:33:49.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.321+0000 7fd63307f640 1 -- 172.21.15.121:0/4189769423 wait complete. 2024-09-17T18:33:49.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.321+0000 7fd63307f640 1 librados: shutdown 2024-09-17T18:33:49.353 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T18:33:49.353 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-17T18:33:49.353 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T18:33:49.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.454+0000 7f7b8e2ed640 10 monclient: get_monmap_and_config 2024-09-17T18:33:49.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.454+0000 7f7b8e2ed640 10 monclient: build_initial_monmap 2024-09-17T18:33:49.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.454+0000 7f7b8e2ed640 10 monclient: monmap: 2024-09-17T18:33:49.456 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:33:49.457 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:49.457 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:33:49.455126+0000 2024-09-17T18:33:49.457 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:33:49.455126+0000 2024-09-17T18:33:49.457 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:33:49.457 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:49.457 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:33:49.457 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:33:49.457 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:33:49.457 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:33:49.457 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:33:49.457 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:49.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.455+0000 7f7b8e2ed640 1 Processor -- start 2024-09-17T18:33:49.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.455+0000 7f7b8e2ed640 1 -- start start 2024-09-17T18:33:49.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.455+0000 7f7b8e2ed640 10 monclient: init 2024-09-17T18:33:49.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.455+0000 7f7b8e2ed640 10 monclient: _reopen_session rank -1 2024-09-17T18:33:49.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.455+0000 7f7b8e2ed640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-17T18:33:49.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.455+0000 7f7b8e2ed640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b8809da70 0x7f7b881361c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:49.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.455+0000 7f7b8e2ed640 10 monclient(hunting): picked mon.noname-b con 0x7f7b8809da70 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:33:49.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.455+0000 7f7b8e2ed640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7b88136700 addr v1:172.21.15.121:6789/0 2024-09-17T18:33:49.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.455+0000 7f7b8e2ed640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f7b88136b00 0x7f7b881333d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:49.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.455+0000 7f7b8e2ed640 10 monclient(hunting): picked mon.noname-c con 0x7f7b88136b00 addr v2:172.21.15.173:3300/0 2024-09-17T18:33:49.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.455+0000 7f7b8e2ed640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b880a2d40 con 0x7f7b88136700 2024-09-17T18:33:49.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.455+0000 7f7b8e2ed640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:49.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.455+0000 7f7b8e2ed640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f7b880a0e20 con 0x7f7b88136b00 2024-09-17T18:33:49.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.455+0000 7f7b8e2ed640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:49.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.455+0000 7f7b8e2ed640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7b880a3020 con 0x7f7b8809da70 2024-09-17T18:33:49.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.455+0000 7f7b8e2ed640 10 monclient(hunting): _renew_subs 2024-09-17T18:33:49.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.455+0000 7f7b8e2ed640 10 monclient(hunting): authenticate will time out at 1633.171810s 2024-09-17T18:33:49.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.455+0000 7f7b877fe640 1 -- >> v1:172.21.15.121:6789/0 conn(0x7f7b88136700 legacy=0x7f7b88132cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.121:6789/0 2024-09-17T18:33:49.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.455+0000 7f7b87fff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b8809da70 0x7f7b881361c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:49.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.455+0000 7f7b87fff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b8809da70 0x7f7b881361c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:60944/0 (socket says 172.21.15.121:60944) 2024-09-17T18:33:49.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.455+0000 7f7b87fff640 1 -- 172.21.15.121:0/434709636 learned_addr learned my addr 172.21.15.121:0/434709636 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:33:49.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.455+0000 7f7b87fff640 10 monclient(hunting): get_auth_request con 0x7f7b8809da70 auth_method 0 2024-09-17T18:33:49.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.455+0000 7f7b87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:49.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.455+0000 7f7b87fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:49.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.455+0000 7f7b87fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:49.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.456+0000 7f7b87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:49.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.456+0000 7f7b87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:49.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.456+0000 7f7b87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:49.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.456+0000 7f7b87fff640 10 monclient(hunting): handle_auth_done global_id 64439 payload 274 2024-09-17T18:33:49.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.456+0000 7f7b8c88a640 1 --2- 172.21.15.121:0/434709636 >> v2:172.21.15.173:3300/0 conn(0x7f7b88136b00 0x7f7b881333d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:49.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.456+0000 7f7b87fff640 1 -- 172.21.15.121:0/434709636 >> v2:172.21.15.173:3300/0 conn(0x7f7b88136b00 msgr2=0x7f7b881333d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:33:49.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.456+0000 7f7b87fff640 1 --2- 172.21.15.121:0/434709636 >> v2:172.21.15.173:3300/0 conn(0x7f7b88136b00 0x7f7b881333d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:49.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.456+0000 7f7b87fff640 1 -- 172.21.15.121:0/434709636 >> v1:172.21.15.121:6789/0 conn(0x7f7b88136700 legacy=0x7f7b88132cc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:33:49.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.456+0000 7f7b87fff640 10 monclient: _finish_hunting 0 2024-09-17T18:33:49.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.456+0000 7f7b87fff640 1 monclient: found mon.noname-b 2024-09-17T18:33:49.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.456+0000 7f7b87fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:33:49.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.456+0000 7f7b87fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:33:49.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.456+0000 7f7b87fff640 1 -- 172.21.15.121:0/434709636 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b880a3300 con 0x7f7b8809da70 2024-09-17T18:33:49.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.458+0000 7f7b87fff640 1 -- 172.21.15.121:0/434709636 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b8809da70 msgr2=0x7f7b881361c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:49.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.458+0000 7f7b87fff640 1 --2- 172.21.15.121:0/434709636 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b8809da70 0x7f7b881361c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7b78009870 tx=0x7f7b7802a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=697f203e0c40c061 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:49.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.459+0000 7f7b86ffd640 1 -- 172.21.15.121:0/434709636 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b78037070 con 0x7f7b8809da70 2024-09-17T18:33:49.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.459+0000 7f7b86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:49.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.459+0000 7f7b86ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:33:49.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.459+0000 7f7b86ffd640 10 monclient: dump: 2024-09-17T18:33:49.465 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:49.466 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:49.466 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:49.466 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:49.466 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:49.466 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:49.466 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:49.466 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:49.466 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:49.466 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:49.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.459+0000 7f7b86ffd640 10 monclient: _finish_auth 0 2024-09-17T18:33:49.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.459+0000 7f7b86ffd640 10 monclient: _check_auth_tickets 2024-09-17T18:33:49.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.459+0000 7f7b86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:33:49.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.459+0000 7f7b8e2ed640 5 monclient: authenticate success, global_id 64439 2024-09-17T18:33:49.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.459+0000 7f7b86ffd640 1 -- 172.21.15.121:0/434709636 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b78032640 con 0x7f7b8809da70 2024-09-17T18:33:49.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.459+0000 7f7b86ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:33:49.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.459+0000 7f7b8e2ed640 10 monclient: get_monmap_and_config success 2024-09-17T18:33:49.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.459+0000 7f7b86ffd640 1 -- 172.21.15.121:0/434709636 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b78032a10 con 0x7f7b8809da70 2024-09-17T18:33:49.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.459+0000 7f7b86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:49.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.459+0000 7f7b8e2ed640 4 set_mon_vals no callback set 2024-09-17T18:33:49.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.459+0000 7f7b86ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:33:49.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.459+0000 7f7b86ffd640 10 monclient: dump: 2024-09-17T18:33:49.467 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:49.467 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:49.467 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:49.467 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:49.467 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:49.467 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:49.468 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:49.468 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:49.468 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:49.468 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:49.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.459+0000 7f7b8e2ed640 10 monclient: shutdown 2024-09-17T18:33:49.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.459+0000 7f7b8e2ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:33:49.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.459+0000 7f7b8e2ed640 1 -- 172.21.15.121:0/434709636 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b8809da70 msgr2=0x7f7b881361c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:49.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.459+0000 7f7b8e2ed640 1 --2- 172.21.15.121:0/434709636 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b8809da70 0x7f7b881361c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7b78009870 tx=0x7f7b7802a0a0 comp rx=0 tx=0).stop 2024-09-17T18:33:49.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.459+0000 7f7b8e2ed640 1 -- 172.21.15.121:0/434709636 shutdown_connections 2024-09-17T18:33:49.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.459+0000 7f7b8e2ed640 1 --2- 172.21.15.121:0/434709636 >> v2:172.21.15.173:3300/0 conn(0x7f7b88136b00 0x7f7b881333d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:49.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.459+0000 7f7b8e2ed640 1 --2- 172.21.15.121:0/434709636 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b8809da70 0x7f7b881361c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:49.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.459+0000 7f7b8e2ed640 1 -- 172.21.15.121:0/434709636 >> 172.21.15.121:0/434709636 conn(0x7f7b880a7830 msgr2=0x7f7b880a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:33:49.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.459+0000 7f7b8e2ed640 1 -- 172.21.15.121:0/434709636 shutdown_connections 2024-09-17T18:33:49.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.459+0000 7f7b8e2ed640 1 -- 172.21.15.121:0/434709636 wait complete. 2024-09-17T18:33:49.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.460+0000 7f7b8e2ed640 10 monclient: build_initial_monmap 2024-09-17T18:33:49.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.460+0000 7f7b8e2ed640 10 monclient: monmap: 2024-09-17T18:33:49.469 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:33:49.469 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:33:49.469 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:33:49.469 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:33:49.469 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:33:49.469 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:49.469 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:33:49.469 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:33:49.469 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:33:49.469 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:49.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.460+0000 7f7b8e2ed640 1 librados: starting msgr at 2024-09-17T18:33:49.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.460+0000 7f7b8e2ed640 1 librados: starting objecter 2024-09-17T18:33:49.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.460+0000 7f7b8e2ed640 1 Processor -- start 2024-09-17T18:33:49.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.460+0000 7f7b8e2ed640 1 -- start start 2024-09-17T18:33:49.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.460+0000 7f7b8e2ed640 1 librados: setting wanted keys 2024-09-17T18:33:49.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.460+0000 7f7b8e2ed640 1 librados: calling monclient init 2024-09-17T18:33:49.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.460+0000 7f7b8e2ed640 10 monclient: init 2024-09-17T18:33:49.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.461+0000 7f7b8e2ed640 10 monclient: _reopen_session rank -1 2024-09-17T18:33:49.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.461+0000 7f7b8e2ed640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T18:33:49.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.461+0000 7f7b8e2ed640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b8809da70 0x7f7b881cb740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:49.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.461+0000 7f7b8e2ed640 10 monclient(hunting): picked mon.noname-a con 0x7f7b8809da70 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:33:49.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.461+0000 7f7b8e2ed640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b88136700 0x7f7b8812fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:49.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.461+0000 7f7b8e2ed640 10 monclient(hunting): picked mon.noname-c con 0x7f7b88136700 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:33:49.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.461+0000 7f7b8e2ed640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b88136b00 0x7f7b88130000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:49.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.461+0000 7f7b8e2ed640 10 monclient(hunting): picked mon.noname-b con 0x7f7b88136b00 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:33:49.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.461+0000 7f7b87fff640 1 -- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b8809da70 msgr2=0x7f7b881cb740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:33:49.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.461+0000 7f7b8e2ed640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:49.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.461+0000 7f7b8e2ed640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7b880a3930 con 0x7f7b8809da70 2024-09-17T18:33:49.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.461+0000 7f7b87fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b8809da70 0x7f7b881cb740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:33:49.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.461+0000 7f7b8e2ed640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:49.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.461+0000 7f7b8e2ed640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7b88133910 con 0x7f7b88136b00 2024-09-17T18:33:49.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.461+0000 7f7b8e2ed640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:49.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.461+0000 7f7b8e2ed640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7b88074020 con 0x7f7b88136700 2024-09-17T18:33:49.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.461+0000 7f7b8e2ed640 10 monclient(hunting): _renew_subs 2024-09-17T18:33:49.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.461+0000 7f7b8e2ed640 10 monclient(hunting): authenticate will time out at 1633.177286s 2024-09-17T18:33:49.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.461+0000 7f7b877fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b88136700 0x7f7b8812fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:49.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.461+0000 7f7b877fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b88136700 0x7f7b8812fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:60958/0 (socket says 172.21.15.121:60958) 2024-09-17T18:33:49.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.461+0000 7f7b877fe640 1 -- 172.21.15.121:0/2276021221 learned_addr learned my addr 172.21.15.121:0/2276021221 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:33:49.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.461+0000 7f7b877fe640 10 monclient(hunting): get_auth_request con 0x7f7b88136700 auth_method 0 2024-09-17T18:33:49.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.461+0000 7f7b877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:49.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.461+0000 7f7b877fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:49.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.461+0000 7f7b877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:49.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.461+0000 7f7b877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:49.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.461+0000 7f7b877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:49.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.461+0000 7f7b877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:49.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.461+0000 7f7b8c88a640 1 --2- 172.21.15.121:0/2276021221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b88136b00 0x7f7b88130000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:49.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.461+0000 7f7b877fe640 10 monclient(hunting): handle_auth_done global_id 64442 payload 824 2024-09-17T18:33:49.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.461+0000 7f7b877fe640 1 -- 172.21.15.121:0/2276021221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b88136b00 msgr2=0x7f7b88130000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:49.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.461+0000 7f7b877fe640 1 --2- 172.21.15.121:0/2276021221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b88136b00 0x7f7b88130000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:49.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.461+0000 7f7b877fe640 1 -- 172.21.15.121:0/2276021221 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b8809da70 msgr2=0x7f7b881cb740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:33:49.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.461+0000 7f7b877fe640 1 --2- 172.21.15.121:0/2276021221 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b8809da70 0x7f7b881cb740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:49.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.461+0000 7f7b877fe640 10 monclient: _finish_hunting 0 2024-09-17T18:33:49.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.461+0000 7f7b877fe640 1 monclient: found mon.noname-c 2024-09-17T18:33:49.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.461+0000 7f7b877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:33:49.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.461+0000 7f7b877fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:33:49.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.461+0000 7f7b877fe640 1 -- 172.21.15.121:0/2276021221 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b8809e240 con 0x7f7b88136700 2024-09-17T18:33:49.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.461+0000 7f7b8c88a640 10 monclient: get_auth_request con 0x7f7b88136b00 auth_method 0 2024-09-17T18:33:49.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.461+0000 7f7b8c88a640 1 --2- 172.21.15.121:0/2276021221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b88136b00 0x7f7b88130000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T18:33:49.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.463+0000 7f7b877fe640 1 -- 172.21.15.121:0/2276021221 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b88136700 msgr2=0x7f7b8812fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:49.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.463+0000 7f7b877fe640 1 --2- 172.21.15.121:0/2276021221 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b88136700 0x7f7b8812fac0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c00b550 tx=0x7f7b7c00ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:49.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.463+0000 7f7b84ff9640 10 client.?.objecter ms_handle_connect 0x7f7b88136700 2024-09-17T18:33:49.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.463+0000 7f7b84ff9640 10 client.?.objecter resend_mon_ops 2024-09-17T18:33:49.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.463+0000 7f7b84ff9640 1 -- 172.21.15.121:0/2276021221 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c005070 con 0x7f7b88136700 2024-09-17T18:33:49.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.463+0000 7f7b84ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:49.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.463+0000 7f7b84ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:33:49.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.463+0000 7f7b84ff9640 10 monclient: dump: 2024-09-17T18:33:49.483 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:49.483 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:49.483 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:49.483 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:49.483 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:49.483 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:49.483 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:49.483 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:49.483 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:49.483 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:49.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.463+0000 7f7b84ff9640 10 monclient: _finish_auth 0 2024-09-17T18:33:49.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.463+0000 7f7b84ff9640 10 monclient: _check_auth_tickets 2024-09-17T18:33:49.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.463+0000 7f7b84ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:33:49.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.464+0000 7f7b8e2ed640 5 monclient: authenticate success, global_id 64442 2024-09-17T18:33:49.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.464+0000 7f7b84ff9640 1 -- 172.21.15.121:0/2276021221 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b7c015070 con 0x7f7b88136700 2024-09-17T18:33:49.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.464+0000 7f7b8e2ed640 10 monclient: _renew_subs 2024-09-17T18:33:49.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.464+0000 7f7b8e2ed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:33:49.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.464+0000 7f7b8e2ed640 1 -- 172.21.15.121:0/2276021221 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b88074300 con 0x7f7b88136700 2024-09-17T18:33:49.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.464+0000 7f7b84ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T18:33:49.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.464+0000 7f7b84ff9640 1 -- 172.21.15.121:0/2276021221 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c005450 con 0x7f7b88136700 2024-09-17T18:33:49.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.464+0000 7f7b84ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:49.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.464+0000 7f7b84ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:33:49.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.464+0000 7f7b84ff9640 10 monclient: dump: 2024-09-17T18:33:49.484 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:49.484 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:49.484 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:49.485 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:49.485 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:49.485 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:49.485 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:49.485 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:49.485 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:49.485 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:49.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.464+0000 7f7b8e2ed640 10 client.64442.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:33:49.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.464+0000 7f7b8e2ed640 10 monclient: _renew_subs 2024-09-17T18:33:49.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.464+0000 7f7b867fc640 4 set_mon_vals no callback set 2024-09-17T18:33:49.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.464+0000 7f7b8e2ed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:33:49.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.464+0000 7f7b8e2ed640 1 -- 172.21.15.121:0/2276021221 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b88072b00 con 0x7f7b88136700 2024-09-17T18:33:49.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.464+0000 7f7b8e2ed640 1 librados: init done 2024-09-17T18:33:49.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.464+0000 7f7b8e2ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:33:49.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.464+0000 7f7b8e2ed640 1 --2- 172.21.15.121:0/2276021221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b88070ca0 0x7f7b8802d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:49.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.464+0000 7f7b8e2ed640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:49.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.464+0000 7f7b8e2ed640 1 -- 172.21.15.121:0/2276021221 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7b88072d00 con 0x7f7b88070ca0 2024-09-17T18:33:49.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.465+0000 7f7b87fff640 1 --2- 172.21.15.121:0/2276021221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b88070ca0 0x7f7b8802d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:49.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.465+0000 7f7b84ff9640 1 -- 172.21.15.121:0/2276021221 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f7b7c0055f0 con 0x7f7b88136700 2024-09-17T18:33:49.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.465+0000 7f7b87fff640 10 monclient: get_auth_request con 0x7f7b88070ca0 auth_method 0 2024-09-17T18:33:49.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.465+0000 7f7b87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:49.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.465+0000 7f7b87fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:49.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.465+0000 7f7b87fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:49.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.465+0000 7f7b87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:49.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.465+0000 7f7b87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:49.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.465+0000 7f7b87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:49.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.465+0000 7f7b84ff9640 1 --2- 172.21.15.121:0/2276021221 >> v2:172.21.15.173:6812/2423786037 conn(0x7f7b74040150 0x7f7b74042610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:49.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.465+0000 7f7b84ff9640 1 -- 172.21.15.121:0/2276021221 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(568..568 src has 247..568) ==== 5897+0+0 (secure 0 0 0) 0x7f7b7c053830 con 0x7f7b88136700 2024-09-17T18:33:49.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.465+0000 7f7b87fff640 10 monclient(hunting): handle_auth_done global_id 84226 payload 824 2024-09-17T18:33:49.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.465+0000 7f7b84ff9640 10 client.64442.objecter ms_dispatch 0x7f7b88000b90 osd_map(568..568 src has 247..568) 2024-09-17T18:33:49.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.465+0000 7f7b84ff9640 3 client.64442.objecter handle_osd_map got epochs [568,568] > 0 2024-09-17T18:33:49.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.465+0000 7f7b84ff9640 3 client.64442.objecter handle_osd_map decoding full epoch 568 2024-09-17T18:33:49.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.465+0000 7f7b87fff640 1 -- 172.21.15.121:0/2276021221 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7b881361c0 con 0x7f7b88070ca0 2024-09-17T18:33:49.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.465+0000 7f7b84ff9640 20 client.64442.objecter dump_active .. 0 homeless 2024-09-17T18:33:49.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.465+0000 7f7b8c88a640 1 --2- 172.21.15.121:0/2276021221 >> v2:172.21.15.173:6812/2423786037 conn(0x7f7b74040150 0x7f7b74042610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:49.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.465+0000 7f7b8c88a640 10 monclient: get_auth_request con 0x7f7b74040150 auth_method 0 2024-09-17T18:33:49.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.467+0000 7f7b84ff9640 10 client.64442.objecter ms_handle_connect 0x7f7b88070ca0 2024-09-17T18:33:49.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.467+0000 7f7b84ff9640 10 client.64442.objecter resend_mon_ops 2024-09-17T18:33:49.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.467+0000 7f7b87fff640 1 -- 172.21.15.121:0/2276021221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b88070ca0 msgr2=0x7f7b8802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:49.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.467+0000 7f7b87fff640 1 --2- 172.21.15.121:0/2276021221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b88070ca0 0x7f7b8802d440 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7b8809e1a0 tx=0x7f7b780046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:49.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.467+0000 7f7b84ff9640 1 -- 172.21.15.121:0/2276021221 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b78041070 con 0x7f7b88070ca0 2024-09-17T18:33:49.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.467+0000 7f7b84ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:49.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.467+0000 7f7b84ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:33:49.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.467+0000 7f7b84ff9640 10 monclient: dump: 2024-09-17T18:33:49.488 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:49.488 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:49.488 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:49.488 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:49.488 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:49.489 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:49.489 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:49.489 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:49.489 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:49.489 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:49.489 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.468+0000 7f7b84ff9640 10 client.64442.objecter ms_handle_connect 0x7f7b74040150 2024-09-17T18:33:49.489 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.468+0000 7f7b8c88a640 1 --2- 172.21.15.121:0/2276021221 >> v2:172.21.15.173:6812/2423786037 conn(0x7f7b74040150 0x7f7b74042610 secure :-1 s=READY pgs=4062 cs=0 l=1 rev1=1 crypto rx=0x7f7b881cc5d0 tx=0x7f7b80007420 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:49.489 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.468+0000 7f7b84ff9640 1 -- 172.21.15.121:0/2276021221 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7b78009490 con 0x7f7b88070ca0 2024-09-17T18:33:49.489 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.468+0000 7f7b84ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:33:49.489 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.468+0000 7f7b84ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:33:49.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.468+0000 7f7b84ff9640 1 -- 172.21.15.121:0/2276021221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b88070ca0 msgr2=0x7f7b8802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:49.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.468+0000 7f7b84ff9640 1 --2- 172.21.15.121:0/2276021221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b88070ca0 0x7f7b8802d440 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7b8809e1a0 tx=0x7f7b780046c0 comp rx=0 tx=0).stop 2024-09-17T18:33:49.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.468+0000 7f7b84ff9640 1 -- 172.21.15.121:0/2276021221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b88070ca0 msgr2=0x7f7b8802d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:33:49.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.468+0000 7f7b84ff9640 1 --2- 172.21.15.121:0/2276021221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b88070ca0 0x7f7b8802d440 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:33:49.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.474+0000 7f7b8e2ed640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:33:49.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.474+0000 7f7b8e2ed640 1 --2- 172.21.15.121:0/2276021221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b88071080 0x7f7b8802d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:33:49.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.474+0000 7f7b8e2ed640 10 monclient(hunting): start opening mon connection 2024-09-17T18:33:49.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.474+0000 7f7b8e2ed640 1 -- 172.21.15.121:0/2276021221 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7b88137040 con 0x7f7b88071080 2024-09-17T18:33:49.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.474+0000 7f7b87fff640 1 --2- 172.21.15.121:0/2276021221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b88071080 0x7f7b8802d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:33:49.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.474+0000 7f7b87fff640 10 monclient: get_auth_request con 0x7f7b88071080 auth_method 0 2024-09-17T18:33:49.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.474+0000 7f7b87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:33:49.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.474+0000 7f7b87fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:33:49.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.474+0000 7f7b87fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:33:49.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.474+0000 7f7b87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:33:49.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.474+0000 7f7b87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:33:49.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.474+0000 7f7b87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:33:49.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.475+0000 7f7b87fff640 10 monclient(hunting): handle_auth_done global_id 84229 payload 824 2024-09-17T18:33:49.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.475+0000 7f7b87fff640 1 -- 172.21.15.121:0/2276021221 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7b88071fa0 con 0x7f7b88071080 2024-09-17T18:33:49.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.477+0000 7f7b84ff9640 10 client.64442.objecter ms_handle_connect 0x7f7b88071080 2024-09-17T18:33:49.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.477+0000 7f7b87fff640 1 -- 172.21.15.121:0/2276021221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b88071080 msgr2=0x7f7b8802d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:33:49.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.477+0000 7f7b84ff9640 10 client.64442.objecter resend_mon_ops 2024-09-17T18:33:49.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.477+0000 7f7b87fff640 1 --2- 172.21.15.121:0/2276021221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b88071080 0x7f7b8802d980 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7b78009870 tx=0x7f7b78009240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:33:49.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.477+0000 7f7b84ff9640 1 -- 172.21.15.121:0/2276021221 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b78037040 con 0x7f7b88071080 2024-09-17T18:33:49.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.477+0000 7f7b84ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:33:49.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.477+0000 7f7b84ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:33:49.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.477+0000 7f7b84ff9640 10 monclient: dump: 2024-09-17T18:33:49.493 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:33:49.493 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:33:49.493 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:49.493 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:33:49.493 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:33:49.493 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:33:49.494 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:33:49.494 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:33:49.494 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:33:49.494 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:33:49.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.478+0000 7f7b84ff9640 1 -- 172.21.15.121:0/2276021221 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7f7b78050440 con 0x7f7b88071080 2024-09-17T18:33:49.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.478+0000 7f7b84ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:33:49.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.478+0000 7f7b84ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:33:49.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.478+0000 7f7b84ff9640 1 -- 172.21.15.121:0/2276021221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b88071080 msgr2=0x7f7b8802d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:49.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.478+0000 7f7b84ff9640 1 --2- 172.21.15.121:0/2276021221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b88071080 0x7f7b8802d980 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7b78009870 tx=0x7f7b78009240 comp rx=0 tx=0).stop 2024-09-17T18:33:49.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.478+0000 7f7b84ff9640 1 -- 172.21.15.121:0/2276021221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b88071080 msgr2=0x7f7b8802d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:33:49.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.478+0000 7f7b84ff9640 1 --2- 172.21.15.121:0/2276021221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b88071080 0x7f7b8802d980 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7b78009870 tx=0x7f7b78009240 comp rx=0 tx=0).stop 2024-09-17T18:33:49.495 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:33:49.495 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:33:49.495 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:33:49.495 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "leader", 2024-09-17T18:33:49.495 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 56, 2024-09-17T18:33:49.495 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 154575, 2024-09-17T18:33:49.495 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:33:49.495 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:33:49.495 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:33:49.495 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:33:49.496 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 3, 2024-09-17T18:33:49.496 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:33:49.496 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:33:49.496 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:33:49.496 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:33:49.496 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:33:49.496 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:33:49.496 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:33:49.496 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:33:49.496 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:33:49.496 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:33:49.496 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:33:49.497 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:33:49.497 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:33:49.497 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:33:49.497 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:33:49.497 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:33:49.497 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:33:49.497 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:33:49.497 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:33:49.497 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:33:49.497 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:33:49.497 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:33:49.498 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:33:49.498 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:33:49.498 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:33:49.498 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:33:49.498 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:33:49.498 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:33:49.498 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:49.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:49.498 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:33:49.498 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:33:49.498 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:33:49.498 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:33:49.498 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:33:49.499 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:33:49.499 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:33:49.499 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:33:49.499 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:33:49.499 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:33:49.499 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:33:49.499 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:33:49.499 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:33:49.500 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:33:49.500 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:33:49.500 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:33:49.500 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:33:49.500 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:33:49.500 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:33:49.500 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:33:49.500 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:33:49.500 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:33:49.500 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:33:49.500 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:33:49.501 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:33:49.501 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:33:49.501 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:33:49.501 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:33:49.501 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:33:49.501 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:33:49.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:49.501 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:33:49.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:49.501 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:33:49.501 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:33:49.502 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:33:49.502 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:33:49.502 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:49.502 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:33:49.502 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:33:49.502 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:49.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:49.502 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:49.502 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:33:49.502 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:33:49.502 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:49.503 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:49.503 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:49.503 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:49.503 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:33:49.503 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:33:49.503 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:33:49.503 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:33:49.503 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:33:49.503 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:49.503 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:49.503 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:33:49.503 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:33:49.503 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:33:49.504 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:33:49.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:49.504 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:33:49.504 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:33:49.504 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:49.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:49.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:49.504 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:33:49.504 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:33:49.504 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:49.504 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:49.504 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:49.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:49.505 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:33:49.505 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:33:49.505 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:33:49.505 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:33:49.505 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:33:49.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:49.505 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:49.505 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:33:49.505 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:33:49.506 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:33:49.506 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:33:49.506 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:49.506 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:33:49.506 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:33:49.506 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:49.506 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:49.506 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:49.506 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:33:49.506 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:33:49.506 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:33:49.506 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:49.507 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:49.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:49.507 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:33:49.507 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:33:49.507 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:33:49.507 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:33:49.507 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:33:49.507 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:49.507 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:49.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:49.507 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:33:49.507 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:33:49.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:49.508 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:33:49.508 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:33:49.508 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:33:49.508 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:49.508 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:33:49.508 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:33:49.508 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:49.508 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:33:49.508 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:33:49.508 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 4 2024-09-17T18:33:49.508 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:49.509 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:33:49.509 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": [ 2024-09-17T18:33:49.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:33:49.509 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:33:49.509 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:33:49.509 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:33:49.509 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:33:49.509 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:33:49.509 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:33:49.509 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:33:49.509 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:33:49.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.478+0000 7f7b8e2ed640 10 librados: watch_flush enter 2024-09-17T18:33:49.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.478+0000 7f7b8e2ed640 10 librados: watch_flush exit 2024-09-17T18:33:49.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.478+0000 7f7b8e2ed640 20 client.64442.objecter shutdown clearing up homeless session... 2024-09-17T18:33:49.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.478+0000 7f7b8e2ed640 10 client.64442.objecter successfully canceled tick 2024-09-17T18:33:49.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.478+0000 7f7b8e2ed640 1 -- 172.21.15.121:0/2276021221 >> v2:172.21.15.173:6812/2423786037 conn(0x7f7b74040150 msgr2=0x7f7b74042610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:49.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.478+0000 7f7b8e2ed640 1 --2- 172.21.15.121:0/2276021221 >> v2:172.21.15.173:6812/2423786037 conn(0x7f7b74040150 0x7f7b74042610 secure :-1 s=READY pgs=4062 cs=0 l=1 rev1=1 crypto rx=0x7f7b881cc5d0 tx=0x7f7b80007420 comp rx=0 tx=0).stop 2024-09-17T18:33:49.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.478+0000 7f7b8e2ed640 10 monclient: shutdown 2024-09-17T18:33:49.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.478+0000 7f7b8e2ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:33:49.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.478+0000 7f7b8e2ed640 1 -- 172.21.15.121:0/2276021221 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b88136700 msgr2=0x7f7b8812fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:33:49.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.478+0000 7f7b8e2ed640 1 --2- 172.21.15.121:0/2276021221 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b88136700 0x7f7b8812fac0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c00b550 tx=0x7f7b7c00ba10 comp rx=0 tx=0).stop 2024-09-17T18:33:49.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.478+0000 7f7b8c88a640 1 -- 172.21.15.121:0/2276021221 reap_dead start 2024-09-17T18:33:49.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.478+0000 7f7b8c88a640 1 -- 172.21.15.121:0/2276021221 reap_dead start 2024-09-17T18:33:49.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.478+0000 7f7b8e2ed640 1 -- 172.21.15.121:0/2276021221 shutdown_connections 2024-09-17T18:33:49.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.478+0000 7f7b8e2ed640 1 -- 172.21.15.121:0/2276021221 >> 172.21.15.121:0/2276021221 conn(0x7f7b880a7830 msgr2=0x7f7b88132bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:33:49.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.478+0000 7f7b8e2ed640 1 -- 172.21.15.121:0/2276021221 shutdown_connections 2024-09-17T18:33:49.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.478+0000 7f7b8e2ed640 1 -- 172.21.15.121:0/2276021221 wait complete. 2024-09-17T18:33:49.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:33:49.478+0000 7f7b8e2ed640 1 librados: shutdown 2024-09-17T18:33:49.513 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T18:33:49.513 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-17T18:35:19.514 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-17T18:35:19.514 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-17T18:35:19.515 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-17T18:35:19.519 INFO:tasks.ceph.mon.a:Started 2024-09-17T18:35:19.519 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T18:35:19.519 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:35:19.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.626+0000 7f1fcf8dd640 10 monclient: get_monmap_and_config 2024-09-17T18:35:19.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.626+0000 7f1fcf8dd640 10 monclient: build_initial_monmap 2024-09-17T18:35:19.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.626+0000 7f1fcf8dd640 10 monclient: monmap: 2024-09-17T18:35:19.628 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:35:19.628 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:19.628 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:35:19.626813+0000 2024-09-17T18:35:19.628 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:35:19.626813+0000 2024-09-17T18:35:19.628 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:35:19.628 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:19.628 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:35:19.629 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:35:19.629 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:35:19.629 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:35:19.629 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:35:19.629 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:19.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.626+0000 7f1fcf8dd640 1 Processor -- start 2024-09-17T18:35:19.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.627+0000 7f1fcf8dd640 1 -- start start 2024-09-17T18:35:19.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.627+0000 7f1fcf8dd640 10 monclient: init 2024-09-17T18:35:19.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.627+0000 7f1fcf8dd640 10 monclient: _reopen_session rank -1 2024-09-17T18:35:19.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.627+0000 7f1fcf8dd640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-17T18:35:19.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.627+0000 7f1fcf8dd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1fc80a5c00 addr v1:172.21.15.173:6789/0 2024-09-17T18:35:19.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.627+0000 7f1fcf8dd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1fc80a5fe0 0x7f1fc80a63c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:19.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.627+0000 7f1fcf8dd640 10 monclient(hunting): picked mon.noname-b con 0x7f1fc80a5fe0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:35:19.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.627+0000 7f1fcf8dd640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f1fc813dc00 0x7f1fc813fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:19.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.627+0000 7f1fcf8dd640 10 monclient(hunting): picked mon.noname-c con 0x7f1fc813dc00 addr v2:172.21.15.173:3300/0 2024-09-17T18:35:19.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.627+0000 7f1fcf8dd640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fc80a8a80 con 0x7f1fc80a5c00 2024-09-17T18:35:19.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.627+0000 7f1fcf8dd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:19.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.627+0000 7f1fcf8dd640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f1fc80a74d0 con 0x7f1fc813dc00 2024-09-17T18:35:19.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.627+0000 7f1fcf8dd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:19.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.627+0000 7f1fcf8dd640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1fc809cb90 con 0x7f1fc80a5fe0 2024-09-17T18:35:19.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.627+0000 7f1fcf8dd640 10 monclient(hunting): _renew_subs 2024-09-17T18:35:19.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.627+0000 7f1fcf8dd640 10 monclient(hunting): authenticate will time out at 1723.343644s 2024-09-17T18:35:19.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.627+0000 7f1fcce78640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1fc80a5fe0 0x7f1fc80a63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:19.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.627+0000 7f1fcce78640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1fc80a5fe0 0x7f1fc80a63c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:53304/0 (socket says 172.21.15.121:53304) 2024-09-17T18:35:19.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.627+0000 7f1fcce78640 1 -- 172.21.15.121:0/3614732085 learned_addr learned my addr 172.21.15.121:0/3614732085 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:35:19.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.627+0000 7f1fcce78640 10 monclient(hunting): get_auth_request con 0x7f1fc80a5fe0 auth_method 0 2024-09-17T18:35:19.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.627+0000 7f1fcce78640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:19.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.627+0000 7f1fcce78640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:19.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.627+0000 7f1fcce78640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:19.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.627+0000 7f1fcce78640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:19.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.627+0000 7f1fcce78640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:19.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.627+0000 7f1fcce78640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:19.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.628+0000 7f1fcde7a640 1 --2- 172.21.15.121:0/3614732085 >> v2:172.21.15.173:3300/0 conn(0x7f1fc813dc00 0x7f1fc813fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:19.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.628+0000 7f1fcce78640 10 monclient(hunting): handle_auth_done global_id 64445 payload 274 2024-09-17T18:35:19.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.628+0000 7f1fcce78640 1 -- 172.21.15.121:0/3614732085 >> v2:172.21.15.173:3300/0 conn(0x7f1fc813dc00 msgr2=0x7f1fc813fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:35:19.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.628+0000 7f1fcce78640 1 --2- 172.21.15.121:0/3614732085 >> v2:172.21.15.173:3300/0 conn(0x7f1fc813dc00 0x7f1fc813fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:19.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.628+0000 7f1fcce78640 1 -- 172.21.15.121:0/3614732085 >> v1:172.21.15.173:6789/0 conn(0x7f1fc80a5c00 legacy=0x7f1fc809d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:35:19.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.628+0000 7f1fcce78640 10 monclient: _finish_hunting 0 2024-09-17T18:35:19.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.628+0000 7f1fcce78640 1 monclient: found mon.noname-b 2024-09-17T18:35:19.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.628+0000 7f1fcce78640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:35:19.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.628+0000 7f1fcce78640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:35:19.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.628+0000 7f1fcce78640 1 -- 172.21.15.121:0/3614732085 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fc809ce70 con 0x7f1fc80a5fe0 2024-09-17T18:35:19.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.628+0000 7f1fcde7a640 10 monclient: get_auth_request con 0x7f1fc813dc00 auth_method 0 2024-09-17T18:35:19.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.628+0000 7f1fcde7a640 1 --2- 172.21.15.121:0/3614732085 >> v2:172.21.15.173:3300/0 conn(0x7f1fc813dc00 0x7f1fc813fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T18:35:19.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.630+0000 7f1fcce78640 1 -- 172.21.15.121:0/3614732085 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1fc80a5fe0 msgr2=0x7f1fc80a63c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:19.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.630+0000 7f1fcce78640 1 --2- 172.21.15.121:0/3614732085 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1fc80a5fe0 0x7f1fc80a63c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1fb8009870 tx=0x7f1fb802a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dff3ff3c982774d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:19.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.631+0000 7f1fbffff640 1 -- 172.21.15.121:0/3614732085 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb8037070 con 0x7f1fc80a5fe0 2024-09-17T18:35:19.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.631+0000 7f1fbffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:19.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.631+0000 7f1fbffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:35:19.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.631+0000 7f1fbffff640 10 monclient: dump: 2024-09-17T18:35:19.640 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:19.640 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:19.641 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:19.641 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:19.641 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:19.641 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:19.641 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:19.641 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:19.641 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:19.641 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:19.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.631+0000 7f1fbffff640 10 monclient: _finish_auth 0 2024-09-17T18:35:19.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.631+0000 7f1fbffff640 10 monclient: _check_auth_tickets 2024-09-17T18:35:19.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.631+0000 7f1fbffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:35:19.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.631+0000 7f1fcf8dd640 5 monclient: authenticate success, global_id 64445 2024-09-17T18:35:19.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.631+0000 7f1fbffff640 1 -- 172.21.15.121:0/3614732085 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fb8032640 con 0x7f1fc80a5fe0 2024-09-17T18:35:19.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.631+0000 7f1fcf8dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:35:19.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.631+0000 7f1fbffff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:35:19.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.631+0000 7f1fcf8dd640 10 monclient: get_monmap_and_config success 2024-09-17T18:35:19.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.631+0000 7f1fcf8dd640 4 set_mon_vals no callback set 2024-09-17T18:35:19.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.631+0000 7f1fbffff640 1 -- 172.21.15.121:0/3614732085 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb8032a10 con 0x7f1fc80a5fe0 2024-09-17T18:35:19.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.631+0000 7f1fbffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:19.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.631+0000 7f1fbffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:35:19.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.631+0000 7f1fbffff640 10 monclient: dump: 2024-09-17T18:35:19.642 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:19.642 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:19.642 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:19.642 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:19.642 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:19.642 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:19.642 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:19.642 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:19.643 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:19.643 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:19.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.631+0000 7f1fcf8dd640 10 monclient: shutdown 2024-09-17T18:35:19.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.631+0000 7f1fcf8dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:35:19.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.631+0000 7f1fcf8dd640 1 -- 172.21.15.121:0/3614732085 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1fc80a5fe0 msgr2=0x7f1fc80a63c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:19.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.631+0000 7f1fcf8dd640 1 --2- 172.21.15.121:0/3614732085 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1fc80a5fe0 0x7f1fc80a63c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1fb8009870 tx=0x7f1fb802a0a0 comp rx=0 tx=0).stop 2024-09-17T18:35:19.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.631+0000 7f1fcf8dd640 1 -- 172.21.15.121:0/3614732085 shutdown_connections 2024-09-17T18:35:19.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.631+0000 7f1fcf8dd640 1 --2- 172.21.15.121:0/3614732085 >> v2:172.21.15.173:3300/0 conn(0x7f1fc813dc00 0x7f1fc813fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:19.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.631+0000 7f1fcf8dd640 1 --2- 172.21.15.121:0/3614732085 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1fc80a5fe0 0x7f1fc80a63c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:19.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.631+0000 7f1fcf8dd640 1 -- 172.21.15.121:0/3614732085 >> 172.21.15.121:0/3614732085 conn(0x7f1fc80ab9b0 msgr2=0x7f1fc80a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:35:19.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.631+0000 7f1fcf8dd640 1 -- 172.21.15.121:0/3614732085 shutdown_connections 2024-09-17T18:35:19.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.631+0000 7f1fcf8dd640 1 -- 172.21.15.121:0/3614732085 wait complete. 2024-09-17T18:35:19.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.632+0000 7f1fcf8dd640 10 monclient: build_initial_monmap 2024-09-17T18:35:19.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.632+0000 7f1fcf8dd640 10 monclient: monmap: 2024-09-17T18:35:19.643 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:35:19.644 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:35:19.644 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:35:19.644 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:35:19.644 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:35:19.644 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:19.644 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:35:19.644 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:35:19.644 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:35:19.644 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:19.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.632+0000 7f1fcf8dd640 1 librados: starting msgr at 2024-09-17T18:35:19.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.632+0000 7f1fcf8dd640 1 librados: starting objecter 2024-09-17T18:35:19.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.632+0000 7f1fcf8dd640 1 Processor -- start 2024-09-17T18:35:19.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.632+0000 7f1fcf8dd640 1 -- start start 2024-09-17T18:35:19.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.632+0000 7f1fcf8dd640 1 librados: setting wanted keys 2024-09-17T18:35:19.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.632+0000 7f1fcf8dd640 1 librados: calling monclient init 2024-09-17T18:35:19.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.632+0000 7f1fcf8dd640 10 monclient: init 2024-09-17T18:35:19.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcf8dd640 10 monclient: _reopen_session rank -1 2024-09-17T18:35:19.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcf8dd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:35:19.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcf8dd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1fc80a5c00 0x7f1fc81387a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:19.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcf8dd640 10 monclient(hunting): picked mon.noname-a con 0x7f1fc80a5c00 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:35:19.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcf8dd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1fc80a5fe0 0x7f1fc8138ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:19.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcf8dd640 10 monclient(hunting): picked mon.noname-b con 0x7f1fc80a5fe0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:35:19.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcf8dd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1fc813dc00 0x7f1fc813b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:19.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcd679640 1 -- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1fc80a5c00 msgr2=0x7f1fc81387a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:35:19.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcf8dd640 10 monclient(hunting): picked mon.noname-c con 0x7f1fc813dc00 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:35:19.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcf8dd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:19.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcf8dd640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc8070550 con 0x7f1fc80a5c00 2024-09-17T18:35:19.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcd679640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1fc80a5c00 0x7f1fc81387a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:35:19.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcf8dd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:19.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcf8dd640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc80703d0 con 0x7f1fc80a5fe0 2024-09-17T18:35:19.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcf8dd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:19.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcf8dd640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1fc8140530 con 0x7f1fc813dc00 2024-09-17T18:35:19.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcf8dd640 10 monclient(hunting): _renew_subs 2024-09-17T18:35:19.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcf8dd640 10 monclient(hunting): authenticate will time out at 1723.349469s 2024-09-17T18:35:19.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcde7a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1fc813dc00 0x7f1fc813b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:19.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcde7a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1fc813dc00 0x7f1fc813b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:53320/0 (socket says 172.21.15.121:53320) 2024-09-17T18:35:19.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcde7a640 1 -- 172.21.15.121:0/1530571620 learned_addr learned my addr 172.21.15.121:0/1530571620 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:35:19.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcde7a640 10 monclient(hunting): get_auth_request con 0x7f1fc813dc00 auth_method 0 2024-09-17T18:35:19.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcde7a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:19.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcde7a640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:19.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcde7a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:19.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcde7a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:19.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcde7a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:19.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcce78640 1 --2- 172.21.15.121:0/1530571620 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1fc80a5fe0 0x7f1fc8138ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:19.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcde7a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:19.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcce78640 10 monclient(hunting): get_auth_request con 0x7f1fc80a5fe0 auth_method 0 2024-09-17T18:35:19.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcce78640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:19.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcce78640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:19.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcce78640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:19.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcce78640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:19.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcce78640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:19.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcce78640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:19.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcde7a640 10 monclient(hunting): handle_auth_done global_id 64448 payload 824 2024-09-17T18:35:19.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcde7a640 1 -- 172.21.15.121:0/1530571620 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1fc80a5fe0 msgr2=0x7f1fc8138ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:19.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcde7a640 1 --2- 172.21.15.121:0/1530571620 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1fc80a5fe0 0x7f1fc8138ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:19.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcde7a640 1 -- 172.21.15.121:0/1530571620 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1fc80a5c00 msgr2=0x7f1fc81387a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:35:19.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcde7a640 1 --2- 172.21.15.121:0/1530571620 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1fc80a5c00 0x7f1fc81387a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:19.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcde7a640 10 monclient: _finish_hunting 0 2024-09-17T18:35:19.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcde7a640 1 monclient: found mon.noname-c 2024-09-17T18:35:19.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcde7a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:35:19.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcde7a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:35:19.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.633+0000 7f1fcde7a640 1 -- 172.21.15.121:0/1530571620 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fc8123040 con 0x7f1fc813dc00 2024-09-17T18:35:19.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.636+0000 7f1fcde7a640 1 -- 172.21.15.121:0/1530571620 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1fc813dc00 msgr2=0x7f1fc813b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:19.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.636+0000 7f1fbdffb640 10 client.?.objecter ms_handle_connect 0x7f1fc813dc00 2024-09-17T18:35:19.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.636+0000 7f1fbdffb640 10 client.?.objecter resend_mon_ops 2024-09-17T18:35:19.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.636+0000 7f1fcde7a640 1 --2- 172.21.15.121:0/1530571620 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1fc813dc00 0x7f1fc813b8b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1fc000cef0 tx=0x7f1fc000a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:19.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.636+0000 7f1fbdffb640 1 -- 172.21.15.121:0/1530571620 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc001a070 con 0x7f1fc813dc00 2024-09-17T18:35:19.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.636+0000 7f1fbdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:19.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.636+0000 7f1fbdffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:35:19.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.636+0000 7f1fbdffb640 10 monclient: dump: 2024-09-17T18:35:19.648 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:19.649 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:19.649 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:19.649 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:19.649 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:19.649 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:19.649 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:19.649 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:19.649 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:19.649 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:19.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.636+0000 7f1fbdffb640 10 monclient: _finish_auth 0 2024-09-17T18:35:19.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.636+0000 7f1fbdffb640 10 monclient: _check_auth_tickets 2024-09-17T18:35:19.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.636+0000 7f1fbdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:35:19.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.636+0000 7f1fcf8dd640 5 monclient: authenticate success, global_id 64448 2024-09-17T18:35:19.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.636+0000 7f1fbdffb640 1 -- 172.21.15.121:0/1530571620 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fc000da70 con 0x7f1fc813dc00 2024-09-17T18:35:19.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.636+0000 7f1fcf8dd640 10 monclient: _renew_subs 2024-09-17T18:35:19.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.636+0000 7f1fcf8dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:19.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.636+0000 7f1fcf8dd640 1 -- 172.21.15.121:0/1530571620 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fc8123320 con 0x7f1fc813dc00 2024-09-17T18:35:19.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.636+0000 7f1fbdffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:35:19.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.636+0000 7f1fbdffb640 1 -- 172.21.15.121:0/1530571620 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc000de60 con 0x7f1fc813dc00 2024-09-17T18:35:19.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.636+0000 7f1fbdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:19.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.636+0000 7f1fbdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:35:19.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.636+0000 7f1fbdffb640 10 monclient: dump: 2024-09-17T18:35:19.650 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:19.650 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:19.650 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:19.650 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:19.650 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:19.650 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:19.650 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:19.650 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:19.651 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:19.651 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:19.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.636+0000 7f1fcf8dd640 10 client.64448.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:35:19.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.636+0000 7f1fbf7fe640 4 set_mon_vals no callback set 2024-09-17T18:35:19.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.636+0000 7f1fcf8dd640 10 monclient: _renew_subs 2024-09-17T18:35:19.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.636+0000 7f1fcf8dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:19.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.636+0000 7f1fcf8dd640 1 -- 172.21.15.121:0/1530571620 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fc80a1f20 con 0x7f1fc813dc00 2024-09-17T18:35:19.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.636+0000 7f1fcf8dd640 1 librados: init done 2024-09-17T18:35:19.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.637+0000 7f1fbdffb640 1 -- 172.21.15.121:0/1530571620 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f1fc001d070 con 0x7f1fc813dc00 2024-09-17T18:35:19.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.637+0000 7f1fcf8dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:19.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.637+0000 7f1fcf8dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:19.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.637+0000 7f1fcf8dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:19.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.637+0000 7f1fbdffb640 1 --2- 172.21.15.121:0/1530571620 >> v2:172.21.15.173:6812/2423786037 conn(0x7f1fac040120 0x7f1fac0425e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:19.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.637+0000 7f1fcf8dd640 1 -- 172.21.15.121:0/1530571620 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fc80a63c0 con 0x7f1fc813dc00 2024-09-17T18:35:19.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.637+0000 7f1fbdffb640 1 -- 172.21.15.121:0/1530571620 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(642..642 src has 247..642) ==== 5917+0+0 (secure 0 0 0) 0x7f1fc00541a0 con 0x7f1fc813dc00 2024-09-17T18:35:19.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.637+0000 7f1fbdffb640 10 client.64448.objecter ms_dispatch 0x7f1fc8000b90 osd_map(642..642 src has 247..642) 2024-09-17T18:35:19.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.637+0000 7f1fbdffb640 3 client.64448.objecter handle_osd_map got epochs [642,642] > 0 2024-09-17T18:35:19.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.637+0000 7f1fbdffb640 3 client.64448.objecter handle_osd_map decoding full epoch 642 2024-09-17T18:35:19.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.637+0000 7f1fbdffb640 20 client.64448.objecter dump_active .. 0 homeless 2024-09-17T18:35:19.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.637+0000 7f1fcd679640 1 --2- 172.21.15.121:0/1530571620 >> v2:172.21.15.173:6812/2423786037 conn(0x7f1fac040120 0x7f1fac0425e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:19.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.637+0000 7f1fcd679640 10 monclient: get_auth_request con 0x7f1fac040120 auth_method 0 2024-09-17T18:35:19.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.640+0000 7f1fbdffb640 10 client.64448.objecter ms_handle_connect 0x7f1fac040120 2024-09-17T18:35:19.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.640+0000 7f1fcd679640 1 --2- 172.21.15.121:0/1530571620 >> v2:172.21.15.173:6812/2423786037 conn(0x7f1fac040120 0x7f1fac0425e0 secure :-1 s=READY pgs=4067 cs=0 l=1 rev1=1 crypto rx=0x7f1fc40036d0 tx=0x7f1fc4007a10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:19.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.646+0000 7f1fbdffb640 1 -- 172.21.15.121:0/1530571620 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f1fc0052730 con 0x7f1fc813dc00 2024-09-17T18:35:19.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.646+0000 7f1fbdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:19.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.646+0000 7f1fbdffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:35:19.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.893+0000 7f1fcf8dd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:35:19.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.893+0000 7f1fcf8dd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:35:19.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.893+0000 7f1fcf8dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:19.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.893+0000 7f1fcf8dd640 1 -- 172.21.15.121:0/1530571620 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1fc8121e80 con 0x7f1fc813dc00 2024-09-17T18:35:19.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.893+0000 7f1fbdffb640 1 -- 172.21.15.121:0/1530571620 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f1fc0052730 con 0x7f1fc813dc00 2024-09-17T18:35:19.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.893+0000 7f1fbdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:35:19.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.893+0000 7f1fbdffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:35:19.895 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":56,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:35:19.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.895+0000 7f1fcf8dd640 10 librados: watch_flush enter 2024-09-17T18:35:19.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.895+0000 7f1fcf8dd640 10 librados: watch_flush exit 2024-09-17T18:35:19.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.895+0000 7f1fcf8dd640 20 client.64448.objecter shutdown clearing up homeless session... 2024-09-17T18:35:19.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.895+0000 7f1fcf8dd640 10 client.64448.objecter successfully canceled tick 2024-09-17T18:35:19.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.896+0000 7f1fcf8dd640 1 -- 172.21.15.121:0/1530571620 >> v2:172.21.15.173:6812/2423786037 conn(0x7f1fac040120 msgr2=0x7f1fac0425e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:19.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.896+0000 7f1fcf8dd640 1 --2- 172.21.15.121:0/1530571620 >> v2:172.21.15.173:6812/2423786037 conn(0x7f1fac040120 0x7f1fac0425e0 secure :-1 s=READY pgs=4067 cs=0 l=1 rev1=1 crypto rx=0x7f1fc40036d0 tx=0x7f1fc4007a10 comp rx=0 tx=0).stop 2024-09-17T18:35:19.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.896+0000 7f1fcf8dd640 10 monclient: shutdown 2024-09-17T18:35:19.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.896+0000 7f1fcf8dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:35:19.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.896+0000 7f1fcf8dd640 1 -- 172.21.15.121:0/1530571620 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1fc813dc00 msgr2=0x7f1fc813b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:19.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.896+0000 7f1fcf8dd640 1 --2- 172.21.15.121:0/1530571620 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1fc813dc00 0x7f1fc813b8b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1fc000cef0 tx=0x7f1fc000a710 comp rx=0 tx=0).stop 2024-09-17T18:35:19.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.896+0000 7f1fcf8dd640 1 -- 172.21.15.121:0/1530571620 shutdown_connections 2024-09-17T18:35:19.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.896+0000 7f1fcf8dd640 1 --2- 172.21.15.121:0/1530571620 >> v2:172.21.15.173:6812/2423786037 conn(0x7f1fac040120 0x7f1fac0425e0 unknown :-1 s=CLOSED pgs=4067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:19.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.896+0000 7f1fcf8dd640 1 --2- 172.21.15.121:0/1530571620 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1fc80a5fe0 0x7f1fc8138ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:19.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.896+0000 7f1fcf8dd640 1 --2- 172.21.15.121:0/1530571620 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1fc813dc00 0x7f1fc813b8b0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:19.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.896+0000 7f1fcf8dd640 1 --2- 172.21.15.121:0/1530571620 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1fc80a5c00 0x7f1fc81387a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:19.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.896+0000 7f1fcf8dd640 1 -- 172.21.15.121:0/1530571620 >> 172.21.15.121:0/1530571620 conn(0x7f1fc80ab9b0 msgr2=0x7f1fc80a0e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:35:19.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.896+0000 7f1fcf8dd640 1 -- 172.21.15.121:0/1530571620 shutdown_connections 2024-09-17T18:35:19.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.896+0000 7f1fcf8dd640 1 -- 172.21.15.121:0/1530571620 wait complete. 2024-09-17T18:35:19.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:19.896+0000 7f1fcf8dd640 1 librados: shutdown 2024-09-17T18:35:22.909 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:35:23.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.014+0000 7fc5bb53d640 10 monclient: get_monmap_and_config 2024-09-17T18:35:23.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.014+0000 7fc5bb53d640 10 monclient: build_initial_monmap 2024-09-17T18:35:23.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.014+0000 7fc5bb53d640 10 monclient: monmap: 2024-09-17T18:35:23.016 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:35:23.016 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:23.016 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:35:23.014882+0000 2024-09-17T18:35:23.016 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:35:23.014882+0000 2024-09-17T18:35:23.016 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:35:23.016 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:23.016 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:35:23.016 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:35:23.016 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:35:23.017 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:35:23.017 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:35:23.017 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:23.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.014+0000 7fc5bb53d640 1 Processor -- start 2024-09-17T18:35:23.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.015+0000 7fc5bb53d640 1 -- start start 2024-09-17T18:35:23.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.015+0000 7fc5bb53d640 10 monclient: init 2024-09-17T18:35:23.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.015+0000 7fc5bb53d640 10 monclient: _reopen_session rank -1 2024-09-17T18:35:23.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.015+0000 7fc5bb53d640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-17T18:35:23.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.015+0000 7fc5bb53d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc5b40a5c00 addr v1:172.21.15.173:6789/0 2024-09-17T18:35:23.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.015+0000 7fc5bb53d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc5b40a5fe0 0x7fc5b40a63c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:23.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.015+0000 7fc5bb53d640 10 monclient(hunting): picked mon.noname-b con 0x7fc5b40a5fe0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:35:23.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.015+0000 7fc5bb53d640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc5b413dc00 0x7fc5b413fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:23.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.015+0000 7fc5bb53d640 10 monclient(hunting): picked mon.noname-a con 0x7fc5b413dc00 addr v2:172.21.15.121:3300/0 2024-09-17T18:35:23.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.015+0000 7fc5bb53d640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5b40a8a80 con 0x7fc5b40a5c00 2024-09-17T18:35:23.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.015+0000 7fc5bb53d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:23.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.015+0000 7fc5bb53d640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fc5b40a74d0 con 0x7fc5b413dc00 2024-09-17T18:35:23.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.015+0000 7fc5bb53d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:23.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.015+0000 7fc5bb53d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc5b409cb90 con 0x7fc5b40a5fe0 2024-09-17T18:35:23.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.015+0000 7fc5bb53d640 10 monclient(hunting): _renew_subs 2024-09-17T18:35:23.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.015+0000 7fc5bb53d640 10 monclient(hunting): authenticate will time out at 1726.731613s 2024-09-17T18:35:23.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.015+0000 7fc5b9ada640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc5b413dc00 0x7fc5b413fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:23.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.015+0000 7fc5b8ad8640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc5b40a5fe0 0x7fc5b40a63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:23.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.015+0000 7fc5b9ada640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc5b413dc00 0x7fc5b413fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54290/0 (socket says 172.21.15.121:54290) 2024-09-17T18:35:23.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.015+0000 7fc5b9ada640 1 -- 172.21.15.121:0/3289848281 learned_addr learned my addr 172.21.15.121:0/3289848281 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:35:23.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.015+0000 7fc5b9ada640 10 monclient(hunting): get_auth_request con 0x7fc5b413dc00 auth_method 0 2024-09-17T18:35:23.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.015+0000 7fc5b9ada640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:23.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.015+0000 7fc5b9ada640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:23.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.015+0000 7fc5b9ada640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:23.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.015+0000 7fc5b8ad8640 10 monclient(hunting): get_auth_request con 0x7fc5b40a5fe0 auth_method 0 2024-09-17T18:35:23.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.015+0000 7fc5b8ad8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:23.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.015+0000 7fc5b8ad8640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:23.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.015+0000 7fc5b8ad8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:23.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.015+0000 7fc5b9ada640 1 -- 172.21.15.121:0/3289848281 >> v2:172.21.15.121:3300/0 conn(0x7fc5b413dc00 msgr2=0x7fc5b413fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T18:35:23.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.015+0000 7fc5b9ada640 1 -- 172.21.15.121:0/3289848281 >> v2:172.21.15.121:3300/0 conn(0x7fc5b413dc00 msgr2=0x7fc5b413fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:35:23.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.015+0000 7fc5b9ada640 1 --2- 172.21.15.121:0/3289848281 >> v2:172.21.15.121:3300/0 conn(0x7fc5b413dc00 0x7fc5b413fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:35:23.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.015+0000 7fc5b8ad8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:23.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.015+0000 7fc5b8ad8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:23.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.015+0000 7fc5b8ad8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:23.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.015+0000 7fc5b9ada640 1 --2- 172.21.15.121:0/3289848281 >> v2:172.21.15.121:3300/0 conn(0x7fc5b413dc00 0x7fc5b413fff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:35:23.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.016+0000 7fc5b8ad8640 10 monclient(hunting): handle_auth_done global_id 64451 payload 274 2024-09-17T18:35:23.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.016+0000 7fc5b8ad8640 1 -- 172.21.15.121:0/3289848281 >> v2:172.21.15.121:3300/0 conn(0x7fc5b413dc00 msgr2=0x7fc5b413fff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:35:23.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.016+0000 7fc5b8ad8640 1 --2- 172.21.15.121:0/3289848281 >> v2:172.21.15.121:3300/0 conn(0x7fc5b413dc00 0x7fc5b413fff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:23.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.016+0000 7fc5b8ad8640 1 -- 172.21.15.121:0/3289848281 >> v1:172.21.15.173:6789/0 conn(0x7fc5b40a5c00 legacy=0x7fc5b409d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:35:23.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.016+0000 7fc5b8ad8640 10 monclient: _finish_hunting 0 2024-09-17T18:35:23.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.016+0000 7fc5b8ad8640 1 monclient: found mon.noname-b 2024-09-17T18:35:23.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.016+0000 7fc5b8ad8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:35:23.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.016+0000 7fc5b8ad8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:35:23.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.016+0000 7fc5b8ad8640 1 -- 172.21.15.121:0/3289848281 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5b409ce70 con 0x7fc5b40a5fe0 2024-09-17T18:35:23.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.018+0000 7fc5b8ad8640 1 -- 172.21.15.121:0/3289848281 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc5b40a5fe0 msgr2=0x7fc5b40a63c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:23.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.018+0000 7fc5b8ad8640 1 --2- 172.21.15.121:0/3289848281 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc5b40a5fe0 0x7fc5b40a63c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc5b00097a0 tx=0x7fc5b0029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf77ff9ee081a643 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:23.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.018+0000 7fc5a7fff640 1 -- 172.21.15.121:0/3289848281 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0037070 con 0x7fc5b40a5fe0 2024-09-17T18:35:23.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.018+0000 7fc5a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:23.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.018+0000 7fc5a7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:35:23.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.018+0000 7fc5a7fff640 10 monclient: dump: 2024-09-17T18:35:23.028 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:23.028 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:23.028 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:23.028 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:23.028 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:23.028 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:23.028 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:23.029 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:23.029 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:23.029 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:23.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.019+0000 7fc5a7fff640 10 monclient: _finish_auth 0 2024-09-17T18:35:23.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.019+0000 7fc5a7fff640 10 monclient: _check_auth_tickets 2024-09-17T18:35:23.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.019+0000 7fc5a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:35:23.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.019+0000 7fc5bb53d640 5 monclient: authenticate success, global_id 64451 2024-09-17T18:35:23.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.019+0000 7fc5bb53d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:35:23.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.019+0000 7fc5a7fff640 1 -- 172.21.15.121:0/3289848281 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5b0032440 con 0x7fc5b40a5fe0 2024-09-17T18:35:23.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.019+0000 7fc5a7fff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:35:23.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.019+0000 7fc5a7fff640 1 -- 172.21.15.121:0/3289848281 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0032810 con 0x7fc5b40a5fe0 2024-09-17T18:35:23.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.019+0000 7fc5bb53d640 10 monclient: get_monmap_and_config success 2024-09-17T18:35:23.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.019+0000 7fc5bb53d640 4 set_mon_vals no callback set 2024-09-17T18:35:23.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.019+0000 7fc5a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:23.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.019+0000 7fc5a7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:35:23.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.019+0000 7fc5a7fff640 10 monclient: dump: 2024-09-17T18:35:23.030 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:23.030 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:23.030 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:23.030 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:23.030 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:23.030 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:23.030 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:23.030 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:23.030 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:23.030 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:23.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.019+0000 7fc5bb53d640 10 monclient: shutdown 2024-09-17T18:35:23.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.019+0000 7fc5bb53d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:35:23.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.019+0000 7fc5bb53d640 1 -- 172.21.15.121:0/3289848281 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc5b40a5fe0 msgr2=0x7fc5b40a63c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:23.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.019+0000 7fc5bb53d640 1 --2- 172.21.15.121:0/3289848281 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc5b40a5fe0 0x7fc5b40a63c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc5b00097a0 tx=0x7fc5b0029fd0 comp rx=0 tx=0).stop 2024-09-17T18:35:23.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.019+0000 7fc5bb53d640 1 -- 172.21.15.121:0/3289848281 shutdown_connections 2024-09-17T18:35:23.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.019+0000 7fc5bb53d640 1 --2- 172.21.15.121:0/3289848281 >> v2:172.21.15.121:3300/0 conn(0x7fc5b413dc00 0x7fc5b413fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:23.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.019+0000 7fc5bb53d640 1 --2- 172.21.15.121:0/3289848281 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc5b40a5fe0 0x7fc5b40a63c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:23.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.019+0000 7fc5bb53d640 1 -- 172.21.15.121:0/3289848281 >> 172.21.15.121:0/3289848281 conn(0x7fc5b40ab9b0 msgr2=0x7fc5b40a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:35:23.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.019+0000 7fc5bb53d640 1 -- 172.21.15.121:0/3289848281 shutdown_connections 2024-09-17T18:35:23.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.019+0000 7fc5bb53d640 1 -- 172.21.15.121:0/3289848281 wait complete. 2024-09-17T18:35:23.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.020+0000 7fc5bb53d640 10 monclient: build_initial_monmap 2024-09-17T18:35:23.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.020+0000 7fc5bb53d640 10 monclient: monmap: 2024-09-17T18:35:23.031 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:35:23.031 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:35:23.031 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:35:23.031 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:35:23.031 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:35:23.031 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:23.032 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:35:23.032 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:35:23.032 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:35:23.032 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:23.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.020+0000 7fc5bb53d640 1 librados: starting msgr at 2024-09-17T18:35:23.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.020+0000 7fc5bb53d640 1 librados: starting objecter 2024-09-17T18:35:23.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.020+0000 7fc5bb53d640 1 Processor -- start 2024-09-17T18:35:23.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.020+0000 7fc5bb53d640 1 -- start start 2024-09-17T18:35:23.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.020+0000 7fc5bb53d640 1 librados: setting wanted keys 2024-09-17T18:35:23.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.020+0000 7fc5bb53d640 1 librados: calling monclient init 2024-09-17T18:35:23.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.020+0000 7fc5bb53d640 10 monclient: init 2024-09-17T18:35:23.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.020+0000 7fc5bb53d640 10 monclient: _reopen_session rank -1 2024-09-17T18:35:23.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.020+0000 7fc5bb53d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:35:23.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.020+0000 7fc5bb53d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc5b40a5c00 0x7fc5b4138880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:23.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.020+0000 7fc5bb53d640 10 monclient(hunting): picked mon.noname-a con 0x7fc5b40a5c00 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:35:23.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.021+0000 7fc5bb53d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5b40a5fe0 0x7fc5b413ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:23.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.021+0000 7fc5bb53d640 10 monclient(hunting): picked mon.noname-b con 0x7fc5b40a5fe0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:35:23.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.021+0000 7fc5bb53d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc5b413dc00 0x7fc5b413bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:23.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.021+0000 7fc5bb53d640 10 monclient(hunting): picked mon.noname-c con 0x7fc5b413dc00 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:35:23.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.021+0000 7fc5bb53d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:23.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.021+0000 7fc5bb53d640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b4070550 con 0x7fc5b40a5c00 2024-09-17T18:35:23.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.021+0000 7fc5bb53d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:23.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.021+0000 7fc5bb53d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b40703d0 con 0x7fc5b40a5fe0 2024-09-17T18:35:23.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.021+0000 7fc5bb53d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:23.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.021+0000 7fc5bb53d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc5b4140530 con 0x7fc5b413dc00 2024-09-17T18:35:23.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.021+0000 7fc5bb53d640 10 monclient(hunting): _renew_subs 2024-09-17T18:35:23.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.021+0000 7fc5bb53d640 10 monclient(hunting): authenticate will time out at 1726.737060s 2024-09-17T18:35:23.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.021+0000 7fc5b92d9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc5b40a5c00 0x7fc5b4138880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:23.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.021+0000 7fc5b9ada640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc5b413dc00 0x7fc5b413bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:23.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.021+0000 7fc5b92d9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc5b40a5c00 0x7fc5b4138880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54294/0 (socket says 172.21.15.121:54294) 2024-09-17T18:35:23.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.021+0000 7fc5b92d9640 1 -- 172.21.15.121:0/104247361 learned_addr learned my addr 172.21.15.121:0/104247361 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:35:23.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.021+0000 7fc5b92d9640 10 monclient(hunting): get_auth_request con 0x7fc5b40a5c00 auth_method 0 2024-09-17T18:35:23.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.021+0000 7fc5b92d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:23.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.021+0000 7fc5b92d9640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:23.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.021+0000 7fc5b92d9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:23.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.021+0000 7fc5b9ada640 10 monclient(hunting): get_auth_request con 0x7fc5b413dc00 auth_method 0 2024-09-17T18:35:23.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.021+0000 7fc5b9ada640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:23.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.021+0000 7fc5b9ada640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:23.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.021+0000 7fc5b9ada640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:23.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.021+0000 7fc5b9ada640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:23.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.021+0000 7fc5b9ada640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:23.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.021+0000 7fc5b92d9640 1 -- 172.21.15.121:0/104247361 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc5b40a5c00 msgr2=0x7fc5b4138880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T18:35:23.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.021+0000 7fc5b92d9640 1 -- 172.21.15.121:0/104247361 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc5b40a5c00 msgr2=0x7fc5b4138880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:35:23.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.021+0000 7fc5b9ada640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:23.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.021+0000 7fc5b92d9640 1 --2- 172.21.15.121:0/104247361 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc5b40a5c00 0x7fc5b4138880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:35:23.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.021+0000 7fc5b92d9640 1 --2- 172.21.15.121:0/104247361 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc5b40a5c00 0x7fc5b4138880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:35:23.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.021+0000 7fc5b9ada640 10 monclient(hunting): handle_auth_done global_id 64454 payload 824 2024-09-17T18:35:23.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.021+0000 7fc5b8ad8640 1 --2- 172.21.15.121:0/104247361 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5b40a5fe0 0x7fc5b413ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:23.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.021+0000 7fc5b9ada640 1 -- 172.21.15.121:0/104247361 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5b40a5fe0 msgr2=0x7fc5b413ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:23.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.021+0000 7fc5b9ada640 1 --2- 172.21.15.121:0/104247361 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5b40a5fe0 0x7fc5b413ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:23.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.021+0000 7fc5b9ada640 1 -- 172.21.15.121:0/104247361 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc5b40a5c00 msgr2=0x7fc5b4138880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:35:23.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.021+0000 7fc5b9ada640 1 --2- 172.21.15.121:0/104247361 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc5b40a5c00 0x7fc5b4138880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:23.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.021+0000 7fc5b9ada640 10 monclient: _finish_hunting 0 2024-09-17T18:35:23.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.021+0000 7fc5b9ada640 1 monclient: found mon.noname-c 2024-09-17T18:35:23.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.021+0000 7fc5b9ada640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:35:23.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.021+0000 7fc5b9ada640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:35:23.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.021+0000 7fc5b9ada640 1 -- 172.21.15.121:0/104247361 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5b40a1f20 con 0x7fc5b413dc00 2024-09-17T18:35:23.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.023+0000 7fc5b9ada640 1 -- 172.21.15.121:0/104247361 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc5b413dc00 msgr2=0x7fc5b413bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:23.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.023+0000 7fc5b9ada640 1 --2- 172.21.15.121:0/104247361 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc5b413dc00 0x7fc5b413bf60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac0096d0 tx=0x7fc5ac007ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:23.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.023+0000 7fc5a5ffb640 10 client.?.objecter ms_handle_connect 0x7fc5b413dc00 2024-09-17T18:35:23.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.023+0000 7fc5a5ffb640 10 client.?.objecter resend_mon_ops 2024-09-17T18:35:23.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.023+0000 7fc5a5ffb640 1 -- 172.21.15.121:0/104247361 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac01a070 con 0x7fc5b413dc00 2024-09-17T18:35:23.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.023+0000 7fc5a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:23.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.023+0000 7fc5a5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:35:23.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.023+0000 7fc5a5ffb640 10 monclient: dump: 2024-09-17T18:35:23.036 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:23.036 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:23.036 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:23.037 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:23.037 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:23.037 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:23.037 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:23.037 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:23.037 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:23.037 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:23.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.023+0000 7fc5a5ffb640 10 monclient: _finish_auth 0 2024-09-17T18:35:23.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.023+0000 7fc5a5ffb640 10 monclient: _check_auth_tickets 2024-09-17T18:35:23.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.023+0000 7fc5a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:35:23.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.023+0000 7fc5a5ffb640 1 -- 172.21.15.121:0/104247361 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5ac00ddc0 con 0x7fc5b413dc00 2024-09-17T18:35:23.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.023+0000 7fc5bb53d640 5 monclient: authenticate success, global_id 64454 2024-09-17T18:35:23.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.023+0000 7fc5a5ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:35:23.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.023+0000 7fc5bb53d640 10 monclient: _renew_subs 2024-09-17T18:35:23.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.023+0000 7fc5a5ffb640 1 -- 172.21.15.121:0/104247361 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac015640 con 0x7fc5b413dc00 2024-09-17T18:35:23.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.023+0000 7fc5bb53d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:23.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.023+0000 7fc5bb53d640 1 -- 172.21.15.121:0/104247361 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5b40a23f0 con 0x7fc5b413dc00 2024-09-17T18:35:23.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.023+0000 7fc5a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:23.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.023+0000 7fc5a77fe640 4 set_mon_vals no callback set 2024-09-17T18:35:23.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.023+0000 7fc5a5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:35:23.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.023+0000 7fc5a5ffb640 10 monclient: dump: 2024-09-17T18:35:23.038 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:23.038 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:23.038 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:23.038 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:23.038 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:23.039 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:23.039 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:23.039 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:23.039 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:23.039 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:23.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.024+0000 7fc5bb53d640 10 client.64454.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:35:23.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.024+0000 7fc5bb53d640 10 monclient: _renew_subs 2024-09-17T18:35:23.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.024+0000 7fc5bb53d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:23.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.024+0000 7fc5bb53d640 1 -- 172.21.15.121:0/104247361 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5b4121850 con 0x7fc5b413dc00 2024-09-17T18:35:23.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.024+0000 7fc5bb53d640 1 librados: init done 2024-09-17T18:35:23.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.024+0000 7fc5a5ffb640 1 -- 172.21.15.121:0/104247361 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fc5ac01d070 con 0x7fc5b413dc00 2024-09-17T18:35:23.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.025+0000 7fc5a5ffb640 1 --2- 172.21.15.121:0/104247361 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc5980401c0 0x7fc598042680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:23.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.025+0000 7fc5bb53d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:23.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.025+0000 7fc5bb53d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:23.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.025+0000 7fc5bb53d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:23.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.025+0000 7fc5bb53d640 1 -- 172.21.15.121:0/104247361 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5b40a63c0 con 0x7fc5b413dc00 2024-09-17T18:35:23.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.025+0000 7fc5a5ffb640 1 -- 172.21.15.121:0/104247361 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(645..645 src has 247..645) ==== 5925+0+0 (secure 0 0 0) 0x7fc5ac053890 con 0x7fc5b413dc00 2024-09-17T18:35:23.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.025+0000 7fc5a5ffb640 10 client.64454.objecter ms_dispatch 0x7fc5b4000b90 osd_map(645..645 src has 247..645) 2024-09-17T18:35:23.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.025+0000 7fc5a5ffb640 3 client.64454.objecter handle_osd_map got epochs [645,645] > 0 2024-09-17T18:35:23.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.025+0000 7fc5a5ffb640 3 client.64454.objecter handle_osd_map decoding full epoch 645 2024-09-17T18:35:23.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.025+0000 7fc5a5ffb640 20 client.64454.objecter dump_active .. 0 homeless 2024-09-17T18:35:23.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.025+0000 7fc5b92d9640 1 --2- 172.21.15.121:0/104247361 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc5980401c0 0x7fc598042680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:23.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.025+0000 7fc5b92d9640 10 monclient: get_auth_request con 0x7fc5980401c0 auth_method 0 2024-09-17T18:35:23.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.028+0000 7fc5a5ffb640 10 client.64454.objecter ms_handle_connect 0x7fc5980401c0 2024-09-17T18:35:23.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.028+0000 7fc5b92d9640 1 --2- 172.21.15.121:0/104247361 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc5980401c0 0x7fc598042680 secure :-1 s=READY pgs=4069 cs=0 l=1 rev1=1 crypto rx=0x7fc5a8006eb0 tx=0x7fc5a8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:23.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.030+0000 7fc5a5ffb640 1 -- 172.21.15.121:0/104247361 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc5b40a23f0 con 0x7fc5b413dc00 2024-09-17T18:35:23.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.030+0000 7fc5a5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:23.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.030+0000 7fc5a5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:35:23.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.273+0000 7fc5bb53d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:35:23.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.273+0000 7fc5bb53d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:35:23.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.273+0000 7fc5bb53d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:23.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.273+0000 7fc5bb53d640 1 -- 172.21.15.121:0/104247361 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc5b4121b70 con 0x7fc5b413dc00 2024-09-17T18:35:23.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.273+0000 7fc5a5ffb640 1 -- 172.21.15.121:0/104247361 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7fc5b40a1f20 con 0x7fc5b413dc00 2024-09-17T18:35:23.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.273+0000 7fc5a5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:35:23.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.273+0000 7fc5a5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:35:23.275 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":56,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:35:23.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.275+0000 7fc5bb53d640 10 librados: watch_flush enter 2024-09-17T18:35:23.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.275+0000 7fc5bb53d640 10 librados: watch_flush exit 2024-09-17T18:35:23.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.275+0000 7fc5bb53d640 20 client.64454.objecter shutdown clearing up homeless session... 2024-09-17T18:35:23.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.275+0000 7fc5bb53d640 10 client.64454.objecter successfully canceled tick 2024-09-17T18:35:23.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.275+0000 7fc5bb53d640 1 -- 172.21.15.121:0/104247361 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc5980401c0 msgr2=0x7fc598042680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:23.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.275+0000 7fc5bb53d640 1 --2- 172.21.15.121:0/104247361 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc5980401c0 0x7fc598042680 secure :-1 s=READY pgs=4069 cs=0 l=1 rev1=1 crypto rx=0x7fc5a8006eb0 tx=0x7fc5a8008040 comp rx=0 tx=0).stop 2024-09-17T18:35:23.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.275+0000 7fc5bb53d640 10 monclient: shutdown 2024-09-17T18:35:23.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.275+0000 7fc5bb53d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:35:23.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.275+0000 7fc5bb53d640 1 -- 172.21.15.121:0/104247361 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc5b413dc00 msgr2=0x7fc5b413bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:23.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.275+0000 7fc5bb53d640 1 --2- 172.21.15.121:0/104247361 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc5b413dc00 0x7fc5b413bf60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac0096d0 tx=0x7fc5ac007ef0 comp rx=0 tx=0).stop 2024-09-17T18:35:23.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.275+0000 7fc5bb53d640 1 -- 172.21.15.121:0/104247361 shutdown_connections 2024-09-17T18:35:23.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.275+0000 7fc5bb53d640 1 --2- 172.21.15.121:0/104247361 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc5980401c0 0x7fc598042680 unknown :-1 s=CLOSED pgs=4069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:23.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.275+0000 7fc5bb53d640 1 --2- 172.21.15.121:0/104247361 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5b40a5fe0 0x7fc5b413ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:23.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.275+0000 7fc5bb53d640 1 --2- 172.21.15.121:0/104247361 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc5b413dc00 0x7fc5b413bf60 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:23.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.275+0000 7fc5bb53d640 1 --2- 172.21.15.121:0/104247361 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc5b40a5c00 0x7fc5b4138880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:23.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.275+0000 7fc5bb53d640 1 -- 172.21.15.121:0/104247361 >> 172.21.15.121:0/104247361 conn(0x7fc5b40ab9b0 msgr2=0x7fc5b413e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:35:23.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.276+0000 7fc5bb53d640 1 -- 172.21.15.121:0/104247361 shutdown_connections 2024-09-17T18:35:23.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.276+0000 7fc5bb53d640 1 -- 172.21.15.121:0/104247361 wait complete. 2024-09-17T18:35:23.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:23.276+0000 7fc5bb53d640 1 librados: shutdown 2024-09-17T18:35:26.288 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:35:26.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.395+0000 7f92c9fbc640 10 monclient: get_monmap_and_config 2024-09-17T18:35:26.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.395+0000 7f92c9fbc640 10 monclient: build_initial_monmap 2024-09-17T18:35:26.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.395+0000 7f92c9fbc640 10 monclient: monmap: 2024-09-17T18:35:26.397 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:35:26.397 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:26.397 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:35:26.396327+0000 2024-09-17T18:35:26.398 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:35:26.396327+0000 2024-09-17T18:35:26.398 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:35:26.398 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:26.398 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:35:26.398 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:35:26.398 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:35:26.398 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:35:26.398 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:35:26.398 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:26.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.396+0000 7f92c9fbc640 1 Processor -- start 2024-09-17T18:35:26.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.396+0000 7f92c9fbc640 1 -- start start 2024-09-17T18:35:26.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.396+0000 7f92c9fbc640 10 monclient: init 2024-09-17T18:35:26.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.397+0000 7f92c9fbc640 10 monclient: _reopen_session rank -1 2024-09-17T18:35:26.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.397+0000 7f92c9fbc640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-17T18:35:26.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.397+0000 7f92c9fbc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f92c40a0a70 addr v1:172.21.15.121:6789/0 2024-09-17T18:35:26.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.397+0000 7f92c9fbc640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f92c41348f0 0x7f92c4139c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:26.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.397+0000 7f92c9fbc640 10 monclient(hunting): picked mon.noname-a con 0x7f92c41348f0 addr v2:172.21.15.121:3300/0 2024-09-17T18:35:26.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.397+0000 7f92c9fbc640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f92c413a150 0x7f92c413c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:26.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.397+0000 7f92c9fbc640 10 monclient(hunting): picked mon.noname-c con 0x7f92c413a150 addr v2:172.21.15.173:3300/0 2024-09-17T18:35:26.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.397+0000 7f92c9fbc640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92c409f5b0 con 0x7f92c40a0a70 2024-09-17T18:35:26.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.397+0000 7f92c9fbc640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:26.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.397+0000 7f92c9fbc640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f92c409ef80 con 0x7f92c41348f0 2024-09-17T18:35:26.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.397+0000 7f92c9fbc640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:26.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.397+0000 7f92c9fbc640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f92c40a1560 con 0x7f92c413a150 2024-09-17T18:35:26.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.397+0000 7f92c9fbc640 10 monclient(hunting): _renew_subs 2024-09-17T18:35:26.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.397+0000 7f92c9fbc640 10 monclient(hunting): authenticate will time out at 1730.113295s 2024-09-17T18:35:26.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.397+0000 7f92c37fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f92c40a0a70 0x7f92c40a0e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:43040/0 (socket says 172.21.15.121:43040) 2024-09-17T18:35:26.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.397+0000 7f92c37fe640 1 -- 172.21.15.121:0/1316026718 learned_addr learned my addr 172.21.15.121:0/1316026718 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:35:26.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.397+0000 7f92c2ffd640 1 --2- 172.21.15.121:0/1316026718 >> v2:172.21.15.121:3300/0 conn(0x7f92c41348f0 0x7f92c4139c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:26.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.397+0000 7f92c2ffd640 10 monclient(hunting): get_auth_request con 0x7f92c41348f0 auth_method 0 2024-09-17T18:35:26.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.397+0000 7f92c2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:26.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.397+0000 7f92c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:26.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.397+0000 7f92c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:26.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.397+0000 7f92c2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:26.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.397+0000 7f92c2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:26.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.397+0000 7f92c2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:26.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.397+0000 7f92c3fff640 1 --2- 172.21.15.121:0/1316026718 >> v2:172.21.15.173:3300/0 conn(0x7f92c413a150 0x7f92c413c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:26.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.397+0000 7f92c3fff640 10 monclient(hunting): get_auth_request con 0x7f92c413a150 auth_method 0 2024-09-17T18:35:26.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.397+0000 7f92c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:26.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.397+0000 7f92c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:26.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.397+0000 7f92c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:26.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.397+0000 7f92c2ffd640 10 monclient(hunting): handle_auth_done global_id 104097 payload 274 2024-09-17T18:35:26.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.397+0000 7f92c2ffd640 1 -- 172.21.15.121:0/1316026718 >> v2:172.21.15.173:3300/0 conn(0x7f92c413a150 msgr2=0x7f92c413c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:35:26.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.397+0000 7f92c2ffd640 1 --2- 172.21.15.121:0/1316026718 >> v2:172.21.15.173:3300/0 conn(0x7f92c413a150 0x7f92c413c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:26.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.397+0000 7f92c2ffd640 1 -- 172.21.15.121:0/1316026718 >> v1:172.21.15.121:6789/0 conn(0x7f92c40a0a70 legacy=0x7f92c40a0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:35:26.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.397+0000 7f92c2ffd640 10 monclient: _finish_hunting 0 2024-09-17T18:35:26.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.397+0000 7f92c2ffd640 1 monclient: found mon.noname-a 2024-09-17T18:35:26.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.397+0000 7f92c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:35:26.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.397+0000 7f92c2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:35:26.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.397+0000 7f92c2ffd640 1 -- 172.21.15.121:0/1316026718 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92c409eca0 con 0x7f92c41348f0 2024-09-17T18:35:26.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.400+0000 7f92c2ffd640 1 -- 172.21.15.121:0/1316026718 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92c41348f0 msgr2=0x7f92c4139c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:26.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.400+0000 7f92c2ffd640 1 --2- 172.21.15.121:0/1316026718 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92c41348f0 0x7f92c4139c10 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f92b80097d0 tx=0x7f92b802a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec38e939e7cfdd35 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:26.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.400+0000 7f92c27fc640 1 -- 172.21.15.121:0/1316026718 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b8037070 con 0x7f92c41348f0 2024-09-17T18:35:26.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.400+0000 7f92c27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:26.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.400+0000 7f92c27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:35:26.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.400+0000 7f92c27fc640 10 monclient: dump: 2024-09-17T18:35:26.409 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:26.409 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:26.409 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:26.409 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:26.409 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:26.410 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:26.410 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:26.410 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:26.410 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:26.410 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:26.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.400+0000 7f92c27fc640 10 monclient: _finish_auth 0 2024-09-17T18:35:26.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.400+0000 7f92c27fc640 10 monclient: _check_auth_tickets 2024-09-17T18:35:26.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.400+0000 7f92c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:35:26.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.400+0000 7f92c27fc640 1 -- 172.21.15.121:0/1316026718 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92b8032440 con 0x7f92c41348f0 2024-09-17T18:35:26.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.400+0000 7f92c9fbc640 5 monclient: authenticate success, global_id 104097 2024-09-17T18:35:26.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.400+0000 7f92c9fbc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:35:26.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.400+0000 7f92c27fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:35:26.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.400+0000 7f92c9fbc640 10 monclient: get_monmap_and_config success 2024-09-17T18:35:26.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.400+0000 7f92c27fc640 1 -- 172.21.15.121:0/1316026718 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b8032810 con 0x7f92c41348f0 2024-09-17T18:35:26.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.400+0000 7f92c9fbc640 4 set_mon_vals no callback set 2024-09-17T18:35:26.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.400+0000 7f92c27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:26.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.400+0000 7f92c27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:35:26.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.400+0000 7f92c27fc640 10 monclient: dump: 2024-09-17T18:35:26.411 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:26.411 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:26.411 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:26.411 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:26.411 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:26.411 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:26.411 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:26.411 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:26.411 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:26.411 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:26.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.401+0000 7f92c9fbc640 10 monclient: shutdown 2024-09-17T18:35:26.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.401+0000 7f92c9fbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:35:26.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.401+0000 7f92c9fbc640 1 -- 172.21.15.121:0/1316026718 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92c41348f0 msgr2=0x7f92c4139c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:26.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.401+0000 7f92c9fbc640 1 --2- 172.21.15.121:0/1316026718 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92c41348f0 0x7f92c4139c10 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f92b80097d0 tx=0x7f92b802a040 comp rx=0 tx=0).stop 2024-09-17T18:35:26.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.401+0000 7f92c9fbc640 1 -- 172.21.15.121:0/1316026718 shutdown_connections 2024-09-17T18:35:26.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.401+0000 7f92c9fbc640 1 --2- 172.21.15.121:0/1316026718 >> v2:172.21.15.173:3300/0 conn(0x7f92c413a150 0x7f92c413c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:26.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.401+0000 7f92c9fbc640 1 --2- 172.21.15.121:0/1316026718 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92c41348f0 0x7f92c4139c10 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:26.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.401+0000 7f92c9fbc640 1 -- 172.21.15.121:0/1316026718 >> 172.21.15.121:0/1316026718 conn(0x7f92c40a3b50 msgr2=0x7f92c40a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:35:26.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.401+0000 7f92c9fbc640 1 -- 172.21.15.121:0/1316026718 shutdown_connections 2024-09-17T18:35:26.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.401+0000 7f92c9fbc640 1 -- 172.21.15.121:0/1316026718 wait complete. 2024-09-17T18:35:26.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.401+0000 7f92c9fbc640 10 monclient: build_initial_monmap 2024-09-17T18:35:26.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.401+0000 7f92c9fbc640 10 monclient: monmap: 2024-09-17T18:35:26.412 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:35:26.412 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:35:26.412 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:35:26.413 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:35:26.413 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:35:26.413 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:26.413 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:35:26.413 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:35:26.413 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:35:26.413 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:26.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.402+0000 7f92c9fbc640 1 librados: starting msgr at 2024-09-17T18:35:26.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.402+0000 7f92c9fbc640 1 librados: starting objecter 2024-09-17T18:35:26.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.402+0000 7f92c9fbc640 1 Processor -- start 2024-09-17T18:35:26.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.402+0000 7f92c9fbc640 1 -- start start 2024-09-17T18:35:26.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.402+0000 7f92c9fbc640 1 librados: setting wanted keys 2024-09-17T18:35:26.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.402+0000 7f92c9fbc640 1 librados: calling monclient init 2024-09-17T18:35:26.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.402+0000 7f92c9fbc640 10 monclient: init 2024-09-17T18:35:26.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.402+0000 7f92c9fbc640 10 monclient: _reopen_session rank -1 2024-09-17T18:35:26.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.402+0000 7f92c9fbc640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T18:35:26.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.402+0000 7f92c9fbc640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92c40a0a70 0x7f92c41cba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:26.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.402+0000 7f92c9fbc640 10 monclient(hunting): picked mon.noname-a con 0x7f92c40a0a70 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:35:26.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.402+0000 7f92c9fbc640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92c41348f0 0x7f92c41cbf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:26.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.402+0000 7f92c9fbc640 10 monclient(hunting): picked mon.noname-c con 0x7f92c41348f0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:35:26.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.402+0000 7f92c9fbc640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92c413a150 0x7f92c41cc490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:26.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.402+0000 7f92c9fbc640 10 monclient(hunting): picked mon.noname-b con 0x7f92c413a150 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:35:26.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.402+0000 7f92c9fbc640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:26.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.402+0000 7f92c9fbc640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f92c409dce0 con 0x7f92c40a0a70 2024-09-17T18:35:26.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.403+0000 7f92c9fbc640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:26.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.403+0000 7f92c9fbc640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f92c4134d60 con 0x7f92c413a150 2024-09-17T18:35:26.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.403+0000 7f92c9fbc640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:26.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.403+0000 7f92c9fbc640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f92c409de60 con 0x7f92c41348f0 2024-09-17T18:35:26.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.403+0000 7f92c9fbc640 10 monclient(hunting): _renew_subs 2024-09-17T18:35:26.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.403+0000 7f92c37fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92c40a0a70 0x7f92c41cba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:26.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.403+0000 7f92c9fbc640 10 monclient(hunting): authenticate will time out at 1730.119029s 2024-09-17T18:35:26.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.403+0000 7f92c37fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92c40a0a70 0x7f92c41cba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54302/0 (socket says 172.21.15.121:54302) 2024-09-17T18:35:26.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.403+0000 7f92c37fe640 1 -- 172.21.15.121:0/3684242094 learned_addr learned my addr 172.21.15.121:0/3684242094 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:35:26.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.403+0000 7f92c37fe640 10 monclient(hunting): get_auth_request con 0x7f92c40a0a70 auth_method 0 2024-09-17T18:35:26.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.403+0000 7f92c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:26.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.403+0000 7f92c37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:26.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.403+0000 7f92c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:26.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.403+0000 7f92c2ffd640 1 --2- 172.21.15.121:0/3684242094 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92c41348f0 0x7f92c41cbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:26.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.403+0000 7f92c2ffd640 10 monclient(hunting): get_auth_request con 0x7f92c41348f0 auth_method 0 2024-09-17T18:35:26.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.403+0000 7f92c2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:26.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.403+0000 7f92c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:26.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.403+0000 7f92c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:26.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.403+0000 7f92c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:26.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.403+0000 7f92c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:26.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.403+0000 7f92c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:26.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.403+0000 7f92c3fff640 1 --2- 172.21.15.121:0/3684242094 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92c413a150 0x7f92c41cc490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:26.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.403+0000 7f92c2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:26.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.403+0000 7f92c2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:26.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.403+0000 7f92c2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:26.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.403+0000 7f92c3fff640 10 monclient(hunting): get_auth_request con 0x7f92c413a150 auth_method 0 2024-09-17T18:35:26.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.403+0000 7f92c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:26.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.403+0000 7f92c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:26.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.403+0000 7f92c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:26.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.403+0000 7f92c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:26.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.403+0000 7f92c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:26.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.403+0000 7f92c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:26.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.403+0000 7f92c37fe640 10 monclient(hunting): handle_auth_done global_id 104100 payload 824 2024-09-17T18:35:26.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.403+0000 7f92c37fe640 1 -- 172.21.15.121:0/3684242094 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92c41348f0 msgr2=0x7f92c41cbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:26.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.403+0000 7f92c37fe640 1 --2- 172.21.15.121:0/3684242094 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92c41348f0 0x7f92c41cbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:26.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.403+0000 7f92c37fe640 1 -- 172.21.15.121:0/3684242094 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92c413a150 msgr2=0x7f92c41cc490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:26.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.403+0000 7f92c37fe640 1 --2- 172.21.15.121:0/3684242094 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92c413a150 0x7f92c41cc490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:26.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.403+0000 7f92c37fe640 10 monclient: _finish_hunting 0 2024-09-17T18:35:26.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.403+0000 7f92c37fe640 1 monclient: found mon.noname-a 2024-09-17T18:35:26.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.403+0000 7f92c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:35:26.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.403+0000 7f92c37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:35:26.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.403+0000 7f92c37fe640 1 -- 172.21.15.121:0/3684242094 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92c409e240 con 0x7f92c40a0a70 2024-09-17T18:35:26.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.405+0000 7f92c37fe640 1 -- 172.21.15.121:0/3684242094 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92c40a0a70 msgr2=0x7f92c41cba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:26.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.405+0000 7f92c37fe640 1 --2- 172.21.15.121:0/3684242094 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92c40a0a70 0x7f92c41cba10 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f92b400cc10 tx=0x7f92b400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:26.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.405+0000 7f92abfff640 10 client.?.objecter ms_handle_connect 0x7f92c40a0a70 2024-09-17T18:35:26.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.405+0000 7f92abfff640 10 client.?.objecter resend_mon_ops 2024-09-17T18:35:26.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.405+0000 7f92abfff640 1 -- 172.21.15.121:0/3684242094 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b4012400 con 0x7f92c40a0a70 2024-09-17T18:35:26.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.405+0000 7f92abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:26.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.405+0000 7f92abfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:35:26.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.405+0000 7f92abfff640 10 monclient: dump: 2024-09-17T18:35:26.418 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:26.418 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:26.418 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:26.418 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:26.418 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:26.418 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:26.418 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:26.418 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:26.418 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:26.418 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:26.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.405+0000 7f92abfff640 10 monclient: _finish_auth 0 2024-09-17T18:35:26.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.405+0000 7f92abfff640 10 monclient: _check_auth_tickets 2024-09-17T18:35:26.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.405+0000 7f92abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:35:26.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.405+0000 7f92c9fbc640 5 monclient: authenticate success, global_id 104100 2024-09-17T18:35:26.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.405+0000 7f92abfff640 1 -- 172.21.15.121:0/3684242094 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92b40125e0 con 0x7f92c40a0a70 2024-09-17T18:35:26.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.405+0000 7f92abfff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:35:26.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.405+0000 7f92abfff640 1 -- 172.21.15.121:0/3684242094 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b40129d0 con 0x7f92c40a0a70 2024-09-17T18:35:26.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.405+0000 7f92c9fbc640 10 monclient: _renew_subs 2024-09-17T18:35:26.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.405+0000 7f92c9fbc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:35:26.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.405+0000 7f92c9fbc640 1 -- 172.21.15.121:0/3684242094 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92c40749a0 con 0x7f92c40a0a70 2024-09-17T18:35:26.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.405+0000 7f92abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:26.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.405+0000 7f92abfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:35:26.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.405+0000 7f92c1ffb640 4 set_mon_vals no callback set 2024-09-17T18:35:26.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.405+0000 7f92abfff640 10 monclient: dump: 2024-09-17T18:35:26.419 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:26.419 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:26.419 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:26.420 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:26.420 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:26.420 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:26.420 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:26.420 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:26.420 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:26.420 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:26.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.405+0000 7f92c9fbc640 10 client.104100.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:35:26.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.405+0000 7f92c9fbc640 10 monclient: _renew_subs 2024-09-17T18:35:26.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.405+0000 7f92c9fbc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:35:26.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.405+0000 7f92c9fbc640 1 -- 172.21.15.121:0/3684242094 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92c411deb0 con 0x7f92c40a0a70 2024-09-17T18:35:26.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.405+0000 7f92c9fbc640 1 librados: init done 2024-09-17T18:35:26.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.406+0000 7f92abfff640 1 -- 172.21.15.121:0/3684242094 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f92b4012bf0 con 0x7f92c40a0a70 2024-09-17T18:35:26.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.406+0000 7f92c9fbc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:26.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.406+0000 7f92c9fbc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:26.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.406+0000 7f92c9fbc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:35:26.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.406+0000 7f92c9fbc640 1 -- 172.21.15.121:0/3684242094 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92c40a0e50 con 0x7f92c40a0a70 2024-09-17T18:35:26.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.406+0000 7f92abfff640 1 --2- 172.21.15.121:0/3684242094 >> v2:172.21.15.173:6812/2423786037 conn(0x7f92b0040170 0x7f92b0042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:26.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.406+0000 7f92abfff640 1 -- 172.21.15.121:0/3684242094 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(646..646 src has 247..646) ==== 5925+0+0 (secure 0 0 0) 0x7f92b4054060 con 0x7f92c40a0a70 2024-09-17T18:35:26.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.406+0000 7f92abfff640 10 client.104100.objecter ms_dispatch 0x7f92c4000b90 osd_map(646..646 src has 247..646) 2024-09-17T18:35:26.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.406+0000 7f92abfff640 3 client.104100.objecter handle_osd_map got epochs [646,646] > 0 2024-09-17T18:35:26.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.406+0000 7f92abfff640 3 client.104100.objecter handle_osd_map decoding full epoch 646 2024-09-17T18:35:26.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.407+0000 7f92abfff640 20 client.104100.objecter dump_active .. 0 homeless 2024-09-17T18:35:26.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.407+0000 7f92c2ffd640 1 --2- 172.21.15.121:0/3684242094 >> v2:172.21.15.173:6812/2423786037 conn(0x7f92b0040170 0x7f92b0042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:26.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.407+0000 7f92c2ffd640 10 monclient: get_auth_request con 0x7f92b0040170 auth_method 0 2024-09-17T18:35:26.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.409+0000 7f92abfff640 10 client.104100.objecter ms_handle_connect 0x7f92b0040170 2024-09-17T18:35:26.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.409+0000 7f92c2ffd640 1 --2- 172.21.15.121:0/3684242094 >> v2:172.21.15.173:6812/2423786037 conn(0x7f92b0040170 0x7f92b0042630 secure :-1 s=READY pgs=4070 cs=0 l=1 rev1=1 crypto rx=0x7f92b8034000 tx=0x7f92b802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:26.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.411+0000 7f92abfff640 1 -- 172.21.15.121:0/3684242094 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f92b40525e0 con 0x7f92c40a0a70 2024-09-17T18:35:26.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.411+0000 7f92abfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:26.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.411+0000 7f92abfff640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:35:26.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.655+0000 7f92c9fbc640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:35:26.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.655+0000 7f92c9fbc640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:35:26.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.655+0000 7f92c9fbc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:35:26.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.655+0000 7f92c9fbc640 1 -- 172.21.15.121:0/3684242094 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f92c406ce60 con 0x7f92c40a0a70 2024-09-17T18:35:26.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.655+0000 7f92abfff640 1 -- 172.21.15.121:0/3684242094 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f92b401bc00 con 0x7f92c40a0a70 2024-09-17T18:35:26.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.655+0000 7f92abfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:35:26.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.655+0000 7f92abfff640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:35:26.658 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":60,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:35:26.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.657+0000 7f92c9fbc640 10 librados: watch_flush enter 2024-09-17T18:35:26.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.657+0000 7f92c9fbc640 10 librados: watch_flush exit 2024-09-17T18:35:26.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.657+0000 7f92c9fbc640 20 client.104100.objecter shutdown clearing up homeless session... 2024-09-17T18:35:26.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.657+0000 7f92c9fbc640 10 client.104100.objecter successfully canceled tick 2024-09-17T18:35:26.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.658+0000 7f92c9fbc640 1 -- 172.21.15.121:0/3684242094 >> v2:172.21.15.173:6812/2423786037 conn(0x7f92b0040170 msgr2=0x7f92b0042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:26.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.658+0000 7f92c9fbc640 1 --2- 172.21.15.121:0/3684242094 >> v2:172.21.15.173:6812/2423786037 conn(0x7f92b0040170 0x7f92b0042630 secure :-1 s=READY pgs=4070 cs=0 l=1 rev1=1 crypto rx=0x7f92b8034000 tx=0x7f92b802e000 comp rx=0 tx=0).stop 2024-09-17T18:35:26.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.658+0000 7f92c9fbc640 10 monclient: shutdown 2024-09-17T18:35:26.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.658+0000 7f92c9fbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:35:26.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.658+0000 7f92c9fbc640 1 -- 172.21.15.121:0/3684242094 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92c40a0a70 msgr2=0x7f92c41cba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:26.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.658+0000 7f92c9fbc640 1 --2- 172.21.15.121:0/3684242094 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92c40a0a70 0x7f92c41cba10 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f92b400cc10 tx=0x7f92b400a430 comp rx=0 tx=0).stop 2024-09-17T18:35:26.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.658+0000 7f92c9fbc640 1 -- 172.21.15.121:0/3684242094 shutdown_connections 2024-09-17T18:35:26.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.658+0000 7f92c9fbc640 1 --2- 172.21.15.121:0/3684242094 >> v2:172.21.15.173:6812/2423786037 conn(0x7f92b0040170 0x7f92b0042630 unknown :-1 s=CLOSED pgs=4070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:26.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.658+0000 7f92c9fbc640 1 --2- 172.21.15.121:0/3684242094 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92c413a150 0x7f92c41cc490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:26.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.658+0000 7f92c9fbc640 1 --2- 172.21.15.121:0/3684242094 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92c41348f0 0x7f92c41cbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:26.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.658+0000 7f92c9fbc640 1 --2- 172.21.15.121:0/3684242094 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92c40a0a70 0x7f92c41cba10 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:26.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.658+0000 7f92c9fbc640 1 -- 172.21.15.121:0/3684242094 >> 172.21.15.121:0/3684242094 conn(0x7f92c40a3b50 msgr2=0x7f92c413c810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:35:26.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.658+0000 7f92c9fbc640 1 -- 172.21.15.121:0/3684242094 shutdown_connections 2024-09-17T18:35:26.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.658+0000 7f92c9fbc640 1 -- 172.21.15.121:0/3684242094 wait complete. 2024-09-17T18:35:26.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.658+0000 7f92c9fbc640 1 librados: shutdown 2024-09-17T18:35:26.670 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T18:35:26.670 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T18:35:26.670 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T18:35:26.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.765+0000 7f61ec189640 10 monclient: get_monmap_and_config 2024-09-17T18:35:26.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.765+0000 7f61ec189640 10 monclient: build_initial_monmap 2024-09-17T18:35:26.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.765+0000 7f61ec189640 10 monclient: monmap: 2024-09-17T18:35:26.767 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:35:26.767 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:26.767 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:35:26.766265+0000 2024-09-17T18:35:26.767 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:35:26.766265+0000 2024-09-17T18:35:26.768 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:35:26.768 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:26.768 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:35:26.768 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:35:26.768 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:35:26.768 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:35:26.768 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:35:26.768 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:26.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.766+0000 7f61ec189640 1 Processor -- start 2024-09-17T18:35:26.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.766+0000 7f61ec189640 1 -- start start 2024-09-17T18:35:26.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.766+0000 7f61ec189640 10 monclient: init 2024-09-17T18:35:26.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.766+0000 7f61ec189640 10 monclient: _reopen_session rank -1 2024-09-17T18:35:26.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.766+0000 7f61ec189640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-17T18:35:26.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.766+0000 7f61ec189640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f61e409e550 0x7f61e409e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:26.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.766+0000 7f61ec189640 10 monclient(hunting): picked mon.noname-a con 0x7f61e409e550 addr v2:172.21.15.121:3300/0 2024-09-17T18:35:26.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.766+0000 7f61ec189640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f61e409ee70 0x7f61e4139840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:26.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.766+0000 7f61ec189640 10 monclient(hunting): picked mon.noname-b con 0x7f61e409ee70 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:35:26.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.766+0000 7f61ec189640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f61e409f2c0 0x7f61e413bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:26.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.766+0000 7f61ec189640 10 monclient(hunting): picked mon.noname-c con 0x7f61e409f2c0 addr v2:172.21.15.173:3300/0 2024-09-17T18:35:26.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.766+0000 7f61ec189640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:26.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.766+0000 7f61ec189640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f61e40a4950 con 0x7f61e409e550 2024-09-17T18:35:26.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.766+0000 7f61ec189640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:26.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.766+0000 7f61ec189640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f61e40a4460 con 0x7f61e409f2c0 2024-09-17T18:35:26.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.766+0000 7f61ec189640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:26.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.766+0000 7f61ec189640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f61e40a2b60 con 0x7f61e409ee70 2024-09-17T18:35:26.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.766+0000 7f61ec189640 10 monclient(hunting): _renew_subs 2024-09-17T18:35:26.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.766+0000 7f61ec189640 10 monclient(hunting): authenticate will time out at 1730.482922s 2024-09-17T18:35:26.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.767+0000 7f61e9724640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f61e409ee70 0x7f61e4139840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:26.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.767+0000 7f61e9f25640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f61e409e550 0x7f61e409e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:26.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.767+0000 7f61e9724640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f61e409ee70 0x7f61e4139840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:53402/0 (socket says 172.21.15.121:53402) 2024-09-17T18:35:26.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.767+0000 7f61e9724640 1 -- 172.21.15.121:0/502825413 learned_addr learned my addr 172.21.15.121:0/502825413 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:35:26.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.767+0000 7f61e9f25640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f61e409e550 0x7f61e409e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54314/0 (socket says 172.21.15.121:54314) 2024-09-17T18:35:26.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.767+0000 7f61e9724640 10 monclient(hunting): get_auth_request con 0x7f61e409ee70 auth_method 0 2024-09-17T18:35:26.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.767+0000 7f61e9724640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:26.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.767+0000 7f61e9724640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:26.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.767+0000 7f61e9724640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:26.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.767+0000 7f61e9f25640 10 monclient(hunting): get_auth_request con 0x7f61e409e550 auth_method 0 2024-09-17T18:35:26.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.767+0000 7f61e9f25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:26.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.767+0000 7f61e9f25640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:26.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.767+0000 7f61e9f25640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:26.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.767+0000 7f61e9724640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:26.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.767+0000 7f61e9724640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:26.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.767+0000 7f61e9724640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:26.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.767+0000 7f61e9f25640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:26.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.767+0000 7f61e9f25640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:26.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.767+0000 7f61e9f25640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:26.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.767+0000 7f61ea726640 1 --2- 172.21.15.121:0/502825413 >> v2:172.21.15.173:3300/0 conn(0x7f61e409f2c0 0x7f61e413bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:26.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.767+0000 7f61ea726640 10 monclient(hunting): get_auth_request con 0x7f61e409f2c0 auth_method 0 2024-09-17T18:35:26.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.767+0000 7f61ea726640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:26.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.767+0000 7f61ea726640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:26.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.767+0000 7f61ea726640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:26.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.767+0000 7f61e9724640 10 monclient(hunting): handle_auth_done global_id 64460 payload 274 2024-09-17T18:35:26.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.767+0000 7f61e9724640 1 -- 172.21.15.121:0/502825413 >> v2:172.21.15.173:3300/0 conn(0x7f61e409f2c0 msgr2=0x7f61e413bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:35:26.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.767+0000 7f61e9724640 1 --2- 172.21.15.121:0/502825413 >> v2:172.21.15.173:3300/0 conn(0x7f61e409f2c0 0x7f61e413bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:26.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.767+0000 7f61e9724640 1 -- 172.21.15.121:0/502825413 >> v2:172.21.15.121:3300/0 conn(0x7f61e409e550 msgr2=0x7f61e409e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:35:26.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.767+0000 7f61e9724640 1 --2- 172.21.15.121:0/502825413 >> v2:172.21.15.121:3300/0 conn(0x7f61e409e550 0x7f61e409e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:26.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.767+0000 7f61e9724640 10 monclient: _finish_hunting 0 2024-09-17T18:35:26.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.767+0000 7f61e9724640 1 monclient: found mon.noname-b 2024-09-17T18:35:26.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.767+0000 7f61e9724640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:35:26.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.767+0000 7f61e9724640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:35:26.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.767+0000 7f61e9724640 1 -- 172.21.15.121:0/502825413 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61e409f840 con 0x7f61e409ee70 2024-09-17T18:35:26.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.767+0000 7f61e9f25640 1 --2- 172.21.15.121:0/502825413 >> v2:172.21.15.121:3300/0 conn(0x7f61e409e550 0x7f61e409e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:35:26.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.770+0000 7f61e9724640 1 -- 172.21.15.121:0/502825413 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f61e409ee70 msgr2=0x7f61e4139840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:26.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.770+0000 7f61e9724640 1 --2- 172.21.15.121:0/502825413 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f61e409ee70 0x7f61e4139840 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f61e40a31f0 tx=0x7f61d402a0b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1aa50c0fa8e2099c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:26.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.770+0000 7f61e8f23640 1 -- 172.21.15.121:0/502825413 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61d4037070 con 0x7f61e409ee70 2024-09-17T18:35:26.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.770+0000 7f61e8f23640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:26.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.770+0000 7f61e8f23640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:35:26.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.770+0000 7f61e8f23640 10 monclient: dump: 2024-09-17T18:35:26.782 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:26.782 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:26.782 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:26.782 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:26.783 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:26.783 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:26.783 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:26.783 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:26.783 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:26.783 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:26.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.770+0000 7f61e8f23640 10 monclient: _finish_auth 0 2024-09-17T18:35:26.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.770+0000 7f61e8f23640 10 monclient: _check_auth_tickets 2024-09-17T18:35:26.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.770+0000 7f61e8f23640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:35:26.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.770+0000 7f61e8f23640 1 -- 172.21.15.121:0/502825413 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61d4032640 con 0x7f61e409ee70 2024-09-17T18:35:26.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.770+0000 7f61ec189640 5 monclient: authenticate success, global_id 64460 2024-09-17T18:35:26.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.770+0000 7f61ec189640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:35:26.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.770+0000 7f61e8f23640 10 monclient: handle_config config(0 keys) 2024-09-17T18:35:26.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.770+0000 7f61ec189640 10 monclient: get_monmap_and_config success 2024-09-17T18:35:26.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.770+0000 7f61e8f23640 1 -- 172.21.15.121:0/502825413 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61d4032a10 con 0x7f61e409ee70 2024-09-17T18:35:26.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.770+0000 7f61e8f23640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:26.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.770+0000 7f61ec189640 4 set_mon_vals no callback set 2024-09-17T18:35:26.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.770+0000 7f61e8f23640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:35:26.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.770+0000 7f61e8f23640 10 monclient: dump: 2024-09-17T18:35:26.784 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:26.784 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:26.784 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:26.784 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:26.784 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:26.784 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:26.784 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:26.785 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:26.785 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:26.785 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:26.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.770+0000 7f61ec189640 10 monclient: shutdown 2024-09-17T18:35:26.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.770+0000 7f61ec189640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:35:26.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.770+0000 7f61ec189640 1 -- 172.21.15.121:0/502825413 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f61e409ee70 msgr2=0x7f61e4139840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:26.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.770+0000 7f61ec189640 1 --2- 172.21.15.121:0/502825413 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f61e409ee70 0x7f61e4139840 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f61e40a31f0 tx=0x7f61d402a0b0 comp rx=0 tx=0).stop 2024-09-17T18:35:26.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.770+0000 7f61ec189640 1 -- 172.21.15.121:0/502825413 shutdown_connections 2024-09-17T18:35:26.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.770+0000 7f61ec189640 1 --2- 172.21.15.121:0/502825413 >> v2:172.21.15.173:3300/0 conn(0x7f61e409f2c0 0x7f61e413bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:26.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.770+0000 7f61ec189640 1 --2- 172.21.15.121:0/502825413 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f61e409ee70 0x7f61e4139840 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:26.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.770+0000 7f61ec189640 1 --2- 172.21.15.121:0/502825413 >> v2:172.21.15.121:3300/0 conn(0x7f61e409e550 0x7f61e409e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:26.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.770+0000 7f61ec189640 1 -- 172.21.15.121:0/502825413 >> 172.21.15.121:0/502825413 conn(0x7f61e40a7830 msgr2=0x7f61e40a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:35:26.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.770+0000 7f61ec189640 1 -- 172.21.15.121:0/502825413 shutdown_connections 2024-09-17T18:35:26.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.770+0000 7f61ec189640 1 -- 172.21.15.121:0/502825413 wait complete. 2024-09-17T18:35:26.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.771+0000 7f61ec189640 10 monclient: build_initial_monmap 2024-09-17T18:35:26.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.771+0000 7f61ec189640 10 monclient: monmap: 2024-09-17T18:35:26.786 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:35:26.786 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:35:26.786 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:35:26.786 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:35:26.786 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:35:26.786 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:26.786 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:35:26.786 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:35:26.786 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:35:26.786 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:26.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.771+0000 7f61ec189640 1 librados: starting msgr at 2024-09-17T18:35:26.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.771+0000 7f61ec189640 1 librados: starting objecter 2024-09-17T18:35:26.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.771+0000 7f61ec189640 1 Processor -- start 2024-09-17T18:35:26.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.771+0000 7f61ec189640 1 -- start start 2024-09-17T18:35:26.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.771+0000 7f61ec189640 1 librados: setting wanted keys 2024-09-17T18:35:26.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.771+0000 7f61ec189640 1 librados: calling monclient init 2024-09-17T18:35:26.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.771+0000 7f61ec189640 10 monclient: init 2024-09-17T18:35:26.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61ec189640 10 monclient: _reopen_session rank -1 2024-09-17T18:35:26.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61ec189640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T18:35:26.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61ec189640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f61e409e550 0x7f61e413b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:26.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61ec189640 10 monclient(hunting): picked mon.noname-b con 0x7f61e409e550 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:35:26.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61ec189640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61e409ee70 0x7f61e4133700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:26.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61ec189640 10 monclient(hunting): picked mon.noname-a con 0x7f61e409ee70 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:35:26.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61ec189640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f61e409f2c0 0x7f61e4138b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:26.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61ec189640 10 monclient(hunting): picked mon.noname-c con 0x7f61e409f2c0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:35:26.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61ec189640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:26.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61ec189640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f61e409de60 con 0x7f61e409ee70 2024-09-17T18:35:26.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61ec189640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:26.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61ec189640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f61e409dce0 con 0x7f61e409e550 2024-09-17T18:35:26.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61ec189640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:26.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61ec189640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f61e40747e0 con 0x7f61e409f2c0 2024-09-17T18:35:26.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61ec189640 10 monclient(hunting): _renew_subs 2024-09-17T18:35:26.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61ec189640 10 monclient(hunting): authenticate will time out at 1730.488261s 2024-09-17T18:35:26.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61e9724640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61e409ee70 0x7f61e4133700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:26.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61ea726640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f61e409f2c0 0x7f61e4138b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:26.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61e9724640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61e409ee70 0x7f61e4133700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54322/0 (socket says 172.21.15.121:54322) 2024-09-17T18:35:26.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61ea726640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f61e409f2c0 0x7f61e4138b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:53410/0 (socket says 172.21.15.121:53410) 2024-09-17T18:35:26.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61e9724640 1 -- 172.21.15.121:0/3987814153 learned_addr learned my addr 172.21.15.121:0/3987814153 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:35:26.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61e9724640 10 monclient(hunting): get_auth_request con 0x7f61e409ee70 auth_method 0 2024-09-17T18:35:26.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61e9724640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:26.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61e9724640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:26.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61e9724640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:26.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61ea726640 10 monclient(hunting): get_auth_request con 0x7f61e409f2c0 auth_method 0 2024-09-17T18:35:26.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61ea726640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:26.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61ea726640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:26.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61ea726640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:26.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61e9724640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:26.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61e9724640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:26.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61e9724640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:26.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61ea726640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:26.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61ea726640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:26.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61ea726640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:26.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61e9f25640 1 --2- 172.21.15.121:0/3987814153 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f61e409e550 0x7f61e413b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:26.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61e9f25640 10 monclient(hunting): get_auth_request con 0x7f61e409e550 auth_method 0 2024-09-17T18:35:26.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61e9f25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:26.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61e9f25640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:26.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61e9f25640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:26.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61e9f25640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:26.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61e9f25640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:26.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61e9f25640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:26.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61e9724640 10 monclient(hunting): handle_auth_done global_id 104106 payload 824 2024-09-17T18:35:26.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61e9724640 1 -- 172.21.15.121:0/3987814153 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f61e409f2c0 msgr2=0x7f61e4138b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:26.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61e9724640 1 --2- 172.21.15.121:0/3987814153 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f61e409f2c0 0x7f61e4138b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:26.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61e9724640 1 -- 172.21.15.121:0/3987814153 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f61e409e550 msgr2=0x7f61e413b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:26.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61e9724640 1 --2- 172.21.15.121:0/3987814153 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f61e409e550 0x7f61e413b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:26.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61e9724640 10 monclient: _finish_hunting 0 2024-09-17T18:35:26.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61e9724640 1 monclient: found mon.noname-a 2024-09-17T18:35:26.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61e9724640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:35:26.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61e9724640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:35:26.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61e9724640 1 -- 172.21.15.121:0/3987814153 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61e413bb40 con 0x7f61e409ee70 2024-09-17T18:35:26.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.772+0000 7f61ea726640 1 --2- 172.21.15.121:0/3987814153 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f61e409f2c0 0x7f61e4138b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:35:26.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.774+0000 7f61e9724640 1 -- 172.21.15.121:0/3987814153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61e409ee70 msgr2=0x7f61e4133700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:26.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.774+0000 7f61e9724640 1 --2- 172.21.15.121:0/3987814153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61e409ee70 0x7f61e4133700 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f61d40090b0 tx=0x7f61d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:26.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.774+0000 7f61da7fc640 10 client.?.objecter ms_handle_connect 0x7f61e409ee70 2024-09-17T18:35:26.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.774+0000 7f61da7fc640 10 client.?.objecter resend_mon_ops 2024-09-17T18:35:26.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.774+0000 7f61da7fc640 1 -- 172.21.15.121:0/3987814153 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61d4041020 con 0x7f61e409ee70 2024-09-17T18:35:26.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.775+0000 7f61da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:26.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.775+0000 7f61da7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:35:26.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.775+0000 7f61da7fc640 10 monclient: dump: 2024-09-17T18:35:26.792 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:26.792 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:26.792 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:26.792 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:26.792 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:26.792 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:26.792 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:26.792 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:26.792 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:26.792 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:26.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.775+0000 7f61da7fc640 10 monclient: _finish_auth 0 2024-09-17T18:35:26.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.775+0000 7f61da7fc640 10 monclient: _check_auth_tickets 2024-09-17T18:35:26.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.775+0000 7f61da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:35:26.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.775+0000 7f61da7fc640 1 -- 172.21.15.121:0/3987814153 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61d403d830 con 0x7f61e409ee70 2024-09-17T18:35:26.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.775+0000 7f61ec189640 5 monclient: authenticate success, global_id 104106 2024-09-17T18:35:26.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.775+0000 7f61da7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:35:26.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.775+0000 7f61da7fc640 1 -- 172.21.15.121:0/3987814153 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61d4037040 con 0x7f61e409ee70 2024-09-17T18:35:26.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.775+0000 7f61ec189640 10 monclient: _renew_subs 2024-09-17T18:35:26.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.775+0000 7f61ec189640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:35:26.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.775+0000 7f61ec189640 1 -- 172.21.15.121:0/3987814153 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61e413a650 con 0x7f61e409ee70 2024-09-17T18:35:26.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.775+0000 7f61da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:26.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.775+0000 7f61da7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:35:26.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.775+0000 7f61da7fc640 10 monclient: dump: 2024-09-17T18:35:26.793 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:26.793 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:26.793 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:26.793 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:26.793 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:26.793 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:26.793 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:26.794 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:26.794 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:26.794 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:26.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.775+0000 7f61dbfff640 4 set_mon_vals no callback set 2024-09-17T18:35:26.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.775+0000 7f61ec189640 10 client.104106.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:35:26.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.775+0000 7f61ec189640 10 monclient: _renew_subs 2024-09-17T18:35:26.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.775+0000 7f61ec189640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:35:26.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.775+0000 7f61ec189640 1 -- 172.21.15.121:0/3987814153 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61e411d7f0 con 0x7f61e409ee70 2024-09-17T18:35:26.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.775+0000 7f61ec189640 1 librados: init done 2024-09-17T18:35:26.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.775+0000 7f61ec189640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:26.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.775+0000 7f61ec189640 1 --2- 172.21.15.121:0/3987814153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61e41390d0 0x7f61e41394b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:26.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.775+0000 7f61ec189640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:26.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.775+0000 7f61ec189640 1 -- 172.21.15.121:0/3987814153 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f61e41cc100 con 0x7f61e41390d0 2024-09-17T18:35:26.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.775+0000 7f61da7fc640 1 -- 172.21.15.121:0/3987814153 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f61d4044020 con 0x7f61e409ee70 2024-09-17T18:35:26.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.775+0000 7f61e9f25640 1 --2- 172.21.15.121:0/3987814153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61e41390d0 0x7f61e41394b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:26.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.775+0000 7f61e9f25640 10 monclient: get_auth_request con 0x7f61e41390d0 auth_method 0 2024-09-17T18:35:26.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.775+0000 7f61e9f25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:26.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.775+0000 7f61e9f25640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:26.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.775+0000 7f61e9f25640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:26.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.776+0000 7f61e9f25640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:26.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.776+0000 7f61e9f25640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:26.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.776+0000 7f61e9f25640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:26.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.776+0000 7f61e9f25640 10 monclient(hunting): handle_auth_done global_id 104109 payload 824 2024-09-17T18:35:26.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.776+0000 7f61da7fc640 1 --2- 172.21.15.121:0/3987814153 >> v2:172.21.15.173:6812/2423786037 conn(0x7f61c80401a0 0x7f61c8042660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:26.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.776+0000 7f61e9f25640 1 -- 172.21.15.121:0/3987814153 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f61e41399f0 con 0x7f61e41390d0 2024-09-17T18:35:26.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.776+0000 7f61da7fc640 1 -- 172.21.15.121:0/3987814153 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(646..646 src has 247..646) ==== 5925+0+0 (secure 0 0 0) 0x7f61d4073e60 con 0x7f61e409ee70 2024-09-17T18:35:26.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.776+0000 7f61da7fc640 10 client.104106.objecter ms_dispatch 0x7f61e4000b90 osd_map(646..646 src has 247..646) 2024-09-17T18:35:26.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.776+0000 7f61da7fc640 3 client.104106.objecter handle_osd_map got epochs [646,646] > 0 2024-09-17T18:35:26.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.776+0000 7f61da7fc640 3 client.104106.objecter handle_osd_map decoding full epoch 646 2024-09-17T18:35:26.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.776+0000 7f61da7fc640 20 client.104106.objecter dump_active .. 0 homeless 2024-09-17T18:35:26.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.776+0000 7f61ea726640 1 --2- 172.21.15.121:0/3987814153 >> v2:172.21.15.173:6812/2423786037 conn(0x7f61c80401a0 0x7f61c8042660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:26.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.776+0000 7f61ea726640 10 monclient: get_auth_request con 0x7f61c80401a0 auth_method 0 2024-09-17T18:35:26.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.778+0000 7f61da7fc640 10 client.104106.objecter ms_handle_connect 0x7f61e41390d0 2024-09-17T18:35:26.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.778+0000 7f61da7fc640 10 client.104106.objecter resend_mon_ops 2024-09-17T18:35:26.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.778+0000 7f61e9f25640 1 -- 172.21.15.121:0/3987814153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61e41390d0 msgr2=0x7f61e41394b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:26.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.778+0000 7f61e9f25640 1 --2- 172.21.15.121:0/3987814153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61e41390d0 0x7f61e41394b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f61e00118e0 tx=0x7f61e0011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:26.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.778+0000 7f61da7fc640 1 -- 172.21.15.121:0/3987814153 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e0014440 con 0x7f61e41390d0 2024-09-17T18:35:26.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.778+0000 7f61da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:26.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.778+0000 7f61da7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:35:26.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.778+0000 7f61da7fc640 10 monclient: dump: 2024-09-17T18:35:26.796 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:26.796 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:26.796 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:26.796 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:26.796 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:26.797 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:26.797 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:26.797 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:26.797 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:26.797 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:26.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.778+0000 7f61da7fc640 1 -- 172.21.15.121:0/3987814153 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f61e0014440 con 0x7f61e41390d0 2024-09-17T18:35:26.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.778+0000 7f61da7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:26.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.778+0000 7f61da7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:35:26.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.778+0000 7f61da7fc640 1 -- 172.21.15.121:0/3987814153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61e41390d0 msgr2=0x7f61e41394b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:26.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.778+0000 7f61da7fc640 1 --2- 172.21.15.121:0/3987814153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61e41390d0 0x7f61e41394b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f61e00118e0 tx=0x7f61e0011da0 comp rx=0 tx=0).stop 2024-09-17T18:35:26.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.778+0000 7f61da7fc640 1 -- 172.21.15.121:0/3987814153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61e41390d0 msgr2=0x7f61e41394b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:35:26.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.778+0000 7f61da7fc640 1 --2- 172.21.15.121:0/3987814153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61e41390d0 0x7f61e41394b0 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f61e00118e0 tx=0x7f61e0011da0 comp rx=0 tx=0).stop 2024-09-17T18:35:26.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.779+0000 7f61ea726640 1 --2- 172.21.15.121:0/3987814153 >> v2:172.21.15.173:6812/2423786037 conn(0x7f61c80401a0 0x7f61c8042660 secure :-1 s=READY pgs=4071 cs=0 l=1 rev1=1 crypto rx=0x7f61e413af40 tx=0x7f61dc009820 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:26.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.779+0000 7f61da7fc640 10 client.104106.objecter ms_handle_connect 0x7f61c80401a0 2024-09-17T18:35:26.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.784+0000 7f61ec189640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:35:26.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.784+0000 7f61ec189640 1 --2- 172.21.15.121:0/3987814153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61e4070930 0x7f61e4070d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:26.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.784+0000 7f61ec189640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:26.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.784+0000 7f61ec189640 1 -- 172.21.15.121:0/3987814153 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f61e409e930 con 0x7f61e4070930 2024-09-17T18:35:26.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.784+0000 7f61e9f25640 1 --2- 172.21.15.121:0/3987814153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61e4070930 0x7f61e4070d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:26.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.784+0000 7f61e9f25640 10 monclient: get_auth_request con 0x7f61e4070930 auth_method 0 2024-09-17T18:35:26.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.784+0000 7f61e9f25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:26.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.784+0000 7f61e9f25640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:26.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.784+0000 7f61e9f25640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:26.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.784+0000 7f61e9f25640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:26.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.784+0000 7f61e9f25640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:26.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.784+0000 7f61e9f25640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:26.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.785+0000 7f61e9f25640 10 monclient(hunting): handle_auth_done global_id 104112 payload 824 2024-09-17T18:35:26.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.785+0000 7f61e9f25640 1 -- 172.21.15.121:0/3987814153 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f61e41c31f0 con 0x7f61e4070930 2024-09-17T18:35:26.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.787+0000 7f61da7fc640 10 client.104106.objecter ms_handle_connect 0x7f61e4070930 2024-09-17T18:35:26.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.787+0000 7f61e9f25640 1 -- 172.21.15.121:0/3987814153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61e4070930 msgr2=0x7f61e4070d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:26.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.787+0000 7f61da7fc640 10 client.104106.objecter resend_mon_ops 2024-09-17T18:35:26.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.787+0000 7f61e9f25640 1 --2- 172.21.15.121:0/3987814153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61e4070930 0x7f61e4070d40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f61e00118e0 tx=0x7f61e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:26.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.787+0000 7f61da7fc640 1 -- 172.21.15.121:0/3987814153 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e00229a0 con 0x7f61e4070930 2024-09-17T18:35:26.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.787+0000 7f61da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:26.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.787+0000 7f61da7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:35:26.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.787+0000 7f61da7fc640 10 monclient: dump: 2024-09-17T18:35:26.801 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:26.801 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:26.801 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:26.801 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:26.801 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:26.801 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:26.801 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:26.801 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:26.801 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:26.801 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:26.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.787+0000 7f61da7fc640 1 -- 172.21.15.121:0/3987814153 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4553 (secure 0 0 0) 0x7f61e001d400 con 0x7f61e4070930 2024-09-17T18:35:26.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.787+0000 7f61da7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:35:26.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.787+0000 7f61da7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:35:26.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.787+0000 7f61da7fc640 1 -- 172.21.15.121:0/3987814153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61e4070930 msgr2=0x7f61e4070d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:26.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.787+0000 7f61da7fc640 1 --2- 172.21.15.121:0/3987814153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61e4070930 0x7f61e4070d40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f61e00118e0 tx=0x7f61e00046c0 comp rx=0 tx=0).stop 2024-09-17T18:35:26.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.787+0000 7f61da7fc640 1 -- 172.21.15.121:0/3987814153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61e4070930 msgr2=0x7f61e4070d40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:35:26.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.787+0000 7f61da7fc640 1 --2- 172.21.15.121:0/3987814153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61e4070930 0x7f61e4070d40 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:26.802 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:35:26.802 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:35:26.802 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:35:26.802 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "leader", 2024-09-17T18:35:26.802 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 60, 2024-09-17T18:35:26.803 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 7150, 2024-09-17T18:35:26.803 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:35:26.803 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:35:26.803 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:35:26.803 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:35:26.803 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:35:26.803 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 0, 2024-09-17T18:35:26.803 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:35:26.803 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:35:26.803 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:35:26.803 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:35:26.803 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:35:26.803 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:35:26.803 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:35:26.803 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:35:26.803 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:35:26.804 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:35:26.804 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:35:26.804 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:35:26.804 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:35:26.804 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:35:26.804 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:35:26.804 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:35:26.804 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:35:26.804 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:35:26.804 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:35:26.804 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:35:26.804 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:35:26.804 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:35:26.804 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:35:26.805 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:35:26.805 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:35:26.805 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:35:26.805 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:35:26.805 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:35:26.805 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:26.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:26.805 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:35:26.805 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:35:26.805 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:35:26.805 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:35:26.805 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:35:26.805 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:35:26.805 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:35:26.805 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:35:26.805 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:35:26.806 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:35:26.806 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:35:26.806 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:35:26.806 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:35:26.806 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:35:26.806 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:35:26.806 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:35:26.806 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:35:26.806 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:35:26.806 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:35:26.806 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:35:26.806 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:35:26.806 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:35:26.806 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:35:26.807 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:35:26.807 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:35:26.807 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:35:26.807 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:35:26.807 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:35:26.807 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:35:26.807 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:35:26.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:26.807 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:35:26.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:26.807 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:35:26.807 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:35:26.807 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:35:26.807 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:35:26.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:26.808 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:35:26.808 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:35:26.808 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:26.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:26.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:26.808 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:35:26.808 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:35:26.808 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:26.808 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:26.808 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:26.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:26.808 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:35:26.808 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:35:26.808 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:35:26.808 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:35:26.809 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:35:26.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:26.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:26.809 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:35:26.809 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:35:26.809 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:35:26.809 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:35:26.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:26.809 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:35:26.809 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:35:26.809 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:26.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:26.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:26.810 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:35:26.810 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:35:26.810 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:26.810 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:26.810 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:26.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:26.810 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:35:26.810 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:35:26.810 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:35:26.810 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:35:26.810 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:35:26.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:26.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:26.810 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:35:26.810 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:35:26.810 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:35:26.811 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:35:26.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:26.811 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:35:26.811 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:35:26.811 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:26.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:26.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:26.811 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:35:26.811 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:35:26.811 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:26.811 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:26.811 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:26.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:26.811 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:35:26.811 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:35:26.811 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:35:26.811 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:35:26.812 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:35:26.812 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:26.812 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:26.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:26.812 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:35:26.812 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:35:26.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:26.812 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:35:26.812 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:35:26.812 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:35:26.812 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:26.812 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:35:26.812 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:35:26.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:26.813 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:35:26.813 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:35:26.813 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-17T18:35:26.813 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:26.813 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:26.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:26.813 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:35:26.813 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:35:26.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.788+0000 7f61ec189640 10 librados: watch_flush enter 2024-09-17T18:35:26.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.788+0000 7f61ec189640 10 librados: watch_flush exit 2024-09-17T18:35:26.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.788+0000 7f61ec189640 20 client.104106.objecter shutdown clearing up homeless session... 2024-09-17T18:35:26.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.788+0000 7f61ec189640 10 client.104106.objecter successfully canceled tick 2024-09-17T18:35:26.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.788+0000 7f61ec189640 1 -- 172.21.15.121:0/3987814153 >> v2:172.21.15.173:6812/2423786037 conn(0x7f61c80401a0 msgr2=0x7f61c8042660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:26.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.788+0000 7f61ec189640 1 --2- 172.21.15.121:0/3987814153 >> v2:172.21.15.173:6812/2423786037 conn(0x7f61c80401a0 0x7f61c8042660 secure :-1 s=READY pgs=4071 cs=0 l=1 rev1=1 crypto rx=0x7f61e413af40 tx=0x7f61dc009820 comp rx=0 tx=0).stop 2024-09-17T18:35:26.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.788+0000 7f61ec189640 10 monclient: shutdown 2024-09-17T18:35:26.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.788+0000 7f61ec189640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:35:26.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.788+0000 7f61ec189640 1 -- 172.21.15.121:0/3987814153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61e409ee70 msgr2=0x7f61e4133700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:26.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.788+0000 7f61ec189640 1 --2- 172.21.15.121:0/3987814153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61e409ee70 0x7f61e4133700 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f61d40090b0 tx=0x7f61d40046c0 comp rx=0 tx=0).stop 2024-09-17T18:35:26.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.788+0000 7f61ea726640 1 -- 172.21.15.121:0/3987814153 reap_dead start 2024-09-17T18:35:26.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.788+0000 7f61ea726640 1 -- 172.21.15.121:0/3987814153 reap_dead start 2024-09-17T18:35:26.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.788+0000 7f61ec189640 1 -- 172.21.15.121:0/3987814153 shutdown_connections 2024-09-17T18:35:26.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.788+0000 7f61ec189640 1 -- 172.21.15.121:0/3987814153 >> 172.21.15.121:0/3987814153 conn(0x7f61e40a7830 msgr2=0x7f61e40a4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:35:26.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.788+0000 7f61ec189640 1 -- 172.21.15.121:0/3987814153 shutdown_connections 2024-09-17T18:35:26.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.788+0000 7f61ec189640 1 -- 172.21.15.121:0/3987814153 wait complete. 2024-09-17T18:35:26.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.788+0000 7f61ec189640 1 librados: shutdown 2024-09-17T18:35:26.816 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:35:26.816 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T18:35:26.816 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T18:35:26.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.913+0000 7fa7e9764640 10 monclient: get_monmap_and_config 2024-09-17T18:35:26.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.913+0000 7fa7e9764640 10 monclient: build_initial_monmap 2024-09-17T18:35:26.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.913+0000 7fa7e9764640 10 monclient: monmap: 2024-09-17T18:35:26.915 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:35:26.915 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:26.915 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:35:26.913777+0000 2024-09-17T18:35:26.915 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:35:26.913777+0000 2024-09-17T18:35:26.915 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:35:26.915 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:26.915 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:35:26.915 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:35:26.915 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:35:26.915 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:35:26.915 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:35:26.915 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:26.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.913+0000 7fa7e9764640 1 Processor -- start 2024-09-17T18:35:26.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.913+0000 7fa7e9764640 1 -- start start 2024-09-17T18:35:26.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.913+0000 7fa7e9764640 10 monclient: init 2024-09-17T18:35:26.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.914+0000 7fa7e9764640 10 monclient: _reopen_session rank -1 2024-09-17T18:35:26.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.914+0000 7fa7e9764640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-17T18:35:26.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.914+0000 7fa7e9764640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa7e40700f0 addr v1:172.21.15.173:6789/0 2024-09-17T18:35:26.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.914+0000 7fa7e9764640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fa7e409e810 0x7fa7e413d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:26.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.914+0000 7fa7e9764640 10 monclient(hunting): picked mon.noname-c con 0x7fa7e409e810 addr v2:172.21.15.173:3300/0 2024-09-17T18:35:26.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.914+0000 7fa7e9764640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa7e409ec60 0x7fa7e413fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:26.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.914+0000 7fa7e9764640 10 monclient(hunting): picked mon.noname-a con 0x7fa7e409ec60 addr v2:172.21.15.121:3300/0 2024-09-17T18:35:26.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.914+0000 7fa7e9764640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7e4135560 con 0x7fa7e40700f0 2024-09-17T18:35:26.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.914+0000 7fa7e9764640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:26.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.914+0000 7fa7e9764640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fa7e40704d0 con 0x7fa7e409ec60 2024-09-17T18:35:26.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.914+0000 7fa7e9764640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:26.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.914+0000 7fa7e9764640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fa7e409ca60 con 0x7fa7e409e810 2024-09-17T18:35:26.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.914+0000 7fa7e9764640 10 monclient(hunting): _renew_subs 2024-09-17T18:35:26.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.914+0000 7fa7e9764640 10 monclient(hunting): authenticate will time out at 1730.630383s 2024-09-17T18:35:26.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.914+0000 7fa7e37fe640 1 -- >> v2:172.21.15.121:3300/0 conn(0x7fa7e409ec60 msgr2=0x7fa7e413fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T18:35:26.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.914+0000 7fa7e37fe640 1 -- >> v2:172.21.15.121:3300/0 conn(0x7fa7e409ec60 msgr2=0x7fa7e413fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:35:26.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.914+0000 7fa7e37fe640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa7e409ec60 0x7fa7e413fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) 2024-09-17T18:35:26.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.914+0000 7fa7e37fe640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa7e409ec60 0x7fa7e413fc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:35:26.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.914+0000 7fa7da7fc640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fa7e409e810 0x7fa7e413d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:26.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.914+0000 7fa7da7fc640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fa7e409e810 0x7fa7e413d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.121:37100/0 (socket says 172.21.15.121:37100) 2024-09-17T18:35:26.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.914+0000 7fa7da7fc640 1 -- 172.21.15.121:0/199264949 learned_addr learned my addr 172.21.15.121:0/199264949 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:35:26.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.914+0000 7fa7da7fc640 10 monclient(hunting): get_auth_request con 0x7fa7e409e810 auth_method 0 2024-09-17T18:35:26.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.914+0000 7fa7da7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:26.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.914+0000 7fa7da7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:26.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.914+0000 7fa7da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:26.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.914+0000 7fa7da7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:26.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.914+0000 7fa7da7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:26.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.914+0000 7fa7da7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:26.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.914+0000 7fa7da7fc640 10 monclient(hunting): handle_auth_done global_id 84247 payload 274 2024-09-17T18:35:26.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.914+0000 7fa7da7fc640 1 -- 172.21.15.121:0/199264949 >> v2:172.21.15.121:3300/0 conn(0x7fa7e409ec60 msgr2=0x7fa7e413fc30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:35:26.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.914+0000 7fa7da7fc640 1 --2- 172.21.15.121:0/199264949 >> v2:172.21.15.121:3300/0 conn(0x7fa7e409ec60 0x7fa7e413fc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:26.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.914+0000 7fa7da7fc640 1 -- 172.21.15.121:0/199264949 >> v1:172.21.15.173:6789/0 conn(0x7fa7e40700f0 legacy=0x7fa7e4134e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:35:26.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.914+0000 7fa7da7fc640 10 monclient: _finish_hunting 0 2024-09-17T18:35:26.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.914+0000 7fa7da7fc640 1 monclient: found mon.noname-c 2024-09-17T18:35:26.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.914+0000 7fa7da7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:35:26.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.914+0000 7fa7da7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.173:3300/0 2024-09-17T18:35:26.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.914+0000 7fa7da7fc640 1 -- 172.21.15.121:0/199264949 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7e409cd20 con 0x7fa7e409e810 2024-09-17T18:35:26.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.917+0000 7fa7da7fc640 1 -- 172.21.15.121:0/199264949 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7e409e810 msgr2=0x7fa7e413d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:26.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.917+0000 7fa7da7fc640 1 --2- 172.21.15.121:0/199264949 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7e409e810 0x7fa7e413d6e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa7e40a89f0 tx=0x7fa7cc02a0b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2edf751565828c18 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:26.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.917+0000 7fa7e27fc640 1 -- 172.21.15.121:0/199264949 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7cc037070 con 0x7fa7e409e810 2024-09-17T18:35:26.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.917+0000 7fa7e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:26.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.917+0000 7fa7e27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:35:26.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.917+0000 7fa7e27fc640 10 monclient: dump: 2024-09-17T18:35:26.929 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:26.929 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:26.929 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:26.929 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:26.929 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:26.929 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:26.929 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:26.930 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:26.930 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:26.930 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:26.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.917+0000 7fa7e27fc640 10 monclient: _finish_auth 0 2024-09-17T18:35:26.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.917+0000 7fa7e27fc640 10 monclient: _check_auth_tickets 2024-09-17T18:35:26.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.917+0000 7fa7e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:35:26.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.918+0000 7fa7e9764640 5 monclient: authenticate success, global_id 84247 2024-09-17T18:35:26.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.918+0000 7fa7e27fc640 1 -- 172.21.15.121:0/199264949 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7cc032640 con 0x7fa7e409e810 2024-09-17T18:35:26.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.918+0000 7fa7e9764640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:35:26.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.918+0000 7fa7e27fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:35:26.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.918+0000 7fa7e9764640 10 monclient: get_monmap_and_config success 2024-09-17T18:35:26.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.918+0000 7fa7e9764640 4 set_mon_vals no callback set 2024-09-17T18:35:26.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.918+0000 7fa7e27fc640 1 -- 172.21.15.121:0/199264949 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7cc032a10 con 0x7fa7e409e810 2024-09-17T18:35:26.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.918+0000 7fa7e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:26.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.918+0000 7fa7e27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:35:26.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.918+0000 7fa7e27fc640 10 monclient: dump: 2024-09-17T18:35:26.931 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:26.931 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:26.931 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:26.931 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:26.931 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:26.931 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:26.931 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:26.931 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:26.931 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:26.931 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:26.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.918+0000 7fa7e9764640 10 monclient: shutdown 2024-09-17T18:35:26.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.918+0000 7fa7e9764640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:35:26.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.918+0000 7fa7e9764640 1 -- 172.21.15.121:0/199264949 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7e409e810 msgr2=0x7fa7e413d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:26.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.918+0000 7fa7e9764640 1 --2- 172.21.15.121:0/199264949 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7e409e810 0x7fa7e413d6e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa7e40a89f0 tx=0x7fa7cc02a0b0 comp rx=0 tx=0).stop 2024-09-17T18:35:26.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.918+0000 7fa7e9764640 1 -- 172.21.15.121:0/199264949 shutdown_connections 2024-09-17T18:35:26.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.918+0000 7fa7e9764640 1 --2- 172.21.15.121:0/199264949 >> v2:172.21.15.121:3300/0 conn(0x7fa7e409ec60 0x7fa7e413fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:26.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.918+0000 7fa7e9764640 1 --2- 172.21.15.121:0/199264949 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7e409e810 0x7fa7e413d6e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:26.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.918+0000 7fa7e9764640 1 -- 172.21.15.121:0/199264949 >> 172.21.15.121:0/199264949 conn(0x7fa7e40ab830 msgr2=0x7fa7e40a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:35:26.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.918+0000 7fa7e9764640 1 -- 172.21.15.121:0/199264949 shutdown_connections 2024-09-17T18:35:26.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.918+0000 7fa7e9764640 1 -- 172.21.15.121:0/199264949 wait complete. 2024-09-17T18:35:26.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.919+0000 7fa7e9764640 10 monclient: build_initial_monmap 2024-09-17T18:35:26.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.919+0000 7fa7e9764640 10 monclient: monmap: 2024-09-17T18:35:26.932 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:35:26.932 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:35:26.932 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:35:26.932 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:35:26.932 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:35:26.933 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:26.933 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:35:26.933 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:35:26.933 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:35:26.933 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:26.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.919+0000 7fa7e9764640 1 librados: starting msgr at 2024-09-17T18:35:26.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.919+0000 7fa7e9764640 1 librados: starting objecter 2024-09-17T18:35:26.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.919+0000 7fa7e9764640 1 Processor -- start 2024-09-17T18:35:26.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.919+0000 7fa7e9764640 1 -- start start 2024-09-17T18:35:26.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.919+0000 7fa7e9764640 1 librados: setting wanted keys 2024-09-17T18:35:26.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.919+0000 7fa7e9764640 1 librados: calling monclient init 2024-09-17T18:35:26.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.919+0000 7fa7e9764640 10 monclient: init 2024-09-17T18:35:26.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7e9764640 10 monclient: _reopen_session rank -1 2024-09-17T18:35:26.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7e9764640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T18:35:26.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7e9764640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7e40700f0 0x7fa7e41cd750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:26.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7e9764640 10 monclient(hunting): picked mon.noname-c con 0x7fa7e40700f0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:35:26.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7e9764640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7e409e810 0x7fa7e413f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:26.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7e9764640 10 monclient(hunting): picked mon.noname-a con 0x7fa7e409e810 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:35:26.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7e9764640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7e409ec60 0x7fa7e413f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:26.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7e9764640 10 monclient(hunting): picked mon.noname-b con 0x7fa7e409ec60 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:35:26.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7e9764640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:26.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7e9764640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e4140170 con 0x7fa7e409e810 2024-09-17T18:35:26.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7e9764640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:26.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7e9764640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e409d2e0 con 0x7fa7e409ec60 2024-09-17T18:35:26.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7e9764640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:26.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7e9764640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa7e41402f0 con 0x7fa7e40700f0 2024-09-17T18:35:26.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7e9764640 10 monclient(hunting): _renew_subs 2024-09-17T18:35:26.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7e9764640 10 monclient(hunting): authenticate will time out at 1730.636213s 2024-09-17T18:35:26.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7da7fc640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7e409e810 0x7fa7e413f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:26.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7e2ffd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7e40700f0 0x7fa7e41cd750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:26.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7da7fc640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7e409e810 0x7fa7e413f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54360/0 (socket says 172.21.15.121:54360) 2024-09-17T18:35:26.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7da7fc640 1 -- 172.21.15.121:0/1000368118 learned_addr learned my addr 172.21.15.121:0/1000368118 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:35:26.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7da7fc640 10 monclient(hunting): get_auth_request con 0x7fa7e409e810 auth_method 0 2024-09-17T18:35:26.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7da7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:26.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7da7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:26.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:26.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7e2ffd640 10 monclient(hunting): get_auth_request con 0x7fa7e40700f0 auth_method 0 2024-09-17T18:35:26.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:26.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7e2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:26.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:26.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7da7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:26.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7da7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:26.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7da7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:26.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7e2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:26.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7e2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:26.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7e2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:26.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7e37fe640 1 --2- 172.21.15.121:0/1000368118 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7e409ec60 0x7fa7e413f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:26.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7e37fe640 10 monclient(hunting): get_auth_request con 0x7fa7e409ec60 auth_method 0 2024-09-17T18:35:26.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:26.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:26.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:26.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:26.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:26.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:26.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7e2ffd640 10 monclient(hunting): handle_auth_done global_id 64466 payload 824 2024-09-17T18:35:26.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7e2ffd640 1 -- 172.21.15.121:0/1000368118 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7e409ec60 msgr2=0x7fa7e413f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:26.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7e2ffd640 1 --2- 172.21.15.121:0/1000368118 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7e409ec60 0x7fa7e413f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:26.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7e2ffd640 1 -- 172.21.15.121:0/1000368118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7e409e810 msgr2=0x7fa7e413f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:26.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7e2ffd640 1 --2- 172.21.15.121:0/1000368118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7e409e810 0x7fa7e413f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:26.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7e2ffd640 10 monclient: _finish_hunting 0 2024-09-17T18:35:26.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7e2ffd640 1 monclient: found mon.noname-c 2024-09-17T18:35:26.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:35:26.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7e2ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:35:26.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7e2ffd640 1 -- 172.21.15.121:0/1000368118 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7e4134780 con 0x7fa7e40700f0 2024-09-17T18:35:26.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.920+0000 7fa7da7fc640 1 --2- 172.21.15.121:0/1000368118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7e409e810 0x7fa7e413f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:35:26.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.922+0000 7fa7e2ffd640 1 -- 172.21.15.121:0/1000368118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7e40700f0 msgr2=0x7fa7e41cd750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:26.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.922+0000 7fa7e2ffd640 1 --2- 172.21.15.121:0/1000368118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7e40700f0 0x7fa7e41cd750 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa7d400c410 tx=0x7fa7d400c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:26.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.922+0000 7fa7dbfff640 10 client.?.objecter ms_handle_connect 0x7fa7e40700f0 2024-09-17T18:35:26.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.922+0000 7fa7dbfff640 10 client.?.objecter resend_mon_ops 2024-09-17T18:35:26.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.922+0000 7fa7dbfff640 1 -- 172.21.15.121:0/1000368118 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d4017070 con 0x7fa7e40700f0 2024-09-17T18:35:26.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.922+0000 7fa7dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:26.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.922+0000 7fa7dbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:35:26.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.922+0000 7fa7dbfff640 10 monclient: dump: 2024-09-17T18:35:26.938 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:26.938 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:26.938 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:26.938 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:26.938 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:26.938 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:26.938 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:26.938 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:26.939 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:26.939 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:26.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.922+0000 7fa7dbfff640 10 monclient: _finish_auth 0 2024-09-17T18:35:26.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.922+0000 7fa7dbfff640 10 monclient: _check_auth_tickets 2024-09-17T18:35:26.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.922+0000 7fa7dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:35:26.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.923+0000 7fa7e9764640 5 monclient: authenticate success, global_id 64466 2024-09-17T18:35:26.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.923+0000 7fa7dbfff640 1 -- 172.21.15.121:0/1000368118 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7d4005230 con 0x7fa7e40700f0 2024-09-17T18:35:26.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.923+0000 7fa7dbfff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:35:26.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.923+0000 7fa7dbfff640 1 -- 172.21.15.121:0/1000368118 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d40117a0 con 0x7fa7e40700f0 2024-09-17T18:35:26.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.923+0000 7fa7e9764640 10 monclient: _renew_subs 2024-09-17T18:35:26.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.923+0000 7fa7e9764640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:26.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.923+0000 7fa7e9764640 1 -- 172.21.15.121:0/1000368118 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7e413feb0 con 0x7fa7e40700f0 2024-09-17T18:35:26.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.923+0000 7fa7e1ffb640 4 set_mon_vals no callback set 2024-09-17T18:35:26.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.923+0000 7fa7dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:26.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.923+0000 7fa7dbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:35:26.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.923+0000 7fa7dbfff640 10 monclient: dump: 2024-09-17T18:35:26.940 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:26.940 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:26.940 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:26.940 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:26.940 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:26.940 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:26.940 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:26.940 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:26.940 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:26.940 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:26.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.923+0000 7fa7e9764640 10 client.64466.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:35:26.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.923+0000 7fa7e9764640 10 monclient: _renew_subs 2024-09-17T18:35:26.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.923+0000 7fa7e9764640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:26.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.923+0000 7fa7e9764640 1 -- 172.21.15.121:0/1000368118 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7e41387d0 con 0x7fa7e40700f0 2024-09-17T18:35:26.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.923+0000 7fa7e9764640 1 librados: init done 2024-09-17T18:35:26.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.923+0000 7fa7e9764640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:26.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.923+0000 7fa7e9764640 1 --2- 172.21.15.121:0/1000368118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7e4136e50 0x7fa7e4137230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:26.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.923+0000 7fa7e9764640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:26.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.923+0000 7fa7e9764640 1 -- 172.21.15.121:0/1000368118 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa7e413a750 con 0x7fa7e4136e50 2024-09-17T18:35:26.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.924+0000 7fa7da7fc640 1 --2- 172.21.15.121:0/1000368118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7e4136e50 0x7fa7e4137230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:26.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.924+0000 7fa7dbfff640 1 -- 172.21.15.121:0/1000368118 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fa7d401a070 con 0x7fa7e40700f0 2024-09-17T18:35:26.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.924+0000 7fa7da7fc640 10 monclient: get_auth_request con 0x7fa7e4136e50 auth_method 0 2024-09-17T18:35:26.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.924+0000 7fa7da7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:26.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.924+0000 7fa7da7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:26.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.924+0000 7fa7da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:26.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.924+0000 7fa7da7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:26.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.924+0000 7fa7da7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:26.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.924+0000 7fa7da7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:26.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.924+0000 7fa7da7fc640 10 monclient(hunting): handle_auth_done global_id 64469 payload 824 2024-09-17T18:35:26.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.924+0000 7fa7dbfff640 1 --2- 172.21.15.121:0/1000368118 >> v2:172.21.15.173:6812/2423786037 conn(0x7fa7c4040170 0x7fa7c4042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:26.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.924+0000 7fa7da7fc640 1 -- 172.21.15.121:0/1000368118 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa7e4134e50 con 0x7fa7e4136e50 2024-09-17T18:35:26.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.924+0000 7fa7dbfff640 1 -- 172.21.15.121:0/1000368118 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(646..646 src has 247..646) ==== 5925+0+0 (secure 0 0 0) 0x7fa7d4052be0 con 0x7fa7e40700f0 2024-09-17T18:35:26.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.924+0000 7fa7dbfff640 10 client.64466.objecter ms_dispatch 0x7fa7e4000b90 osd_map(646..646 src has 247..646) 2024-09-17T18:35:26.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.924+0000 7fa7dbfff640 3 client.64466.objecter handle_osd_map got epochs [646,646] > 0 2024-09-17T18:35:26.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.924+0000 7fa7dbfff640 3 client.64466.objecter handle_osd_map decoding full epoch 646 2024-09-17T18:35:26.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.924+0000 7fa7dbfff640 20 client.64466.objecter dump_active .. 0 homeless 2024-09-17T18:35:26.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.924+0000 7fa7e37fe640 1 --2- 172.21.15.121:0/1000368118 >> v2:172.21.15.173:6812/2423786037 conn(0x7fa7c4040170 0x7fa7c4042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:26.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.924+0000 7fa7e37fe640 10 monclient: get_auth_request con 0x7fa7c4040170 auth_method 0 2024-09-17T18:35:26.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.926+0000 7fa7dbfff640 10 client.64466.objecter ms_handle_connect 0x7fa7e4136e50 2024-09-17T18:35:26.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.926+0000 7fa7da7fc640 1 -- 172.21.15.121:0/1000368118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7e4136e50 msgr2=0x7fa7e4137230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:26.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.926+0000 7fa7dbfff640 10 client.64466.objecter resend_mon_ops 2024-09-17T18:35:26.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.926+0000 7fa7da7fc640 1 --2- 172.21.15.121:0/1000368118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7e4136e50 0x7fa7e4137230 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc035040 tx=0x7fa7cc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:26.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.926+0000 7fa7dbfff640 1 -- 172.21.15.121:0/1000368118 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7cc037040 con 0x7fa7e4136e50 2024-09-17T18:35:26.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.926+0000 7fa7dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:26.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.926+0000 7fa7dbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:35:26.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.926+0000 7fa7dbfff640 10 monclient: dump: 2024-09-17T18:35:26.943 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:26.943 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:26.943 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:26.943 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:26.943 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:26.943 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:26.943 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:26.943 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:26.943 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:26.944 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:26.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.926+0000 7fa7dbfff640 1 -- 172.21.15.121:0/1000368118 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa7cc03ca30 con 0x7fa7e4136e50 2024-09-17T18:35:26.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.926+0000 7fa7dbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:26.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.926+0000 7fa7dbfff640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:35:26.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.926+0000 7fa7dbfff640 1 -- 172.21.15.121:0/1000368118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7e4136e50 msgr2=0x7fa7e4137230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:26.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.926+0000 7fa7dbfff640 1 --2- 172.21.15.121:0/1000368118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7e4136e50 0x7fa7e4137230 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc035040 tx=0x7fa7cc0046c0 comp rx=0 tx=0).stop 2024-09-17T18:35:26.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.926+0000 7fa7dbfff640 1 -- 172.21.15.121:0/1000368118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7e4136e50 msgr2=0x7fa7e4137230 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:35:26.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.926+0000 7fa7dbfff640 1 --2- 172.21.15.121:0/1000368118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7e4136e50 0x7fa7e4137230 secure :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc035040 tx=0x7fa7cc0046c0 comp rx=0 tx=0).stop 2024-09-17T18:35:26.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.927+0000 7fa7dbfff640 10 client.64466.objecter ms_handle_connect 0x7fa7c4040170 2024-09-17T18:35:26.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.927+0000 7fa7e37fe640 1 --2- 172.21.15.121:0/1000368118 >> v2:172.21.15.173:6812/2423786037 conn(0x7fa7c4040170 0x7fa7c4042630 secure :-1 s=READY pgs=4072 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4139560 tx=0x7fa7dc0095d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:26.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.932+0000 7fa7e9764640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:35:26.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.932+0000 7fa7e9764640 1 --2- 172.21.15.121:0/1000368118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7e4137770 0x7fa7e409f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:26.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.932+0000 7fa7e9764640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:26.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.932+0000 7fa7e9764640 1 -- 172.21.15.121:0/1000368118 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa7e40a2420 con 0x7fa7e4137770 2024-09-17T18:35:26.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.933+0000 7fa7da7fc640 1 --2- 172.21.15.121:0/1000368118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7e4137770 0x7fa7e409f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:26.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.933+0000 7fa7da7fc640 10 monclient: get_auth_request con 0x7fa7e4137770 auth_method 0 2024-09-17T18:35:26.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.933+0000 7fa7da7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:26.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.933+0000 7fa7da7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:26.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.933+0000 7fa7da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:26.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.933+0000 7fa7da7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:26.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.933+0000 7fa7da7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:26.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.933+0000 7fa7da7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:26.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.933+0000 7fa7da7fc640 10 monclient(hunting): handle_auth_done global_id 64472 payload 824 2024-09-17T18:35:26.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.933+0000 7fa7da7fc640 1 -- 172.21.15.121:0/1000368118 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa7e41396d0 con 0x7fa7e4137770 2024-09-17T18:35:26.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.935+0000 7fa7da7fc640 1 -- 172.21.15.121:0/1000368118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7e4137770 msgr2=0x7fa7e409f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:26.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.935+0000 7fa7dbfff640 10 client.64466.objecter ms_handle_connect 0x7fa7e4137770 2024-09-17T18:35:26.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.935+0000 7fa7dbfff640 10 client.64466.objecter resend_mon_ops 2024-09-17T18:35:26.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.935+0000 7fa7da7fc640 1 --2- 172.21.15.121:0/1000368118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7e4137770 0x7fa7e409f0a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa7e40a23f0 tx=0x7fa7cc008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:26.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.935+0000 7fa7dbfff640 1 -- 172.21.15.121:0/1000368118 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7cc037040 con 0x7fa7e4137770 2024-09-17T18:35:26.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.935+0000 7fa7dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:26.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.935+0000 7fa7dbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:35:26.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.935+0000 7fa7dbfff640 10 monclient: dump: 2024-09-17T18:35:26.947 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:26.947 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:26.947 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:26.947 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:26.947 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:26.947 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:26.948 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:26.948 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:26.948 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:26.948 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:26.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.935+0000 7fa7dbfff640 1 -- 172.21.15.121:0/1000368118 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7fa7cc046da0 con 0x7fa7e4137770 2024-09-17T18:35:26.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.935+0000 7fa7dbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:35:26.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.935+0000 7fa7dbfff640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:35:26.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.935+0000 7fa7dbfff640 1 -- 172.21.15.121:0/1000368118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7e4137770 msgr2=0x7fa7e409f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:26.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.935+0000 7fa7dbfff640 1 --2- 172.21.15.121:0/1000368118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7e4137770 0x7fa7e409f0a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa7e40a23f0 tx=0x7fa7cc008030 comp rx=0 tx=0).stop 2024-09-17T18:35:26.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.935+0000 7fa7dbfff640 1 -- 172.21.15.121:0/1000368118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7e4137770 msgr2=0x7fa7e409f0a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:35:26.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.935+0000 7fa7dbfff640 1 --2- 172.21.15.121:0/1000368118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7e4137770 0x7fa7e409f0a0 secure :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa7e40a23f0 tx=0x7fa7cc008030 comp rx=0 tx=0).stop 2024-09-17T18:35:26.948 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:35:26.949 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:35:26.949 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:35:26.949 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:35:26.949 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 60, 2024-09-17T18:35:26.949 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 505667, 2024-09-17T18:35:26.949 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:35:26.949 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:35:26.949 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:35:26.949 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:35:26.949 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:35:26.949 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 0, 2024-09-17T18:35:26.949 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:35:26.949 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:35:26.950 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:35:26.950 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:35:26.950 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:35:26.950 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:35:26.950 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:35:26.950 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:35:26.950 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:35:26.950 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:35:26.950 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:35:26.950 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:35:26.950 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:35:26.950 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:35:26.950 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:35:26.950 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:35:26.951 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:35:26.951 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:35:26.951 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:35:26.951 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:35:26.951 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:35:26.951 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:35:26.951 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:35:26.951 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:35:26.951 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:35:26.951 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:35:26.951 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:35:26.951 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:35:26.951 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:26.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:26.951 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:35:26.952 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:35:26.952 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:35:26.952 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:35:26.952 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:35:26.952 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:35:26.952 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:35:26.952 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:35:26.952 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:35:26.952 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:35:26.952 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:35:26.952 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:35:26.952 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:35:26.952 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:35:26.952 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:35:26.952 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:35:26.953 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:35:26.953 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:35:26.953 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:35:26.953 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:35:26.953 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:35:26.953 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:35:26.953 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:35:26.953 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:35:26.953 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:35:26.953 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:35:26.953 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:35:26.953 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:35:26.953 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:35:26.953 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:35:26.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:26.954 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:35:26.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:26.954 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:35:26.954 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:35:26.954 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:35:26.954 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:35:26.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:26.954 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:35:26.954 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:35:26.954 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:26.954 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:26.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:26.954 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:35:26.954 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:35:26.954 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:26.955 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:26.955 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:26.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:26.955 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:35:26.955 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:35:26.955 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:35:26.955 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:35:26.955 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:35:26.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:26.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:26.955 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:35:26.955 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:35:26.955 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:35:26.955 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:35:26.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:26.956 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:35:26.956 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:35:26.956 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:26.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:26.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:26.956 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:35:26.956 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:35:26.956 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:26.956 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:26.956 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:26.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:26.956 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:35:26.957 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:35:26.957 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:35:26.957 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:35:26.957 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:35:26.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:26.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:26.957 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:35:26.957 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:35:26.957 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:35:26.957 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:35:26.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:26.957 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:35:26.957 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:35:26.957 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:26.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:26.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:26.958 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:35:26.958 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:35:26.958 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:26.958 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:26.958 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:26.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:26.958 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:35:26.958 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:35:26.958 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:35:26.958 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:35:26.958 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:35:26.958 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:26.958 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:26.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:26.959 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:35:26.959 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:35:26.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:26.959 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:35:26.959 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:35:26.959 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:35:26.959 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:26.959 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:35:26.959 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:35:26.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:26.959 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:35:26.959 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:35:26.959 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 6 2024-09-17T18:35:26.959 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:26.960 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:35:26.960 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:35:26.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:26.960 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:35:26.960 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:35:26.960 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 4 2024-09-17T18:35:26.960 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:26.960 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:26.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:26.960 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:35:26.960 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:35:26.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.936+0000 7fa7e9764640 10 librados: watch_flush enter 2024-09-17T18:35:26.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.936+0000 7fa7e9764640 10 librados: watch_flush exit 2024-09-17T18:35:26.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.936+0000 7fa7e9764640 20 client.64466.objecter shutdown clearing up homeless session... 2024-09-17T18:35:26.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.936+0000 7fa7e9764640 10 client.64466.objecter successfully canceled tick 2024-09-17T18:35:26.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.936+0000 7fa7e9764640 1 -- 172.21.15.121:0/1000368118 >> v2:172.21.15.173:6812/2423786037 conn(0x7fa7c4040170 msgr2=0x7fa7c4042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:26.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.936+0000 7fa7e9764640 1 --2- 172.21.15.121:0/1000368118 >> v2:172.21.15.173:6812/2423786037 conn(0x7fa7c4040170 0x7fa7c4042630 secure :-1 s=READY pgs=4072 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4139560 tx=0x7fa7dc0095d0 comp rx=0 tx=0).stop 2024-09-17T18:35:26.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.936+0000 7fa7e9764640 10 monclient: shutdown 2024-09-17T18:35:26.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.936+0000 7fa7e9764640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:35:26.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.936+0000 7fa7e9764640 1 -- 172.21.15.121:0/1000368118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7e40700f0 msgr2=0x7fa7e41cd750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:26.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.936+0000 7fa7e9764640 1 --2- 172.21.15.121:0/1000368118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7e40700f0 0x7fa7e41cd750 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa7d400c410 tx=0x7fa7d400c8d0 comp rx=0 tx=0).stop 2024-09-17T18:35:26.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.936+0000 7fa7e37fe640 1 -- 172.21.15.121:0/1000368118 reap_dead start 2024-09-17T18:35:26.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.936+0000 7fa7e9764640 1 -- 172.21.15.121:0/1000368118 shutdown_connections 2024-09-17T18:35:26.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.936+0000 7fa7e9764640 1 --2- 172.21.15.121:0/1000368118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7e40700f0 0x7fa7e41cd750 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:26.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.936+0000 7fa7e9764640 1 -- 172.21.15.121:0/1000368118 >> 172.21.15.121:0/1000368118 conn(0x7fa7e40ab830 msgr2=0x7fa7e4133940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:35:26.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.936+0000 7fa7e9764640 1 -- 172.21.15.121:0/1000368118 shutdown_connections 2024-09-17T18:35:26.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.936+0000 7fa7e9764640 1 -- 172.21.15.121:0/1000368118 wait complete. 2024-09-17T18:35:26.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:26.936+0000 7fa7e9764640 1 librados: shutdown 2024-09-17T18:35:26.963 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:35:26.963 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T18:35:26.963 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T18:35:27.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.059+0000 7fb7936d6640 10 monclient: get_monmap_and_config 2024-09-17T18:35:27.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.059+0000 7fb7936d6640 10 monclient: build_initial_monmap 2024-09-17T18:35:27.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.059+0000 7fb7936d6640 10 monclient: monmap: 2024-09-17T18:35:27.061 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:35:27.061 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:27.061 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:35:27.059999+0000 2024-09-17T18:35:27.061 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:35:27.059999+0000 2024-09-17T18:35:27.061 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:35:27.061 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:27.061 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:35:27.061 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:35:27.062 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:35:27.062 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:35:27.062 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:35:27.062 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:27.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.059+0000 7fb7936d6640 1 Processor -- start 2024-09-17T18:35:27.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.060+0000 7fb7936d6640 1 -- start start 2024-09-17T18:35:27.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.060+0000 7fb7936d6640 10 monclient: init 2024-09-17T18:35:27.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.060+0000 7fb7936d6640 10 monclient: _reopen_session rank -1 2024-09-17T18:35:27.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.060+0000 7fb7936d6640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-17T18:35:27.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.060+0000 7fb7936d6640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb78c135440 0x7fb78c138880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:27.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.060+0000 7fb7936d6640 10 monclient(hunting): picked mon.noname-b con 0x7fb78c135440 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:35:27.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.060+0000 7fb7936d6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb78c135820 addr v1:172.21.15.173:6789/0 2024-09-17T18:35:27.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.060+0000 7fb7936d6640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb78c135c20 0x7fb78c0a3950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:27.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.060+0000 7fb7936d6640 10 monclient(hunting): picked mon.noname-a con 0x7fb78c135c20 addr v2:172.21.15.121:3300/0 2024-09-17T18:35:27.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.060+0000 7fb7936d6640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb78c0a13a0 con 0x7fb78c135820 2024-09-17T18:35:27.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.060+0000 7fb7936d6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:27.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.060+0000 7fb7936d6640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fb78c077300 con 0x7fb78c135c20 2024-09-17T18:35:27.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.060+0000 7fb7936d6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:27.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.060+0000 7fb7936d6640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb78c06fc80 con 0x7fb78c135440 2024-09-17T18:35:27.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.060+0000 7fb7936d6640 10 monclient(hunting): _renew_subs 2024-09-17T18:35:27.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.060+0000 7fb7936d6640 10 monclient(hunting): authenticate will time out at 1730.776657s 2024-09-17T18:35:27.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.060+0000 7fb791472640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb78c135440 0x7fb78c138880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:27.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.060+0000 7fb791c73640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb78c135c20 0x7fb78c0a3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:27.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.060+0000 7fb791c73640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb78c135c20 0x7fb78c0a3950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54368/0 (socket says 172.21.15.121:54368) 2024-09-17T18:35:27.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.060+0000 7fb791472640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb78c135440 0x7fb78c138880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:53464/0 (socket says 172.21.15.121:53464) 2024-09-17T18:35:27.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.060+0000 7fb791c73640 1 -- 172.21.15.121:0/493558069 learned_addr learned my addr 172.21.15.121:0/493558069 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:35:27.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.060+0000 7fb791c73640 10 monclient(hunting): get_auth_request con 0x7fb78c135c20 auth_method 0 2024-09-17T18:35:27.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.060+0000 7fb791c73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:27.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.060+0000 7fb791c73640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:27.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.060+0000 7fb791c73640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:27.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.060+0000 7fb791472640 10 monclient(hunting): get_auth_request con 0x7fb78c135440 auth_method 0 2024-09-17T18:35:27.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.060+0000 7fb791472640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:27.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.060+0000 7fb791472640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:27.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.060+0000 7fb791472640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:27.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.060+0000 7fb791472640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:27.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.060+0000 7fb791472640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:27.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.060+0000 7fb791472640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:27.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.060+0000 7fb791c73640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:27.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.060+0000 7fb791c73640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:27.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.060+0000 7fb791c73640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:27.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.061+0000 7fb791472640 10 monclient(hunting): handle_auth_done global_id 64475 payload 274 2024-09-17T18:35:27.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.061+0000 7fb791472640 1 -- 172.21.15.121:0/493558069 >> v2:172.21.15.121:3300/0 conn(0x7fb78c135c20 msgr2=0x7fb78c0a3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:35:27.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.061+0000 7fb791472640 1 --2- 172.21.15.121:0/493558069 >> v2:172.21.15.121:3300/0 conn(0x7fb78c135c20 0x7fb78c0a3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:27.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.061+0000 7fb791472640 1 -- 172.21.15.121:0/493558069 >> v1:172.21.15.173:6789/0 conn(0x7fb78c135820 legacy=0x7fb78c0a3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:35:27.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.061+0000 7fb791472640 10 monclient: _finish_hunting 0 2024-09-17T18:35:27.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.061+0000 7fb791472640 1 monclient: found mon.noname-b 2024-09-17T18:35:27.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.061+0000 7fb791472640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:35:27.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.061+0000 7fb791472640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:35:27.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.061+0000 7fb791472640 1 -- 172.21.15.121:0/493558069 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb78c06ff60 con 0x7fb78c135440 2024-09-17T18:35:27.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.061+0000 7fb791c73640 1 --2- 172.21.15.121:0/493558069 >> v2:172.21.15.121:3300/0 conn(0x7fb78c135c20 0x7fb78c0a3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:35:27.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.063+0000 7fb791472640 1 -- 172.21.15.121:0/493558069 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb78c135440 msgr2=0x7fb78c138880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:27.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.063+0000 7fb791472640 1 --2- 172.21.15.121:0/493558069 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb78c135440 0x7fb78c138880 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb784009d90 tx=0x7fb78402a490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a9c125e8e850e5b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:27.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.063+0000 7fb783fff640 1 -- 172.21.15.121:0/493558069 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb784007950 con 0x7fb78c135440 2024-09-17T18:35:27.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.063+0000 7fb783fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:27.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.064+0000 7fb783fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:35:27.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.064+0000 7fb783fff640 10 monclient: dump: 2024-09-17T18:35:27.074 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:27.074 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:27.074 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:27.074 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:27.074 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:27.074 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:27.074 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:27.074 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:27.074 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:27.074 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:27.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.064+0000 7fb783fff640 10 monclient: _finish_auth 0 2024-09-17T18:35:27.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.064+0000 7fb783fff640 10 monclient: _check_auth_tickets 2024-09-17T18:35:27.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.064+0000 7fb783fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:35:27.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.064+0000 7fb783fff640 1 -- 172.21.15.121:0/493558069 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb784007af0 con 0x7fb78c135440 2024-09-17T18:35:27.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.064+0000 7fb7936d6640 5 monclient: authenticate success, global_id 64475 2024-09-17T18:35:27.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.064+0000 7fb7936d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:35:27.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.064+0000 7fb783fff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:35:27.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.064+0000 7fb7936d6640 10 monclient: get_monmap_and_config success 2024-09-17T18:35:27.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.064+0000 7fb783fff640 1 -- 172.21.15.121:0/493558069 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb784031400 con 0x7fb78c135440 2024-09-17T18:35:27.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.064+0000 7fb7936d6640 4 set_mon_vals no callback set 2024-09-17T18:35:27.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.064+0000 7fb783fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:27.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.064+0000 7fb783fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:35:27.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.064+0000 7fb783fff640 10 monclient: dump: 2024-09-17T18:35:27.075 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:27.076 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:27.076 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:27.076 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:27.076 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:27.076 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:27.076 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:27.076 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:27.076 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:27.076 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:27.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.064+0000 7fb7936d6640 10 monclient: shutdown 2024-09-17T18:35:27.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.064+0000 7fb7936d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:35:27.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.064+0000 7fb7936d6640 1 -- 172.21.15.121:0/493558069 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb78c135440 msgr2=0x7fb78c138880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:27.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.064+0000 7fb7936d6640 1 --2- 172.21.15.121:0/493558069 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb78c135440 0x7fb78c138880 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb784009d90 tx=0x7fb78402a490 comp rx=0 tx=0).stop 2024-09-17T18:35:27.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.064+0000 7fb7936d6640 1 -- 172.21.15.121:0/493558069 shutdown_connections 2024-09-17T18:35:27.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.064+0000 7fb7936d6640 1 --2- 172.21.15.121:0/493558069 >> v2:172.21.15.121:3300/0 conn(0x7fb78c135c20 0x7fb78c0a3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:27.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.064+0000 7fb7936d6640 1 --2- 172.21.15.121:0/493558069 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb78c135440 0x7fb78c138880 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:27.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.064+0000 7fb7936d6640 1 -- 172.21.15.121:0/493558069 >> 172.21.15.121:0/493558069 conn(0x7fb78c0a49c0 msgr2=0x7fb78c0768d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:35:27.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.064+0000 7fb7936d6640 1 -- 172.21.15.121:0/493558069 shutdown_connections 2024-09-17T18:35:27.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.064+0000 7fb7936d6640 1 -- 172.21.15.121:0/493558069 wait complete. 2024-09-17T18:35:27.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.065+0000 7fb7936d6640 10 monclient: build_initial_monmap 2024-09-17T18:35:27.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.065+0000 7fb7936d6640 10 monclient: monmap: 2024-09-17T18:35:27.077 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:35:27.077 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:35:27.077 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:35:27.077 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:35:27.077 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:35:27.077 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:27.077 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:35:27.077 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:35:27.077 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:35:27.078 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:27.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.065+0000 7fb7936d6640 1 librados: starting msgr at 2024-09-17T18:35:27.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.065+0000 7fb7936d6640 1 librados: starting objecter 2024-09-17T18:35:27.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.065+0000 7fb7936d6640 1 Processor -- start 2024-09-17T18:35:27.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.065+0000 7fb7936d6640 1 -- start start 2024-09-17T18:35:27.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.065+0000 7fb7936d6640 1 librados: setting wanted keys 2024-09-17T18:35:27.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.065+0000 7fb7936d6640 1 librados: calling monclient init 2024-09-17T18:35:27.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.065+0000 7fb7936d6640 10 monclient: init 2024-09-17T18:35:27.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.065+0000 7fb7936d6640 10 monclient: _reopen_session rank -1 2024-09-17T18:35:27.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.065+0000 7fb7936d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T18:35:27.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.065+0000 7fb7936d6640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb78c135440 0x7fb78c134160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:27.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.065+0000 7fb7936d6640 10 monclient(hunting): picked mon.noname-a con 0x7fb78c135440 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:35:27.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.065+0000 7fb7936d6640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb78c135820 0x7fb78c11de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:27.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.065+0000 7fb7936d6640 10 monclient(hunting): picked mon.noname-c con 0x7fb78c135820 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:35:27.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.065+0000 7fb7936d6640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb78c135c20 0x7fb78c11e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:27.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.066+0000 7fb7936d6640 10 monclient(hunting): picked mon.noname-b con 0x7fb78c135c20 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:35:27.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.066+0000 7fb7936d6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:27.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.066+0000 7fb7936d6640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb78c06d820 con 0x7fb78c135440 2024-09-17T18:35:27.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.066+0000 7fb7936d6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:27.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.066+0000 7fb7936d6640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb78c06d6a0 con 0x7fb78c135c20 2024-09-17T18:35:27.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.066+0000 7fb7936d6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:27.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.066+0000 7fb7936d6640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb78c0a3e90 con 0x7fb78c135820 2024-09-17T18:35:27.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.066+0000 7fb7936d6640 10 monclient(hunting): _renew_subs 2024-09-17T18:35:27.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.066+0000 7fb7936d6640 10 monclient(hunting): authenticate will time out at 1730.782027s 2024-09-17T18:35:27.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.066+0000 7fb791472640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb78c135440 0x7fb78c134160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:27.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.066+0000 7fb791472640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb78c135440 0x7fb78c134160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54372/0 (socket says 172.21.15.121:54372) 2024-09-17T18:35:27.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.066+0000 7fb791472640 1 -- 172.21.15.121:0/1561245050 learned_addr learned my addr 172.21.15.121:0/1561245050 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:35:27.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.066+0000 7fb790c71640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb78c135820 0x7fb78c11de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:27.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.066+0000 7fb791472640 10 monclient(hunting): get_auth_request con 0x7fb78c135440 auth_method 0 2024-09-17T18:35:27.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.066+0000 7fb791472640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:27.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.066+0000 7fb791472640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:27.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.066+0000 7fb791472640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:27.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.066+0000 7fb790c71640 10 monclient(hunting): get_auth_request con 0x7fb78c135820 auth_method 0 2024-09-17T18:35:27.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.066+0000 7fb790c71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:27.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.066+0000 7fb790c71640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:27.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.066+0000 7fb790c71640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:27.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.066+0000 7fb790c71640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:27.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.066+0000 7fb790c71640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:27.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.066+0000 7fb790c71640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:27.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.066+0000 7fb791472640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:27.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.066+0000 7fb791472640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:27.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.066+0000 7fb791472640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:27.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.066+0000 7fb790c71640 10 monclient(hunting): handle_auth_done global_id 64478 payload 824 2024-09-17T18:35:27.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.066+0000 7fb790c71640 1 -- 172.21.15.121:0/1561245050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb78c135c20 msgr2=0x7fb78c11e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:27.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.066+0000 7fb790c71640 1 --2- 172.21.15.121:0/1561245050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb78c135c20 0x7fb78c11e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:27.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.066+0000 7fb790c71640 1 -- 172.21.15.121:0/1561245050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb78c135440 msgr2=0x7fb78c134160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:27.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.066+0000 7fb790c71640 1 --2- 172.21.15.121:0/1561245050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb78c135440 0x7fb78c134160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:27.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.066+0000 7fb790c71640 10 monclient: _finish_hunting 0 2024-09-17T18:35:27.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.066+0000 7fb790c71640 1 monclient: found mon.noname-c 2024-09-17T18:35:27.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.066+0000 7fb790c71640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:35:27.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.066+0000 7fb790c71640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:35:27.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.066+0000 7fb790c71640 1 -- 172.21.15.121:0/1561245050 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb78c074c70 con 0x7fb78c135820 2024-09-17T18:35:27.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.066+0000 7fb791472640 1 --2- 172.21.15.121:0/1561245050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb78c135440 0x7fb78c134160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:35:27.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.068+0000 7fb790c71640 1 -- 172.21.15.121:0/1561245050 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb78c135820 msgr2=0x7fb78c11de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:27.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.068+0000 7fb790c71640 1 --2- 172.21.15.121:0/1561245050 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb78c135820 0x7fb78c11de80 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb77c00c3e0 tx=0x7fb77c00c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:27.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.068+0000 7fb781ffb640 10 client.?.objecter ms_handle_connect 0x7fb78c135820 2024-09-17T18:35:27.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.068+0000 7fb781ffb640 10 client.?.objecter resend_mon_ops 2024-09-17T18:35:27.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.068+0000 7fb781ffb640 1 -- 172.21.15.121:0/1561245050 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77c0119a0 con 0x7fb78c135820 2024-09-17T18:35:27.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.068+0000 7fb781ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:27.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.068+0000 7fb781ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:35:27.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.068+0000 7fb781ffb640 10 monclient: dump: 2024-09-17T18:35:27.082 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:27.082 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:27.082 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:27.082 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:27.082 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:27.082 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:27.082 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:27.082 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:27.082 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:27.082 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:27.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.068+0000 7fb781ffb640 10 monclient: _finish_auth 0 2024-09-17T18:35:27.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.068+0000 7fb781ffb640 10 monclient: _check_auth_tickets 2024-09-17T18:35:27.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.068+0000 7fb781ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:35:27.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.068+0000 7fb781ffb640 1 -- 172.21.15.121:0/1561245050 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb77c017070 con 0x7fb78c135820 2024-09-17T18:35:27.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.068+0000 7fb781ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:35:27.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.068+0000 7fb781ffb640 1 -- 172.21.15.121:0/1561245050 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77c011d80 con 0x7fb78c135820 2024-09-17T18:35:27.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.068+0000 7fb781ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:27.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.068+0000 7fb781ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:35:27.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.068+0000 7fb781ffb640 10 monclient: dump: 2024-09-17T18:35:27.083 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:27.083 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:27.083 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:27.083 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:27.083 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:27.084 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:27.084 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:27.084 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:27.084 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:27.084 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:27.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.068+0000 7fb7936d6640 5 monclient: authenticate success, global_id 64478 2024-09-17T18:35:27.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.068+0000 7fb7936d6640 10 monclient: _renew_subs 2024-09-17T18:35:27.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.068+0000 7fb7936d6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:27.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.068+0000 7fb7936d6640 1 -- 172.21.15.121:0/1561245050 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb78c121110 con 0x7fb78c135820 2024-09-17T18:35:27.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.068+0000 7fb7837fe640 4 set_mon_vals no callback set 2024-09-17T18:35:27.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.069+0000 7fb7936d6640 10 client.64478.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:35:27.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.069+0000 7fb7936d6640 10 monclient: _renew_subs 2024-09-17T18:35:27.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.069+0000 7fb7936d6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:27.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.069+0000 7fb7936d6640 1 -- 172.21.15.121:0/1561245050 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb78c1201b0 con 0x7fb78c135820 2024-09-17T18:35:27.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.069+0000 7fb7936d6640 1 librados: init done 2024-09-17T18:35:27.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.069+0000 7fb781ffb640 1 -- 172.21.15.121:0/1561245050 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fb77c01c460 con 0x7fb78c135820 2024-09-17T18:35:27.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.069+0000 7fb7936d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:27.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.069+0000 7fb7936d6640 1 --2- 172.21.15.121:0/1561245050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb78c1346a0 0x7fb78c11e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:27.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.069+0000 7fb7936d6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:27.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.069+0000 7fb7936d6640 1 -- 172.21.15.121:0/1561245050 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb78c137dc0 con 0x7fb78c1346a0 2024-09-17T18:35:27.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.069+0000 7fb781ffb640 1 --2- 172.21.15.121:0/1561245050 >> v2:172.21.15.173:6812/2423786037 conn(0x7fb770040170 0x7fb770042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:27.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.069+0000 7fb781ffb640 1 -- 172.21.15.121:0/1561245050 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(646..646 src has 247..646) ==== 5925+0+0 (secure 0 0 0) 0x7fb77c053740 con 0x7fb78c135820 2024-09-17T18:35:27.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.069+0000 7fb781ffb640 10 client.64478.objecter ms_dispatch 0x7fb78c000b90 osd_map(646..646 src has 247..646) 2024-09-17T18:35:27.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.069+0000 7fb781ffb640 3 client.64478.objecter handle_osd_map got epochs [646,646] > 0 2024-09-17T18:35:27.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.069+0000 7fb781ffb640 3 client.64478.objecter handle_osd_map decoding full epoch 646 2024-09-17T18:35:27.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.070+0000 7fb781ffb640 20 client.64478.objecter dump_active .. 0 homeless 2024-09-17T18:35:27.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.070+0000 7fb791472640 1 --2- 172.21.15.121:0/1561245050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb78c1346a0 0x7fb78c11e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:27.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.070+0000 7fb791472640 10 monclient: get_auth_request con 0x7fb78c1346a0 auth_method 0 2024-09-17T18:35:27.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.070+0000 7fb791472640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:27.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.070+0000 7fb791472640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:27.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.070+0000 7fb791472640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:27.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.070+0000 7fb791c73640 1 --2- 172.21.15.121:0/1561245050 >> v2:172.21.15.173:6812/2423786037 conn(0x7fb770040170 0x7fb770042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:27.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.070+0000 7fb791c73640 10 monclient: get_auth_request con 0x7fb770040170 auth_method 0 2024-09-17T18:35:27.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.070+0000 7fb791472640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:27.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.070+0000 7fb791472640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:27.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.070+0000 7fb791472640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:27.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.070+0000 7fb791472640 10 monclient(hunting): handle_auth_done global_id 84253 payload 824 2024-09-17T18:35:27.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.070+0000 7fb791472640 1 -- 172.21.15.121:0/1561245050 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb78c0a6cc0 con 0x7fb78c1346a0 2024-09-17T18:35:27.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.072+0000 7fb791472640 1 -- 172.21.15.121:0/1561245050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb78c1346a0 msgr2=0x7fb78c11e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:27.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.072+0000 7fb791472640 1 --2- 172.21.15.121:0/1561245050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb78c1346a0 0x7fb78c11e900 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb78402a4d0 tx=0x7fb784004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:27.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.072+0000 7fb791c73640 1 --2- 172.21.15.121:0/1561245050 >> v2:172.21.15.173:6812/2423786037 conn(0x7fb770040170 0x7fb770042630 secure :-1 s=READY pgs=4073 cs=0 l=1 rev1=1 crypto rx=0x7fb788009630 tx=0x7fb788006e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:27.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.072+0000 7fb781ffb640 10 client.64478.objecter ms_handle_connect 0x7fb78c1346a0 2024-09-17T18:35:27.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.072+0000 7fb781ffb640 10 client.64478.objecter resend_mon_ops 2024-09-17T18:35:27.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.072+0000 7fb781ffb640 10 client.64478.objecter ms_handle_connect 0x7fb770040170 2024-09-17T18:35:27.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.073+0000 7fb781ffb640 1 -- 172.21.15.121:0/1561245050 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb784007e30 con 0x7fb78c1346a0 2024-09-17T18:35:27.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.073+0000 7fb781ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:27.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.073+0000 7fb781ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:35:27.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.073+0000 7fb781ffb640 10 monclient: dump: 2024-09-17T18:35:27.089 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:27.089 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:27.089 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:27.089 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:27.089 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:27.090 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:27.090 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:27.090 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:27.090 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:27.090 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:27.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.073+0000 7fb781ffb640 1 -- 172.21.15.121:0/1561245050 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb784007e30 con 0x7fb78c1346a0 2024-09-17T18:35:27.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.073+0000 7fb781ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:27.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.073+0000 7fb781ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:35:27.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.073+0000 7fb781ffb640 1 -- 172.21.15.121:0/1561245050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb78c1346a0 msgr2=0x7fb78c11e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:27.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.073+0000 7fb781ffb640 1 --2- 172.21.15.121:0/1561245050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb78c1346a0 0x7fb78c11e900 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb78402a4d0 tx=0x7fb784004a40 comp rx=0 tx=0).stop 2024-09-17T18:35:27.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.073+0000 7fb781ffb640 1 -- 172.21.15.121:0/1561245050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb78c1346a0 msgr2=0x7fb78c11e900 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:35:27.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.073+0000 7fb781ffb640 1 --2- 172.21.15.121:0/1561245050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb78c1346a0 0x7fb78c11e900 secure :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb78402a4d0 tx=0x7fb784004a40 comp rx=0 tx=0).stop 2024-09-17T18:35:27.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.079+0000 7fb7936d6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:35:27.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.079+0000 7fb7936d6640 1 --2- 172.21.15.121:0/1561245050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb78c11ee40 0x7fb78c1be1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:27.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.079+0000 7fb7936d6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:27.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.079+0000 7fb7936d6640 1 -- 172.21.15.121:0/1561245050 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb78c11f250 con 0x7fb78c11ee40 2024-09-17T18:35:27.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.080+0000 7fb791472640 1 --2- 172.21.15.121:0/1561245050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb78c11ee40 0x7fb78c1be1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:27.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.080+0000 7fb791472640 10 monclient: get_auth_request con 0x7fb78c11ee40 auth_method 0 2024-09-17T18:35:27.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.080+0000 7fb791472640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:27.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.080+0000 7fb791472640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:27.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.080+0000 7fb791472640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:27.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.080+0000 7fb791472640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:27.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.080+0000 7fb791472640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:27.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.080+0000 7fb791472640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:27.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.080+0000 7fb791472640 10 monclient(hunting): handle_auth_done global_id 84256 payload 824 2024-09-17T18:35:27.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.080+0000 7fb791472640 1 -- 172.21.15.121:0/1561245050 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb78c138880 con 0x7fb78c11ee40 2024-09-17T18:35:27.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.082+0000 7fb791472640 1 -- 172.21.15.121:0/1561245050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb78c11ee40 msgr2=0x7fb78c1be1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:27.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.082+0000 7fb791472640 1 --2- 172.21.15.121:0/1561245050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb78c11ee40 0x7fb78c1be1a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb784043000 tx=0x7fb784030000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:27.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.082+0000 7fb781ffb640 10 client.64478.objecter ms_handle_connect 0x7fb78c11ee40 2024-09-17T18:35:27.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.082+0000 7fb781ffb640 10 client.64478.objecter resend_mon_ops 2024-09-17T18:35:27.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.082+0000 7fb781ffb640 1 -- 172.21.15.121:0/1561245050 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb78404a040 con 0x7fb78c11ee40 2024-09-17T18:35:27.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.082+0000 7fb781ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:27.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.082+0000 7fb781ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:35:27.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.082+0000 7fb781ffb640 10 monclient: dump: 2024-09-17T18:35:27.092 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:27.092 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:27.092 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:27.092 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:27.092 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:27.092 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:27.093 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:27.093 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:27.093 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:27.093 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:27.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.083+0000 7fb781ffb640 1 -- 172.21.15.121:0/1561245050 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7fb784055470 con 0x7fb78c11ee40 2024-09-17T18:35:27.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.083+0000 7fb781ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:35:27.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.083+0000 7fb781ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:35:27.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.083+0000 7fb781ffb640 1 -- 172.21.15.121:0/1561245050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb78c11ee40 msgr2=0x7fb78c1be1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:27.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.083+0000 7fb781ffb640 1 --2- 172.21.15.121:0/1561245050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb78c11ee40 0x7fb78c1be1a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb784043000 tx=0x7fb784030000 comp rx=0 tx=0).stop 2024-09-17T18:35:27.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.083+0000 7fb781ffb640 1 -- 172.21.15.121:0/1561245050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb78c11ee40 msgr2=0x7fb78c1be1a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:35:27.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.083+0000 7fb781ffb640 1 --2- 172.21.15.121:0/1561245050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb78c11ee40 0x7fb78c1be1a0 secure :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb784043000 tx=0x7fb784030000 comp rx=0 tx=0).stop 2024-09-17T18:35:27.094 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:35:27.094 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:35:27.094 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:35:27.094 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:35:27.094 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 60, 2024-09-17T18:35:27.094 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 252181, 2024-09-17T18:35:27.094 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:35:27.094 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:35:27.094 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:35:27.094 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:35:27.094 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:35:27.094 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 1, 2024-09-17T18:35:27.094 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:35:27.095 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:35:27.095 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:35:27.095 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:35:27.095 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:35:27.095 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:35:27.095 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:35:27.095 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:35:27.095 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:35:27.095 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:35:27.095 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:35:27.095 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:35:27.095 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:35:27.095 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:35:27.095 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:35:27.095 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:35:27.096 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:35:27.096 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:35:27.096 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:35:27.096 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:35:27.096 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:35:27.096 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:35:27.096 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:35:27.096 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:35:27.096 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:35:27.096 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:35:27.096 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:35:27.096 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:35:27.096 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:27.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:27.096 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:35:27.097 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:35:27.097 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:35:27.097 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:35:27.097 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:35:27.097 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:35:27.097 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:35:27.097 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:35:27.097 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:35:27.097 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:35:27.097 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:35:27.097 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:35:27.097 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:35:27.097 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:35:27.097 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:35:27.097 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:35:27.098 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:35:27.098 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:35:27.098 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:35:27.098 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:35:27.098 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:35:27.098 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:35:27.098 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:35:27.098 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:35:27.098 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:35:27.098 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:35:27.098 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:35:27.098 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:35:27.098 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:35:27.098 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:35:27.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:27.099 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:35:27.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:27.099 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:35:27.099 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:35:27.099 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:35:27.099 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:35:27.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:27.099 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:35:27.099 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:35:27.099 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:27.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:27.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:27.099 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:35:27.099 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:35:27.100 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:27.100 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:27.100 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:27.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:27.100 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:35:27.100 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:35:27.100 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:35:27.100 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:35:27.100 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:35:27.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:27.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:27.100 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:35:27.100 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:35:27.100 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:35:27.101 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:35:27.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:27.101 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:35:27.101 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:35:27.101 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:27.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:27.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:27.101 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:35:27.101 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:35:27.101 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:27.101 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:27.101 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:27.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:27.101 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:35:27.101 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:35:27.102 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:35:27.102 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:35:27.102 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:35:27.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:27.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:27.102 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:35:27.102 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:35:27.102 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:35:27.102 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:35:27.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:27.102 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:35:27.102 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:35:27.102 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:27.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:27.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:27.102 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:35:27.103 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:35:27.103 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:27.103 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:27.103 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:27.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:27.103 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:35:27.103 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:35:27.103 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:35:27.103 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:35:27.103 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:35:27.103 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:27.103 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:27.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:27.103 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:35:27.104 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:35:27.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:27.104 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:35:27.104 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:35:27.104 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:35:27.104 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:27.104 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:35:27.104 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:35:27.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:27.104 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:35:27.104 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:35:27.104 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 4 2024-09-17T18:35:27.104 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:27.104 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:35:27.104 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": [ 2024-09-17T18:35:27.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:27.105 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:35:27.105 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:35:27.105 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:35:27.105 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:27.105 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:27.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:27.105 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:35:27.105 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:35:27.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.083+0000 7fb7936d6640 10 librados: watch_flush enter 2024-09-17T18:35:27.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.083+0000 7fb7936d6640 10 librados: watch_flush exit 2024-09-17T18:35:27.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.083+0000 7fb7936d6640 20 client.64478.objecter shutdown clearing up homeless session... 2024-09-17T18:35:27.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.083+0000 7fb7936d6640 10 client.64478.objecter successfully canceled tick 2024-09-17T18:35:27.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.083+0000 7fb7936d6640 1 -- 172.21.15.121:0/1561245050 >> v2:172.21.15.173:6812/2423786037 conn(0x7fb770040170 msgr2=0x7fb770042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:27.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.083+0000 7fb7936d6640 1 --2- 172.21.15.121:0/1561245050 >> v2:172.21.15.173:6812/2423786037 conn(0x7fb770040170 0x7fb770042630 secure :-1 s=READY pgs=4073 cs=0 l=1 rev1=1 crypto rx=0x7fb788009630 tx=0x7fb788006e90 comp rx=0 tx=0).stop 2024-09-17T18:35:27.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.083+0000 7fb7936d6640 10 monclient: shutdown 2024-09-17T18:35:27.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.083+0000 7fb7936d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:35:27.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.083+0000 7fb7936d6640 1 -- 172.21.15.121:0/1561245050 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb78c135820 msgr2=0x7fb78c11de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:27.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.083+0000 7fb7936d6640 1 --2- 172.21.15.121:0/1561245050 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb78c135820 0x7fb78c11de80 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb77c00c3e0 tx=0x7fb77c00c8a0 comp rx=0 tx=0).stop 2024-09-17T18:35:27.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.083+0000 7fb791c73640 1 -- 172.21.15.121:0/1561245050 reap_dead start 2024-09-17T18:35:27.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.083+0000 7fb791c73640 1 -- 172.21.15.121:0/1561245050 reap_dead start 2024-09-17T18:35:27.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.083+0000 7fb7936d6640 1 -- 172.21.15.121:0/1561245050 shutdown_connections 2024-09-17T18:35:27.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.083+0000 7fb7936d6640 1 -- 172.21.15.121:0/1561245050 >> 172.21.15.121:0/1561245050 conn(0x7fb78c0a49c0 msgr2=0x7fb78c0a5730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:35:27.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.083+0000 7fb7936d6640 1 -- 172.21.15.121:0/1561245050 shutdown_connections 2024-09-17T18:35:27.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.083+0000 7fb7936d6640 1 -- 172.21.15.121:0/1561245050 wait complete. 2024-09-17T18:35:27.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.083+0000 7fb7936d6640 1 librados: shutdown 2024-09-17T18:35:27.108 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:35:27.108 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-17T18:35:27.108 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-17T18:35:27.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.205+0000 7efdf8231640 10 monclient: get_monmap_and_config 2024-09-17T18:35:27.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.205+0000 7efdf8231640 10 monclient: build_initial_monmap 2024-09-17T18:35:27.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.205+0000 7efdf8231640 10 monclient: monmap: 2024-09-17T18:35:27.207 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:35:27.207 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:27.207 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:35:27.205618+0000 2024-09-17T18:35:27.207 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:35:27.205618+0000 2024-09-17T18:35:27.207 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:35:27.207 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:27.207 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:35:27.207 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:35:27.207 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:35:27.207 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:35:27.207 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:35:27.207 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:27.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.205+0000 7efdf8231640 1 Processor -- start 2024-09-17T18:35:27.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.205+0000 7efdf8231640 1 -- start start 2024-09-17T18:35:27.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.205+0000 7efdf8231640 10 monclient: init 2024-09-17T18:35:27.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.206+0000 7efdf8231640 10 monclient: _reopen_session rank -1 2024-09-17T18:35:27.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.206+0000 7efdf8231640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-17T18:35:27.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.206+0000 7efdf8231640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efdf01386b0 addr v1:172.21.15.121:6789/0 2024-09-17T18:35:27.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.206+0000 7efdf8231640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7efdf006ec00 0x7efdf0138a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:27.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.206+0000 7efdf8231640 10 monclient(hunting): picked mon.noname-c con 0x7efdf006ec00 addr v2:172.21.15.173:3300/0 2024-09-17T18:35:27.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.206+0000 7efdf8231640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdf0138fd0 0x7efdf013fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:27.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.206+0000 7efdf8231640 10 monclient(hunting): picked mon.noname-b con 0x7efdf0138fd0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:35:27.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.206+0000 7efdf8231640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efdf00a5570 con 0x7efdf01386b0 2024-09-17T18:35:27.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.206+0000 7efdf8231640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:27.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.206+0000 7efdf8231640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7efdf006f830 con 0x7efdf006ec00 2024-09-17T18:35:27.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.206+0000 7efdf8231640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:27.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.206+0000 7efdf8231640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7efdf006fb10 con 0x7efdf0138fd0 2024-09-17T18:35:27.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.206+0000 7efdf8231640 10 monclient(hunting): _renew_subs 2024-09-17T18:35:27.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.206+0000 7efdf8231640 10 monclient(hunting): authenticate will time out at 1730.922321s 2024-09-17T18:35:27.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.206+0000 7efdf5fcd640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7efdf01386b0 0x7efdf00707d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:43046/0 (socket says 172.21.15.121:43046) 2024-09-17T18:35:27.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.206+0000 7efdf67ce640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdf0138fd0 0x7efdf013fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:27.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.206+0000 7efdf5fcd640 1 -- 172.21.15.121:0/3035989877 learned_addr learned my addr 172.21.15.121:0/3035989877 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:35:27.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.206+0000 7efdf67ce640 10 monclient(hunting): get_auth_request con 0x7efdf0138fd0 auth_method 0 2024-09-17T18:35:27.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.206+0000 7efdf67ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:27.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.206+0000 7efdf67ce640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:27.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.206+0000 7efdf67ce640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:27.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.206+0000 7efdf67ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:27.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.206+0000 7efdf67ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:27.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.206+0000 7efdf67ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:27.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.206+0000 7efdf57cc640 1 --2- 172.21.15.121:0/3035989877 >> v2:172.21.15.173:3300/0 conn(0x7efdf006ec00 0x7efdf0138a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:27.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.206+0000 7efdf57cc640 10 monclient(hunting): get_auth_request con 0x7efdf006ec00 auth_method 0 2024-09-17T18:35:27.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.206+0000 7efdf57cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:27.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.206+0000 7efdf57cc640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:27.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.206+0000 7efdf57cc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:27.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.206+0000 7efdf67ce640 10 monclient(hunting): handle_auth_done global_id 64481 payload 274 2024-09-17T18:35:27.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.206+0000 7efdf67ce640 1 -- 172.21.15.121:0/3035989877 >> v2:172.21.15.173:3300/0 conn(0x7efdf006ec00 msgr2=0x7efdf0138a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:35:27.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.206+0000 7efdf67ce640 1 --2- 172.21.15.121:0/3035989877 >> v2:172.21.15.173:3300/0 conn(0x7efdf006ec00 0x7efdf0138a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:27.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.206+0000 7efdf67ce640 1 -- 172.21.15.121:0/3035989877 >> v1:172.21.15.121:6789/0 conn(0x7efdf01386b0 legacy=0x7efdf00707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:35:27.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.206+0000 7efdf67ce640 10 monclient: _finish_hunting 0 2024-09-17T18:35:27.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.206+0000 7efdf67ce640 1 monclient: found mon.noname-b 2024-09-17T18:35:27.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.206+0000 7efdf67ce640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:35:27.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.206+0000 7efdf67ce640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:35:27.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.206+0000 7efdf67ce640 1 -- 172.21.15.121:0/3035989877 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdf0139460 con 0x7efdf0138fd0 2024-09-17T18:35:27.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.209+0000 7efdf67ce640 1 -- 172.21.15.121:0/3035989877 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdf0138fd0 msgr2=0x7efdf013fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:27.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.209+0000 7efdf67ce640 1 --2- 172.21.15.121:0/3035989877 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdf0138fd0 0x7efdf013fbf0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7efde80097d0 tx=0x7efde802a080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2aa3a636115c5ca5 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:27.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.209+0000 7efdf4fcb640 1 -- 172.21.15.121:0/3035989877 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde8037070 con 0x7efdf0138fd0 2024-09-17T18:35:27.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.209+0000 7efdf4fcb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:27.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.209+0000 7efdf4fcb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:35:27.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.209+0000 7efdf4fcb640 10 monclient: dump: 2024-09-17T18:35:27.219 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:27.219 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:27.219 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:27.219 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:27.219 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:27.219 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:27.219 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:27.219 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:27.219 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:27.219 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:27.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.209+0000 7efdf4fcb640 10 monclient: _finish_auth 0 2024-09-17T18:35:27.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.209+0000 7efdf4fcb640 10 monclient: _check_auth_tickets 2024-09-17T18:35:27.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.209+0000 7efdf4fcb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:35:27.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.209+0000 7efdf4fcb640 1 -- 172.21.15.121:0/3035989877 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efde8032440 con 0x7efdf0138fd0 2024-09-17T18:35:27.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.209+0000 7efdf8231640 5 monclient: authenticate success, global_id 64481 2024-09-17T18:35:27.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.209+0000 7efdf8231640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:35:27.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.209+0000 7efdf4fcb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:35:27.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.209+0000 7efdf8231640 10 monclient: get_monmap_and_config success 2024-09-17T18:35:27.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.209+0000 7efdf4fcb640 1 -- 172.21.15.121:0/3035989877 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde8032810 con 0x7efdf0138fd0 2024-09-17T18:35:27.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.209+0000 7efdf8231640 4 set_mon_vals no callback set 2024-09-17T18:35:27.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.209+0000 7efdf4fcb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:27.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.209+0000 7efdf4fcb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:35:27.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.209+0000 7efdf4fcb640 10 monclient: dump: 2024-09-17T18:35:27.220 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:27.220 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:27.221 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:27.221 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:27.221 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:27.221 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:27.221 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:27.221 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:27.221 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:27.221 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:27.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.210+0000 7efdf8231640 10 monclient: shutdown 2024-09-17T18:35:27.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.210+0000 7efdf8231640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:35:27.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.210+0000 7efdf8231640 1 -- 172.21.15.121:0/3035989877 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdf0138fd0 msgr2=0x7efdf013fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:27.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.210+0000 7efdf8231640 1 --2- 172.21.15.121:0/3035989877 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdf0138fd0 0x7efdf013fbf0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7efde80097d0 tx=0x7efde802a080 comp rx=0 tx=0).stop 2024-09-17T18:35:27.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.210+0000 7efdf8231640 1 -- 172.21.15.121:0/3035989877 shutdown_connections 2024-09-17T18:35:27.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.210+0000 7efdf8231640 1 --2- 172.21.15.121:0/3035989877 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdf0138fd0 0x7efdf013fbf0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:27.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.210+0000 7efdf8231640 1 --2- 172.21.15.121:0/3035989877 >> v2:172.21.15.173:3300/0 conn(0x7efdf006ec00 0x7efdf0138a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:27.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.210+0000 7efdf8231640 1 -- 172.21.15.121:0/3035989877 >> 172.21.15.121:0/3035989877 conn(0x7efdf00a91e0 msgr2=0x7efdf00a95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:35:27.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.210+0000 7efdf8231640 1 -- 172.21.15.121:0/3035989877 shutdown_connections 2024-09-17T18:35:27.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.210+0000 7efdf8231640 1 -- 172.21.15.121:0/3035989877 wait complete. 2024-09-17T18:35:27.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.211+0000 7efdf8231640 10 monclient: build_initial_monmap 2024-09-17T18:35:27.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.211+0000 7efdf8231640 10 monclient: monmap: 2024-09-17T18:35:27.222 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:35:27.222 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:35:27.222 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:35:27.222 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:35:27.222 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:35:27.222 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:27.222 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:35:27.223 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:35:27.223 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:35:27.223 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:27.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.211+0000 7efdf8231640 1 librados: starting msgr at 2024-09-17T18:35:27.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.211+0000 7efdf8231640 1 librados: starting objecter 2024-09-17T18:35:27.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.211+0000 7efdf8231640 1 Processor -- start 2024-09-17T18:35:27.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.211+0000 7efdf8231640 1 -- start start 2024-09-17T18:35:27.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.211+0000 7efdf8231640 1 librados: setting wanted keys 2024-09-17T18:35:27.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.211+0000 7efdf8231640 1 librados: calling monclient init 2024-09-17T18:35:27.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.211+0000 7efdf8231640 10 monclient: init 2024-09-17T18:35:27.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.211+0000 7efdf8231640 10 monclient: _reopen_session rank -1 2024-09-17T18:35:27.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.211+0000 7efdf8231640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T18:35:27.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.211+0000 7efdf8231640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efdf006ec00 0x7efdf00a2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:27.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.211+0000 7efdf8231640 10 monclient(hunting): picked mon.noname-b con 0x7efdf006ec00 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:35:27.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.211+0000 7efdf8231640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdf01386b0 0x7efdf0073060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:27.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.211+0000 7efdf8231640 10 monclient(hunting): picked mon.noname-c con 0x7efdf01386b0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:35:27.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.211+0000 7efdf8231640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efdf0138fd0 0x7efdf00735a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:27.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.211+0000 7efdf8231640 10 monclient(hunting): picked mon.noname-a con 0x7efdf0138fd0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:35:27.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.211+0000 7efdf8231640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:27.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.211+0000 7efdf8231640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7efdf01404c0 con 0x7efdf0138fd0 2024-09-17T18:35:27.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.211+0000 7efdf8231640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:27.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.211+0000 7efdf8231640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7efdf0140340 con 0x7efdf006ec00 2024-09-17T18:35:27.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.211+0000 7efdf8231640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:27.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.211+0000 7efdf8231640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7efdf0140640 con 0x7efdf01386b0 2024-09-17T18:35:27.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.211+0000 7efdf8231640 10 monclient(hunting): _renew_subs 2024-09-17T18:35:27.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.211+0000 7efdf8231640 10 monclient(hunting): authenticate will time out at 1730.927884s 2024-09-17T18:35:27.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.211+0000 7efdf57cc640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdf01386b0 0x7efdf0073060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:27.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.211+0000 7efdf67ce640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efdf0138fd0 0x7efdf00735a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:27.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.211+0000 7efdf57cc640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdf01386b0 0x7efdf0073060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:53494/0 (socket says 172.21.15.121:53494) 2024-09-17T18:35:27.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.211+0000 7efdf67ce640 10 monclient(hunting): get_auth_request con 0x7efdf0138fd0 auth_method 0 2024-09-17T18:35:27.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.211+0000 7efdf67ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:27.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.211+0000 7efdf57cc640 1 -- 172.21.15.121:0/3511975594 learned_addr learned my addr 172.21.15.121:0/3511975594 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:35:27.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.211+0000 7efdf67ce640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:27.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.211+0000 7efdf67ce640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:27.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.211+0000 7efdf57cc640 10 monclient(hunting): get_auth_request con 0x7efdf01386b0 auth_method 0 2024-09-17T18:35:27.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.212+0000 7efdf57cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:27.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.212+0000 7efdf57cc640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:27.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.212+0000 7efdf57cc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:27.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.212+0000 7efdf67ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:27.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.212+0000 7efdf67ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:27.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.212+0000 7efdf67ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:27.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.212+0000 7efdf57cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:27.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.212+0000 7efdf57cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:27.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.212+0000 7efdf57cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:27.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.212+0000 7efdf57cc640 10 monclient(hunting): handle_auth_done global_id 64484 payload 824 2024-09-17T18:35:27.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.212+0000 7efdf57cc640 1 -- 172.21.15.121:0/3511975594 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efdf006ec00 msgr2=0x7efdf00a2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:27.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.212+0000 7efdf5fcd640 1 --2- 172.21.15.121:0/3511975594 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efdf006ec00 0x7efdf00a2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:27.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.212+0000 7efdf57cc640 1 --2- 172.21.15.121:0/3511975594 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efdf006ec00 0x7efdf00a2c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:27.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.212+0000 7efdf57cc640 1 -- 172.21.15.121:0/3511975594 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efdf0138fd0 msgr2=0x7efdf00735a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:27.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.212+0000 7efdf57cc640 1 --2- 172.21.15.121:0/3511975594 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efdf0138fd0 0x7efdf00735a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:27.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.212+0000 7efdf57cc640 10 monclient: _finish_hunting 0 2024-09-17T18:35:27.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.212+0000 7efdf57cc640 1 monclient: found mon.noname-c 2024-09-17T18:35:27.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.212+0000 7efdf57cc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:35:27.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.212+0000 7efdf57cc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:35:27.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.212+0000 7efdf57cc640 1 -- 172.21.15.121:0/3511975594 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdf0074980 con 0x7efdf01386b0 2024-09-17T18:35:27.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.212+0000 7efdf67ce640 1 --2- 172.21.15.121:0/3511975594 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efdf0138fd0 0x7efdf00735a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:35:27.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.214+0000 7efdf57cc640 1 -- 172.21.15.121:0/3511975594 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdf01386b0 msgr2=0x7efdf0073060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:27.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.214+0000 7efdf57cc640 1 --2- 172.21.15.121:0/3511975594 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdf01386b0 0x7efdf0073060 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7efde0007e70 tx=0x7efde000e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:27.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.214+0000 7efdee7fc640 10 client.?.objecter ms_handle_connect 0x7efdf01386b0 2024-09-17T18:35:27.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.214+0000 7efdee7fc640 10 client.?.objecter resend_mon_ops 2024-09-17T18:35:27.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.214+0000 7efdee7fc640 1 -- 172.21.15.121:0/3511975594 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde0014620 con 0x7efdf01386b0 2024-09-17T18:35:27.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.214+0000 7efdee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:27.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.214+0000 7efdee7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:35:27.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.214+0000 7efdee7fc640 10 monclient: dump: 2024-09-17T18:35:27.227 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:27.227 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:27.227 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:27.227 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:27.227 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:27.227 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:27.227 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:27.227 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:27.227 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:27.227 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:27.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.214+0000 7efdee7fc640 10 monclient: _finish_auth 0 2024-09-17T18:35:27.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.214+0000 7efdee7fc640 10 monclient: _check_auth_tickets 2024-09-17T18:35:27.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.214+0000 7efdee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:35:27.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.214+0000 7efdee7fc640 1 -- 172.21.15.121:0/3511975594 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efde0019070 con 0x7efdf01386b0 2024-09-17T18:35:27.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.214+0000 7efdf8231640 5 monclient: authenticate success, global_id 64484 2024-09-17T18:35:27.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.214+0000 7efdf8231640 10 monclient: _renew_subs 2024-09-17T18:35:27.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.214+0000 7efdf8231640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:27.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.214+0000 7efdf8231640 1 -- 172.21.15.121:0/3511975594 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdf00a8380 con 0x7efdf01386b0 2024-09-17T18:35:27.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.214+0000 7efdee7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:35:27.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.214+0000 7efdee7fc640 1 -- 172.21.15.121:0/3511975594 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde0014a00 con 0x7efdf01386b0 2024-09-17T18:35:27.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.214+0000 7efdee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:27.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.214+0000 7efdee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:35:27.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.214+0000 7efdee7fc640 10 monclient: dump: 2024-09-17T18:35:27.228 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:27.229 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:27.229 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:27.229 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:27.229 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:27.229 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:27.229 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:27.229 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:27.229 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:27.229 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:27.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.214+0000 7efdf8231640 10 client.64484.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:35:27.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.214+0000 7efdf8231640 10 monclient: _renew_subs 2024-09-17T18:35:27.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.214+0000 7efdf8231640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:27.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.214+0000 7efdeffff640 4 set_mon_vals no callback set 2024-09-17T18:35:27.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.214+0000 7efdf8231640 1 -- 172.21.15.121:0/3511975594 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efdf00a03d0 con 0x7efdf01386b0 2024-09-17T18:35:27.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.214+0000 7efdf8231640 1 librados: init done 2024-09-17T18:35:27.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.215+0000 7efdf8231640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:27.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.215+0000 7efdf8231640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:27.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.215+0000 7efdf8231640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:27.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.215+0000 7efdf8231640 1 -- 172.21.15.121:0/3511975594 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdf0138a90 con 0x7efdf01386b0 2024-09-17T18:35:27.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.215+0000 7efdee7fc640 1 -- 172.21.15.121:0/3511975594 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7efde0014ba0 con 0x7efdf01386b0 2024-09-17T18:35:27.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.215+0000 7efdee7fc640 1 --2- 172.21.15.121:0/3511975594 >> v2:172.21.15.173:6812/2423786037 conn(0x7efddc0401c0 0x7efddc042680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:27.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.216+0000 7efdee7fc640 1 -- 172.21.15.121:0/3511975594 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(646..646 src has 247..646) ==== 5925+0+0 (secure 0 0 0) 0x7efde0054140 con 0x7efdf01386b0 2024-09-17T18:35:27.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.216+0000 7efdee7fc640 10 client.64484.objecter ms_dispatch 0x7efdf0000b90 osd_map(646..646 src has 247..646) 2024-09-17T18:35:27.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.216+0000 7efdee7fc640 3 client.64484.objecter handle_osd_map got epochs [646,646] > 0 2024-09-17T18:35:27.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.216+0000 7efdee7fc640 3 client.64484.objecter handle_osd_map decoding full epoch 646 2024-09-17T18:35:27.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.216+0000 7efdee7fc640 20 client.64484.objecter dump_active .. 0 homeless 2024-09-17T18:35:27.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.216+0000 7efdf5fcd640 1 --2- 172.21.15.121:0/3511975594 >> v2:172.21.15.173:6812/2423786037 conn(0x7efddc0401c0 0x7efddc042680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:27.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.216+0000 7efdf5fcd640 10 monclient: get_auth_request con 0x7efddc0401c0 auth_method 0 2024-09-17T18:35:27.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.219+0000 7efdf5fcd640 1 --2- 172.21.15.121:0/3511975594 >> v2:172.21.15.173:6812/2423786037 conn(0x7efddc0401c0 0x7efddc042680 secure :-1 s=READY pgs=4074 cs=0 l=1 rev1=1 crypto rx=0x7efde4004ff0 tx=0x7efde4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:27.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.219+0000 7efdee7fc640 10 client.64484.objecter ms_handle_connect 0x7efddc0401c0 2024-09-17T18:35:27.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.220+0000 7efdee7fc640 1 -- 172.21.15.121:0/3511975594 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7efde0012cf0 con 0x7efdf01386b0 2024-09-17T18:35:27.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.220+0000 7efdee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:27.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.220+0000 7efdee7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:35:27.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.465+0000 7efdf8231640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-17T18:35:27.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.465+0000 7efdf8231640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-17T18:35:27.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.465+0000 7efdf8231640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:27.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.465+0000 7efdf8231640 1 -- 172.21.15.121:0/3511975594 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7efdf006cd40 con 0x7efdf01386b0 2024-09-17T18:35:27.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.466+0000 7efdee7fc640 1 -- 172.21.15.121:0/3511975594 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7efde001b310 con 0x7efdf01386b0 2024-09-17T18:35:27.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.466+0000 7efdee7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-17T18:35:27.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.466+0000 7efdee7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:35:27.469 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.467+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.469 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.467+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,config=2,health=12,kv=12,logm=54} crc {auth=1698847108,config=2586702516,health=6513301,kv=3128395109,logm=2935324592}) 2024-09-17T18:35:27.469 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.467+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=20,config=2,health=12,kv=14,logm=52} crc {auth=1654673058,config=2586702516,health=3950847391,kv=2790417503,logm=775693337}) 2024-09-17T18:35:27.469 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.467+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.469 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.467+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,config=2,health=12,kv=12,logm=54} crc {auth=1698847108,config=2586702516,health=6513301,kv=3128395109,logm=2935324592}) 2024-09-17T18:35:27.469 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.467+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=20,config=2,health=11,kv=14,logm=53} crc {auth=1654673058,config=2586702516,health=2307765799,kv=2790417503,logm=3939703873}) 2024-09-17T18:35:27.470 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.468+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.470 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.468+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=500856444}) 2024-09-17T18:35:27.470 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.468+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3393701693}) 2024-09-17T18:35:27.470 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.468+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.470 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.468+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=500856444}) 2024-09-17T18:35:27.470 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.468+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2897343419}) 2024-09-17T18:35:27.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.468+0000 7efdf8231640 10 librados: watch_flush enter 2024-09-17T18:35:27.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.468+0000 7efdf8231640 10 librados: watch_flush exit 2024-09-17T18:35:27.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.468+0000 7efdf8231640 20 client.64484.objecter shutdown clearing up homeless session... 2024-09-17T18:35:27.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.468+0000 7efdf8231640 10 client.64484.objecter successfully canceled tick 2024-09-17T18:35:27.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.468+0000 7efdf8231640 1 -- 172.21.15.121:0/3511975594 >> v2:172.21.15.173:6812/2423786037 conn(0x7efddc0401c0 msgr2=0x7efddc042680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:27.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.468+0000 7efdf8231640 1 --2- 172.21.15.121:0/3511975594 >> v2:172.21.15.173:6812/2423786037 conn(0x7efddc0401c0 0x7efddc042680 secure :-1 s=READY pgs=4074 cs=0 l=1 rev1=1 crypto rx=0x7efde4004ff0 tx=0x7efde4009040 comp rx=0 tx=0).stop 2024-09-17T18:35:27.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.468+0000 7efdf8231640 10 monclient: shutdown 2024-09-17T18:35:27.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.468+0000 7efdf8231640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:35:27.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.468+0000 7efdf8231640 1 -- 172.21.15.121:0/3511975594 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdf01386b0 msgr2=0x7efdf0073060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:27.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.468+0000 7efdf8231640 1 --2- 172.21.15.121:0/3511975594 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdf01386b0 0x7efdf0073060 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7efde0007e70 tx=0x7efde000e4b0 comp rx=0 tx=0).stop 2024-09-17T18:35:27.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.468+0000 7efdf8231640 1 -- 172.21.15.121:0/3511975594 shutdown_connections 2024-09-17T18:35:27.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.468+0000 7efdf8231640 1 --2- 172.21.15.121:0/3511975594 >> v2:172.21.15.173:6812/2423786037 conn(0x7efddc0401c0 0x7efddc042680 unknown :-1 s=CLOSED pgs=4074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:27.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.468+0000 7efdf8231640 1 --2- 172.21.15.121:0/3511975594 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efdf0138fd0 0x7efdf00735a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:27.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.468+0000 7efdf8231640 1 --2- 172.21.15.121:0/3511975594 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdf01386b0 0x7efdf0073060 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:27.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.468+0000 7efdf8231640 1 --2- 172.21.15.121:0/3511975594 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efdf006ec00 0x7efdf00a2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:27.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.468+0000 7efdf8231640 1 -- 172.21.15.121:0/3511975594 >> 172.21.15.121:0/3511975594 conn(0x7efdf00a91e0 msgr2=0x7efdf0131fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:35:27.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.469+0000 7efdf8231640 1 -- 172.21.15.121:0/3511975594 shutdown_connections 2024-09-17T18:35:27.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.469+0000 7efdf8231640 1 -- 172.21.15.121:0/3511975594 wait complete. 2024-09-17T18:35:27.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:27.469+0000 7efdf8231640 1 librados: shutdown 2024-09-17T18:35:27.474 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.469+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.474 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.469+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2586811088}) 2024-09-17T18:35:27.474 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.469+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=805186341}) 2024-09-17T18:35:27.474 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.469+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.474 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.469+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2586811088}) 2024-09-17T18:35:27.475 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.469+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1206323662}) 2024-09-17T18:35:27.475 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.470+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.475 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.470+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4134285684}) 2024-09-17T18:35:27.475 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.470+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1579966475}) 2024-09-17T18:35:27.475 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.470+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.475 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.470+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4134285684}) 2024-09-17T18:35:27.475 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.470+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=194182751}) 2024-09-17T18:35:27.475 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.471+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.475 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.471+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3632424983}) 2024-09-17T18:35:27.475 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.471+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2926267906}) 2024-09-17T18:35:27.475 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.471+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.475 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.471+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3632424983}) 2024-09-17T18:35:27.475 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.471+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=698255687}) 2024-09-17T18:35:27.475 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.472+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.475 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.472+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3567416020}) 2024-09-17T18:35:27.476 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.472+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=794205983}) 2024-09-17T18:35:27.476 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.472+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.476 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.472+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3567416020}) 2024-09-17T18:35:27.476 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.472+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=229882868}) 2024-09-17T18:35:27.476 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.472+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.476 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.472+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2777086210}) 2024-09-17T18:35:27.476 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.472+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3062747468}) 2024-09-17T18:35:27.476 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.472+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.476 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.472+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2777086210}) 2024-09-17T18:35:27.476 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.472+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3173015607}) 2024-09-17T18:35:27.477 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.473+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.478 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.473+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1256963382}) 2024-09-17T18:35:27.478 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.473+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2316487711}) 2024-09-17T18:35:27.478 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.473+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.478 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.473+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1256963382}) 2024-09-17T18:35:27.478 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.473+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4254303038}) 2024-09-17T18:35:27.478 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.474+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.478 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.474+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2795100359}) 2024-09-17T18:35:27.478 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.474+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1448592078}) 2024-09-17T18:35:27.478 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.474+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.478 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.474+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2795100359}) 2024-09-17T18:35:27.478 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.474+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1970291280}) 2024-09-17T18:35:27.478 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.475+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.478 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.475+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1234133685}) 2024-09-17T18:35:27.479 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.475+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2088999146}) 2024-09-17T18:35:27.479 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.475+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.479 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.475+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1234133685}) 2024-09-17T18:35:27.479 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.475+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1515300634}) 2024-09-17T18:35:27.479 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.476+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.479 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.476+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1579608825}) 2024-09-17T18:35:27.479 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.476+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3351219911}) 2024-09-17T18:35:27.479 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.476+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.479 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.476+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1579608825}) 2024-09-17T18:35:27.479 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.476+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=785214907}) 2024-09-17T18:35:27.480 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.476+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.480 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.476+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4185187229}) 2024-09-17T18:35:27.481 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.476+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3556793732}) 2024-09-17T18:35:27.481 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.476+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.481 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.476+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4185187229}) 2024-09-17T18:35:27.481 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.476+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=469765844}) 2024-09-17T18:35:27.481 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.477+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.481 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.477+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2764190304}) 2024-09-17T18:35:27.481 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.477+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2315761101}) 2024-09-17T18:35:27.481 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.477+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.481 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.477+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2764190304}) 2024-09-17T18:35:27.481 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.477+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1016486278}) 2024-09-17T18:35:27.481 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.478+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.481 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.478+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4055971176}) 2024-09-17T18:35:27.481 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.478+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2949899497}) 2024-09-17T18:35:27.481 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.478+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.481 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.478+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4055971176}) 2024-09-17T18:35:27.482 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.478+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=169223312}) 2024-09-17T18:35:27.483 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.479+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.483 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.479+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3124971845}) 2024-09-17T18:35:27.483 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.479+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3940585958}) 2024-09-17T18:35:27.483 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.479+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.483 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.479+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3124971845}) 2024-09-17T18:35:27.483 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.479+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1145683936}) 2024-09-17T18:35:27.483 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.480+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.483 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.480+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=24383345}) 2024-09-17T18:35:27.483 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.480+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4293653857}) 2024-09-17T18:35:27.483 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.480+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.483 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.480+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=24383345}) 2024-09-17T18:35:27.483 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.480+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=288136334}) 2024-09-17T18:35:27.484 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-17T18:35:27.485 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.480+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.486 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.480+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1062745897}) 2024-09-17T18:35:27.486 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.480+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3343850449}) 2024-09-17T18:35:27.486 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.480+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.486 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.481+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1062745897}) 2024-09-17T18:35:27.486 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.481+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2147424201}) 2024-09-17T18:35:27.486 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.481+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.486 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.481+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4204996973}) 2024-09-17T18:35:27.486 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.481+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=194146886}) 2024-09-17T18:35:27.486 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.481+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.486 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.481+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4204996973}) 2024-09-17T18:35:27.486 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.481+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=843502423}) 2024-09-17T18:35:27.486 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.482+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.486 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.482+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3002385400}) 2024-09-17T18:35:27.486 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.482+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2441615181}) 2024-09-17T18:35:27.487 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.482+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.487 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.482+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3002385400}) 2024-09-17T18:35:27.487 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.482+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=701227323}) 2024-09-17T18:35:27.487 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.482+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.487 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.482+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4156513992}) 2024-09-17T18:35:27.487 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.482+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4026149165}) 2024-09-17T18:35:27.487 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.482+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.487 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.482+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4156513992}) 2024-09-17T18:35:27.487 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.483+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2962061266}) 2024-09-17T18:35:27.487 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.483+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.487 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.483+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1280087944}) 2024-09-17T18:35:27.487 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.483+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1007818964}) 2024-09-17T18:35:27.487 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.483+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.487 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.483+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1280087944}) 2024-09-17T18:35:27.487 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.483+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4254926466}) 2024-09-17T18:35:27.751 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.748+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.751 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.748+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=572055608}) 2024-09-17T18:35:27.751 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.748+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3954638259}) 2024-09-17T18:35:27.751 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.748+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.752 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.748+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=572055608}) 2024-09-17T18:35:27.752 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.748+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4279186022}) 2024-09-17T18:35:27.799 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.796+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.799 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.796+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4253026997}) 2024-09-17T18:35:27.799 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.796+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=828860072}) 2024-09-17T18:35:27.799 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.796+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.799 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.796+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4253026997}) 2024-09-17T18:35:27.799 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.796+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3110686060}) 2024-09-17T18:35:27.878 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.875+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.878 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.875+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2772849470}) 2024-09-17T18:35:27.878 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.875+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1921101936}) 2024-09-17T18:35:27.879 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.875+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:27.879 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.875+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2772849470}) 2024-09-17T18:35:27.879 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:27.875+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2003418842}) 2024-09-17T18:35:28.062 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.059+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.062 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.059+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1253910835}) 2024-09-17T18:35:28.062 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.059+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1390888431}) 2024-09-17T18:35:28.062 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.059+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.062 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.059+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1253910835}) 2024-09-17T18:35:28.062 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.059+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3619215236}) 2024-09-17T18:35:28.063 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.060+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.063 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.060+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1585755161}) 2024-09-17T18:35:28.063 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.060+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3515440755}) 2024-09-17T18:35:28.063 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.060+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.063 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.060+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1585755161}) 2024-09-17T18:35:28.063 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.060+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3377925740}) 2024-09-17T18:35:28.064 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.061+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.064 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.061+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3844692311}) 2024-09-17T18:35:28.064 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.061+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1294263728}) 2024-09-17T18:35:28.064 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.061+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.065 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.061+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3844692311}) 2024-09-17T18:35:28.065 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.061+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=623508162}) 2024-09-17T18:35:28.065 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.062+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.065 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.062+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1295137315}) 2024-09-17T18:35:28.065 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.062+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1843216524}) 2024-09-17T18:35:28.065 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.062+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.066 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.062+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1295137315}) 2024-09-17T18:35:28.066 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.062+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2471947420}) 2024-09-17T18:35:28.067 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.063+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.067 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.063+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2857048682}) 2024-09-17T18:35:28.067 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.063+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1823130326}) 2024-09-17T18:35:28.067 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.063+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.067 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.063+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2857048682}) 2024-09-17T18:35:28.067 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.063+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1779213093}) 2024-09-17T18:35:28.067 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.064+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.067 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.064+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3683025854}) 2024-09-17T18:35:28.067 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.064+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3194137503}) 2024-09-17T18:35:28.068 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.064+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.068 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.064+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3683025854}) 2024-09-17T18:35:28.068 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.064+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1613650732}) 2024-09-17T18:35:28.069 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.065+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.069 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.065+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4235464241}) 2024-09-17T18:35:28.069 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.065+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=420586599}) 2024-09-17T18:35:28.069 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.065+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.069 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.065+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4235464241}) 2024-09-17T18:35:28.069 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.065+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3382744032}) 2024-09-17T18:35:28.069 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.066+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.069 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.066+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2631997632}) 2024-09-17T18:35:28.070 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.066+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1639117388}) 2024-09-17T18:35:28.070 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.066+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.070 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.066+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2631997632}) 2024-09-17T18:35:28.070 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.066+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2948059654}) 2024-09-17T18:35:28.071 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.067+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.071 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.067+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1571192560}) 2024-09-17T18:35:28.071 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.067+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1264172580}) 2024-09-17T18:35:28.071 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.067+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.071 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.067+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1571192560}) 2024-09-17T18:35:28.071 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.067+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1131213841}) 2024-09-17T18:35:28.071 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.068+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.071 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.068+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=935027487}) 2024-09-17T18:35:28.071 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.068+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=610618655}) 2024-09-17T18:35:28.071 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.068+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.072 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.068+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=935027487}) 2024-09-17T18:35:28.072 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.068+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3429786234}) 2024-09-17T18:35:28.072 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.069+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.072 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.069+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2063102112}) 2024-09-17T18:35:28.072 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.069+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2149235005}) 2024-09-17T18:35:28.073 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.069+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.073 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.069+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2063102112}) 2024-09-17T18:35:28.073 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.069+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1509460502}) 2024-09-17T18:35:28.074 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.070+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.074 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.070+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1013480121}) 2024-09-17T18:35:28.075 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.070+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3492761223}) 2024-09-17T18:35:28.075 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.070+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.075 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.070+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1013480121}) 2024-09-17T18:35:28.075 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.070+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2580289804}) 2024-09-17T18:35:28.075 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.071+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.075 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.071+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1225190852}) 2024-09-17T18:35:28.075 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.071+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2627013013}) 2024-09-17T18:35:28.075 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.071+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.075 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.071+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1225190852}) 2024-09-17T18:35:28.075 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.071+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2165294847}) 2024-09-17T18:35:28.076 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.072+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.076 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.072+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2542210175}) 2024-09-17T18:35:28.076 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.072+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2461277614}) 2024-09-17T18:35:28.076 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.072+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.076 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.072+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2542210175}) 2024-09-17T18:35:28.076 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.072+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3081686454}) 2024-09-17T18:35:28.079 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.072+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.079 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.072+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3685774527}) 2024-09-17T18:35:28.080 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.072+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2770221077}) 2024-09-17T18:35:28.080 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.072+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.080 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.073+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3685774527}) 2024-09-17T18:35:28.080 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.073+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=49040661}) 2024-09-17T18:35:28.080 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.073+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.080 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.073+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=212926752}) 2024-09-17T18:35:28.080 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.073+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3256241141}) 2024-09-17T18:35:28.080 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.073+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.080 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.073+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=212926752}) 2024-09-17T18:35:28.080 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.073+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2964666066}) 2024-09-17T18:35:28.080 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.074+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.080 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.074+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=390798243}) 2024-09-17T18:35:28.081 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.074+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=306136885}) 2024-09-17T18:35:28.081 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.074+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.081 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.074+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=390798243}) 2024-09-17T18:35:28.081 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.074+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4107082193}) 2024-09-17T18:35:28.081 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.075+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.081 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.075+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2770746830}) 2024-09-17T18:35:28.081 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.075+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1091069533}) 2024-09-17T18:35:28.081 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.075+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.081 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.075+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2770746830}) 2024-09-17T18:35:28.081 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.075+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4181142622}) 2024-09-17T18:35:28.082 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.075+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.082 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.075+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3012890299}) 2024-09-17T18:35:28.082 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.075+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3815769085}) 2024-09-17T18:35:28.082 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.075+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.082 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.075+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3012890299}) 2024-09-17T18:35:28.082 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.075+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=846311148}) 2024-09-17T18:35:28.150 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.147+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.150 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.147+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1328086147}) 2024-09-17T18:35:28.150 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.147+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4282963750}) 2024-09-17T18:35:28.150 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.147+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.150 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.147+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1328086147}) 2024-09-17T18:35:28.150 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.147+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=135180093}) 2024-09-17T18:35:28.154 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.151+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.154 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.151+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3370874500}) 2024-09-17T18:35:28.154 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.151+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1308859518}) 2024-09-17T18:35:28.154 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.151+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.154 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.151+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3370874500}) 2024-09-17T18:35:28.154 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.151+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1379723615}) 2024-09-17T18:35:28.165 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.162+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.165 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.162+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1188655570}) 2024-09-17T18:35:28.166 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.162+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=71465509}) 2024-09-17T18:35:28.166 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.163+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.166 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.163+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1188655570}) 2024-09-17T18:35:28.166 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.163+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1745638757}) 2024-09-17T18:35:28.398 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.395+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.398 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.395+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2319619979}) 2024-09-17T18:35:28.398 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.395+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2997908669}) 2024-09-17T18:35:28.398 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.395+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.398 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.395+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2319619979}) 2024-09-17T18:35:28.398 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.395+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2824868069}) 2024-09-17T18:35:28.401 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.396+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.402 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.396+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=303684241}) 2024-09-17T18:35:28.402 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.396+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3271643061}) 2024-09-17T18:35:28.402 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.396+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.402 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.396+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=303684241}) 2024-09-17T18:35:28.402 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.396+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3483475905}) 2024-09-17T18:35:28.402 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.396+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.402 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.396+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1875147405}) 2024-09-17T18:35:28.402 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.396+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=893750560}) 2024-09-17T18:35:28.402 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.396+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.402 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.396+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1875147405}) 2024-09-17T18:35:28.402 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.397+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3405210795}) 2024-09-17T18:35:28.402 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.397+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.402 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.397+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4178926665}) 2024-09-17T18:35:28.402 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.397+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2045875752}) 2024-09-17T18:35:28.403 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.397+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.403 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.397+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4178926665}) 2024-09-17T18:35:28.403 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.397+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1195904448}) 2024-09-17T18:35:28.403 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.398+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.403 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.398+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1049074799}) 2024-09-17T18:35:28.403 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.398+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=514753467}) 2024-09-17T18:35:28.403 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.398+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.403 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.398+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1049074799}) 2024-09-17T18:35:28.403 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.398+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3595303393}) 2024-09-17T18:35:28.403 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.399+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.403 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.399+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2324589298}) 2024-09-17T18:35:28.403 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.399+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1851640971}) 2024-09-17T18:35:28.404 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.399+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.404 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.399+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2324589298}) 2024-09-17T18:35:28.404 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.399+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3478912241}) 2024-09-17T18:35:28.411 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.399+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.411 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.399+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=593105036}) 2024-09-17T18:35:28.411 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.399+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2655828526}) 2024-09-17T18:35:28.411 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.399+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.411 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.399+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=593105036}) 2024-09-17T18:35:28.411 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.399+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2884930009}) 2024-09-17T18:35:28.411 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.400+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.411 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.400+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3396797764}) 2024-09-17T18:35:28.411 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.400+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3809527199}) 2024-09-17T18:35:28.411 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.400+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.412 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.400+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3396797764}) 2024-09-17T18:35:28.412 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.400+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2293608600}) 2024-09-17T18:35:28.412 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.401+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.412 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.401+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=621402995}) 2024-09-17T18:35:28.412 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.401+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1185660443}) 2024-09-17T18:35:28.412 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.401+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.412 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.401+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=621402995}) 2024-09-17T18:35:28.412 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.401+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1501174658}) 2024-09-17T18:35:28.412 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.401+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.412 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.401+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=492431137}) 2024-09-17T18:35:28.412 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.401+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2355681717}) 2024-09-17T18:35:28.412 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.401+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.413 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.401+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=492431137}) 2024-09-17T18:35:28.413 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.401+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2801498400}) 2024-09-17T18:35:28.413 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.402+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.413 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.402+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1025059696}) 2024-09-17T18:35:28.413 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.402+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3461873874}) 2024-09-17T18:35:28.413 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.402+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.413 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.402+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1025059696}) 2024-09-17T18:35:28.413 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.402+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2449752406}) 2024-09-17T18:35:28.413 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.403+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.413 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.403+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=446197526}) 2024-09-17T18:35:28.413 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.403+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=762361694}) 2024-09-17T18:35:28.413 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.403+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.413 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.403+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=446197526}) 2024-09-17T18:35:28.414 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.403+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=299746770}) 2024-09-17T18:35:28.414 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.404+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.414 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.404+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2876167534}) 2024-09-17T18:35:28.414 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.404+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1943316323}) 2024-09-17T18:35:28.414 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.404+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.414 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.404+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2876167534}) 2024-09-17T18:35:28.414 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.404+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=102330125}) 2024-09-17T18:35:28.414 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.404+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.414 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.404+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=907605656}) 2024-09-17T18:35:28.414 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.404+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2661450570}) 2024-09-17T18:35:28.414 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.404+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.414 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.404+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=907605656}) 2024-09-17T18:35:28.414 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.404+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=304220639}) 2024-09-17T18:35:28.414 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.405+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.415 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.405+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3120978788}) 2024-09-17T18:35:28.415 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.405+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3746234189}) 2024-09-17T18:35:28.415 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.405+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.415 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.405+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3120978788}) 2024-09-17T18:35:28.415 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.405+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2195274185}) 2024-09-17T18:35:28.415 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.406+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.415 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.406+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2771048296}) 2024-09-17T18:35:28.415 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.406+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2351457277}) 2024-09-17T18:35:28.415 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.406+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.415 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.406+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2771048296}) 2024-09-17T18:35:28.415 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.406+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1943516540}) 2024-09-17T18:35:28.415 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.406+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.415 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.406+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=431149727}) 2024-09-17T18:35:28.416 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.406+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1065587890}) 2024-09-17T18:35:28.416 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.406+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.416 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.406+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=431149727}) 2024-09-17T18:35:28.416 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.406+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4115806208}) 2024-09-17T18:35:28.416 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.407+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.416 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.407+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2717569170}) 2024-09-17T18:35:28.416 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.407+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1244283794}) 2024-09-17T18:35:28.416 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.407+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.416 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.407+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2717569170}) 2024-09-17T18:35:28.416 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.407+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2617380472}) 2024-09-17T18:35:28.416 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.408+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.416 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.408+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=45,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=1} crc {logm=383166691,mdsmap=761013554,mgr=3361498112,mgr_command_descs=1957073259,mgr_metadata=402068389,mgrstat=443857597,mon_config_key=163209300,monmap=2277444362,nvmeofgw=598337686,osd_metadata=121115626,osd_pg_creating=1889573101,osd_snap=3823785925}) 2024-09-17T18:35:28.416 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.408+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=45,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=17,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=3} crc {logm=3932116090,mdsmap=761013554,mgr=3361498112,mgr_command_descs=1957073259,mgr_metadata=402068389,mgrstat=343470182,mon_config_key=1582663684,monmap=344458555,nvmeofgw=598337686,osd_metadata=121115626,osd_pg_creating=1889573101,osd_snap=1447875684}) 2024-09-17T18:35:28.416 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.408+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.417 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.408+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=45,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=1} crc {logm=383166691,mdsmap=761013554,mgr=3361498112,mgr_command_descs=1957073259,mgr_metadata=402068389,mgrstat=443857597,mon_config_key=163209300,monmap=2277444362,nvmeofgw=598337686,osd_metadata=121115626,osd_pg_creating=1889573101,osd_snap=3823785925}) 2024-09-17T18:35:28.417 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.408+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=42,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=15,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=8} crc {logm=2638839502,mdsmap=761013554,mgr=3361498112,mgr_command_descs=1957073259,mgr_metadata=402068389,mgrstat=624887647,mon_config_key=1994770153,monmap=3140514009,nvmeofgw=598337686,osd_metadata=121115626,osd_pg_creating=1889573101,osd_snap=128455566}) 2024-09-17T18:35:28.420 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.409+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.420 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.409+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=32,osdmap=68} crc {osd_snap=24946121,osdmap=4064273114}) 2024-09-17T18:35:28.420 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.409+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=34,osdmap=66} crc {osd_snap=2178417005,osdmap=4222756454}) 2024-09-17T18:35:28.420 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.409+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.420 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.409+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=32,osdmap=68} crc {osd_snap=24946121,osdmap=4064273114}) 2024-09-17T18:35:28.420 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.409+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=36,osdmap=64} crc {osd_snap=1688983095,osdmap=2440673367}) 2024-09-17T18:35:28.420 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.410+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.420 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.410+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2262369418}) 2024-09-17T18:35:28.420 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.410+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3744064905}) 2024-09-17T18:35:28.420 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.410+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.420 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.410+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2262369418}) 2024-09-17T18:35:28.421 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.410+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3486072057}) 2024-09-17T18:35:28.421 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.410+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.421 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.410+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3393367643}) 2024-09-17T18:35:28.421 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.410+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=217476356}) 2024-09-17T18:35:28.421 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.410+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.421 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.410+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3393367643}) 2024-09-17T18:35:28.421 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.410+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3251127700}) 2024-09-17T18:35:28.421 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.411+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.421 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.411+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1117430173}) 2024-09-17T18:35:28.421 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.411+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3203685274}) 2024-09-17T18:35:28.421 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.411+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.421 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.411+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1117430173}) 2024-09-17T18:35:28.421 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.411+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3052223626}) 2024-09-17T18:35:28.421 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.413+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.421 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.413+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=768531855}) 2024-09-17T18:35:28.421 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.413+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=382783679}) 2024-09-17T18:35:28.422 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.413+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.422 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.413+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=768531855}) 2024-09-17T18:35:28.422 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.413+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1047378008}) 2024-09-17T18:35:28.422 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.415+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.422 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.415+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3497428580}) 2024-09-17T18:35:28.422 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.415+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3928555280}) 2024-09-17T18:35:28.422 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.415+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.422 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.415+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3497428580}) 2024-09-17T18:35:28.422 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.415+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=663121195}) 2024-09-17T18:35:28.423 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.420+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.423 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.420+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=16} crc {osdmap=2875378822}) 2024-09-17T18:35:28.423 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.420+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=15} crc {osdmap=3687943027}) 2024-09-17T18:35:28.423 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.420+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:35:28.423 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.420+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=16} crc {osdmap=2875378822}) 2024-09-17T18:35:28.423 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:35:28.420+0000 7f81e4013640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=16} crc {osdmap=1798602627}) 2024-09-17T18:35:28.485 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T18:35:28.485 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:35:28.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.580+0000 7fdc905e2640 10 monclient: get_monmap_and_config 2024-09-17T18:35:28.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.580+0000 7fdc905e2640 10 monclient: build_initial_monmap 2024-09-17T18:35:28.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.580+0000 7fdc905e2640 10 monclient: monmap: 2024-09-17T18:35:28.583 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:35:28.583 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:28.583 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:35:28.581883+0000 2024-09-17T18:35:28.583 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:35:28.581883+0000 2024-09-17T18:35:28.583 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:35:28.583 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:28.583 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:35:28.583 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:35:28.583 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:35:28.584 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:35:28.584 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:35:28.584 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:28.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.580+0000 7fdc905e2640 1 Processor -- start 2024-09-17T18:35:28.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.580+0000 7fdc905e2640 1 -- start start 2024-09-17T18:35:28.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.580+0000 7fdc905e2640 10 monclient: init 2024-09-17T18:35:28.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.581+0000 7fdc905e2640 10 monclient: _reopen_session rank -1 2024-09-17T18:35:28.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.581+0000 7fdc905e2640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-17T18:35:28.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.581+0000 7fdc905e2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdc880a5c00 addr v1:172.21.15.173:6789/0 2024-09-17T18:35:28.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.581+0000 7fdc905e2640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fdc880a5fe0 0x7fdc880a63f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:28.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.581+0000 7fdc905e2640 10 monclient(hunting): picked mon.noname-c con 0x7fdc880a5fe0 addr v2:172.21.15.173:3300/0 2024-09-17T18:35:28.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.581+0000 7fdc905e2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdc8813dc00 addr v1:172.21.15.121:6789/0 2024-09-17T18:35:28.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.581+0000 7fdc905e2640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc880a8a80 con 0x7fdc8813dc00 2024-09-17T18:35:28.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.581+0000 7fdc905e2640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc880a74d0 con 0x7fdc880a5c00 2024-09-17T18:35:28.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.581+0000 7fdc905e2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:28.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.581+0000 7fdc905e2640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fdc8809cb90 con 0x7fdc880a5fe0 2024-09-17T18:35:28.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.581+0000 7fdc905e2640 10 monclient(hunting): _renew_subs 2024-09-17T18:35:28.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.581+0000 7fdc905e2640 10 monclient(hunting): authenticate will time out at 1732.298656s 2024-09-17T18:35:28.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.581+0000 7fdc8eb7f640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fdc8813dc00 0x7fdc8813fff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:43054/0 (socket says 172.21.15.121:43054) 2024-09-17T18:35:28.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.581+0000 7fdc8eb7f640 1 -- 172.21.15.121:0/251393516 learned_addr learned my addr 172.21.15.121:0/251393516 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:35:28.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.581+0000 7fdc8db7d640 1 --2- 172.21.15.121:0/251393516 >> v2:172.21.15.173:3300/0 conn(0x7fdc880a5fe0 0x7fdc880a63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:28.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.581+0000 7fdc8db7d640 10 monclient(hunting): get_auth_request con 0x7fdc880a5fe0 auth_method 0 2024-09-17T18:35:28.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.581+0000 7fdc8db7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:28.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.581+0000 7fdc8db7d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:28.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.581+0000 7fdc8db7d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:28.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.582+0000 7fdc8db7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:28.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.582+0000 7fdc8db7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:28.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.582+0000 7fdc8db7d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:28.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.582+0000 7fdc8db7d640 10 monclient(hunting): handle_auth_done global_id 84262 payload 274 2024-09-17T18:35:28.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.582+0000 7fdc8db7d640 1 -- 172.21.15.121:0/251393516 >> v1:172.21.15.173:6789/0 conn(0x7fdc880a5c00 legacy=0x7fdc8809d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:35:28.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.582+0000 7fdc8db7d640 1 -- 172.21.15.121:0/251393516 >> v1:172.21.15.121:6789/0 conn(0x7fdc8813dc00 legacy=0x7fdc8813fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:35:28.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.582+0000 7fdc8db7d640 10 monclient: _finish_hunting 0 2024-09-17T18:35:28.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.582+0000 7fdc8db7d640 1 monclient: found mon.noname-c 2024-09-17T18:35:28.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.582+0000 7fdc8db7d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:35:28.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.582+0000 7fdc8db7d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.173:3300/0 2024-09-17T18:35:28.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.582+0000 7fdc8db7d640 1 -- 172.21.15.121:0/251393516 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc8809ce70 con 0x7fdc880a5fe0 2024-09-17T18:35:28.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.585+0000 7fdc8db7d640 1 -- 172.21.15.121:0/251393516 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc880a5fe0 msgr2=0x7fdc880a63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:28.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.585+0000 7fdc8db7d640 1 --2- 172.21.15.121:0/251393516 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc880a5fe0 0x7fdc880a63f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdc840097a0 tx=0x7fdc84029fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3824aca57ef3553a server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:28.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.585+0000 7fdc8d37c640 1 -- 172.21.15.121:0/251393516 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc84037070 con 0x7fdc880a5fe0 2024-09-17T18:35:28.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.585+0000 7fdc8d37c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:28.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.585+0000 7fdc8d37c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:35:28.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.585+0000 7fdc8d37c640 10 monclient: dump: 2024-09-17T18:35:28.598 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:28.598 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:28.598 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:28.598 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:28.598 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:28.598 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:28.599 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:28.599 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:28.599 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:28.599 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:28.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.585+0000 7fdc8d37c640 10 monclient: _finish_auth 0 2024-09-17T18:35:28.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.585+0000 7fdc8d37c640 10 monclient: _check_auth_tickets 2024-09-17T18:35:28.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.585+0000 7fdc8d37c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:35:28.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.585+0000 7fdc8d37c640 1 -- 172.21.15.121:0/251393516 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc84032640 con 0x7fdc880a5fe0 2024-09-17T18:35:28.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.585+0000 7fdc905e2640 5 monclient: authenticate success, global_id 84262 2024-09-17T18:35:28.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.585+0000 7fdc905e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:35:28.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.585+0000 7fdc8d37c640 10 monclient: handle_config config(0 keys) 2024-09-17T18:35:28.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.585+0000 7fdc8d37c640 1 -- 172.21.15.121:0/251393516 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc84032a10 con 0x7fdc880a5fe0 2024-09-17T18:35:28.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.585+0000 7fdc905e2640 10 monclient: get_monmap_and_config success 2024-09-17T18:35:28.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.585+0000 7fdc8d37c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:28.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.585+0000 7fdc905e2640 4 set_mon_vals no callback set 2024-09-17T18:35:28.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.585+0000 7fdc8d37c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:35:28.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.585+0000 7fdc8d37c640 10 monclient: dump: 2024-09-17T18:35:28.600 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:28.600 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:28.600 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:28.600 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:28.600 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:28.600 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:28.600 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:28.600 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:28.600 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:28.600 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:28.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.585+0000 7fdc905e2640 10 monclient: shutdown 2024-09-17T18:35:28.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.585+0000 7fdc905e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:35:28.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.585+0000 7fdc905e2640 1 -- 172.21.15.121:0/251393516 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc880a5fe0 msgr2=0x7fdc880a63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:28.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.585+0000 7fdc905e2640 1 --2- 172.21.15.121:0/251393516 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc880a5fe0 0x7fdc880a63f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdc840097a0 tx=0x7fdc84029fd0 comp rx=0 tx=0).stop 2024-09-17T18:35:28.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.586+0000 7fdc905e2640 1 -- 172.21.15.121:0/251393516 shutdown_connections 2024-09-17T18:35:28.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.586+0000 7fdc905e2640 1 --2- 172.21.15.121:0/251393516 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc880a5fe0 0x7fdc880a63f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:28.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.586+0000 7fdc905e2640 1 -- 172.21.15.121:0/251393516 >> 172.21.15.121:0/251393516 conn(0x7fdc880ab9b0 msgr2=0x7fdc880a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:35:28.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.586+0000 7fdc905e2640 1 -- 172.21.15.121:0/251393516 shutdown_connections 2024-09-17T18:35:28.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.586+0000 7fdc905e2640 1 -- 172.21.15.121:0/251393516 wait complete. 2024-09-17T18:35:28.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.586+0000 7fdc905e2640 10 monclient: build_initial_monmap 2024-09-17T18:35:28.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.586+0000 7fdc905e2640 10 monclient: monmap: 2024-09-17T18:35:28.601 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:35:28.601 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:35:28.601 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:35:28.601 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:35:28.601 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:35:28.601 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:28.601 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:35:28.602 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:35:28.602 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:35:28.602 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:28.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.586+0000 7fdc905e2640 1 librados: starting msgr at 2024-09-17T18:35:28.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.586+0000 7fdc905e2640 1 librados: starting objecter 2024-09-17T18:35:28.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.586+0000 7fdc905e2640 1 Processor -- start 2024-09-17T18:35:28.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc905e2640 1 -- start start 2024-09-17T18:35:28.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc905e2640 1 librados: setting wanted keys 2024-09-17T18:35:28.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc905e2640 1 librados: calling monclient init 2024-09-17T18:35:28.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc905e2640 10 monclient: init 2024-09-17T18:35:28.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc905e2640 10 monclient: _reopen_session rank -1 2024-09-17T18:35:28.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc905e2640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T18:35:28.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc905e2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc880a5c00 0x7fdc8813cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:28.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc905e2640 10 monclient(hunting): picked mon.noname-b con 0x7fdc880a5c00 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:35:28.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc905e2640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc880a5fe0 0x7fdc881cde40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:28.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc905e2640 10 monclient(hunting): picked mon.noname-a con 0x7fdc880a5fe0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:35:28.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc905e2640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc8813dc00 0x7fdc8813a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:28.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc905e2640 10 monclient(hunting): picked mon.noname-c con 0x7fdc8813dc00 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:35:28.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc905e2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:28.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc905e2640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fdc88070550 con 0x7fdc880a5fe0 2024-09-17T18:35:28.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc905e2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:28.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc905e2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdc880703d0 con 0x7fdc880a5c00 2024-09-17T18:35:28.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc905e2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:28.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc905e2640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fdc88141710 con 0x7fdc8813dc00 2024-09-17T18:35:28.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc905e2640 10 monclient(hunting): _renew_subs 2024-09-17T18:35:28.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc905e2640 10 monclient(hunting): authenticate will time out at 1732.304621s 2024-09-17T18:35:28.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc8db7d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc880a5fe0 0x7fdc881cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:28.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc8eb7f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc8813dc00 0x7fdc8813a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:28.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc8db7d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc880a5fe0 0x7fdc881cde40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54396/0 (socket says 172.21.15.121:54396) 2024-09-17T18:35:28.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc8db7d640 1 -- 172.21.15.121:0/1883119035 learned_addr learned my addr 172.21.15.121:0/1883119035 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:35:28.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc8db7d640 10 monclient(hunting): get_auth_request con 0x7fdc880a5fe0 auth_method 0 2024-09-17T18:35:28.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc8db7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:28.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc8db7d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:28.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc8db7d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:28.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc8eb7f640 10 monclient(hunting): get_auth_request con 0x7fdc8813dc00 auth_method 0 2024-09-17T18:35:28.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc8eb7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:28.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc8eb7f640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:28.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc8eb7f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:28.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc8e37e640 1 --2- 172.21.15.121:0/1883119035 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc880a5c00 0x7fdc8813cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:28.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc8db7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:28.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc8db7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:28.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc8db7d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:28.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc8e37e640 10 monclient(hunting): get_auth_request con 0x7fdc880a5c00 auth_method 0 2024-09-17T18:35:28.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc8e37e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:28.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc8e37e640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:28.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc8e37e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:28.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc8eb7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:28.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc8eb7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:28.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc8eb7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:28.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc8e37e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:28.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc8e37e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:28.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.587+0000 7fdc8e37e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:28.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.588+0000 7fdc8db7d640 10 monclient(hunting): handle_auth_done global_id 104127 payload 824 2024-09-17T18:35:28.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.588+0000 7fdc8db7d640 1 -- 172.21.15.121:0/1883119035 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc8813dc00 msgr2=0x7fdc8813a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:28.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.588+0000 7fdc8db7d640 1 --2- 172.21.15.121:0/1883119035 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc8813dc00 0x7fdc8813a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:28.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.588+0000 7fdc8db7d640 1 -- 172.21.15.121:0/1883119035 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc880a5c00 msgr2=0x7fdc8813cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:28.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.588+0000 7fdc8db7d640 1 --2- 172.21.15.121:0/1883119035 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc880a5c00 0x7fdc8813cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:28.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.588+0000 7fdc8db7d640 10 monclient: _finish_hunting 0 2024-09-17T18:35:28.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.588+0000 7fdc8db7d640 1 monclient: found mon.noname-a 2024-09-17T18:35:28.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.588+0000 7fdc8db7d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:35:28.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.588+0000 7fdc8db7d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:35:28.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.588+0000 7fdc8db7d640 1 -- 172.21.15.121:0/1883119035 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc8813d780 con 0x7fdc880a5fe0 2024-09-17T18:35:28.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.588+0000 7fdc8eb7f640 1 --2- 172.21.15.121:0/1883119035 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc8813dc00 0x7fdc8813a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:35:28.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.590+0000 7fdc8db7d640 1 -- 172.21.15.121:0/1883119035 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc880a5fe0 msgr2=0x7fdc881cde40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:28.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.590+0000 7fdc8db7d640 1 --2- 172.21.15.121:0/1883119035 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc880a5fe0 0x7fdc881cde40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fdc840097a0 tx=0x7fdc840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:28.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.590+0000 7fdc7affd640 10 client.?.objecter ms_handle_connect 0x7fdc880a5fe0 2024-09-17T18:35:28.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.590+0000 7fdc7affd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:35:28.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.590+0000 7fdc7affd640 1 -- 172.21.15.121:0/1883119035 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc84041070 con 0x7fdc880a5fe0 2024-09-17T18:35:28.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.590+0000 7fdc7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:28.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.590+0000 7fdc7affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:35:28.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.590+0000 7fdc7affd640 10 monclient: dump: 2024-09-17T18:35:28.606 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:28.606 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:28.606 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:28.606 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:28.606 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:28.606 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:28.606 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:28.607 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:28.607 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:28.607 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:28.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.590+0000 7fdc7affd640 10 monclient: _finish_auth 0 2024-09-17T18:35:28.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.590+0000 7fdc7affd640 10 monclient: _check_auth_tickets 2024-09-17T18:35:28.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.590+0000 7fdc7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:35:28.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.590+0000 7fdc7affd640 1 -- 172.21.15.121:0/1883119035 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc8403c880 con 0x7fdc880a5fe0 2024-09-17T18:35:28.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.590+0000 7fdc905e2640 5 monclient: authenticate success, global_id 104127 2024-09-17T18:35:28.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.590+0000 7fdc905e2640 10 monclient: _renew_subs 2024-09-17T18:35:28.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.590+0000 7fdc905e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:35:28.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.590+0000 7fdc905e2640 1 -- 172.21.15.121:0/1883119035 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc8813c290 con 0x7fdc880a5fe0 2024-09-17T18:35:28.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.590+0000 7fdc7affd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:35:28.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.590+0000 7fdc7affd640 1 -- 172.21.15.121:0/1883119035 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc84037040 con 0x7fdc880a5fe0 2024-09-17T18:35:28.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.590+0000 7fdc7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:28.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.590+0000 7fdc7affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:35:28.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.590+0000 7fdc7affd640 10 monclient: dump: 2024-09-17T18:35:28.608 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:28.608 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:28.608 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:28.608 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:28.608 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:28.608 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:28.608 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:28.608 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:28.608 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:28.608 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:28.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.590+0000 7fdc905e2640 10 client.104127.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:35:28.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.590+0000 7fdc905e2640 10 monclient: _renew_subs 2024-09-17T18:35:28.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.590+0000 7fdc905e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:35:28.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.590+0000 7fdc905e2640 1 -- 172.21.15.121:0/1883119035 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc881c66c0 con 0x7fdc880a5fe0 2024-09-17T18:35:28.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.590+0000 7fdc8cb7b640 4 set_mon_vals no callback set 2024-09-17T18:35:28.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.590+0000 7fdc905e2640 1 librados: init done 2024-09-17T18:35:28.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.591+0000 7fdc905e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:28.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.591+0000 7fdc905e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:28.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.591+0000 7fdc905e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:35:28.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.591+0000 7fdc905e2640 1 -- 172.21.15.121:0/1883119035 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc880a63f0 con 0x7fdc880a5fe0 2024-09-17T18:35:28.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.591+0000 7fdc7affd640 1 -- 172.21.15.121:0/1883119035 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fdc84044070 con 0x7fdc880a5fe0 2024-09-17T18:35:28.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.591+0000 7fdc7affd640 1 --2- 172.21.15.121:0/1883119035 >> v2:172.21.15.173:6812/2423786037 conn(0x7fdc6c040170 0x7fdc6c042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:28.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.591+0000 7fdc7affd640 1 -- 172.21.15.121:0/1883119035 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(646..646 src has 247..646) ==== 5925+0+0 (secure 0 0 0) 0x7fdc84074320 con 0x7fdc880a5fe0 2024-09-17T18:35:28.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.591+0000 7fdc7affd640 10 client.104127.objecter ms_dispatch 0x7fdc88000b90 osd_map(646..646 src has 247..646) 2024-09-17T18:35:28.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.591+0000 7fdc7affd640 3 client.104127.objecter handle_osd_map got epochs [646,646] > 0 2024-09-17T18:35:28.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.591+0000 7fdc7affd640 3 client.104127.objecter handle_osd_map decoding full epoch 646 2024-09-17T18:35:28.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.591+0000 7fdc7affd640 20 client.104127.objecter dump_active .. 0 homeless 2024-09-17T18:35:28.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.592+0000 7fdc8e37e640 1 --2- 172.21.15.121:0/1883119035 >> v2:172.21.15.173:6812/2423786037 conn(0x7fdc6c040170 0x7fdc6c042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:28.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.592+0000 7fdc8e37e640 10 monclient: get_auth_request con 0x7fdc6c040170 auth_method 0 2024-09-17T18:35:28.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.594+0000 7fdc7affd640 10 client.104127.objecter ms_handle_connect 0x7fdc6c040170 2024-09-17T18:35:28.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.594+0000 7fdc8e37e640 1 --2- 172.21.15.121:0/1883119035 >> v2:172.21.15.173:6812/2423786037 conn(0x7fdc6c040170 0x7fdc6c042630 secure :-1 s=READY pgs=4075 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c004f60 tx=0x7fdc7c007e30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:28.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.596+0000 7fdc7affd640 1 -- 172.21.15.121:0/1883119035 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fdc8403d3f0 con 0x7fdc880a5fe0 2024-09-17T18:35:28.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.596+0000 7fdc7affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:28.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.596+0000 7fdc7affd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:35:28.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.839+0000 7fdc905e2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:35:28.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.839+0000 7fdc905e2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:35:28.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.839+0000 7fdc905e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:35:28.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.839+0000 7fdc905e2640 1 -- 172.21.15.121:0/1883119035 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fdc8806cd40 con 0x7fdc880a5fe0 2024-09-17T18:35:28.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.839+0000 7fdc7affd640 1 -- 172.21.15.121:0/1883119035 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fdc840740d0 con 0x7fdc880a5fe0 2024-09-17T18:35:28.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.839+0000 7fdc7affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:35:28.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.839+0000 7fdc7affd640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:35:28.843 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":60,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:35:28.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.841+0000 7fdc905e2640 10 librados: watch_flush enter 2024-09-17T18:35:28.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.841+0000 7fdc905e2640 10 librados: watch_flush exit 2024-09-17T18:35:28.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.841+0000 7fdc905e2640 20 client.104127.objecter shutdown clearing up homeless session... 2024-09-17T18:35:28.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.841+0000 7fdc905e2640 10 client.104127.objecter successfully canceled tick 2024-09-17T18:35:28.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.841+0000 7fdc905e2640 1 -- 172.21.15.121:0/1883119035 >> v2:172.21.15.173:6812/2423786037 conn(0x7fdc6c040170 msgr2=0x7fdc6c042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:28.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.841+0000 7fdc905e2640 1 --2- 172.21.15.121:0/1883119035 >> v2:172.21.15.173:6812/2423786037 conn(0x7fdc6c040170 0x7fdc6c042630 secure :-1 s=READY pgs=4075 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c004f60 tx=0x7fdc7c007e30 comp rx=0 tx=0).stop 2024-09-17T18:35:28.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.841+0000 7fdc905e2640 10 monclient: shutdown 2024-09-17T18:35:28.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.841+0000 7fdc905e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:35:28.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.841+0000 7fdc905e2640 1 -- 172.21.15.121:0/1883119035 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc880a5fe0 msgr2=0x7fdc881cde40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:28.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.841+0000 7fdc905e2640 1 --2- 172.21.15.121:0/1883119035 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc880a5fe0 0x7fdc881cde40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fdc840097a0 tx=0x7fdc840046c0 comp rx=0 tx=0).stop 2024-09-17T18:35:28.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.841+0000 7fdc905e2640 1 -- 172.21.15.121:0/1883119035 shutdown_connections 2024-09-17T18:35:28.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.841+0000 7fdc905e2640 1 --2- 172.21.15.121:0/1883119035 >> v2:172.21.15.173:6812/2423786037 conn(0x7fdc6c040170 0x7fdc6c042630 unknown :-1 s=CLOSED pgs=4075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:28.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.841+0000 7fdc905e2640 1 --2- 172.21.15.121:0/1883119035 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc8813dc00 0x7fdc8813a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:28.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.841+0000 7fdc905e2640 1 --2- 172.21.15.121:0/1883119035 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc880a5fe0 0x7fdc881cde40 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:28.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.841+0000 7fdc905e2640 1 --2- 172.21.15.121:0/1883119035 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc880a5c00 0x7fdc8813cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:28.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.841+0000 7fdc905e2640 1 -- 172.21.15.121:0/1883119035 >> 172.21.15.121:0/1883119035 conn(0x7fdc880ab9b0 msgr2=0x7fdc880a4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:35:28.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.841+0000 7fdc905e2640 1 -- 172.21.15.121:0/1883119035 shutdown_connections 2024-09-17T18:35:28.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.841+0000 7fdc905e2640 1 -- 172.21.15.121:0/1883119035 wait complete. 2024-09-17T18:35:28.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.841+0000 7fdc905e2640 1 librados: shutdown 2024-09-17T18:35:28.854 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T18:35:28.854 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-17T18:35:28.854 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T18:35:28.855 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T18:35:28.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.949+0000 7ffbdacef640 10 monclient: get_monmap_and_config 2024-09-17T18:35:28.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.949+0000 7ffbdacef640 10 monclient: build_initial_monmap 2024-09-17T18:35:28.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.949+0000 7ffbdacef640 10 monclient: monmap: 2024-09-17T18:35:28.952 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:35:28.952 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:28.953 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:35:28.951205+0000 2024-09-17T18:35:28.953 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:35:28.951205+0000 2024-09-17T18:35:28.953 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:35:28.953 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:28.953 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:35:28.953 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:35:28.953 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:35:28.953 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:35:28.953 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:35:28.953 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:28.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.950+0000 7ffbdacef640 1 Processor -- start 2024-09-17T18:35:28.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.950+0000 7ffbdacef640 1 -- start start 2024-09-17T18:35:28.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.950+0000 7ffbdacef640 10 monclient: init 2024-09-17T18:35:28.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.950+0000 7ffbdacef640 10 monclient: _reopen_session rank -1 2024-09-17T18:35:28.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.950+0000 7ffbdacef640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-17T18:35:28.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.950+0000 7ffbdacef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffbd40a0a00 addr v1:172.21.15.173:6789/0 2024-09-17T18:35:28.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.950+0000 7ffbdacef640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ffbd4137c90 0x7ffbd413a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:28.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.950+0000 7ffbdacef640 10 monclient(hunting): picked mon.noname-a con 0x7ffbd4137c90 addr v2:172.21.15.121:3300/0 2024-09-17T18:35:28.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.950+0000 7ffbdacef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffbd413a5c0 addr v1:172.21.15.121:6789/0 2024-09-17T18:35:28.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.950+0000 7ffbdacef640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbd40a3820 con 0x7ffbd413a5c0 2024-09-17T18:35:28.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.950+0000 7ffbdacef640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbd40a2ea0 con 0x7ffbd40a0a00 2024-09-17T18:35:28.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.950+0000 7ffbdacef640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:28.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.950+0000 7ffbdacef640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7ffbd409e800 con 0x7ffbd4137c90 2024-09-17T18:35:28.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.950+0000 7ffbdacef640 10 monclient(hunting): _renew_subs 2024-09-17T18:35:28.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.950+0000 7ffbdacef640 10 monclient(hunting): authenticate will time out at 1732.667917s 2024-09-17T18:35:28.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.951+0000 7ffbcbfff640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ffbd4137c90 0x7ffbd413a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:28.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.951+0000 7ffbcbfff640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ffbd4137c90 0x7ffbd413a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54410/0 (socket says 172.21.15.121:54410) 2024-09-17T18:35:28.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.951+0000 7ffbd928c640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7ffbd413a5c0 0x7ffbd413c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:43056/0 (socket says 172.21.15.121:43056) 2024-09-17T18:35:28.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.951+0000 7ffbcbfff640 1 -- 172.21.15.121:0/3514110915 learned_addr learned my addr 172.21.15.121:0/3514110915 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:35:28.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.951+0000 7ffbcbfff640 10 monclient(hunting): get_auth_request con 0x7ffbd4137c90 auth_method 0 2024-09-17T18:35:28.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.951+0000 7ffbcbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:28.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.951+0000 7ffbcbfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:28.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.951+0000 7ffbcbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:28.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.951+0000 7ffbcbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:28.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.951+0000 7ffbcbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:28.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.951+0000 7ffbcbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:28.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.951+0000 7ffbcbfff640 10 monclient(hunting): handle_auth_done global_id 104130 payload 274 2024-09-17T18:35:28.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.951+0000 7ffbcbfff640 1 -- 172.21.15.121:0/3514110915 >> v1:172.21.15.173:6789/0 conn(0x7ffbd40a0a00 legacy=0x7ffbd40a0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:35:28.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.951+0000 7ffbcbfff640 1 -- 172.21.15.121:0/3514110915 >> v1:172.21.15.121:6789/0 conn(0x7ffbd413a5c0 legacy=0x7ffbd413c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:35:28.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.951+0000 7ffbcbfff640 10 monclient: _finish_hunting 0 2024-09-17T18:35:28.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.951+0000 7ffbcbfff640 1 monclient: found mon.noname-a 2024-09-17T18:35:28.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.951+0000 7ffbcbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:35:28.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.951+0000 7ffbcbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:35:28.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.951+0000 7ffbcbfff640 1 -- 172.21.15.121:0/3514110915 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbd40a2c30 con 0x7ffbd4137c90 2024-09-17T18:35:28.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.954+0000 7ffbcbfff640 1 -- 172.21.15.121:0/3514110915 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffbd4137c90 msgr2=0x7ffbd413a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:28.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.954+0000 7ffbcbfff640 1 --2- 172.21.15.121:0/3514110915 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffbd4137c90 0x7ffbd413a080 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ffbc40098a0 tx=0x7ffbc402a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=815b76acb22ce398 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:28.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.954+0000 7ffbcb7fe640 1 -- 172.21.15.121:0/3514110915 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc4037070 con 0x7ffbd4137c90 2024-09-17T18:35:28.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.954+0000 7ffbcb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:28.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.954+0000 7ffbcb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:35:28.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.954+0000 7ffbcb7fe640 10 monclient: dump: 2024-09-17T18:35:28.964 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:28.964 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:28.964 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:28.964 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:28.964 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:28.964 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:28.965 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:28.965 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:28.965 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:28.965 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:28.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.954+0000 7ffbcb7fe640 10 monclient: _finish_auth 0 2024-09-17T18:35:28.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.954+0000 7ffbcb7fe640 10 monclient: _check_auth_tickets 2024-09-17T18:35:28.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.954+0000 7ffbcb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:35:28.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.954+0000 7ffbcb7fe640 1 -- 172.21.15.121:0/3514110915 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbc4032640 con 0x7ffbd4137c90 2024-09-17T18:35:28.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.954+0000 7ffbdacef640 5 monclient: authenticate success, global_id 104130 2024-09-17T18:35:28.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.954+0000 7ffbdacef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:35:28.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.954+0000 7ffbcb7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:35:28.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.954+0000 7ffbdacef640 10 monclient: get_monmap_and_config success 2024-09-17T18:35:28.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.954+0000 7ffbcb7fe640 1 -- 172.21.15.121:0/3514110915 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc4032a10 con 0x7ffbd4137c90 2024-09-17T18:35:28.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.954+0000 7ffbdacef640 4 set_mon_vals no callback set 2024-09-17T18:35:28.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.954+0000 7ffbcb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:28.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.954+0000 7ffbcb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:35:28.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.954+0000 7ffbcb7fe640 10 monclient: dump: 2024-09-17T18:35:28.966 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:28.966 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:28.966 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:28.966 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:28.966 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:28.966 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:28.966 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:28.966 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:28.966 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:28.966 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:28.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.954+0000 7ffbdacef640 10 monclient: shutdown 2024-09-17T18:35:28.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.955+0000 7ffbdacef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:35:28.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.955+0000 7ffbdacef640 1 -- 172.21.15.121:0/3514110915 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffbd4137c90 msgr2=0x7ffbd413a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:28.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.955+0000 7ffbdacef640 1 --2- 172.21.15.121:0/3514110915 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffbd4137c90 0x7ffbd413a080 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ffbc40098a0 tx=0x7ffbc402a0d0 comp rx=0 tx=0).stop 2024-09-17T18:35:28.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.955+0000 7ffbdacef640 1 -- 172.21.15.121:0/3514110915 shutdown_connections 2024-09-17T18:35:28.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.955+0000 7ffbdacef640 1 --2- 172.21.15.121:0/3514110915 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffbd4137c90 0x7ffbd413a080 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:28.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.955+0000 7ffbdacef640 1 -- 172.21.15.121:0/3514110915 >> 172.21.15.121:0/3514110915 conn(0x7ffbd40a7830 msgr2=0x7ffbd40a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:35:28.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.955+0000 7ffbdacef640 1 -- 172.21.15.121:0/3514110915 shutdown_connections 2024-09-17T18:35:28.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.955+0000 7ffbdacef640 1 -- 172.21.15.121:0/3514110915 wait complete. 2024-09-17T18:35:28.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.955+0000 7ffbdacef640 10 monclient: build_initial_monmap 2024-09-17T18:35:28.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.955+0000 7ffbdacef640 10 monclient: monmap: 2024-09-17T18:35:28.967 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:35:28.967 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:35:28.967 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:35:28.968 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:35:28.968 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:35:28.968 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:28.968 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:35:28.968 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:35:28.968 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:35:28.968 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:28.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.955+0000 7ffbdacef640 1 librados: starting msgr at 2024-09-17T18:35:28.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.955+0000 7ffbdacef640 1 librados: starting objecter 2024-09-17T18:35:28.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.955+0000 7ffbdacef640 1 Processor -- start 2024-09-17T18:35:28.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.955+0000 7ffbdacef640 1 -- start start 2024-09-17T18:35:28.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.955+0000 7ffbdacef640 1 librados: setting wanted keys 2024-09-17T18:35:28.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.955+0000 7ffbdacef640 1 librados: calling monclient init 2024-09-17T18:35:28.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.955+0000 7ffbdacef640 10 monclient: init 2024-09-17T18:35:28.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbdacef640 10 monclient: _reopen_session rank -1 2024-09-17T18:35:28.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbdacef640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T18:35:28.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbdacef640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffbd40a0a00 0x7ffbd41cba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:28.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbdacef640 10 monclient(hunting): picked mon.noname-b con 0x7ffbd40a0a00 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:35:28.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbdacef640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffbd4137c90 0x7ffbd41cbf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:28.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbdacef640 10 monclient(hunting): picked mon.noname-c con 0x7ffbd4137c90 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:35:28.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbdacef640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffbd413a5c0 0x7ffbd411f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:28.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbdacef640 10 monclient(hunting): picked mon.noname-a con 0x7ffbd413a5c0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:35:28.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbdacef640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:28.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbdacef640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd409de60 con 0x7ffbd413a5c0 2024-09-17T18:35:28.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbdacef640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:28.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbdacef640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd409dce0 con 0x7ffbd40a0a00 2024-09-17T18:35:28.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbdacef640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:28.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbdacef640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ffbd40a3120 con 0x7ffbd4137c90 2024-09-17T18:35:28.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbdacef640 10 monclient(hunting): _renew_subs 2024-09-17T18:35:28.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbdacef640 10 monclient(hunting): authenticate will time out at 1732.673572s 2024-09-17T18:35:28.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbd928c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffbd413a5c0 0x7ffbd411f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:28.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbcbfff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffbd4137c90 0x7ffbd41cbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:28.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbcbfff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffbd4137c90 0x7ffbd41cbf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:53510/0 (socket says 172.21.15.121:53510) 2024-09-17T18:35:28.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbd928c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffbd413a5c0 0x7ffbd411f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54418/0 (socket says 172.21.15.121:54418) 2024-09-17T18:35:28.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbcbfff640 1 -- 172.21.15.121:0/290418684 learned_addr learned my addr 172.21.15.121:0/290418684 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:35:28.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbcbfff640 10 monclient(hunting): get_auth_request con 0x7ffbd4137c90 auth_method 0 2024-09-17T18:35:28.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbcbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:28.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbcbfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:28.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbcbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:28.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbd928c640 10 monclient(hunting): get_auth_request con 0x7ffbd413a5c0 auth_method 0 2024-09-17T18:35:28.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbd928c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:28.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbd928c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:28.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbd928c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:28.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbcbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:28.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbcbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:28.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbd8a8b640 1 --2- 172.21.15.121:0/290418684 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffbd40a0a00 0x7ffbd41cba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:28.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbcbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:28.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbd928c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:28.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbd928c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:28.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbd928c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:28.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbd8a8b640 10 monclient(hunting): get_auth_request con 0x7ffbd40a0a00 auth_method 0 2024-09-17T18:35:28.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbd8a8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:28.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbd8a8b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:28.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbd8a8b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:28.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbd8a8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:28.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbd8a8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:28.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbd8a8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:28.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbd928c640 10 monclient(hunting): handle_auth_done global_id 104133 payload 824 2024-09-17T18:35:28.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbd928c640 1 -- 172.21.15.121:0/290418684 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffbd4137c90 msgr2=0x7ffbd41cbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:28.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.956+0000 7ffbd928c640 1 --2- 172.21.15.121:0/290418684 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffbd4137c90 0x7ffbd41cbf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:28.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.957+0000 7ffbd928c640 1 -- 172.21.15.121:0/290418684 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffbd40a0a00 msgr2=0x7ffbd41cba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:28.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.957+0000 7ffbd928c640 1 --2- 172.21.15.121:0/290418684 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffbd40a0a00 0x7ffbd41cba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:28.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.957+0000 7ffbd928c640 10 monclient: _finish_hunting 0 2024-09-17T18:35:28.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.957+0000 7ffbd928c640 1 monclient: found mon.noname-a 2024-09-17T18:35:28.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.957+0000 7ffbd928c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:35:28.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.957+0000 7ffbd928c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:35:28.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.957+0000 7ffbd928c640 1 -- 172.21.15.121:0/290418684 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbd41353d0 con 0x7ffbd413a5c0 2024-09-17T18:35:28.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.957+0000 7ffbcbfff640 1 --2- 172.21.15.121:0/290418684 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffbd4137c90 0x7ffbd41cbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:35:28.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.959+0000 7ffbd928c640 1 -- 172.21.15.121:0/290418684 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffbd413a5c0 msgr2=0x7ffbd411f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:28.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.959+0000 7ffbd928c640 1 --2- 172.21.15.121:0/290418684 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffbd413a5c0 0x7ffbd411f010 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc00c780 tx=0x7ffbcc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:28.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.959+0000 7ffbc97fa640 10 client.?.objecter ms_handle_connect 0x7ffbd413a5c0 2024-09-17T18:35:28.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.959+0000 7ffbc97fa640 10 client.?.objecter resend_mon_ops 2024-09-17T18:35:28.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.959+0000 7ffbc97fa640 1 -- 172.21.15.121:0/290418684 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbcc017070 con 0x7ffbd413a5c0 2024-09-17T18:35:28.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.959+0000 7ffbc97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:28.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.959+0000 7ffbc97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:35:28.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.959+0000 7ffbc97fa640 10 monclient: dump: 2024-09-17T18:35:28.973 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:28.973 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:28.973 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:28.973 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:28.973 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:28.973 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:28.973 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:28.974 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:28.974 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:28.974 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:28.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.959+0000 7ffbc97fa640 10 monclient: _finish_auth 0 2024-09-17T18:35:28.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.959+0000 7ffbc97fa640 10 monclient: _check_auth_tickets 2024-09-17T18:35:28.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.959+0000 7ffbc97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:35:28.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.959+0000 7ffbc97fa640 1 -- 172.21.15.121:0/290418684 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbcc00a840 con 0x7ffbd413a5c0 2024-09-17T18:35:28.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.959+0000 7ffbdacef640 5 monclient: authenticate success, global_id 104133 2024-09-17T18:35:28.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.959+0000 7ffbdacef640 10 monclient: _renew_subs 2024-09-17T18:35:28.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.959+0000 7ffbdacef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:35:28.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.959+0000 7ffbdacef640 1 -- 172.21.15.121:0/290418684 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbd409e240 con 0x7ffbd413a5c0 2024-09-17T18:35:28.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.959+0000 7ffbc97fa640 10 monclient: handle_config config(0 keys) 2024-09-17T18:35:28.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.959+0000 7ffbc97fa640 1 -- 172.21.15.121:0/290418684 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbcc00ac30 con 0x7ffbd413a5c0 2024-09-17T18:35:28.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.959+0000 7ffbc97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:28.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.959+0000 7ffbc97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:35:28.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.959+0000 7ffbc97fa640 10 monclient: dump: 2024-09-17T18:35:28.975 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:28.975 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:28.975 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:28.975 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:28.975 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:28.975 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:28.975 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:28.975 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:28.975 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:28.975 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:28.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.959+0000 7ffbdacef640 10 client.104133.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:35:28.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.959+0000 7ffbdacef640 10 monclient: _renew_subs 2024-09-17T18:35:28.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.959+0000 7ffbcaffd640 4 set_mon_vals no callback set 2024-09-17T18:35:28.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.959+0000 7ffbdacef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:35:28.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.959+0000 7ffbdacef640 1 -- 172.21.15.121:0/290418684 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbd4122de0 con 0x7ffbd413a5c0 2024-09-17T18:35:28.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.959+0000 7ffbdacef640 1 librados: init done 2024-09-17T18:35:28.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.960+0000 7ffbdacef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:28.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.960+0000 7ffbdacef640 1 --2- 172.21.15.121:0/290418684 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffbd40712d0 0x7ffbd40716b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:28.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.960+0000 7ffbdacef640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:28.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.960+0000 7ffbdacef640 1 -- 172.21.15.121:0/290418684 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd40a5cf0 con 0x7ffbd40712d0 2024-09-17T18:35:28.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.960+0000 7ffbd8a8b640 1 --2- 172.21.15.121:0/290418684 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffbd40712d0 0x7ffbd40716b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:28.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.960+0000 7ffbd8a8b640 10 monclient: get_auth_request con 0x7ffbd40712d0 auth_method 0 2024-09-17T18:35:28.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.960+0000 7ffbd8a8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:28.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.960+0000 7ffbd8a8b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:28.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.960+0000 7ffbd8a8b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:28.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.960+0000 7ffbd8a8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:28.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.960+0000 7ffbd8a8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:28.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.960+0000 7ffbc97fa640 1 -- 172.21.15.121:0/290418684 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7ffbcc01a070 con 0x7ffbd413a5c0 2024-09-17T18:35:28.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.960+0000 7ffbd8a8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:28.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.960+0000 7ffbd8a8b640 10 monclient(hunting): handle_auth_done global_id 104136 payload 824 2024-09-17T18:35:28.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.960+0000 7ffbd8a8b640 1 -- 172.21.15.121:0/290418684 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ffbd40a1fa0 con 0x7ffbd40712d0 2024-09-17T18:35:28.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.960+0000 7ffbc97fa640 1 --2- 172.21.15.121:0/290418684 >> v2:172.21.15.173:6812/2423786037 conn(0x7ffbc00401a0 0x7ffbc0042660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:28.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.960+0000 7ffbc97fa640 1 -- 172.21.15.121:0/290418684 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(646..646 src has 247..646) ==== 5925+0+0 (secure 0 0 0) 0x7ffbcc052f10 con 0x7ffbd413a5c0 2024-09-17T18:35:28.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.960+0000 7ffbc97fa640 10 client.104133.objecter ms_dispatch 0x7ffbd4000b90 osd_map(646..646 src has 247..646) 2024-09-17T18:35:28.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.960+0000 7ffbc97fa640 3 client.104133.objecter handle_osd_map got epochs [646,646] > 0 2024-09-17T18:35:28.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.960+0000 7ffbc97fa640 3 client.104133.objecter handle_osd_map decoding full epoch 646 2024-09-17T18:35:28.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.960+0000 7ffbc97fa640 20 client.104133.objecter dump_active .. 0 homeless 2024-09-17T18:35:28.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.961+0000 7ffbcbfff640 1 --2- 172.21.15.121:0/290418684 >> v2:172.21.15.173:6812/2423786037 conn(0x7ffbc00401a0 0x7ffbc0042660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:28.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.961+0000 7ffbcbfff640 10 monclient: get_auth_request con 0x7ffbc00401a0 auth_method 0 2024-09-17T18:35:28.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.962+0000 7ffbc97fa640 10 client.104133.objecter ms_handle_connect 0x7ffbd40712d0 2024-09-17T18:35:28.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.962+0000 7ffbc97fa640 10 client.104133.objecter resend_mon_ops 2024-09-17T18:35:28.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.962+0000 7ffbd8a8b640 1 -- 172.21.15.121:0/290418684 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffbd40712d0 msgr2=0x7ffbd40716b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:28.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.962+0000 7ffbd8a8b640 1 --2- 172.21.15.121:0/290418684 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffbd40712d0 0x7ffbd40716b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ffbd000f7a0 tx=0x7ffbd000fc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:28.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.962+0000 7ffbc97fa640 1 -- 172.21.15.121:0/290418684 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbd0018030 con 0x7ffbd40712d0 2024-09-17T18:35:28.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.962+0000 7ffbc97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:28.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.962+0000 7ffbc97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:35:28.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.962+0000 7ffbc97fa640 10 monclient: dump: 2024-09-17T18:35:28.981 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:28.981 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:28.981 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:28.981 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:28.981 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:28.981 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:28.981 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:28.981 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:28.981 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:28.981 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:28.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.963+0000 7ffbc97fa640 1 -- 172.21.15.121:0/290418684 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ffbd000d9b0 con 0x7ffbd40712d0 2024-09-17T18:35:28.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.963+0000 7ffbc97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:28.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.963+0000 7ffbc97fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:35:28.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.963+0000 7ffbc97fa640 1 -- 172.21.15.121:0/290418684 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffbd40712d0 msgr2=0x7ffbd40716b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:28.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.963+0000 7ffbc97fa640 1 --2- 172.21.15.121:0/290418684 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffbd40712d0 0x7ffbd40716b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ffbd000f7a0 tx=0x7ffbd000fc60 comp rx=0 tx=0).stop 2024-09-17T18:35:28.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.963+0000 7ffbc97fa640 1 -- 172.21.15.121:0/290418684 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffbd40712d0 msgr2=0x7ffbd40716b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:35:28.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.963+0000 7ffbc97fa640 1 --2- 172.21.15.121:0/290418684 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffbd40712d0 0x7ffbd40716b0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:28.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.963+0000 7ffbc97fa640 10 client.104133.objecter ms_handle_connect 0x7ffbc00401a0 2024-09-17T18:35:28.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.963+0000 7ffbcbfff640 1 --2- 172.21.15.121:0/290418684 >> v2:172.21.15.173:6812/2423786037 conn(0x7ffbc00401a0 0x7ffbc0042660 secure :-1 s=READY pgs=4076 cs=0 l=1 rev1=1 crypto rx=0x7ffbc4035040 tx=0x7ffbc402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:28.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.969+0000 7ffbdacef640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:35:28.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.969+0000 7ffbdacef640 1 --2- 172.21.15.121:0/290418684 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffbd4071bf0 0x7ffbd411f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:28.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.969+0000 7ffbdacef640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:28.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.969+0000 7ffbdacef640 1 -- 172.21.15.121:0/290418684 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd41215e0 con 0x7ffbd4071bf0 2024-09-17T18:35:28.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.969+0000 7ffbd8a8b640 1 --2- 172.21.15.121:0/290418684 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffbd4071bf0 0x7ffbd411f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:28.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.969+0000 7ffbd8a8b640 10 monclient: get_auth_request con 0x7ffbd4071bf0 auth_method 0 2024-09-17T18:35:28.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.969+0000 7ffbd8a8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:28.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.969+0000 7ffbd8a8b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:28.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.969+0000 7ffbd8a8b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:28.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.969+0000 7ffbd8a8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:28.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.969+0000 7ffbd8a8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:28.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.969+0000 7ffbd8a8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:28.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.969+0000 7ffbd8a8b640 10 monclient(hunting): handle_auth_done global_id 104139 payload 824 2024-09-17T18:35:28.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.969+0000 7ffbd8a8b640 1 -- 172.21.15.121:0/290418684 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ffbd406cd40 con 0x7ffbd4071bf0 2024-09-17T18:35:28.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.971+0000 7ffbd8a8b640 1 -- 172.21.15.121:0/290418684 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffbd4071bf0 msgr2=0x7ffbd411f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:28.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.971+0000 7ffbc97fa640 10 client.104133.objecter ms_handle_connect 0x7ffbd4071bf0 2024-09-17T18:35:28.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.971+0000 7ffbc97fa640 10 client.104133.objecter resend_mon_ops 2024-09-17T18:35:28.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.971+0000 7ffbd8a8b640 1 --2- 172.21.15.121:0/290418684 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffbd4071bf0 0x7ffbd411f550 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ffbd000f7a0 tx=0x7ffbd0003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:28.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.971+0000 7ffbc97fa640 1 -- 172.21.15.121:0/290418684 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbd001dd70 con 0x7ffbd4071bf0 2024-09-17T18:35:28.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.971+0000 7ffbc97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:28.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.971+0000 7ffbc97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:35:28.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.971+0000 7ffbc97fa640 10 monclient: dump: 2024-09-17T18:35:28.984 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:28.984 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:28.984 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:28.984 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:28.984 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:28.984 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:28.984 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:28.984 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:28.984 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:28.984 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:28.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.972+0000 7ffbc97fa640 1 -- 172.21.15.121:0/290418684 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4553 (secure 0 0 0) 0x7ffbd0004000 con 0x7ffbd4071bf0 2024-09-17T18:35:28.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.972+0000 7ffbc97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:35:28.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.972+0000 7ffbc97fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:35:28.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.972+0000 7ffbc97fa640 1 -- 172.21.15.121:0/290418684 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffbd4071bf0 msgr2=0x7ffbd411f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:28.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.972+0000 7ffbc97fa640 1 --2- 172.21.15.121:0/290418684 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffbd4071bf0 0x7ffbd411f550 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ffbd000f7a0 tx=0x7ffbd0003760 comp rx=0 tx=0).stop 2024-09-17T18:35:28.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.972+0000 7ffbc97fa640 1 -- 172.21.15.121:0/290418684 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffbd4071bf0 msgr2=0x7ffbd411f550 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:35:28.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.972+0000 7ffbc97fa640 1 --2- 172.21.15.121:0/290418684 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffbd4071bf0 0x7ffbd411f550 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ffbd000f7a0 tx=0x7ffbd0003760 comp rx=0 tx=0).stop 2024-09-17T18:35:28.985 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:35:28.985 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:35:28.985 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:35:28.985 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "leader", 2024-09-17T18:35:28.985 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 60, 2024-09-17T18:35:28.986 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 9335, 2024-09-17T18:35:28.986 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:35:28.986 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:35:28.986 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:35:28.986 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:35:28.986 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:35:28.986 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 2, 2024-09-17T18:35:28.986 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:35:28.986 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:35:28.986 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:35:28.986 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:35:28.986 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:35:28.986 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:35:28.986 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:35:28.986 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:35:28.987 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:35:28.987 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:35:28.987 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:35:28.987 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:35:28.987 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:35:28.987 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:35:28.987 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:35:28.987 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:35:28.987 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:35:28.987 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:35:28.987 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:35:28.987 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:35:28.987 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:35:28.987 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:35:28.988 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:35:28.988 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:35:28.988 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:35:28.988 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:35:28.988 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:35:28.988 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:35:28.988 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:28.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:28.988 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:35:28.988 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:35:28.988 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:35:28.988 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:35:28.988 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:35:28.988 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:35:28.989 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:35:28.989 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:35:28.989 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:35:28.989 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:35:28.989 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:35:28.989 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:35:28.989 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:35:28.989 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:35:28.989 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:35:28.989 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:35:28.989 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:35:28.989 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:35:28.989 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:35:28.989 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:35:28.990 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:35:28.990 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:35:28.990 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:35:28.990 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:35:28.990 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:35:28.990 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:35:28.990 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:35:28.990 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:35:28.990 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:35:28.990 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:35:28.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:28.990 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:35:28.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:28.990 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:35:28.991 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:35:28.991 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:35:28.991 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:35:28.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:28.991 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:35:28.991 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:35:28.991 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:28.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:28.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:28.991 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:35:28.991 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:35:28.991 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:28.991 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:28.991 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:28.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:28.992 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:35:28.992 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:35:28.992 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:35:28.992 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:35:28.992 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:35:28.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:28.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:28.992 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:35:28.992 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:35:28.992 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:35:28.992 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:35:28.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:28.992 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:35:28.993 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:35:28.993 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:28.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:28.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:28.996 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:35:28.996 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:35:28.996 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:28.996 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:28.996 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:28.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:28.996 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:35:28.996 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:35:28.996 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:35:28.997 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:35:28.997 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:35:28.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:28.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:28.997 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:35:28.997 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:35:28.997 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:35:28.997 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:35:28.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:28.997 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:35:28.997 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:35:28.997 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:28.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:28.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:28.998 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:35:28.998 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:35:28.998 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:28.998 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:28.998 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:28.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:28.998 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:35:28.998 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:35:28.998 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:35:28.998 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:35:28.998 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:35:28.998 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:28.998 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:28.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:28.998 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:35:28.999 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:35:28.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:28.999 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:35:28.999 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:35:28.999 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:35:28.999 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:28.999 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:35:28.999 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:35:28.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:28.999 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:35:28.999 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:35:28.999 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-17T18:35:28.999 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:28.999 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:28.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:29.000 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:35:29.000 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:35:29.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.972+0000 7ffbdacef640 10 librados: watch_flush enter 2024-09-17T18:35:29.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.972+0000 7ffbdacef640 10 librados: watch_flush exit 2024-09-17T18:35:29.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.972+0000 7ffbdacef640 20 client.104133.objecter shutdown clearing up homeless session... 2024-09-17T18:35:29.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.972+0000 7ffbdacef640 10 client.104133.objecter successfully canceled tick 2024-09-17T18:35:29.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.972+0000 7ffbdacef640 1 -- 172.21.15.121:0/290418684 >> v2:172.21.15.173:6812/2423786037 conn(0x7ffbc00401a0 msgr2=0x7ffbc0042660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:29.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.972+0000 7ffbdacef640 1 --2- 172.21.15.121:0/290418684 >> v2:172.21.15.173:6812/2423786037 conn(0x7ffbc00401a0 0x7ffbc0042660 secure :-1 s=READY pgs=4076 cs=0 l=1 rev1=1 crypto rx=0x7ffbc4035040 tx=0x7ffbc402e000 comp rx=0 tx=0).stop 2024-09-17T18:35:29.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.972+0000 7ffbdacef640 10 monclient: shutdown 2024-09-17T18:35:29.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.972+0000 7ffbdacef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:35:29.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.972+0000 7ffbdacef640 1 -- 172.21.15.121:0/290418684 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffbd413a5c0 msgr2=0x7ffbd411f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:29.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.972+0000 7ffbdacef640 1 --2- 172.21.15.121:0/290418684 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffbd413a5c0 0x7ffbd411f010 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc00c780 tx=0x7ffbcc00cc40 comp rx=0 tx=0).stop 2024-09-17T18:35:29.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.972+0000 7ffbd928c640 1 -- 172.21.15.121:0/290418684 reap_dead start 2024-09-17T18:35:29.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.972+0000 7ffbd928c640 1 -- 172.21.15.121:0/290418684 reap_dead start 2024-09-17T18:35:29.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.972+0000 7ffbdacef640 1 -- 172.21.15.121:0/290418684 shutdown_connections 2024-09-17T18:35:29.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.972+0000 7ffbdacef640 1 -- 172.21.15.121:0/290418684 >> 172.21.15.121:0/290418684 conn(0x7ffbd40a7830 msgr2=0x7ffbd4139e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:35:29.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.972+0000 7ffbdacef640 1 -- 172.21.15.121:0/290418684 shutdown_connections 2024-09-17T18:35:29.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.972+0000 7ffbdacef640 1 -- 172.21.15.121:0/290418684 wait complete. 2024-09-17T18:35:29.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:28.972+0000 7ffbdacef640 1 librados: shutdown 2024-09-17T18:35:29.003 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:35:29.003 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T18:35:29.003 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T18:35:29.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.098+0000 7f3dfffec640 10 monclient: get_monmap_and_config 2024-09-17T18:35:29.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.098+0000 7f3dfffec640 10 monclient: build_initial_monmap 2024-09-17T18:35:29.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.098+0000 7f3dfffec640 10 monclient: monmap: 2024-09-17T18:35:29.101 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:35:29.101 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:29.101 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:35:29.099802+0000 2024-09-17T18:35:29.101 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:35:29.099802+0000 2024-09-17T18:35:29.101 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:35:29.101 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:29.101 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:35:29.101 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:35:29.101 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:35:29.101 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:35:29.101 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:35:29.102 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:29.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.098+0000 7f3dfffec640 1 Processor -- start 2024-09-17T18:35:29.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.099+0000 7f3dfffec640 1 -- start start 2024-09-17T18:35:29.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.099+0000 7f3dfffec640 10 monclient: init 2024-09-17T18:35:29.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.099+0000 7f3dfffec640 10 monclient: _reopen_session rank -1 2024-09-17T18:35:29.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.099+0000 7f3dfffec640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-17T18:35:29.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.099+0000 7f3dfffec640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3df80700f0 addr v1:172.21.15.121:6789/0 2024-09-17T18:35:29.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.099+0000 7f3dfffec640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3df809e810 addr v1:172.21.15.173:6789/0 2024-09-17T18:35:29.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.099+0000 7f3dfffec640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3df809ec60 0x7f3df8140e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:29.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.099+0000 7f3dfffec640 10 monclient(hunting): picked mon.noname-b con 0x7f3df809ec60 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:35:29.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.099+0000 7f3dfffec640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3df8135560 con 0x7f3df80700f0 2024-09-17T18:35:29.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.099+0000 7f3dfffec640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3df809ca60 con 0x7f3df809e810 2024-09-17T18:35:29.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.099+0000 7f3dfffec640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:29.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.099+0000 7f3dfffec640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3df80704d0 con 0x7f3df809ec60 2024-09-17T18:35:29.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.099+0000 7f3dfffec640 10 monclient(hunting): _renew_subs 2024-09-17T18:35:29.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.099+0000 7f3dfffec640 10 monclient(hunting): authenticate will time out at 1732.816646s 2024-09-17T18:35:29.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.099+0000 7f3dfdd88640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f3df80700f0 0x7f3df8134e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:57114/0 (socket says 172.21.15.121:57114) 2024-09-17T18:35:29.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.099+0000 7f3dfdd88640 1 -- 172.21.15.121:0/2471695629 learned_addr learned my addr 172.21.15.121:0/2471695629 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:35:29.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.099+0000 7f3dfe589640 1 --2- 172.21.15.121:0/2471695629 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3df809ec60 0x7f3df8140e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:29.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.099+0000 7f3dfe589640 10 monclient(hunting): get_auth_request con 0x7f3df809ec60 auth_method 0 2024-09-17T18:35:29.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.099+0000 7f3dfe589640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:29.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.099+0000 7f3dfe589640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:29.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.099+0000 7f3dfe589640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:29.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.099+0000 7f3dfe589640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:29.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.099+0000 7f3dfe589640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:29.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.099+0000 7f3dfe589640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:29.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.099+0000 7f3dfe589640 10 monclient(hunting): handle_auth_done global_id 64493 payload 274 2024-09-17T18:35:29.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.099+0000 7f3dfe589640 1 -- 172.21.15.121:0/2471695629 >> v1:172.21.15.173:6789/0 conn(0x7f3df809e810 legacy=0x7f3df813d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:35:29.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.099+0000 7f3dfe589640 1 -- 172.21.15.121:0/2471695629 >> v1:172.21.15.121:6789/0 conn(0x7f3df80700f0 legacy=0x7f3df8134e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:35:29.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.099+0000 7f3dfe589640 10 monclient: _finish_hunting 0 2024-09-17T18:35:29.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.099+0000 7f3dfe589640 1 monclient: found mon.noname-b 2024-09-17T18:35:29.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.099+0000 7f3dfe589640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:35:29.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.099+0000 7f3dfe589640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:35:29.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.099+0000 7f3dfe589640 1 -- 172.21.15.121:0/2471695629 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3df809cd20 con 0x7f3df809ec60 2024-09-17T18:35:29.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.102+0000 7f3dfe589640 1 -- 172.21.15.121:0/2471695629 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3df809ec60 msgr2=0x7f3df8140e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:29.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.102+0000 7f3dfe589640 1 --2- 172.21.15.121:0/2471695629 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3df809ec60 0x7f3df8140e10 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3df0009dc0 tx=0x7f3df002a470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53d2b82a0f2816b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:29.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.102+0000 7f3dfcd86640 1 -- 172.21.15.121:0/2471695629 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df0007a30 con 0x7f3df809ec60 2024-09-17T18:35:29.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.102+0000 7f3dfcd86640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:29.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.102+0000 7f3dfcd86640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:35:29.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.102+0000 7f3dfcd86640 10 monclient: dump: 2024-09-17T18:35:29.113 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:29.113 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:29.113 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.113 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.113 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:29.113 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:29.113 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:29.114 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:29.114 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:29.114 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:29.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.102+0000 7f3dfcd86640 10 monclient: _finish_auth 0 2024-09-17T18:35:29.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.102+0000 7f3dfcd86640 10 monclient: _check_auth_tickets 2024-09-17T18:35:29.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.102+0000 7f3dfcd86640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:35:29.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.102+0000 7f3dfcd86640 1 -- 172.21.15.121:0/2471695629 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3df0007bd0 con 0x7f3df809ec60 2024-09-17T18:35:29.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.102+0000 7f3dfcd86640 10 monclient: handle_config config(0 keys) 2024-09-17T18:35:29.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.102+0000 7f3dfcd86640 1 -- 172.21.15.121:0/2471695629 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df0031400 con 0x7f3df809ec60 2024-09-17T18:35:29.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.102+0000 7f3dfffec640 5 monclient: authenticate success, global_id 64493 2024-09-17T18:35:29.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.102+0000 7f3dfffec640 10 monclient: get_monmap_and_config success 2024-09-17T18:35:29.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.102+0000 7f3dfffec640 4 set_mon_vals no callback set 2024-09-17T18:35:29.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.102+0000 7f3dfcd86640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:29.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.102+0000 7f3dfcd86640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:35:29.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.102+0000 7f3dfcd86640 10 monclient: dump: 2024-09-17T18:35:29.115 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:29.115 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:29.115 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.115 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.115 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:29.115 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:29.115 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:29.115 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:29.115 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:29.115 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:29.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.103+0000 7f3dfffec640 10 monclient: shutdown 2024-09-17T18:35:29.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.103+0000 7f3dfffec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:35:29.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.103+0000 7f3dfffec640 1 -- 172.21.15.121:0/2471695629 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3df809ec60 msgr2=0x7f3df8140e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:29.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.103+0000 7f3dfffec640 1 --2- 172.21.15.121:0/2471695629 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3df809ec60 0x7f3df8140e10 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3df0009dc0 tx=0x7f3df002a470 comp rx=0 tx=0).stop 2024-09-17T18:35:29.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.103+0000 7f3dfffec640 1 -- 172.21.15.121:0/2471695629 shutdown_connections 2024-09-17T18:35:29.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.103+0000 7f3dfffec640 1 --2- 172.21.15.121:0/2471695629 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3df809ec60 0x7f3df8140e10 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:29.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.103+0000 7f3dfffec640 1 -- 172.21.15.121:0/2471695629 >> 172.21.15.121:0/2471695629 conn(0x7f3df80ab830 msgr2=0x7f3df80a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:35:29.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.103+0000 7f3dfffec640 1 -- 172.21.15.121:0/2471695629 shutdown_connections 2024-09-17T18:35:29.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.103+0000 7f3dfffec640 1 -- 172.21.15.121:0/2471695629 wait complete. 2024-09-17T18:35:29.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.104+0000 7f3dfffec640 10 monclient: build_initial_monmap 2024-09-17T18:35:29.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.104+0000 7f3dfffec640 10 monclient: monmap: 2024-09-17T18:35:29.116 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:35:29.116 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:35:29.116 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:35:29.116 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:35:29.116 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:35:29.116 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:29.116 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:35:29.116 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:35:29.117 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:35:29.117 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:29.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.104+0000 7f3dfffec640 1 librados: starting msgr at 2024-09-17T18:35:29.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.104+0000 7f3dfffec640 1 librados: starting objecter 2024-09-17T18:35:29.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.104+0000 7f3dfffec640 1 Processor -- start 2024-09-17T18:35:29.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.104+0000 7f3dfffec640 1 -- start start 2024-09-17T18:35:29.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.104+0000 7f3dfffec640 1 librados: setting wanted keys 2024-09-17T18:35:29.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.104+0000 7f3dfffec640 1 librados: calling monclient init 2024-09-17T18:35:29.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.104+0000 7f3dfffec640 10 monclient: init 2024-09-17T18:35:29.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.104+0000 7f3dfffec640 10 monclient: _reopen_session rank -1 2024-09-17T18:35:29.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.104+0000 7f3dfffec640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T18:35:29.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.104+0000 7f3dfffec640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3df80700f0 0x7f3df8131d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:29.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.104+0000 7f3dfffec640 10 monclient(hunting): picked mon.noname-c con 0x7f3df80700f0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:35:29.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.104+0000 7f3dfffec640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3df809e810 0x7f3df8132290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:29.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.104+0000 7f3dfffec640 10 monclient(hunting): picked mon.noname-b con 0x7f3df809e810 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:35:29.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.104+0000 7f3dfffec640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3df809ec60 0x7f3df809fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:29.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.104+0000 7f3dfffec640 10 monclient(hunting): picked mon.noname-a con 0x7f3df809ec60 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:35:29.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.104+0000 7f3dfffec640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:29.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.104+0000 7f3dfffec640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3df8141350 con 0x7f3df809ec60 2024-09-17T18:35:29.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.104+0000 7f3dfffec640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:29.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.104+0000 7f3dfffec640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3df81414d0 con 0x7f3df809e810 2024-09-17T18:35:29.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.105+0000 7f3dfffec640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:29.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.105+0000 7f3dfffec640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3df8141650 con 0x7f3df80700f0 2024-09-17T18:35:29.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.105+0000 7f3dfffec640 10 monclient(hunting): _renew_subs 2024-09-17T18:35:29.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.105+0000 7f3dfffec640 10 monclient(hunting): authenticate will time out at 1732.822028s 2024-09-17T18:35:29.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.105+0000 7f3dfdd88640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3df80700f0 0x7f3df8131d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:29.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.105+0000 7f3dfe589640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3df809ec60 0x7f3df809fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:29.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.105+0000 7f3dfdd88640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3df80700f0 0x7f3df8131d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:53906/0 (socket says 172.21.15.121:53906) 2024-09-17T18:35:29.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.105+0000 7f3dfdd88640 1 -- 172.21.15.121:0/3832228560 learned_addr learned my addr 172.21.15.121:0/3832228560 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:35:29.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.105+0000 7f3dfdd88640 10 monclient(hunting): get_auth_request con 0x7f3df80700f0 auth_method 0 2024-09-17T18:35:29.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.105+0000 7f3dfe589640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3df809ec60 0x7f3df809fee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33796/0 (socket says 172.21.15.121:33796) 2024-09-17T18:35:29.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.105+0000 7f3dfdd88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:29.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.105+0000 7f3dfdd88640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:29.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.105+0000 7f3dfdd88640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:29.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.105+0000 7f3dfe589640 10 monclient(hunting): get_auth_request con 0x7f3df809ec60 auth_method 0 2024-09-17T18:35:29.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.105+0000 7f3dfe589640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:29.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.105+0000 7f3dfe589640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:29.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.105+0000 7f3dfe589640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:29.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.105+0000 7f3dfdd88640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:29.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.105+0000 7f3dfdd88640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:29.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.105+0000 7f3dfdd88640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:29.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.105+0000 7f3dfe589640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:29.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.105+0000 7f3dfe589640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:29.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.105+0000 7f3dfe589640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:29.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.105+0000 7f3dfd587640 1 --2- 172.21.15.121:0/3832228560 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3df809e810 0x7f3df8132290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:29.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.105+0000 7f3dfdd88640 10 monclient(hunting): handle_auth_done global_id 64496 payload 824 2024-09-17T18:35:29.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.105+0000 7f3dfdd88640 1 -- 172.21.15.121:0/3832228560 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3df809e810 msgr2=0x7f3df8132290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:29.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.105+0000 7f3dfdd88640 1 --2- 172.21.15.121:0/3832228560 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3df809e810 0x7f3df8132290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:29.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.105+0000 7f3dfdd88640 1 -- 172.21.15.121:0/3832228560 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3df809ec60 msgr2=0x7f3df809fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:29.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.105+0000 7f3dfdd88640 1 --2- 172.21.15.121:0/3832228560 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3df809ec60 0x7f3df809fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:29.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.105+0000 7f3dfdd88640 10 monclient: _finish_hunting 0 2024-09-17T18:35:29.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.105+0000 7f3dfdd88640 1 monclient: found mon.noname-c 2024-09-17T18:35:29.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.105+0000 7f3dfdd88640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:35:29.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.105+0000 7f3dfdd88640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:35:29.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.105+0000 7f3dfdd88640 1 -- 172.21.15.121:0/3832228560 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3df813ab40 con 0x7f3df80700f0 2024-09-17T18:35:29.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.105+0000 7f3dfe589640 1 --2- 172.21.15.121:0/3832228560 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3df809ec60 0x7f3df809fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:35:29.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.107+0000 7f3dfdd88640 1 -- 172.21.15.121:0/3832228560 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3df80700f0 msgr2=0x7f3df8131d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:29.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.107+0000 7f3dfdd88640 1 --2- 172.21.15.121:0/3832228560 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3df80700f0 0x7f3df8131d50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3dec00c460 tx=0x7f3dec00c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:29.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.107+0000 7f3dea7fc640 10 client.?.objecter ms_handle_connect 0x7f3df80700f0 2024-09-17T18:35:29.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.107+0000 7f3dea7fc640 10 client.?.objecter resend_mon_ops 2024-09-17T18:35:29.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.107+0000 7f3dea7fc640 1 -- 172.21.15.121:0/3832228560 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dec007070 con 0x7f3df80700f0 2024-09-17T18:35:29.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.107+0000 7f3dea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:29.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.107+0000 7f3dea7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:35:29.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.107+0000 7f3dea7fc640 10 monclient: dump: 2024-09-17T18:35:29.121 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:29.121 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:29.121 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.121 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.121 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:29.121 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:29.121 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:29.121 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:29.121 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:29.121 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:29.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.107+0000 7f3dea7fc640 10 monclient: _finish_auth 0 2024-09-17T18:35:29.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.107+0000 7f3dea7fc640 10 monclient: _check_auth_tickets 2024-09-17T18:35:29.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.107+0000 7f3dea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:35:29.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.107+0000 7f3dea7fc640 1 -- 172.21.15.121:0/3832228560 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dec007230 con 0x7f3df80700f0 2024-09-17T18:35:29.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.107+0000 7f3dea7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:35:29.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.107+0000 7f3dea7fc640 1 -- 172.21.15.121:0/3832228560 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dec007620 con 0x7f3df80700f0 2024-09-17T18:35:29.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.107+0000 7f3dfffec640 5 monclient: authenticate success, global_id 64496 2024-09-17T18:35:29.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.107+0000 7f3dfffec640 10 monclient: _renew_subs 2024-09-17T18:35:29.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.107+0000 7f3dfffec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:29.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.107+0000 7f3dfffec640 1 -- 172.21.15.121:0/3832228560 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3df813ae20 con 0x7f3df80700f0 2024-09-17T18:35:29.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.107+0000 7f3dea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:29.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.107+0000 7f3dea7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:35:29.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.107+0000 7f3debfff640 4 set_mon_vals no callback set 2024-09-17T18:35:29.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.107+0000 7f3dea7fc640 10 monclient: dump: 2024-09-17T18:35:29.123 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:29.123 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:29.123 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.123 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.123 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:29.123 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:29.123 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:29.123 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:29.123 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:29.123 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:29.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.107+0000 7f3dfffec640 10 client.64496.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:35:29.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.107+0000 7f3dfffec640 10 monclient: _renew_subs 2024-09-17T18:35:29.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.107+0000 7f3dfffec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:29.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.107+0000 7f3dfffec640 1 -- 172.21.15.121:0/3832228560 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3df8139620 con 0x7f3df80700f0 2024-09-17T18:35:29.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.107+0000 7f3dfffec640 1 librados: init done 2024-09-17T18:35:29.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.108+0000 7f3dfffec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:29.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.108+0000 7f3dfffec640 1 --2- 172.21.15.121:0/3832228560 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3df80a0420 0x7f3df80a0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:29.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.108+0000 7f3dfffec640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:29.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.108+0000 7f3dfffec640 1 -- 172.21.15.121:0/3832228560 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3df809d0b0 con 0x7f3df80a0420 2024-09-17T18:35:29.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.108+0000 7f3dfd587640 1 --2- 172.21.15.121:0/3832228560 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3df80a0420 0x7f3df80a0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:29.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.108+0000 7f3dea7fc640 1 -- 172.21.15.121:0/3832228560 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f3dec01b400 con 0x7f3df80700f0 2024-09-17T18:35:29.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.108+0000 7f3dfd587640 10 monclient: get_auth_request con 0x7f3df80a0420 auth_method 0 2024-09-17T18:35:29.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.108+0000 7f3dfd587640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:29.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.108+0000 7f3dfd587640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:29.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.108+0000 7f3dfd587640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:29.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.108+0000 7f3dfd587640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:29.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.108+0000 7f3dfd587640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:29.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.108+0000 7f3dfd587640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:29.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.108+0000 7f3dea7fc640 1 --2- 172.21.15.121:0/3832228560 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3ddc0401c0 0x7f3ddc042680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:29.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.108+0000 7f3dfd587640 10 monclient(hunting): handle_auth_done global_id 64499 payload 824 2024-09-17T18:35:29.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.108+0000 7f3dfd587640 1 -- 172.21.15.121:0/3832228560 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3df8134e50 con 0x7f3df80a0420 2024-09-17T18:35:29.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.108+0000 7f3dea7fc640 1 -- 172.21.15.121:0/3832228560 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(646..646 src has 247..646) ==== 5925+0+0 (secure 0 0 0) 0x7f3dec053590 con 0x7f3df80700f0 2024-09-17T18:35:29.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.108+0000 7f3dea7fc640 10 client.64496.objecter ms_dispatch 0x7f3df8000b90 osd_map(646..646 src has 247..646) 2024-09-17T18:35:29.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.109+0000 7f3dea7fc640 3 client.64496.objecter handle_osd_map got epochs [646,646] > 0 2024-09-17T18:35:29.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.109+0000 7f3dea7fc640 3 client.64496.objecter handle_osd_map decoding full epoch 646 2024-09-17T18:35:29.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.109+0000 7f3dea7fc640 20 client.64496.objecter dump_active .. 0 homeless 2024-09-17T18:35:29.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.109+0000 7f3dfe589640 1 --2- 172.21.15.121:0/3832228560 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3ddc0401c0 0x7f3ddc042680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:29.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.109+0000 7f3dfe589640 10 monclient: get_auth_request con 0x7f3ddc0401c0 auth_method 0 2024-09-17T18:35:29.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.111+0000 7f3dea7fc640 10 client.64496.objecter ms_handle_connect 0x7f3df80a0420 2024-09-17T18:35:29.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.111+0000 7f3dfd587640 1 -- 172.21.15.121:0/3832228560 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3df80a0420 msgr2=0x7f3df80a0800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:29.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.111+0000 7f3dea7fc640 10 client.64496.objecter resend_mon_ops 2024-09-17T18:35:29.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.111+0000 7f3dfd587640 1 --2- 172.21.15.121:0/3832228560 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3df80a0420 0x7f3df80a0800 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3df81395b0 tx=0x7f3df400d810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:29.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.111+0000 7f3dea7fc640 1 -- 172.21.15.121:0/3832228560 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df4016020 con 0x7f3df80a0420 2024-09-17T18:35:29.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.111+0000 7f3dea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:29.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.111+0000 7f3dea7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:35:29.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.111+0000 7f3dea7fc640 10 monclient: dump: 2024-09-17T18:35:29.128 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:29.128 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:29.128 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.128 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.128 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:29.128 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:29.128 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:29.129 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:29.129 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:29.129 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:29.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.111+0000 7f3dea7fc640 1 -- 172.21.15.121:0/3832228560 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3df4012a10 con 0x7f3df80a0420 2024-09-17T18:35:29.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.111+0000 7f3dea7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:29.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.111+0000 7f3dea7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:35:29.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.111+0000 7f3dea7fc640 1 -- 172.21.15.121:0/3832228560 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3df80a0420 msgr2=0x7f3df80a0800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:29.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.111+0000 7f3dea7fc640 1 --2- 172.21.15.121:0/3832228560 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3df80a0420 0x7f3df80a0800 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3df81395b0 tx=0x7f3df400d810 comp rx=0 tx=0).stop 2024-09-17T18:35:29.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.111+0000 7f3dea7fc640 1 -- 172.21.15.121:0/3832228560 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3df80a0420 msgr2=0x7f3df80a0800 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:35:29.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.111+0000 7f3dea7fc640 1 --2- 172.21.15.121:0/3832228560 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3df80a0420 0x7f3df80a0800 secure :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3df81395b0 tx=0x7f3df400d810 comp rx=0 tx=0).stop 2024-09-17T18:35:29.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.111+0000 7f3dfe589640 1 --2- 172.21.15.121:0/3832228560 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3ddc0401c0 0x7f3ddc042680 secure :-1 s=READY pgs=4077 cs=0 l=1 rev1=1 crypto rx=0x7f3df8077b10 tx=0x7f3df002f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:29.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.111+0000 7f3dea7fc640 10 client.64496.objecter ms_handle_connect 0x7f3ddc0401c0 2024-09-17T18:35:29.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.117+0000 7f3dfffec640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:35:29.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.117+0000 7f3dfffec640 1 --2- 172.21.15.121:0/3832228560 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3df80a0d40 0x7f3df8135f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:29.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.117+0000 7f3dfffec640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:29.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.117+0000 7f3dfffec640 1 -- 172.21.15.121:0/3832228560 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3df80a1150 con 0x7f3df80a0d40 2024-09-17T18:35:29.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.117+0000 7f3dfd587640 1 --2- 172.21.15.121:0/3832228560 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3df80a0d40 0x7f3df8135f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:29.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.117+0000 7f3dfd587640 10 monclient: get_auth_request con 0x7f3df80a0d40 auth_method 0 2024-09-17T18:35:29.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.117+0000 7f3dfd587640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:29.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.117+0000 7f3dfd587640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:29.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.117+0000 7f3dfd587640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:29.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.117+0000 7f3dfd587640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:29.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.117+0000 7f3dfd587640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:29.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.117+0000 7f3dfd587640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:29.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.117+0000 7f3dfd587640 10 monclient(hunting): handle_auth_done global_id 64502 payload 824 2024-09-17T18:35:29.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.117+0000 7f3dfd587640 1 -- 172.21.15.121:0/3832228560 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3df81327d0 con 0x7f3df80a0d40 2024-09-17T18:35:29.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.120+0000 7f3dea7fc640 10 client.64496.objecter ms_handle_connect 0x7f3df80a0d40 2024-09-17T18:35:29.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.120+0000 7f3dea7fc640 10 client.64496.objecter resend_mon_ops 2024-09-17T18:35:29.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.120+0000 7f3dfd587640 1 -- 172.21.15.121:0/3832228560 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3df80a0d40 msgr2=0x7f3df8135f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:29.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.120+0000 7f3dfd587640 1 --2- 172.21.15.121:0/3832228560 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3df80a0d40 0x7f3df8135f30 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3df4017000 tx=0x7f3df4003760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:29.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.120+0000 7f3dea7fc640 1 -- 172.21.15.121:0/3832228560 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df4016050 con 0x7f3df80a0d40 2024-09-17T18:35:29.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.120+0000 7f3dea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:29.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.120+0000 7f3dea7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:35:29.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.120+0000 7f3dea7fc640 10 monclient: dump: 2024-09-17T18:35:29.131 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:29.131 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:29.131 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.131 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.131 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:29.131 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:29.131 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:29.131 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:29.131 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:29.131 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:29.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.120+0000 7f3dea7fc640 1 -- 172.21.15.121:0/3832228560 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7f3df401b830 con 0x7f3df80a0d40 2024-09-17T18:35:29.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.120+0000 7f3dea7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:35:29.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.120+0000 7f3dea7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:35:29.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.120+0000 7f3dea7fc640 1 -- 172.21.15.121:0/3832228560 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3df80a0d40 msgr2=0x7f3df8135f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:29.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.120+0000 7f3dea7fc640 1 --2- 172.21.15.121:0/3832228560 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3df80a0d40 0x7f3df8135f30 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3df4017000 tx=0x7f3df4003760 comp rx=0 tx=0).stop 2024-09-17T18:35:29.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.120+0000 7f3dea7fc640 1 -- 172.21.15.121:0/3832228560 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3df80a0d40 msgr2=0x7f3df8135f30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:35:29.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.120+0000 7f3dea7fc640 1 --2- 172.21.15.121:0/3832228560 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3df80a0d40 0x7f3df8135f30 secure :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3df4017000 tx=0x7f3df4003760 comp rx=0 tx=0).stop 2024-09-17T18:35:29.132 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:35:29.132 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:35:29.132 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:35:29.132 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:35:29.133 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 60, 2024-09-17T18:35:29.133 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 507853, 2024-09-17T18:35:29.133 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:35:29.133 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:35:29.133 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:35:29.133 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:35:29.133 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:35:29.133 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 3, 2024-09-17T18:35:29.133 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:35:29.133 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:35:29.133 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:35:29.133 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:35:29.133 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:35:29.133 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:35:29.134 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:35:29.134 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:35:29.134 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:35:29.134 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:35:29.134 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:35:29.134 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:35:29.134 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:35:29.134 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:35:29.134 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:35:29.134 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:35:29.134 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:35:29.134 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:35:29.134 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:35:29.134 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:35:29.135 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:35:29.135 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:35:29.135 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:35:29.135 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:35:29.135 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:35:29.135 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:35:29.135 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:35:29.135 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:35:29.135 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:29.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:29.135 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:35:29.135 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:35:29.135 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:35:29.135 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:35:29.135 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:35:29.136 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:35:29.136 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:35:29.136 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:35:29.136 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:35:29.136 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:35:29.136 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:35:29.136 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:35:29.136 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:35:29.136 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:35:29.136 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:35:29.136 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:35:29.136 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:35:29.136 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:35:29.136 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:35:29.137 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:35:29.137 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:35:29.137 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:35:29.137 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:35:29.137 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:35:29.137 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:35:29.137 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:35:29.137 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:35:29.137 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:35:29.137 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:35:29.137 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:35:29.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:29.137 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:35:29.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:29.137 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:35:29.138 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:35:29.138 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:35:29.138 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:35:29.138 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:29.138 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:35:29.138 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:35:29.138 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:29.138 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:29.138 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:29.138 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:35:29.138 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:35:29.138 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:29.138 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:29.138 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:29.138 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:29.139 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:35:29.139 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:35:29.139 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:35:29.139 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:35:29.139 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:35:29.139 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:29.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:29.139 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:35:29.139 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:35:29.139 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:35:29.139 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:35:29.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:29.139 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:35:29.139 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:35:29.139 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:29.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:29.140 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:29.140 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:35:29.140 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:35:29.140 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:29.140 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:29.140 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:29.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:29.140 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:35:29.140 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:35:29.140 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:35:29.140 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:35:29.140 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:35:29.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:29.140 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:29.141 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:35:29.141 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:35:29.141 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:35:29.141 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:35:29.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:29.141 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:35:29.141 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:35:29.141 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:29.141 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:29.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:29.141 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:35:29.141 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:35:29.141 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:29.141 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:29.141 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:29.142 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:29.142 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:35:29.142 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:35:29.142 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:35:29.142 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:35:29.142 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:35:29.142 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:29.142 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:29.142 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:29.142 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:35:29.142 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:35:29.142 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:29.142 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:35:29.142 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:35:29.142 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:35:29.142 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:29.143 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:35:29.143 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:35:29.143 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:29.143 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:35:29.143 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:35:29.143 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 6 2024-09-17T18:35:29.143 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:29.143 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:35:29.143 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:35:29.143 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:29.143 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:35:29.143 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:35:29.143 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 4 2024-09-17T18:35:29.143 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:29.143 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:29.143 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:29.144 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:35:29.144 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:35:29.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.120+0000 7f3dfffec640 10 librados: watch_flush enter 2024-09-17T18:35:29.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.120+0000 7f3dfffec640 10 librados: watch_flush exit 2024-09-17T18:35:29.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.120+0000 7f3dfffec640 20 client.64496.objecter shutdown clearing up homeless session... 2024-09-17T18:35:29.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.120+0000 7f3dfffec640 10 client.64496.objecter successfully canceled tick 2024-09-17T18:35:29.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.120+0000 7f3dfffec640 1 -- 172.21.15.121:0/3832228560 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3ddc0401c0 msgr2=0x7f3ddc042680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:29.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.120+0000 7f3dfffec640 1 --2- 172.21.15.121:0/3832228560 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3ddc0401c0 0x7f3ddc042680 secure :-1 s=READY pgs=4077 cs=0 l=1 rev1=1 crypto rx=0x7f3df8077b10 tx=0x7f3df002f000 comp rx=0 tx=0).stop 2024-09-17T18:35:29.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.120+0000 7f3dfffec640 10 monclient: shutdown 2024-09-17T18:35:29.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.120+0000 7f3dfffec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:35:29.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.120+0000 7f3dfffec640 1 -- 172.21.15.121:0/3832228560 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3df80700f0 msgr2=0x7f3df8131d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:29.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.120+0000 7f3dfffec640 1 --2- 172.21.15.121:0/3832228560 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3df80700f0 0x7f3df8131d50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3dec00c460 tx=0x7f3dec00c920 comp rx=0 tx=0).stop 2024-09-17T18:35:29.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.120+0000 7f3dfe589640 1 -- 172.21.15.121:0/3832228560 reap_dead start 2024-09-17T18:35:29.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.120+0000 7f3dfe589640 1 -- 172.21.15.121:0/3832228560 reap_dead start 2024-09-17T18:35:29.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.120+0000 7f3dfffec640 1 -- 172.21.15.121:0/3832228560 shutdown_connections 2024-09-17T18:35:29.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.120+0000 7f3dfffec640 1 -- 172.21.15.121:0/3832228560 >> 172.21.15.121:0/3832228560 conn(0x7f3df80ab830 msgr2=0x7f3df80a9070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:35:29.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.120+0000 7f3dfffec640 1 -- 172.21.15.121:0/3832228560 shutdown_connections 2024-09-17T18:35:29.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.120+0000 7f3dfffec640 1 -- 172.21.15.121:0/3832228560 wait complete. 2024-09-17T18:35:29.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.120+0000 7f3dfffec640 1 librados: shutdown 2024-09-17T18:35:29.147 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:35:29.147 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T18:35:29.147 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T18:35:29.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.245+0000 7fd921c95640 10 monclient: get_monmap_and_config 2024-09-17T18:35:29.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.245+0000 7fd921c95640 10 monclient: build_initial_monmap 2024-09-17T18:35:29.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.245+0000 7fd921c95640 10 monclient: monmap: 2024-09-17T18:35:29.249 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:35:29.249 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:29.249 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:35:29.247549+0000 2024-09-17T18:35:29.249 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:35:29.247549+0000 2024-09-17T18:35:29.249 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:35:29.249 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:29.249 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:35:29.249 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:35:29.249 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:35:29.249 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:35:29.249 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:35:29.249 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:29.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.246+0000 7fd921c95640 1 Processor -- start 2024-09-17T18:35:29.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.246+0000 7fd921c95640 1 -- start start 2024-09-17T18:35:29.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.246+0000 7fd921c95640 10 monclient: init 2024-09-17T18:35:29.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd921c95640 10 monclient: _reopen_session rank -1 2024-09-17T18:35:29.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd921c95640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-17T18:35:29.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd921c95640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd91c0a0a00 0x7fd91c0a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:29.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd921c95640 10 monclient(hunting): picked mon.noname-a con 0x7fd91c0a0a00 addr v2:172.21.15.121:3300/0 2024-09-17T18:35:29.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd921c95640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd91c0a1320 0x7fd91c134980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:29.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd921c95640 10 monclient(hunting): picked mon.noname-b con 0x7fd91c0a1320 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:35:29.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd921c95640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fd91c134ec0 0x7fd91c135330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:29.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd921c95640 10 monclient(hunting): picked mon.noname-c con 0x7fd91c134ec0 addr v2:172.21.15.173:3300/0 2024-09-17T18:35:29.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd921c95640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:29.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd921c95640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fd91c0a3820 con 0x7fd91c0a0a00 2024-09-17T18:35:29.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd921c95640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:29.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd921c95640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fd91c0a2ea0 con 0x7fd91c134ec0 2024-09-17T18:35:29.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd921c95640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:29.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd921c95640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd91c09e800 con 0x7fd91c0a1320 2024-09-17T18:35:29.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd921c95640 10 monclient(hunting): _renew_subs 2024-09-17T18:35:29.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd921c95640 10 monclient(hunting): authenticate will time out at 1732.964328s 2024-09-17T18:35:29.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd91b7fe640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd91c0a0a00 0x7fd91c0a0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:29.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd91affd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd91c0a1320 0x7fd91c134980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:29.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd91affd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd91c0a1320 0x7fd91c134980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:53934/0 (socket says 172.21.15.121:53934) 2024-09-17T18:35:29.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd91b7fe640 10 monclient(hunting): get_auth_request con 0x7fd91c0a0a00 auth_method 0 2024-09-17T18:35:29.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd91affd640 1 -- 172.21.15.121:0/2587644139 learned_addr learned my addr 172.21.15.121:0/2587644139 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:35:29.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd91b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:29.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd91b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:29.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd91b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:29.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd91affd640 10 monclient(hunting): get_auth_request con 0x7fd91c0a1320 auth_method 0 2024-09-17T18:35:29.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd91affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:29.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd91affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:29.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd91affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:29.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd91b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:29.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd91b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:29.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd91b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:29.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd91affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:29.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd91affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:29.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd91affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:29.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd91bfff640 1 --2- 172.21.15.121:0/2587644139 >> v2:172.21.15.173:3300/0 conn(0x7fd91c134ec0 0x7fd91c135330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:29.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd91b7fe640 10 monclient(hunting): handle_auth_done global_id 104145 payload 274 2024-09-17T18:35:29.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd91b7fe640 1 -- 172.21.15.121:0/2587644139 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd91c0a1320 msgr2=0x7fd91c134980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:35:29.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd91b7fe640 1 --2- 172.21.15.121:0/2587644139 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd91c0a1320 0x7fd91c134980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:29.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd91b7fe640 1 -- 172.21.15.121:0/2587644139 >> v2:172.21.15.173:3300/0 conn(0x7fd91c134ec0 msgr2=0x7fd91c135330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:35:29.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd91b7fe640 1 --2- 172.21.15.121:0/2587644139 >> v2:172.21.15.173:3300/0 conn(0x7fd91c134ec0 0x7fd91c135330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:29.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd91b7fe640 10 monclient: _finish_hunting 0 2024-09-17T18:35:29.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd91b7fe640 1 monclient: found mon.noname-a 2024-09-17T18:35:29.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd91b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:35:29.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd91b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:35:29.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd91b7fe640 1 -- 172.21.15.121:0/2587644139 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd91c0a2c30 con 0x7fd91c0a0a00 2024-09-17T18:35:29.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd91bfff640 10 monclient: get_auth_request con 0x7fd91c134ec0 auth_method 0 2024-09-17T18:35:29.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd91affd640 1 --2- 172.21.15.121:0/2587644139 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd91c0a1320 0x7fd91c134980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:35:29.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.247+0000 7fd91bfff640 1 --2- 172.21.15.121:0/2587644139 >> v2:172.21.15.173:3300/0 conn(0x7fd91c134ec0 0x7fd91c135330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T18:35:29.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.250+0000 7fd91b7fe640 1 -- 172.21.15.121:0/2587644139 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd91c0a0a00 msgr2=0x7fd91c0a0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:29.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.250+0000 7fd91b7fe640 1 --2- 172.21.15.121:0/2587644139 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd91c0a0a00 0x7fd91c0a0de0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd91c0a4e90 tx=0x7fd90c02a010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e95e01248fd0058 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:29.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.250+0000 7fd91a7fc640 1 -- 172.21.15.121:0/2587644139 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd90c037070 con 0x7fd91c0a0a00 2024-09-17T18:35:29.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.250+0000 7fd91a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:29.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.250+0000 7fd91a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:35:29.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.250+0000 7fd91a7fc640 10 monclient: dump: 2024-09-17T18:35:29.261 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:29.261 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:29.261 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.261 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.261 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:29.261 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:29.261 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:29.261 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:29.261 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:29.262 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:29.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.250+0000 7fd91a7fc640 10 monclient: _finish_auth 0 2024-09-17T18:35:29.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.250+0000 7fd91a7fc640 10 monclient: _check_auth_tickets 2024-09-17T18:35:29.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.250+0000 7fd91a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:35:29.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.250+0000 7fd91a7fc640 1 -- 172.21.15.121:0/2587644139 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd90c032440 con 0x7fd91c0a0a00 2024-09-17T18:35:29.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.250+0000 7fd921c95640 5 monclient: authenticate success, global_id 104145 2024-09-17T18:35:29.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.250+0000 7fd921c95640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:35:29.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.250+0000 7fd91a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:35:29.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.250+0000 7fd921c95640 10 monclient: get_monmap_and_config success 2024-09-17T18:35:29.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.250+0000 7fd91a7fc640 1 -- 172.21.15.121:0/2587644139 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd90c032810 con 0x7fd91c0a0a00 2024-09-17T18:35:29.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.250+0000 7fd921c95640 4 set_mon_vals no callback set 2024-09-17T18:35:29.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.250+0000 7fd91a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:29.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.250+0000 7fd91a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:35:29.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.250+0000 7fd91a7fc640 10 monclient: dump: 2024-09-17T18:35:29.262 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:29.262 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:29.263 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.263 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.263 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:29.263 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:29.263 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:29.263 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:29.263 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:29.263 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:29.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.251+0000 7fd921c95640 10 monclient: shutdown 2024-09-17T18:35:29.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.251+0000 7fd921c95640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:35:29.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.251+0000 7fd921c95640 1 -- 172.21.15.121:0/2587644139 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd91c0a0a00 msgr2=0x7fd91c0a0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:29.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.251+0000 7fd921c95640 1 --2- 172.21.15.121:0/2587644139 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd91c0a0a00 0x7fd91c0a0de0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd91c0a4e90 tx=0x7fd90c02a010 comp rx=0 tx=0).stop 2024-09-17T18:35:29.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.251+0000 7fd921c95640 1 -- 172.21.15.121:0/2587644139 shutdown_connections 2024-09-17T18:35:29.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.251+0000 7fd921c95640 1 --2- 172.21.15.121:0/2587644139 >> v2:172.21.15.173:3300/0 conn(0x7fd91c134ec0 0x7fd91c135330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:29.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.251+0000 7fd921c95640 1 --2- 172.21.15.121:0/2587644139 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd91c0a1320 0x7fd91c134980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:29.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.251+0000 7fd921c95640 1 --2- 172.21.15.121:0/2587644139 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd91c0a0a00 0x7fd91c0a0de0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:29.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.251+0000 7fd921c95640 1 -- 172.21.15.121:0/2587644139 >> 172.21.15.121:0/2587644139 conn(0x7fd91c0a7830 msgr2=0x7fd91c0a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:35:29.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.251+0000 7fd921c95640 1 -- 172.21.15.121:0/2587644139 shutdown_connections 2024-09-17T18:35:29.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.251+0000 7fd921c95640 1 -- 172.21.15.121:0/2587644139 wait complete. 2024-09-17T18:35:29.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.251+0000 7fd921c95640 10 monclient: build_initial_monmap 2024-09-17T18:35:29.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.251+0000 7fd921c95640 10 monclient: monmap: 2024-09-17T18:35:29.264 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:35:29.264 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:35:29.264 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:35:29.264 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:35:29.264 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:35:29.264 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:29.264 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:35:29.264 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:35:29.265 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:35:29.265 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:29.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.251+0000 7fd921c95640 1 librados: starting msgr at 2024-09-17T18:35:29.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.251+0000 7fd921c95640 1 librados: starting objecter 2024-09-17T18:35:29.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.252+0000 7fd921c95640 1 Processor -- start 2024-09-17T18:35:29.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.252+0000 7fd921c95640 1 -- start start 2024-09-17T18:35:29.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.252+0000 7fd921c95640 1 librados: setting wanted keys 2024-09-17T18:35:29.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.252+0000 7fd921c95640 1 librados: calling monclient init 2024-09-17T18:35:29.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.252+0000 7fd921c95640 10 monclient: init 2024-09-17T18:35:29.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.252+0000 7fd921c95640 10 monclient: _reopen_session rank -1 2024-09-17T18:35:29.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.252+0000 7fd921c95640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:35:29.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.252+0000 7fd921c95640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd91c0a0a00 0x7fd91c1cbb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:29.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.252+0000 7fd921c95640 10 monclient(hunting): picked mon.noname-a con 0x7fd91c0a0a00 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:35:29.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.252+0000 7fd921c95640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd91c0a1320 0x7fd91c1cc040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:29.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.252+0000 7fd921c95640 10 monclient(hunting): picked mon.noname-b con 0x7fd91c0a1320 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:35:29.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.252+0000 7fd921c95640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd91c134ec0 0x7fd91c1220a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:29.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.252+0000 7fd921c95640 10 monclient(hunting): picked mon.noname-c con 0x7fd91c134ec0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:35:29.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.252+0000 7fd921c95640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:29.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.252+0000 7fd921c95640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd91c09de60 con 0x7fd91c0a0a00 2024-09-17T18:35:29.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.252+0000 7fd921c95640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:29.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.252+0000 7fd921c95640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd91c09dce0 con 0x7fd91c0a1320 2024-09-17T18:35:29.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.252+0000 7fd921c95640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:29.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.252+0000 7fd921c95640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd91c0a3120 con 0x7fd91c134ec0 2024-09-17T18:35:29.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.252+0000 7fd921c95640 10 monclient(hunting): _renew_subs 2024-09-17T18:35:29.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.252+0000 7fd921c95640 10 monclient(hunting): authenticate will time out at 1732.969740s 2024-09-17T18:35:29.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.252+0000 7fd91b7fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd91c0a0a00 0x7fd91c1cbb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:29.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.252+0000 7fd91bfff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd91c134ec0 0x7fd91c1220a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:29.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.252+0000 7fd91b7fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd91c0a0a00 0x7fd91c1cbb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33804/0 (socket says 172.21.15.121:33804) 2024-09-17T18:35:29.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.252+0000 7fd91b7fe640 1 -- 172.21.15.121:0/2283904277 learned_addr learned my addr 172.21.15.121:0/2283904277 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:35:29.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.252+0000 7fd91b7fe640 10 monclient(hunting): get_auth_request con 0x7fd91c0a0a00 auth_method 0 2024-09-17T18:35:29.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.252+0000 7fd91b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:29.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.252+0000 7fd91b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:29.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.252+0000 7fd91b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:29.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.252+0000 7fd91bfff640 10 monclient(hunting): get_auth_request con 0x7fd91c134ec0 auth_method 0 2024-09-17T18:35:29.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.252+0000 7fd91bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:29.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.252+0000 7fd91bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:29.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.252+0000 7fd91bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:29.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.252+0000 7fd91b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:29.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.252+0000 7fd91b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:29.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.252+0000 7fd91b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:29.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.252+0000 7fd91bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:29.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.252+0000 7fd91bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:29.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.252+0000 7fd91bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:29.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.253+0000 7fd91bfff640 10 monclient(hunting): handle_auth_done global_id 64508 payload 824 2024-09-17T18:35:29.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.253+0000 7fd91bfff640 1 -- 172.21.15.121:0/2283904277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd91c0a1320 msgr2=0x7fd91c1cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:29.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.253+0000 7fd91bfff640 1 --2- 172.21.15.121:0/2283904277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd91c0a1320 0x7fd91c1cc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:29.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.253+0000 7fd91bfff640 1 -- 172.21.15.121:0/2283904277 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd91c0a0a00 msgr2=0x7fd91c1cbb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:29.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.253+0000 7fd91bfff640 1 --2- 172.21.15.121:0/2283904277 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd91c0a0a00 0x7fd91c1cbb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:29.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.253+0000 7fd91bfff640 10 monclient: _finish_hunting 0 2024-09-17T18:35:29.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.253+0000 7fd91bfff640 1 monclient: found mon.noname-c 2024-09-17T18:35:29.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.253+0000 7fd91bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:35:29.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.253+0000 7fd91bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:35:29.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.253+0000 7fd91bfff640 1 -- 172.21.15.121:0/2283904277 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd91c125ea0 con 0x7fd91c134ec0 2024-09-17T18:35:29.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.253+0000 7fd91b7fe640 1 --2- 172.21.15.121:0/2283904277 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd91c0a0a00 0x7fd91c1cbb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:35:29.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.255+0000 7fd91bfff640 1 -- 172.21.15.121:0/2283904277 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd91c134ec0 msgr2=0x7fd91c1220a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:29.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.255+0000 7fd91bfff640 1 --2- 172.21.15.121:0/2283904277 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd91c134ec0 0x7fd91c1220a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd91400ce20 tx=0x7fd91400a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:29.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.255+0000 7fd903fff640 10 client.?.objecter ms_handle_connect 0x7fd91c134ec0 2024-09-17T18:35:29.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.255+0000 7fd903fff640 10 client.?.objecter resend_mon_ops 2024-09-17T18:35:29.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.255+0000 7fd903fff640 1 -- 172.21.15.121:0/2283904277 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd91401a070 con 0x7fd91c134ec0 2024-09-17T18:35:29.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.255+0000 7fd903fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:29.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.255+0000 7fd903fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:35:29.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.255+0000 7fd903fff640 10 monclient: dump: 2024-09-17T18:35:29.269 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:29.269 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:29.269 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.269 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.269 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:29.269 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:29.269 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:29.269 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:29.269 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:29.269 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:29.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.255+0000 7fd903fff640 10 monclient: _finish_auth 0 2024-09-17T18:35:29.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.255+0000 7fd903fff640 10 monclient: _check_auth_tickets 2024-09-17T18:35:29.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.255+0000 7fd903fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:35:29.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.255+0000 7fd903fff640 1 -- 172.21.15.121:0/2283904277 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd91400da50 con 0x7fd91c134ec0 2024-09-17T18:35:29.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.255+0000 7fd921c95640 5 monclient: authenticate success, global_id 64508 2024-09-17T18:35:29.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.255+0000 7fd921c95640 10 monclient: _renew_subs 2024-09-17T18:35:29.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.255+0000 7fd921c95640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:29.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.255+0000 7fd921c95640 1 -- 172.21.15.121:0/2283904277 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd91c09e240 con 0x7fd91c134ec0 2024-09-17T18:35:29.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.255+0000 7fd903fff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:35:29.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.255+0000 7fd903fff640 1 -- 172.21.15.121:0/2283904277 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd91400de40 con 0x7fd91c134ec0 2024-09-17T18:35:29.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.255+0000 7fd903fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:29.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.255+0000 7fd903fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:35:29.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.255+0000 7fd903fff640 10 monclient: dump: 2024-09-17T18:35:29.270 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:29.270 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:29.270 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.270 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.270 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:29.270 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:29.270 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:29.271 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:29.271 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:29.271 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:29.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.255+0000 7fd919ffb640 4 set_mon_vals no callback set 2024-09-17T18:35:29.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.255+0000 7fd921c95640 10 client.64508.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:35:29.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.255+0000 7fd921c95640 10 monclient: _renew_subs 2024-09-17T18:35:29.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.255+0000 7fd921c95640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:29.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.255+0000 7fd921c95640 1 -- 172.21.15.121:0/2283904277 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd91c123b40 con 0x7fd91c134ec0 2024-09-17T18:35:29.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.255+0000 7fd921c95640 1 librados: init done 2024-09-17T18:35:29.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.256+0000 7fd903fff640 1 -- 172.21.15.121:0/2283904277 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fd91401d070 con 0x7fd91c134ec0 2024-09-17T18:35:29.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.256+0000 7fd921c95640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:29.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.256+0000 7fd921c95640 1 --2- 172.21.15.121:0/2283904277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd91c122da0 0x7fd91c02d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:29.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.256+0000 7fd921c95640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:29.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.256+0000 7fd921c95640 1 -- 172.21.15.121:0/2283904277 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd91c0a5290 con 0x7fd91c122da0 2024-09-17T18:35:29.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.256+0000 7fd903fff640 1 --2- 172.21.15.121:0/2283904277 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd908040170 0x7fd908042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:29.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.256+0000 7fd903fff640 1 -- 172.21.15.121:0/2283904277 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(646..646 src has 247..646) ==== 5925+0+0 (secure 0 0 0) 0x7fd9140540e0 con 0x7fd91c134ec0 2024-09-17T18:35:29.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.256+0000 7fd91b7fe640 1 --2- 172.21.15.121:0/2283904277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd91c122da0 0x7fd91c02d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:29.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.256+0000 7fd903fff640 10 client.64508.objecter ms_dispatch 0x7fd91c000b90 osd_map(646..646 src has 247..646) 2024-09-17T18:35:29.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.256+0000 7fd903fff640 3 client.64508.objecter handle_osd_map got epochs [646,646] > 0 2024-09-17T18:35:29.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.256+0000 7fd903fff640 3 client.64508.objecter handle_osd_map decoding full epoch 646 2024-09-17T18:35:29.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.256+0000 7fd91b7fe640 10 monclient: get_auth_request con 0x7fd91c122da0 auth_method 0 2024-09-17T18:35:29.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.256+0000 7fd91b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:29.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.256+0000 7fd91b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:29.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.256+0000 7fd91b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:29.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.256+0000 7fd903fff640 20 client.64508.objecter dump_active .. 0 homeless 2024-09-17T18:35:29.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.256+0000 7fd91b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:29.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.256+0000 7fd91b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:29.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.256+0000 7fd91b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:29.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.256+0000 7fd91affd640 1 --2- 172.21.15.121:0/2283904277 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd908040170 0x7fd908042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:29.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.256+0000 7fd91affd640 10 monclient: get_auth_request con 0x7fd908040170 auth_method 0 2024-09-17T18:35:29.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.257+0000 7fd91b7fe640 10 monclient(hunting): handle_auth_done global_id 84271 payload 824 2024-09-17T18:35:29.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.257+0000 7fd91b7fe640 1 -- 172.21.15.121:0/2283904277 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd91c0a1f10 con 0x7fd91c122da0 2024-09-17T18:35:29.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.259+0000 7fd903fff640 10 client.64508.objecter ms_handle_connect 0x7fd91c122da0 2024-09-17T18:35:29.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.259+0000 7fd91b7fe640 1 -- 172.21.15.121:0/2283904277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd91c122da0 msgr2=0x7fd91c02d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:29.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.259+0000 7fd903fff640 10 client.64508.objecter resend_mon_ops 2024-09-17T18:35:29.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.259+0000 7fd91b7fe640 1 --2- 172.21.15.121:0/2283904277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd91c122da0 0x7fd91c02d440 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd90c0097a0 tx=0x7fd90c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:29.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.259+0000 7fd903fff640 10 client.64508.objecter ms_handle_connect 0x7fd908040170 2024-09-17T18:35:29.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.259+0000 7fd91affd640 1 --2- 172.21.15.121:0/2283904277 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd908040170 0x7fd908042630 secure :-1 s=READY pgs=4078 cs=0 l=1 rev1=1 crypto rx=0x7fd91000a7d0 tx=0x7fd910006e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:29.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.259+0000 7fd903fff640 1 -- 172.21.15.121:0/2283904277 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd90c043070 con 0x7fd91c122da0 2024-09-17T18:35:29.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.259+0000 7fd903fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:29.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.259+0000 7fd903fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:35:29.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.259+0000 7fd903fff640 10 monclient: dump: 2024-09-17T18:35:29.276 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:29.276 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:29.276 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.276 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.276 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:29.276 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:29.276 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:29.276 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:29.276 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:29.276 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:29.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.259+0000 7fd903fff640 1 -- 172.21.15.121:0/2283904277 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd90c03ca60 con 0x7fd91c122da0 2024-09-17T18:35:29.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.259+0000 7fd903fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:29.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.259+0000 7fd903fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:35:29.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.259+0000 7fd903fff640 1 -- 172.21.15.121:0/2283904277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd91c122da0 msgr2=0x7fd91c02d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:29.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.259+0000 7fd903fff640 1 --2- 172.21.15.121:0/2283904277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd91c122da0 0x7fd91c02d440 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd90c0097a0 tx=0x7fd90c0046c0 comp rx=0 tx=0).stop 2024-09-17T18:35:29.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.259+0000 7fd903fff640 1 -- 172.21.15.121:0/2283904277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd91c122da0 msgr2=0x7fd91c02d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:35:29.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.259+0000 7fd903fff640 1 --2- 172.21.15.121:0/2283904277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd91c122da0 0x7fd91c02d440 secure :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd90c0097a0 tx=0x7fd90c0046c0 comp rx=0 tx=0).stop 2024-09-17T18:35:29.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.265+0000 7fd921c95640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:35:29.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.265+0000 7fd921c95640 1 --2- 172.21.15.121:0/2283904277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd91c123180 0x7fd91c02d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:29.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.265+0000 7fd921c95640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:29.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.265+0000 7fd921c95640 1 -- 172.21.15.121:0/2283904277 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd91c125c80 con 0x7fd91c123180 2024-09-17T18:35:29.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.266+0000 7fd91b7fe640 1 --2- 172.21.15.121:0/2283904277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd91c123180 0x7fd91c02d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:29.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.266+0000 7fd91b7fe640 10 monclient: get_auth_request con 0x7fd91c123180 auth_method 0 2024-09-17T18:35:29.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.266+0000 7fd91b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:29.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.266+0000 7fd91b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:29.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.266+0000 7fd91b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:29.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.266+0000 7fd91b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:29.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.266+0000 7fd91b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:29.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.266+0000 7fd91b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:29.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.266+0000 7fd91b7fe640 10 monclient(hunting): handle_auth_done global_id 84274 payload 824 2024-09-17T18:35:29.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.266+0000 7fd91b7fe640 1 -- 172.21.15.121:0/2283904277 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd91c0729e0 con 0x7fd91c123180 2024-09-17T18:35:29.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.268+0000 7fd91b7fe640 1 -- 172.21.15.121:0/2283904277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd91c123180 msgr2=0x7fd91c02d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:29.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.268+0000 7fd91b7fe640 1 --2- 172.21.15.121:0/2283904277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd91c123180 0x7fd91c02d980 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd91c0a5410 tx=0x7fd90c009190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:29.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.268+0000 7fd903fff640 10 client.64508.objecter ms_handle_connect 0x7fd91c123180 2024-09-17T18:35:29.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.268+0000 7fd903fff640 10 client.64508.objecter resend_mon_ops 2024-09-17T18:35:29.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.269+0000 7fd903fff640 1 -- 172.21.15.121:0/2283904277 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd90c037040 con 0x7fd91c123180 2024-09-17T18:35:29.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.269+0000 7fd903fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:29.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.269+0000 7fd903fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:35:29.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.269+0000 7fd903fff640 10 monclient: dump: 2024-09-17T18:35:29.278 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:29.278 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:29.278 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.278 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.278 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:29.278 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:29.279 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:29.279 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:29.279 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:29.279 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:29.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.269+0000 7fd903fff640 1 -- 172.21.15.121:0/2283904277 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7fd90c03b5b0 con 0x7fd91c123180 2024-09-17T18:35:29.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.269+0000 7fd903fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:35:29.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.269+0000 7fd903fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:35:29.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.269+0000 7fd903fff640 1 -- 172.21.15.121:0/2283904277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd91c123180 msgr2=0x7fd91c02d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:29.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.269+0000 7fd903fff640 1 --2- 172.21.15.121:0/2283904277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd91c123180 0x7fd91c02d980 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd91c0a5410 tx=0x7fd90c009190 comp rx=0 tx=0).stop 2024-09-17T18:35:29.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.269+0000 7fd903fff640 1 -- 172.21.15.121:0/2283904277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd91c123180 msgr2=0x7fd91c02d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:35:29.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.269+0000 7fd903fff640 1 --2- 172.21.15.121:0/2283904277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd91c123180 0x7fd91c02d980 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:29.279 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:35:29.280 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:35:29.280 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:35:29.280 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:35:29.280 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 60, 2024-09-17T18:35:29.280 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 254368, 2024-09-17T18:35:29.280 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:35:29.280 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:35:29.280 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:35:29.280 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:35:29.280 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:35:29.280 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 3, 2024-09-17T18:35:29.280 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:35:29.280 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:35:29.280 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:35:29.281 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:35:29.281 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:35:29.281 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:35:29.281 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:35:29.281 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:35:29.281 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:35:29.281 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:35:29.281 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:35:29.281 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:35:29.281 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:35:29.281 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:35:29.281 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:35:29.281 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:35:29.281 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:35:29.281 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:35:29.282 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:35:29.282 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:35:29.282 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:35:29.282 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:35:29.282 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:35:29.282 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:35:29.282 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:35:29.282 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:35:29.282 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:35:29.282 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:35:29.282 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:29.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:29.282 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:35:29.282 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:35:29.282 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:35:29.283 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:35:29.283 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:35:29.283 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:35:29.283 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:35:29.283 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:35:29.283 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:35:29.283 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:35:29.283 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:35:29.283 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:35:29.283 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:35:29.283 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:35:29.283 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:35:29.283 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:35:29.283 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:35:29.283 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:35:29.283 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:35:29.284 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:35:29.284 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:35:29.284 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:35:29.284 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:35:29.284 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:35:29.284 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:35:29.284 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:35:29.284 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:35:29.284 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:35:29.284 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:35:29.284 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:35:29.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:29.284 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:35:29.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:29.284 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:35:29.285 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:35:29.285 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:35:29.285 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:35:29.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:29.285 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:35:29.285 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:35:29.285 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:29.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:29.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:29.285 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:35:29.285 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:35:29.285 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:29.285 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:29.285 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:29.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:29.286 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:35:29.286 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:35:29.286 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:35:29.286 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:35:29.286 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:35:29.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:29.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:29.286 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:35:29.286 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:35:29.286 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:35:29.286 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:35:29.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:29.286 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:35:29.286 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:35:29.286 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:29.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:29.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:29.287 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:35:29.287 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:35:29.287 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:29.287 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:29.287 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:29.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:29.287 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:35:29.287 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:35:29.287 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:35:29.287 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:35:29.287 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:35:29.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:29.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:29.287 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:35:29.287 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:35:29.288 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:35:29.288 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:35:29.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:29.288 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:35:29.288 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:35:29.288 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:29.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:29.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:29.288 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:35:29.288 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:35:29.288 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:29.288 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:29.288 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:29.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:29.288 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:35:29.289 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:35:29.289 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:35:29.289 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:35:29.289 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:35:29.289 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:29.289 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:29.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:29.289 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:35:29.289 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:35:29.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:29.289 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:35:29.289 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:35:29.289 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:35:29.290 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:29.290 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:35:29.290 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:35:29.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:29.290 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:35:29.290 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:35:29.290 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 4 2024-09-17T18:35:29.290 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:29.290 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:35:29.290 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": [ 2024-09-17T18:35:29.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:29.290 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:35:29.290 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:35:29.290 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:35:29.291 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:29.291 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:29.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:29.291 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:35:29.291 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:35:29.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.269+0000 7fd921c95640 10 librados: watch_flush enter 2024-09-17T18:35:29.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.269+0000 7fd921c95640 10 librados: watch_flush exit 2024-09-17T18:35:29.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.269+0000 7fd921c95640 20 client.64508.objecter shutdown clearing up homeless session... 2024-09-17T18:35:29.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.269+0000 7fd921c95640 10 client.64508.objecter successfully canceled tick 2024-09-17T18:35:29.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.269+0000 7fd921c95640 1 -- 172.21.15.121:0/2283904277 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd908040170 msgr2=0x7fd908042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:29.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.269+0000 7fd921c95640 1 --2- 172.21.15.121:0/2283904277 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd908040170 0x7fd908042630 secure :-1 s=READY pgs=4078 cs=0 l=1 rev1=1 crypto rx=0x7fd91000a7d0 tx=0x7fd910006e90 comp rx=0 tx=0).stop 2024-09-17T18:35:29.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.269+0000 7fd921c95640 10 monclient: shutdown 2024-09-17T18:35:29.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.269+0000 7fd921c95640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:35:29.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.269+0000 7fd921c95640 1 -- 172.21.15.121:0/2283904277 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd91c134ec0 msgr2=0x7fd91c1220a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:29.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.269+0000 7fd921c95640 1 --2- 172.21.15.121:0/2283904277 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd91c134ec0 0x7fd91c1220a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd91400ce20 tx=0x7fd91400a710 comp rx=0 tx=0).stop 2024-09-17T18:35:29.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.269+0000 7fd91bfff640 1 -- 172.21.15.121:0/2283904277 reap_dead start 2024-09-17T18:35:29.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.269+0000 7fd921c95640 1 -- 172.21.15.121:0/2283904277 shutdown_connections 2024-09-17T18:35:29.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.269+0000 7fd921c95640 1 --2- 172.21.15.121:0/2283904277 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd91c134ec0 0x7fd91c1220a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:29.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.269+0000 7fd921c95640 1 -- 172.21.15.121:0/2283904277 >> 172.21.15.121:0/2283904277 conn(0x7fd91c0a7830 msgr2=0x7fd91c134140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:35:29.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.269+0000 7fd921c95640 1 -- 172.21.15.121:0/2283904277 shutdown_connections 2024-09-17T18:35:29.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.269+0000 7fd921c95640 1 -- 172.21.15.121:0/2283904277 wait complete. 2024-09-17T18:35:29.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.269+0000 7fd921c95640 1 librados: shutdown 2024-09-17T18:35:29.294 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:35:29.294 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-17T18:35:29.294 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-17T18:35:29.294 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-17T18:35:29.294 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a store 2024-09-17T18:35:29.294 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a sync_force --yes-i-really-mean-it 2024-09-17T18:35:29.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.391+0000 7f94d78ed640 10 monclient: get_monmap_and_config 2024-09-17T18:35:29.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.391+0000 7f94d78ed640 10 monclient: build_initial_monmap 2024-09-17T18:35:29.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.391+0000 7f94d78ed640 10 monclient: monmap: 2024-09-17T18:35:29.394 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:35:29.394 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:29.394 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:35:29.392746+0000 2024-09-17T18:35:29.394 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:35:29.392746+0000 2024-09-17T18:35:29.394 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:35:29.394 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:29.394 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:35:29.394 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:35:29.395 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:35:29.395 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:35:29.395 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:35:29.395 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:29.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.391+0000 7f94d78ed640 1 Processor -- start 2024-09-17T18:35:29.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.391+0000 7f94d78ed640 1 -- start start 2024-09-17T18:35:29.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.391+0000 7f94d78ed640 10 monclient: init 2024-09-17T18:35:29.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.392+0000 7f94d78ed640 10 monclient: _reopen_session rank -1 2024-09-17T18:35:29.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.392+0000 7f94d78ed640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-17T18:35:29.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.392+0000 7f94d78ed640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f94d00a5b90 addr v1:172.21.15.173:6789/0 2024-09-17T18:35:29.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.392+0000 7f94d78ed640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f94d00a71c0 0x7f94d013dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:29.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.392+0000 7f94d78ed640 10 monclient(hunting): picked mon.noname-c con 0x7f94d00a71c0 addr v2:172.21.15.173:3300/0 2024-09-17T18:35:29.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.392+0000 7f94d78ed640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f94d00a6680 0x7f94d01401c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:29.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.392+0000 7f94d78ed640 10 monclient(hunting): picked mon.noname-b con 0x7f94d00a6680 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:35:29.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.392+0000 7f94d78ed640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94d009cb90 con 0x7f94d00a5b90 2024-09-17T18:35:29.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.392+0000 7f94d78ed640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:29.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.392+0000 7f94d78ed640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f94d009d4c0 con 0x7f94d00a71c0 2024-09-17T18:35:29.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.392+0000 7f94d78ed640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:29.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.392+0000 7f94d78ed640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f94d00a6c50 con 0x7f94d00a6680 2024-09-17T18:35:29.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.392+0000 7f94d78ed640 10 monclient(hunting): _renew_subs 2024-09-17T18:35:29.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.392+0000 7f94d78ed640 10 monclient(hunting): authenticate will time out at 1733.109518s 2024-09-17T18:35:29.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.392+0000 7f94d5e8a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f94d00a6680 0x7f94d01401c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:29.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.392+0000 7f94d5e8a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f94d00a6680 0x7f94d01401c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:53952/0 (socket says 172.21.15.121:53952) 2024-09-17T18:35:29.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.392+0000 7f94d5e8a640 1 -- 172.21.15.121:0/3564405233 learned_addr learned my addr 172.21.15.121:0/3564405233 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:35:29.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.392+0000 7f94d5e8a640 10 monclient(hunting): get_auth_request con 0x7f94d00a6680 auth_method 0 2024-09-17T18:35:29.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.392+0000 7f94d5e8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:29.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.392+0000 7f94d5e8a640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:29.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.392+0000 7f94d5e8a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:29.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.392+0000 7f94d5e8a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:29.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.392+0000 7f94d5e8a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:29.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.392+0000 7f94d4e88640 1 --2- 172.21.15.121:0/3564405233 >> v2:172.21.15.173:3300/0 conn(0x7f94d00a71c0 0x7f94d013dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:29.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.392+0000 7f94d5e8a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:29.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.392+0000 7f94d4e88640 10 monclient(hunting): get_auth_request con 0x7f94d00a71c0 auth_method 0 2024-09-17T18:35:29.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.392+0000 7f94d4e88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:29.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.392+0000 7f94d4e88640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:29.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.392+0000 7f94d4e88640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:29.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.392+0000 7f94d4e88640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:29.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.392+0000 7f94d4e88640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:29.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.392+0000 7f94d4e88640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:29.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.392+0000 7f94d5e8a640 10 monclient(hunting): handle_auth_done global_id 64511 payload 274 2024-09-17T18:35:29.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.392+0000 7f94d5e8a640 1 -- 172.21.15.121:0/3564405233 >> v2:172.21.15.173:3300/0 conn(0x7f94d00a71c0 msgr2=0x7f94d013dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:35:29.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.392+0000 7f94d5e8a640 1 --2- 172.21.15.121:0/3564405233 >> v2:172.21.15.173:3300/0 conn(0x7f94d00a71c0 0x7f94d013dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:29.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.392+0000 7f94d5e8a640 1 -- 172.21.15.121:0/3564405233 >> v1:172.21.15.173:6789/0 conn(0x7f94d00a5b90 legacy=0x7f94d00a5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:35:29.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.392+0000 7f94d5e8a640 10 monclient: _finish_hunting 0 2024-09-17T18:35:29.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.392+0000 7f94d5e8a640 1 monclient: found mon.noname-b 2024-09-17T18:35:29.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.392+0000 7f94d5e8a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:35:29.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.392+0000 7f94d5e8a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:35:29.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.392+0000 7f94d5e8a640 1 -- 172.21.15.121:0/3564405233 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94d0070550 con 0x7f94d00a6680 2024-09-17T18:35:29.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.395+0000 7f94d5e8a640 1 -- 172.21.15.121:0/3564405233 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f94d00a6680 msgr2=0x7f94d01401c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:29.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.395+0000 7f94d5e8a640 1 --2- 172.21.15.121:0/3564405233 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f94d00a6680 0x7f94d01401c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f94c8009e50 tx=0x7f94c802a560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36d2a71224633cbc server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:29.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.395+0000 7f94c7fff640 1 -- 172.21.15.121:0/3564405233 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94c8036070 con 0x7f94d00a6680 2024-09-17T18:35:29.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.395+0000 7f94c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:29.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.395+0000 7f94c7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:35:29.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.395+0000 7f94c7fff640 10 monclient: dump: 2024-09-17T18:35:29.409 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:29.409 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:29.409 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.409 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.409 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:29.410 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:29.410 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:29.410 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:29.410 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:29.410 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:29.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.395+0000 7f94c7fff640 10 monclient: _finish_auth 0 2024-09-17T18:35:29.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.395+0000 7f94c7fff640 10 monclient: _check_auth_tickets 2024-09-17T18:35:29.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.395+0000 7f94c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:35:29.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.395+0000 7f94d78ed640 5 monclient: authenticate success, global_id 64511 2024-09-17T18:35:29.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.395+0000 7f94d78ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:35:29.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.395+0000 7f94c7fff640 1 -- 172.21.15.121:0/3564405233 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94c8007800 con 0x7f94d00a6680 2024-09-17T18:35:29.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.395+0000 7f94c7fff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:35:29.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.395+0000 7f94d78ed640 10 monclient: get_monmap_and_config success 2024-09-17T18:35:29.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.395+0000 7f94d78ed640 4 set_mon_vals no callback set 2024-09-17T18:35:29.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.395+0000 7f94c7fff640 1 -- 172.21.15.121:0/3564405233 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94c8007bd0 con 0x7f94d00a6680 2024-09-17T18:35:29.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.395+0000 7f94c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:29.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.395+0000 7f94c7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:35:29.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.395+0000 7f94c7fff640 10 monclient: dump: 2024-09-17T18:35:29.411 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:29.411 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:29.411 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.411 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.411 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:29.411 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:29.411 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:29.411 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:29.411 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:29.411 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:29.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.396+0000 7f94d78ed640 10 monclient: shutdown 2024-09-17T18:35:29.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.396+0000 7f94d78ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:35:29.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.396+0000 7f94d78ed640 1 -- 172.21.15.121:0/3564405233 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f94d00a6680 msgr2=0x7f94d01401c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:29.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.396+0000 7f94d78ed640 1 --2- 172.21.15.121:0/3564405233 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f94d00a6680 0x7f94d01401c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f94c8009e50 tx=0x7f94c802a560 comp rx=0 tx=0).stop 2024-09-17T18:35:29.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.396+0000 7f94d78ed640 1 -- 172.21.15.121:0/3564405233 shutdown_connections 2024-09-17T18:35:29.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.396+0000 7f94d78ed640 1 --2- 172.21.15.121:0/3564405233 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f94d00a6680 0x7f94d01401c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:29.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.396+0000 7f94d78ed640 1 --2- 172.21.15.121:0/3564405233 >> v2:172.21.15.173:3300/0 conn(0x7f94d00a71c0 0x7f94d013dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:29.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.396+0000 7f94d78ed640 1 -- 172.21.15.121:0/3564405233 >> 172.21.15.121:0/3564405233 conn(0x7f94d00a7c90 msgr2=0x7f94d00a80a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:35:29.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.396+0000 7f94d78ed640 1 -- 172.21.15.121:0/3564405233 shutdown_connections 2024-09-17T18:35:29.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.396+0000 7f94d78ed640 1 -- 172.21.15.121:0/3564405233 wait complete. 2024-09-17T18:35:29.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.397+0000 7f94d78ed640 10 monclient: build_initial_monmap 2024-09-17T18:35:29.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.397+0000 7f94d78ed640 10 monclient: monmap: 2024-09-17T18:35:29.412 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:35:29.412 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:35:29.412 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:35:29.412 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:35:29.412 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:35:29.413 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:29.413 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:35:29.413 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:35:29.413 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:35:29.413 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:29.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.397+0000 7f94d78ed640 1 librados: starting msgr at 2024-09-17T18:35:29.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.397+0000 7f94d78ed640 1 librados: starting objecter 2024-09-17T18:35:29.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.397+0000 7f94d78ed640 1 Processor -- start 2024-09-17T18:35:29.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.397+0000 7f94d78ed640 1 -- start start 2024-09-17T18:35:29.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.397+0000 7f94d78ed640 1 librados: setting wanted keys 2024-09-17T18:35:29.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.397+0000 7f94d78ed640 1 librados: calling monclient init 2024-09-17T18:35:29.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.397+0000 7f94d78ed640 10 monclient: init 2024-09-17T18:35:29.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.397+0000 7f94d78ed640 10 monclient: _reopen_session rank -1 2024-09-17T18:35:29.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.397+0000 7f94d78ed640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T18:35:29.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.397+0000 7f94d78ed640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f94d00a5b90 0x7f94d01383e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:29.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.397+0000 7f94d78ed640 10 monclient(hunting): picked mon.noname-c con 0x7f94d00a5b90 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:35:29.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.397+0000 7f94d78ed640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94d00a6680 0x7f94d0138920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:29.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.397+0000 7f94d78ed640 10 monclient(hunting): picked mon.noname-b con 0x7f94d00a6680 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:35:29.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.397+0000 7f94d78ed640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94d00a71c0 0x7f94d013db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:29.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.397+0000 7f94d78ed640 10 monclient(hunting): picked mon.noname-a con 0x7f94d00a71c0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:35:29.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.397+0000 7f94d78ed640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:29.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.397+0000 7f94d78ed640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f94d00a8ba0 con 0x7f94d00a71c0 2024-09-17T18:35:29.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.397+0000 7f94d78ed640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:29.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.397+0000 7f94d78ed640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f94d00a8a20 con 0x7f94d00a6680 2024-09-17T18:35:29.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.397+0000 7f94d78ed640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:29.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.397+0000 7f94d78ed640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f94d0140700 con 0x7f94d00a5b90 2024-09-17T18:35:29.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.397+0000 7f94d78ed640 10 monclient(hunting): _renew_subs 2024-09-17T18:35:29.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.397+0000 7f94d78ed640 10 monclient(hunting): authenticate will time out at 1733.114921s 2024-09-17T18:35:29.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.397+0000 7f94d5689640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f94d00a5b90 0x7f94d01383e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:29.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.398+0000 7f94d5689640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f94d00a5b90 0x7f94d01383e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:53964/0 (socket says 172.21.15.121:53964) 2024-09-17T18:35:29.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.398+0000 7f94d5689640 1 -- 172.21.15.121:0/2679654835 learned_addr learned my addr 172.21.15.121:0/2679654835 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:35:29.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.398+0000 7f94d5e8a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94d00a71c0 0x7f94d013db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:29.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.398+0000 7f94d5689640 10 monclient(hunting): get_auth_request con 0x7f94d00a5b90 auth_method 0 2024-09-17T18:35:29.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.398+0000 7f94d5689640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:29.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.398+0000 7f94d5689640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:29.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.398+0000 7f94d5689640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:29.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.398+0000 7f94d5e8a640 10 monclient(hunting): get_auth_request con 0x7f94d00a71c0 auth_method 0 2024-09-17T18:35:29.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.398+0000 7f94d5e8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:29.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.398+0000 7f94d5e8a640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:29.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.398+0000 7f94d5e8a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:29.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.398+0000 7f94d5689640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:29.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.398+0000 7f94d5689640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:29.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.398+0000 7f94d5689640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:29.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.398+0000 7f94d5e8a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:29.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.398+0000 7f94d5e8a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:29.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.398+0000 7f94d4e88640 1 --2- 172.21.15.121:0/2679654835 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94d00a6680 0x7f94d0138920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:29.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.398+0000 7f94d5e8a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:29.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.398+0000 7f94d4e88640 10 monclient(hunting): get_auth_request con 0x7f94d00a6680 auth_method 0 2024-09-17T18:35:29.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.398+0000 7f94d4e88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:29.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.398+0000 7f94d4e88640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:29.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.398+0000 7f94d4e88640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:29.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.398+0000 7f94d4e88640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:29.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.398+0000 7f94d4e88640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:29.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.398+0000 7f94d4e88640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:29.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.398+0000 7f94d5689640 10 monclient(hunting): handle_auth_done global_id 64514 payload 824 2024-09-17T18:35:29.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.398+0000 7f94d5689640 1 -- 172.21.15.121:0/2679654835 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94d00a6680 msgr2=0x7f94d0138920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:29.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.398+0000 7f94d5689640 1 --2- 172.21.15.121:0/2679654835 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94d00a6680 0x7f94d0138920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:29.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.398+0000 7f94d5689640 1 -- 172.21.15.121:0/2679654835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94d00a71c0 msgr2=0x7f94d013db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:29.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.398+0000 7f94d5689640 1 --2- 172.21.15.121:0/2679654835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94d00a71c0 0x7f94d013db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:29.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.398+0000 7f94d5689640 10 monclient: _finish_hunting 0 2024-09-17T18:35:29.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.398+0000 7f94d5689640 1 monclient: found mon.noname-c 2024-09-17T18:35:29.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.398+0000 7f94d5689640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:35:29.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.398+0000 7f94d5689640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:35:29.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.398+0000 7f94d5689640 1 -- 172.21.15.121:0/2679654835 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94d0123040 con 0x7f94d00a5b90 2024-09-17T18:35:29.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.398+0000 7f94d5e8a640 1 --2- 172.21.15.121:0/2679654835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94d00a71c0 0x7f94d013db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:35:29.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.400+0000 7f94d5689640 1 -- 172.21.15.121:0/2679654835 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f94d00a5b90 msgr2=0x7f94d01383e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:29.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.400+0000 7f94d5689640 1 --2- 172.21.15.121:0/2679654835 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f94d00a5b90 0x7f94d01383e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f94cc00c430 tx=0x7f94cc00c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:29.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.400+0000 7f94c5ffb640 10 client.?.objecter ms_handle_connect 0x7f94d00a5b90 2024-09-17T18:35:29.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.400+0000 7f94c5ffb640 10 client.?.objecter resend_mon_ops 2024-09-17T18:35:29.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.400+0000 7f94c5ffb640 1 -- 172.21.15.121:0/2679654835 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94cc0119a0 con 0x7f94d00a5b90 2024-09-17T18:35:29.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.400+0000 7f94c5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:29.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.400+0000 7f94c5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:35:29.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.400+0000 7f94c5ffb640 10 monclient: dump: 2024-09-17T18:35:29.417 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:29.417 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:29.417 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.418 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.418 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:29.418 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:29.418 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:29.418 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:29.418 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:29.418 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:29.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.400+0000 7f94c5ffb640 10 monclient: _finish_auth 0 2024-09-17T18:35:29.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.400+0000 7f94c5ffb640 10 monclient: _check_auth_tickets 2024-09-17T18:35:29.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.400+0000 7f94c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:35:29.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.400+0000 7f94c5ffb640 1 -- 172.21.15.121:0/2679654835 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94cc017070 con 0x7f94d00a5b90 2024-09-17T18:35:29.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.400+0000 7f94d78ed640 5 monclient: authenticate success, global_id 64514 2024-09-17T18:35:29.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.400+0000 7f94d78ed640 10 monclient: _renew_subs 2024-09-17T18:35:29.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.400+0000 7f94d78ed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:29.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.400+0000 7f94d78ed640 1 -- 172.21.15.121:0/2679654835 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94d0123320 con 0x7f94d00a5b90 2024-09-17T18:35:29.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.400+0000 7f94c5ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:35:29.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.400+0000 7f94c5ffb640 1 -- 172.21.15.121:0/2679654835 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94cc011d80 con 0x7f94d00a5b90 2024-09-17T18:35:29.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.400+0000 7f94c5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:29.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.400+0000 7f94c5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:35:29.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.400+0000 7f94c5ffb640 10 monclient: dump: 2024-09-17T18:35:29.419 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:29.419 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:29.419 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.419 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.419 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:29.419 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:29.419 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:29.419 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:29.419 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:29.419 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:29.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.400+0000 7f94c77fe640 4 set_mon_vals no callback set 2024-09-17T18:35:29.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.400+0000 7f94d78ed640 10 client.64514.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:35:29.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.400+0000 7f94d78ed640 10 monclient: _renew_subs 2024-09-17T18:35:29.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.400+0000 7f94d78ed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:29.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.400+0000 7f94d78ed640 1 -- 172.21.15.121:0/2679654835 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f94d0122d60 con 0x7f94d00a5b90 2024-09-17T18:35:29.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.400+0000 7f94d78ed640 1 librados: init done 2024-09-17T18:35:29.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.401+0000 7f94d78ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:29.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.401+0000 7f94d78ed640 1 --2- 172.21.15.121:0/2679654835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94d013e0a0 0x7f94d013e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:29.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.401+0000 7f94d78ed640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:29.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.401+0000 7f94d78ed640 1 -- 172.21.15.121:0/2679654835 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f94d0140210 con 0x7f94d013e0a0 2024-09-17T18:35:29.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.401+0000 7f94c5ffb640 1 -- 172.21.15.121:0/2679654835 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f94cc01c460 con 0x7f94d00a5b90 2024-09-17T18:35:29.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.401+0000 7f94d4e88640 1 --2- 172.21.15.121:0/2679654835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94d013e0a0 0x7f94d013e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:29.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.401+0000 7f94d4e88640 10 monclient: get_auth_request con 0x7f94d013e0a0 auth_method 0 2024-09-17T18:35:29.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.401+0000 7f94d4e88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:29.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.401+0000 7f94d4e88640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:29.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.401+0000 7f94d4e88640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:29.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.402+0000 7f94d4e88640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:29.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.402+0000 7f94d4e88640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:29.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.402+0000 7f94d4e88640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:29.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.402+0000 7f94c5ffb640 1 --2- 172.21.15.121:0/2679654835 >> v2:172.21.15.173:6812/2423786037 conn(0x7f94b40401c0 0x7f94b4042680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:29.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.402+0000 7f94c5ffb640 1 -- 172.21.15.121:0/2679654835 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(646..646 src has 247..646) ==== 5925+0+0 (secure 0 0 0) 0x7f94cc053770 con 0x7f94d00a5b90 2024-09-17T18:35:29.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.402+0000 7f94c5ffb640 10 client.64514.objecter ms_dispatch 0x7f94d0000b90 osd_map(646..646 src has 247..646) 2024-09-17T18:35:29.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.402+0000 7f94c5ffb640 3 client.64514.objecter handle_osd_map got epochs [646,646] > 0 2024-09-17T18:35:29.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.402+0000 7f94c5ffb640 3 client.64514.objecter handle_osd_map decoding full epoch 646 2024-09-17T18:35:29.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.402+0000 7f94d4e88640 10 monclient(hunting): handle_auth_done global_id 104154 payload 824 2024-09-17T18:35:29.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.402+0000 7f94d4e88640 1 -- 172.21.15.121:0/2679654835 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f94d009f490 con 0x7f94d013e0a0 2024-09-17T18:35:29.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.402+0000 7f94c5ffb640 20 client.64514.objecter dump_active .. 0 homeless 2024-09-17T18:35:29.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.402+0000 7f94d5e8a640 1 --2- 172.21.15.121:0/2679654835 >> v2:172.21.15.173:6812/2423786037 conn(0x7f94b40401c0 0x7f94b4042680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:29.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.402+0000 7f94d5e8a640 10 monclient: get_auth_request con 0x7f94b40401c0 auth_method 0 2024-09-17T18:35:29.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.404+0000 7f94c5ffb640 10 client.64514.objecter ms_handle_connect 0x7f94d013e0a0 2024-09-17T18:35:29.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.404+0000 7f94d4e88640 1 -- 172.21.15.121:0/2679654835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94d013e0a0 msgr2=0x7f94d013e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:29.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.404+0000 7f94c5ffb640 10 client.64514.objecter resend_mon_ops 2024-09-17T18:35:29.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.404+0000 7f94d4e88640 1 --2- 172.21.15.121:0/2679654835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94d013e0a0 0x7f94d013e480 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f94d0121ab0 tx=0x7f94c0011d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:29.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.404+0000 7f94c5ffb640 1 -- 172.21.15.121:0/2679654835 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94c0019070 con 0x7f94d013e0a0 2024-09-17T18:35:29.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.404+0000 7f94c5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:29.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.404+0000 7f94c5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:35:29.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.404+0000 7f94c5ffb640 10 monclient: dump: 2024-09-17T18:35:29.422 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:29.422 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:29.422 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.422 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.422 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:29.422 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:29.422 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:29.422 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:29.422 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:29.423 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:29.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.404+0000 7f94c5ffb640 1 -- 172.21.15.121:0/2679654835 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f94c000f890 con 0x7f94d013e0a0 2024-09-17T18:35:29.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.404+0000 7f94c5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:29.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.404+0000 7f94c5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:35:29.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.404+0000 7f94c5ffb640 1 -- 172.21.15.121:0/2679654835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94d013e0a0 msgr2=0x7f94d013e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:29.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.404+0000 7f94c5ffb640 1 --2- 172.21.15.121:0/2679654835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94d013e0a0 0x7f94d013e480 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f94d0121ab0 tx=0x7f94c0011d40 comp rx=0 tx=0).stop 2024-09-17T18:35:29.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.404+0000 7f94c5ffb640 1 -- 172.21.15.121:0/2679654835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94d013e0a0 msgr2=0x7f94d013e480 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:35:29.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.404+0000 7f94c5ffb640 1 --2- 172.21.15.121:0/2679654835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94d013e0a0 0x7f94d013e480 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:29.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.404+0000 7f94c5ffb640 10 client.64514.objecter ms_handle_connect 0x7f94b40401c0 2024-09-17T18:35:29.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.404+0000 7f94d5e8a640 1 --2- 172.21.15.121:0/2679654835 >> v2:172.21.15.173:6812/2423786037 conn(0x7f94b40401c0 0x7f94b4042680 secure :-1 s=READY pgs=4079 cs=0 l=1 rev1=1 crypto rx=0x7f94d00770d0 tx=0x7f94c8009000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:29.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.411+0000 7f94d78ed640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-17T18:35:29.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.411+0000 7f94d78ed640 1 --2- 172.21.15.121:0/2679654835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94d013e9c0 0x7f94d0139c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:29.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.411+0000 7f94d78ed640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:29.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.411+0000 7f94d78ed640 1 -- 172.21.15.121:0/2679654835 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f94d00a5f70 con 0x7f94d013e9c0 2024-09-17T18:35:29.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.411+0000 7f94d4e88640 1 --2- 172.21.15.121:0/2679654835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94d013e9c0 0x7f94d0139c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:29.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.411+0000 7f94d4e88640 10 monclient: get_auth_request con 0x7f94d013e9c0 auth_method 0 2024-09-17T18:35:29.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.411+0000 7f94d4e88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:29.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.411+0000 7f94d4e88640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:29.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.411+0000 7f94d4e88640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:29.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.412+0000 7f94d4e88640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:29.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.412+0000 7f94d4e88640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:29.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.412+0000 7f94d4e88640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:29.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.412+0000 7f94d4e88640 10 monclient(hunting): handle_auth_done global_id 104157 payload 824 2024-09-17T18:35:29.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.412+0000 7f94d4e88640 1 -- 172.21.15.121:0/2679654835 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7f94d006cd40 con 0x7f94d013e9c0 2024-09-17T18:35:29.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.414+0000 7f94c5ffb640 10 client.64514.objecter ms_handle_connect 0x7f94d013e9c0 2024-09-17T18:35:29.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.414+0000 7f94d4e88640 1 -- 172.21.15.121:0/2679654835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94d013e9c0 msgr2=0x7f94d0139c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:29.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.414+0000 7f94c5ffb640 10 client.64514.objecter resend_mon_ops 2024-09-17T18:35:29.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.414+0000 7f94d4e88640 1 --2- 172.21.15.121:0/2679654835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94d013e9c0 0x7f94d0139c50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f94d0121ab0 tx=0x7f94c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:29.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.414+0000 7f94c5ffb640 1 -- 172.21.15.121:0/2679654835 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94c002e080 con 0x7f94d013e9c0 2024-09-17T18:35:29.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.414+0000 7f94c5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:29.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.414+0000 7f94c5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:35:29.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.414+0000 7f94c5ffb640 10 monclient: dump: 2024-09-17T18:35:29.426 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:29.426 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:29.426 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.426 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.426 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:29.427 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:29.427 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:29.427 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:29.427 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:29.427 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:29.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.414+0000 7f94c5ffb640 1 -- 172.21.15.121:0/2679654835 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (secure 0 0 0) 0x7f94c001e660 con 0x7f94d013e9c0 2024-09-17T18:35:29.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.414+0000 7f94c5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-17T18:35:29.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.414+0000 7f94c5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:35:29.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.414+0000 7f94c5ffb640 1 -- 172.21.15.121:0/2679654835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94d013e9c0 msgr2=0x7f94d0139c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:29.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.414+0000 7f94c5ffb640 1 --2- 172.21.15.121:0/2679654835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94d013e9c0 0x7f94d0139c50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f94d0121ab0 tx=0x7f94c00046c0 comp rx=0 tx=0).stop 2024-09-17T18:35:29.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.414+0000 7f94c5ffb640 1 -- 172.21.15.121:0/2679654835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94d013e9c0 msgr2=0x7f94d0139c50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:35:29.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.414+0000 7f94c5ffb640 1 --2- 172.21.15.121:0/2679654835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94d013e9c0 0x7f94d0139c50 secure :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f94d0121ab0 tx=0x7f94c00046c0 comp rx=0 tx=0).stop 2024-09-17T18:35:29.428 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:35:29.428 INFO:teuthology.orchestra.run.smithi121.stdout: "ret": 0, 2024-09-17T18:35:29.428 INFO:teuthology.orchestra.run.smithi121.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-09-17T18:35:29.428 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:35:29.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.415+0000 7f94d78ed640 10 librados: watch_flush enter 2024-09-17T18:35:29.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.415+0000 7f94d78ed640 10 librados: watch_flush exit 2024-09-17T18:35:29.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.415+0000 7f94d78ed640 20 client.64514.objecter shutdown clearing up homeless session... 2024-09-17T18:35:29.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.415+0000 7f94d78ed640 10 client.64514.objecter successfully canceled tick 2024-09-17T18:35:29.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.415+0000 7f94d78ed640 1 -- 172.21.15.121:0/2679654835 >> v2:172.21.15.173:6812/2423786037 conn(0x7f94b40401c0 msgr2=0x7f94b4042680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:29.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.415+0000 7f94d78ed640 1 --2- 172.21.15.121:0/2679654835 >> v2:172.21.15.173:6812/2423786037 conn(0x7f94b40401c0 0x7f94b4042680 secure :-1 s=READY pgs=4079 cs=0 l=1 rev1=1 crypto rx=0x7f94d00770d0 tx=0x7f94c8009000 comp rx=0 tx=0).stop 2024-09-17T18:35:29.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.415+0000 7f94d78ed640 10 monclient: shutdown 2024-09-17T18:35:29.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.415+0000 7f94d78ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:35:29.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.415+0000 7f94d78ed640 1 -- 172.21.15.121:0/2679654835 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f94d00a5b90 msgr2=0x7f94d01383e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:29.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.415+0000 7f94d78ed640 1 --2- 172.21.15.121:0/2679654835 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f94d00a5b90 0x7f94d01383e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f94cc00c430 tx=0x7f94cc00c8f0 comp rx=0 tx=0).stop 2024-09-17T18:35:29.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.415+0000 7f94d5e8a640 1 -- 172.21.15.121:0/2679654835 reap_dead start 2024-09-17T18:35:29.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.415+0000 7f94d5e8a640 1 -- 172.21.15.121:0/2679654835 reap_dead start 2024-09-17T18:35:29.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.415+0000 7f94d78ed640 1 -- 172.21.15.121:0/2679654835 shutdown_connections 2024-09-17T18:35:29.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.415+0000 7f94d78ed640 1 -- 172.21.15.121:0/2679654835 >> 172.21.15.121:0/2679654835 conn(0x7f94d00a7c90 msgr2=0x7f94d00a13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:35:29.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.415+0000 7f94d78ed640 1 -- 172.21.15.121:0/2679654835 shutdown_connections 2024-09-17T18:35:29.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.415+0000 7f94d78ed640 1 -- 172.21.15.121:0/2679654835 wait complete. 2024-09-17T18:35:29.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.415+0000 7f94d78ed640 1 librados: shutdown 2024-09-17T18:35:29.431 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-17T18:35:29.431 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-17T18:35:29.431 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T18:35:29.441 INFO:tasks.ceph.mon.a:Stopped 2024-09-17T18:35:29.441 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-17T18:35:29.441 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:35:29.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.575+0000 7f9aaf3ec640 10 monclient: get_monmap_and_config 2024-09-17T18:35:29.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.575+0000 7f9aaf3ec640 10 monclient: build_initial_monmap 2024-09-17T18:35:29.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.575+0000 7f9aaf3ec640 10 monclient: monmap: 2024-09-17T18:35:29.578 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:35:29.578 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:29.578 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:35:29.576823+0000 2024-09-17T18:35:29.578 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:35:29.576823+0000 2024-09-17T18:35:29.579 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:35:29.579 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:29.579 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:35:29.579 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:35:29.579 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:35:29.579 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:35:29.579 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:35:29.579 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:29.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.575+0000 7f9aaf3ec640 1 Processor -- start 2024-09-17T18:35:29.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.576+0000 7f9aaf3ec640 1 -- start start 2024-09-17T18:35:29.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.576+0000 7f9aaf3ec640 10 monclient: init 2024-09-17T18:35:29.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.576+0000 7f9aaf3ec640 10 monclient: _reopen_session rank -1 2024-09-17T18:35:29.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.576+0000 7f9aaf3ec640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-17T18:35:29.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.576+0000 7f9aaf3ec640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9aa80a5c00 0x7f9aa809d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:29.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.576+0000 7f9aaf3ec640 10 monclient(hunting): picked mon.noname-b con 0x7f9aa80a5c00 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:35:29.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.576+0000 7f9aaf3ec640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9aa80a5fe0 addr v1:172.21.15.173:6789/0 2024-09-17T18:35:29.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.576+0000 7f9aaf3ec640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9aa813dc00 addr v1:172.21.15.121:6789/0 2024-09-17T18:35:29.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.576+0000 7f9aaf3ec640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9aa80a8a80 con 0x7f9aa813dc00 2024-09-17T18:35:29.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.576+0000 7f9aaf3ec640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9aa809de00 con 0x7f9aa80a5fe0 2024-09-17T18:35:29.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.576+0000 7f9aaf3ec640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:29.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.576+0000 7f9aaf3ec640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9aa80a7110 con 0x7f9aa80a5c00 2024-09-17T18:35:29.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.576+0000 7f9aaf3ec640 10 monclient(hunting): _renew_subs 2024-09-17T18:35:29.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.576+0000 7f9aaf3ec640 10 monclient(hunting): authenticate will time out at 1733.293634s 2024-09-17T18:35:29.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.576+0000 7f9aad989640 1 -- >> v1:172.21.15.121:6789/0 conn(0x7f9aa813dc00 legacy=0x7f9aa813fff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.121:6789/0 2024-09-17T18:35:29.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.576+0000 7f9aad188640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9aa80a5c00 0x7f9aa809d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:29.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.576+0000 7f9aad188640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9aa80a5c00 0x7f9aa809d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:53972/0 (socket says 172.21.15.121:53972) 2024-09-17T18:35:29.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.576+0000 7f9aad188640 1 -- 172.21.15.121:0/934230342 learned_addr learned my addr 172.21.15.121:0/934230342 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:35:29.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.576+0000 7f9aad188640 10 monclient(hunting): get_auth_request con 0x7f9aa80a5c00 auth_method 0 2024-09-17T18:35:29.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.576+0000 7f9aad188640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:29.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.576+0000 7f9aad188640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:29.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.576+0000 7f9aad188640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:29.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.576+0000 7f9aad188640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:29.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.576+0000 7f9aad188640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:29.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.576+0000 7f9aad188640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:29.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.576+0000 7f9aad188640 10 monclient(hunting): handle_auth_done global_id 64517 payload 274 2024-09-17T18:35:29.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.576+0000 7f9aad188640 1 -- 172.21.15.121:0/934230342 >> v1:172.21.15.173:6789/0 conn(0x7f9aa80a5fe0 legacy=0x7f9aa80a63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:35:29.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.576+0000 7f9aad188640 1 -- 172.21.15.121:0/934230342 >> v1:172.21.15.121:6789/0 conn(0x7f9aa813dc00 legacy=0x7f9aa813fff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:35:29.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.576+0000 7f9aad188640 10 monclient: _finish_hunting 0 2024-09-17T18:35:29.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.576+0000 7f9aad188640 1 monclient: found mon.noname-b 2024-09-17T18:35:29.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.576+0000 7f9aad188640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:35:29.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.576+0000 7f9aad188640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:35:29.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.576+0000 7f9aad188640 1 -- 172.21.15.121:0/934230342 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aa80a74d0 con 0x7f9aa80a5c00 2024-09-17T18:35:29.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.579+0000 7f9aad188640 1 -- 172.21.15.121:0/934230342 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9aa80a5c00 msgr2=0x7f9aa809d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:29.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.579+0000 7f9aad188640 1 --2- 172.21.15.121:0/934230342 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9aa80a5c00 0x7f9aa809d8c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c009870 tx=0x7f9a9c02a050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e43ac70b0cdbe66c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:29.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.579+0000 7f9a9bfff640 1 -- 172.21.15.121:0/934230342 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c037070 con 0x7f9aa80a5c00 2024-09-17T18:35:29.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.579+0000 7f9a9bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:29.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.579+0000 7f9a9bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:35:29.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.579+0000 7f9a9bfff640 10 monclient: dump: 2024-09-17T18:35:29.589 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:29.589 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:29.589 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.589 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.589 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:29.590 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:29.590 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:29.590 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:29.590 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:29.590 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:29.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.579+0000 7f9a9bfff640 10 monclient: _finish_auth 0 2024-09-17T18:35:29.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.579+0000 7f9a9bfff640 10 monclient: _check_auth_tickets 2024-09-17T18:35:29.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.579+0000 7f9a9bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:35:29.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.579+0000 7f9aaf3ec640 5 monclient: authenticate success, global_id 64517 2024-09-17T18:35:29.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.579+0000 7f9aaf3ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:35:29.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.579+0000 7f9a9bfff640 1 -- 172.21.15.121:0/934230342 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a9c032640 con 0x7f9aa80a5c00 2024-09-17T18:35:29.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.579+0000 7f9a9bfff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:35:29.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.579+0000 7f9a9bfff640 1 -- 172.21.15.121:0/934230342 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c032a10 con 0x7f9aa80a5c00 2024-09-17T18:35:29.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.579+0000 7f9aaf3ec640 10 monclient: get_monmap_and_config success 2024-09-17T18:35:29.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.579+0000 7f9aaf3ec640 4 set_mon_vals no callback set 2024-09-17T18:35:29.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.579+0000 7f9a9bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:29.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.579+0000 7f9a9bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:35:29.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.579+0000 7f9a9bfff640 10 monclient: dump: 2024-09-17T18:35:29.591 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:29.591 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:29.591 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.591 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.591 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:29.591 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:29.591 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:29.591 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:29.591 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:29.591 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:29.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.580+0000 7f9aaf3ec640 10 monclient: shutdown 2024-09-17T18:35:29.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.580+0000 7f9aaf3ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:35:29.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.580+0000 7f9aaf3ec640 1 -- 172.21.15.121:0/934230342 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9aa80a5c00 msgr2=0x7f9aa809d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:29.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.580+0000 7f9aaf3ec640 1 --2- 172.21.15.121:0/934230342 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9aa80a5c00 0x7f9aa809d8c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c009870 tx=0x7f9a9c02a050 comp rx=0 tx=0).stop 2024-09-17T18:35:29.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.580+0000 7f9aaf3ec640 1 -- 172.21.15.121:0/934230342 shutdown_connections 2024-09-17T18:35:29.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.580+0000 7f9aaf3ec640 1 --2- 172.21.15.121:0/934230342 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9aa80a5c00 0x7f9aa809d8c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:29.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.580+0000 7f9aaf3ec640 1 -- 172.21.15.121:0/934230342 >> 172.21.15.121:0/934230342 conn(0x7f9aa80ab9b0 msgr2=0x7f9aa80a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:35:29.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.580+0000 7f9aaf3ec640 1 -- 172.21.15.121:0/934230342 shutdown_connections 2024-09-17T18:35:29.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.580+0000 7f9aaf3ec640 1 -- 172.21.15.121:0/934230342 wait complete. 2024-09-17T18:35:29.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.581+0000 7f9aaf3ec640 10 monclient: build_initial_monmap 2024-09-17T18:35:29.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.581+0000 7f9aaf3ec640 10 monclient: monmap: 2024-09-17T18:35:29.592 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:35:29.592 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:35:29.593 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:35:29.593 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:35:29.593 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:35:29.593 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:29.593 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:35:29.593 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:35:29.593 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:35:29.593 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:29.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.581+0000 7f9aaf3ec640 1 librados: starting msgr at 2024-09-17T18:35:29.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.581+0000 7f9aaf3ec640 1 librados: starting objecter 2024-09-17T18:35:29.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.581+0000 7f9aaf3ec640 1 Processor -- start 2024-09-17T18:35:29.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.581+0000 7f9aaf3ec640 1 -- start start 2024-09-17T18:35:29.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.581+0000 7f9aaf3ec640 1 librados: setting wanted keys 2024-09-17T18:35:29.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.581+0000 7f9aaf3ec640 1 librados: calling monclient init 2024-09-17T18:35:29.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.581+0000 7f9aaf3ec640 10 monclient: init 2024-09-17T18:35:29.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.581+0000 7f9aaf3ec640 10 monclient: _reopen_session rank -1 2024-09-17T18:35:29.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.581+0000 7f9aaf3ec640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T18:35:29.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.581+0000 7f9aaf3ec640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9aa80a5c00 0x7f9aa81381a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:29.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.581+0000 7f9aaf3ec640 10 monclient(hunting): picked mon.noname-c con 0x7f9aa80a5c00 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:35:29.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.581+0000 7f9aaf3ec640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9aa80a5fe0 0x7f9aa81386e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:29.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.581+0000 7f9aaf3ec640 10 monclient(hunting): picked mon.noname-a con 0x7f9aa80a5fe0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:35:29.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.581+0000 7f9aaf3ec640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9aa813dc00 0x7f9aa813a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:29.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.581+0000 7f9aaf3ec640 10 monclient(hunting): picked mon.noname-b con 0x7f9aa813dc00 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:35:29.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.581+0000 7f9aaf3ec640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:29.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.581+0000 7f9aaf3ec640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa80703d0 con 0x7f9aa80a5fe0 2024-09-17T18:35:29.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.581+0000 7f9aaf3ec640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:29.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.581+0000 7f9aaf3ec640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa8070550 con 0x7f9aa813dc00 2024-09-17T18:35:29.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.581+0000 7f9aaf3ec640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:29.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.581+0000 7f9aaf3ec640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9aa8141710 con 0x7f9aa80a5c00 2024-09-17T18:35:29.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.581+0000 7f9aaf3ec640 10 monclient(hunting): _renew_subs 2024-09-17T18:35:29.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.581+0000 7f9aaf3ec640 10 monclient(hunting): authenticate will time out at 1733.298943s 2024-09-17T18:35:29.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.581+0000 7f9aac987640 1 -- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9aa80a5fe0 msgr2=0x7f9aa81386e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:35:29.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.581+0000 7f9aac987640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9aa80a5fe0 0x7f9aa81386e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:35:29.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.581+0000 7f9aad188640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9aa80a5c00 0x7f9aa81381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:29.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.582+0000 7f9aad188640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9aa80a5c00 0x7f9aa81381a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:53988/0 (socket says 172.21.15.121:53988) 2024-09-17T18:35:29.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.582+0000 7f9aad188640 1 -- 172.21.15.121:0/1607213043 learned_addr learned my addr 172.21.15.121:0/1607213043 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:35:29.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.582+0000 7f9aad188640 10 monclient(hunting): get_auth_request con 0x7f9aa80a5c00 auth_method 0 2024-09-17T18:35:29.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.582+0000 7f9aad188640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:29.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.582+0000 7f9aad188640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:29.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.582+0000 7f9aad188640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:29.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.582+0000 7f9aad188640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:29.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.582+0000 7f9aad188640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:29.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.582+0000 7f9aad188640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:29.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.582+0000 7f9aad188640 10 monclient(hunting): handle_auth_done global_id 64520 payload 824 2024-09-17T18:35:29.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.582+0000 7f9aad989640 1 --2- 172.21.15.121:0/1607213043 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9aa813dc00 0x7f9aa813a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:29.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.582+0000 7f9aad188640 1 -- 172.21.15.121:0/1607213043 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9aa813dc00 msgr2=0x7f9aa813a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:29.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.582+0000 7f9aad188640 1 --2- 172.21.15.121:0/1607213043 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9aa813dc00 0x7f9aa813a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:29.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.582+0000 7f9aad188640 1 -- 172.21.15.121:0/1607213043 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9aa80a5fe0 msgr2=0x7f9aa81386e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:35:29.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.582+0000 7f9aad188640 1 --2- 172.21.15.121:0/1607213043 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9aa80a5fe0 0x7f9aa81386e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:29.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.582+0000 7f9aad188640 10 monclient: _finish_hunting 0 2024-09-17T18:35:29.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.582+0000 7f9aad188640 1 monclient: found mon.noname-c 2024-09-17T18:35:29.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.582+0000 7f9aad188640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:35:29.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.582+0000 7f9aad188640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:35:29.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.582+0000 7f9aad188640 1 -- 172.21.15.121:0/1607213043 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aa8121bd0 con 0x7f9aa80a5c00 2024-09-17T18:35:29.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.584+0000 7f9aad188640 1 -- 172.21.15.121:0/1607213043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9aa80a5c00 msgr2=0x7f9aa81381a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:29.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.584+0000 7f9aad188640 1 --2- 172.21.15.121:0/1607213043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9aa80a5c00 0x7f9aa81381a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c0090b0 tx=0x7f9a9c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:29.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.584+0000 7f9a99ffb640 10 client.?.objecter ms_handle_connect 0x7f9aa80a5c00 2024-09-17T18:35:29.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.584+0000 7f9a99ffb640 10 client.?.objecter resend_mon_ops 2024-09-17T18:35:29.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.584+0000 7f9a99ffb640 1 -- 172.21.15.121:0/1607213043 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c041070 con 0x7f9aa80a5c00 2024-09-17T18:35:29.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.584+0000 7f9a99ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:29.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.584+0000 7f9a99ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:35:29.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.584+0000 7f9a99ffb640 10 monclient: dump: 2024-09-17T18:35:29.597 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:29.597 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:29.597 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.597 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.597 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:29.597 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:29.597 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:29.597 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:29.597 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:29.597 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:29.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.584+0000 7f9a99ffb640 10 monclient: _finish_auth 0 2024-09-17T18:35:29.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.584+0000 7f9a99ffb640 10 monclient: _check_auth_tickets 2024-09-17T18:35:29.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.584+0000 7f9a99ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:35:29.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.584+0000 7f9a99ffb640 1 -- 172.21.15.121:0/1607213043 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a9c03ca80 con 0x7f9aa80a5c00 2024-09-17T18:35:29.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.584+0000 7f9aaf3ec640 5 monclient: authenticate success, global_id 64520 2024-09-17T18:35:29.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.584+0000 7f9aaf3ec640 10 monclient: _renew_subs 2024-09-17T18:35:29.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.584+0000 7f9aaf3ec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:29.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.584+0000 7f9aaf3ec640 1 -- 172.21.15.121:0/1607213043 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9aa8121eb0 con 0x7f9aa80a5c00 2024-09-17T18:35:29.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.584+0000 7f9a99ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:35:29.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.584+0000 7f9a99ffb640 1 -- 172.21.15.121:0/1607213043 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c037040 con 0x7f9aa80a5c00 2024-09-17T18:35:29.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.584+0000 7f9a99ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:29.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.584+0000 7f9a99ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:35:29.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.584+0000 7f9a99ffb640 10 monclient: dump: 2024-09-17T18:35:29.598 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:29.598 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:29.599 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.599 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:29.599 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:29.599 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:29.599 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:29.599 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:29.599 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:29.599 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:29.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.584+0000 7f9a9b7fe640 4 set_mon_vals no callback set 2024-09-17T18:35:29.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.584+0000 7f9aaf3ec640 10 client.64520.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:35:29.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.584+0000 7f9aaf3ec640 10 monclient: _renew_subs 2024-09-17T18:35:29.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.584+0000 7f9aaf3ec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:29.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.584+0000 7f9aaf3ec640 1 -- 172.21.15.121:0/1607213043 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9aa81218f0 con 0x7f9aa80a5c00 2024-09-17T18:35:29.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.584+0000 7f9aaf3ec640 1 librados: init done 2024-09-17T18:35:29.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.585+0000 7f9a99ffb640 1 -- 172.21.15.121:0/1607213043 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f9a9c044070 con 0x7f9aa80a5c00 2024-09-17T18:35:29.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.585+0000 7f9aaf3ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:29.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.585+0000 7f9aaf3ec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:29.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.585+0000 7f9aaf3ec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:29.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.585+0000 7f9aaf3ec640 1 -- 172.21.15.121:0/1607213043 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9aa809d8c0 con 0x7f9aa80a5c00 2024-09-17T18:35:29.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.586+0000 7f9a99ffb640 1 --2- 172.21.15.121:0/1607213043 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9a8c0401a0 0x7f9a8c042660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:29.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.586+0000 7f9a99ffb640 1 -- 172.21.15.121:0/1607213043 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(646..646 src has 247..646) ==== 5925+0+0 (secure 0 0 0) 0x7f9a9c074250 con 0x7f9aa80a5c00 2024-09-17T18:35:29.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.586+0000 7f9a99ffb640 10 client.64520.objecter ms_dispatch 0x7f9aa8000b90 osd_map(646..646 src has 247..646) 2024-09-17T18:35:29.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.586+0000 7f9a99ffb640 3 client.64520.objecter handle_osd_map got epochs [646,646] > 0 2024-09-17T18:35:29.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.586+0000 7f9a99ffb640 3 client.64520.objecter handle_osd_map decoding full epoch 646 2024-09-17T18:35:29.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.586+0000 7f9a99ffb640 20 client.64520.objecter dump_active .. 0 homeless 2024-09-17T18:35:29.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.586+0000 7f9aac987640 1 --2- 172.21.15.121:0/1607213043 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9a8c0401a0 0x7f9a8c042660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:29.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.586+0000 7f9aac987640 10 monclient: get_auth_request con 0x7f9a8c0401a0 auth_method 0 2024-09-17T18:35:29.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.588+0000 7f9a99ffb640 10 client.64520.objecter ms_handle_connect 0x7f9a8c0401a0 2024-09-17T18:35:29.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.588+0000 7f9aac987640 1 --2- 172.21.15.121:0/1607213043 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9a8c0401a0 0x7f9a8c042660 secure :-1 s=READY pgs=4080 cs=0 l=1 rev1=1 crypto rx=0x7f9aa813db50 tx=0x7f9aa4007a10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:29.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.590+0000 7f9a99ffb640 1 -- 172.21.15.121:0/1607213043 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9a9c03a360 con 0x7f9aa80a5c00 2024-09-17T18:35:29.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.590+0000 7f9a99ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:29.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.590+0000 7f9a99ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:35:29.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.830+0000 7f9aaf3ec640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:35:29.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.830+0000 7f9aaf3ec640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:35:29.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.830+0000 7f9aaf3ec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:29.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.830+0000 7f9aaf3ec640 1 -- 172.21.15.121:0/1607213043 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9aa813cf80 con 0x7f9aa80a5c00 2024-09-17T18:35:29.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.830+0000 7f9a99ffb640 1 -- 172.21.15.121:0/1607213043 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f9a9c03ce70 con 0x7f9aa80a5c00 2024-09-17T18:35:29.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.830+0000 7f9a99ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:35:29.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.830+0000 7f9a99ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:35:29.833 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":60,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:35:29.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.832+0000 7f9aaf3ec640 10 librados: watch_flush enter 2024-09-17T18:35:29.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.832+0000 7f9aaf3ec640 10 librados: watch_flush exit 2024-09-17T18:35:29.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.832+0000 7f9aaf3ec640 20 client.64520.objecter shutdown clearing up homeless session... 2024-09-17T18:35:29.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.832+0000 7f9aaf3ec640 10 client.64520.objecter successfully canceled tick 2024-09-17T18:35:29.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.832+0000 7f9aaf3ec640 1 -- 172.21.15.121:0/1607213043 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9a8c0401a0 msgr2=0x7f9a8c042660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:29.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.832+0000 7f9aaf3ec640 1 --2- 172.21.15.121:0/1607213043 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9a8c0401a0 0x7f9a8c042660 secure :-1 s=READY pgs=4080 cs=0 l=1 rev1=1 crypto rx=0x7f9aa813db50 tx=0x7f9aa4007a10 comp rx=0 tx=0).stop 2024-09-17T18:35:29.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.832+0000 7f9aaf3ec640 10 monclient: shutdown 2024-09-17T18:35:29.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.832+0000 7f9aaf3ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:35:29.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.832+0000 7f9aaf3ec640 1 -- 172.21.15.121:0/1607213043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9aa80a5c00 msgr2=0x7f9aa81381a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:29.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.832+0000 7f9aaf3ec640 1 --2- 172.21.15.121:0/1607213043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9aa80a5c00 0x7f9aa81381a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c0090b0 tx=0x7f9a9c0046c0 comp rx=0 tx=0).stop 2024-09-17T18:35:29.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.832+0000 7f9aaf3ec640 1 -- 172.21.15.121:0/1607213043 shutdown_connections 2024-09-17T18:35:29.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.832+0000 7f9aaf3ec640 1 --2- 172.21.15.121:0/1607213043 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9a8c0401a0 0x7f9a8c042660 unknown :-1 s=CLOSED pgs=4080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:29.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.832+0000 7f9aaf3ec640 1 --2- 172.21.15.121:0/1607213043 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9aa813dc00 0x7f9aa813a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:29.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.832+0000 7f9aaf3ec640 1 --2- 172.21.15.121:0/1607213043 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9aa80a5fe0 0x7f9aa81386e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:29.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.832+0000 7f9aaf3ec640 1 --2- 172.21.15.121:0/1607213043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9aa80a5c00 0x7f9aa81381a0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:29.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.832+0000 7f9aaf3ec640 1 -- 172.21.15.121:0/1607213043 >> 172.21.15.121:0/1607213043 conn(0x7f9aa80ab9b0 msgr2=0x7f9aa80a9dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:35:29.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.832+0000 7f9aaf3ec640 1 -- 172.21.15.121:0/1607213043 shutdown_connections 2024-09-17T18:35:29.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.832+0000 7f9aaf3ec640 1 -- 172.21.15.121:0/1607213043 wait complete. 2024-09-17T18:35:29.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:29.832+0000 7f9aaf3ec640 1 librados: shutdown 2024-09-17T18:35:32.846 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:35:32.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.951+0000 7f84fcbd3640 10 monclient: get_monmap_and_config 2024-09-17T18:35:32.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.951+0000 7f84fcbd3640 10 monclient: build_initial_monmap 2024-09-17T18:35:32.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.951+0000 7f84fcbd3640 10 monclient: monmap: 2024-09-17T18:35:32.954 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:35:32.954 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:32.955 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:35:32.953333+0000 2024-09-17T18:35:32.955 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:35:32.953333+0000 2024-09-17T18:35:32.955 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:35:32.955 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:32.955 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:35:32.955 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:35:32.955 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:35:32.955 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:35:32.955 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:35:32.955 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:32.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.952+0000 7f84fcbd3640 1 Processor -- start 2024-09-17T18:35:32.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.952+0000 7f84fcbd3640 1 -- start start 2024-09-17T18:35:32.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.952+0000 7f84fcbd3640 10 monclient: init 2024-09-17T18:35:32.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.953+0000 7f84fcbd3640 10 monclient: _reopen_session rank -1 2024-09-17T18:35:32.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.953+0000 7f84fcbd3640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-17T18:35:32.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.953+0000 7f84fcbd3640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f84f8070200 0x7f84f80a5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:32.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.953+0000 7f84fcbd3640 10 monclient(hunting): picked mon.noname-a con 0x7f84f8070200 addr v2:172.21.15.121:3300/0 2024-09-17T18:35:32.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.953+0000 7f84fcbd3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f84f80a61e0 addr v1:172.21.15.173:6789/0 2024-09-17T18:35:32.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.953+0000 7f84fcbd3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f84f80a65f0 addr v1:172.21.15.121:6789/0 2024-09-17T18:35:32.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.953+0000 7f84fcbd3640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84f80a5740 con 0x7f84f80a65f0 2024-09-17T18:35:32.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.953+0000 7f84fcbd3640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84f80a6dd0 con 0x7f84f80a61e0 2024-09-17T18:35:32.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.953+0000 7f84fcbd3640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:32.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.953+0000 7f84fcbd3640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f84f80705e0 con 0x7f84f8070200 2024-09-17T18:35:32.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.953+0000 7f84fcbd3640 10 monclient(hunting): _renew_subs 2024-09-17T18:35:32.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.953+0000 7f84fcbd3640 10 monclient(hunting): authenticate will time out at 1736.670429s 2024-09-17T18:35:32.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.953+0000 7f84f659c640 1 -- >> v2:172.21.15.121:3300/0 conn(0x7f84f8070200 msgr2=0x7f84f80a5ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:35:32.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.953+0000 7f84f6d9d640 1 -- >> v1:172.21.15.121:6789/0 conn(0x7f84f80a65f0 legacy=0x7f84f81401b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.121:6789/0 2024-09-17T18:35:32.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.953+0000 7f84f659c640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f84f8070200 0x7f84f80a5ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:35:32.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.953+0000 7f84f5d9b640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f84f80a61e0 0x7f84f813ca80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.121:55698/0 (socket says 172.21.15.121:55698) 2024-09-17T18:35:32.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.953+0000 7f84f5d9b640 1 -- 172.21.15.121:0/1798411920 learned_addr learned my addr 172.21.15.121:0/1798411920 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:35:32.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f5d9b640 1 -- 172.21.15.121:0/1798411920 >> v1:172.21.15.173:6789/0 conn(0x7f84f80a61e0 legacy=0x7f84f813ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:32.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f559a640 1 -- 172.21.15.121:0/1798411920 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4165860662 0 0) 0x7f84f80a6dd0 con 0x7f84f80a61e0 2024-09-17T18:35:32.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f559a640 10 monclient(hunting): _negotiate 2024-09-17T18:35:32.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f559a640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:32.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:32.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f559a640 10 monclient(hunting): my global_id is 84283 2024-09-17T18:35:32.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f559a640 1 -- 172.21.15.121:0/1798411920 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f84e0003520 con 0x7f84f80a61e0 2024-09-17T18:35:32.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f559a640 1 -- 172.21.15.121:0/1798411920 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2375378445 0 0) 0x7f84e0003520 con 0x7f84f80a61e0 2024-09-17T18:35:32.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f559a640 1 -- 172.21.15.121:0/1798411920 >> v2:172.21.15.121:3300/0 conn(0x7f84f8070200 msgr2=0x7f84f80a5ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:35:32.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f559a640 1 --2- 172.21.15.121:0/1798411920 >> v2:172.21.15.121:3300/0 conn(0x7f84f8070200 0x7f84f80a5ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:32.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f559a640 1 -- 172.21.15.121:0/1798411920 >> v1:172.21.15.121:6789/0 conn(0x7f84f80a65f0 legacy=0x7f84f81401b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:35:32.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f559a640 10 monclient: _finish_hunting 0 2024-09-17T18:35:32.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f559a640 1 monclient: found mon.noname-c-legacy 2024-09-17T18:35:32.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:35:32.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f559a640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.173:6789/0 2024-09-17T18:35:32.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f559a640 1 -- 172.21.15.121:0/1798411920 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84f80a2bf0 con 0x7f84f80a61e0 2024-09-17T18:35:32.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f559a640 10 monclient: _finish_auth 0 2024-09-17T18:35:32.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f559a640 10 monclient: _check_auth_tickets 2024-09-17T18:35:32.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:35:32.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f559a640 1 -- 172.21.15.121:0/1798411920 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4002019445 0 0) 0x7f84ec001d20 con 0x7f84f80a61e0 2024-09-17T18:35:32.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84fcbd3640 5 monclient: authenticate success, global_id 84283 2024-09-17T18:35:32.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84fcbd3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:35:32.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:32.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f559a640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-17T18:35:32.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f559a640 10 monclient: dump: 2024-09-17T18:35:32.963 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:32.964 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:32.964 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:32.964 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:32.964 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:32.964 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:32.964 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:32.964 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:32.964 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:32.964 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:32.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f559a640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] but i'm connected to v1:172.21.15.173:6789/0, reconnecting 2024-09-17T18:35:32.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f559a640 10 monclient: _reopen_session rank -1 2024-09-17T18:35:32.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f559a640 1 -- 172.21.15.121:0/1798411920 >> v1:172.21.15.173:6789/0 conn(0x7f84f80a61e0 legacy=0x7f84f813ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:32.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f559a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:35:32.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f559a640 1 --2- 172.21.15.121:0/1798411920 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f84e0003a50 0x7f84e0003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:32.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f559a640 10 monclient(hunting): picked mon.a con 0x7f84e0003a50 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:35:32.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f559a640 1 --2- 172.21.15.121:0/1798411920 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f84e0004800 0x7f84e0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:32.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f559a640 10 monclient(hunting): picked mon.b con 0x7f84e0004800 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:35:32.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f559a640 1 --2- 172.21.15.121:0/1798411920 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f84e00099b0 0x7f84e000bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:32.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f559a640 10 monclient(hunting): picked mon.c con 0x7f84e00099b0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:35:32.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f559a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:32.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f559a640 1 -- 172.21.15.121:0/1798411920 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f84f80705e0 con 0x7f84e0003a50 2024-09-17T18:35:32.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f559a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:32.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f559a640 1 -- 172.21.15.121:0/1798411920 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f84e000c3c0 con 0x7f84e0004800 2024-09-17T18:35:32.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f559a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:32.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f559a640 1 -- 172.21.15.121:0/1798411920 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f84e000c5c0 con 0x7f84e00099b0 2024-09-17T18:35:32.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f559a640 10 monclient(hunting): _renew_subs 2024-09-17T18:35:32.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f559a640 10 monclient(hunting): _finish_auth 0 2024-09-17T18:35:32.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84fcbd3640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-17T18:35:32.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f659c640 1 -- 172.21.15.121:0/1798411920 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f84e0003a50 msgr2=0x7f84e0003e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:35:32.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.956+0000 7f84f659c640 1 --2- 172.21.15.121:0/1798411920 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f84e0003a50 0x7f84e0003e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:35:32.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.957+0000 7f84f6d9d640 1 --2- 172.21.15.121:0/1798411920 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f84e00099b0 0x7f84e000bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:32.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.957+0000 7f84f6d9d640 10 monclient(hunting): get_auth_request con 0x7f84e00099b0 auth_method 0 2024-09-17T18:35:32.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.957+0000 7f84f6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:32.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.957+0000 7f84f6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:32.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.957+0000 7f84f6d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:35:32.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.957+0000 7f84f5d9b640 1 --2- 172.21.15.121:0/1798411920 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f84e0004800 0x7f84e0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:32.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.957+0000 7f84f5d9b640 10 monclient(hunting): get_auth_request con 0x7f84e0004800 auth_method 0 2024-09-17T18:35:32.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.957+0000 7f84f5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:32.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.957+0000 7f84f5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:32.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.957+0000 7f84f5d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:35:32.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.957+0000 7f84f6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:32.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.957+0000 7f84f6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:32.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.957+0000 7f84f6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:35:32.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.957+0000 7f84f5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:32.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.957+0000 7f84f5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:32.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.957+0000 7f84f5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:35:32.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.957+0000 7f84f6d9d640 10 monclient(hunting): handle_auth_done global_id 84283 payload 293 2024-09-17T18:35:32.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.957+0000 7f84f6d9d640 1 -- 172.21.15.121:0/1798411920 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f84e0004800 msgr2=0x7f84e0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:35:32.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.957+0000 7f84f6d9d640 1 --2- 172.21.15.121:0/1798411920 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f84e0004800 0x7f84e0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:32.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.957+0000 7f84f6d9d640 1 -- 172.21.15.121:0/1798411920 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f84e0003a50 msgr2=0x7f84e0003e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:35:32.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.957+0000 7f84f6d9d640 1 --2- 172.21.15.121:0/1798411920 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f84e0003a50 0x7f84e0003e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:32.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.957+0000 7f84f6d9d640 10 monclient: _finish_hunting 0 2024-09-17T18:35:32.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.957+0000 7f84f6d9d640 1 monclient: found mon.c 2024-09-17T18:35:32.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.957+0000 7f84f6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:35:32.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.957+0000 7f84f6d9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:32.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.957+0000 7f84f6d9d640 1 -- 172.21.15.121:0/1798411920 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84e0003520 con 0x7f84e00099b0 2024-09-17T18:35:32.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.957+0000 7f84f6d9d640 10 monclient: _finish_auth 0 2024-09-17T18:35:32.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.957+0000 7f84f6d9d640 10 monclient: _check_auth_tickets 2024-09-17T18:35:32.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.957+0000 7f84f6d9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:35:32.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.957+0000 7f84f6d9d640 1 -- 172.21.15.121:0/1798411920 reap_dead start 2024-09-17T18:35:32.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.960+0000 7f84f6d9d640 1 -- 172.21.15.121:0/1798411920 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f84e00099b0 msgr2=0x7f84e000bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:32.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.960+0000 7f84f6d9d640 1 --2- 172.21.15.121:0/1798411920 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f84e00099b0 0x7f84e000bda0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f84e8009830 tx=0x7f84e0005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f660c489bc8f3cc0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:32.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.960+0000 7f84f559a640 1 -- 172.21.15.121:0/1798411920 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e8009430 con 0x7f84e00099b0 2024-09-17T18:35:32.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.960+0000 7f84f559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:32.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.960+0000 7f84f559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:35:32.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.960+0000 7f84f559a640 10 monclient: dump: 2024-09-17T18:35:32.973 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:32.973 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:32.973 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:32.973 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:32.974 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:32.974 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:32.974 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:32.974 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:32.974 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:32.974 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:32.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.960+0000 7f84f559a640 1 -- 172.21.15.121:0/1798411920 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84e802e070 con 0x7f84e00099b0 2024-09-17T18:35:32.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.960+0000 7f84fcbd3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:35:32.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.960+0000 7f84f559a640 10 monclient: handle_config config(0 keys) 2024-09-17T18:35:32.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.960+0000 7f84fcbd3640 10 monclient: get_monmap_and_config success 2024-09-17T18:35:32.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.960+0000 7f84f559a640 1 -- 172.21.15.121:0/1798411920 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e802a850 con 0x7f84e00099b0 2024-09-17T18:35:32.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.960+0000 7f84f559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:32.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.960+0000 7f84fcbd3640 4 set_mon_vals no callback set 2024-09-17T18:35:32.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.960+0000 7f84f559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:35:32.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.960+0000 7f84f559a640 10 monclient: dump: 2024-09-17T18:35:32.975 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:32.975 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:32.975 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:32.975 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:32.975 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:32.975 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:32.975 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:32.975 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:32.975 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:32.975 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:32.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.960+0000 7f84fcbd3640 10 monclient: shutdown 2024-09-17T18:35:32.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.960+0000 7f84fcbd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:35:32.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.960+0000 7f84fcbd3640 1 -- 172.21.15.121:0/1798411920 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f84e00099b0 msgr2=0x7f84e000bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:32.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.960+0000 7f84fcbd3640 1 --2- 172.21.15.121:0/1798411920 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f84e00099b0 0x7f84e000bda0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f84e8009830 tx=0x7f84e0005830 comp rx=0 tx=0).stop 2024-09-17T18:35:32.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.961+0000 7f84fcbd3640 1 -- 172.21.15.121:0/1798411920 shutdown_connections 2024-09-17T18:35:32.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.961+0000 7f84fcbd3640 1 --2- 172.21.15.121:0/1798411920 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f84e00099b0 0x7f84e000bda0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:32.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.961+0000 7f84fcbd3640 1 -- 172.21.15.121:0/1798411920 >> 172.21.15.121:0/1798411920 conn(0x7f84f80aba80 msgr2=0x7f84f80a7da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:35:32.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.961+0000 7f84fcbd3640 1 -- 172.21.15.121:0/1798411920 shutdown_connections 2024-09-17T18:35:32.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.961+0000 7f84fcbd3640 1 -- 172.21.15.121:0/1798411920 wait complete. 2024-09-17T18:35:32.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.961+0000 7f84fcbd3640 10 monclient: build_initial_monmap 2024-09-17T18:35:32.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.961+0000 7f84fcbd3640 10 monclient: monmap: 2024-09-17T18:35:32.976 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:35:32.976 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:35:32.976 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:35:32.976 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:35:32.976 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:35:32.976 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:32.976 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:35:32.976 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:35:32.977 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:35:32.977 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:32.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.962+0000 7f84fcbd3640 1 librados: starting msgr at 2024-09-17T18:35:32.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.962+0000 7f84fcbd3640 1 librados: starting objecter 2024-09-17T18:35:32.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.962+0000 7f84fcbd3640 1 Processor -- start 2024-09-17T18:35:32.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.962+0000 7f84fcbd3640 1 -- start start 2024-09-17T18:35:32.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.962+0000 7f84fcbd3640 1 librados: setting wanted keys 2024-09-17T18:35:32.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.962+0000 7f84fcbd3640 1 librados: calling monclient init 2024-09-17T18:35:32.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.962+0000 7f84fcbd3640 10 monclient: init 2024-09-17T18:35:32.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.962+0000 7f84fcbd3640 10 monclient: _reopen_session rank -1 2024-09-17T18:35:32.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.962+0000 7f84fcbd3640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T18:35:32.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.962+0000 7f84fcbd3640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f84e00099b0 0x7f84f81bd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:32.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.962+0000 7f84fcbd3640 10 monclient(hunting): picked mon.noname-a con 0x7f84e00099b0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:35:32.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.962+0000 7f84fcbd3640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f84f81a7d40 0x7f84f81a81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:32.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.962+0000 7f84fcbd3640 10 monclient(hunting): picked mon.noname-c con 0x7f84f81a7d40 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:35:32.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.962+0000 7f84fcbd3640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f84f81a8700 0x7f84f81d3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:32.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.962+0000 7f84fcbd3640 10 monclient(hunting): picked mon.noname-b con 0x7f84f81a8700 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:35:32.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.962+0000 7f84fcbd3640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:32.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.962+0000 7f84fcbd3640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f84f80a6a40 con 0x7f84e00099b0 2024-09-17T18:35:32.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.962+0000 7f84fcbd3640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:32.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.962+0000 7f84fcbd3640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f84f80a31b0 con 0x7f84f81a8700 2024-09-17T18:35:32.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.962+0000 7f84fcbd3640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:32.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.962+0000 7f84fcbd3640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f84f80a8540 con 0x7f84f81a7d40 2024-09-17T18:35:32.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.962+0000 7f84fcbd3640 10 monclient(hunting): _renew_subs 2024-09-17T18:35:32.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.962+0000 7f84fcbd3640 10 monclient(hunting): authenticate will time out at 1736.679736s 2024-09-17T18:35:32.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.962+0000 7f84f659c640 1 -- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f84e00099b0 msgr2=0x7f84f81bd3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:35:32.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.962+0000 7f84f659c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f84e00099b0 0x7f84f81bd3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:35:32.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.962+0000 7f84e6ffd640 1 -- DispatchQueue::entry inject delay of 0.002000 2024-09-17T18:35:32.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.962+0000 7f84f5d9b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f84f81a7d40 0x7f84f81a81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:32.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.962+0000 7f84f5d9b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f84f81a7d40 0x7f84f81a81c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:54000/0 (socket says 172.21.15.121:54000) 2024-09-17T18:35:32.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.962+0000 7f84f5d9b640 1 -- 172.21.15.121:0/1961366608 learned_addr learned my addr 172.21.15.121:0/1961366608 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:35:32.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.962+0000 7f84f5d9b640 10 monclient(hunting): get_auth_request con 0x7f84f81a7d40 auth_method 0 2024-09-17T18:35:32.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.962+0000 7f84f5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:32.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.962+0000 7f84f5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:32.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.962+0000 7f84f5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:32.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.962+0000 7f84f5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:32.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.962+0000 7f84f5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:32.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.962+0000 7f84f5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:32.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.963+0000 7f84f5d9b640 10 monclient(hunting): handle_auth_done global_id 64523 payload 824 2024-09-17T18:35:32.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.963+0000 7f84f6d9d640 1 --2- 172.21.15.121:0/1961366608 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f84f81a8700 0x7f84f81d3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:32.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.963+0000 7f84f5d9b640 1 -- 172.21.15.121:0/1961366608 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f84f81a8700 msgr2=0x7f84f81d3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:32.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.963+0000 7f84f5d9b640 1 --2- 172.21.15.121:0/1961366608 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f84f81a8700 0x7f84f81d3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:32.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.963+0000 7f84f5d9b640 1 -- 172.21.15.121:0/1961366608 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f84e00099b0 msgr2=0x7f84f81bd3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:35:32.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.963+0000 7f84f5d9b640 1 --2- 172.21.15.121:0/1961366608 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f84e00099b0 0x7f84f81bd3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:32.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.963+0000 7f84f5d9b640 10 monclient: _finish_hunting 0 2024-09-17T18:35:32.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.963+0000 7f84f5d9b640 1 monclient: found mon.noname-c 2024-09-17T18:35:32.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.963+0000 7f84f5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:35:32.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.963+0000 7f84f5d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:35:32.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.963+0000 7f84f5d9b640 1 -- 172.21.15.121:0/1961366608 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84f80a2490 con 0x7f84f81a7d40 2024-09-17T18:35:32.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.963+0000 7f84f6d9d640 10 monclient: get_auth_request con 0x7f84f81a8700 auth_method 0 2024-09-17T18:35:32.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.963+0000 7f84f6d9d640 1 --2- 172.21.15.121:0/1961366608 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f84f81a8700 0x7f84f81d3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T18:35:32.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.965+0000 7f84f5d9b640 1 -- 172.21.15.121:0/1961366608 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f84f81a7d40 msgr2=0x7f84f81a81c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:32.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.965+0000 7f84f5d9b640 1 --2- 172.21.15.121:0/1961366608 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f84f81a7d40 0x7f84f81a81c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f84ec00a940 tx=0x7f84ec00ae00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:32.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.965+0000 7f84e6ffd640 10 client.?.objecter ms_handle_connect 0x7f84f81a7d40 2024-09-17T18:35:32.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.965+0000 7f84e6ffd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:35:32.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.965+0000 7f84e6ffd640 1 -- 172.21.15.121:0/1961366608 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84ec017020 con 0x7f84f81a7d40 2024-09-17T18:35:32.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.965+0000 7f84e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:32.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.965+0000 7f84e6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:35:32.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.965+0000 7f84e6ffd640 10 monclient: dump: 2024-09-17T18:35:32.981 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:32.981 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:32.981 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:32.981 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:32.981 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:32.981 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:32.981 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:32.981 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:32.981 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:32.981 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:32.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.965+0000 7f84e6ffd640 10 monclient: _finish_auth 0 2024-09-17T18:35:32.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.965+0000 7f84e6ffd640 10 monclient: _check_auth_tickets 2024-09-17T18:35:32.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.965+0000 7f84e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:35:32.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.965+0000 7f84fcbd3640 5 monclient: authenticate success, global_id 64523 2024-09-17T18:35:32.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.965+0000 7f84e6ffd640 1 -- 172.21.15.121:0/1961366608 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84ec013670 con 0x7f84f81a7d40 2024-09-17T18:35:32.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.965+0000 7f84e6ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:35:32.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.965+0000 7f84e6ffd640 1 -- 172.21.15.121:0/1961366608 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84ec013a60 con 0x7f84f81a7d40 2024-09-17T18:35:32.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.965+0000 7f84e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:32.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.965+0000 7f84e6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:35:32.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.965+0000 7f84e6ffd640 10 monclient: dump: 2024-09-17T18:35:32.982 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:32.982 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:32.982 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:32.982 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:32.982 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:32.982 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:32.982 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:32.982 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:32.982 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:32.982 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:32.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.965+0000 7f84fcbd3640 10 monclient: _renew_subs 2024-09-17T18:35:32.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.965+0000 7f84f4d99640 4 set_mon_vals no callback set 2024-09-17T18:35:32.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.965+0000 7f84fcbd3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:32.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.965+0000 7f84fcbd3640 1 -- 172.21.15.121:0/1961366608 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84f81aac10 con 0x7f84f81a7d40 2024-09-17T18:35:32.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.965+0000 7f84fcbd3640 10 client.64523.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:35:32.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.965+0000 7f84fcbd3640 10 monclient: _renew_subs 2024-09-17T18:35:32.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.965+0000 7f84fcbd3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:32.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.965+0000 7f84fcbd3640 1 -- 172.21.15.121:0/1961366608 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f84f81ab1e0 con 0x7f84f81a7d40 2024-09-17T18:35:32.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.965+0000 7f84fcbd3640 1 librados: init done 2024-09-17T18:35:32.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.966+0000 7f84e6ffd640 1 -- 172.21.15.121:0/1961366608 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f84ec01a020 con 0x7f84f81a7d40 2024-09-17T18:35:32.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.966+0000 7f84fcbd3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:32.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.966+0000 7f84fcbd3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:32.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.966+0000 7f84fcbd3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:32.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.966+0000 7f84fcbd3640 1 -- 172.21.15.121:0/1961366608 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84f80a5ca0 con 0x7f84f81a7d40 2024-09-17T18:35:32.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.966+0000 7f84e6ffd640 1 --2- 172.21.15.121:0/1961366608 >> v2:172.21.15.173:6812/2423786037 conn(0x7f84d8040170 0x7f84d8042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:32.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.966+0000 7f84e6ffd640 1 -- 172.21.15.121:0/1961366608 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(646..646 src has 247..646) ==== 5925+0+0 (secure 0 0 0) 0x7f84ec054900 con 0x7f84f81a7d40 2024-09-17T18:35:32.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.966+0000 7f84e6ffd640 10 client.64523.objecter ms_dispatch 0x7f84f8000b90 osd_map(646..646 src has 247..646) 2024-09-17T18:35:32.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.966+0000 7f84e6ffd640 3 client.64523.objecter handle_osd_map got epochs [646,646] > 0 2024-09-17T18:35:32.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.966+0000 7f84e6ffd640 3 client.64523.objecter handle_osd_map decoding full epoch 646 2024-09-17T18:35:32.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.967+0000 7f84e6ffd640 20 client.64523.objecter dump_active .. 0 homeless 2024-09-17T18:35:32.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.967+0000 7f84f659c640 1 --2- 172.21.15.121:0/1961366608 >> v2:172.21.15.173:6812/2423786037 conn(0x7f84d8040170 0x7f84d8042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:32.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.967+0000 7f84f659c640 10 monclient: get_auth_request con 0x7f84d8040170 auth_method 0 2024-09-17T18:35:32.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.969+0000 7f84f659c640 1 --2- 172.21.15.121:0/1961366608 >> v2:172.21.15.173:6812/2423786037 conn(0x7f84d8040170 0x7f84d8042630 secure :-1 s=READY pgs=4081 cs=0 l=1 rev1=1 crypto rx=0x7f84f0006fd0 tx=0x7f84f0006e00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:32.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.969+0000 7f84e6ffd640 10 client.64523.objecter ms_handle_connect 0x7f84d8040170 2024-09-17T18:35:32.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.971+0000 7f84e6ffd640 1 -- 172.21.15.121:0/1961366608 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f84ec021020 con 0x7f84f81a7d40 2024-09-17T18:35:32.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.971+0000 7f84e6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:32.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:32.971+0000 7f84e6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:35:33.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:33.215+0000 7f84fcbd3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:35:33.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:33.215+0000 7f84fcbd3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:35:33.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:33.215+0000 7f84fcbd3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:33.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:33.215+0000 7f84fcbd3640 1 -- 172.21.15.121:0/1961366608 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f84f806cd40 con 0x7f84f81a7d40 2024-09-17T18:35:33.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:33.215+0000 7f84e6ffd640 1 -- 172.21.15.121:0/1961366608 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f84ec00e040 con 0x7f84f81a7d40 2024-09-17T18:35:33.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:33.215+0000 7f84e6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:35:33.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:33.215+0000 7f84e6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:35:33.219 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":60,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:35:33.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:33.217+0000 7f84fcbd3640 10 librados: watch_flush enter 2024-09-17T18:35:33.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:33.217+0000 7f84fcbd3640 10 librados: watch_flush exit 2024-09-17T18:35:33.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:33.217+0000 7f84fcbd3640 20 client.64523.objecter shutdown clearing up homeless session... 2024-09-17T18:35:33.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:33.217+0000 7f84fcbd3640 10 client.64523.objecter successfully canceled tick 2024-09-17T18:35:33.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:33.218+0000 7f84fcbd3640 1 -- 172.21.15.121:0/1961366608 >> v2:172.21.15.173:6812/2423786037 conn(0x7f84d8040170 msgr2=0x7f84d8042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:33.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:33.218+0000 7f84fcbd3640 1 --2- 172.21.15.121:0/1961366608 >> v2:172.21.15.173:6812/2423786037 conn(0x7f84d8040170 0x7f84d8042630 secure :-1 s=READY pgs=4081 cs=0 l=1 rev1=1 crypto rx=0x7f84f0006fd0 tx=0x7f84f0006e00 comp rx=0 tx=0).stop 2024-09-17T18:35:33.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:33.218+0000 7f84fcbd3640 10 monclient: shutdown 2024-09-17T18:35:33.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:33.218+0000 7f84fcbd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:35:33.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:33.218+0000 7f84fcbd3640 1 -- 172.21.15.121:0/1961366608 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f84f81a7d40 msgr2=0x7f84f81a81c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:33.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:33.218+0000 7f84fcbd3640 1 --2- 172.21.15.121:0/1961366608 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f84f81a7d40 0x7f84f81a81c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f84ec00a940 tx=0x7f84ec00ae00 comp rx=0 tx=0).stop 2024-09-17T18:35:33.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:33.218+0000 7f84fcbd3640 1 -- 172.21.15.121:0/1961366608 shutdown_connections 2024-09-17T18:35:33.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:33.218+0000 7f84fcbd3640 1 --2- 172.21.15.121:0/1961366608 >> v2:172.21.15.173:6812/2423786037 conn(0x7f84d8040170 0x7f84d8042630 unknown :-1 s=CLOSED pgs=4081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:33.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:33.218+0000 7f84fcbd3640 1 --2- 172.21.15.121:0/1961366608 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f84f81a8700 0x7f84f81d3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:33.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:33.218+0000 7f84fcbd3640 1 --2- 172.21.15.121:0/1961366608 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f84f81a7d40 0x7f84f81a81c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:33.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:33.218+0000 7f84fcbd3640 1 --2- 172.21.15.121:0/1961366608 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f84e00099b0 0x7f84f81bd3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:33.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:33.218+0000 7f84fcbd3640 1 -- 172.21.15.121:0/1961366608 >> 172.21.15.121:0/1961366608 conn(0x7f84f80aba80 msgr2=0x7f84f80aab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:35:33.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:33.218+0000 7f84fcbd3640 1 -- 172.21.15.121:0/1961366608 shutdown_connections 2024-09-17T18:35:33.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:33.218+0000 7f84fcbd3640 1 -- 172.21.15.121:0/1961366608 wait complete. 2024-09-17T18:35:33.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:33.218+0000 7f84fcbd3640 1 librados: shutdown 2024-09-17T18:35:36.232 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:35:36.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.338+0000 7fec849c2640 10 monclient: get_monmap_and_config 2024-09-17T18:35:36.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.338+0000 7fec849c2640 10 monclient: build_initial_monmap 2024-09-17T18:35:36.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.338+0000 7fec849c2640 10 monclient: monmap: 2024-09-17T18:35:36.341 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:35:36.341 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:36.341 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:35:36.339634+0000 2024-09-17T18:35:36.341 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:35:36.339634+0000 2024-09-17T18:35:36.341 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:35:36.341 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:36.341 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:35:36.341 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:35:36.341 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:35:36.341 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:35:36.341 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:35:36.341 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:36.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.338+0000 7fec849c2640 1 Processor -- start 2024-09-17T18:35:36.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.338+0000 7fec849c2640 1 -- start start 2024-09-17T18:35:36.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.338+0000 7fec849c2640 10 monclient: init 2024-09-17T18:35:36.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.339+0000 7fec849c2640 10 monclient: _reopen_session rank -1 2024-09-17T18:35:36.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.339+0000 7fec849c2640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-17T18:35:36.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.339+0000 7fec849c2640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec800a5c00 0x7fec8009d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:36.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.339+0000 7fec849c2640 10 monclient(hunting): picked mon.noname-b con 0x7fec800a5c00 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:35:36.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.339+0000 7fec849c2640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fec800a5fe0 0x7fec800a63c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:36.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.339+0000 7fec849c2640 10 monclient(hunting): picked mon.noname-c con 0x7fec800a5fe0 addr v2:172.21.15.173:3300/0 2024-09-17T18:35:36.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.339+0000 7fec849c2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fec801389c0 addr v1:172.21.15.173:6789/0 2024-09-17T18:35:36.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.339+0000 7fec849c2640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec800a8a80 con 0x7fec801389c0 2024-09-17T18:35:36.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.339+0000 7fec849c2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:36.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.339+0000 7fec849c2640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fec8009de00 con 0x7fec800a5fe0 2024-09-17T18:35:36.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.339+0000 7fec849c2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:36.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.339+0000 7fec849c2640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fec800a74d0 con 0x7fec800a5c00 2024-09-17T18:35:36.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.339+0000 7fec849c2640 10 monclient(hunting): _renew_subs 2024-09-17T18:35:36.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.339+0000 7fec849c2640 10 monclient(hunting): authenticate will time out at 1740.056552s 2024-09-17T18:35:36.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.339+0000 7fec7e59c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec800a5c00 0x7fec8009d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:36.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.339+0000 7fec7e59c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec800a5c00 0x7fec8009d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:54014/0 (socket says 172.21.15.121:54014) 2024-09-17T18:35:36.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.339+0000 7fec7e59c640 1 -- 172.21.15.121:0/866576794 learned_addr learned my addr 172.21.15.121:0/866576794 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:35:36.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.339+0000 7fec7e59c640 10 monclient(hunting): get_auth_request con 0x7fec800a5c00 auth_method 0 2024-09-17T18:35:36.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.339+0000 7fec7e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:36.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.339+0000 7fec7e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:36.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.339+0000 7fec7e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:36.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.339+0000 7fec7e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:36.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.339+0000 7fec7e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:36.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.339+0000 7fec7e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:36.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.339+0000 7fec7e59c640 10 monclient(hunting): handle_auth_done global_id 64526 payload 274 2024-09-17T18:35:36.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.339+0000 7fec7e59c640 1 -- 172.21.15.121:0/866576794 >> v2:172.21.15.173:3300/0 conn(0x7fec800a5fe0 msgr2=0x7fec800a63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:35:36.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.339+0000 7fec7dd9b640 1 --2- 172.21.15.121:0/866576794 >> v2:172.21.15.173:3300/0 conn(0x7fec800a5fe0 0x7fec800a63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:36.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.339+0000 7fec7e59c640 1 --2- 172.21.15.121:0/866576794 >> v2:172.21.15.173:3300/0 conn(0x7fec800a5fe0 0x7fec800a63c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:36.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.339+0000 7fec7e59c640 1 -- 172.21.15.121:0/866576794 >> v1:172.21.15.173:6789/0 conn(0x7fec801389c0 legacy=0x7fec80138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:35:36.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.339+0000 7fec7e59c640 10 monclient: _finish_hunting 0 2024-09-17T18:35:36.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.339+0000 7fec7e59c640 1 monclient: found mon.noname-b 2024-09-17T18:35:36.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.339+0000 7fec7e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:35:36.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.340+0000 7fec7e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:35:36.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.340+0000 7fec7e59c640 1 -- 172.21.15.121:0/866576794 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec8009cb90 con 0x7fec800a5c00 2024-09-17T18:35:36.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.342+0000 7fec7e59c640 1 -- 172.21.15.121:0/866576794 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec800a5c00 msgr2=0x7fec8009d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:36.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.342+0000 7fec7e59c640 1 --2- 172.21.15.121:0/866576794 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec800a5c00 0x7fec8009d8c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fec70009e50 tx=0x7fec7002a560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7235c9a409bcf2cd server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:36.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.343+0000 7fec7d59a640 1 -- 172.21.15.121:0/866576794 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec70036070 con 0x7fec800a5c00 2024-09-17T18:35:36.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.343+0000 7fec7d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:36.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.343+0000 7fec7d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:35:36.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.343+0000 7fec7d59a640 10 monclient: dump: 2024-09-17T18:35:36.354 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:36.354 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:36.354 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:36.354 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:36.354 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:36.354 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:36.354 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:36.354 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:36.354 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:36.354 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:36.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.343+0000 7fec7d59a640 10 monclient: _finish_auth 0 2024-09-17T18:35:36.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.343+0000 7fec7d59a640 10 monclient: _check_auth_tickets 2024-09-17T18:35:36.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.343+0000 7fec7d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:35:36.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.343+0000 7fec7d59a640 1 -- 172.21.15.121:0/866576794 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec70007800 con 0x7fec800a5c00 2024-09-17T18:35:36.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.343+0000 7fec849c2640 5 monclient: authenticate success, global_id 64526 2024-09-17T18:35:36.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.343+0000 7fec849c2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:35:36.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.343+0000 7fec7d59a640 10 monclient: handle_config config(0 keys) 2024-09-17T18:35:36.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.343+0000 7fec7d59a640 1 -- 172.21.15.121:0/866576794 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec70007bd0 con 0x7fec800a5c00 2024-09-17T18:35:36.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.343+0000 7fec849c2640 10 monclient: get_monmap_and_config success 2024-09-17T18:35:36.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.343+0000 7fec849c2640 4 set_mon_vals no callback set 2024-09-17T18:35:36.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.343+0000 7fec7d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:36.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.343+0000 7fec7d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:35:36.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.343+0000 7fec7d59a640 10 monclient: dump: 2024-09-17T18:35:36.355 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:36.355 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:36.355 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:36.355 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:36.356 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:36.356 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:36.356 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:36.356 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:36.356 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:36.356 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:36.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.343+0000 7fec849c2640 10 monclient: shutdown 2024-09-17T18:35:36.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.343+0000 7fec849c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:35:36.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.343+0000 7fec849c2640 1 -- 172.21.15.121:0/866576794 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec800a5c00 msgr2=0x7fec8009d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:36.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.343+0000 7fec849c2640 1 --2- 172.21.15.121:0/866576794 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec800a5c00 0x7fec8009d8c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fec70009e50 tx=0x7fec7002a560 comp rx=0 tx=0).stop 2024-09-17T18:35:36.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.343+0000 7fec849c2640 1 -- 172.21.15.121:0/866576794 shutdown_connections 2024-09-17T18:35:36.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.343+0000 7fec849c2640 1 --2- 172.21.15.121:0/866576794 >> v2:172.21.15.173:3300/0 conn(0x7fec800a5fe0 0x7fec800a63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:36.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.343+0000 7fec849c2640 1 --2- 172.21.15.121:0/866576794 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec800a5c00 0x7fec8009d8c0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:36.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.343+0000 7fec849c2640 1 -- 172.21.15.121:0/866576794 >> 172.21.15.121:0/866576794 conn(0x7fec800ab9b0 msgr2=0x7fec800a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:35:36.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.344+0000 7fec849c2640 1 -- 172.21.15.121:0/866576794 shutdown_connections 2024-09-17T18:35:36.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.344+0000 7fec849c2640 1 -- 172.21.15.121:0/866576794 wait complete. 2024-09-17T18:35:36.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.344+0000 7fec849c2640 10 monclient: build_initial_monmap 2024-09-17T18:35:36.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.344+0000 7fec849c2640 10 monclient: monmap: 2024-09-17T18:35:36.357 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:35:36.357 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:35:36.357 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:35:36.357 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:35:36.357 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:35:36.357 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:36.357 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:35:36.357 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:35:36.357 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:35:36.357 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:36.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.344+0000 7fec849c2640 1 librados: starting msgr at 2024-09-17T18:35:36.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.345+0000 7fec849c2640 1 librados: starting objecter 2024-09-17T18:35:36.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.345+0000 7fec849c2640 1 Processor -- start 2024-09-17T18:35:36.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.345+0000 7fec849c2640 1 -- start start 2024-09-17T18:35:36.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.345+0000 7fec849c2640 1 librados: setting wanted keys 2024-09-17T18:35:36.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.345+0000 7fec849c2640 1 librados: calling monclient init 2024-09-17T18:35:36.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.345+0000 7fec849c2640 10 monclient: init 2024-09-17T18:35:36.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.345+0000 7fec849c2640 10 monclient: _reopen_session rank -1 2024-09-17T18:35:36.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.345+0000 7fec849c2640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:35:36.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.345+0000 7fec849c2640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec800a5c00 0x7fec801250a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:36.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.345+0000 7fec849c2640 10 monclient(hunting): picked mon.noname-a con 0x7fec800a5c00 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:35:36.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.345+0000 7fec849c2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec800a5fe0 0x7fec801255e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:36.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.345+0000 7fec849c2640 10 monclient(hunting): picked mon.noname-b con 0x7fec800a5fe0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:35:36.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.345+0000 7fec849c2640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec801389c0 0x7fec80125b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:36.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.345+0000 7fec7e59c640 1 -- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec800a5c00 msgr2=0x7fec801250a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:35:36.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.345+0000 7fec849c2640 10 monclient(hunting): picked mon.noname-c con 0x7fec801389c0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:35:36.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.345+0000 7fec849c2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:36.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.345+0000 7fec849c2640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fec800a6900 con 0x7fec800a5c00 2024-09-17T18:35:36.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.345+0000 7fec7e59c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec800a5c00 0x7fec801250a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:35:36.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.345+0000 7fec849c2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:36.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.345+0000 7fec849c2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fec800703d0 con 0x7fec800a5fe0 2024-09-17T18:35:36.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.345+0000 7fec849c2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:36.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.345+0000 7fec849c2640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fec800aa7f0 con 0x7fec801389c0 2024-09-17T18:35:36.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.345+0000 7fec849c2640 10 monclient(hunting): _renew_subs 2024-09-17T18:35:36.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.345+0000 7fec849c2640 10 monclient(hunting): authenticate will time out at 1740.062725s 2024-09-17T18:35:36.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.345+0000 7fec7ed9d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec801389c0 0x7fec80125b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:36.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.345+0000 7fec7ed9d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec801389c0 0x7fec80125b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:54016/0 (socket says 172.21.15.121:54016) 2024-09-17T18:35:36.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.345+0000 7fec7ed9d640 1 -- 172.21.15.121:0/1893014603 learned_addr learned my addr 172.21.15.121:0/1893014603 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:35:36.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.345+0000 7fec7ed9d640 10 monclient(hunting): get_auth_request con 0x7fec801389c0 auth_method 0 2024-09-17T18:35:36.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.345+0000 7fec7ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:36.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.345+0000 7fec7ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:36.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.345+0000 7fec7ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:36.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.345+0000 7fec7ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:36.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.345+0000 7fec7ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:36.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.345+0000 7fec7ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:36.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.345+0000 7fec7dd9b640 1 --2- 172.21.15.121:0/1893014603 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec800a5fe0 0x7fec801255e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:36.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.345+0000 7fec7dd9b640 10 monclient(hunting): get_auth_request con 0x7fec800a5fe0 auth_method 0 2024-09-17T18:35:36.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.345+0000 7fec7dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:36.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.345+0000 7fec7dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:36.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.345+0000 7fec7dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:36.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.346+0000 7fec7dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:36.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.346+0000 7fec7dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:36.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.346+0000 7fec7dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:36.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.346+0000 7fec7ed9d640 10 monclient(hunting): handle_auth_done global_id 64529 payload 824 2024-09-17T18:35:36.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.346+0000 7fec7ed9d640 1 -- 172.21.15.121:0/1893014603 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec800a5fe0 msgr2=0x7fec801255e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:36.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.346+0000 7fec7ed9d640 1 --2- 172.21.15.121:0/1893014603 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec800a5fe0 0x7fec801255e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:36.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.346+0000 7fec7ed9d640 1 -- 172.21.15.121:0/1893014603 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec800a5c00 msgr2=0x7fec801250a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:35:36.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.346+0000 7fec7ed9d640 1 --2- 172.21.15.121:0/1893014603 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec800a5c00 0x7fec801250a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:36.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.346+0000 7fec7ed9d640 10 monclient: _finish_hunting 0 2024-09-17T18:35:36.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.346+0000 7fec7ed9d640 1 monclient: found mon.noname-c 2024-09-17T18:35:36.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.346+0000 7fec7ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:35:36.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.346+0000 7fec7ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:35:36.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.346+0000 7fec7ed9d640 1 -- 172.21.15.121:0/1893014603 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec80128be0 con 0x7fec801389c0 2024-09-17T18:35:36.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.348+0000 7fec7ed9d640 1 -- 172.21.15.121:0/1893014603 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec801389c0 msgr2=0x7fec80125b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:36.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.348+0000 7fec7ed9d640 1 --2- 172.21.15.121:0/1893014603 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec801389c0 0x7fec80125b20 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fec7800c390 tx=0x7fec7800c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:36.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.348+0000 7fec76ffd640 10 client.?.objecter ms_handle_connect 0x7fec801389c0 2024-09-17T18:35:36.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.348+0000 7fec76ffd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:35:36.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.348+0000 7fec76ffd640 1 -- 172.21.15.121:0/1893014603 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec78017070 con 0x7fec801389c0 2024-09-17T18:35:36.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.348+0000 7fec76ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:36.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.348+0000 7fec76ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:35:36.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.348+0000 7fec76ffd640 10 monclient: dump: 2024-09-17T18:35:36.362 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:36.362 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:36.362 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:36.362 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:36.362 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:36.362 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:36.362 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:36.362 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:36.363 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:36.363 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:36.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.348+0000 7fec76ffd640 10 monclient: _finish_auth 0 2024-09-17T18:35:36.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.348+0000 7fec76ffd640 10 monclient: _check_auth_tickets 2024-09-17T18:35:36.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.348+0000 7fec76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:35:36.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.348+0000 7fec76ffd640 1 -- 172.21.15.121:0/1893014603 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec780037f0 con 0x7fec801389c0 2024-09-17T18:35:36.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.348+0000 7fec849c2640 5 monclient: authenticate success, global_id 64529 2024-09-17T18:35:36.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.348+0000 7fec849c2640 10 monclient: _renew_subs 2024-09-17T18:35:36.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.348+0000 7fec849c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:36.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.348+0000 7fec849c2640 1 -- 172.21.15.121:0/1893014603 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec80128ec0 con 0x7fec801389c0 2024-09-17T18:35:36.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.348+0000 7fec76ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:35:36.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.348+0000 7fec76ffd640 1 -- 172.21.15.121:0/1893014603 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec78003be0 con 0x7fec801389c0 2024-09-17T18:35:36.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.348+0000 7fec76ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:36.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.348+0000 7fec76ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:35:36.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.348+0000 7fec76ffd640 10 monclient: dump: 2024-09-17T18:35:36.363 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:36.364 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:36.364 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:36.364 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:36.364 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:36.364 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:36.364 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:36.364 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:36.364 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:36.364 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:36.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.348+0000 7fec7cd99640 4 set_mon_vals no callback set 2024-09-17T18:35:36.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.348+0000 7fec849c2640 10 client.64529.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:35:36.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.348+0000 7fec849c2640 10 monclient: _renew_subs 2024-09-17T18:35:36.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.348+0000 7fec849c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:36.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.348+0000 7fec849c2640 1 -- 172.21.15.121:0/1893014603 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fec80128050 con 0x7fec801389c0 2024-09-17T18:35:36.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.348+0000 7fec849c2640 1 librados: init done 2024-09-17T18:35:36.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.349+0000 7fec76ffd640 1 -- 172.21.15.121:0/1893014603 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fec7801a070 con 0x7fec801389c0 2024-09-17T18:35:36.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.349+0000 7fec849c2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:36.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.349+0000 7fec849c2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:36.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.349+0000 7fec849c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:36.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.349+0000 7fec849c2640 1 -- 172.21.15.121:0/1893014603 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec80139660 con 0x7fec801389c0 2024-09-17T18:35:36.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.349+0000 7fec76ffd640 1 --2- 172.21.15.121:0/1893014603 >> v2:172.21.15.173:6812/2423786037 conn(0x7fec60040120 0x7fec600425e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:36.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.349+0000 7fec76ffd640 1 -- 172.21.15.121:0/1893014603 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(646..646 src has 247..646) ==== 5925+0+0 (secure 0 0 0) 0x7fec78052da0 con 0x7fec801389c0 2024-09-17T18:35:36.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.349+0000 7fec76ffd640 10 client.64529.objecter ms_dispatch 0x7fec80000b90 osd_map(646..646 src has 247..646) 2024-09-17T18:35:36.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.349+0000 7fec76ffd640 3 client.64529.objecter handle_osd_map got epochs [646,646] > 0 2024-09-17T18:35:36.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.349+0000 7fec76ffd640 3 client.64529.objecter handle_osd_map decoding full epoch 646 2024-09-17T18:35:36.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.349+0000 7fec76ffd640 20 client.64529.objecter dump_active .. 0 homeless 2024-09-17T18:35:36.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.350+0000 7fec7e59c640 1 --2- 172.21.15.121:0/1893014603 >> v2:172.21.15.173:6812/2423786037 conn(0x7fec60040120 0x7fec600425e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:36.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.350+0000 7fec7e59c640 10 monclient: get_auth_request con 0x7fec60040120 auth_method 0 2024-09-17T18:35:36.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.352+0000 7fec7e59c640 1 --2- 172.21.15.121:0/1893014603 >> v2:172.21.15.173:6812/2423786037 conn(0x7fec60040120 0x7fec600425e0 secure :-1 s=READY pgs=4082 cs=0 l=1 rev1=1 crypto rx=0x7fec8009d210 tx=0x7fec70009000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:36.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.352+0000 7fec76ffd640 1 -- 172.21.15.121:0/1893014603 DispatchQueue::entry inject delay of 0.002000 2024-09-17T18:35:36.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.354+0000 7fec76ffd640 10 client.64529.objecter ms_handle_connect 0x7fec60040120 2024-09-17T18:35:36.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.354+0000 7fec76ffd640 1 -- 172.21.15.121:0/1893014603 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fec78003e00 con 0x7fec801389c0 2024-09-17T18:35:36.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.354+0000 7fec76ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:36.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.354+0000 7fec76ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:35:36.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.596+0000 7fec849c2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:35:36.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.596+0000 7fec849c2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:35:36.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.596+0000 7fec849c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:36.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.596+0000 7fec849c2640 1 -- 172.21.15.121:0/1893014603 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fec8006cd40 con 0x7fec801389c0 2024-09-17T18:35:36.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.597+0000 7fec76ffd640 1 -- 172.21.15.121:0/1893014603 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7fec78051320 con 0x7fec801389c0 2024-09-17T18:35:36.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.597+0000 7fec76ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:35:36.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.597+0000 7fec76ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:35:36.600 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":60,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:35:36.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.599+0000 7fec849c2640 10 librados: watch_flush enter 2024-09-17T18:35:36.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.599+0000 7fec849c2640 10 librados: watch_flush exit 2024-09-17T18:35:36.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.599+0000 7fec849c2640 20 client.64529.objecter shutdown clearing up homeless session... 2024-09-17T18:35:36.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.599+0000 7fec849c2640 10 client.64529.objecter successfully canceled tick 2024-09-17T18:35:36.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.599+0000 7fec849c2640 1 -- 172.21.15.121:0/1893014603 >> v2:172.21.15.173:6812/2423786037 conn(0x7fec60040120 msgr2=0x7fec600425e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:36.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.599+0000 7fec849c2640 1 --2- 172.21.15.121:0/1893014603 >> v2:172.21.15.173:6812/2423786037 conn(0x7fec60040120 0x7fec600425e0 secure :-1 s=READY pgs=4082 cs=0 l=1 rev1=1 crypto rx=0x7fec8009d210 tx=0x7fec70009000 comp rx=0 tx=0).stop 2024-09-17T18:35:36.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.599+0000 7fec849c2640 10 monclient: shutdown 2024-09-17T18:35:36.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.599+0000 7fec849c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:35:36.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.599+0000 7fec849c2640 1 -- 172.21.15.121:0/1893014603 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec801389c0 msgr2=0x7fec80125b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:36.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.599+0000 7fec849c2640 1 --2- 172.21.15.121:0/1893014603 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec801389c0 0x7fec80125b20 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fec7800c390 tx=0x7fec7800c850 comp rx=0 tx=0).stop 2024-09-17T18:35:36.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.599+0000 7fec849c2640 1 -- 172.21.15.121:0/1893014603 shutdown_connections 2024-09-17T18:35:36.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.599+0000 7fec849c2640 1 --2- 172.21.15.121:0/1893014603 >> v2:172.21.15.173:6812/2423786037 conn(0x7fec60040120 0x7fec600425e0 unknown :-1 s=CLOSED pgs=4082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:36.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.599+0000 7fec849c2640 1 --2- 172.21.15.121:0/1893014603 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec800a5fe0 0x7fec801255e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:36.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.599+0000 7fec849c2640 1 --2- 172.21.15.121:0/1893014603 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec801389c0 0x7fec80125b20 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:36.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.599+0000 7fec849c2640 1 --2- 172.21.15.121:0/1893014603 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec800a5c00 0x7fec801250a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:36.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.599+0000 7fec849c2640 1 -- 172.21.15.121:0/1893014603 >> 172.21.15.121:0/1893014603 conn(0x7fec800ab9b0 msgr2=0x7fec800a0e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:35:36.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.599+0000 7fec849c2640 1 -- 172.21.15.121:0/1893014603 shutdown_connections 2024-09-17T18:35:36.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.599+0000 7fec849c2640 1 -- 172.21.15.121:0/1893014603 wait complete. 2024-09-17T18:35:36.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:36.599+0000 7fec849c2640 1 librados: shutdown 2024-09-17T18:35:39.613 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:35:39.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:39.717+0000 7f9d17197640 10 monclient: get_monmap_and_config 2024-09-17T18:35:39.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:39.717+0000 7f9d17197640 10 monclient: build_initial_monmap 2024-09-17T18:35:39.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:39.717+0000 7f9d17197640 10 monclient: monmap: 2024-09-17T18:35:39.720 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:35:39.720 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:39.720 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:35:39.719233+0000 2024-09-17T18:35:39.720 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:35:39.719233+0000 2024-09-17T18:35:39.721 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:35:39.721 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:39.721 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:35:39.721 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:35:39.721 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:35:39.721 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:35:39.721 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:35:39.721 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:39.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:39.718+0000 7f9d17197640 1 Processor -- start 2024-09-17T18:35:39.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:39.718+0000 7f9d17197640 1 -- start start 2024-09-17T18:35:39.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:39.718+0000 7f9d17197640 10 monclient: init 2024-09-17T18:35:39.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:39.718+0000 7f9d17197640 10 monclient: _reopen_session rank -1 2024-09-17T18:35:39.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:39.718+0000 7f9d17197640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-17T18:35:39.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:39.718+0000 7f9d17197640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f9d10136800 0x7f9d10136170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:39.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:39.718+0000 7f9d17197640 10 monclient(hunting): picked mon.noname-c con 0x7f9d10136800 addr v2:172.21.15.173:3300/0 2024-09-17T18:35:39.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:39.718+0000 7f9d17197640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9d10136be0 addr v1:172.21.15.173:6789/0 2024-09-17T18:35:39.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:39.718+0000 7f9d17197640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9d10133360 addr v1:172.21.15.121:6789/0 2024-09-17T18:35:39.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:39.718+0000 7f9d17197640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d100a0c20 con 0x7f9d10133360 2024-09-17T18:35:39.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:39.718+0000 7f9d17197640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d100a2cb0 con 0x7f9d10136be0 2024-09-17T18:35:39.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:39.718+0000 7f9d17197640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:39.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:39.718+0000 7f9d17197640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f9d100a2f90 con 0x7f9d10136800 2024-09-17T18:35:39.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:39.718+0000 7f9d17197640 10 monclient(hunting): _renew_subs 2024-09-17T18:35:39.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:39.718+0000 7f9d17197640 10 monclient(hunting): authenticate will time out at 1743.435965s 2024-09-17T18:35:39.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:39.718+0000 7f9d15734640 1 -- >> v1:172.21.15.121:6789/0 conn(0x7f9d10133360 legacy=0x7f9d1013e510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.121:6789/0 2024-09-17T18:35:39.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:39.719+0000 7f9d14f33640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f9d10136800 0x7f9d10136170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:39.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:39.719+0000 7f9d14f33640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f9d10136800 0x7f9d10136170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.121:48896/0 (socket says 172.21.15.121:48896) 2024-09-17T18:35:39.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:39.719+0000 7f9d07fff640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f9d10136be0 0x7f9d10132c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.121:56666/0 (socket says 172.21.15.121:56666) 2024-09-17T18:35:39.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:39.719+0000 7f9d14f33640 1 -- 172.21.15.121:0/668759241 learned_addr learned my addr 172.21.15.121:0/668759241 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:35:39.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:39.719+0000 7f9d14f33640 10 monclient(hunting): get_auth_request con 0x7f9d10136800 auth_method 0 2024-09-17T18:35:39.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:39.719+0000 7f9d14f33640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:39.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:39.719+0000 7f9d14f33640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:39.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:39.719+0000 7f9d14f33640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:39.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:39.719+0000 7f9d14f33640 1 -- 172.21.15.121:0/668759241 >> v2:172.21.15.173:3300/0 conn(0x7f9d10136800 msgr2=0x7f9d10136170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:35:39.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:39.719+0000 7f9d14f33640 1 -- 172.21.15.121:0/668759241 >> v2:172.21.15.173:3300/0 conn(0x7f9d10136800 msgr2=0x7f9d10136170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:35:39.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:39.719+0000 7f9d14f33640 1 --2- 172.21.15.121:0/668759241 >> v2:172.21.15.173:3300/0 conn(0x7f9d10136800 0x7f9d10136170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:35:39.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:39.719+0000 7f9d14f33640 1 --2- 172.21.15.121:0/668759241 >> v2:172.21.15.173:3300/0 conn(0x7f9d10136800 0x7f9d10136170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:35:39.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:39.721+0000 7f9d07fff640 1 -- 172.21.15.121:0/668759241 >> v1:172.21.15.173:6789/0 conn(0x7f9d10136be0 legacy=0x7f9d10132c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:39.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:39.920+0000 7f9d15734640 1 -- 172.21.15.121:0/668759241 >> v1:172.21.15.121:6789/0 conn(0x7f9d10133360 legacy=0x7f9d1013e510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.121:6789/0 2024-09-17T18:35:39.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:39.921+0000 7f9d14f33640 1 --2- 172.21.15.121:0/668759241 >> v2:172.21.15.173:3300/0 conn(0x7f9d10136800 0x7f9d10136170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:39.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:39.921+0000 7f9d14f33640 10 monclient(hunting): get_auth_request con 0x7f9d10136800 auth_method 0 2024-09-17T18:35:39.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:39.921+0000 7f9d14f33640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:39.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:39.921+0000 7f9d14f33640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:39.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:39.921+0000 7f9d14f33640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:35:39.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:39.921+0000 7f9d14f33640 1 -- 172.21.15.121:0/668759241 >> v2:172.21.15.173:3300/0 conn(0x7f9d10136800 msgr2=0x7f9d10136170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:35:39.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:39.921+0000 7f9d14f33640 1 -- 172.21.15.121:0/668759241 >> v2:172.21.15.173:3300/0 conn(0x7f9d10136800 msgr2=0x7f9d10136170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:35:39.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:39.921+0000 7f9d14f33640 1 --2- 172.21.15.121:0/668759241 >> v2:172.21.15.173:3300/0 conn(0x7f9d10136800 0x7f9d10136170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:35:39.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:39.921+0000 7f9d14f33640 1 --2- 172.21.15.121:0/668759241 >> v2:172.21.15.173:3300/0 conn(0x7f9d10136800 0x7f9d10136170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T18:35:40.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:40.321+0000 7f9d15734640 1 -- 172.21.15.121:0/668759241 >> v1:172.21.15.121:6789/0 conn(0x7f9d10133360 legacy=0x7f9d1013e510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.121:6789/0 2024-09-17T18:35:40.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:40.323+0000 7f9d14f33640 1 --2- 172.21.15.121:0/668759241 >> v2:172.21.15.173:3300/0 conn(0x7f9d10136800 0x7f9d10136170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:40.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:40.323+0000 7f9d14f33640 10 monclient(hunting): get_auth_request con 0x7f9d10136800 auth_method 0 2024-09-17T18:35:40.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:40.323+0000 7f9d14f33640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:40.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:40.323+0000 7f9d14f33640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:40.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:40.323+0000 7f9d14f33640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:35:40.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:40.323+0000 7f9d14f33640 1 -- 172.21.15.121:0/668759241 >> v2:172.21.15.173:3300/0 conn(0x7f9d10136800 msgr2=0x7f9d10136170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:35:40.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:40.323+0000 7f9d14f33640 1 -- 172.21.15.121:0/668759241 >> v2:172.21.15.173:3300/0 conn(0x7f9d10136800 msgr2=0x7f9d10136170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:35:40.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:40.323+0000 7f9d14f33640 1 --2- 172.21.15.121:0/668759241 >> v2:172.21.15.173:3300/0 conn(0x7f9d10136800 0x7f9d10136170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:35:40.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:40.323+0000 7f9d14f33640 1 --2- 172.21.15.121:0/668759241 >> v2:172.21.15.173:3300/0 conn(0x7f9d10136800 0x7f9d10136170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T18:35:41.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:41.123+0000 7f9d15734640 1 -- 172.21.15.121:0/668759241 >> v1:172.21.15.121:6789/0 conn(0x7f9d10133360 legacy=0x7f9d1013e510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.121:6789/0 2024-09-17T18:35:41.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:41.124+0000 7f9d14f33640 1 --2- 172.21.15.121:0/668759241 >> v2:172.21.15.173:3300/0 conn(0x7f9d10136800 0x7f9d10136170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:41.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:41.124+0000 7f9d14f33640 10 monclient(hunting): get_auth_request con 0x7f9d10136800 auth_method 0 2024-09-17T18:35:41.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:41.124+0000 7f9d14f33640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:41.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:41.124+0000 7f9d14f33640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:41.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:41.124+0000 7f9d14f33640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:35:41.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:41.124+0000 7f9d14f33640 1 -- 172.21.15.121:0/668759241 >> v2:172.21.15.173:3300/0 conn(0x7f9d10136800 msgr2=0x7f9d10136170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:35:41.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:41.124+0000 7f9d14f33640 1 -- 172.21.15.121:0/668759241 >> v2:172.21.15.173:3300/0 conn(0x7f9d10136800 msgr2=0x7f9d10136170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:35:41.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:41.124+0000 7f9d14f33640 1 --2- 172.21.15.121:0/668759241 >> v2:172.21.15.173:3300/0 conn(0x7f9d10136800 0x7f9d10136170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:35:41.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:41.125+0000 7f9d14f33640 1 --2- 172.21.15.121:0/668759241 >> v2:172.21.15.173:3300/0 conn(0x7f9d10136800 0x7f9d10136170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T18:35:42.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:42.719+0000 7f9d067fc640 10 monclient(hunting): tick 2024-09-17T18:35:42.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:42.719+0000 7f9d067fc640 10 monclient(hunting): _check_auth_tickets 2024-09-17T18:35:42.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:42.719+0000 7f9d067fc640 1 monclient(hunting): continuing hunt 2024-09-17T18:35:42.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:42.719+0000 7f9d067fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T18:35:42.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:42.719+0000 7f9d067fc640 1 -- 172.21.15.121:0/668759241 >> v2:172.21.15.173:3300/0 conn(0x7f9d10136800 msgr2=0x7f9d10136170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:35:42.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:42.719+0000 7f9d067fc640 1 --2- 172.21.15.121:0/668759241 >> v2:172.21.15.173:3300/0 conn(0x7f9d10136800 0x7f9d10136170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:42.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:42.719+0000 7f9d067fc640 1 -- 172.21.15.121:0/668759241 >> v1:172.21.15.173:6789/0 conn(0x7f9d10136be0 legacy=0x7f9d10132c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:42.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:42.719+0000 7f9d067fc640 1 -- 172.21.15.121:0/668759241 >> v1:172.21.15.121:6789/0 conn(0x7f9d10133360 legacy=0x7f9d1013e510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:35:42.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:42.719+0000 7f9d067fc640 10 monclient: _add_conns ranks=[2,3] 2024-09-17T18:35:42.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:42.719+0000 7f9d067fc640 1 --2- 172.21.15.121:0/668759241 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cfc003720 0x7f9cfc005b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:42.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:42.719+0000 7f9d067fc640 10 monclient(hunting): picked mon.noname-b con 0x7f9cfc003720 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:35:42.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:42.719+0000 7f9d067fc640 1 --2- 172.21.15.121:0/668759241 >> v2:172.21.15.121:3300/0 conn(0x7f9cfc006230 0x7f9cfc0086f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:42.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:42.719+0000 7f9d067fc640 10 monclient(hunting): picked mon.noname-a con 0x7f9cfc006230 addr v2:172.21.15.121:3300/0 2024-09-17T18:35:42.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:42.719+0000 7f9d067fc640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:42.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:42.719+0000 7f9d067fc640 1 -- 172.21.15.121:0/668759241 --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f9d100a2f90 con 0x7f9cfc006230 2024-09-17T18:35:42.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:42.719+0000 7f9d067fc640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:42.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:42.719+0000 7f9d067fc640 1 -- 172.21.15.121:0/668759241 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9cfc008db0 con 0x7f9cfc003720 2024-09-17T18:35:42.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:42.719+0000 7f9d067fc640 10 monclient(hunting): _renew_subs 2024-09-17T18:35:42.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:42.719+0000 7f9d07fff640 1 -- 172.21.15.121:0/668759241 >> v2:172.21.15.121:3300/0 conn(0x7f9cfc006230 msgr2=0x7f9cfc0086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:35:42.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:42.719+0000 7f9d07fff640 1 --2- 172.21.15.121:0/668759241 >> v2:172.21.15.121:3300/0 conn(0x7f9cfc006230 0x7f9cfc0086f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:35:42.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:42.719+0000 7f9d14f33640 1 --2- 172.21.15.121:0/668759241 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cfc003720 0x7f9cfc005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:42.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:42.719+0000 7f9d14f33640 10 monclient(hunting): get_auth_request con 0x7f9cfc003720 auth_method 0 2024-09-17T18:35:42.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:42.719+0000 7f9d14f33640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:42.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:42.719+0000 7f9d14f33640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:42.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:42.719+0000 7f9d14f33640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:42.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:42.719+0000 7f9d14f33640 1 -- 172.21.15.121:0/668759241 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cfc003720 msgr2=0x7f9cfc005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:35:42.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:42.719+0000 7f9d14f33640 1 -- 172.21.15.121:0/668759241 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cfc003720 msgr2=0x7f9cfc005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:35:42.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:42.719+0000 7f9d14f33640 1 --2- 172.21.15.121:0/668759241 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cfc003720 0x7f9cfc005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:35:42.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:42.719+0000 7f9d14f33640 1 --2- 172.21.15.121:0/668759241 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cfc003720 0x7f9cfc005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:35:42.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:42.920+0000 7f9d07fff640 1 -- 172.21.15.121:0/668759241 >> v2:172.21.15.121:3300/0 conn(0x7f9cfc006230 msgr2=0x7f9cfc0086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:35:42.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:42.920+0000 7f9d07fff640 1 --2- 172.21.15.121:0/668759241 >> v2:172.21.15.121:3300/0 conn(0x7f9cfc006230 0x7f9cfc0086f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T18:35:42.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:42.921+0000 7f9d14f33640 1 --2- 172.21.15.121:0/668759241 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cfc003720 0x7f9cfc005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:42.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:42.921+0000 7f9d14f33640 10 monclient(hunting): get_auth_request con 0x7f9cfc003720 auth_method 0 2024-09-17T18:35:42.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:42.921+0000 7f9d14f33640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:42.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:42.921+0000 7f9d14f33640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:42.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:42.921+0000 7f9d14f33640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:35:42.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:42.921+0000 7f9d14f33640 1 -- 172.21.15.121:0/668759241 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cfc003720 msgr2=0x7f9cfc005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:35:42.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:42.921+0000 7f9d14f33640 1 -- 172.21.15.121:0/668759241 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cfc003720 msgr2=0x7f9cfc005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:35:42.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:42.921+0000 7f9d14f33640 1 --2- 172.21.15.121:0/668759241 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cfc003720 0x7f9cfc005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:35:42.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:42.921+0000 7f9d14f33640 1 --2- 172.21.15.121:0/668759241 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cfc003720 0x7f9cfc005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T18:35:43.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:43.322+0000 7f9d07fff640 1 -- 172.21.15.121:0/668759241 >> v2:172.21.15.121:3300/0 conn(0x7f9cfc006230 msgr2=0x7f9cfc0086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:35:43.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:43.322+0000 7f9d07fff640 1 --2- 172.21.15.121:0/668759241 >> v2:172.21.15.121:3300/0 conn(0x7f9cfc006230 0x7f9cfc0086f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T18:35:43.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:43.322+0000 7f9d14f33640 1 --2- 172.21.15.121:0/668759241 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cfc003720 0x7f9cfc005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:43.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:43.322+0000 7f9d14f33640 10 monclient(hunting): get_auth_request con 0x7f9cfc003720 auth_method 0 2024-09-17T18:35:43.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:43.322+0000 7f9d14f33640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:43.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:43.322+0000 7f9d14f33640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:43.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:43.322+0000 7f9d14f33640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:35:43.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:43.322+0000 7f9d14f33640 1 -- 172.21.15.121:0/668759241 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cfc003720 msgr2=0x7f9cfc005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:35:43.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:43.322+0000 7f9d14f33640 1 -- 172.21.15.121:0/668759241 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cfc003720 msgr2=0x7f9cfc005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:35:43.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:43.322+0000 7f9d14f33640 1 --2- 172.21.15.121:0/668759241 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cfc003720 0x7f9cfc005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:35:43.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:43.322+0000 7f9d14f33640 1 --2- 172.21.15.121:0/668759241 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cfc003720 0x7f9cfc005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T18:35:44.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:44.123+0000 7f9d07fff640 1 -- 172.21.15.121:0/668759241 >> v2:172.21.15.121:3300/0 conn(0x7f9cfc006230 msgr2=0x7f9cfc0086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:35:44.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:44.123+0000 7f9d07fff640 1 --2- 172.21.15.121:0/668759241 >> v2:172.21.15.121:3300/0 conn(0x7f9cfc006230 0x7f9cfc0086f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T18:35:44.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:44.124+0000 7f9d14f33640 1 --2- 172.21.15.121:0/668759241 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cfc003720 0x7f9cfc005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:44.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:44.124+0000 7f9d14f33640 10 monclient(hunting): get_auth_request con 0x7f9cfc003720 auth_method 0 2024-09-17T18:35:44.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:44.124+0000 7f9d14f33640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:44.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:44.124+0000 7f9d14f33640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:44.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:44.124+0000 7f9d14f33640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:35:44.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:44.124+0000 7f9d14f33640 1 -- 172.21.15.121:0/668759241 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cfc003720 msgr2=0x7f9cfc005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T18:35:44.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:44.124+0000 7f9d14f33640 1 -- 172.21.15.121:0/668759241 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cfc003720 msgr2=0x7f9cfc005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:35:44.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:44.124+0000 7f9d14f33640 1 --2- 172.21.15.121:0/668759241 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cfc003720 0x7f9cfc005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:35:44.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:44.124+0000 7f9d14f33640 1 --2- 172.21.15.121:0/668759241 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cfc003720 0x7f9cfc005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T18:35:45.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.719+0000 7f9d067fc640 10 monclient(hunting): tick 2024-09-17T18:35:45.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.719+0000 7f9d067fc640 10 monclient(hunting): _check_auth_tickets 2024-09-17T18:35:45.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.719+0000 7f9d067fc640 1 monclient(hunting): continuing hunt 2024-09-17T18:35:45.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.719+0000 7f9d067fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T18:35:45.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.719+0000 7f9d067fc640 1 -- 172.21.15.121:0/668759241 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cfc003720 msgr2=0x7f9cfc005b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:35:45.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.719+0000 7f9d067fc640 1 --2- 172.21.15.121:0/668759241 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cfc003720 0x7f9cfc005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:45.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.719+0000 7f9d067fc640 1 -- 172.21.15.121:0/668759241 >> v2:172.21.15.121:3300/0 conn(0x7f9cfc006230 msgr2=0x7f9cfc0086f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:35:45.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.719+0000 7f9d067fc640 1 --2- 172.21.15.121:0/668759241 >> v2:172.21.15.121:3300/0 conn(0x7f9cfc006230 0x7f9cfc0086f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:45.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.719+0000 7f9d067fc640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-17T18:35:45.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.719+0000 7f9d15734640 1 -- 172.21.15.121:0/668759241 reap_dead start 2024-09-17T18:35:45.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.719+0000 7f9d067fc640 1 --2- 172.21.15.121:0/668759241 >> v2:172.21.15.173:3300/0 conn(0x7f9d10136800 0x7f9cfc00b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:45.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.719+0000 7f9d067fc640 10 monclient(hunting): picked mon.noname-c con 0x7f9d10136800 addr v2:172.21.15.173:3300/0 2024-09-17T18:35:45.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.719+0000 7f9d067fc640 1 --2- 172.21.15.121:0/668759241 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cfc003720 0x7f9cfc005b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:45.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.719+0000 7f9d067fc640 10 monclient(hunting): picked mon.noname-b con 0x7f9cfc003720 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:35:45.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.719+0000 7f9d067fc640 1 --2- 172.21.15.121:0/668759241 >> v2:172.21.15.121:3300/0 conn(0x7f9cfc0066e0 0x7f9cfc00b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:45.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.719+0000 7f9d067fc640 10 monclient(hunting): picked mon.noname-a con 0x7f9cfc0066e0 addr v2:172.21.15.121:3300/0 2024-09-17T18:35:45.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.719+0000 7f9d067fc640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:45.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.719+0000 7f9d067fc640 1 -- 172.21.15.121:0/668759241 --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f9d100a2f90 con 0x7f9cfc0066e0 2024-09-17T18:35:45.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.719+0000 7f9d067fc640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:45.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.719+0000 7f9d067fc640 1 -- 172.21.15.121:0/668759241 --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f9cfc008db0 con 0x7f9d10136800 2024-09-17T18:35:45.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.719+0000 7f9d067fc640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:45.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.719+0000 7f9d067fc640 1 -- 172.21.15.121:0/668759241 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9cfc00bd80 con 0x7f9cfc003720 2024-09-17T18:35:45.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.719+0000 7f9d067fc640 10 monclient(hunting): _renew_subs 2024-09-17T18:35:45.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.719+0000 7f9d15734640 1 -- 172.21.15.121:0/668759241 >> v2:172.21.15.121:3300/0 conn(0x7f9cfc0066e0 msgr2=0x7f9cfc00b840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:35:45.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.719+0000 7f9d15734640 1 --2- 172.21.15.121:0/668759241 >> v2:172.21.15.121:3300/0 conn(0x7f9cfc0066e0 0x7f9cfc00b840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:35:45.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.719+0000 7f9d07fff640 1 --2- 172.21.15.121:0/668759241 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cfc003720 0x7f9cfc005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:45.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.719+0000 7f9d07fff640 10 monclient(hunting): get_auth_request con 0x7f9cfc003720 auth_method 0 2024-09-17T18:35:45.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.719+0000 7f9d07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:45.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.719+0000 7f9d07fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:45.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.719+0000 7f9d07fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:45.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.720+0000 7f9d07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:45.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.720+0000 7f9d07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:45.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.720+0000 7f9d07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:45.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.720+0000 7f9d14f33640 1 --2- 172.21.15.121:0/668759241 >> v2:172.21.15.173:3300/0 conn(0x7f9d10136800 0x7f9cfc00b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:45.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.720+0000 7f9d14f33640 10 monclient(hunting): get_auth_request con 0x7f9d10136800 auth_method 0 2024-09-17T18:35:45.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.720+0000 7f9d14f33640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:45.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.720+0000 7f9d14f33640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:45.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.720+0000 7f9d14f33640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:45.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.720+0000 7f9d07fff640 10 monclient(hunting): handle_auth_done global_id 64532 payload 274 2024-09-17T18:35:45.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.720+0000 7f9d07fff640 1 -- 172.21.15.121:0/668759241 >> v2:172.21.15.173:3300/0 conn(0x7f9d10136800 msgr2=0x7f9cfc00b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:35:45.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.720+0000 7f9d07fff640 1 --2- 172.21.15.121:0/668759241 >> v2:172.21.15.173:3300/0 conn(0x7f9d10136800 0x7f9cfc00b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:45.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.720+0000 7f9d07fff640 1 -- 172.21.15.121:0/668759241 >> v2:172.21.15.121:3300/0 conn(0x7f9cfc0066e0 msgr2=0x7f9cfc00b840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:35:45.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.720+0000 7f9d07fff640 1 --2- 172.21.15.121:0/668759241 >> v2:172.21.15.121:3300/0 conn(0x7f9cfc0066e0 0x7f9cfc00b840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:45.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.720+0000 7f9d07fff640 10 monclient: _finish_hunting 0 2024-09-17T18:35:45.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.720+0000 7f9d07fff640 1 monclient: found mon.noname-b 2024-09-17T18:35:45.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.720+0000 7f9d07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:35:45.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.720+0000 7f9d07fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:35:45.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.720+0000 7f9d07fff640 1 -- 172.21.15.121:0/668759241 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d100a3270 con 0x7f9cfc003720 2024-09-17T18:35:45.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.723+0000 7f9d07fff640 1 -- 172.21.15.121:0/668759241 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cfc003720 msgr2=0x7f9cfc005b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:45.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.723+0000 7f9d07fff640 1 --2- 172.21.15.121:0/668759241 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cfc003720 0x7f9cfc005b70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9d1006f3b0 tx=0x7f9d0002a440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9a96a917c84a1e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:45.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.723+0000 7f9d077fe640 1 -- 172.21.15.121:0/668759241 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d00007330 con 0x7f9cfc003720 2024-09-17T18:35:45.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.723+0000 7f9d077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:45.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.723+0000 7f9d077fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:35:45.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.723+0000 7f9d077fe640 10 monclient: dump: 2024-09-17T18:35:45.732 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:45.732 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:45.732 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:45.732 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:45.732 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:45.732 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:45.732 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:45.732 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:45.732 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:45.733 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:45.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.723+0000 7f9d077fe640 10 monclient: _finish_auth 0 2024-09-17T18:35:45.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.723+0000 7f9d077fe640 10 monclient: _check_auth_tickets 2024-09-17T18:35:45.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.723+0000 7f9d077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:35:45.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.723+0000 7f9d077fe640 1 -- 172.21.15.121:0/668759241 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d000074d0 con 0x7f9cfc003720 2024-09-17T18:35:45.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.723+0000 7f9d077fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:35:45.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.723+0000 7f9d077fe640 1 -- 172.21.15.121:0/668759241 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d000049e0 con 0x7f9cfc003720 2024-09-17T18:35:45.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.723+0000 7f9d17197640 5 monclient: authenticate success, global_id 64532 2024-09-17T18:35:45.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.723+0000 7f9d17197640 10 monclient: get_monmap_and_config success 2024-09-17T18:35:45.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.723+0000 7f9d17197640 4 set_mon_vals no callback set 2024-09-17T18:35:45.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.723+0000 7f9d077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:45.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.723+0000 7f9d077fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:35:45.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.723+0000 7f9d077fe640 10 monclient: dump: 2024-09-17T18:35:45.734 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:45.734 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:45.734 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:45.734 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:45.734 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:45.734 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:45.734 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:45.734 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:45.734 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:45.734 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:45.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.724+0000 7f9d17197640 10 monclient: shutdown 2024-09-17T18:35:45.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.724+0000 7f9d17197640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:35:45.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.724+0000 7f9d17197640 1 -- 172.21.15.121:0/668759241 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cfc003720 msgr2=0x7f9cfc005b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:45.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.724+0000 7f9d17197640 1 --2- 172.21.15.121:0/668759241 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cfc003720 0x7f9cfc005b70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9d1006f3b0 tx=0x7f9d0002a440 comp rx=0 tx=0).stop 2024-09-17T18:35:45.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.724+0000 7f9d17197640 1 -- 172.21.15.121:0/668759241 shutdown_connections 2024-09-17T18:35:45.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.724+0000 7f9d17197640 1 --2- 172.21.15.121:0/668759241 >> v2:172.21.15.121:3300/0 conn(0x7f9cfc0066e0 0x7f9cfc00b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:45.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.724+0000 7f9d17197640 1 --2- 172.21.15.121:0/668759241 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cfc003720 0x7f9cfc005b70 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:45.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.724+0000 7f9d17197640 1 --2- 172.21.15.121:0/668759241 >> v2:172.21.15.173:3300/0 conn(0x7f9d10136800 0x7f9cfc00b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:45.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.724+0000 7f9d17197640 1 -- 172.21.15.121:0/668759241 >> 172.21.15.121:0/668759241 conn(0x7f9d100a3b50 msgr2=0x7f9d100a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:35:45.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.724+0000 7f9d17197640 1 -- 172.21.15.121:0/668759241 shutdown_connections 2024-09-17T18:35:45.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.724+0000 7f9d17197640 1 -- 172.21.15.121:0/668759241 wait complete. 2024-09-17T18:35:45.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.725+0000 7f9d17197640 10 monclient: build_initial_monmap 2024-09-17T18:35:45.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.725+0000 7f9d17197640 10 monclient: monmap: 2024-09-17T18:35:45.735 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:35:45.735 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:35:45.735 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:35:45.735 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:35:45.735 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:35:45.735 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:45.735 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:35:45.736 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:35:45.736 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:35:45.736 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:45.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.725+0000 7f9d17197640 1 librados: starting msgr at 2024-09-17T18:35:45.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.725+0000 7f9d17197640 1 librados: starting objecter 2024-09-17T18:35:45.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.725+0000 7f9d17197640 1 Processor -- start 2024-09-17T18:35:45.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.725+0000 7f9d17197640 1 -- start start 2024-09-17T18:35:45.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.725+0000 7f9d17197640 1 librados: setting wanted keys 2024-09-17T18:35:45.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.725+0000 7f9d17197640 1 librados: calling monclient init 2024-09-17T18:35:45.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.725+0000 7f9d17197640 10 monclient: init 2024-09-17T18:35:45.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.726+0000 7f9d17197640 10 monclient: _reopen_session rank -1 2024-09-17T18:35:45.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.726+0000 7f9d17197640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T18:35:45.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.726+0000 7f9d17197640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cfc003720 0x7f9d101cb600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:45.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.726+0000 7f9d17197640 10 monclient(hunting): picked mon.noname-c con 0x7f9cfc003720 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:35:45.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.726+0000 7f9d17197640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9cfc0066e0 0x7f9d1012fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:45.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.726+0000 7f9d17197640 10 monclient(hunting): picked mon.noname-a con 0x7f9cfc0066e0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:35:45.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.726+0000 7f9d17197640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d10136800 0x7f9d101301b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:45.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.726+0000 7f9d17197640 10 monclient(hunting): picked mon.noname-b con 0x7f9d10136800 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:35:45.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.726+0000 7f9d17197640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:45.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.726+0000 7f9d17197640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1009dce0 con 0x7f9cfc0066e0 2024-09-17T18:35:45.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.726+0000 7f9d17197640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:45.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.726+0000 7f9d17197640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9d10137050 con 0x7f9d10136800 2024-09-17T18:35:45.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.726+0000 7f9d17197640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:45.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.726+0000 7f9d17197640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9d1009de60 con 0x7f9cfc003720 2024-09-17T18:35:45.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.726+0000 7f9d17197640 10 monclient(hunting): _renew_subs 2024-09-17T18:35:45.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.726+0000 7f9d17197640 10 monclient(hunting): authenticate will time out at 1749.443648s 2024-09-17T18:35:45.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.726+0000 7f9d07fff640 1 -- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9cfc0066e0 msgr2=0x7f9d1012fc70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:35:45.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.726+0000 7f9d07fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9cfc0066e0 0x7f9d1012fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:35:45.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.726+0000 7f9d14f33640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cfc003720 0x7f9d101cb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:45.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.726+0000 7f9d14f33640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cfc003720 0x7f9d101cb600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:37092/0 (socket says 172.21.15.121:37092) 2024-09-17T18:35:45.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.726+0000 7f9d14f33640 1 -- 172.21.15.121:0/4270666950 learned_addr learned my addr 172.21.15.121:0/4270666950 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:35:45.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.726+0000 7f9d14f33640 10 monclient(hunting): get_auth_request con 0x7f9cfc003720 auth_method 0 2024-09-17T18:35:45.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.726+0000 7f9d14f33640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:45.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.726+0000 7f9d14f33640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:45.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.726+0000 7f9d14f33640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:45.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.726+0000 7f9d14f33640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:45.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.726+0000 7f9d14f33640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:45.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.726+0000 7f9d14f33640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:45.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.727+0000 7f9d15734640 1 --2- 172.21.15.121:0/4270666950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d10136800 0x7f9d101301b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:45.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.727+0000 7f9d15734640 10 monclient(hunting): get_auth_request con 0x7f9d10136800 auth_method 0 2024-09-17T18:35:45.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.727+0000 7f9d15734640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:45.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.727+0000 7f9d15734640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:45.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.727+0000 7f9d15734640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:45.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.727+0000 7f9d14f33640 10 monclient(hunting): handle_auth_done global_id 64535 payload 824 2024-09-17T18:35:45.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.727+0000 7f9d14f33640 1 -- 172.21.15.121:0/4270666950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d10136800 msgr2=0x7f9d101301b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:45.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.727+0000 7f9d14f33640 1 --2- 172.21.15.121:0/4270666950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d10136800 0x7f9d101301b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:45.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.727+0000 7f9d14f33640 1 -- 172.21.15.121:0/4270666950 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9cfc0066e0 msgr2=0x7f9d1012fc70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:35:45.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.727+0000 7f9d14f33640 1 --2- 172.21.15.121:0/4270666950 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9cfc0066e0 0x7f9d1012fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:45.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.727+0000 7f9d14f33640 10 monclient: _finish_hunting 0 2024-09-17T18:35:45.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.727+0000 7f9d14f33640 1 monclient: found mon.noname-c 2024-09-17T18:35:45.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.727+0000 7f9d14f33640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:35:45.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.727+0000 7f9d14f33640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:35:45.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.727+0000 7f9d14f33640 1 -- 172.21.15.121:0/4270666950 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d10121670 con 0x7f9cfc003720 2024-09-17T18:35:45.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.727+0000 7f9d15734640 1 --2- 172.21.15.121:0/4270666950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d10136800 0x7f9d101301b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T18:35:45.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.729+0000 7f9d14f33640 1 -- 172.21.15.121:0/4270666950 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cfc003720 msgr2=0x7f9d101cb600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:45.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.729+0000 7f9d14f33640 1 --2- 172.21.15.121:0/4270666950 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cfc003720 0x7f9d101cb600 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c0198a0 tx=0x7f9d0c019d60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:45.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.729+0000 7f9d057fa640 10 client.?.objecter ms_handle_connect 0x7f9cfc003720 2024-09-17T18:35:45.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.729+0000 7f9d057fa640 10 client.?.objecter resend_mon_ops 2024-09-17T18:35:45.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.729+0000 7f9d057fa640 1 -- 172.21.15.121:0/4270666950 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d0c00f040 con 0x7f9cfc003720 2024-09-17T18:35:45.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.729+0000 7f9d057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:45.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.729+0000 7f9d057fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:35:45.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.729+0000 7f9d057fa640 10 monclient: dump: 2024-09-17T18:35:45.745 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:45.745 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:45.745 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:45.745 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:45.745 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:45.745 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:45.746 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:45.746 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:45.746 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:45.746 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:45.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.729+0000 7f9d057fa640 10 monclient: _finish_auth 0 2024-09-17T18:35:45.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.729+0000 7f9d057fa640 10 monclient: _check_auth_tickets 2024-09-17T18:35:45.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.729+0000 7f9d057fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:35:45.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.729+0000 7f9d057fa640 1 -- 172.21.15.121:0/4270666950 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d0c028070 con 0x7f9cfc003720 2024-09-17T18:35:45.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.729+0000 7f9d17197640 5 monclient: authenticate success, global_id 64535 2024-09-17T18:35:45.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.729+0000 7f9d17197640 10 monclient: _renew_subs 2024-09-17T18:35:45.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.729+0000 7f9d17197640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:45.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.729+0000 7f9d17197640 1 -- 172.21.15.121:0/4270666950 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d1009e240 con 0x7f9cfc003720 2024-09-17T18:35:45.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.729+0000 7f9d057fa640 10 monclient: handle_config config(0 keys) 2024-09-17T18:35:45.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.729+0000 7f9d057fa640 1 -- 172.21.15.121:0/4270666950 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d0c0235f0 con 0x7f9cfc003720 2024-09-17T18:35:45.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.729+0000 7f9d057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:45.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.729+0000 7f9d057fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:35:45.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.729+0000 7f9d057fa640 10 monclient: dump: 2024-09-17T18:35:45.747 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:45.747 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:45.747 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:45.747 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:45.747 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:45.747 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:45.747 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:45.748 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:45.748 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:45.748 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:45.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.729+0000 7f9d06ffd640 4 set_mon_vals no callback set 2024-09-17T18:35:45.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.729+0000 7f9d17197640 10 client.64535.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:35:45.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.729+0000 7f9d17197640 10 monclient: _renew_subs 2024-09-17T18:35:45.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.729+0000 7f9d17197640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:45.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.729+0000 7f9d17197640 1 -- 172.21.15.121:0/4270666950 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d10121390 con 0x7f9cfc003720 2024-09-17T18:35:45.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.729+0000 7f9d17197640 1 librados: init done 2024-09-17T18:35:45.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.730+0000 7f9d057fa640 1 -- 172.21.15.121:0/4270666950 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f9d0c023810 con 0x7f9cfc003720 2024-09-17T18:35:45.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.731+0000 7f9d057fa640 1 --2- 172.21.15.121:0/4270666950 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9cfc0415a0 0x7f9cfc043a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:45.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.731+0000 7f9d057fa640 1 -- 172.21.15.121:0/4270666950 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(648..648 src has 247..648) ==== 5917+0+0 (secure 0 0 0) 0x7f9d0c055770 con 0x7f9cfc003720 2024-09-17T18:35:45.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.731+0000 7f9d057fa640 10 client.64535.objecter ms_dispatch 0x7f9d10000b90 osd_map(648..648 src has 247..648) 2024-09-17T18:35:45.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.731+0000 7f9d17197640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:45.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.731+0000 7f9d057fa640 3 client.64535.objecter handle_osd_map got epochs [648,648] > 0 2024-09-17T18:35:45.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.731+0000 7f9d057fa640 3 client.64535.objecter handle_osd_map decoding full epoch 648 2024-09-17T18:35:45.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.731+0000 7f9d17197640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:45.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.731+0000 7f9d17197640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:45.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.731+0000 7f9d17197640 1 -- 172.21.15.121:0/4270666950 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d101338d0 con 0x7f9cfc003720 2024-09-17T18:35:45.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.731+0000 7f9d057fa640 20 client.64535.objecter dump_active .. 0 homeless 2024-09-17T18:35:45.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.731+0000 7f9d07fff640 1 --2- 172.21.15.121:0/4270666950 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9cfc0415a0 0x7f9cfc043a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:45.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.731+0000 7f9d07fff640 10 monclient: get_auth_request con 0x7f9cfc0415a0 auth_method 0 2024-09-17T18:35:45.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.734+0000 7f9d057fa640 10 client.64535.objecter ms_handle_connect 0x7f9cfc0415a0 2024-09-17T18:35:45.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.734+0000 7f9d07fff640 1 --2- 172.21.15.121:0/4270666950 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9cfc0415a0 0x7f9cfc043a60 secure :-1 s=READY pgs=4083 cs=0 l=1 rev1=1 crypto rx=0x7f9d00007fd0 tx=0x7f9d00009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:45.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.736+0000 7f9d057fa640 1 -- 172.21.15.121:0/4270666950 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9d0c020070 con 0x7f9cfc003720 2024-09-17T18:35:45.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.736+0000 7f9d057fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:45.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.736+0000 7f9d057fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:35:45.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.982+0000 7f9d17197640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:35:45.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.982+0000 7f9d17197640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:35:45.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.982+0000 7f9d17197640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:45.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.982+0000 7f9d17197640 1 -- 172.21.15.121:0/4270666950 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9d1006ce60 con 0x7f9cfc003720 2024-09-17T18:35:45.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.982+0000 7f9d057fa640 1 -- 172.21.15.121:0/4270666950 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f9d0c015360 con 0x7f9cfc003720 2024-09-17T18:35:45.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.982+0000 7f9d057fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:35:45.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.982+0000 7f9d057fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:35:45.986 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":62,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:35:45.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.984+0000 7f9d17197640 10 librados: watch_flush enter 2024-09-17T18:35:45.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.984+0000 7f9d17197640 10 librados: watch_flush exit 2024-09-17T18:35:45.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.984+0000 7f9d17197640 20 client.64535.objecter shutdown clearing up homeless session... 2024-09-17T18:35:45.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.984+0000 7f9d17197640 10 client.64535.objecter successfully canceled tick 2024-09-17T18:35:45.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.984+0000 7f9d17197640 1 -- 172.21.15.121:0/4270666950 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9cfc0415a0 msgr2=0x7f9cfc043a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:45.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.984+0000 7f9d17197640 1 --2- 172.21.15.121:0/4270666950 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9cfc0415a0 0x7f9cfc043a60 secure :-1 s=READY pgs=4083 cs=0 l=1 rev1=1 crypto rx=0x7f9d00007fd0 tx=0x7f9d00009030 comp rx=0 tx=0).stop 2024-09-17T18:35:45.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.984+0000 7f9d17197640 10 monclient: shutdown 2024-09-17T18:35:45.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.984+0000 7f9d17197640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:35:45.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.984+0000 7f9d17197640 1 -- 172.21.15.121:0/4270666950 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cfc003720 msgr2=0x7f9d101cb600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:45.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.984+0000 7f9d17197640 1 --2- 172.21.15.121:0/4270666950 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cfc003720 0x7f9d101cb600 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c0198a0 tx=0x7f9d0c019d60 comp rx=0 tx=0).stop 2024-09-17T18:35:45.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.984+0000 7f9d17197640 1 -- 172.21.15.121:0/4270666950 shutdown_connections 2024-09-17T18:35:45.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.984+0000 7f9d17197640 1 --2- 172.21.15.121:0/4270666950 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9cfc0415a0 0x7f9cfc043a60 unknown :-1 s=CLOSED pgs=4083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:45.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.984+0000 7f9d17197640 1 --2- 172.21.15.121:0/4270666950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d10136800 0x7f9d101301b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:45.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.984+0000 7f9d17197640 1 --2- 172.21.15.121:0/4270666950 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9cfc0066e0 0x7f9d1012fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:45.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.984+0000 7f9d17197640 1 --2- 172.21.15.121:0/4270666950 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cfc003720 0x7f9d101cb600 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:45.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.984+0000 7f9d17197640 1 -- 172.21.15.121:0/4270666950 >> 172.21.15.121:0/4270666950 conn(0x7f9d100a3b50 msgr2=0x7f9d10131990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:35:45.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.984+0000 7f9d17197640 1 -- 172.21.15.121:0/4270666950 shutdown_connections 2024-09-17T18:35:45.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.984+0000 7f9d17197640 1 -- 172.21.15.121:0/4270666950 wait complete. 2024-09-17T18:35:45.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:45.984+0000 7f9d17197640 1 librados: shutdown 2024-09-17T18:35:45.998 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-17T18:35:45.998 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-17T18:35:45.999 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T18:35:46.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.094+0000 7fb4917a2640 10 monclient: get_monmap_and_config 2024-09-17T18:35:46.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.094+0000 7fb4917a2640 10 monclient: build_initial_monmap 2024-09-17T18:35:46.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.094+0000 7fb4917a2640 10 monclient: monmap: 2024-09-17T18:35:46.097 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:35:46.097 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:46.097 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:35:46.096207+0000 2024-09-17T18:35:46.097 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:35:46.096207+0000 2024-09-17T18:35:46.098 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:35:46.098 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:46.098 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:35:46.098 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:35:46.098 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:35:46.098 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:35:46.098 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:35:46.098 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:46.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.095+0000 7fb4917a2640 1 Processor -- start 2024-09-17T18:35:46.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.095+0000 7fb4917a2640 1 -- start start 2024-09-17T18:35:46.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.095+0000 7fb4917a2640 10 monclient: init 2024-09-17T18:35:46.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.095+0000 7fb4917a2640 10 monclient: _reopen_session rank -1 2024-09-17T18:35:46.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.095+0000 7fb4917a2640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-17T18:35:46.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.095+0000 7fb4917a2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb48c0a5b90 addr v1:172.21.15.121:6789/0 2024-09-17T18:35:46.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.095+0000 7fb4917a2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb48c09d830 addr v1:172.21.15.173:6789/0 2024-09-17T18:35:46.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.095+0000 7fb4917a2640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb48c0a7080 0x7fb48c1412c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:46.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.095+0000 7fb4917a2640 10 monclient(hunting): picked mon.noname-b con 0x7fb48c0a7080 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:35:46.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.095+0000 7fb48affd640 1 -- >> v1:172.21.15.121:6789/0 conn(0x7fb48c0a5b90 legacy=0x7fb48c0a5f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.121:6789/0 2024-09-17T18:35:46.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.095+0000 7fb4917a2640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb48c0a8690 con 0x7fb48c0a5b90 2024-09-17T18:35:46.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.095+0000 7fb4917a2640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb48c09cb00 con 0x7fb48c09d830 2024-09-17T18:35:46.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.095+0000 7fb4917a2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:46.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.095+0000 7fb4917a2640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb48c09cde0 con 0x7fb48c0a7080 2024-09-17T18:35:46.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.095+0000 7fb4917a2640 10 monclient(hunting): _renew_subs 2024-09-17T18:35:46.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.095+0000 7fb4917a2640 10 monclient(hunting): authenticate will time out at 1749.812834s 2024-09-17T18:35:46.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.095+0000 7fb48b7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb48c0a7080 0x7fb48c1412c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:46.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.095+0000 7fb48b7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb48c0a7080 0x7fb48c1412c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:37096/0 (socket says 172.21.15.121:37096) 2024-09-17T18:35:46.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.095+0000 7fb48b7fe640 1 -- 172.21.15.121:0/3065784217 learned_addr learned my addr 172.21.15.121:0/3065784217 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:35:46.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.095+0000 7fb48b7fe640 10 monclient(hunting): get_auth_request con 0x7fb48c0a7080 auth_method 0 2024-09-17T18:35:46.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.095+0000 7fb48b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:46.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.095+0000 7fb48b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:46.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.095+0000 7fb48b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:46.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.096+0000 7fb48b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:46.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.096+0000 7fb48b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:46.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.096+0000 7fb48b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:46.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.096+0000 7fb48b7fe640 10 monclient(hunting): handle_auth_done global_id 64538 payload 274 2024-09-17T18:35:46.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.096+0000 7fb48b7fe640 1 -- 172.21.15.121:0/3065784217 >> v1:172.21.15.173:6789/0 conn(0x7fb48c09d830 legacy=0x7fb48c13db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:35:46.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.096+0000 7fb48b7fe640 1 -- 172.21.15.121:0/3065784217 >> v1:172.21.15.121:6789/0 conn(0x7fb48c0a5b90 legacy=0x7fb48c0a5f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:35:46.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.096+0000 7fb48b7fe640 10 monclient: _finish_hunting 0 2024-09-17T18:35:46.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.096+0000 7fb48b7fe640 1 monclient: found mon.noname-b 2024-09-17T18:35:46.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.096+0000 7fb48b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:35:46.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.096+0000 7fb48b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:35:46.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.096+0000 7fb48b7fe640 1 -- 172.21.15.121:0/3065784217 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb48c09d0c0 con 0x7fb48c0a7080 2024-09-17T18:35:46.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.098+0000 7fb48b7fe640 1 -- 172.21.15.121:0/3065784217 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb48c0a7080 msgr2=0x7fb48c1412c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:46.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.099+0000 7fb48b7fe640 1 --2- 172.21.15.121:0/3065784217 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb48c0a7080 0x7fb48c1412c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb484009e80 tx=0x7fb48402a530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=669e165d8f260108 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:46.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.099+0000 7fb489ffb640 1 -- 172.21.15.121:0/3065784217 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb484036070 con 0x7fb48c0a7080 2024-09-17T18:35:46.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.099+0000 7fb489ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:46.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.099+0000 7fb489ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:35:46.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.099+0000 7fb489ffb640 10 monclient: dump: 2024-09-17T18:35:46.111 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:46.111 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:46.111 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:46.111 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:46.111 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:46.111 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:46.111 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:46.111 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:46.111 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:46.111 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:46.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.099+0000 7fb489ffb640 10 monclient: _finish_auth 0 2024-09-17T18:35:46.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.099+0000 7fb489ffb640 10 monclient: _check_auth_tickets 2024-09-17T18:35:46.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.099+0000 7fb489ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:35:46.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.099+0000 7fb489ffb640 1 -- 172.21.15.121:0/3065784217 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb484007800 con 0x7fb48c0a7080 2024-09-17T18:35:46.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.099+0000 7fb489ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:35:46.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.099+0000 7fb489ffb640 1 -- 172.21.15.121:0/3065784217 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb484007bd0 con 0x7fb48c0a7080 2024-09-17T18:35:46.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.099+0000 7fb4917a2640 5 monclient: authenticate success, global_id 64538 2024-09-17T18:35:46.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.099+0000 7fb4917a2640 10 monclient: get_monmap_and_config success 2024-09-17T18:35:46.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.099+0000 7fb4917a2640 4 set_mon_vals no callback set 2024-09-17T18:35:46.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.099+0000 7fb489ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:46.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.099+0000 7fb489ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:35:46.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.099+0000 7fb489ffb640 10 monclient: dump: 2024-09-17T18:35:46.112 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:46.112 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:46.113 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:46.113 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:46.113 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:46.113 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:46.113 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:46.113 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:46.113 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:46.113 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:46.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.099+0000 7fb4917a2640 10 monclient: shutdown 2024-09-17T18:35:46.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.099+0000 7fb4917a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:35:46.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.099+0000 7fb4917a2640 1 -- 172.21.15.121:0/3065784217 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb48c0a7080 msgr2=0x7fb48c1412c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:46.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.099+0000 7fb4917a2640 1 --2- 172.21.15.121:0/3065784217 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb48c0a7080 0x7fb48c1412c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb484009e80 tx=0x7fb48402a530 comp rx=0 tx=0).stop 2024-09-17T18:35:46.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.099+0000 7fb4917a2640 1 -- 172.21.15.121:0/3065784217 shutdown_connections 2024-09-17T18:35:46.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.099+0000 7fb4917a2640 1 --2- 172.21.15.121:0/3065784217 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb48c0a7080 0x7fb48c1412c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:46.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.099+0000 7fb4917a2640 1 -- 172.21.15.121:0/3065784217 >> 172.21.15.121:0/3065784217 conn(0x7fb48c0ab880 msgr2=0x7fb48c0a7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:35:46.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.099+0000 7fb4917a2640 1 -- 172.21.15.121:0/3065784217 shutdown_connections 2024-09-17T18:35:46.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.099+0000 7fb4917a2640 1 -- 172.21.15.121:0/3065784217 wait complete. 2024-09-17T18:35:46.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.100+0000 7fb4917a2640 10 monclient: build_initial_monmap 2024-09-17T18:35:46.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.100+0000 7fb4917a2640 10 monclient: monmap: 2024-09-17T18:35:46.114 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:35:46.114 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:35:46.114 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:35:46.114 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:35:46.114 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:35:46.114 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:46.114 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:35:46.114 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:35:46.114 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:35:46.114 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:46.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.100+0000 7fb4917a2640 1 librados: starting msgr at 2024-09-17T18:35:46.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.100+0000 7fb4917a2640 1 librados: starting objecter 2024-09-17T18:35:46.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.100+0000 7fb4917a2640 1 Processor -- start 2024-09-17T18:35:46.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.100+0000 7fb4917a2640 1 -- start start 2024-09-17T18:35:46.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.100+0000 7fb4917a2640 1 librados: setting wanted keys 2024-09-17T18:35:46.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.100+0000 7fb4917a2640 1 librados: calling monclient init 2024-09-17T18:35:46.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.100+0000 7fb4917a2640 10 monclient: init 2024-09-17T18:35:46.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.101+0000 7fb4917a2640 10 monclient: _reopen_session rank -1 2024-09-17T18:35:46.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.101+0000 7fb4917a2640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:35:46.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.101+0000 7fb4917a2640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb48c09d830 0x7fb48c13fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:46.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.101+0000 7fb4917a2640 10 monclient(hunting): picked mon.noname-a con 0x7fb48c09d830 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:35:46.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.101+0000 7fb4917a2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb48c0a5b90 0x7fb48c1cdea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:46.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.101+0000 7fb4917a2640 10 monclient(hunting): picked mon.noname-b con 0x7fb48c0a5b90 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:35:46.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.101+0000 7fb4917a2640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb48c0a7080 0x7fb48c13d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:46.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.101+0000 7fb4917a2640 10 monclient(hunting): picked mon.noname-c con 0x7fb48c0a7080 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:35:46.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.101+0000 7fb4917a2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:46.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.101+0000 7fb4917a2640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb48c0a6800 con 0x7fb48c09d830 2024-09-17T18:35:46.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.101+0000 7fb4917a2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:46.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.101+0000 7fb4917a2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb48c0a6680 con 0x7fb48c0a5b90 2024-09-17T18:35:46.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.101+0000 7fb4917a2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:46.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.101+0000 7fb4917a2640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb48c0a6980 con 0x7fb48c0a7080 2024-09-17T18:35:46.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.101+0000 7fb4917a2640 10 monclient(hunting): _renew_subs 2024-09-17T18:35:46.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.101+0000 7fb4917a2640 10 monclient(hunting): authenticate will time out at 1749.818293s 2024-09-17T18:35:46.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.101+0000 7fb48affd640 1 -- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb48c09d830 msgr2=0x7fb48c13fd30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:35:46.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.101+0000 7fb48affd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb48c09d830 0x7fb48c13fd30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:35:46.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.101+0000 7fb48b7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb48c0a7080 0x7fb48c13d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:46.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.101+0000 7fb48b7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb48c0a7080 0x7fb48c13d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:37102/0 (socket says 172.21.15.121:37102) 2024-09-17T18:35:46.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.101+0000 7fb48b7fe640 1 -- 172.21.15.121:0/999112370 learned_addr learned my addr 172.21.15.121:0/999112370 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:35:46.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.101+0000 7fb48b7fe640 10 monclient(hunting): get_auth_request con 0x7fb48c0a7080 auth_method 0 2024-09-17T18:35:46.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.101+0000 7fb48b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:46.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.101+0000 7fb48b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:46.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.101+0000 7fb48b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:46.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.101+0000 7fb48b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:46.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.101+0000 7fb48b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:46.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.101+0000 7fb48b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:46.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.101+0000 7fb48b7fe640 10 monclient(hunting): handle_auth_done global_id 64541 payload 824 2024-09-17T18:35:46.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.101+0000 7fb48a7fc640 1 --2- 172.21.15.121:0/999112370 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb48c0a5b90 0x7fb48c1cdea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:46.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.101+0000 7fb48b7fe640 1 -- 172.21.15.121:0/999112370 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb48c0a5b90 msgr2=0x7fb48c1cdea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:46.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.101+0000 7fb48b7fe640 1 --2- 172.21.15.121:0/999112370 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb48c0a5b90 0x7fb48c1cdea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:46.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.101+0000 7fb48b7fe640 1 -- 172.21.15.121:0/999112370 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb48c09d830 msgr2=0x7fb48c13fd30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:35:46.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.101+0000 7fb48b7fe640 1 --2- 172.21.15.121:0/999112370 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb48c09d830 0x7fb48c13fd30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:46.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.101+0000 7fb48b7fe640 10 monclient: _finish_hunting 0 2024-09-17T18:35:46.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.101+0000 7fb48b7fe640 1 monclient: found mon.noname-c 2024-09-17T18:35:46.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.101+0000 7fb48b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:35:46.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.101+0000 7fb48b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:35:46.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.101+0000 7fb48b7fe640 1 -- 172.21.15.121:0/999112370 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb48c121b80 con 0x7fb48c0a7080 2024-09-17T18:35:46.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.103+0000 7fb48b7fe640 1 -- 172.21.15.121:0/999112370 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb48c0a7080 msgr2=0x7fb48c13d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:46.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.103+0000 7fb48b7fe640 1 --2- 172.21.15.121:0/999112370 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb48c0a7080 0x7fb48c13d920 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb4840093e0 tx=0x7fb484004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:46.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.103+0000 7fb4737fe640 10 client.?.objecter ms_handle_connect 0x7fb48c0a7080 2024-09-17T18:35:46.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.103+0000 7fb4737fe640 10 client.?.objecter resend_mon_ops 2024-09-17T18:35:46.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.104+0000 7fb4737fe640 1 -- 172.21.15.121:0/999112370 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb484041020 con 0x7fb48c0a7080 2024-09-17T18:35:46.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.104+0000 7fb4737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:46.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.104+0000 7fb48b7fe640 1 --2- 172.21.15.121:0/999112370 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb48c0a7080 0x7fb48c13d920 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb4840093e0 tx=0x7fb484004a40 comp rx=0 tx=0).queue_received will delay after 1.7266e+09 on 0x7fb484031940 config(0 keys) 2024-09-17T18:35:46.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.104+0000 7fb4737fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:35:46.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.104+0000 7fb4737fe640 10 monclient: dump: 2024-09-17T18:35:46.118 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:46.119 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:46.119 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:46.119 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:46.119 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:46.119 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:46.119 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:46.119 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:46.119 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:46.119 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:46.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.104+0000 7fb4737fe640 10 monclient: _finish_auth 0 2024-09-17T18:35:46.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.104+0000 7fb4737fe640 10 monclient: _check_auth_tickets 2024-09-17T18:35:46.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.104+0000 7fb4737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:35:46.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.104+0000 7fb4737fe640 1 -- 172.21.15.121:0/999112370 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb484031940 con 0x7fb48c0a7080 2024-09-17T18:35:46.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.104+0000 7fb4917a2640 5 monclient: authenticate success, global_id 64541 2024-09-17T18:35:46.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.104+0000 7fb4737fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:35:46.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.104+0000 7fb4917a2640 10 monclient: _renew_subs 2024-09-17T18:35:46.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.104+0000 7fb4917a2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:46.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.104+0000 7fb4917a2640 1 -- 172.21.15.121:0/999112370 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb48c121e60 con 0x7fb48c0a7080 2024-09-17T18:35:46.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.104+0000 7fb4897fa640 4 set_mon_vals no callback set 2024-09-17T18:35:46.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.104+0000 7fb4917a2640 10 client.64541.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:35:46.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.104+0000 7fb4917a2640 10 monclient: _renew_subs 2024-09-17T18:35:46.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.104+0000 7fb4917a2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:46.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.104+0000 7fb4917a2640 1 -- 172.21.15.121:0/999112370 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb48c13f1d0 con 0x7fb48c0a7080 2024-09-17T18:35:46.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.104+0000 7fb4917a2640 1 librados: init done 2024-09-17T18:35:46.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.105+0000 7fb4917a2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:46.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.105+0000 7fb4917a2640 1 --2- 172.21.15.121:0/999112370 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb48c1ce3e0 0x7fb48c13de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:46.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.105+0000 7fb4917a2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:46.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.105+0000 7fb4917a2640 1 -- 172.21.15.121:0/999112370 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb48c1218a0 con 0x7fb48c1ce3e0 2024-09-17T18:35:46.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.105+0000 7fb4737fe640 1 -- 172.21.15.121:0/999112370 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb484036040 con 0x7fb48c0a7080 2024-09-17T18:35:46.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.105+0000 7fb4737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:46.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.105+0000 7fb4737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:35:46.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.105+0000 7fb4737fe640 10 monclient: dump: 2024-09-17T18:35:46.121 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:46.121 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:46.121 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:46.121 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:46.121 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:46.121 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:46.121 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:46.122 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:46.122 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:46.122 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:46.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.105+0000 7fb4737fe640 1 -- 172.21.15.121:0/999112370 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fb484044020 con 0x7fb48c0a7080 2024-09-17T18:35:46.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.105+0000 7fb48affd640 1 --2- 172.21.15.121:0/999112370 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb48c1ce3e0 0x7fb48c13de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:46.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.105+0000 7fb48affd640 10 monclient: get_auth_request con 0x7fb48c1ce3e0 auth_method 0 2024-09-17T18:35:46.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.105+0000 7fb48affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:46.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.105+0000 7fb48affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:46.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.105+0000 7fb48affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:46.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.105+0000 7fb48affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:46.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.105+0000 7fb48affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:46.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.105+0000 7fb48affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:46.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.105+0000 7fb4737fe640 1 --2- 172.21.15.121:0/999112370 >> v2:172.21.15.173:6812/2423786037 conn(0x7fb478040150 0x7fb478042610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:46.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.105+0000 7fb48affd640 10 monclient(hunting): handle_auth_done global_id 64544 payload 824 2024-09-17T18:35:46.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.105+0000 7fb48affd640 1 -- 172.21.15.121:0/999112370 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb48c0a5f70 con 0x7fb48c1ce3e0 2024-09-17T18:35:46.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.105+0000 7fb48a7fc640 1 --2- 172.21.15.121:0/999112370 >> v2:172.21.15.173:6812/2423786037 conn(0x7fb478040150 0x7fb478042610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:46.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.105+0000 7fb48a7fc640 10 monclient: get_auth_request con 0x7fb478040150 auth_method 0 2024-09-17T18:35:46.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.107+0000 7fb4737fe640 10 client.64541.objecter ms_handle_connect 0x7fb48c1ce3e0 2024-09-17T18:35:46.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.107+0000 7fb48affd640 1 -- 172.21.15.121:0/999112370 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb48c1ce3e0 msgr2=0x7fb48c13de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:46.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.107+0000 7fb4737fe640 10 client.64541.objecter resend_mon_ops 2024-09-17T18:35:46.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.107+0000 7fb48affd640 1 --2- 172.21.15.121:0/999112370 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb48c1ce3e0 0x7fb48c13de60 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb47c003850 tx=0x7fb47c00b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:46.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.107+0000 7fb4737fe640 1 -- 172.21.15.121:0/999112370 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb47c016070 con 0x7fb48c1ce3e0 2024-09-17T18:35:46.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.107+0000 7fb4737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:46.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.107+0000 7fb4737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:35:46.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.107+0000 7fb4737fe640 10 monclient: dump: 2024-09-17T18:35:46.123 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:46.123 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:46.123 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:46.123 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:46.124 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:46.124 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:46.124 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:46.124 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:46.124 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:46.124 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:46.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.108+0000 7fb4737fe640 1 -- 172.21.15.121:0/999112370 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb47c005270 con 0x7fb48c1ce3e0 2024-09-17T18:35:46.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.108+0000 7fb4737fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:46.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.108+0000 7fb4737fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:35:46.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.108+0000 7fb4737fe640 1 -- 172.21.15.121:0/999112370 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb48c1ce3e0 msgr2=0x7fb48c13de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:46.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.108+0000 7fb4737fe640 1 --2- 172.21.15.121:0/999112370 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb48c1ce3e0 0x7fb48c13de60 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb47c003850 tx=0x7fb47c00b810 comp rx=0 tx=0).stop 2024-09-17T18:35:46.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.108+0000 7fb4737fe640 1 -- 172.21.15.121:0/999112370 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb48c1ce3e0 msgr2=0x7fb48c13de60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:35:46.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.108+0000 7fb4737fe640 1 --2- 172.21.15.121:0/999112370 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb48c1ce3e0 0x7fb48c13de60 secure :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb47c003850 tx=0x7fb47c00b810 comp rx=0 tx=0).stop 2024-09-17T18:35:46.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.108+0000 7fb4737fe640 10 client.64541.objecter ms_handle_connect 0x7fb478040150 2024-09-17T18:35:46.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.108+0000 7fb48a7fc640 1 --2- 172.21.15.121:0/999112370 >> v2:172.21.15.173:6812/2423786037 conn(0x7fb478040150 0x7fb478042610 secure :-1 s=READY pgs=4084 cs=0 l=1 rev1=1 crypto rx=0x7fb48c13fcc0 tx=0x7fb480008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:46.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.114+0000 7fb4917a2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:35:46.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.114+0000 7fb4917a2640 1 --2- 172.21.15.121:0/999112370 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb48c13e3a0 0x7fb48c139b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:46.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.114+0000 7fb4917a2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:46.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.114+0000 7fb4917a2640 1 -- 172.21.15.121:0/999112370 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb48c13e7b0 con 0x7fb48c13e3a0 2024-09-17T18:35:46.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.114+0000 7fb48affd640 1 --2- 172.21.15.121:0/999112370 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb48c13e3a0 0x7fb48c139b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:46.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.114+0000 7fb48affd640 10 monclient: get_auth_request con 0x7fb48c13e3a0 auth_method 0 2024-09-17T18:35:46.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.114+0000 7fb48affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:46.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.114+0000 7fb48affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:46.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.114+0000 7fb48affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:46.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.114+0000 7fb48affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:46.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.114+0000 7fb48affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:46.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.114+0000 7fb48affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:46.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.114+0000 7fb48affd640 10 monclient(hunting): handle_auth_done global_id 64547 payload 824 2024-09-17T18:35:46.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.114+0000 7fb48affd640 1 -- 172.21.15.121:0/999112370 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb48c06cd40 con 0x7fb48c13e3a0 2024-09-17T18:35:46.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.116+0000 7fb48affd640 1 -- 172.21.15.121:0/999112370 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb48c13e3a0 msgr2=0x7fb48c139b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:46.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.116+0000 7fb48affd640 1 --2- 172.21.15.121:0/999112370 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb48c13e3a0 0x7fb48c139b70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb47c003850 tx=0x7fb47c0059d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:46.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.116+0000 7fb4737fe640 10 client.64541.objecter ms_handle_connect 0x7fb48c13e3a0 2024-09-17T18:35:46.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.116+0000 7fb4737fe640 10 client.64541.objecter resend_mon_ops 2024-09-17T18:35:46.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.117+0000 7fb4737fe640 1 -- 172.21.15.121:0/999112370 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb47c016040 con 0x7fb48c13e3a0 2024-09-17T18:35:46.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.117+0000 7fb4737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:46.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.117+0000 7fb4737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:35:46.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.117+0000 7fb4737fe640 10 monclient: dump: 2024-09-17T18:35:46.128 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:46.128 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:46.129 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:46.129 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:46.129 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:46.129 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:46.129 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:46.129 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:46.129 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:46.129 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:46.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.117+0000 7fb4737fe640 1 -- 172.21.15.121:0/999112370 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4709 (secure 0 0 0) 0x7fb47c018ab0 con 0x7fb48c13e3a0 2024-09-17T18:35:46.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.117+0000 7fb4737fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:35:46.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.117+0000 7fb4737fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:35:46.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.117+0000 7fb4737fe640 1 -- 172.21.15.121:0/999112370 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb48c13e3a0 msgr2=0x7fb48c139b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:46.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.117+0000 7fb4737fe640 1 --2- 172.21.15.121:0/999112370 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb48c13e3a0 0x7fb48c139b70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb47c003850 tx=0x7fb47c0059d0 comp rx=0 tx=0).stop 2024-09-17T18:35:46.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.117+0000 7fb4737fe640 1 -- 172.21.15.121:0/999112370 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb48c13e3a0 msgr2=0x7fb48c139b70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:35:46.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.117+0000 7fb4737fe640 1 --2- 172.21.15.121:0/999112370 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb48c13e3a0 0x7fb48c139b70 secure :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb47c003850 tx=0x7fb47c0059d0 comp rx=0 tx=0).stop 2024-09-17T18:35:46.130 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:35:46.130 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:35:46.130 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:35:46.130 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:35:46.130 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 62, 2024-09-17T18:35:46.130 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 524850, 2024-09-17T18:35:46.130 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:35:46.130 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:35:46.130 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:35:46.131 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:35:46.131 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 1, 2024-09-17T18:35:46.131 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:35:46.131 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:35:46.131 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:35:46.131 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:35:46.131 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:35:46.131 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:35:46.131 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:35:46.131 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:35:46.131 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:35:46.131 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:35:46.131 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:35:46.131 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:35:46.132 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:35:46.132 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:35:46.132 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:35:46.132 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:35:46.132 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:35:46.132 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:35:46.132 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:35:46.132 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:35:46.132 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:35:46.132 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:35:46.132 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:35:46.132 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:35:46.132 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:35:46.132 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:35:46.133 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:35:46.133 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:35:46.133 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:46.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:46.133 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:35:46.133 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:35:46.133 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:35:46.133 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:35:46.133 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:35:46.133 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:35:46.133 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:35:46.133 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:35:46.133 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:35:46.133 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:35:46.134 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:35:46.134 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:35:46.134 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:35:46.134 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:35:46.134 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:35:46.134 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:35:46.134 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:35:46.134 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:35:46.134 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:35:46.134 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:35:46.134 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:35:46.134 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:35:46.134 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:35:46.135 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:35:46.135 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:35:46.135 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:35:46.135 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:35:46.135 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:35:46.135 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:35:46.135 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:35:46.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:46.135 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:35:46.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:46.135 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:35:46.135 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:35:46.135 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:35:46.135 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:35:46.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:46.136 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:35:46.136 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:35:46.136 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:46.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:46.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:46.136 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:35:46.136 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:35:46.136 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:46.136 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:46.136 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:46.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:46.136 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:35:46.136 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:35:46.136 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:35:46.137 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:35:46.137 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:35:46.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:46.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:46.137 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:35:46.137 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:35:46.137 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:35:46.137 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:35:46.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:46.137 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:35:46.137 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:35:46.137 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:46.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:46.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:46.137 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:35:46.138 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:35:46.138 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:46.138 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:46.138 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:46.138 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:46.138 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:35:46.138 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:35:46.138 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:35:46.138 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:35:46.138 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:35:46.138 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:46.138 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:46.138 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:35:46.138 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:35:46.138 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:35:46.139 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:35:46.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:46.139 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:35:46.139 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:35:46.139 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:46.139 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:46.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:46.139 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:35:46.139 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:35:46.139 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:46.139 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:46.139 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:46.139 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:46.139 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:35:46.140 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:35:46.140 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:35:46.140 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:35:46.140 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:35:46.140 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:46.140 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:46.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:46.140 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:35:46.140 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:35:46.140 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:46.140 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:35:46.140 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:35:46.140 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:35:46.140 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:46.141 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:35:46.141 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:35:46.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:46.141 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:35:46.141 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:35:46.141 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 6 2024-09-17T18:35:46.141 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:46.141 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:35:46.141 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:35:46.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:46.141 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:35:46.141 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:35:46.141 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 4 2024-09-17T18:35:46.141 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:46.141 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:46.142 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:46.142 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:35:46.142 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:35:46.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.117+0000 7fb4917a2640 10 librados: watch_flush enter 2024-09-17T18:35:46.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.117+0000 7fb4917a2640 10 librados: watch_flush exit 2024-09-17T18:35:46.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.117+0000 7fb4917a2640 20 client.64541.objecter shutdown clearing up homeless session... 2024-09-17T18:35:46.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.117+0000 7fb4917a2640 10 client.64541.objecter successfully canceled tick 2024-09-17T18:35:46.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.117+0000 7fb4917a2640 1 -- 172.21.15.121:0/999112370 >> v2:172.21.15.173:6812/2423786037 conn(0x7fb478040150 msgr2=0x7fb478042610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:46.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.117+0000 7fb4917a2640 1 --2- 172.21.15.121:0/999112370 >> v2:172.21.15.173:6812/2423786037 conn(0x7fb478040150 0x7fb478042610 secure :-1 s=READY pgs=4084 cs=0 l=1 rev1=1 crypto rx=0x7fb48c13fcc0 tx=0x7fb480008040 comp rx=0 tx=0).stop 2024-09-17T18:35:46.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.117+0000 7fb4917a2640 10 monclient: shutdown 2024-09-17T18:35:46.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.117+0000 7fb4917a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:35:46.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.117+0000 7fb4917a2640 1 -- 172.21.15.121:0/999112370 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb48c0a7080 msgr2=0x7fb48c13d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:46.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.117+0000 7fb4917a2640 1 --2- 172.21.15.121:0/999112370 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb48c0a7080 0x7fb48c13d920 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb4840093e0 tx=0x7fb484004a40 comp rx=0 tx=0).stop 2024-09-17T18:35:46.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.117+0000 7fb48b7fe640 1 -- 172.21.15.121:0/999112370 reap_dead start 2024-09-17T18:35:46.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.117+0000 7fb4737fe640 1 -- 172.21.15.121:0/999112370 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(648..648 src has 247..648) ==== 5917+0+0 (unknown 0 0 0) 0x7fb484074a40 con 0x7fb48c0a7080 2024-09-17T18:35:46.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.117+0000 7fb4737fe640 10 client.64541.objecter ms_dispatch 0x7fb48c000b90 osd_map(648..648 src has 247..648) 2024-09-17T18:35:46.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.117+0000 7fb4737fe640 10 librados: disconnected, discarding osd_map(648..648 src has 247..648) 2024-09-17T18:35:46.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.117+0000 7fb4917a2640 1 -- 172.21.15.121:0/999112370 shutdown_connections 2024-09-17T18:35:46.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.117+0000 7fb4917a2640 1 --2- 172.21.15.121:0/999112370 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb48c0a7080 0x7fb48c13d920 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:46.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.117+0000 7fb4917a2640 1 -- 172.21.15.121:0/999112370 >> 172.21.15.121:0/999112370 conn(0x7fb48c0ab880 msgr2=0x7fb48c141010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:35:46.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.117+0000 7fb4917a2640 1 -- 172.21.15.121:0/999112370 shutdown_connections 2024-09-17T18:35:46.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.117+0000 7fb4917a2640 1 -- 172.21.15.121:0/999112370 wait complete. 2024-09-17T18:35:46.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.117+0000 7fb4917a2640 1 librados: shutdown 2024-09-17T18:35:46.145 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T18:35:46.145 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-17T18:35:46.146 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T18:35:46.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.240+0000 7f513000f640 10 monclient: get_monmap_and_config 2024-09-17T18:35:46.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.240+0000 7f513000f640 10 monclient: build_initial_monmap 2024-09-17T18:35:46.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.240+0000 7f513000f640 10 monclient: monmap: 2024-09-17T18:35:46.244 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:35:46.244 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:46.244 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:35:46.242309+0000 2024-09-17T18:35:46.244 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:35:46.242309+0000 2024-09-17T18:35:46.244 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:35:46.244 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:46.244 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:35:46.244 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:35:46.244 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:35:46.244 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:35:46.245 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:35:46.245 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:46.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.241+0000 7f513000f640 1 Processor -- start 2024-09-17T18:35:46.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.241+0000 7f513000f640 1 -- start start 2024-09-17T18:35:46.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.241+0000 7f513000f640 10 monclient: init 2024-09-17T18:35:46.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.241+0000 7f513000f640 10 monclient: _reopen_session rank -1 2024-09-17T18:35:46.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.241+0000 7f513000f640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-17T18:35:46.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.241+0000 7f513000f640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f512809da70 0x7f51281361c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:46.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.241+0000 7f513000f640 10 monclient(hunting): picked mon.noname-a con 0x7f512809da70 addr v2:172.21.15.121:3300/0 2024-09-17T18:35:46.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.241+0000 7f513000f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5128136700 addr v1:172.21.15.173:6789/0 2024-09-17T18:35:46.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.242+0000 7f513000f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5128136b30 addr v1:172.21.15.121:6789/0 2024-09-17T18:35:46.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.242+0000 7f513000f640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5128136fc0 con 0x7f5128136b30 2024-09-17T18:35:46.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.242+0000 7f513000f640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51280a2d40 con 0x7f5128136700 2024-09-17T18:35:46.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.242+0000 7f513000f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:46.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.242+0000 7f513000f640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f51280a0e20 con 0x7f512809da70 2024-09-17T18:35:46.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.242+0000 7f513000f640 10 monclient(hunting): _renew_subs 2024-09-17T18:35:46.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.242+0000 7f513000f640 10 monclient(hunting): authenticate will time out at 1749.959047s 2024-09-17T18:35:46.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.242+0000 7f512ddab640 1 -- >> v2:172.21.15.121:3300/0 conn(0x7f512809da70 msgr2=0x7f51281361c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:35:46.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.242+0000 7f512e5ac640 1 -- >> v1:172.21.15.121:6789/0 conn(0x7f5128136b30 legacy=0x7f51281333d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.121:6789/0 2024-09-17T18:35:46.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.242+0000 7f512ddab640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f512809da70 0x7f51281361c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:35:46.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.242+0000 7f512d5aa640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f5128136700 0x7f5128132cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.121:56690/0 (socket says 172.21.15.121:56690) 2024-09-17T18:35:46.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.242+0000 7f512d5aa640 1 -- 172.21.15.121:0/2806545976 learned_addr learned my addr 172.21.15.121:0/2806545976 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:35:46.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.244+0000 7f512d5aa640 1 -- 172.21.15.121:0/2806545976 >> v1:172.21.15.173:6789/0 conn(0x7f5128136700 legacy=0x7f5128132cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:46.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.246+0000 7f512cda9640 1 -- 172.21.15.121:0/2806545976 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2097884343 0 0) 0x7f51280a2d40 con 0x7f5128136700 2024-09-17T18:35:46.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.246+0000 7f512cda9640 10 monclient(hunting): _negotiate 2024-09-17T18:35:46.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.246+0000 7f512cda9640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:46.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.246+0000 7f512cda9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:46.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.246+0000 7f512cda9640 10 monclient(hunting): my global_id is 84295 2024-09-17T18:35:46.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.246+0000 7f512cda9640 1 -- 172.21.15.121:0/2806545976 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5114003520 con 0x7f5128136700 2024-09-17T18:35:46.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512cda9640 1 -- 172.21.15.121:0/2806545976 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 50703368 0 0) 0x7f5114003520 con 0x7f5128136700 2024-09-17T18:35:46.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512cda9640 1 -- 172.21.15.121:0/2806545976 >> v2:172.21.15.121:3300/0 conn(0x7f512809da70 msgr2=0x7f51281361c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:35:46.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512cda9640 1 --2- 172.21.15.121:0/2806545976 >> v2:172.21.15.121:3300/0 conn(0x7f512809da70 0x7f51281361c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:46.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512cda9640 1 -- 172.21.15.121:0/2806545976 >> v1:172.21.15.121:6789/0 conn(0x7f5128136b30 legacy=0x7f51281333d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:35:46.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512cda9640 10 monclient: _finish_hunting 0 2024-09-17T18:35:46.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512cda9640 1 monclient: found mon.noname-c-legacy 2024-09-17T18:35:46.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512cda9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:35:46.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512cda9640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.173:6789/0 2024-09-17T18:35:46.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512cda9640 1 -- 172.21.15.121:0/2806545976 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51280a3020 con 0x7f5128136700 2024-09-17T18:35:46.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512cda9640 10 monclient: _finish_auth 0 2024-09-17T18:35:46.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512cda9640 10 monclient: _check_auth_tickets 2024-09-17T18:35:46.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512cda9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:35:46.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512cda9640 1 -- 172.21.15.121:0/2806545976 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4002019445 0 0) 0x7f51180019a0 con 0x7f5128136700 2024-09-17T18:35:46.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f513000f640 5 monclient: authenticate success, global_id 84295 2024-09-17T18:35:46.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f513000f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:35:46.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512cda9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:46.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512cda9640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-17T18:35:46.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512cda9640 10 monclient: dump: 2024-09-17T18:35:46.253 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:46.253 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:46.253 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:46.253 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:46.254 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:46.254 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:46.254 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:46.254 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:46.254 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:46.254 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:46.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512cda9640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] but i'm connected to v1:172.21.15.173:6789/0, reconnecting 2024-09-17T18:35:46.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512cda9640 10 monclient: _reopen_session rank -1 2024-09-17T18:35:46.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512cda9640 1 -- 172.21.15.121:0/2806545976 >> v1:172.21.15.173:6789/0 conn(0x7f5128136700 legacy=0x7f5128132cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:46.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512cda9640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:35:46.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512cda9640 1 --2- 172.21.15.121:0/2806545976 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5114003a70 0x7f5114003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:46.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512ddab640 1 -- 172.21.15.121:0/2806545976 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5114003a70 msgr2=0x7f5114003e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:35:46.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512cda9640 10 monclient(hunting): picked mon.a con 0x7f5114003a70 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:35:46.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512ddab640 1 --2- 172.21.15.121:0/2806545976 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5114003a70 0x7f5114003e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:35:46.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512cda9640 1 --2- 172.21.15.121:0/2806545976 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5114004800 0x7f5114004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:46.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512cda9640 10 monclient(hunting): picked mon.b con 0x7f5114004800 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:35:46.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512cda9640 1 --2- 172.21.15.121:0/2806545976 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f51140099b0 0x7f511400bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:46.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512cda9640 10 monclient(hunting): picked mon.c con 0x7f51140099b0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:35:46.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512cda9640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:46.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512cda9640 1 -- 172.21.15.121:0/2806545976 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f51280a0e20 con 0x7f5114003a70 2024-09-17T18:35:46.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512cda9640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:46.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512cda9640 1 -- 172.21.15.121:0/2806545976 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f511400c3c0 con 0x7f5114004800 2024-09-17T18:35:46.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512cda9640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:46.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512cda9640 1 -- 172.21.15.121:0/2806545976 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f511400c5c0 con 0x7f51140099b0 2024-09-17T18:35:46.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512cda9640 10 monclient(hunting): _renew_subs 2024-09-17T18:35:46.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512cda9640 10 monclient(hunting): _finish_auth 0 2024-09-17T18:35:46.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f513000f640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-17T18:35:46.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512e5ac640 1 --2- 172.21.15.121:0/2806545976 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f51140099b0 0x7f511400bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:46.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512e5ac640 10 monclient(hunting): get_auth_request con 0x7f51140099b0 auth_method 0 2024-09-17T18:35:46.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512e5ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:46.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512e5ac640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:46.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512e5ac640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:35:46.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512e5ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:46.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512e5ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:46.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512e5ac640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:35:46.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512e5ac640 10 monclient(hunting): handle_auth_done global_id 84295 payload 293 2024-09-17T18:35:46.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512e5ac640 1 -- 172.21.15.121:0/2806545976 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5114004800 msgr2=0x7f5114004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:35:46.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512e5ac640 1 --2- 172.21.15.121:0/2806545976 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5114004800 0x7f5114004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:46.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512e5ac640 1 -- 172.21.15.121:0/2806545976 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5114003a70 msgr2=0x7f5114003e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T18:35:46.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512e5ac640 1 --2- 172.21.15.121:0/2806545976 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5114003a70 0x7f5114003e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:46.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512e5ac640 10 monclient: _finish_hunting 0 2024-09-17T18:35:46.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512e5ac640 1 monclient: found mon.c 2024-09-17T18:35:46.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512e5ac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:35:46.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512e5ac640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:46.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512e5ac640 1 -- 172.21.15.121:0/2806545976 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5114003520 con 0x7f51140099b0 2024-09-17T18:35:46.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512e5ac640 10 monclient: _finish_auth 0 2024-09-17T18:35:46.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512e5ac640 10 monclient: _check_auth_tickets 2024-09-17T18:35:46.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.247+0000 7f512e5ac640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:35:46.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.248+0000 7f512e5ac640 1 -- 172.21.15.121:0/2806545976 reap_dead start 2024-09-17T18:35:46.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.250+0000 7f512e5ac640 1 -- 172.21.15.121:0/2806545976 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f51140099b0 msgr2=0x7f511400bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:46.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.250+0000 7f512e5ac640 1 --2- 172.21.15.121:0/2806545976 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f51140099b0 0x7f511400bda0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5120009e30 tx=0x7f5114005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70f44915a804dcd2 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:46.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.250+0000 7f512cda9640 1 -- 172.21.15.121:0/2806545976 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5120035070 con 0x7f51140099b0 2024-09-17T18:35:46.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.250+0000 7f512cda9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:46.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.250+0000 7f512cda9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:35:46.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.250+0000 7f512cda9640 10 monclient: dump: 2024-09-17T18:35:46.265 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:46.265 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:46.265 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:46.265 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:46.265 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:46.265 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:46.265 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:46.265 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:46.265 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:46.266 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:46.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.250+0000 7f512cda9640 1 -- 172.21.15.121:0/2806545976 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f512002c040 con 0x7f51140099b0 2024-09-17T18:35:46.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.250+0000 7f513000f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:35:46.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.250+0000 7f512cda9640 10 monclient: handle_config config(0 keys) 2024-09-17T18:35:46.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.250+0000 7f513000f640 10 monclient: get_monmap_and_config success 2024-09-17T18:35:46.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.250+0000 7f513000f640 4 set_mon_vals no callback set 2024-09-17T18:35:46.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.250+0000 7f512cda9640 1 -- 172.21.15.121:0/2806545976 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5120009a30 con 0x7f51140099b0 2024-09-17T18:35:46.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.251+0000 7f512cda9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:46.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.251+0000 7f512cda9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:35:46.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.251+0000 7f512cda9640 10 monclient: dump: 2024-09-17T18:35:46.266 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:46.266 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:46.266 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:46.266 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:46.266 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:46.267 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:46.267 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:46.267 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:46.267 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:46.267 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:46.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.251+0000 7f513000f640 10 monclient: shutdown 2024-09-17T18:35:46.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.251+0000 7f513000f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:35:46.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.251+0000 7f513000f640 1 -- 172.21.15.121:0/2806545976 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f51140099b0 msgr2=0x7f511400bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:46.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.251+0000 7f513000f640 1 --2- 172.21.15.121:0/2806545976 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f51140099b0 0x7f511400bda0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5120009e30 tx=0x7f5114005830 comp rx=0 tx=0).stop 2024-09-17T18:35:46.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.251+0000 7f513000f640 1 -- 172.21.15.121:0/2806545976 shutdown_connections 2024-09-17T18:35:46.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.251+0000 7f513000f640 1 --2- 172.21.15.121:0/2806545976 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f51140099b0 0x7f511400bda0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:46.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.251+0000 7f513000f640 1 -- 172.21.15.121:0/2806545976 >> 172.21.15.121:0/2806545976 conn(0x7f51280a7830 msgr2=0x7f51280a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:35:46.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.251+0000 7f513000f640 1 -- 172.21.15.121:0/2806545976 shutdown_connections 2024-09-17T18:35:46.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.251+0000 7f513000f640 1 -- 172.21.15.121:0/2806545976 wait complete. 2024-09-17T18:35:46.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.252+0000 7f513000f640 10 monclient: build_initial_monmap 2024-09-17T18:35:46.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.252+0000 7f513000f640 10 monclient: monmap: 2024-09-17T18:35:46.268 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:35:46.268 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:35:46.268 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:35:46.268 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:35:46.268 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:35:46.268 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:46.268 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:35:46.268 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:35:46.268 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:35:46.268 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:46.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.252+0000 7f513000f640 1 librados: starting msgr at 2024-09-17T18:35:46.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.252+0000 7f513000f640 1 librados: starting objecter 2024-09-17T18:35:46.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.252+0000 7f513000f640 1 Processor -- start 2024-09-17T18:35:46.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.252+0000 7f513000f640 1 -- start start 2024-09-17T18:35:46.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.252+0000 7f513000f640 1 librados: setting wanted keys 2024-09-17T18:35:46.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.252+0000 7f513000f640 1 librados: calling monclient init 2024-09-17T18:35:46.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.252+0000 7f513000f640 10 monclient: init 2024-09-17T18:35:46.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f513000f640 10 monclient: _reopen_session rank -1 2024-09-17T18:35:46.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f513000f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:35:46.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f513000f640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f51140099b0 0x7f51281cb420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:46.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f513000f640 10 monclient(hunting): picked mon.noname-a con 0x7f51140099b0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:35:46.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f513000f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f512812f9a0 0x7f512812fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:46.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f513000f640 10 monclient(hunting): picked mon.noname-b con 0x7f512812f9a0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:35:46.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f513000f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f512811d6a0 0x7f512811db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:46.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f513000f640 10 monclient(hunting): picked mon.noname-c con 0x7f512811d6a0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:35:46.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f513000f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:46.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f513000f640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f51280a3360 con 0x7f51140099b0 2024-09-17T18:35:46.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f513000f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:46.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f513000f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f51280a3930 con 0x7f512812f9a0 2024-09-17T18:35:46.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f513000f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:46.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f513000f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f51281361c0 con 0x7f512811d6a0 2024-09-17T18:35:46.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f513000f640 10 monclient(hunting): _renew_subs 2024-09-17T18:35:46.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f513000f640 10 monclient(hunting): authenticate will time out at 1749.970434s 2024-09-17T18:35:46.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f512ddab640 1 -- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f51140099b0 msgr2=0x7f51281cb420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:3300/0 2024-09-17T18:35:46.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f512ddab640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f51140099b0 0x7f51281cb420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:35:46.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f512e5ac640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f512811d6a0 0x7f512811db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:46.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f512e5ac640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f512811d6a0 0x7f512811db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:37132/0 (socket says 172.21.15.121:37132) 2024-09-17T18:35:46.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f512e5ac640 1 -- 172.21.15.121:0/2494123449 learned_addr learned my addr 172.21.15.121:0/2494123449 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:35:46.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f512e5ac640 10 monclient(hunting): get_auth_request con 0x7f512811d6a0 auth_method 0 2024-09-17T18:35:46.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f512e5ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:46.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f512e5ac640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:46.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f512e5ac640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:46.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f512e5ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:46.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f512e5ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:46.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f512d5aa640 1 --2- 172.21.15.121:0/2494123449 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f512812f9a0 0x7f512812fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:46.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f512e5ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:46.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f512d5aa640 10 monclient(hunting): get_auth_request con 0x7f512812f9a0 auth_method 0 2024-09-17T18:35:46.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f512d5aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:46.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f512d5aa640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:46.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f512d5aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:46.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f512d5aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:46.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f512d5aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:46.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f512d5aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:46.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f512e5ac640 10 monclient(hunting): handle_auth_done global_id 64550 payload 824 2024-09-17T18:35:46.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f512e5ac640 1 -- 172.21.15.121:0/2494123449 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f512812f9a0 msgr2=0x7f512812fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:46.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f512e5ac640 1 --2- 172.21.15.121:0/2494123449 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f512812f9a0 0x7f512812fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:46.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f512e5ac640 1 -- 172.21.15.121:0/2494123449 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f51140099b0 msgr2=0x7f51281cb420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:35:46.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f512e5ac640 1 --2- 172.21.15.121:0/2494123449 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f51140099b0 0x7f51281cb420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:46.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f512e5ac640 10 monclient: _finish_hunting 0 2024-09-17T18:35:46.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f512e5ac640 1 monclient: found mon.noname-c 2024-09-17T18:35:46.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f512e5ac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:35:46.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f512e5ac640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:35:46.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.253+0000 7f512e5ac640 1 -- 172.21.15.121:0/2494123449 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f512811f040 con 0x7f512811d6a0 2024-09-17T18:35:46.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.255+0000 7f512e5ac640 1 -- 172.21.15.121:0/2494123449 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f512811d6a0 msgr2=0x7f512811db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:46.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.256+0000 7f512e5ac640 1 --2- 172.21.15.121:0/2494123449 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f512811d6a0 0x7f512811db10 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5120009f50 tx=0x7f5120005cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:46.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.255+0000 7f511e7fc640 10 client.?.objecter ms_handle_connect 0x7f512811d6a0 2024-09-17T18:35:46.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.256+0000 7f511e7fc640 10 client.?.objecter resend_mon_ops 2024-09-17T18:35:46.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.256+0000 7f511e7fc640 1 -- 172.21.15.121:0/2494123449 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5120035040 con 0x7f512811d6a0 2024-09-17T18:35:46.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.256+0000 7f511e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:46.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.256+0000 7f511e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:35:46.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.256+0000 7f511e7fc640 10 monclient: dump: 2024-09-17T18:35:46.273 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:46.273 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:46.273 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:46.273 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:46.273 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:46.273 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:46.273 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:46.273 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:46.273 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:46.274 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:46.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.256+0000 7f511e7fc640 10 monclient: _finish_auth 0 2024-09-17T18:35:46.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.256+0000 7f511e7fc640 10 monclient: _check_auth_tickets 2024-09-17T18:35:46.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.256+0000 7f511e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:35:46.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.256+0000 7f513000f640 5 monclient: authenticate success, global_id 64550 2024-09-17T18:35:46.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.256+0000 7f511e7fc640 1 -- 172.21.15.121:0/2494123449 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5120041070 con 0x7f512811d6a0 2024-09-17T18:35:46.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.256+0000 7f513000f640 10 monclient: _renew_subs 2024-09-17T18:35:46.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.256+0000 7f513000f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:46.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.256+0000 7f513000f640 1 -- 172.21.15.121:0/2494123449 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f512809e240 con 0x7f512811d6a0 2024-09-17T18:35:46.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.256+0000 7f511e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:35:46.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.256+0000 7f511e7fc640 1 -- 172.21.15.121:0/2494123449 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5120030a80 con 0x7f512811d6a0 2024-09-17T18:35:46.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.256+0000 7f511e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:46.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.256+0000 7f511e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:35:46.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.256+0000 7f511e7fc640 10 monclient: dump: 2024-09-17T18:35:46.274 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:46.275 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:46.275 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:46.275 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:46.275 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:46.275 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:46.275 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:46.275 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:46.275 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:46.275 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:46.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.256+0000 7f513000f640 10 client.64550.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:35:46.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.256+0000 7f511ffff640 4 set_mon_vals no callback set 2024-09-17T18:35:46.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.256+0000 7f513000f640 10 monclient: _renew_subs 2024-09-17T18:35:46.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.256+0000 7f513000f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:35:46.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.256+0000 7f513000f640 1 -- 172.21.15.121:0/2494123449 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5128120b00 con 0x7f512811d6a0 2024-09-17T18:35:46.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.256+0000 7f513000f640 1 librados: init done 2024-09-17T18:35:46.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.256+0000 7f511e7fc640 1 -- 172.21.15.121:0/2494123449 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f5120030c20 con 0x7f512811d6a0 2024-09-17T18:35:46.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.256+0000 7f513000f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:46.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.256+0000 7f513000f640 1 --2- 172.21.15.121:0/2494123449 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f512811e050 0x7f512806f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:46.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.257+0000 7f513000f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:46.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.257+0000 7f513000f640 1 -- 172.21.15.121:0/2494123449 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f51280a6be0 con 0x7f512811e050 2024-09-17T18:35:46.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.257+0000 7f511e7fc640 1 --2- 172.21.15.121:0/2494123449 >> v2:172.21.15.173:6812/2423786037 conn(0x7f510c040120 0x7f510c0425e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:46.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.257+0000 7f511e7fc640 1 -- 172.21.15.121:0/2494123449 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(649..649 src has 247..649) ==== 5544+0+0 (secure 0 0 0) 0x7f5120074700 con 0x7f512811d6a0 2024-09-17T18:35:46.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.257+0000 7f511e7fc640 10 client.64550.objecter ms_dispatch 0x7f5128000b90 osd_map(649..649 src has 247..649) 2024-09-17T18:35:46.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.257+0000 7f511e7fc640 3 client.64550.objecter handle_osd_map got epochs [649,649] > 0 2024-09-17T18:35:46.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.257+0000 7f511e7fc640 3 client.64550.objecter handle_osd_map decoding full epoch 649 2024-09-17T18:35:46.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.257+0000 7f512ddab640 1 --2- 172.21.15.121:0/2494123449 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f512811e050 0x7f512806f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:46.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.257+0000 7f511e7fc640 20 client.64550.objecter dump_active .. 0 homeless 2024-09-17T18:35:46.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.257+0000 7f512ddab640 10 monclient: get_auth_request con 0x7f512811e050 auth_method 0 2024-09-17T18:35:46.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.257+0000 7f512ddab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:46.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.257+0000 7f512ddab640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:46.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.257+0000 7f512ddab640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:46.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.257+0000 7f512d5aa640 1 --2- 172.21.15.121:0/2494123449 >> v2:172.21.15.173:6812/2423786037 conn(0x7f510c040120 0x7f510c0425e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:46.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.257+0000 7f512d5aa640 10 monclient: get_auth_request con 0x7f510c040120 auth_method 0 2024-09-17T18:35:46.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.257+0000 7f512ddab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:46.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.257+0000 7f512ddab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:46.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.257+0000 7f512ddab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:46.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.257+0000 7f512ddab640 10 monclient(hunting): handle_auth_done global_id 84301 payload 824 2024-09-17T18:35:46.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.257+0000 7f512ddab640 1 -- 172.21.15.121:0/2494123449 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f512811fba0 con 0x7f512811e050 2024-09-17T18:35:46.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.260+0000 7f511e7fc640 10 client.64550.objecter ms_handle_connect 0x7f510c040120 2024-09-17T18:35:46.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.260+0000 7f512d5aa640 1 --2- 172.21.15.121:0/2494123449 >> v2:172.21.15.173:6812/2423786037 conn(0x7f510c040120 0x7f510c0425e0 secure :-1 s=READY pgs=4085 cs=0 l=1 rev1=1 crypto rx=0x7f51180010e0 tx=0x7f5118001890 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:46.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.260+0000 7f511e7fc640 10 client.64550.objecter ms_handle_connect 0x7f512811e050 2024-09-17T18:35:46.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.260+0000 7f511e7fc640 10 client.64550.objecter resend_mon_ops 2024-09-17T18:35:46.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.260+0000 7f512ddab640 1 -- 172.21.15.121:0/2494123449 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f512811e050 msgr2=0x7f512806f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:46.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.260+0000 7f512ddab640 1 --2- 172.21.15.121:0/2494123449 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f512811e050 0x7f512806f2c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5124003980 tx=0x7f512400b8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:46.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.260+0000 7f511e7fc640 1 -- 172.21.15.121:0/2494123449 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5124005270 con 0x7f512811e050 2024-09-17T18:35:46.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.260+0000 7f511e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:46.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.260+0000 7f511e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:35:46.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.260+0000 7f511e7fc640 10 monclient: dump: 2024-09-17T18:35:46.278 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:46.278 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:46.278 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:46.278 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:46.278 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:46.278 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:46.278 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:46.278 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:46.278 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:46.279 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:46.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.260+0000 7f511e7fc640 1 -- 172.21.15.121:0/2494123449 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5124015070 con 0x7f512811e050 2024-09-17T18:35:46.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.260+0000 7f511e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:35:46.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.260+0000 7f511e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:35:46.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.260+0000 7f511e7fc640 1 -- 172.21.15.121:0/2494123449 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f512811e050 msgr2=0x7f512806f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:46.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.260+0000 7f511e7fc640 1 --2- 172.21.15.121:0/2494123449 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f512811e050 0x7f512806f2c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5124003980 tx=0x7f512400b8e0 comp rx=0 tx=0).stop 2024-09-17T18:35:46.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.260+0000 7f511e7fc640 1 -- 172.21.15.121:0/2494123449 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f512811e050 msgr2=0x7f512806f2c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:35:46.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.260+0000 7f511e7fc640 1 --2- 172.21.15.121:0/2494123449 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f512811e050 0x7f512806f2c0 secure :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5124003980 tx=0x7f512400b8e0 comp rx=0 tx=0).stop 2024-09-17T18:35:46.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.266+0000 7f513000f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:35:46.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.266+0000 7f513000f640 1 --2- 172.21.15.121:0/2494123449 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f512806f800 0x7f512806fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:35:46.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.266+0000 7f513000f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:35:46.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.266+0000 7f513000f640 1 -- 172.21.15.121:0/2494123449 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f512811e430 con 0x7f512806f800 2024-09-17T18:35:46.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.267+0000 7f512ddab640 1 --2- 172.21.15.121:0/2494123449 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f512806f800 0x7f512806fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:35:46.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.267+0000 7f512ddab640 10 monclient: get_auth_request con 0x7f512806f800 auth_method 0 2024-09-17T18:35:46.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.267+0000 7f512ddab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:35:46.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.267+0000 7f512ddab640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:35:46.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.267+0000 7f512ddab640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:35:46.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.267+0000 7f512ddab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:35:46.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.267+0000 7f512ddab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:35:46.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.267+0000 7f512ddab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:35:46.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.267+0000 7f512ddab640 10 monclient(hunting): handle_auth_done global_id 84304 payload 824 2024-09-17T18:35:46.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.267+0000 7f512ddab640 1 -- 172.21.15.121:0/2494123449 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f512806cd40 con 0x7f512806f800 2024-09-17T18:35:46.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.270+0000 7f512ddab640 1 -- 172.21.15.121:0/2494123449 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f512806f800 msgr2=0x7f512806fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:35:46.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.270+0000 7f511e7fc640 10 client.64550.objecter ms_handle_connect 0x7f512806f800 2024-09-17T18:35:46.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.270+0000 7f511e7fc640 10 client.64550.objecter resend_mon_ops 2024-09-17T18:35:46.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.270+0000 7f512ddab640 1 --2- 172.21.15.121:0/2494123449 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f512806f800 0x7f512806fc10 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5124003980 tx=0x7f5124005a60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:35:46.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.270+0000 7f511e7fc640 1 -- 172.21.15.121:0/2494123449 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5124024410 con 0x7f512806f800 2024-09-17T18:35:46.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.270+0000 7f511e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:35:46.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.270+0000 7f511e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:35:46.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.270+0000 7f511e7fc640 10 monclient: dump: 2024-09-17T18:35:46.282 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:35:46.282 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:35:46.282 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:46.282 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:35:46.283 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:35:46.283 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:35:46.283 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:35:46.283 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:35:46.283 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:35:46.283 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:35:46.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.270+0000 7f511e7fc640 1 -- 172.21.15.121:0/2494123449 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7f5124010400 con 0x7f512806f800 2024-09-17T18:35:46.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.270+0000 7f511e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:35:46.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.270+0000 7f511e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:35:46.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.270+0000 7f511e7fc640 1 -- 172.21.15.121:0/2494123449 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f512806f800 msgr2=0x7f512806fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:46.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.270+0000 7f511e7fc640 1 --2- 172.21.15.121:0/2494123449 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f512806f800 0x7f512806fc10 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5124003980 tx=0x7f5124005a60 comp rx=0 tx=0).stop 2024-09-17T18:35:46.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.270+0000 7f511e7fc640 1 -- 172.21.15.121:0/2494123449 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f512806f800 msgr2=0x7f512806fc10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:35:46.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.270+0000 7f511e7fc640 1 --2- 172.21.15.121:0/2494123449 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f512806f800 0x7f512806fc10 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:35:46.284 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:35:46.284 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:35:46.284 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:35:46.284 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "leader", 2024-09-17T18:35:46.284 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 62, 2024-09-17T18:35:46.284 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 271368, 2024-09-17T18:35:46.284 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:35:46.284 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:35:46.284 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:35:46.284 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:35:46.284 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 2, 2024-09-17T18:35:46.285 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:35:46.285 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:35:46.285 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:35:46.285 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:35:46.285 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:35:46.285 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:35:46.285 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:35:46.285 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:35:46.285 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:35:46.285 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:35:46.285 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:35:46.285 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:35:46.285 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:35:46.286 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:35:46.286 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:35:46.286 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:35:46.286 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:35:46.286 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:35:46.286 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:35:46.286 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:35:46.286 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:35:46.286 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:35:46.286 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:35:46.286 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:35:46.286 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:35:46.286 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:35:46.286 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:35:46.286 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:35:46.287 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:46.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:46.287 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:35:46.287 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:35:46.287 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:35:46.287 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:35:46.287 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:35:46.287 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:35:46.287 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:35:46.287 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:35:46.287 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:35:46.287 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:35:46.287 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:35:46.287 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:35:46.288 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:35:46.288 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:35:46.288 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:35:46.288 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:35:46.288 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:35:46.288 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:35:46.288 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:35:46.288 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:35:46.288 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:35:46.288 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:35:46.288 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:35:46.288 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:35:46.288 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:35:46.288 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:35:46.289 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:35:46.289 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:35:46.289 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:35:46.289 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:35:46.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:46.289 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:35:46.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:46.289 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:35:46.289 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:35:46.289 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:35:46.289 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:35:46.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:46.289 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:35:46.289 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:35:46.289 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:46.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:46.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:46.290 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:35:46.290 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:35:46.290 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:46.290 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:46.290 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:46.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:46.290 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:35:46.290 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:35:46.290 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:35:46.290 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:35:46.290 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:35:46.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:46.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:46.290 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:35:46.291 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:35:46.291 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:35:46.291 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:35:46.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:46.291 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:35:46.291 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:35:46.291 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:46.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:46.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:46.291 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:35:46.291 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:35:46.291 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:46.291 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:46.291 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:46.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:46.291 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:35:46.292 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:35:46.292 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:35:46.292 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:35:46.292 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:35:46.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:46.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:46.292 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:35:46.292 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:35:46.292 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:35:46.292 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:35:46.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:46.292 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:35:46.292 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:35:46.292 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:46.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:46.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:46.293 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:35:46.293 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:35:46.293 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:35:46.293 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:46.293 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:46.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:46.293 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:35:46.293 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:35:46.293 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:35:46.293 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:35:46.293 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:35:46.293 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:46.293 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:46.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:46.294 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:35:46.294 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:35:46.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:46.294 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:35:46.294 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:35:46.294 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:35:46.294 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:46.294 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:35:46.294 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:35:46.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:46.294 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:35:46.294 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:35:46.294 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 4 2024-09-17T18:35:46.295 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:46.295 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:35:46.295 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": [ 2024-09-17T18:35:46.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:35:46.295 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:35:46.295 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:35:46.295 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:35:46.295 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:35:46.295 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:35:46.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:35:46.295 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:35:46.295 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:35:46.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.270+0000 7f513000f640 10 librados: watch_flush enter 2024-09-17T18:35:46.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.270+0000 7f513000f640 10 librados: watch_flush exit 2024-09-17T18:35:46.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.270+0000 7f513000f640 20 client.64550.objecter shutdown clearing up homeless session... 2024-09-17T18:35:46.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.270+0000 7f513000f640 10 client.64550.objecter successfully canceled tick 2024-09-17T18:35:46.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.270+0000 7f513000f640 1 -- 172.21.15.121:0/2494123449 >> v2:172.21.15.173:6812/2423786037 conn(0x7f510c040120 msgr2=0x7f510c0425e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:46.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.270+0000 7f513000f640 1 --2- 172.21.15.121:0/2494123449 >> v2:172.21.15.173:6812/2423786037 conn(0x7f510c040120 0x7f510c0425e0 secure :-1 s=READY pgs=4085 cs=0 l=1 rev1=1 crypto rx=0x7f51180010e0 tx=0x7f5118001890 comp rx=0 tx=0).stop 2024-09-17T18:35:46.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.270+0000 7f513000f640 10 monclient: shutdown 2024-09-17T18:35:46.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.270+0000 7f513000f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:35:46.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.270+0000 7f513000f640 1 -- 172.21.15.121:0/2494123449 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f512811d6a0 msgr2=0x7f512811db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:35:46.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.270+0000 7f513000f640 1 --2- 172.21.15.121:0/2494123449 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f512811d6a0 0x7f512811db10 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5120009f50 tx=0x7f5120005cc0 comp rx=0 tx=0).stop 2024-09-17T18:35:46.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.270+0000 7f512e5ac640 1 -- 172.21.15.121:0/2494123449 reap_dead start 2024-09-17T18:35:46.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.270+0000 7f512e5ac640 1 -- 172.21.15.121:0/2494123449 reap_dead start 2024-09-17T18:35:46.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.270+0000 7f513000f640 1 -- 172.21.15.121:0/2494123449 shutdown_connections 2024-09-17T18:35:46.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.270+0000 7f513000f640 1 -- 172.21.15.121:0/2494123449 >> 172.21.15.121:0/2494123449 conn(0x7f51280a7830 msgr2=0x7f51280a5bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:35:46.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.270+0000 7f513000f640 1 -- 172.21.15.121:0/2494123449 shutdown_connections 2024-09-17T18:35:46.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.270+0000 7f513000f640 1 -- 172.21.15.121:0/2494123449 wait complete. 2024-09-17T18:35:46.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:35:46.270+0000 7f513000f640 1 librados: shutdown 2024-09-17T18:35:46.299 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T18:35:46.299 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-17T18:36:25.445 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (24062 ms) 2024-09-17T18:36:25.446 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-09-17T18:36:25.446 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (26511 ms) 2024-09-17T18:36:25.446 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-09-17T18:36:25.446 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (8367 ms) 2024-09-17T18:36:25.446 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-09-17T18:36:25.446 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (12624 ms) 2024-09-17T18:36:25.446 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-09-17T18:36:25.446 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (8042 ms) 2024-09-17T18:36:25.446 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-09-17T18:36:25.446 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (7999 ms) 2024-09-17T18:36:25.446 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-09-17T18:36:25.447 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: hmm, no HitSet yet 2024-09-17T18:36:25.447 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: ok, hit_set contains 332:602f83fe:::foo:head 2024-09-17T18:36:25.447 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (8966 ms) 2024-09-17T18:36:25.447 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-09-17T18:36:25.447 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: got ls 1726598091,0 2024-09-17T18:36:25.447 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: first is 1726598091 2024-09-17T18:36:25.447 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: got ls 1726598091,0 2024-09-17T18:36:25.447 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: got ls 1726598091,0 2024-09-17T18:36:25.447 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: got ls 1726598091,1726598093,1726598094,0 2024-09-17T18:36:25.447 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: got ls 1726598091,1726598093,1726598094,0 2024-09-17T18:36:25.447 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: got ls 1726598091,1726598093,1726598094,0 2024-09-17T18:36:25.447 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: got ls 1726598091,1726598093,1726598094,1726598096,1726598097,0 2024-09-17T18:36:25.447 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: got ls 1726598091,1726598093,1726598094,1726598096,1726598097,0 2024-09-17T18:36:25.448 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: got ls 1726598091,1726598093,1726598094,1726598096,1726598097,0 2024-09-17T18:36:25.448 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: got ls 1726598091,1726598093,1726598094,1726598096,1726598097,1726598099,1726598100,0 2024-09-17T18:36:25.448 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: got ls 1726598091,1726598093,1726598094,1726598096,1726598097,1726598099,1726598100,0 2024-09-17T18:36:25.448 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: got ls 1726598091,1726598093,1726598094,1726598096,1726598097,1726598099,1726598100,0 2024-09-17T18:36:25.448 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: got ls 1726598094,1726598096,1726598097,1726598099,1726598100,1726598102,1726598103,0 2024-09-17T18:36:25.448 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: first now 1726598094, trimmed 2024-09-17T18:36:25.448 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (21029 ms) 2024-09-17T18:36:25.448 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-09-17T18:36:25.448 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: foo0 2024-09-17T18:36:25.448 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-09-17T18:36:25.448 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (13978 ms) 2024-09-17T18:36:25.448 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-09-17T18:36:25.448 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (27832 ms) 2024-09-17T18:36:25.449 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-09-17T18:36:25.449 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (22200 ms) 2024-09-17T18:36:25.449 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-09-17T18:36:25.449 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (4980 ms) 2024-09-17T18:36:25.449 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-09-17T18:36:25.449 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (3029 ms) 2024-09-17T18:36:25.449 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-09-17T18:36:25.449 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (3462 ms) 2024-09-17T18:36:25.449 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-09-17T18:36:25.449 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (3013 ms) 2024-09-17T18:36:25.449 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (272075 ms total) 2024-09-17T18:36:25.449 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: 2024-09-17T18:36:25.449 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-09-17T18:36:25.449 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (984620 ms total) 2024-09-17T18:36:25.450 INFO:tasks.workunit.client.0.smithi173.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-09-17T18:36:25.455 INFO:tasks.workunit.client.0.smithi173.stderr:+ for t in "${!pids[@]}" 2024-09-17T18:36:25.455 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=34968 2024-09-17T18:36:25.455 INFO:tasks.workunit.client.0.smithi173.stderr:+ wait 34968 2024-09-17T18:36:25.455 INFO:tasks.workunit.client.0.smithi173.stderr:+ for t in "${!pids[@]}" 2024-09-17T18:36:25.455 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35151 2024-09-17T18:36:25.455 INFO:tasks.workunit.client.0.smithi173.stderr:+ wait 35151 2024-09-17T18:36:25.455 INFO:tasks.workunit.client.0.smithi173.stderr:+ for t in "${!pids[@]}" 2024-09-17T18:36:25.456 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35628 2024-09-17T18:36:25.456 INFO:tasks.workunit.client.0.smithi173.stderr:+ wait 35628 2024-09-17T18:36:25.456 INFO:tasks.workunit.client.0.smithi173.stderr:+ for t in "${!pids[@]}" 2024-09-17T18:36:25.456 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35340 2024-09-17T18:36:25.456 INFO:tasks.workunit.client.0.smithi173.stderr:+ wait 35340 2024-09-17T18:36:25.456 INFO:tasks.workunit.client.0.smithi173.stderr:+ for t in "${!pids[@]}" 2024-09-17T18:36:25.456 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35243 2024-09-17T18:36:25.456 INFO:tasks.workunit.client.0.smithi173.stderr:+ wait 35243 2024-09-17T18:36:25.456 INFO:tasks.workunit.client.0.smithi173.stderr:+ for t in "${!pids[@]}" 2024-09-17T18:36:25.456 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35038 2024-09-17T18:36:25.456 INFO:tasks.workunit.client.0.smithi173.stderr:+ wait 35038 2024-09-17T18:36:25.456 INFO:tasks.workunit.client.0.smithi173.stderr:+ for t in "${!pids[@]}" 2024-09-17T18:36:25.456 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35277 2024-09-17T18:36:25.456 INFO:tasks.workunit.client.0.smithi173.stderr:+ wait 35277 2024-09-17T18:36:25.456 INFO:tasks.workunit.client.0.smithi173.stderr:+ for t in "${!pids[@]}" 2024-09-17T18:36:25.457 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35800 2024-09-17T18:36:25.457 INFO:tasks.workunit.client.0.smithi173.stderr:+ wait 35800 2024-09-17T18:36:25.457 INFO:tasks.workunit.client.0.smithi173.stderr:+ for t in "${!pids[@]}" 2024-09-17T18:36:25.457 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35817 2024-09-17T18:36:25.457 INFO:tasks.workunit.client.0.smithi173.stderr:+ wait 35817 2024-09-17T18:36:25.457 INFO:tasks.workunit.client.0.smithi173.stderr:+ for t in "${!pids[@]}" 2024-09-17T18:36:25.457 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35057 2024-09-17T18:36:25.457 INFO:tasks.workunit.client.0.smithi173.stderr:+ wait 35057 2024-09-17T18:36:25.457 INFO:tasks.workunit.client.0.smithi173.stderr:+ for t in "${!pids[@]}" 2024-09-17T18:36:25.457 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35688 2024-09-17T18:36:25.457 INFO:tasks.workunit.client.0.smithi173.stderr:+ wait 35688 2024-09-17T18:36:25.457 INFO:tasks.workunit.client.0.smithi173.stderr:+ for t in "${!pids[@]}" 2024-09-17T18:36:25.457 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=34940 2024-09-17T18:36:25.458 INFO:tasks.workunit.client.0.smithi173.stderr:+ wait 34940 2024-09-17T18:36:25.458 INFO:tasks.workunit.client.0.smithi173.stderr:+ for t in "${!pids[@]}" 2024-09-17T18:36:25.458 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35003 2024-09-17T18:36:25.458 INFO:tasks.workunit.client.0.smithi173.stderr:+ wait 35003 2024-09-17T18:36:25.458 INFO:tasks.workunit.client.0.smithi173.stderr:+ for t in "${!pids[@]}" 2024-09-17T18:36:25.458 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35552 2024-09-17T18:36:25.458 INFO:tasks.workunit.client.0.smithi173.stderr:+ wait 35552 2024-09-17T18:36:25.458 INFO:tasks.workunit.client.0.smithi173.stderr:+ for t in "${!pids[@]}" 2024-09-17T18:36:25.458 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=34931 2024-09-17T18:36:25.458 INFO:tasks.workunit.client.0.smithi173.stderr:+ wait 34931 2024-09-17T18:36:25.458 INFO:tasks.workunit.client.0.smithi173.stderr:+ for t in "${!pids[@]}" 2024-09-17T18:36:25.458 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=34953 2024-09-17T18:36:25.458 INFO:tasks.workunit.client.0.smithi173.stderr:+ wait 34953 2024-09-17T18:36:25.459 INFO:tasks.workunit.client.0.smithi173.stderr:+ for t in "${!pids[@]}" 2024-09-17T18:36:25.459 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=36020 2024-09-17T18:36:25.459 INFO:tasks.workunit.client.0.smithi173.stderr:+ wait 36020 2024-09-17T18:36:25.459 INFO:tasks.workunit.client.0.smithi173.stderr:+ for t in "${!pids[@]}" 2024-09-17T18:36:25.459 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35190 2024-09-17T18:36:25.459 INFO:tasks.workunit.client.0.smithi173.stderr:+ wait 35190 2024-09-17T18:36:25.459 INFO:tasks.workunit.client.0.smithi173.stderr:+ for t in "${!pids[@]}" 2024-09-17T18:36:25.459 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=36195 2024-09-17T18:36:25.459 INFO:tasks.workunit.client.0.smithi173.stderr:+ wait 36195 2024-09-17T18:36:25.459 INFO:tasks.workunit.client.0.smithi173.stderr:+ for t in "${!pids[@]}" 2024-09-17T18:36:25.459 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35264 2024-09-17T18:36:25.459 INFO:tasks.workunit.client.0.smithi173.stderr:+ wait 35264 2024-09-17T18:36:25.459 INFO:tasks.workunit.client.0.smithi173.stderr:+ for t in "${!pids[@]}" 2024-09-17T18:36:25.459 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35766 2024-09-17T18:36:25.460 INFO:tasks.workunit.client.0.smithi173.stderr:+ wait 35766 2024-09-17T18:36:25.460 INFO:tasks.workunit.client.0.smithi173.stderr:+ for t in "${!pids[@]}" 2024-09-17T18:36:25.460 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=36274 2024-09-17T18:36:25.460 INFO:tasks.workunit.client.0.smithi173.stderr:+ wait 36274 2024-09-17T18:36:25.460 INFO:tasks.workunit.client.0.smithi173.stderr:+ for t in "${!pids[@]}" 2024-09-17T18:36:25.460 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35216 2024-09-17T18:36:25.460 INFO:tasks.workunit.client.0.smithi173.stderr:+ wait 35216 2024-09-17T18:36:25.460 INFO:tasks.workunit.client.0.smithi173.stderr:+ for t in "${!pids[@]}" 2024-09-17T18:36:25.460 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35599 2024-09-17T18:36:25.460 INFO:tasks.workunit.client.0.smithi173.stderr:+ wait 35599 2024-09-17T18:36:25.460 INFO:tasks.workunit.client.0.smithi173.stderr:+ for t in "${!pids[@]}" 2024-09-17T18:36:25.460 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35931 2024-09-17T18:36:25.460 INFO:tasks.workunit.client.0.smithi173.stderr:+ wait 35931 2024-09-17T18:36:25.460 INFO:tasks.workunit.client.0.smithi173.stderr:+ for t in "${!pids[@]}" 2024-09-17T18:36:25.461 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35718 2024-09-17T18:36:25.461 INFO:tasks.workunit.client.0.smithi173.stderr:+ wait 35718 2024-09-17T18:36:25.461 INFO:tasks.workunit.client.0.smithi173.stderr:+ for t in "${!pids[@]}" 2024-09-17T18:36:25.461 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=36115 2024-09-17T18:36:25.461 INFO:tasks.workunit.client.0.smithi173.stderr:+ wait 36115 2024-09-17T18:36:25.461 INFO:tasks.workunit.client.0.smithi173.stderr:+ for t in "${!pids[@]}" 2024-09-17T18:36:25.461 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35518 2024-09-17T18:36:25.461 INFO:tasks.workunit.client.0.smithi173.stderr:+ wait 35518 2024-09-17T18:36:25.461 INFO:tasks.workunit.client.0.smithi173.stderr:+ for t in "${!pids[@]}" 2024-09-17T18:36:25.461 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35109 2024-09-17T18:36:25.461 INFO:tasks.workunit.client.0.smithi173.stderr:+ wait 35109 2024-09-17T18:36:25.461 INFO:tasks.workunit.client.0.smithi173.stderr:+ for t in "${!pids[@]}" 2024-09-17T18:36:25.461 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=36260 2024-09-17T18:36:25.462 INFO:tasks.workunit.client.0.smithi173.stderr:+ wait 36260 2024-09-17T18:36:25.462 INFO:tasks.workunit.client.0.smithi173.stderr:+ for t in "${!pids[@]}" 2024-09-17T18:36:25.462 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35387 2024-09-17T18:36:25.462 INFO:tasks.workunit.client.0.smithi173.stderr:+ wait 35387 2024-09-17T18:36:25.462 INFO:tasks.workunit.client.0.smithi173.stderr:+ for t in "${!pids[@]}" 2024-09-17T18:36:25.462 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=34924 2024-09-17T18:36:25.462 INFO:tasks.workunit.client.0.smithi173.stderr:+ wait 34924 2024-09-17T18:36:25.462 INFO:tasks.workunit.client.0.smithi173.stderr:+ for t in "${!pids[@]}" 2024-09-17T18:36:25.462 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=34985 2024-09-17T18:36:25.462 INFO:tasks.workunit.client.0.smithi173.stderr:+ wait 34985 2024-09-17T18:36:25.462 INFO:tasks.workunit.client.0.smithi173.stderr:+ for t in "${!pids[@]}" 2024-09-17T18:36:25.462 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35311 2024-09-17T18:36:25.462 INFO:tasks.workunit.client.0.smithi173.stderr:+ wait 35311 2024-09-17T18:36:25.463 INFO:tasks.workunit.client.0.smithi173.stderr:+ for t in "${!pids[@]}" 2024-09-17T18:36:25.463 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35020 2024-09-17T18:36:25.463 INFO:tasks.workunit.client.0.smithi173.stderr:+ wait 35020 2024-09-17T18:36:25.463 INFO:tasks.workunit.client.0.smithi173.stderr:+ for t in "${!pids[@]}" 2024-09-17T18:36:25.463 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35439 2024-09-17T18:36:25.463 INFO:tasks.workunit.client.0.smithi173.stderr:+ wait 35439 2024-09-17T18:36:25.463 INFO:tasks.workunit.client.0.smithi173.stderr:+ for t in "${!pids[@]}" 2024-09-17T18:36:25.463 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35464 2024-09-17T18:36:25.463 INFO:tasks.workunit.client.0.smithi173.stderr:+ wait 35464 2024-09-17T18:36:25.463 INFO:tasks.workunit.client.0.smithi173.stderr:+ for t in "${!pids[@]}" 2024-09-17T18:36:25.463 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=35829 2024-09-17T18:36:25.463 INFO:tasks.workunit.client.0.smithi173.stderr:+ wait 35829 2024-09-17T18:36:25.463 INFO:tasks.workunit.client.0.smithi173.stderr:+ for t in "${!pids[@]}" 2024-09-17T18:36:25.463 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=36313 2024-09-17T18:36:25.463 INFO:tasks.workunit.client.0.smithi173.stderr:+ wait 36313 2024-09-17T18:36:25.463 INFO:tasks.workunit.client.0.smithi173.stderr:+ exit 0 2024-09-17T18:36:25.464 INFO:tasks.workunit.client.0.smithi173.stderr:+ cleanup 2024-09-17T18:36:25.464 INFO:tasks.workunit.client.0.smithi173.stderr:+ pkill -P 34918 2024-09-17T18:36:25.468 INFO:tasks.workunit.client.0.smithi173.stderr:+ true 2024-09-17T18:36:25.468 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-17T18:36:25.468 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-17T18:36:25.497 INFO:tasks.workunit:Stopping ['rados/test.sh unit_test_scan'] on client.0... 2024-09-17T18:36:25.497 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-17T18:36:25.821 DEBUG:teuthology.parallel:result is None 2024-09-17T18:36:25.821 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-17T18:36:25.845 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-17T18:36:25.846 DEBUG:teuthology.orchestra.run.smithi173:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-17T18:36:25.914 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-17T18:36:25.914 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-09-17T18:36:25.927 INFO:tasks.mon_thrash:joining mon_thrasher 2024-09-17T18:37:16.299 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-17T18:37:16.300 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-17T18:37:16.300 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-17T18:37:16.303 INFO:tasks.ceph.mon.a:Started 2024-09-17T18:37:16.303 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T18:37:16.303 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:37:16.403 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:16.401+0000 7f8d84b75c80 -1 mon.a@-1(???).log v0 log_external_backlog rewinding external_log_to from 677 -> 0 (sync_force? mon rebuild?) 2024-09-17T18:37:16.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.409+0000 7fcac3c2b640 10 monclient: get_monmap_and_config 2024-09-17T18:37:16.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.409+0000 7fcac3c2b640 10 monclient: build_initial_monmap 2024-09-17T18:37:16.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.409+0000 7fcac3c2b640 10 monclient: monmap: 2024-09-17T18:37:16.412 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:16.412 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:16.412 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:37:16.410910+0000 2024-09-17T18:37:16.412 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:37:16.410910+0000 2024-09-17T18:37:16.412 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:16.412 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:16.412 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:37:16.412 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:37:16.412 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:37:16.412 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:37:16.412 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:37:16.412 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:16.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.409+0000 7fcac3c2b640 1 Processor -- start 2024-09-17T18:37:16.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.410+0000 7fcac3c2b640 1 -- start start 2024-09-17T18:37:16.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.410+0000 7fcac3c2b640 10 monclient: init 2024-09-17T18:37:16.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.410+0000 7fcac3c2b640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:16.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.410+0000 7fcac3c2b640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-17T18:37:16.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.410+0000 7fcac3c2b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcabc0a5c00 addr v1:172.21.15.173:6789/0 2024-09-17T18:37:16.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.410+0000 7fcac3c2b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcabc0a5fe0 0x7fcabc0a63c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:16.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.410+0000 7fcac3c2b640 10 monclient(hunting): picked mon.noname-b con 0x7fcabc0a5fe0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:16.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.410+0000 7fcac3c2b640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fcabc13dc00 0x7fcabc13fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:16.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.410+0000 7fcac3c2b640 10 monclient(hunting): picked mon.noname-c con 0x7fcabc13dc00 addr v2:172.21.15.173:3300/0 2024-09-17T18:37:16.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.410+0000 7fcac3c2b640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcabc0a8a80 con 0x7fcabc0a5c00 2024-09-17T18:37:16.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.410+0000 7fcac3c2b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:16.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.410+0000 7fcac3c2b640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fcabc0a74d0 con 0x7fcabc13dc00 2024-09-17T18:37:16.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.410+0000 7fcac3c2b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:16.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.410+0000 7fcac3c2b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcabc09cb90 con 0x7fcabc0a5fe0 2024-09-17T18:37:16.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.410+0000 7fcac3c2b640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:16.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.410+0000 7fcac3c2b640 10 monclient(hunting): authenticate will time out at 1840.127636s 2024-09-17T18:37:16.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.410+0000 7fcac11c6640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcabc0a5fe0 0x7fcabc0a63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:16.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.410+0000 7fcac11c6640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcabc0a5fe0 0x7fcabc0a63c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:50838/0 (socket says 172.21.15.121:50838) 2024-09-17T18:37:16.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.410+0000 7fcac11c6640 1 -- 172.21.15.121:0/1072859701 learned_addr learned my addr 172.21.15.121:0/1072859701 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:37:16.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.410+0000 7fcac11c6640 10 monclient(hunting): get_auth_request con 0x7fcabc0a5fe0 auth_method 0 2024-09-17T18:37:16.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.410+0000 7fcac11c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:16.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.410+0000 7fcac11c6640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:16.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.410+0000 7fcac11c6640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:16.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.410+0000 7fcac11c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:16.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.410+0000 7fcac11c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:16.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.410+0000 7fcac11c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:16.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.410+0000 7fcac11c6640 10 monclient(hunting): handle_auth_done global_id 64553 payload 274 2024-09-17T18:37:16.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.410+0000 7fcac21c8640 1 --2- 172.21.15.121:0/1072859701 >> v2:172.21.15.173:3300/0 conn(0x7fcabc13dc00 0x7fcabc13fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:16.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.410+0000 7fcac11c6640 1 -- 172.21.15.121:0/1072859701 >> v2:172.21.15.173:3300/0 conn(0x7fcabc13dc00 msgr2=0x7fcabc13fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:16.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.410+0000 7fcac11c6640 1 --2- 172.21.15.121:0/1072859701 >> v2:172.21.15.173:3300/0 conn(0x7fcabc13dc00 0x7fcabc13fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:16.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.410+0000 7fcac11c6640 1 -- 172.21.15.121:0/1072859701 >> v1:172.21.15.173:6789/0 conn(0x7fcabc0a5c00 legacy=0x7fcabc09d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:16.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.411+0000 7fcac11c6640 10 monclient: _finish_hunting 0 2024-09-17T18:37:16.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.411+0000 7fcac11c6640 1 monclient: found mon.noname-b 2024-09-17T18:37:16.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.411+0000 7fcac11c6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:16.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.411+0000 7fcac11c6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:37:16.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.411+0000 7fcac11c6640 1 -- 172.21.15.121:0/1072859701 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcabc09ce70 con 0x7fcabc0a5fe0 2024-09-17T18:37:16.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.413+0000 7fcac11c6640 1 -- 172.21.15.121:0/1072859701 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcabc0a5fe0 msgr2=0x7fcabc0a63c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:16.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.413+0000 7fcac11c6640 1 --2- 172.21.15.121:0/1072859701 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcabc0a5fe0 0x7fcabc0a63c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcaac009870 tx=0x7fcaac02a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e47ea194d957b0e server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:16.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.413+0000 7fcac09c5640 1 -- 172.21.15.121:0/1072859701 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaac037070 con 0x7fcabc0a5fe0 2024-09-17T18:37:16.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.413+0000 7fcac09c5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:16.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.413+0000 7fcac09c5640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:37:16.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.413+0000 7fcac09c5640 10 monclient: dump: 2024-09-17T18:37:16.422 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:16.422 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:16.422 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:16.422 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:16.422 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:16.422 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:16.422 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:16.422 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:16.422 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:16.422 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:16.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.413+0000 7fcac09c5640 10 monclient: _finish_auth 0 2024-09-17T18:37:16.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.413+0000 7fcac09c5640 10 monclient: _check_auth_tickets 2024-09-17T18:37:16.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.413+0000 7fcac09c5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:16.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.413+0000 7fcac09c5640 1 -- 172.21.15.121:0/1072859701 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcaac032640 con 0x7fcabc0a5fe0 2024-09-17T18:37:16.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.413+0000 7fcac3c2b640 5 monclient: authenticate success, global_id 64553 2024-09-17T18:37:16.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.413+0000 7fcac3c2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:37:16.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.413+0000 7fcac09c5640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:16.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.413+0000 7fcac3c2b640 10 monclient: get_monmap_and_config success 2024-09-17T18:37:16.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.413+0000 7fcac09c5640 1 -- 172.21.15.121:0/1072859701 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaac032a10 con 0x7fcabc0a5fe0 2024-09-17T18:37:16.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.413+0000 7fcac09c5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:16.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.413+0000 7fcac3c2b640 4 set_mon_vals no callback set 2024-09-17T18:37:16.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.413+0000 7fcac09c5640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:37:16.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.413+0000 7fcac09c5640 10 monclient: dump: 2024-09-17T18:37:16.423 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:16.423 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:16.423 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:16.424 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:16.424 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:16.424 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:16.424 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:16.424 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:16.424 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:16.424 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:16.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.414+0000 7fcac3c2b640 10 monclient: shutdown 2024-09-17T18:37:16.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.414+0000 7fcac3c2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:16.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.414+0000 7fcac3c2b640 1 -- 172.21.15.121:0/1072859701 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcabc0a5fe0 msgr2=0x7fcabc0a63c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:16.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.414+0000 7fcac3c2b640 1 --2- 172.21.15.121:0/1072859701 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcabc0a5fe0 0x7fcabc0a63c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcaac009870 tx=0x7fcaac02a0a0 comp rx=0 tx=0).stop 2024-09-17T18:37:16.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.414+0000 7fcac3c2b640 1 -- 172.21.15.121:0/1072859701 shutdown_connections 2024-09-17T18:37:16.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.414+0000 7fcac3c2b640 1 --2- 172.21.15.121:0/1072859701 >> v2:172.21.15.173:3300/0 conn(0x7fcabc13dc00 0x7fcabc13fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:16.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.414+0000 7fcac3c2b640 1 --2- 172.21.15.121:0/1072859701 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcabc0a5fe0 0x7fcabc0a63c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:16.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.414+0000 7fcac3c2b640 1 -- 172.21.15.121:0/1072859701 >> 172.21.15.121:0/1072859701 conn(0x7fcabc0ab9b0 msgr2=0x7fcabc0a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:16.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.414+0000 7fcac3c2b640 1 -- 172.21.15.121:0/1072859701 shutdown_connections 2024-09-17T18:37:16.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.414+0000 7fcac3c2b640 1 -- 172.21.15.121:0/1072859701 wait complete. 2024-09-17T18:37:16.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.415+0000 7fcac3c2b640 10 monclient: build_initial_monmap 2024-09-17T18:37:16.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.415+0000 7fcac3c2b640 10 monclient: monmap: 2024-09-17T18:37:16.425 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:16.425 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:37:16.425 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:37:16.425 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:37:16.425 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:16.425 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:16.425 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:37:16.425 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:37:16.425 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:37:16.425 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:16.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.415+0000 7fcac3c2b640 1 librados: starting msgr at 2024-09-17T18:37:16.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.415+0000 7fcac3c2b640 1 librados: starting objecter 2024-09-17T18:37:16.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.415+0000 7fcac3c2b640 1 Processor -- start 2024-09-17T18:37:16.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.415+0000 7fcac3c2b640 1 -- start start 2024-09-17T18:37:16.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.415+0000 7fcac3c2b640 1 librados: setting wanted keys 2024-09-17T18:37:16.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.415+0000 7fcac3c2b640 1 librados: calling monclient init 2024-09-17T18:37:16.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.415+0000 7fcac3c2b640 10 monclient: init 2024-09-17T18:37:16.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.415+0000 7fcac3c2b640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:16.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.415+0000 7fcac3c2b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T18:37:16.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.415+0000 7fcac3c2b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcabc0a5c00 0x7fcabc1387f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:16.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.415+0000 7fcac3c2b640 10 monclient(hunting): picked mon.noname-c con 0x7fcabc0a5c00 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:16.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.415+0000 7fcac3c2b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcabc0a5fe0 0x7fcabc138d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:16.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.415+0000 7fcac3c2b640 10 monclient(hunting): picked mon.noname-b con 0x7fcabc0a5fe0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:37:16.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.415+0000 7fcac3c2b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcabc13dc00 0x7fcabc13b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:16.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.415+0000 7fcac3c2b640 10 monclient(hunting): picked mon.noname-a con 0x7fcabc13dc00 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:37:16.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.415+0000 7fcac3c2b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:16.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.415+0000 7fcac3c2b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcabc070550 con 0x7fcabc13dc00 2024-09-17T18:37:16.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.415+0000 7fcac3c2b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:16.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.415+0000 7fcac3c2b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcabc0703d0 con 0x7fcabc0a5fe0 2024-09-17T18:37:16.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.415+0000 7fcac3c2b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:16.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.415+0000 7fcac3c2b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcabc140530 con 0x7fcabc0a5c00 2024-09-17T18:37:16.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.415+0000 7fcac3c2b640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:16.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.415+0000 7fcac3c2b640 10 monclient(hunting): authenticate will time out at 1840.132931s 2024-09-17T18:37:16.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.416+0000 7fcac19c7640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcabc0a5c00 0x7fcabc1387f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:16.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.416+0000 7fcac19c7640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcabc0a5c00 0x7fcabc1387f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:50850/0 (socket says 172.21.15.121:50850) 2024-09-17T18:37:16.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.416+0000 7fcac19c7640 1 -- 172.21.15.121:0/1048227125 learned_addr learned my addr 172.21.15.121:0/1048227125 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:37:16.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.416+0000 7fcac19c7640 10 monclient(hunting): get_auth_request con 0x7fcabc0a5c00 auth_method 0 2024-09-17T18:37:16.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.416+0000 7fcac19c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:16.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.416+0000 7fcac19c7640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:16.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.416+0000 7fcac19c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:16.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.416+0000 7fcac21c8640 1 --2- 172.21.15.121:0/1048227125 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcabc13dc00 0x7fcabc13b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:16.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.416+0000 7fcac21c8640 10 monclient(hunting): get_auth_request con 0x7fcabc13dc00 auth_method 0 2024-09-17T18:37:16.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.416+0000 7fcac21c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:16.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.416+0000 7fcac21c8640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:16.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.416+0000 7fcac21c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:16.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.416+0000 7fcac19c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:16.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.416+0000 7fcac19c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:16.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.416+0000 7fcac19c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:16.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.416+0000 7fcac21c8640 1 -- 172.21.15.121:0/1048227125 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcabc13dc00 msgr2=0x7fcabc13b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-17T18:37:16.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.416+0000 7fcac21c8640 1 -- 172.21.15.121:0/1048227125 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcabc13dc00 msgr2=0x7fcabc13b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:37:16.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.416+0000 7fcac21c8640 1 --2- 172.21.15.121:0/1048227125 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcabc13dc00 0x7fcabc13b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:37:16.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.416+0000 7fcac21c8640 1 --2- 172.21.15.121:0/1048227125 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcabc13dc00 0x7fcabc13b990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:37:16.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.416+0000 7fcac11c6640 1 --2- 172.21.15.121:0/1048227125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcabc0a5fe0 0x7fcabc138d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:16.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.416+0000 7fcac11c6640 10 monclient(hunting): get_auth_request con 0x7fcabc0a5fe0 auth_method 0 2024-09-17T18:37:16.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.416+0000 7fcac11c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:16.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.416+0000 7fcac11c6640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:16.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.416+0000 7fcac11c6640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:16.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.416+0000 7fcac19c7640 10 monclient(hunting): handle_auth_done global_id 64556 payload 824 2024-09-17T18:37:16.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.416+0000 7fcac19c7640 1 -- 172.21.15.121:0/1048227125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcabc0a5fe0 msgr2=0x7fcabc138d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:16.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.416+0000 7fcac19c7640 1 --2- 172.21.15.121:0/1048227125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcabc0a5fe0 0x7fcabc138d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:16.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.416+0000 7fcac19c7640 1 -- 172.21.15.121:0/1048227125 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcabc13dc00 msgr2=0x7fcabc13b990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:37:16.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.416+0000 7fcac19c7640 1 --2- 172.21.15.121:0/1048227125 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcabc13dc00 0x7fcabc13b990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:16.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.416+0000 7fcac19c7640 10 monclient: _finish_hunting 0 2024-09-17T18:37:16.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.416+0000 7fcac19c7640 1 monclient: found mon.noname-c 2024-09-17T18:37:16.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.416+0000 7fcac19c7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:16.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.416+0000 7fcac19c7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:37:16.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.416+0000 7fcac19c7640 1 -- 172.21.15.121:0/1048227125 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcabc0a23f0 con 0x7fcabc0a5c00 2024-09-17T18:37:16.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.418+0000 7fcac19c7640 1 -- 172.21.15.121:0/1048227125 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcabc0a5c00 msgr2=0x7fcabc1387f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:16.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.418+0000 7fcac19c7640 1 --2- 172.21.15.121:0/1048227125 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcabc0a5c00 0x7fcabc1387f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fcab800b410 tx=0x7fcab800b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:16.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.418+0000 7fcab27fc640 1 -- 172.21.15.121:0/1048227125 DispatchQueue::entry inject delay of 0.002000 2024-09-17T18:37:16.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.420+0000 7fcab27fc640 10 client.?.objecter ms_handle_connect 0x7fcabc0a5c00 2024-09-17T18:37:16.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.420+0000 7fcab27fc640 10 client.?.objecter resend_mon_ops 2024-09-17T18:37:16.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.420+0000 7fcab27fc640 1 -- 172.21.15.121:0/1048227125 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab8015020 con 0x7fcabc0a5c00 2024-09-17T18:37:16.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.420+0000 7fcab27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:16.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.420+0000 7fcab27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:37:16.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.420+0000 7fcab27fc640 10 monclient: dump: 2024-09-17T18:37:16.432 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:16.432 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:16.432 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:16.432 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:16.432 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:16.432 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:16.432 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:16.432 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:16.432 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:16.432 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:16.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.420+0000 7fcab27fc640 10 monclient: _finish_auth 0 2024-09-17T18:37:16.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.420+0000 7fcab27fc640 10 monclient: _check_auth_tickets 2024-09-17T18:37:16.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.420+0000 7fcab27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:16.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.420+0000 7fcab27fc640 1 -- 172.21.15.121:0/1048227125 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcab8011a40 con 0x7fcabc0a5c00 2024-09-17T18:37:16.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.420+0000 7fcab27fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:16.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.420+0000 7fcab27fc640 1 -- 172.21.15.121:0/1048227125 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab8011e30 con 0x7fcabc0a5c00 2024-09-17T18:37:16.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.420+0000 7fcab27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:16.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.420+0000 7fcab27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:37:16.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.420+0000 7fcab27fc640 10 monclient: dump: 2024-09-17T18:37:16.433 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:16.433 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:16.433 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:16.433 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:16.433 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:16.433 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:16.433 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:16.433 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:16.433 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:16.433 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:16.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.420+0000 7fcac3c2b640 5 monclient: authenticate success, global_id 64556 2024-09-17T18:37:16.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.420+0000 7fcac3c2b640 10 monclient: _renew_subs 2024-09-17T18:37:16.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.420+0000 7fcac3c2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:37:16.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.420+0000 7fcac3c2b640 1 -- 172.21.15.121:0/1048227125 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcabc13d3c0 con 0x7fcabc0a5c00 2024-09-17T18:37:16.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.420+0000 7fcab3fff640 4 set_mon_vals no callback set 2024-09-17T18:37:16.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.420+0000 7fcac3c2b640 10 client.64556.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:37:16.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.420+0000 7fcac3c2b640 10 monclient: _renew_subs 2024-09-17T18:37:16.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.420+0000 7fcac3c2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:37:16.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.420+0000 7fcac3c2b640 1 -- 172.21.15.121:0/1048227125 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcabc121850 con 0x7fcabc0a5c00 2024-09-17T18:37:16.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.420+0000 7fcac3c2b640 1 librados: init done 2024-09-17T18:37:16.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.421+0000 7fcac3c2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:16.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.421+0000 7fcac3c2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:16.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.421+0000 7fcac3c2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:37:16.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.421+0000 7fcab27fc640 1 -- 172.21.15.121:0/1048227125 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fcab8018020 con 0x7fcabc0a5c00 2024-09-17T18:37:16.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.421+0000 7fcac3c2b640 1 -- 172.21.15.121:0/1048227125 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcabc0a63c0 con 0x7fcabc0a5c00 2024-09-17T18:37:16.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.422+0000 7fcab27fc640 1 --2- 172.21.15.121:0/1048227125 >> v2:172.21.15.173:6812/2423786037 conn(0x7fcaa00401c0 0x7fcaa0042680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:16.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.422+0000 7fcab27fc640 1 -- 172.21.15.121:0/1048227125 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(679..679 src has 247..679) ==== 4685+0+0 (secure 0 0 0) 0x7fcab8053420 con 0x7fcabc0a5c00 2024-09-17T18:37:16.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.422+0000 7fcab27fc640 10 client.64556.objecter ms_dispatch 0x7fcabc000b90 osd_map(679..679 src has 247..679) 2024-09-17T18:37:16.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.422+0000 7fcab27fc640 3 client.64556.objecter handle_osd_map got epochs [679,679] > 0 2024-09-17T18:37:16.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.422+0000 7fcab27fc640 3 client.64556.objecter handle_osd_map decoding full epoch 679 2024-09-17T18:37:16.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.422+0000 7fcab27fc640 20 client.64556.objecter dump_active .. 0 homeless 2024-09-17T18:37:16.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.422+0000 7fcac11c6640 1 --2- 172.21.15.121:0/1048227125 >> v2:172.21.15.173:6812/2423786037 conn(0x7fcaa00401c0 0x7fcaa0042680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:16.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.422+0000 7fcac11c6640 10 monclient: get_auth_request con 0x7fcaa00401c0 auth_method 0 2024-09-17T18:37:16.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.425+0000 7fcac11c6640 1 --2- 172.21.15.121:0/1048227125 >> v2:172.21.15.173:6812/2423786037 conn(0x7fcaa00401c0 0x7fcaa0042680 secure :-1 s=READY pgs=4087 cs=0 l=1 rev1=1 crypto rx=0x7fcabc06a3f0 tx=0x7fcaac02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:16.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.425+0000 7fcab27fc640 10 client.64556.objecter ms_handle_connect 0x7fcaa00401c0 2024-09-17T18:37:16.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.426+0000 7fcab27fc640 1 -- 172.21.15.121:0/1048227125 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fcab801f020 con 0x7fcabc0a5c00 2024-09-17T18:37:16.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.426+0000 7fcab27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:16.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.426+0000 7fcab27fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:37:16.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.668+0000 7fcac3c2b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:37:16.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.668+0000 7fcac3c2b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:37:16.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.668+0000 7fcac3c2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:37:16.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.668+0000 7fcac3c2b640 1 -- 172.21.15.121:0/1048227125 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fcabc06cd40 con 0x7fcabc0a5c00 2024-09-17T18:37:16.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.668+0000 7fcab27fc640 1 -- 172.21.15.121:0/1048227125 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7fcab8051e80 con 0x7fcabc0a5c00 2024-09-17T18:37:16.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.668+0000 7fcab27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:37:16.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.668+0000 7fcab27fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:37:16.672 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":62,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":92,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:37:16.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.670+0000 7fcac3c2b640 10 librados: watch_flush enter 2024-09-17T18:37:16.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.670+0000 7fcac3c2b640 10 librados: watch_flush exit 2024-09-17T18:37:16.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.670+0000 7fcac3c2b640 20 client.64556.objecter shutdown clearing up homeless session... 2024-09-17T18:37:16.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.670+0000 7fcac3c2b640 10 client.64556.objecter successfully canceled tick 2024-09-17T18:37:16.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.671+0000 7fcac3c2b640 1 -- 172.21.15.121:0/1048227125 >> v2:172.21.15.173:6812/2423786037 conn(0x7fcaa00401c0 msgr2=0x7fcaa0042680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:16.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.671+0000 7fcac3c2b640 1 --2- 172.21.15.121:0/1048227125 >> v2:172.21.15.173:6812/2423786037 conn(0x7fcaa00401c0 0x7fcaa0042680 secure :-1 s=READY pgs=4087 cs=0 l=1 rev1=1 crypto rx=0x7fcabc06a3f0 tx=0x7fcaac02e000 comp rx=0 tx=0).stop 2024-09-17T18:37:16.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.671+0000 7fcac3c2b640 10 monclient: shutdown 2024-09-17T18:37:16.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.671+0000 7fcac3c2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:16.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.671+0000 7fcac3c2b640 1 -- 172.21.15.121:0/1048227125 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcabc0a5c00 msgr2=0x7fcabc1387f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:16.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.671+0000 7fcac3c2b640 1 --2- 172.21.15.121:0/1048227125 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcabc0a5c00 0x7fcabc1387f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fcab800b410 tx=0x7fcab800b8d0 comp rx=0 tx=0).stop 2024-09-17T18:37:16.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.671+0000 7fcac3c2b640 1 -- 172.21.15.121:0/1048227125 shutdown_connections 2024-09-17T18:37:16.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.671+0000 7fcac3c2b640 1 --2- 172.21.15.121:0/1048227125 >> v2:172.21.15.173:6812/2423786037 conn(0x7fcaa00401c0 0x7fcaa0042680 unknown :-1 s=CLOSED pgs=4087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:16.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.671+0000 7fcac3c2b640 1 --2- 172.21.15.121:0/1048227125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcabc0a5fe0 0x7fcabc138d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:16.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.671+0000 7fcac3c2b640 1 --2- 172.21.15.121:0/1048227125 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcabc13dc00 0x7fcabc13b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:16.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.671+0000 7fcac3c2b640 1 --2- 172.21.15.121:0/1048227125 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcabc0a5c00 0x7fcabc1387f0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:16.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.671+0000 7fcac3c2b640 1 -- 172.21.15.121:0/1048227125 >> 172.21.15.121:0/1048227125 conn(0x7fcabc0ab9b0 msgr2=0x7fcabc13e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:16.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.671+0000 7fcac3c2b640 1 -- 172.21.15.121:0/1048227125 shutdown_connections 2024-09-17T18:37:16.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.671+0000 7fcac3c2b640 1 -- 172.21.15.121:0/1048227125 wait complete. 2024-09-17T18:37:16.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:16.671+0000 7fcac3c2b640 1 librados: shutdown 2024-09-17T18:37:19.684 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:37:19.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.789+0000 7fec3f73d640 10 monclient: get_monmap_and_config 2024-09-17T18:37:19.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.789+0000 7fec3f73d640 10 monclient: build_initial_monmap 2024-09-17T18:37:19.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.789+0000 7fec3f73d640 10 monclient: monmap: 2024-09-17T18:37:19.791 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:19.791 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:19.791 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:37:19.790312+0000 2024-09-17T18:37:19.791 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:37:19.790312+0000 2024-09-17T18:37:19.791 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:19.792 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:19.792 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:37:19.792 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:37:19.792 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:37:19.792 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:37:19.792 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:37:19.792 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:19.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.790+0000 7fec3f73d640 1 Processor -- start 2024-09-17T18:37:19.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.790+0000 7fec3f73d640 1 -- start start 2024-09-17T18:37:19.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.790+0000 7fec3f73d640 10 monclient: init 2024-09-17T18:37:19.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.791+0000 7fec3f73d640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:19.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.791+0000 7fec3f73d640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-17T18:37:19.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.791+0000 7fec3f73d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fec380a1a50 addr v1:172.21.15.121:6789/0 2024-09-17T18:37:19.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.791+0000 7fec3f73d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec380a2540 0x7fec38142050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:19.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.791+0000 7fec3f73d640 10 monclient(hunting): picked mon.noname-b con 0x7fec380a2540 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:19.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.791+0000 7fec3f73d640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fec38142590 0x7fec38144980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:19.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.791+0000 7fec3f73d640 10 monclient(hunting): picked mon.noname-c con 0x7fec38142590 addr v2:172.21.15.173:3300/0 2024-09-17T18:37:19.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.791+0000 7fec3f73d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec380a0960 con 0x7fec380a1a50 2024-09-17T18:37:19.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.791+0000 7fec3f73d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:19.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.791+0000 7fec3f73d640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fec380a3000 con 0x7fec38142590 2024-09-17T18:37:19.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.791+0000 7fec3f73d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:19.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.791+0000 7fec3f73d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fec38077730 con 0x7fec380a2540 2024-09-17T18:37:19.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.791+0000 7fec3f73d640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:19.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.791+0000 7fec3f73d640 10 monclient(hunting): authenticate will time out at 1843.507108s 2024-09-17T18:37:19.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.791+0000 7fec3ccd8640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec380a2540 0x7fec38142050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:19.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.791+0000 7fec3d4d9640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fec380a1a50 0x7fec380a1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:47280/0 (socket says 172.21.15.121:47280) 2024-09-17T18:37:19.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.791+0000 7fec3d4d9640 1 -- 172.21.15.121:0/3374705014 learned_addr learned my addr 172.21.15.121:0/3374705014 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:37:19.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.791+0000 7fec3ccd8640 10 monclient(hunting): get_auth_request con 0x7fec380a2540 auth_method 0 2024-09-17T18:37:19.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.791+0000 7fec3ccd8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:19.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.791+0000 7fec3ccd8640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:19.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.791+0000 7fec3ccd8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:19.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.791+0000 7fec3ccd8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:19.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.791+0000 7fec3ccd8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:19.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.791+0000 7fec3ccd8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:19.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.791+0000 7fec3ccd8640 10 monclient(hunting): handle_auth_done global_id 64559 payload 274 2024-09-17T18:37:19.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.791+0000 7fec3ccd8640 1 -- 172.21.15.121:0/3374705014 >> v2:172.21.15.173:3300/0 conn(0x7fec38142590 msgr2=0x7fec38144980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:19.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.791+0000 7fec3dcda640 1 --2- 172.21.15.121:0/3374705014 >> v2:172.21.15.173:3300/0 conn(0x7fec38142590 0x7fec38144980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:19.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.791+0000 7fec3ccd8640 1 --2- 172.21.15.121:0/3374705014 >> v2:172.21.15.173:3300/0 conn(0x7fec38142590 0x7fec38144980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:19.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.791+0000 7fec3ccd8640 1 -- 172.21.15.121:0/3374705014 >> v1:172.21.15.121:6789/0 conn(0x7fec380a1a50 legacy=0x7fec380a1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:19.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.791+0000 7fec3ccd8640 10 monclient: _finish_hunting 0 2024-09-17T18:37:19.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.791+0000 7fec3ccd8640 1 monclient: found mon.noname-b 2024-09-17T18:37:19.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.791+0000 7fec3ccd8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:19.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.791+0000 7fec3ccd8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:37:19.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.791+0000 7fec3ccd8640 1 -- 172.21.15.121:0/3374705014 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec3809e6f0 con 0x7fec380a2540 2024-09-17T18:37:19.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.791+0000 7fec3dcda640 10 monclient: get_auth_request con 0x7fec38142590 auth_method 0 2024-09-17T18:37:19.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.791+0000 7fec3dcda640 1 --2- 172.21.15.121:0/3374705014 >> v2:172.21.15.173:3300/0 conn(0x7fec38142590 0x7fec38144980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T18:37:19.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.794+0000 7fec3ccd8640 1 -- 172.21.15.121:0/3374705014 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec380a2540 msgr2=0x7fec38142050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:19.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.794+0000 7fec3ccd8640 1 --2- 172.21.15.121:0/3374705014 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec380a2540 0x7fec38142050 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fec340097a0 tx=0x7fec34029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d93d57832facb83 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:19.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.794+0000 7fec2ffff640 1 -- 172.21.15.121:0/3374705014 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec34037070 con 0x7fec380a2540 2024-09-17T18:37:19.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.794+0000 7fec2ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:19.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.794+0000 7fec2ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:37:19.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.794+0000 7fec2ffff640 10 monclient: dump: 2024-09-17T18:37:19.803 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:19.803 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:19.803 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:19.803 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:19.803 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:19.804 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:19.804 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:19.804 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:19.804 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:19.804 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:19.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.794+0000 7fec2ffff640 10 monclient: _finish_auth 0 2024-09-17T18:37:19.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.794+0000 7fec2ffff640 10 monclient: _check_auth_tickets 2024-09-17T18:37:19.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.794+0000 7fec2ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:19.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.794+0000 7fec2ffff640 1 -- 172.21.15.121:0/3374705014 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec34032640 con 0x7fec380a2540 2024-09-17T18:37:19.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.794+0000 7fec2ffff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:19.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.794+0000 7fec2ffff640 1 -- 172.21.15.121:0/3374705014 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec34032a10 con 0x7fec380a2540 2024-09-17T18:37:19.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.794+0000 7fec3f73d640 5 monclient: authenticate success, global_id 64559 2024-09-17T18:37:19.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.794+0000 7fec3f73d640 10 monclient: get_monmap_and_config success 2024-09-17T18:37:19.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.794+0000 7fec3f73d640 4 set_mon_vals no callback set 2024-09-17T18:37:19.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.794+0000 7fec2ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:19.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.794+0000 7fec2ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:37:19.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.794+0000 7fec2ffff640 10 monclient: dump: 2024-09-17T18:37:19.805 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:19.805 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:19.805 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:19.805 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:19.805 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:19.805 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:19.805 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:19.805 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:19.805 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:19.805 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:19.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.795+0000 7fec3f73d640 10 monclient: shutdown 2024-09-17T18:37:19.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.795+0000 7fec3f73d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:19.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.795+0000 7fec3f73d640 1 -- 172.21.15.121:0/3374705014 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec380a2540 msgr2=0x7fec38142050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:19.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.795+0000 7fec3f73d640 1 --2- 172.21.15.121:0/3374705014 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec380a2540 0x7fec38142050 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fec340097a0 tx=0x7fec34029fd0 comp rx=0 tx=0).stop 2024-09-17T18:37:19.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.795+0000 7fec3f73d640 1 -- 172.21.15.121:0/3374705014 shutdown_connections 2024-09-17T18:37:19.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.795+0000 7fec3f73d640 1 --2- 172.21.15.121:0/3374705014 >> v2:172.21.15.173:3300/0 conn(0x7fec38142590 0x7fec38144980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:19.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.795+0000 7fec3f73d640 1 --2- 172.21.15.121:0/3374705014 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec380a2540 0x7fec38142050 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:19.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.795+0000 7fec3f73d640 1 -- 172.21.15.121:0/3374705014 >> 172.21.15.121:0/3374705014 conn(0x7fec380a3b50 msgr2=0x7fec380a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:19.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.795+0000 7fec3f73d640 1 -- 172.21.15.121:0/3374705014 shutdown_connections 2024-09-17T18:37:19.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.795+0000 7fec3f73d640 1 -- 172.21.15.121:0/3374705014 wait complete. 2024-09-17T18:37:19.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.795+0000 7fec3f73d640 10 monclient: build_initial_monmap 2024-09-17T18:37:19.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.795+0000 7fec3f73d640 10 monclient: monmap: 2024-09-17T18:37:19.806 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:19.806 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:37:19.806 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:37:19.807 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:37:19.807 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:19.807 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:19.807 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:37:19.807 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:37:19.807 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:37:19.807 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:19.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3f73d640 1 librados: starting msgr at 2024-09-17T18:37:19.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3f73d640 1 librados: starting objecter 2024-09-17T18:37:19.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3f73d640 1 Processor -- start 2024-09-17T18:37:19.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3f73d640 1 -- start start 2024-09-17T18:37:19.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3f73d640 1 librados: setting wanted keys 2024-09-17T18:37:19.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3f73d640 1 librados: calling monclient init 2024-09-17T18:37:19.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3f73d640 10 monclient: init 2024-09-17T18:37:19.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3f73d640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:19.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3f73d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T18:37:19.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3f73d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec380a1a50 0x7fec381d1c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:19.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3f73d640 10 monclient(hunting): picked mon.noname-b con 0x7fec380a1a50 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:37:19.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3f73d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec380a2540 0x7fec3813e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:19.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3f73d640 10 monclient(hunting): picked mon.noname-a con 0x7fec380a2540 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:37:19.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3f73d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec38142590 0x7fec3813eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:19.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3f73d640 10 monclient(hunting): picked mon.noname-c con 0x7fec38142590 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:19.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3f73d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:19.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3f73d640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fec380a2950 con 0x7fec380a2540 2024-09-17T18:37:19.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3f73d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:19.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3f73d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fec3809dce0 con 0x7fec380a1a50 2024-09-17T18:37:19.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3f73d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:19.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3f73d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fec38129ed0 con 0x7fec38142590 2024-09-17T18:37:19.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3f73d640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:19.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3f73d640 10 monclient(hunting): authenticate will time out at 1843.512651s 2024-09-17T18:37:19.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3dcda640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec38142590 0x7fec3813eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:19.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3ccd8640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec380a2540 0x7fec3813e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:19.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3ccd8640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec380a2540 0x7fec3813e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51098/0 (socket says 172.21.15.121:51098) 2024-09-17T18:37:19.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3dcda640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec38142590 0x7fec3813eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:36630/0 (socket says 172.21.15.121:36630) 2024-09-17T18:37:19.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3ccd8640 1 -- 172.21.15.121:0/2945964339 learned_addr learned my addr 172.21.15.121:0/2945964339 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:37:19.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3ccd8640 10 monclient(hunting): get_auth_request con 0x7fec380a2540 auth_method 0 2024-09-17T18:37:19.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3ccd8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:19.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3ccd8640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:19.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3ccd8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:19.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3dcda640 10 monclient(hunting): get_auth_request con 0x7fec38142590 auth_method 0 2024-09-17T18:37:19.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3dcda640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:19.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3dcda640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:19.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3dcda640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:19.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3dcda640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:19.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3ccd8640 1 -- 172.21.15.121:0/2945964339 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec380a2540 msgr2=0x7fec3813e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T18:37:19.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3dcda640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:19.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3ccd8640 1 -- 172.21.15.121:0/2945964339 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec380a2540 msgr2=0x7fec3813e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:37:19.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3ccd8640 1 --2- 172.21.15.121:0/2945964339 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec380a2540 0x7fec3813e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:37:19.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3dcda640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:19.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3ccd8640 1 --2- 172.21.15.121:0/2945964339 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec380a2540 0x7fec3813e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:37:19.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3d4d9640 1 --2- 172.21.15.121:0/2945964339 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec380a1a50 0x7fec381d1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:19.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3d4d9640 10 monclient(hunting): get_auth_request con 0x7fec380a1a50 auth_method 0 2024-09-17T18:37:19.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3d4d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:19.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3d4d9640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:19.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.796+0000 7fec3d4d9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:19.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.797+0000 7fec3d4d9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:19.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.797+0000 7fec3d4d9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:19.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.797+0000 7fec3d4d9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:19.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.797+0000 7fec3dcda640 10 monclient(hunting): handle_auth_done global_id 64562 payload 824 2024-09-17T18:37:19.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.797+0000 7fec3dcda640 1 -- 172.21.15.121:0/2945964339 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec380a1a50 msgr2=0x7fec381d1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:19.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.797+0000 7fec3dcda640 1 --2- 172.21.15.121:0/2945964339 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec380a1a50 0x7fec381d1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:19.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.797+0000 7fec3dcda640 1 -- 172.21.15.121:0/2945964339 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec380a2540 msgr2=0x7fec3813e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:37:19.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.797+0000 7fec3dcda640 1 --2- 172.21.15.121:0/2945964339 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec380a2540 0x7fec3813e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:19.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.797+0000 7fec3dcda640 10 monclient: _finish_hunting 0 2024-09-17T18:37:19.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.797+0000 7fec3dcda640 1 monclient: found mon.noname-c 2024-09-17T18:37:19.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.797+0000 7fec3dcda640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:19.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.797+0000 7fec3dcda640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:37:19.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.797+0000 7fec3dcda640 1 -- 172.21.15.121:0/2945964339 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec3809e240 con 0x7fec38142590 2024-09-17T18:37:19.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.799+0000 7fec3dcda640 1 -- 172.21.15.121:0/2945964339 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec38142590 msgr2=0x7fec3813eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:19.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.799+0000 7fec3dcda640 1 --2- 172.21.15.121:0/2945964339 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec38142590 0x7fec3813eaf0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fec3000cdf0 tx=0x7fec3000a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:19.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.799+0000 7fec2dffb640 10 client.?.objecter ms_handle_connect 0x7fec38142590 2024-09-17T18:37:19.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.799+0000 7fec2dffb640 10 client.?.objecter resend_mon_ops 2024-09-17T18:37:19.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.799+0000 7fec2dffb640 1 -- 172.21.15.121:0/2945964339 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec3001a070 con 0x7fec38142590 2024-09-17T18:37:19.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.799+0000 7fec2dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:19.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.799+0000 7fec2dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:37:19.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.799+0000 7fec2dffb640 10 monclient: dump: 2024-09-17T18:37:19.812 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:19.812 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:19.812 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:19.812 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:19.812 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:19.812 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:19.813 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:19.813 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:19.813 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:19.813 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:19.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.799+0000 7fec2dffb640 10 monclient: _finish_auth 0 2024-09-17T18:37:19.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.799+0000 7fec2dffb640 10 monclient: _check_auth_tickets 2024-09-17T18:37:19.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.799+0000 7fec2dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:19.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.799+0000 7fec2dffb640 1 -- 172.21.15.121:0/2945964339 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec3000da50 con 0x7fec38142590 2024-09-17T18:37:19.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.799+0000 7fec2dffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:19.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.799+0000 7fec2dffb640 1 -- 172.21.15.121:0/2945964339 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec3000de40 con 0x7fec38142590 2024-09-17T18:37:19.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.799+0000 7fec3f73d640 5 monclient: authenticate success, global_id 64562 2024-09-17T18:37:19.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.799+0000 7fec2dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:19.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.799+0000 7fec2dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:37:19.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.799+0000 7fec2dffb640 10 monclient: dump: 2024-09-17T18:37:19.814 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:19.814 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:19.814 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:19.814 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:19.814 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:19.814 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:19.814 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:19.814 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:19.814 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:19.814 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:19.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.799+0000 7fec3f73d640 10 monclient: _renew_subs 2024-09-17T18:37:19.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.799+0000 7fec2f7fe640 4 set_mon_vals no callback set 2024-09-17T18:37:19.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.799+0000 7fec3f73d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:37:19.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.799+0000 7fec3f73d640 1 -- 172.21.15.121:0/2945964339 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec381d2150 con 0x7fec38142590 2024-09-17T18:37:19.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.799+0000 7fec3f73d640 10 client.64562.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:37:19.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.799+0000 7fec3f73d640 10 monclient: _renew_subs 2024-09-17T18:37:19.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.799+0000 7fec3f73d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:37:19.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.799+0000 7fec3f73d640 1 -- 172.21.15.121:0/2945964339 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fec3812af60 con 0x7fec38142590 2024-09-17T18:37:19.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.799+0000 7fec3f73d640 1 librados: init done 2024-09-17T18:37:19.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.800+0000 7fec2dffb640 1 -- 172.21.15.121:0/2945964339 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fec3001d070 con 0x7fec38142590 2024-09-17T18:37:19.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.800+0000 7fec3f73d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:19.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.800+0000 7fec3f73d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:19.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.800+0000 7fec3f73d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:37:19.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.800+0000 7fec2dffb640 1 --2- 172.21.15.121:0/2945964339 >> v2:172.21.15.173:6812/2423786037 conn(0x7fec1c0401c0 0x7fec1c042680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:19.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.800+0000 7fec3f73d640 1 -- 172.21.15.121:0/2945964339 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec380a6df0 con 0x7fec38142590 2024-09-17T18:37:19.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.801+0000 7fec2dffb640 1 -- 172.21.15.121:0/2945964339 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(679..679 src has 247..679) ==== 4685+0+0 (secure 0 0 0) 0x7fec30053c80 con 0x7fec38142590 2024-09-17T18:37:19.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.801+0000 7fec2dffb640 10 client.64562.objecter ms_dispatch 0x7fec38000b90 osd_map(679..679 src has 247..679) 2024-09-17T18:37:19.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.801+0000 7fec2dffb640 3 client.64562.objecter handle_osd_map got epochs [679,679] > 0 2024-09-17T18:37:19.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.801+0000 7fec2dffb640 3 client.64562.objecter handle_osd_map decoding full epoch 679 2024-09-17T18:37:19.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.801+0000 7fec2dffb640 20 client.64562.objecter dump_active .. 0 homeless 2024-09-17T18:37:19.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.801+0000 7fec3d4d9640 1 --2- 172.21.15.121:0/2945964339 >> v2:172.21.15.173:6812/2423786037 conn(0x7fec1c0401c0 0x7fec1c042680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:19.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.801+0000 7fec3d4d9640 10 monclient: get_auth_request con 0x7fec1c0401c0 auth_method 0 2024-09-17T18:37:19.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.803+0000 7fec3d4d9640 1 --2- 172.21.15.121:0/2945964339 >> v2:172.21.15.173:6812/2423786037 conn(0x7fec1c0401c0 0x7fec1c042680 secure :-1 s=READY pgs=4088 cs=0 l=1 rev1=1 crypto rx=0x7fec28007c90 tx=0x7fec28007be0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:19.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.803+0000 7fec2dffb640 10 client.64562.objecter ms_handle_connect 0x7fec1c0401c0 2024-09-17T18:37:19.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.805+0000 7fec2dffb640 1 -- 172.21.15.121:0/2945964339 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fec380a6df0 con 0x7fec38142590 2024-09-17T18:37:19.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.805+0000 7fec2dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:19.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:19.805+0000 7fec2dffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:37:20.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:20.046+0000 7fec3f73d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:37:20.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:20.047+0000 7fec3f73d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:37:20.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:20.047+0000 7fec3f73d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:37:20.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:20.047+0000 7fec3f73d640 1 -- 172.21.15.121:0/2945964339 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fec380749b0 con 0x7fec38142590 2024-09-17T18:37:20.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:20.047+0000 7fec2dffb640 1 -- 172.21.15.121:0/2945964339 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7fec300526e0 con 0x7fec38142590 2024-09-17T18:37:20.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:20.047+0000 7fec2dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:37:20.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:20.047+0000 7fec2dffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:37:20.049 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":62,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":95,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:37:20.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:20.049+0000 7fec3f73d640 10 librados: watch_flush enter 2024-09-17T18:37:20.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:20.049+0000 7fec3f73d640 10 librados: watch_flush exit 2024-09-17T18:37:20.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:20.049+0000 7fec3f73d640 20 client.64562.objecter shutdown clearing up homeless session... 2024-09-17T18:37:20.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:20.049+0000 7fec3f73d640 10 client.64562.objecter successfully canceled tick 2024-09-17T18:37:20.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:20.049+0000 7fec3f73d640 1 -- 172.21.15.121:0/2945964339 >> v2:172.21.15.173:6812/2423786037 conn(0x7fec1c0401c0 msgr2=0x7fec1c042680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:20.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:20.049+0000 7fec3f73d640 1 --2- 172.21.15.121:0/2945964339 >> v2:172.21.15.173:6812/2423786037 conn(0x7fec1c0401c0 0x7fec1c042680 secure :-1 s=READY pgs=4088 cs=0 l=1 rev1=1 crypto rx=0x7fec28007c90 tx=0x7fec28007be0 comp rx=0 tx=0).stop 2024-09-17T18:37:20.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:20.049+0000 7fec3f73d640 10 monclient: shutdown 2024-09-17T18:37:20.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:20.049+0000 7fec3f73d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:20.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:20.049+0000 7fec3f73d640 1 -- 172.21.15.121:0/2945964339 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec38142590 msgr2=0x7fec3813eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:20.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:20.049+0000 7fec3f73d640 1 --2- 172.21.15.121:0/2945964339 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec38142590 0x7fec3813eaf0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fec3000cdf0 tx=0x7fec3000a710 comp rx=0 tx=0).stop 2024-09-17T18:37:20.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:20.049+0000 7fec3f73d640 1 -- 172.21.15.121:0/2945964339 shutdown_connections 2024-09-17T18:37:20.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:20.049+0000 7fec3f73d640 1 --2- 172.21.15.121:0/2945964339 >> v2:172.21.15.173:6812/2423786037 conn(0x7fec1c0401c0 0x7fec1c042680 unknown :-1 s=CLOSED pgs=4088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:20.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:20.049+0000 7fec3f73d640 1 --2- 172.21.15.121:0/2945964339 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec38142590 0x7fec3813eaf0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:20.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:20.049+0000 7fec3f73d640 1 --2- 172.21.15.121:0/2945964339 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec380a2540 0x7fec3813e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:20.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:20.049+0000 7fec3f73d640 1 --2- 172.21.15.121:0/2945964339 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec380a1a50 0x7fec381d1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:20.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:20.049+0000 7fec3f73d640 1 -- 172.21.15.121:0/2945964339 >> 172.21.15.121:0/2945964339 conn(0x7fec380a3b50 msgr2=0x7fec380a6420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:20.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:20.049+0000 7fec3f73d640 1 -- 172.21.15.121:0/2945964339 shutdown_connections 2024-09-17T18:37:20.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:20.049+0000 7fec3f73d640 1 -- 172.21.15.121:0/2945964339 wait complete. 2024-09-17T18:37:20.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:20.050+0000 7fec3f73d640 1 librados: shutdown 2024-09-17T18:37:23.063 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:37:23.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.166+0000 7fc78552b640 10 monclient: get_monmap_and_config 2024-09-17T18:37:23.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.166+0000 7fc78552b640 10 monclient: build_initial_monmap 2024-09-17T18:37:23.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.166+0000 7fc78552b640 10 monclient: monmap: 2024-09-17T18:37:23.168 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:23.168 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:23.168 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:37:23.166740+0000 2024-09-17T18:37:23.168 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:37:23.166740+0000 2024-09-17T18:37:23.168 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:23.168 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:23.168 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:37:23.168 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:37:23.168 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:37:23.168 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:37:23.168 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:37:23.169 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:23.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.166+0000 7fc78552b640 1 Processor -- start 2024-09-17T18:37:23.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.166+0000 7fc78552b640 1 -- start start 2024-09-17T18:37:23.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.166+0000 7fc78552b640 10 monclient: init 2024-09-17T18:37:23.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.167+0000 7fc78552b640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:23.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.167+0000 7fc78552b640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-17T18:37:23.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.167+0000 7fc78552b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc7800a5c00 addr v1:172.21.15.173:6789/0 2024-09-17T18:37:23.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.167+0000 7fc78552b640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc7800a5fe0 0x7fc7800a63f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:23.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.167+0000 7fc78552b640 10 monclient(hunting): picked mon.noname-a con 0x7fc7800a5fe0 addr v2:172.21.15.121:3300/0 2024-09-17T18:37:23.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.167+0000 7fc78552b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc78013dc00 0x7fc78013fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:23.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.167+0000 7fc78552b640 10 monclient(hunting): picked mon.noname-b con 0x7fc78013dc00 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:23.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.167+0000 7fc78552b640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7800a8a80 con 0x7fc7800a5c00 2024-09-17T18:37:23.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.167+0000 7fc78552b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:23.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.167+0000 7fc78552b640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fc7800a7110 con 0x7fc7800a5fe0 2024-09-17T18:37:23.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.167+0000 7fc78552b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:23.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.167+0000 7fc78552b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc7800a74d0 con 0x7fc78013dc00 2024-09-17T18:37:23.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.167+0000 7fc78552b640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:23.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.167+0000 7fc78552b640 10 monclient(hunting): authenticate will time out at 1846.883497s 2024-09-17T18:37:23.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.167+0000 7fc77f7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc78013dc00 0x7fc78013fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:23.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.167+0000 7fc777fff640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc7800a5fe0 0x7fc7800a63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:23.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.167+0000 7fc777fff640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc7800a5fe0 0x7fc7800a63f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51122/0 (socket says 172.21.15.121:51122) 2024-09-17T18:37:23.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.167+0000 7fc777fff640 1 -- 172.21.15.121:0/3370335154 learned_addr learned my addr 172.21.15.121:0/3370335154 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:37:23.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.167+0000 7fc77f7fe640 10 monclient(hunting): get_auth_request con 0x7fc78013dc00 auth_method 0 2024-09-17T18:37:23.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.167+0000 7fc77f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:23.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.167+0000 7fc77f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:23.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.167+0000 7fc77f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:23.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.167+0000 7fc777fff640 10 monclient(hunting): get_auth_request con 0x7fc7800a5fe0 auth_method 0 2024-09-17T18:37:23.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.167+0000 7fc777fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:23.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.167+0000 7fc777fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:23.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.167+0000 7fc777fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:23.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.167+0000 7fc777fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:23.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.167+0000 7fc777fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:23.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.167+0000 7fc777fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:23.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.167+0000 7fc77f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:23.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.167+0000 7fc77f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:23.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.167+0000 7fc77f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:23.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.167+0000 7fc777fff640 10 monclient(hunting): handle_auth_done global_id 114099 payload 274 2024-09-17T18:37:23.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.167+0000 7fc777fff640 1 -- 172.21.15.121:0/3370335154 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc78013dc00 msgr2=0x7fc78013fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:23.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.167+0000 7fc777fff640 1 --2- 172.21.15.121:0/3370335154 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc78013dc00 0x7fc78013fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:23.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.167+0000 7fc777fff640 1 -- 172.21.15.121:0/3370335154 >> v1:172.21.15.173:6789/0 conn(0x7fc7800a5c00 legacy=0x7fc78009d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:23.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.167+0000 7fc777fff640 10 monclient: _finish_hunting 0 2024-09-17T18:37:23.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.167+0000 7fc777fff640 1 monclient: found mon.noname-a 2024-09-17T18:37:23.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.167+0000 7fc777fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:23.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.167+0000 7fc777fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:37:23.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.167+0000 7fc777fff640 1 -- 172.21.15.121:0/3370335154 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc78009cb90 con 0x7fc7800a5fe0 2024-09-17T18:37:23.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.167+0000 7fc77f7fe640 1 --2- 172.21.15.121:0/3370335154 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc78013dc00 0x7fc78013fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:37:23.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.170+0000 7fc777fff640 1 -- 172.21.15.121:0/3370335154 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7800a5fe0 msgr2=0x7fc7800a63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:23.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.170+0000 7fc777fff640 1 --2- 172.21.15.121:0/3370335154 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7800a5fe0 0x7fc7800a63f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc768009870 tx=0x7fc76802a0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7118425f9afbca5b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:23.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.170+0000 7fc777fff640 1 -- 172.21.15.121:0/3370335154 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7800a5fe0 msgr2=0x7fc7800a63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-17T18:37:23.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.170+0000 7fc777fff640 1 -- 172.21.15.121:0/3370335154 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7800a5fe0 msgr2=0x7fc7800a63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:37:23.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.170+0000 7fc777fff640 1 --2- 172.21.15.121:0/3370335154 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7800a5fe0 0x7fc7800a63f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc768009870 tx=0x7fc76802a0a0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:37:23.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.170+0000 7fc777fff640 1 --2- 172.21.15.121:0/3370335154 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7800a5fe0 0x7fc7800a63f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc768009870 tx=0x7fc76802a0a0 comp rx=0 tx=0).stop 2024-09-17T18:37:23.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.170+0000 7fc77e7fc640 10 monclient: ms_handle_reset current mon [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:37:23.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.170+0000 7fc77e7fc640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:23.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.170+0000 7fc77e7fc640 1 -- 172.21.15.121:0/3370335154 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7800a5fe0 msgr2=0x7fc7800a63f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:37:23.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.170+0000 7fc77e7fc640 1 --2- 172.21.15.121:0/3370335154 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7800a5fe0 0x7fc7800a63f0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:23.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.170+0000 7fc77e7fc640 10 monclient: _add_conns ranks=[0,4] 2024-09-17T18:37:23.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.170+0000 7fc77e7fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc76c003500 addr v1:172.21.15.121:6789/0 2024-09-17T18:37:23.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.170+0000 7fc77e7fc640 1 --2- 172.21.15.121:0/3370335154 >> v2:172.21.15.173:3300/0 conn(0x7fc76c007490 0x7fc76c009950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:23.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.170+0000 7fc77e7fc640 10 monclient(hunting): picked mon.noname-c con 0x7fc76c007490 addr v2:172.21.15.173:3300/0 2024-09-17T18:37:23.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.170+0000 7fc77e7fc640 1 -- 172.21.15.121:0/3370335154 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc76c00a220 con 0x7fc76c003500 2024-09-17T18:37:23.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.170+0000 7fc77e7fc640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:23.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.170+0000 7fc77e7fc640 1 -- 172.21.15.121:0/3370335154 --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fc76c00b420 con 0x7fc76c007490 2024-09-17T18:37:23.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.170+0000 7fc77e7fc640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:23.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.170+0000 7fc77e7fc640 10 monclient(hunting): ms_handle_reset stray mon [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:37:23.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.171+0000 7fc777fff640 1 --2- 172.21.15.121:0/3370335154 >> v2:172.21.15.173:3300/0 conn(0x7fc76c007490 0x7fc76c009950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:23.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.171+0000 7fc777fff640 10 monclient(hunting): get_auth_request con 0x7fc76c007490 auth_method 0 2024-09-17T18:37:23.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.171+0000 7fc777fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:23.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.171+0000 7fc777fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:23.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.171+0000 7fc777fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:37:23.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.171+0000 7fc777fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:23.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.171+0000 7fc777fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:23.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.171+0000 7fc777fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:37:23.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.171+0000 7fc777fff640 10 monclient(hunting): handle_auth_done global_id 114099 payload 293 2024-09-17T18:37:23.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.171+0000 7fc777fff640 1 -- 172.21.15.121:0/3370335154 >> v1:172.21.15.121:6789/0 conn(0x7fc76c003500 legacy=0x7fc76c005980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:23.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.171+0000 7fc777fff640 10 monclient: _finish_hunting 0 2024-09-17T18:37:23.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.171+0000 7fc777fff640 1 monclient: found mon.noname-c 2024-09-17T18:37:23.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.171+0000 7fc777fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:23.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.171+0000 7fc777fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.173:3300/0 2024-09-17T18:37:23.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.171+0000 7fc777fff640 1 -- 172.21.15.121:0/3370335154 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc76c00b690 con 0x7fc76c007490 2024-09-17T18:37:23.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.173+0000 7fc777fff640 1 -- 172.21.15.121:0/3370335154 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc76c007490 msgr2=0x7fc76c009950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:23.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.173+0000 7fc777fff640 1 --2- 172.21.15.121:0/3370335154 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc76c007490 0x7fc76c009950 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc78006a3f0 tx=0x7fc768009240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23d2f812cfedd4e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:23.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.174+0000 7fc77e7fc640 1 -- 172.21.15.121:0/3370335154 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc768037070 con 0x7fc76c007490 2024-09-17T18:37:23.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.174+0000 7fc77e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:23.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.174+0000 7fc77e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:37:23.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.174+0000 7fc77e7fc640 10 monclient: dump: 2024-09-17T18:37:23.185 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:23.185 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:23.185 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.185 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.185 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:23.185 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:23.185 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:23.185 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:23.185 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:23.186 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:23.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.174+0000 7fc77e7fc640 10 monclient: _finish_auth 0 2024-09-17T18:37:23.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.174+0000 7fc77e7fc640 10 monclient: _check_auth_tickets 2024-09-17T18:37:23.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.174+0000 7fc77e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:23.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.174+0000 7fc77e7fc640 1 -- 172.21.15.121:0/3370335154 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc768003610 con 0x7fc76c007490 2024-09-17T18:37:23.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.174+0000 7fc78552b640 5 monclient: authenticate success, global_id 114099 2024-09-17T18:37:23.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.174+0000 7fc78552b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:37:23.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.174+0000 7fc77e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:23.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.174+0000 7fc78552b640 10 monclient: get_monmap_and_config success 2024-09-17T18:37:23.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.174+0000 7fc77e7fc640 1 -- 172.21.15.121:0/3370335154 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc768003a00 con 0x7fc76c007490 2024-09-17T18:37:23.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.174+0000 7fc77e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:23.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.174+0000 7fc78552b640 4 set_mon_vals no callback set 2024-09-17T18:37:23.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.174+0000 7fc77e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:37:23.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.174+0000 7fc77e7fc640 10 monclient: dump: 2024-09-17T18:37:23.186 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:23.187 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:23.187 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.187 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.187 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:23.187 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:23.187 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:23.187 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:23.187 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:23.187 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:23.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.174+0000 7fc78552b640 10 monclient: shutdown 2024-09-17T18:37:23.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.174+0000 7fc78552b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:23.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.174+0000 7fc78552b640 1 -- 172.21.15.121:0/3370335154 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc76c007490 msgr2=0x7fc76c009950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:23.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.174+0000 7fc78552b640 1 --2- 172.21.15.121:0/3370335154 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc76c007490 0x7fc76c009950 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc78006a3f0 tx=0x7fc768009240 comp rx=0 tx=0).stop 2024-09-17T18:37:23.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.174+0000 7fc77f7fe640 1 -- 172.21.15.121:0/3370335154 reap_dead start 2024-09-17T18:37:23.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.174+0000 7fc78552b640 1 -- 172.21.15.121:0/3370335154 shutdown_connections 2024-09-17T18:37:23.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.174+0000 7fc78552b640 1 --2- 172.21.15.121:0/3370335154 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc76c007490 0x7fc76c009950 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:23.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.174+0000 7fc78552b640 1 --2- 172.21.15.121:0/3370335154 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7800a5fe0 0x7fc7800a63f0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:23.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.174+0000 7fc78552b640 1 -- 172.21.15.121:0/3370335154 >> 172.21.15.121:0/3370335154 conn(0x7fc7800ab9b0 msgr2=0x7fc7800a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:23.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.174+0000 7fc78552b640 1 -- 172.21.15.121:0/3370335154 shutdown_connections 2024-09-17T18:37:23.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.174+0000 7fc78552b640 1 -- 172.21.15.121:0/3370335154 wait complete. 2024-09-17T18:37:23.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.175+0000 7fc78552b640 10 monclient: build_initial_monmap 2024-09-17T18:37:23.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.175+0000 7fc78552b640 10 monclient: monmap: 2024-09-17T18:37:23.188 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:23.188 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:37:23.188 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:37:23.188 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:37:23.188 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:23.188 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:23.188 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:37:23.188 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:37:23.188 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:37:23.189 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:23.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.175+0000 7fc78552b640 1 librados: starting msgr at 2024-09-17T18:37:23.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.175+0000 7fc78552b640 1 librados: starting objecter 2024-09-17T18:37:23.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.175+0000 7fc78552b640 1 Processor -- start 2024-09-17T18:37:23.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.175+0000 7fc78552b640 1 -- start start 2024-09-17T18:37:23.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.175+0000 7fc78552b640 1 librados: setting wanted keys 2024-09-17T18:37:23.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.175+0000 7fc78552b640 1 librados: calling monclient init 2024-09-17T18:37:23.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.175+0000 7fc78552b640 10 monclient: init 2024-09-17T18:37:23.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc78552b640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:23.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc78552b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T18:37:23.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc78552b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7800a5fe0 0x7fc780138230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:23.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc78552b640 10 monclient(hunting): picked mon.noname-c con 0x7fc7800a5fe0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:23.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc78552b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc76c007490 0x7fc780138770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:23.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc78552b640 10 monclient(hunting): picked mon.noname-b con 0x7fc76c007490 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:37:23.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc78552b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc78013b870 0x7fc78013bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:23.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc78552b640 10 monclient(hunting): picked mon.noname-a con 0x7fc78013b870 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:37:23.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc78552b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:23.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc78552b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc780070550 con 0x7fc78013b870 2024-09-17T18:37:23.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc78552b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:23.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc78552b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc7800703d0 con 0x7fc76c007490 2024-09-17T18:37:23.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc78552b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:23.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc78552b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc780140530 con 0x7fc7800a5fe0 2024-09-17T18:37:23.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc78552b640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:23.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc78552b640 10 monclient(hunting): authenticate will time out at 1846.892096s 2024-09-17T18:37:23.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc77effd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7800a5fe0 0x7fc780138230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:23.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc77f7fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc78013b870 0x7fc78013bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:23.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc77effd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7800a5fe0 0x7fc780138230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:36666/0 (socket says 172.21.15.121:36666) 2024-09-17T18:37:23.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc77effd640 1 -- 172.21.15.121:0/1452989506 learned_addr learned my addr 172.21.15.121:0/1452989506 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:37:23.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc77effd640 10 monclient(hunting): get_auth_request con 0x7fc7800a5fe0 auth_method 0 2024-09-17T18:37:23.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc77effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:23.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc77effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:23.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc77effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:23.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc77f7fe640 10 monclient(hunting): get_auth_request con 0x7fc78013b870 auth_method 0 2024-09-17T18:37:23.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc77f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:23.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc77f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:23.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc77f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:23.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc777fff640 1 --2- 172.21.15.121:0/1452989506 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc76c007490 0x7fc780138770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:23.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc77effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:23.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc77effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:23.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc77effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:23.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc77f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:23.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc77f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:23.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc77f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:23.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc777fff640 10 monclient(hunting): get_auth_request con 0x7fc76c007490 auth_method 0 2024-09-17T18:37:23.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc777fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:23.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc777fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:23.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc777fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:23.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc777fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:23.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc777fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:23.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc777fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:23.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc77f7fe640 10 monclient(hunting): handle_auth_done global_id 114102 payload 824 2024-09-17T18:37:23.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc77f7fe640 1 -- 172.21.15.121:0/1452989506 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7800a5fe0 msgr2=0x7fc780138230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:23.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc77f7fe640 1 --2- 172.21.15.121:0/1452989506 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7800a5fe0 0x7fc780138230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:23.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc77f7fe640 1 -- 172.21.15.121:0/1452989506 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc76c007490 msgr2=0x7fc780138770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:23.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc77f7fe640 1 --2- 172.21.15.121:0/1452989506 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc76c007490 0x7fc780138770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:23.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc77f7fe640 10 monclient: _finish_hunting 0 2024-09-17T18:37:23.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc77f7fe640 1 monclient: found mon.noname-a 2024-09-17T18:37:23.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc77f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:23.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc77f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:37:23.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc77f7fe640 1 -- 172.21.15.121:0/1452989506 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc780122c90 con 0x7fc78013b870 2024-09-17T18:37:23.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.176+0000 7fc77effd640 1 --2- 172.21.15.121:0/1452989506 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7800a5fe0 0x7fc780138230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:37:23.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.178+0000 7fc77f7fe640 1 -- 172.21.15.121:0/1452989506 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc78013b870 msgr2=0x7fc78013bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:23.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.178+0000 7fc77f7fe640 1 --2- 172.21.15.121:0/1452989506 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc78013b870 0x7fc78013bcf0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc7800a8550 tx=0x7fc77800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:23.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.178+0000 7fc7777fe640 10 client.?.objecter ms_handle_connect 0x7fc78013b870 2024-09-17T18:37:23.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.178+0000 7fc7777fe640 10 client.?.objecter resend_mon_ops 2024-09-17T18:37:23.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.178+0000 7fc7777fe640 1 -- 172.21.15.121:0/1452989506 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc778018020 con 0x7fc78013b870 2024-09-17T18:37:23.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.178+0000 7fc7777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:23.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.178+0000 7fc7777fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:37:23.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.178+0000 7fc7777fe640 10 monclient: dump: 2024-09-17T18:37:23.193 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:23.193 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:23.193 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.193 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.193 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:23.193 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:23.193 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:23.194 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:23.194 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:23.194 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:23.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.178+0000 7fc7777fe640 10 monclient: _finish_auth 0 2024-09-17T18:37:23.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.178+0000 7fc7777fe640 10 monclient: _check_auth_tickets 2024-09-17T18:37:23.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.178+0000 7fc7777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:23.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.178+0000 7fc7777fe640 1 -- 172.21.15.121:0/1452989506 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7780146a0 con 0x7fc78013b870 2024-09-17T18:37:23.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.178+0000 7fc78552b640 5 monclient: authenticate success, global_id 114102 2024-09-17T18:37:23.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.178+0000 7fc7777fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:23.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.178+0000 7fc7777fe640 1 -- 172.21.15.121:0/1452989506 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc778014a90 con 0x7fc78013b870 2024-09-17T18:37:23.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.178+0000 7fc78552b640 10 monclient: _renew_subs 2024-09-17T18:37:23.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.178+0000 7fc78552b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:23.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.178+0000 7fc78552b640 1 -- 172.21.15.121:0/1452989506 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc780122f70 con 0x7fc78013b870 2024-09-17T18:37:23.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.178+0000 7fc77dffb640 4 set_mon_vals no callback set 2024-09-17T18:37:23.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.178+0000 7fc7777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:23.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.178+0000 7fc7777fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:37:23.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.178+0000 7fc7777fe640 10 monclient: dump: 2024-09-17T18:37:23.195 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:23.195 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:23.195 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.195 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.195 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:23.195 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:23.195 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:23.195 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:23.195 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:23.195 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:23.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.178+0000 7fc78552b640 10 client.114102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:37:23.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.178+0000 7fc78552b640 10 monclient: _renew_subs 2024-09-17T18:37:23.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.178+0000 7fc78552b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:23.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.178+0000 7fc78552b640 1 -- 172.21.15.121:0/1452989506 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc780123250 con 0x7fc78013b870 2024-09-17T18:37:23.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.178+0000 7fc78552b640 1 librados: init done 2024-09-17T18:37:23.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.179+0000 7fc78552b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:23.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.179+0000 7fc78552b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:23.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.179+0000 7fc78552b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:23.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.179+0000 7fc7777fe640 1 -- 172.21.15.121:0/1452989506 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fc778005d10 con 0x7fc78013b870 2024-09-17T18:37:23.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.179+0000 7fc78552b640 1 -- 172.21.15.121:0/1452989506 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7800a63f0 con 0x7fc78013b870 2024-09-17T18:37:23.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.180+0000 7fc7777fe640 1 --2- 172.21.15.121:0/1452989506 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc760040170 0x7fc760042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:23.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.180+0000 7fc7777fe640 1 -- 172.21.15.121:0/1452989506 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(679..679 src has 247..679) ==== 4685+0+0 (secure 0 0 0) 0x7fc778053de0 con 0x7fc78013b870 2024-09-17T18:37:23.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.180+0000 7fc7777fe640 10 client.114102.objecter ms_dispatch 0x7fc780000b90 osd_map(679..679 src has 247..679) 2024-09-17T18:37:23.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.180+0000 7fc7777fe640 3 client.114102.objecter handle_osd_map got epochs [679,679] > 0 2024-09-17T18:37:23.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.180+0000 7fc7777fe640 3 client.114102.objecter handle_osd_map decoding full epoch 679 2024-09-17T18:37:23.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.180+0000 7fc7777fe640 20 client.114102.objecter dump_active .. 0 homeless 2024-09-17T18:37:23.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.180+0000 7fc77effd640 1 --2- 172.21.15.121:0/1452989506 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc760040170 0x7fc760042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:23.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.180+0000 7fc77effd640 10 monclient: get_auth_request con 0x7fc760040170 auth_method 0 2024-09-17T18:37:23.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.183+0000 7fc7777fe640 10 client.114102.objecter ms_handle_connect 0x7fc760040170 2024-09-17T18:37:23.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.183+0000 7fc77effd640 1 --2- 172.21.15.121:0/1452989506 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc760040170 0x7fc760042630 secure :-1 s=READY pgs=4090 cs=0 l=1 rev1=1 crypto rx=0x7fc770004360 tx=0x7fc770007e70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:23.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.184+0000 7fc7777fe640 1 -- 172.21.15.121:0/1452989506 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc778022070 con 0x7fc78013b870 2024-09-17T18:37:23.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.184+0000 7fc7777fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:23.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.184+0000 7fc7777fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:37:23.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.426+0000 7fc78552b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:37:23.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.426+0000 7fc78552b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:37:23.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.426+0000 7fc78552b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:23.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.426+0000 7fc78552b640 1 -- 172.21.15.121:0/1452989506 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc780122a10 con 0x7fc78013b870 2024-09-17T18:37:23.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.426+0000 7fc7777fe640 1 -- 172.21.15.121:0/1452989506 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fc778010040 con 0x7fc78013b870 2024-09-17T18:37:23.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.426+0000 7fc7777fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:37:23.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.426+0000 7fc7777fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:37:23.428 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":66,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:37:23.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.428+0000 7fc78552b640 10 librados: watch_flush enter 2024-09-17T18:37:23.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.428+0000 7fc78552b640 10 librados: watch_flush exit 2024-09-17T18:37:23.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.428+0000 7fc78552b640 20 client.114102.objecter shutdown clearing up homeless session... 2024-09-17T18:37:23.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.428+0000 7fc78552b640 10 client.114102.objecter successfully canceled tick 2024-09-17T18:37:23.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.429+0000 7fc78552b640 1 -- 172.21.15.121:0/1452989506 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc760040170 msgr2=0x7fc760042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:23.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.429+0000 7fc78552b640 1 --2- 172.21.15.121:0/1452989506 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc760040170 0x7fc760042630 secure :-1 s=READY pgs=4090 cs=0 l=1 rev1=1 crypto rx=0x7fc770004360 tx=0x7fc770007e70 comp rx=0 tx=0).stop 2024-09-17T18:37:23.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.429+0000 7fc78552b640 10 monclient: shutdown 2024-09-17T18:37:23.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.429+0000 7fc78552b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:23.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.429+0000 7fc78552b640 1 -- 172.21.15.121:0/1452989506 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc78013b870 msgr2=0x7fc78013bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:23.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.429+0000 7fc78552b640 1 --2- 172.21.15.121:0/1452989506 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc78013b870 0x7fc78013bcf0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc7800a8550 tx=0x7fc77800c490 comp rx=0 tx=0).stop 2024-09-17T18:37:23.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.429+0000 7fc78552b640 1 -- 172.21.15.121:0/1452989506 shutdown_connections 2024-09-17T18:37:23.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.429+0000 7fc78552b640 1 --2- 172.21.15.121:0/1452989506 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc760040170 0x7fc760042630 unknown :-1 s=CLOSED pgs=4090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:23.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.429+0000 7fc78552b640 1 --2- 172.21.15.121:0/1452989506 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc76c007490 0x7fc780138770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:23.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.429+0000 7fc78552b640 1 --2- 172.21.15.121:0/1452989506 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc78013b870 0x7fc78013bcf0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:23.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.429+0000 7fc78552b640 1 --2- 172.21.15.121:0/1452989506 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7800a5fe0 0x7fc780138230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:23.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.429+0000 7fc78552b640 1 -- 172.21.15.121:0/1452989506 >> 172.21.15.121:0/1452989506 conn(0x7fc7800ab9b0 msgr2=0x7fc7800a90e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:23.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.429+0000 7fc78552b640 1 -- 172.21.15.121:0/1452989506 shutdown_connections 2024-09-17T18:37:23.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.429+0000 7fc78552b640 1 -- 172.21.15.121:0/1452989506 wait complete. 2024-09-17T18:37:23.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.429+0000 7fc78552b640 1 librados: shutdown 2024-09-17T18:37:23.441 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T18:37:23.441 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T18:37:23.441 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T18:37:23.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.537+0000 7f3523dd3640 10 monclient: get_monmap_and_config 2024-09-17T18:37:23.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.537+0000 7f3523dd3640 10 monclient: build_initial_monmap 2024-09-17T18:37:23.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.537+0000 7f3523dd3640 10 monclient: monmap: 2024-09-17T18:37:23.539 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:23.539 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:23.539 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:37:23.537788+0000 2024-09-17T18:37:23.539 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:37:23.537788+0000 2024-09-17T18:37:23.539 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:23.539 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:23.539 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:37:23.539 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:37:23.539 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:37:23.539 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:37:23.539 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:37:23.540 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:23.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.537+0000 7f3523dd3640 1 Processor -- start 2024-09-17T18:37:23.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.538+0000 7f3523dd3640 1 -- start start 2024-09-17T18:37:23.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.538+0000 7f3523dd3640 10 monclient: init 2024-09-17T18:37:23.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.538+0000 7f3523dd3640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:23.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.538+0000 7f3523dd3640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-17T18:37:23.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.538+0000 7f3523dd3640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f351c09e550 0x7f351c09e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:23.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.538+0000 7f3523dd3640 10 monclient(hunting): picked mon.noname-a con 0x7f351c09e550 addr v2:172.21.15.121:3300/0 2024-09-17T18:37:23.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.538+0000 7f3523dd3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f351c09ee70 addr v1:172.21.15.173:6789/0 2024-09-17T18:37:23.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.538+0000 7f3523dd3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f351c09f2c0 addr v1:172.21.15.121:6789/0 2024-09-17T18:37:23.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.538+0000 7f3523dd3640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f351c0a4460 con 0x7f351c09f2c0 2024-09-17T18:37:23.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.538+0000 7f3523dd3640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f351c0a2b60 con 0x7f351c09ee70 2024-09-17T18:37:23.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.538+0000 7f3523dd3640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:23.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.538+0000 7f3523dd3640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f351c0a4950 con 0x7f351c09e550 2024-09-17T18:37:23.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.538+0000 7f3523dd3640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:23.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.538+0000 7f3523dd3640 10 monclient(hunting): authenticate will time out at 1847.254539s 2024-09-17T18:37:23.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.538+0000 7f3521b6f640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f351c09e550 0x7f351c09e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:23.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.538+0000 7f3522370640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f351c09f2c0 0x7f351c13cf70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:47294/0 (socket says 172.21.15.121:47294) 2024-09-17T18:37:23.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.538+0000 7f3522370640 1 -- 172.21.15.121:0/2224678126 learned_addr learned my addr 172.21.15.121:0/2224678126 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:37:23.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.538+0000 7f3521b6f640 10 monclient(hunting): get_auth_request con 0x7f351c09e550 auth_method 0 2024-09-17T18:37:23.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.538+0000 7f3521b6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:23.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.538+0000 7f3521b6f640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:23.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.538+0000 7f3521b6f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:23.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.538+0000 7f3521b6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:23.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.538+0000 7f3521b6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:23.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.538+0000 7f3521b6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:23.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.538+0000 7f3521b6f640 10 monclient(hunting): handle_auth_done global_id 114105 payload 274 2024-09-17T18:37:23.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.538+0000 7f3521b6f640 1 -- 172.21.15.121:0/2224678126 >> v1:172.21.15.173:6789/0 conn(0x7f351c09ee70 legacy=0x7f351c139840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:23.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.538+0000 7f3521b6f640 1 -- 172.21.15.121:0/2224678126 >> v1:172.21.15.121:6789/0 conn(0x7f351c09f2c0 legacy=0x7f351c13cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:23.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.538+0000 7f3521b6f640 10 monclient: _finish_hunting 0 2024-09-17T18:37:23.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.538+0000 7f3521b6f640 1 monclient: found mon.noname-a 2024-09-17T18:37:23.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.538+0000 7f3521b6f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:23.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.538+0000 7f3521b6f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:37:23.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.538+0000 7f3521b6f640 1 -- 172.21.15.121:0/2224678126 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f351c09f840 con 0x7f351c09e550 2024-09-17T18:37:23.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.541+0000 7f3521b6f640 1 -- 172.21.15.121:0/2224678126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f351c09e550 msgr2=0x7f351c09e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:23.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.541+0000 7f3521b6f640 1 --2- 172.21.15.121:0/2224678126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f351c09e550 0x7f351c09e930 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f350c009d90 tx=0x7f350c02a490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5afcb2441c45f93b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:23.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.541+0000 7f3520b6d640 1 -- 172.21.15.121:0/2224678126 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f350c007a30 con 0x7f351c09e550 2024-09-17T18:37:23.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.541+0000 7f3520b6d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:23.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.541+0000 7f3520b6d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:37:23.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.541+0000 7f3520b6d640 10 monclient: dump: 2024-09-17T18:37:23.555 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:23.555 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:23.555 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.555 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.555 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:23.555 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:23.555 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:23.555 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:23.555 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:23.555 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:23.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.542+0000 7f3520b6d640 10 monclient: _finish_auth 0 2024-09-17T18:37:23.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.542+0000 7f3520b6d640 10 monclient: _check_auth_tickets 2024-09-17T18:37:23.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.542+0000 7f3520b6d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:23.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.542+0000 7f3520b6d640 1 -- 172.21.15.121:0/2224678126 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f350c007bd0 con 0x7f351c09e550 2024-09-17T18:37:23.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.542+0000 7f3523dd3640 5 monclient: authenticate success, global_id 114105 2024-09-17T18:37:23.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.542+0000 7f3523dd3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:37:23.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.542+0000 7f3520b6d640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:23.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.542+0000 7f3523dd3640 10 monclient: get_monmap_and_config success 2024-09-17T18:37:23.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.542+0000 7f3520b6d640 1 -- 172.21.15.121:0/2224678126 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f350c031400 con 0x7f351c09e550 2024-09-17T18:37:23.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.542+0000 7f3520b6d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:23.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.542+0000 7f3523dd3640 4 set_mon_vals no callback set 2024-09-17T18:37:23.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.542+0000 7f3520b6d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:37:23.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.542+0000 7f3520b6d640 10 monclient: dump: 2024-09-17T18:37:23.556 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:23.556 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:23.556 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.557 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.557 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:23.557 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:23.557 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:23.557 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:23.557 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:23.557 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:23.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.542+0000 7f3523dd3640 10 monclient: shutdown 2024-09-17T18:37:23.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.542+0000 7f3523dd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:23.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.542+0000 7f3523dd3640 1 -- 172.21.15.121:0/2224678126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f351c09e550 msgr2=0x7f351c09e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:23.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.542+0000 7f3523dd3640 1 --2- 172.21.15.121:0/2224678126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f351c09e550 0x7f351c09e930 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f350c009d90 tx=0x7f350c02a490 comp rx=0 tx=0).stop 2024-09-17T18:37:23.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.542+0000 7f3523dd3640 1 -- 172.21.15.121:0/2224678126 shutdown_connections 2024-09-17T18:37:23.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.542+0000 7f3523dd3640 1 --2- 172.21.15.121:0/2224678126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f351c09e550 0x7f351c09e930 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:23.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.542+0000 7f3523dd3640 1 -- 172.21.15.121:0/2224678126 >> 172.21.15.121:0/2224678126 conn(0x7f351c0a7830 msgr2=0x7f351c0a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:23.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.542+0000 7f3523dd3640 1 -- 172.21.15.121:0/2224678126 shutdown_connections 2024-09-17T18:37:23.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.542+0000 7f3523dd3640 1 -- 172.21.15.121:0/2224678126 wait complete. 2024-09-17T18:37:23.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.543+0000 7f3523dd3640 10 monclient: build_initial_monmap 2024-09-17T18:37:23.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.543+0000 7f3523dd3640 10 monclient: monmap: 2024-09-17T18:37:23.558 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:23.558 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:37:23.558 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:37:23.558 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:37:23.558 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:23.558 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:23.558 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:37:23.558 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:37:23.558 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:37:23.558 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:23.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.543+0000 7f3523dd3640 1 librados: starting msgr at 2024-09-17T18:37:23.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.543+0000 7f3523dd3640 1 librados: starting objecter 2024-09-17T18:37:23.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.543+0000 7f3523dd3640 1 Processor -- start 2024-09-17T18:37:23.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.543+0000 7f3523dd3640 1 -- start start 2024-09-17T18:37:23.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.543+0000 7f3523dd3640 1 librados: setting wanted keys 2024-09-17T18:37:23.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.543+0000 7f3523dd3640 1 librados: calling monclient init 2024-09-17T18:37:23.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.543+0000 7f3523dd3640 10 monclient: init 2024-09-17T18:37:23.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.543+0000 7f3523dd3640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:23.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.543+0000 7f3523dd3640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T18:37:23.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.543+0000 7f3523dd3640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f351c09e550 0x7f351c128c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:23.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.543+0000 7f3523dd3640 10 monclient(hunting): picked mon.noname-b con 0x7f351c09e550 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:37:23.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.543+0000 7f3523dd3640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f351c09ee70 0x7f351c126ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:23.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.543+0000 7f3523dd3640 10 monclient(hunting): picked mon.noname-a con 0x7f351c09ee70 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:37:23.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.543+0000 7f3523dd3640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f351c09f2c0 0x7f351c13fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:23.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.543+0000 7f3523dd3640 10 monclient(hunting): picked mon.noname-c con 0x7f351c09f2c0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:23.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.544+0000 7f3523dd3640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:23.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.544+0000 7f3523dd3640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f351c09de60 con 0x7f351c09ee70 2024-09-17T18:37:23.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.544+0000 7f3523dd3640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:23.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.544+0000 7f3523dd3640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f351c09dce0 con 0x7f351c09e550 2024-09-17T18:37:23.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.544+0000 7f3523dd3640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:23.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.544+0000 7f3523dd3640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f351c0747e0 con 0x7f351c09f2c0 2024-09-17T18:37:23.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.544+0000 7f3523dd3640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:23.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.544+0000 7f3523dd3640 10 monclient(hunting): authenticate will time out at 1847.260018s 2024-09-17T18:37:23.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.544+0000 7f352136e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f351c09ee70 0x7f351c126ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:23.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.544+0000 7f352136e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f351c09ee70 0x7f351c126ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51146/0 (socket says 172.21.15.121:51146) 2024-09-17T18:37:23.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.544+0000 7f352136e640 1 -- 172.21.15.121:0/1905323667 learned_addr learned my addr 172.21.15.121:0/1905323667 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:37:23.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.544+0000 7f352136e640 10 monclient(hunting): get_auth_request con 0x7f351c09ee70 auth_method 0 2024-09-17T18:37:23.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.544+0000 7f352136e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:23.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.544+0000 7f352136e640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:23.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.544+0000 7f352136e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:23.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.544+0000 7f3522370640 1 --2- 172.21.15.121:0/1905323667 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f351c09f2c0 0x7f351c13fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:23.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.544+0000 7f3522370640 10 monclient(hunting): get_auth_request con 0x7f351c09f2c0 auth_method 0 2024-09-17T18:37:23.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.544+0000 7f3522370640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:23.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.544+0000 7f3522370640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:23.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.544+0000 7f3522370640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:23.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.544+0000 7f352136e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:23.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.544+0000 7f352136e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:23.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.544+0000 7f352136e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:23.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.544+0000 7f3522370640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:23.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.544+0000 7f3522370640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:23.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.544+0000 7f3522370640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:23.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.544+0000 7f3521b6f640 1 --2- 172.21.15.121:0/1905323667 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f351c09e550 0x7f351c128c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:23.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.544+0000 7f352136e640 10 monclient(hunting): handle_auth_done global_id 114108 payload 824 2024-09-17T18:37:23.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.544+0000 7f352136e640 1 -- 172.21.15.121:0/1905323667 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f351c09f2c0 msgr2=0x7f351c13fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:23.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.544+0000 7f352136e640 1 --2- 172.21.15.121:0/1905323667 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f351c09f2c0 0x7f351c13fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:23.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.544+0000 7f352136e640 1 -- 172.21.15.121:0/1905323667 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f351c09e550 msgr2=0x7f351c128c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:23.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.544+0000 7f352136e640 1 --2- 172.21.15.121:0/1905323667 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f351c09e550 0x7f351c128c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:23.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.544+0000 7f352136e640 10 monclient: _finish_hunting 0 2024-09-17T18:37:23.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.544+0000 7f352136e640 1 monclient: found mon.noname-a 2024-09-17T18:37:23.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.544+0000 7f352136e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:23.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.544+0000 7f352136e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:37:23.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.544+0000 7f352136e640 1 -- 172.21.15.121:0/1905323667 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f351c1280c0 con 0x7f351c09ee70 2024-09-17T18:37:23.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.544+0000 7f3521b6f640 10 monclient: get_auth_request con 0x7f351c09e550 auth_method 0 2024-09-17T18:37:23.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.544+0000 7f3521b6f640 1 --2- 172.21.15.121:0/1905323667 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f351c09e550 0x7f351c128c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T18:37:23.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.546+0000 7f352136e640 1 -- 172.21.15.121:0/1905323667 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f351c09ee70 msgr2=0x7f351c126ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:23.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.546+0000 7f352136e640 1 --2- 172.21.15.121:0/1905323667 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f351c09ee70 0x7f351c126ae0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f351000c460 tx=0x7f351000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:23.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.546+0000 7f351a7fc640 10 client.?.objecter ms_handle_connect 0x7f351c09ee70 2024-09-17T18:37:23.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.546+0000 7f351a7fc640 10 client.?.objecter resend_mon_ops 2024-09-17T18:37:23.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.546+0000 7f351a7fc640 1 -- 172.21.15.121:0/1905323667 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3510016020 con 0x7f351c09ee70 2024-09-17T18:37:23.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.546+0000 7f351a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:23.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.546+0000 7f351a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:37:23.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.546+0000 7f351a7fc640 10 monclient: dump: 2024-09-17T18:37:23.563 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:23.563 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:23.563 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.563 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.563 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:23.563 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:23.563 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:23.564 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:23.564 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:23.564 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:23.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.546+0000 7f351a7fc640 10 monclient: _finish_auth 0 2024-09-17T18:37:23.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.546+0000 7f351a7fc640 10 monclient: _check_auth_tickets 2024-09-17T18:37:23.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.546+0000 7f351a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:23.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.546+0000 7f3523dd3640 5 monclient: authenticate success, global_id 114108 2024-09-17T18:37:23.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.546+0000 7f3523dd3640 10 monclient: _renew_subs 2024-09-17T18:37:23.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.546+0000 7f351a7fc640 1 -- 172.21.15.121:0/1905323667 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3510012a80 con 0x7f351c09ee70 2024-09-17T18:37:23.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.546+0000 7f3523dd3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:23.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.546+0000 7f3523dd3640 1 -- 172.21.15.121:0/1905323667 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f351c127a90 con 0x7f351c09ee70 2024-09-17T18:37:23.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.546+0000 7f351a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:23.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.546+0000 7f351a7fc640 1 -- 172.21.15.121:0/1905323667 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3510011380 con 0x7f351c09ee70 2024-09-17T18:37:23.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.546+0000 7f351a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:23.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.546+0000 7f351a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:37:23.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.546+0000 7f351a7fc640 10 monclient: dump: 2024-09-17T18:37:23.565 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:23.565 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:23.565 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.565 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.565 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:23.565 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:23.565 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:23.565 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:23.565 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:23.565 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:23.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.546+0000 7f3523dd3640 10 client.114108.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:37:23.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.546+0000 7f3523dd3640 10 monclient: _renew_subs 2024-09-17T18:37:23.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.546+0000 7f351bfff640 4 set_mon_vals no callback set 2024-09-17T18:37:23.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.546+0000 7f3523dd3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:23.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.546+0000 7f3523dd3640 1 -- 172.21.15.121:0/1905323667 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f351c129b80 con 0x7f351c09ee70 2024-09-17T18:37:23.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.546+0000 7f3523dd3640 1 librados: init done 2024-09-17T18:37:23.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.546+0000 7f352136e640 1 -- 172.21.15.121:0/1905323667 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f351c09ee70 msgr2=0x7f351c126ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T18:37:23.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.546+0000 7f352136e640 1 -- 172.21.15.121:0/1905323667 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f351c09ee70 msgr2=0x7f351c126ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:37:23.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.546+0000 7f352136e640 1 --2- 172.21.15.121:0/1905323667 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f351c09ee70 0x7f351c126ae0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f351000c460 tx=0x7f351000c920 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:37:23.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.546+0000 7f352136e640 1 --2- 172.21.15.121:0/1905323667 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f351c09ee70 0x7f351c126ae0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f351000c460 tx=0x7f351000c920 comp rx=0 tx=0).stop 2024-09-17T18:37:23.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f351a7fc640 10 monclient: ms_handle_reset current mon [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:37:23.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f351a7fc640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:23.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f351a7fc640 1 -- 172.21.15.121:0/1905323667 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f351c09ee70 msgr2=0x7f351c126ae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:37:23.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f351a7fc640 1 --2- 172.21.15.121:0/1905323667 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f351c09ee70 0x7f351c126ae0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:23.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f351a7fc640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T18:37:23.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f351a7fc640 1 --2- 172.21.15.121:0/1905323667 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f351c09f2c0 0x7f3508007230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:23.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f351a7fc640 10 monclient(hunting): picked mon.c con 0x7f351c09f2c0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:23.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f351a7fc640 1 --2- 172.21.15.121:0/1905323667 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3508007c00 0x7f350800a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:23.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f351a7fc640 10 monclient(hunting): picked mon.a con 0x7f3508007c00 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:37:23.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f351a7fc640 1 --2- 172.21.15.121:0/1905323667 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f351c09e550 0x7f350800cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:23.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f351a7fc640 10 monclient(hunting): picked mon.b con 0x7f351c09e550 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:37:23.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f351a7fc640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:23.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f351a7fc640 1 -- 172.21.15.121:0/1905323667 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f350800d510 con 0x7f3508007c00 2024-09-17T18:37:23.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f351a7fc640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:23.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f351a7fc640 1 -- 172.21.15.121:0/1905323667 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f350800d690 con 0x7f351c09e550 2024-09-17T18:37:23.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f351a7fc640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:23.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f351a7fc640 1 -- 172.21.15.121:0/1905323667 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f350800d810 con 0x7f351c09f2c0 2024-09-17T18:37:23.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f351a7fc640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:23.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f3521b6f640 1 --2- 172.21.15.121:0/1905323667 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f351c09f2c0 0x7f3508007230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:23.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f352136e640 1 --2- 172.21.15.121:0/1905323667 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3508007c00 0x7f350800a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:23.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f352136e640 10 monclient(hunting): get_auth_request con 0x7f3508007c00 auth_method 0 2024-09-17T18:37:23.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f352136e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:23.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f352136e640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:23.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f352136e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:37:23.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f3521b6f640 10 monclient(hunting): get_auth_request con 0x7f351c09f2c0 auth_method 0 2024-09-17T18:37:23.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f3521b6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:23.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f3521b6f640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:23.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f3521b6f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:37:23.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f352136e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:23.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f352136e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:23.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f352136e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:37:23.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f3521b6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:23.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f3521b6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:23.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f3521b6f640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:37:23.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f3523dd3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:23.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f3522370640 1 --2- 172.21.15.121:0/1905323667 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f351c09e550 0x7f350800cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:23.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f3523dd3640 1 --2- 172.21.15.121:0/1905323667 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f351c126ac0 0x7f351c128c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:23.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f3523dd3640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:23.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f3523dd3640 1 -- 172.21.15.121:0/1905323667 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f351c126ea0 con 0x7f351c126ac0 2024-09-17T18:37:23.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f3522370640 10 monclient(hunting): get_auth_request con 0x7f351c09e550 auth_method 0 2024-09-17T18:37:23.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f3522370640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:23.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f3522370640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:23.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f3522370640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:37:23.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f352136e640 10 monclient(hunting): handle_auth_done global_id 114108 payload 293 2024-09-17T18:37:23.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f352136e640 1 -- 172.21.15.121:0/1905323667 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f351c09f2c0 msgr2=0x7f3508007230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:23.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f352136e640 1 --2- 172.21.15.121:0/1905323667 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f351c09f2c0 0x7f3508007230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:23.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f352136e640 1 -- 172.21.15.121:0/1905323667 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f351c09e550 msgr2=0x7f350800cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:23.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f352136e640 1 --2- 172.21.15.121:0/1905323667 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f351c09e550 0x7f350800cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:23.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f352136e640 10 monclient: _finish_hunting 0 2024-09-17T18:37:23.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f352136e640 1 monclient: found mon.a 2024-09-17T18:37:23.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f352136e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:23.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f352136e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:23.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f352136e640 1 -- 172.21.15.121:0/1905323667 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7f350800db00 con 0x7f3508007c00 2024-09-17T18:37:23.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f352136e640 10 monclient: _finish_auth 0 2024-09-17T18:37:23.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f352136e640 10 monclient: _check_auth_tickets 2024-09-17T18:37:23.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f352136e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:23.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f3522370640 1 --2- 172.21.15.121:0/1905323667 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f351c09e550 0x7f350800cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T18:37:23.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f3521b6f640 1 --2- 172.21.15.121:0/1905323667 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f351c126ac0 0x7f351c128c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:23.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f3521b6f640 10 monclient: get_auth_request con 0x7f351c126ac0 auth_method 0 2024-09-17T18:37:23.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f3521b6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:23.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f3521b6f640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:23.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f3521b6f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:23.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f3521b6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:23.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f3521b6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:23.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.547+0000 7f3521b6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:23.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.548+0000 7f3521b6f640 10 monclient(hunting): handle_auth_done global_id 114111 payload 824 2024-09-17T18:37:23.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.548+0000 7f3521b6f640 1 -- 172.21.15.121:0/1905323667 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f351c09e930 con 0x7f351c126ac0 2024-09-17T18:37:23.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.550+0000 7f352136e640 1 -- 172.21.15.121:0/1905323667 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3508007c00 msgr2=0x7f350800a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:23.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.550+0000 7f351a7fc640 10 client.114108.objecter ms_handle_connect 0x7f3508007c00 2024-09-17T18:37:23.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.550+0000 7f351a7fc640 10 client.114108.objecter resend_mon_ops 2024-09-17T18:37:23.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.550+0000 7f352136e640 1 --2- 172.21.15.121:0/1905323667 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3508007c00 0x7f350800a090 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3510017000 tx=0x7f3510007ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:23.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.550+0000 7f351a7fc640 1 -- 172.21.15.121:0/1905323667 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3510011770 con 0x7f3508007c00 2024-09-17T18:37:23.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.550+0000 7f351a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:23.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.550+0000 7f351a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:37:23.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.550+0000 7f351a7fc640 10 monclient: dump: 2024-09-17T18:37:23.572 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:23.572 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:23.572 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.572 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.572 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:23.572 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:23.572 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:23.572 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:23.572 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:23.572 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:23.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.550+0000 7f351a7fc640 1 -- 172.21.15.121:0/1905323667 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3510011910 con 0x7f3508007c00 2024-09-17T18:37:23.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.550+0000 7f351a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:23.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.550+0000 7f351b7fe640 4 set_mon_vals no callback set 2024-09-17T18:37:23.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.550+0000 7f351a7fc640 10 client.114108.objecter ms_handle_connect 0x7f351c126ac0 2024-09-17T18:37:23.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.550+0000 7f351a7fc640 10 client.114108.objecter resend_mon_ops 2024-09-17T18:37:23.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.550+0000 7f3521b6f640 1 -- 172.21.15.121:0/1905323667 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f351c126ac0 msgr2=0x7f351c128c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:23.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.550+0000 7f3521b6f640 1 --2- 172.21.15.121:0/1905323667 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f351c126ac0 0x7f351c128c20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f351c077910 tx=0x7f350c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:23.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.550+0000 7f351a7fc640 1 -- 172.21.15.121:0/1905323667 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f350c031cd0 con 0x7f351c126ac0 2024-09-17T18:37:23.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.550+0000 7f351a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:23.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.550+0000 7f351a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:37:23.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.550+0000 7f351a7fc640 10 monclient: dump: 2024-09-17T18:37:23.573 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:23.573 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:23.573 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.573 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.573 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:23.573 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:23.573 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:23.574 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:23.574 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:23.574 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:23.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.550+0000 7f351a7fc640 1 -- 172.21.15.121:0/1905323667 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f3510019020 con 0x7f3508007c00 2024-09-17T18:37:23.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.551+0000 7f351a7fc640 1 --2- 172.21.15.121:0/1905323667 >> v2:172.21.15.173:6812/2423786037 conn(0x7f351c09ee70 0x7f350804a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:23.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.551+0000 7f351a7fc640 1 -- 172.21.15.121:0/1905323667 <== mon.0 v2:172.21.15.121:3300/0 4 ==== osd_map(679..679 src has 247..679) ==== 4685+0+0 (secure 0 0 0) 0x7f3510054b70 con 0x7f3508007c00 2024-09-17T18:37:23.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.551+0000 7f351a7fc640 10 client.114108.objecter ms_dispatch 0x7f351c000b90 osd_map(679..679 src has 247..679) 2024-09-17T18:37:23.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.551+0000 7f351a7fc640 3 client.114108.objecter handle_osd_map got epochs [679,679] > 0 2024-09-17T18:37:23.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.551+0000 7f351a7fc640 3 client.114108.objecter handle_osd_map decoding full epoch 679 2024-09-17T18:37:23.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.551+0000 7f351a7fc640 20 client.114108.objecter dump_active .. 0 homeless 2024-09-17T18:37:23.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.551+0000 7f351a7fc640 1 -- 172.21.15.121:0/1905323667 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f350c03c420 con 0x7f351c126ac0 2024-09-17T18:37:23.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.551+0000 7f351a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:23.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.551+0000 7f351a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:37:23.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.551+0000 7f351a7fc640 1 -- 172.21.15.121:0/1905323667 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f351c126ac0 msgr2=0x7f351c128c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:23.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.551+0000 7f351a7fc640 1 --2- 172.21.15.121:0/1905323667 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f351c126ac0 0x7f351c128c20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f351c077910 tx=0x7f350c004a40 comp rx=0 tx=0).stop 2024-09-17T18:37:23.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.551+0000 7f351a7fc640 1 -- 172.21.15.121:0/1905323667 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f351c126ac0 msgr2=0x7f351c128c20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:37:23.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.551+0000 7f351a7fc640 1 --2- 172.21.15.121:0/1905323667 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f351c126ac0 0x7f351c128c20 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f351c077910 tx=0x7f350c004a40 comp rx=0 tx=0).stop 2024-09-17T18:37:23.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.551+0000 7f3522370640 1 --2- 172.21.15.121:0/1905323667 >> v2:172.21.15.173:6812/2423786037 conn(0x7f351c09ee70 0x7f350804a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:23.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.551+0000 7f3522370640 10 monclient: get_auth_request con 0x7f351c09ee70 auth_method 0 2024-09-17T18:37:23.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.553+0000 7f351a7fc640 10 client.114108.objecter ms_handle_connect 0x7f351c09ee70 2024-09-17T18:37:23.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.553+0000 7f3522370640 1 --2- 172.21.15.121:0/1905323667 >> v2:172.21.15.173:6812/2423786037 conn(0x7f351c09ee70 0x7f350804a6f0 secure :-1 s=READY pgs=4091 cs=0 l=1 rev1=1 crypto rx=0x7f3514007c90 tx=0x7f3514007c00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:23.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.557+0000 7f3523dd3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:37:23.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.557+0000 7f3523dd3640 1 --2- 172.21.15.121:0/1905323667 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f351c13fa90 0x7f351c13fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:23.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.557+0000 7f3523dd3640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:23.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.557+0000 7f3523dd3640 1 -- 172.21.15.121:0/1905323667 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f351c0a5f20 con 0x7f351c13fa90 2024-09-17T18:37:23.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.557+0000 7f3521b6f640 1 --2- 172.21.15.121:0/1905323667 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f351c13fa90 0x7f351c13fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:23.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.557+0000 7f3521b6f640 10 monclient: get_auth_request con 0x7f351c13fa90 auth_method 0 2024-09-17T18:37:23.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.557+0000 7f3521b6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:23.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.557+0000 7f3521b6f640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:23.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.557+0000 7f3521b6f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:23.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.557+0000 7f3521b6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:23.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.557+0000 7f3521b6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:23.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.557+0000 7f3521b6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:23.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.557+0000 7f3521b6f640 10 monclient(hunting): handle_auth_done global_id 114114 payload 824 2024-09-17T18:37:23.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.557+0000 7f3521b6f640 1 -- 172.21.15.121:0/1905323667 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f351c129f60 con 0x7f351c13fa90 2024-09-17T18:37:23.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.559+0000 7f3521b6f640 1 -- 172.21.15.121:0/1905323667 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f351c13fa90 msgr2=0x7f351c13fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:23.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.559+0000 7f351a7fc640 10 client.114108.objecter ms_handle_connect 0x7f351c13fa90 2024-09-17T18:37:23.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.559+0000 7f351a7fc640 10 client.114108.objecter resend_mon_ops 2024-09-17T18:37:23.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.559+0000 7f3521b6f640 1 --2- 172.21.15.121:0/1905323667 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f351c13fa90 0x7f351c13fe70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f350c02a4d0 tx=0x7f350c02f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:23.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.560+0000 7f351a7fc640 1 -- 172.21.15.121:0/1905323667 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f350c004660 con 0x7f351c13fa90 2024-09-17T18:37:23.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.560+0000 7f351a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:23.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.560+0000 7f351a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:37:23.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.560+0000 7f351a7fc640 10 monclient: dump: 2024-09-17T18:37:23.578 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:23.578 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:23.578 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.578 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.579 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:23.579 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:23.579 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:23.579 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:23.579 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:23.579 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:23.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.560+0000 7f351a7fc640 1 -- 172.21.15.121:0/1905323667 <== mon.0 v2:172.21.15.121:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4553 (secure 0 0 0) 0x7f350c04c040 con 0x7f351c13fa90 2024-09-17T18:37:23.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.560+0000 7f351a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:37:23.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.560+0000 7f351a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:37:23.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.560+0000 7f351a7fc640 1 -- 172.21.15.121:0/1905323667 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f351c13fa90 msgr2=0x7f351c13fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:23.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.560+0000 7f351a7fc640 1 --2- 172.21.15.121:0/1905323667 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f351c13fa90 0x7f351c13fe70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f350c02a4d0 tx=0x7f350c02f000 comp rx=0 tx=0).stop 2024-09-17T18:37:23.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.560+0000 7f351a7fc640 1 -- 172.21.15.121:0/1905323667 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f351c13fa90 msgr2=0x7f351c13fe70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:37:23.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.560+0000 7f351a7fc640 1 --2- 172.21.15.121:0/1905323667 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f351c13fa90 0x7f351c13fe70 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f350c02a4d0 tx=0x7f350c02f000 comp rx=0 tx=0).stop 2024-09-17T18:37:23.580 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:37:23.580 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:37:23.580 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:37:23.580 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "leader", 2024-09-17T18:37:23.580 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 66, 2024-09-17T18:37:23.580 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 7171, 2024-09-17T18:37:23.580 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:37:23.580 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:37:23.580 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:37:23.580 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:37:23.580 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:37:23.580 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 0, 2024-09-17T18:37:23.580 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:37:23.581 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:37:23.581 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:37:23.581 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:37:23.581 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:37:23.581 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:37:23.581 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:37:23.581 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:37:23.581 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:37:23.581 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:37:23.581 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:37:23.581 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:37:23.581 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:37:23.581 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:37:23.581 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:37:23.581 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:37:23.582 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:37:23.582 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:37:23.582 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:37:23.582 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:37:23.582 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:37:23.582 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:37:23.582 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:37:23.582 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:37:23.582 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:37:23.582 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:37:23.582 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:37:23.582 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:37:23.582 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:37:23.582 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:37:23.582 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:37:23.583 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:37:23.583 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:37:23.583 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:37:23.583 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:37:23.583 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:37:23.583 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:37:23.583 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:37:23.583 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:37:23.583 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:37:23.583 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:37:23.583 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:37:23.583 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:37:23.583 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:37:23.583 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:37:23.583 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:37:23.584 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:37:23.584 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:37:23.584 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:37:23.584 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:37:23.584 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:37:23.584 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:37:23.584 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:37:23.584 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:37:23.584 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:37:23.584 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:37:23.584 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:37:23.584 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:37:23.584 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:37:23.584 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:37:23.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:37:23.585 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:37:23.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:37:23.585 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:37:23.585 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:37:23.585 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:37:23.585 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:37:23.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:37:23.585 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:37:23.585 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:37:23.585 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:37:23.585 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:37:23.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:37:23.586 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:37:23.586 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:37:23.586 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:37:23.586 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:37:23.586 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:37:23.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:37:23.586 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:37:23.586 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:37:23.586 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:37:23.586 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:37:23.586 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:37:23.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:37:23.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:37:23.586 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:37:23.586 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:37:23.586 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:37:23.587 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:37:23.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:37:23.587 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:37:23.587 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:37:23.587 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:37:23.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:37:23.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:37:23.587 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:37:23.587 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:37:23.587 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:37:23.587 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:37:23.587 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:37:23.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:37:23.587 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:37:23.587 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:37:23.588 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:37:23.588 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:37:23.588 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:37:23.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:37:23.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:37:23.588 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:37:23.588 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:37:23.588 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:37:23.588 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:37:23.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:37:23.588 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:37:23.588 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:37:23.588 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:37:23.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:37:23.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:37:23.588 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:37:23.589 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:37:23.589 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:37:23.589 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:37:23.589 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:37:23.589 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:37:23.589 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:37:23.589 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:37:23.589 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:37:23.589 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:37:23.589 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:37:23.589 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:37:23.589 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:37:23.589 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:37:23.589 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:37:23.589 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:37:23.589 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:37:23.590 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:37:23.590 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:37:23.590 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:37:23.590 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:37:23.590 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:37:23.590 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:37:23.590 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:37:23.590 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:37:23.590 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:37:23.590 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-17T18:37:23.590 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:37:23.590 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:37:23.590 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:37:23.590 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:37:23.590 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:37:23.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.560+0000 7f3523dd3640 10 librados: watch_flush enter 2024-09-17T18:37:23.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.560+0000 7f3523dd3640 10 librados: watch_flush exit 2024-09-17T18:37:23.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.560+0000 7f3523dd3640 20 client.114108.objecter shutdown clearing up homeless session... 2024-09-17T18:37:23.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.560+0000 7f3523dd3640 10 client.114108.objecter successfully canceled tick 2024-09-17T18:37:23.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.560+0000 7f3523dd3640 1 -- 172.21.15.121:0/1905323667 >> v2:172.21.15.173:6812/2423786037 conn(0x7f351c09ee70 msgr2=0x7f350804a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:23.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.560+0000 7f3523dd3640 1 --2- 172.21.15.121:0/1905323667 >> v2:172.21.15.173:6812/2423786037 conn(0x7f351c09ee70 0x7f350804a6f0 secure :-1 s=READY pgs=4091 cs=0 l=1 rev1=1 crypto rx=0x7f3514007c90 tx=0x7f3514007c00 comp rx=0 tx=0).stop 2024-09-17T18:37:23.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.560+0000 7f3523dd3640 10 monclient: shutdown 2024-09-17T18:37:23.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.560+0000 7f3523dd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:23.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.560+0000 7f3523dd3640 1 -- 172.21.15.121:0/1905323667 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3508007c00 msgr2=0x7f350800a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:23.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.560+0000 7f3523dd3640 1 --2- 172.21.15.121:0/1905323667 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3508007c00 0x7f350800a090 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3510017000 tx=0x7f3510007ef0 comp rx=0 tx=0).stop 2024-09-17T18:37:23.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.560+0000 7f3522370640 1 -- 172.21.15.121:0/1905323667 reap_dead start 2024-09-17T18:37:23.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.560+0000 7f3522370640 1 -- 172.21.15.121:0/1905323667 reap_dead start 2024-09-17T18:37:23.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.560+0000 7f3523dd3640 1 -- 172.21.15.121:0/1905323667 shutdown_connections 2024-09-17T18:37:23.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.560+0000 7f3523dd3640 1 -- 172.21.15.121:0/1905323667 >> 172.21.15.121:0/1905323667 conn(0x7f351c0a7830 msgr2=0x7f351c072bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:23.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.560+0000 7f3523dd3640 1 -- 172.21.15.121:0/1905323667 shutdown_connections 2024-09-17T18:37:23.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.560+0000 7f3523dd3640 1 -- 172.21.15.121:0/1905323667 wait complete. 2024-09-17T18:37:23.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.560+0000 7f3523dd3640 1 librados: shutdown 2024-09-17T18:37:23.593 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:37:23.593 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T18:37:23.593 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T18:37:23.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.688+0000 7fc09a42b640 10 monclient: get_monmap_and_config 2024-09-17T18:37:23.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.688+0000 7fc09a42b640 10 monclient: build_initial_monmap 2024-09-17T18:37:23.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.688+0000 7fc09a42b640 10 monclient: monmap: 2024-09-17T18:37:23.691 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:23.691 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:23.691 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:37:23.689535+0000 2024-09-17T18:37:23.691 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:37:23.689535+0000 2024-09-17T18:37:23.691 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:23.691 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:23.691 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:37:23.691 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:37:23.691 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:37:23.691 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:37:23.691 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:37:23.691 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:23.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.689+0000 7fc09a42b640 1 Processor -- start 2024-09-17T18:37:23.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.689+0000 7fc09a42b640 1 -- start start 2024-09-17T18:37:23.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.689+0000 7fc09a42b640 10 monclient: init 2024-09-17T18:37:23.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.690+0000 7fc09a42b640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:23.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.690+0000 7fc09a42b640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-17T18:37:23.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.690+0000 7fc09a42b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc09409da70 addr v1:172.21.15.121:6789/0 2024-09-17T18:37:23.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.690+0000 7fc09a42b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc0941368d0 addr v1:172.21.15.173:6789/0 2024-09-17T18:37:23.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.690+0000 7fc09a42b640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc094136d00 0x7fc09413ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:23.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.690+0000 7fc09a42b640 10 monclient(hunting): picked mon.noname-a con 0x7fc094136d00 addr v2:172.21.15.121:3300/0 2024-09-17T18:37:23.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.690+0000 7fc09a42b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0940a2d40 con 0x7fc09409da70 2024-09-17T18:37:23.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.690+0000 7fc09a42b640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0940a3020 con 0x7fc0941368d0 2024-09-17T18:37:23.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.690+0000 7fc09a42b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:23.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.690+0000 7fc09a42b640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fc09409de50 con 0x7fc094136d00 2024-09-17T18:37:23.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.690+0000 7fc09a42b640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:23.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.690+0000 7fc09a42b640 10 monclient(hunting): authenticate will time out at 1847.406392s 2024-09-17T18:37:23.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.690+0000 7fc0989c8640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc094136d00 0x7fc09413ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:23.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.690+0000 7fc093fff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fc09409da70 0x7fc0941361c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:47310/0 (socket says 172.21.15.121:47310) 2024-09-17T18:37:23.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.690+0000 7fc093fff640 1 -- 172.21.15.121:0/2234900505 learned_addr learned my addr 172.21.15.121:0/2234900505 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:37:23.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.690+0000 7fc0989c8640 1 --2- 172.21.15.121:0/2234900505 >> v2:172.21.15.121:3300/0 conn(0x7fc094136d00 0x7fc09413ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51184/0 (socket says 172.21.15.121:51184) 2024-09-17T18:37:23.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.690+0000 7fc0989c8640 10 monclient(hunting): get_auth_request con 0x7fc094136d00 auth_method 0 2024-09-17T18:37:23.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.690+0000 7fc0989c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:23.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.690+0000 7fc0989c8640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:23.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.690+0000 7fc0989c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:23.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.690+0000 7fc0989c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:23.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.690+0000 7fc0989c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:23.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.690+0000 7fc0989c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:23.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.690+0000 7fc0989c8640 10 monclient(hunting): handle_auth_done global_id 114117 payload 274 2024-09-17T18:37:23.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.690+0000 7fc0989c8640 1 -- 172.21.15.121:0/2234900505 >> v1:172.21.15.173:6789/0 conn(0x7fc0941368d0 legacy=0x7fc09413b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:23.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.690+0000 7fc0989c8640 1 -- 172.21.15.121:0/2234900505 >> v1:172.21.15.121:6789/0 conn(0x7fc09409da70 legacy=0x7fc0941361c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:23.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.690+0000 7fc0989c8640 10 monclient: _finish_hunting 0 2024-09-17T18:37:23.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.690+0000 7fc0989c8640 1 monclient: found mon.noname-a 2024-09-17T18:37:23.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.690+0000 7fc0989c8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:23.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.690+0000 7fc0989c8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:37:23.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.690+0000 7fc0989c8640 1 -- 172.21.15.121:0/2234900505 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0940a3300 con 0x7fc094136d00 2024-09-17T18:37:23.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.693+0000 7fc0989c8640 1 -- 172.21.15.121:0/2234900505 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc094136d00 msgr2=0x7fc09413ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:23.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.693+0000 7fc0989c8640 1 --2- 172.21.15.121:0/2234900505 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc094136d00 0x7fc09413ec70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc08c009d90 tx=0x7fc08c02a490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88680ded2d36a4f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:23.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.693+0000 7fc092ffd640 1 -- 172.21.15.121:0/2234900505 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc08c007a30 con 0x7fc094136d00 2024-09-17T18:37:23.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.693+0000 7fc092ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:23.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.693+0000 7fc092ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:37:23.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.693+0000 7fc092ffd640 10 monclient: dump: 2024-09-17T18:37:23.700 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:23.700 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:23.700 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.700 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.700 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:23.700 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:23.700 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:23.700 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:23.700 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:23.701 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:23.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.693+0000 7fc092ffd640 10 monclient: _finish_auth 0 2024-09-17T18:37:23.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.693+0000 7fc092ffd640 10 monclient: _check_auth_tickets 2024-09-17T18:37:23.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.693+0000 7fc092ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:23.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.693+0000 7fc092ffd640 1 -- 172.21.15.121:0/2234900505 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc08c007bd0 con 0x7fc094136d00 2024-09-17T18:37:23.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.693+0000 7fc092ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:23.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.693+0000 7fc092ffd640 1 -- 172.21.15.121:0/2234900505 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc08c031400 con 0x7fc094136d00 2024-09-17T18:37:23.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.693+0000 7fc09a42b640 5 monclient: authenticate success, global_id 114117 2024-09-17T18:37:23.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.693+0000 7fc092ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:23.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.693+0000 7fc092ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:37:23.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.693+0000 7fc092ffd640 10 monclient: dump: 2024-09-17T18:37:23.701 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:23.701 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:23.701 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.701 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.702 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:23.702 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:23.702 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:23.702 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:23.702 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:23.702 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:23.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.693+0000 7fc09a42b640 10 monclient: get_monmap_and_config success 2024-09-17T18:37:23.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.693+0000 7fc09a42b640 4 set_mon_vals no callback set 2024-09-17T18:37:23.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.694+0000 7fc09a42b640 10 monclient: shutdown 2024-09-17T18:37:23.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.694+0000 7fc09a42b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:23.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.694+0000 7fc09a42b640 1 -- 172.21.15.121:0/2234900505 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc094136d00 msgr2=0x7fc09413ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:23.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.694+0000 7fc09a42b640 1 --2- 172.21.15.121:0/2234900505 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc094136d00 0x7fc09413ec70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc08c009d90 tx=0x7fc08c02a490 comp rx=0 tx=0).stop 2024-09-17T18:37:23.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.694+0000 7fc09a42b640 1 -- 172.21.15.121:0/2234900505 shutdown_connections 2024-09-17T18:37:23.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.694+0000 7fc09a42b640 1 --2- 172.21.15.121:0/2234900505 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc094136d00 0x7fc09413ec70 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:23.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.694+0000 7fc09a42b640 1 -- 172.21.15.121:0/2234900505 >> 172.21.15.121:0/2234900505 conn(0x7fc0940a7830 msgr2=0x7fc0940a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:23.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.694+0000 7fc09a42b640 1 -- 172.21.15.121:0/2234900505 shutdown_connections 2024-09-17T18:37:23.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.694+0000 7fc09a42b640 1 -- 172.21.15.121:0/2234900505 wait complete. 2024-09-17T18:37:23.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.695+0000 7fc09a42b640 10 monclient: build_initial_monmap 2024-09-17T18:37:23.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.695+0000 7fc09a42b640 10 monclient: monmap: 2024-09-17T18:37:23.703 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:23.703 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:37:23.703 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:37:23.703 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:37:23.703 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:23.703 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:23.703 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:37:23.703 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:37:23.703 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:37:23.703 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:23.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.695+0000 7fc09a42b640 1 librados: starting msgr at 2024-09-17T18:37:23.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.695+0000 7fc09a42b640 1 librados: starting objecter 2024-09-17T18:37:23.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.695+0000 7fc09a42b640 1 Processor -- start 2024-09-17T18:37:23.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.695+0000 7fc09a42b640 1 -- start start 2024-09-17T18:37:23.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.695+0000 7fc09a42b640 1 librados: setting wanted keys 2024-09-17T18:37:23.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.695+0000 7fc09a42b640 1 librados: calling monclient init 2024-09-17T18:37:23.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.695+0000 7fc09a42b640 10 monclient: init 2024-09-17T18:37:23.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc09a42b640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:23.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc09a42b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T18:37:23.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc09a42b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc09409da70 0x7fc09412fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:23.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc09a42b640 10 monclient(hunting): picked mon.noname-a con 0x7fc09409da70 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:37:23.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc09a42b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0941368d0 0x7fc09412ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:23.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc09a42b640 10 monclient(hunting): picked mon.noname-c con 0x7fc0941368d0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:23.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc09a42b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc094136d00 0x7fc094137190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:23.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc09a42b640 10 monclient(hunting): picked mon.noname-b con 0x7fc094136d00 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:37:23.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc09a42b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:23.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc09a42b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc09413f1b0 con 0x7fc09409da70 2024-09-17T18:37:23.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc09a42b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:23.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc09a42b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc0940a3930 con 0x7fc094136d00 2024-09-17T18:37:23.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc09a42b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:23.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc09a42b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc09413f330 con 0x7fc0941368d0 2024-09-17T18:37:23.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc09a42b640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:23.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc09a42b640 10 monclient(hunting): authenticate will time out at 1847.412125s 2024-09-17T18:37:23.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc093fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc09409da70 0x7fc09412fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:23.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc0937fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0941368d0 0x7fc09412ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:23.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc093fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc09409da70 0x7fc09412fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51190/0 (socket says 172.21.15.121:51190) 2024-09-17T18:37:23.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc093fff640 1 -- 172.21.15.121:0/1236720723 learned_addr learned my addr 172.21.15.121:0/1236720723 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:37:23.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc093fff640 10 monclient(hunting): get_auth_request con 0x7fc09409da70 auth_method 0 2024-09-17T18:37:23.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc093fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:23.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc093fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:23.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc093fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:23.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc0937fe640 10 monclient(hunting): get_auth_request con 0x7fc0941368d0 auth_method 0 2024-09-17T18:37:23.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc0937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:23.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc0937fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:23.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc0937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:23.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc093fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:23.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc093fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:23.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc093fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:23.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc0937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:23.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc0937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:23.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc0937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:23.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc0989c8640 1 --2- 172.21.15.121:0/1236720723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc094136d00 0x7fc094137190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:23.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc0989c8640 10 monclient(hunting): get_auth_request con 0x7fc094136d00 auth_method 0 2024-09-17T18:37:23.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc0989c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:23.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc0989c8640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:23.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc0989c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:23.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc093fff640 10 monclient(hunting): handle_auth_done global_id 114120 payload 824 2024-09-17T18:37:23.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc093fff640 1 -- 172.21.15.121:0/1236720723 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0941368d0 msgr2=0x7fc09412ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:23.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc093fff640 1 --2- 172.21.15.121:0/1236720723 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0941368d0 0x7fc09412ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:23.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc093fff640 1 -- 172.21.15.121:0/1236720723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc094136d00 msgr2=0x7fc094137190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:23.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc093fff640 1 --2- 172.21.15.121:0/1236720723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc094136d00 0x7fc094137190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:23.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc093fff640 10 monclient: _finish_hunting 0 2024-09-17T18:37:23.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc093fff640 1 monclient: found mon.noname-a 2024-09-17T18:37:23.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc093fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:23.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc093fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:37:23.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc093fff640 1 -- 172.21.15.121:0/1236720723 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc09413eaa0 con 0x7fc09409da70 2024-09-17T18:37:23.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc0937fe640 1 --2- 172.21.15.121:0/1236720723 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0941368d0 0x7fc09412ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:37:23.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.696+0000 7fc0989c8640 1 --2- 172.21.15.121:0/1236720723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc094136d00 0x7fc094137190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T18:37:23.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.698+0000 7fc093fff640 1 -- 172.21.15.121:0/1236720723 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc09409da70 msgr2=0x7fc09412fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:23.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.698+0000 7fc093fff640 1 --2- 172.21.15.121:0/1236720723 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc09409da70 0x7fc09412fa30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc08400c460 tx=0x7fc08400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:23.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.698+0000 7fc090ff9640 10 client.?.objecter ms_handle_connect 0x7fc09409da70 2024-09-17T18:37:23.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.698+0000 7fc090ff9640 10 client.?.objecter resend_mon_ops 2024-09-17T18:37:23.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.698+0000 7fc090ff9640 1 -- 172.21.15.121:0/1236720723 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc084007070 con 0x7fc09409da70 2024-09-17T18:37:23.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.698+0000 7fc090ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:23.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.698+0000 7fc090ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:37:23.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.698+0000 7fc090ff9640 10 monclient: dump: 2024-09-17T18:37:23.712 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:23.712 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:23.712 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.712 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.712 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:23.712 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:23.712 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:23.712 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:23.713 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:23.713 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:23.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.698+0000 7fc090ff9640 10 monclient: _finish_auth 0 2024-09-17T18:37:23.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.698+0000 7fc090ff9640 10 monclient: _check_auth_tickets 2024-09-17T18:37:23.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.698+0000 7fc090ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:23.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.698+0000 7fc09a42b640 5 monclient: authenticate success, global_id 114120 2024-09-17T18:37:23.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.698+0000 7fc090ff9640 1 -- 172.21.15.121:0/1236720723 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc084007230 con 0x7fc09409da70 2024-09-17T18:37:23.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.698+0000 7fc090ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:23.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.698+0000 7fc090ff9640 1 -- 172.21.15.121:0/1236720723 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc084007620 con 0x7fc09409da70 2024-09-17T18:37:23.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.698+0000 7fc09a42b640 10 monclient: _renew_subs 2024-09-17T18:37:23.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.698+0000 7fc09a42b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:23.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.698+0000 7fc09a42b640 1 -- 172.21.15.121:0/1236720723 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc094138780 con 0x7fc09409da70 2024-09-17T18:37:23.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.698+0000 7fc090ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:23.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.698+0000 7fc090ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:37:23.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.698+0000 7fc090ff9640 10 monclient: dump: 2024-09-17T18:37:23.714 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:23.714 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:23.714 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.714 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.714 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:23.714 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:23.714 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:23.714 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:23.714 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:23.714 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:23.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.698+0000 7fc0927fc640 4 set_mon_vals no callback set 2024-09-17T18:37:23.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.698+0000 7fc09a42b640 10 client.114120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:37:23.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.699+0000 7fc09a42b640 10 monclient: _renew_subs 2024-09-17T18:37:23.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.699+0000 7fc09a42b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:23.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.699+0000 7fc09a42b640 1 -- 172.21.15.121:0/1236720723 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc09413a240 con 0x7fc09409da70 2024-09-17T18:37:23.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.699+0000 7fc09a42b640 1 librados: init done 2024-09-17T18:37:23.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.699+0000 7fc09a42b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:23.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.699+0000 7fc09a42b640 1 --2- 172.21.15.121:0/1236720723 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0941376d0 0x7fc094137ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:23.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.699+0000 7fc09a42b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:23.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.699+0000 7fc09a42b640 1 -- 172.21.15.121:0/1236720723 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc0941361c0 con 0x7fc0941376d0 2024-09-17T18:37:23.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.699+0000 7fc090ff9640 1 -- 172.21.15.121:0/1236720723 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fc08401b400 con 0x7fc09409da70 2024-09-17T18:37:23.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.699+0000 7fc0937fe640 1 --2- 172.21.15.121:0/1236720723 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0941376d0 0x7fc094137ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:23.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.699+0000 7fc0937fe640 10 monclient: get_auth_request con 0x7fc0941376d0 auth_method 0 2024-09-17T18:37:23.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.699+0000 7fc0937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:23.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.699+0000 7fc0937fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:23.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.699+0000 7fc0937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:23.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.699+0000 7fc0937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:23.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.699+0000 7fc0937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:23.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.699+0000 7fc0937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:23.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.700+0000 7fc090ff9640 1 --2- 172.21.15.121:0/1236720723 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc0800401a0 0x7fc080042660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:23.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.700+0000 7fc0937fe640 10 monclient(hunting): handle_auth_done global_id 64577 payload 824 2024-09-17T18:37:23.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.700+0000 7fc0937fe640 1 -- 172.21.15.121:0/1236720723 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc09413a5e0 con 0x7fc0941376d0 2024-09-17T18:37:23.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.700+0000 7fc090ff9640 1 -- 172.21.15.121:0/1236720723 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(679..679 src has 247..679) ==== 4685+0+0 (secure 0 0 0) 0x7fc0840530d0 con 0x7fc09409da70 2024-09-17T18:37:23.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.700+0000 7fc090ff9640 10 client.114120.objecter ms_dispatch 0x7fc094000b90 osd_map(679..679 src has 247..679) 2024-09-17T18:37:23.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.700+0000 7fc090ff9640 3 client.114120.objecter handle_osd_map got epochs [679,679] > 0 2024-09-17T18:37:23.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.700+0000 7fc090ff9640 3 client.114120.objecter handle_osd_map decoding full epoch 679 2024-09-17T18:37:23.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.700+0000 7fc090ff9640 20 client.114120.objecter dump_active .. 0 homeless 2024-09-17T18:37:23.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.700+0000 7fc0989c8640 1 --2- 172.21.15.121:0/1236720723 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc0800401a0 0x7fc080042660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:23.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.700+0000 7fc0989c8640 10 monclient: get_auth_request con 0x7fc0800401a0 auth_method 0 2024-09-17T18:37:23.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.702+0000 7fc090ff9640 10 client.114120.objecter ms_handle_connect 0x7fc0941376d0 2024-09-17T18:37:23.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.702+0000 7fc090ff9640 10 client.114120.objecter resend_mon_ops 2024-09-17T18:37:23.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.702+0000 7fc0937fe640 1 -- 172.21.15.121:0/1236720723 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0941376d0 msgr2=0x7fc094137ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:23.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.702+0000 7fc0937fe640 1 --2- 172.21.15.121:0/1236720723 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0941376d0 0x7fc094137ab0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc0940744f0 tx=0x7fc08800ff70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:23.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.702+0000 7fc090ff9640 1 -- 172.21.15.121:0/1236720723 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc08800dd60 con 0x7fc0941376d0 2024-09-17T18:37:23.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.702+0000 7fc090ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:23.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.702+0000 7fc090ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:37:23.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.702+0000 7fc090ff9640 10 monclient: dump: 2024-09-17T18:37:23.717 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:23.717 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:23.717 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.717 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.717 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:23.717 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:23.717 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:23.718 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:23.718 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:23.718 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:23.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.702+0000 7fc090ff9640 1 -- 172.21.15.121:0/1236720723 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc08800dd60 con 0x7fc0941376d0 2024-09-17T18:37:23.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.702+0000 7fc090ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:23.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.702+0000 7fc090ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:37:23.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.702+0000 7fc090ff9640 1 -- 172.21.15.121:0/1236720723 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0941376d0 msgr2=0x7fc094137ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:23.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.702+0000 7fc090ff9640 1 --2- 172.21.15.121:0/1236720723 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0941376d0 0x7fc094137ab0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc0940744f0 tx=0x7fc08800ff70 comp rx=0 tx=0).stop 2024-09-17T18:37:23.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.702+0000 7fc090ff9640 1 -- 172.21.15.121:0/1236720723 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0941376d0 msgr2=0x7fc094137ab0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:37:23.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.702+0000 7fc090ff9640 1 --2- 172.21.15.121:0/1236720723 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0941376d0 0x7fc094137ab0 secure :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc0940744f0 tx=0x7fc08800ff70 comp rx=0 tx=0).stop 2024-09-17T18:37:23.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.702+0000 7fc090ff9640 10 client.114120.objecter ms_handle_connect 0x7fc0800401a0 2024-09-17T18:37:23.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.702+0000 7fc0989c8640 1 --2- 172.21.15.121:0/1236720723 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc0800401a0 0x7fc080042660 secure :-1 s=READY pgs=4092 cs=0 l=1 rev1=1 crypto rx=0x7fc08c034000 tx=0x7fc08c02f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:23.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.708+0000 7fc09a42b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:37:23.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.708+0000 7fc09a42b640 1 --2- 172.21.15.121:0/1236720723 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc094137ff0 0x7fc0940702d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:23.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.708+0000 7fc09a42b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:23.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.708+0000 7fc09a42b640 1 -- 172.21.15.121:0/1236720723 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc094139060 con 0x7fc094137ff0 2024-09-17T18:37:23.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.708+0000 7fc0937fe640 1 --2- 172.21.15.121:0/1236720723 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc094137ff0 0x7fc0940702d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:23.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.708+0000 7fc0937fe640 10 monclient: get_auth_request con 0x7fc094137ff0 auth_method 0 2024-09-17T18:37:23.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.708+0000 7fc0937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:23.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.708+0000 7fc0937fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:23.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.708+0000 7fc0937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:23.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.708+0000 7fc0937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:23.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.708+0000 7fc0937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:23.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.708+0000 7fc0937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:23.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.709+0000 7fc0937fe640 10 monclient(hunting): handle_auth_done global_id 64580 payload 824 2024-09-17T18:37:23.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.709+0000 7fc0937fe640 1 -- 172.21.15.121:0/1236720723 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc09406cd40 con 0x7fc094137ff0 2024-09-17T18:37:23.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.711+0000 7fc0937fe640 1 -- 172.21.15.121:0/1236720723 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc094137ff0 msgr2=0x7fc0940702d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:23.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.711+0000 7fc090ff9640 10 client.114120.objecter ms_handle_connect 0x7fc094137ff0 2024-09-17T18:37:23.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.711+0000 7fc090ff9640 10 client.114120.objecter resend_mon_ops 2024-09-17T18:37:23.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.711+0000 7fc0937fe640 1 --2- 172.21.15.121:0/1236720723 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc094137ff0 0x7fc0940702d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc088018000 tx=0x7fc088003760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:23.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.711+0000 7fc090ff9640 1 -- 172.21.15.121:0/1236720723 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc08801e030 con 0x7fc094137ff0 2024-09-17T18:37:23.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.711+0000 7fc090ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:23.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.711+0000 7fc090ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:37:23.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.711+0000 7fc090ff9640 10 monclient: dump: 2024-09-17T18:37:23.721 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:23.721 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:23.721 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.721 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.721 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:23.721 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:23.721 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:23.721 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:23.721 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:23.721 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:23.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.711+0000 7fc090ff9640 1 -- 172.21.15.121:0/1236720723 <== mon.2 v2:172.21.15.121:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7fc08801e030 con 0x7fc094137ff0 2024-09-17T18:37:23.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.711+0000 7fc090ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:37:23.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.711+0000 7fc090ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:37:23.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.711+0000 7fc090ff9640 1 -- 172.21.15.121:0/1236720723 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc094137ff0 msgr2=0x7fc0940702d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:23.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.711+0000 7fc090ff9640 1 --2- 172.21.15.121:0/1236720723 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc094137ff0 0x7fc0940702d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc088018000 tx=0x7fc088003760 comp rx=0 tx=0).stop 2024-09-17T18:37:23.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.711+0000 7fc090ff9640 1 -- 172.21.15.121:0/1236720723 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc094137ff0 msgr2=0x7fc0940702d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:37:23.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.711+0000 7fc090ff9640 1 --2- 172.21.15.121:0/1236720723 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc094137ff0 0x7fc0940702d0 secure :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc088018000 tx=0x7fc088003760 comp rx=0 tx=0).stop 2024-09-17T18:37:23.722 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:37:23.722 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:37:23.722 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:37:23.722 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:37:23.722 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 66, 2024-09-17T18:37:23.723 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 622444, 2024-09-17T18:37:23.723 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:37:23.723 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:37:23.723 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:37:23.723 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:37:23.723 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:37:23.723 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 0, 2024-09-17T18:37:23.723 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:37:23.723 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:37:23.723 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:37:23.723 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:37:23.723 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:37:23.723 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:37:23.723 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:37:23.723 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:37:23.724 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:37:23.724 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:37:23.724 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:37:23.724 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:37:23.724 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:37:23.724 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:37:23.724 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:37:23.724 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:37:23.724 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:37:23.724 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:37:23.724 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:37:23.724 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:37:23.724 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:37:23.724 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:37:23.725 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:37:23.725 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:37:23.725 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:37:23.725 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:37:23.725 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:37:23.725 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:37:23.725 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:37:23.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:37:23.725 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:37:23.725 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:37:23.725 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:37:23.725 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:37:23.725 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:37:23.725 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:37:23.725 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:37:23.725 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:37:23.726 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:37:23.726 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:37:23.726 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:37:23.726 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:37:23.726 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:37:23.726 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:37:23.726 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:37:23.726 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:37:23.726 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:37:23.726 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:37:23.726 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:37:23.726 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:37:23.726 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:37:23.726 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:37:23.726 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:37:23.726 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:37:23.727 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:37:23.727 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:37:23.727 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:37:23.727 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:37:23.727 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:37:23.727 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:37:23.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:37:23.727 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:37:23.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:37:23.727 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:37:23.727 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:37:23.727 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:37:23.727 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:37:23.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:37:23.727 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:37:23.728 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:37:23.728 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:37:23.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:37:23.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:37:23.728 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:37:23.728 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:37:23.728 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:37:23.728 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:37:23.728 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:37:23.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:37:23.728 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:37:23.728 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:37:23.728 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:37:23.728 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:37:23.728 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:37:23.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:37:23.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:37:23.729 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:37:23.729 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:37:23.729 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:37:23.729 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:37:23.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:37:23.729 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:37:23.729 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:37:23.729 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:37:23.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:37:23.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:37:23.729 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:37:23.729 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:37:23.729 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:37:23.729 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:37:23.730 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:37:23.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:37:23.730 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:37:23.730 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:37:23.730 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:37:23.730 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:37:23.730 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:37:23.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:37:23.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:37:23.730 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:37:23.730 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:37:23.730 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:37:23.730 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:37:23.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:37:23.730 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:37:23.731 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:37:23.731 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:37:23.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:37:23.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:37:23.731 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:37:23.731 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:37:23.731 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:37:23.731 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:37:23.731 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:37:23.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:37:23.731 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:37:23.731 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:37:23.731 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:37:23.731 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:37:23.731 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:37:23.731 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:37:23.732 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:37:23.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:37:23.732 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:37:23.732 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:37:23.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:37:23.732 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:37:23.732 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:37:23.732 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:37:23.732 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:37:23.732 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:37:23.732 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-17T18:37:23.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:37:23.732 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:37:23.732 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:37:23.732 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 6 2024-09-17T18:37:23.732 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:37:23.733 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:37:23.733 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:37:23.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:37:23.733 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:37:23.733 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:37:23.733 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 3 2024-09-17T18:37:23.733 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:37:23.733 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:37:23.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:37:23.733 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:37:23.733 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:37:23.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.711+0000 7fc09a42b640 10 librados: watch_flush enter 2024-09-17T18:37:23.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.711+0000 7fc09a42b640 10 librados: watch_flush exit 2024-09-17T18:37:23.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.711+0000 7fc09a42b640 20 client.114120.objecter shutdown clearing up homeless session... 2024-09-17T18:37:23.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.711+0000 7fc09a42b640 10 client.114120.objecter successfully canceled tick 2024-09-17T18:37:23.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.711+0000 7fc09a42b640 1 -- 172.21.15.121:0/1236720723 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc0800401a0 msgr2=0x7fc080042660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:23.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.711+0000 7fc09a42b640 1 --2- 172.21.15.121:0/1236720723 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc0800401a0 0x7fc080042660 secure :-1 s=READY pgs=4092 cs=0 l=1 rev1=1 crypto rx=0x7fc08c034000 tx=0x7fc08c02f000 comp rx=0 tx=0).stop 2024-09-17T18:37:23.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.711+0000 7fc09a42b640 10 monclient: shutdown 2024-09-17T18:37:23.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.711+0000 7fc09a42b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:23.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.711+0000 7fc09a42b640 1 -- 172.21.15.121:0/1236720723 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc09409da70 msgr2=0x7fc09412fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:23.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.711+0000 7fc09a42b640 1 --2- 172.21.15.121:0/1236720723 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc09409da70 0x7fc09412fa30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc08400c460 tx=0x7fc08400c920 comp rx=0 tx=0).stop 2024-09-17T18:37:23.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.711+0000 7fc0989c8640 1 -- 172.21.15.121:0/1236720723 reap_dead start 2024-09-17T18:37:23.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.711+0000 7fc0989c8640 1 -- 172.21.15.121:0/1236720723 reap_dead start 2024-09-17T18:37:23.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.711+0000 7fc09a42b640 1 -- 172.21.15.121:0/1236720723 shutdown_connections 2024-09-17T18:37:23.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.711+0000 7fc09a42b640 1 -- 172.21.15.121:0/1236720723 >> 172.21.15.121:0/1236720723 conn(0x7fc0940a7830 msgr2=0x7fc0940a4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:23.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.712+0000 7fc09a42b640 1 -- 172.21.15.121:0/1236720723 shutdown_connections 2024-09-17T18:37:23.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.712+0000 7fc09a42b640 1 -- 172.21.15.121:0/1236720723 wait complete. 2024-09-17T18:37:23.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.712+0000 7fc09a42b640 1 librados: shutdown 2024-09-17T18:37:23.736 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:37:23.736 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T18:37:23.736 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T18:37:23.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.831+0000 7f075f3d3640 10 monclient: get_monmap_and_config 2024-09-17T18:37:23.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.831+0000 7f075f3d3640 10 monclient: build_initial_monmap 2024-09-17T18:37:23.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.831+0000 7f075f3d3640 10 monclient: monmap: 2024-09-17T18:37:23.833 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:23.833 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:23.833 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:37:23.832019+0000 2024-09-17T18:37:23.833 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:37:23.832019+0000 2024-09-17T18:37:23.833 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:23.834 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:23.834 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:37:23.834 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:37:23.834 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:37:23.834 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:37:23.834 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:37:23.834 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:23.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.831+0000 7f075f3d3640 1 Processor -- start 2024-09-17T18:37:23.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.832+0000 7f075f3d3640 1 -- start start 2024-09-17T18:37:23.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.832+0000 7f075f3d3640 10 monclient: init 2024-09-17T18:37:23.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.832+0000 7f075f3d3640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:23.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.832+0000 7f075f3d3640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-17T18:37:23.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.832+0000 7f075f3d3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0758135440 addr v1:172.21.15.121:6789/0 2024-09-17T18:37:23.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.832+0000 7f075f3d3640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0758138880 0x7f075813d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:23.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.832+0000 7f075f3d3640 10 monclient(hunting): picked mon.noname-a con 0x7f0758138880 addr v2:172.21.15.121:3300/0 2024-09-17T18:37:23.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.832+0000 7f075f3d3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f075813dc00 addr v1:172.21.15.173:6789/0 2024-09-17T18:37:23.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.832+0000 7f075f3d3640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07580a13a0 con 0x7f0758135440 2024-09-17T18:37:23.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.832+0000 7f075f3d3640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0758077300 con 0x7f075813dc00 2024-09-17T18:37:23.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.832+0000 7f075f3d3640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:23.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.832+0000 7f075f3d3640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f075806fc80 con 0x7f0758138880 2024-09-17T18:37:23.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.832+0000 7f075f3d3640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:23.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.832+0000 7f075f3d3640 10 monclient(hunting): authenticate will time out at 1847.548695s 2024-09-17T18:37:23.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.832+0000 7f075c96e640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0758138880 0x7f075813d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:23.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.832+0000 7f075d16f640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f0758135440 0x7f0758135820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:47324/0 (socket says 172.21.15.121:47324) 2024-09-17T18:37:23.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.832+0000 7f075c96e640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0758138880 0x7f075813d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51198/0 (socket says 172.21.15.121:51198) 2024-09-17T18:37:23.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.832+0000 7f075d16f640 1 -- 172.21.15.121:0/1627027488 learned_addr learned my addr 172.21.15.121:0/1627027488 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:37:23.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.832+0000 7f075c96e640 10 monclient(hunting): get_auth_request con 0x7f0758138880 auth_method 0 2024-09-17T18:37:23.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.832+0000 7f075c96e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:23.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.832+0000 7f075c96e640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:23.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.832+0000 7f075c96e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:23.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.832+0000 7f075c96e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:23.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.832+0000 7f075c96e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:23.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.832+0000 7f075c96e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:23.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.833+0000 7f075c96e640 10 monclient(hunting): handle_auth_done global_id 114123 payload 274 2024-09-17T18:37:23.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.833+0000 7f075c96e640 1 -- 172.21.15.121:0/1627027488 >> v1:172.21.15.173:6789/0 conn(0x7f075813dc00 legacy=0x7f075813fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:23.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.833+0000 7f075c96e640 1 -- 172.21.15.121:0/1627027488 >> v1:172.21.15.121:6789/0 conn(0x7f0758135440 legacy=0x7f0758135820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:23.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.833+0000 7f075c96e640 10 monclient: _finish_hunting 0 2024-09-17T18:37:23.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.833+0000 7f075c96e640 1 monclient: found mon.noname-a 2024-09-17T18:37:23.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.833+0000 7f075c96e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:23.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.833+0000 7f075c96e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:37:23.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.833+0000 7f075c96e640 1 -- 172.21.15.121:0/1627027488 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f075806ff60 con 0x7f0758138880 2024-09-17T18:37:23.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.835+0000 7f075c96e640 1 -- 172.21.15.121:0/1627027488 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0758138880 msgr2=0x7f075813d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:23.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.835+0000 7f075c96e640 1 --2- 172.21.15.121:0/1627027488 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0758138880 0x7f075813d6c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f07580a1aa0 tx=0x7f0754029fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d0da5c4e1c71ffe server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:23.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.836+0000 7f074bfff640 1 -- 172.21.15.121:0/1627027488 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0754037070 con 0x7f0758138880 2024-09-17T18:37:23.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.836+0000 7f074bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:23.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.836+0000 7f074bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:37:23.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.836+0000 7f074bfff640 10 monclient: dump: 2024-09-17T18:37:23.845 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:23.845 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:23.845 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.845 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.845 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:23.845 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:23.845 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:23.845 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:23.845 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:23.845 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:23.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.836+0000 7f074bfff640 10 monclient: _finish_auth 0 2024-09-17T18:37:23.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.836+0000 7f074bfff640 10 monclient: _check_auth_tickets 2024-09-17T18:37:23.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.836+0000 7f074bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:23.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.836+0000 7f075f3d3640 5 monclient: authenticate success, global_id 114123 2024-09-17T18:37:23.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.836+0000 7f075f3d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:37:23.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.836+0000 7f074bfff640 1 -- 172.21.15.121:0/1627027488 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0754032440 con 0x7f0758138880 2024-09-17T18:37:23.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.836+0000 7f074bfff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:23.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.836+0000 7f074bfff640 1 -- 172.21.15.121:0/1627027488 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0754032810 con 0x7f0758138880 2024-09-17T18:37:23.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.836+0000 7f075f3d3640 10 monclient: get_monmap_and_config success 2024-09-17T18:37:23.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.836+0000 7f075f3d3640 4 set_mon_vals no callback set 2024-09-17T18:37:23.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.836+0000 7f074bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:23.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.836+0000 7f074bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:37:23.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.836+0000 7f074bfff640 10 monclient: dump: 2024-09-17T18:37:23.846 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:23.846 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:23.846 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.846 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.847 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:23.847 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:23.847 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:23.847 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:23.847 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:23.847 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:23.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.836+0000 7f075f3d3640 10 monclient: shutdown 2024-09-17T18:37:23.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.836+0000 7f075f3d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:23.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.836+0000 7f075f3d3640 1 -- 172.21.15.121:0/1627027488 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0758138880 msgr2=0x7f075813d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:23.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.836+0000 7f075f3d3640 1 --2- 172.21.15.121:0/1627027488 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0758138880 0x7f075813d6c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f07580a1aa0 tx=0x7f0754029fb0 comp rx=0 tx=0).stop 2024-09-17T18:37:23.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.836+0000 7f075f3d3640 1 -- 172.21.15.121:0/1627027488 shutdown_connections 2024-09-17T18:37:23.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.836+0000 7f075f3d3640 1 --2- 172.21.15.121:0/1627027488 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0758138880 0x7f075813d6c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:23.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.836+0000 7f075f3d3640 1 -- 172.21.15.121:0/1627027488 >> 172.21.15.121:0/1627027488 conn(0x7f07580a49c0 msgr2=0x7f07580768d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:23.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.836+0000 7f075f3d3640 1 -- 172.21.15.121:0/1627027488 shutdown_connections 2024-09-17T18:37:23.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.836+0000 7f075f3d3640 1 -- 172.21.15.121:0/1627027488 wait complete. 2024-09-17T18:37:23.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.837+0000 7f075f3d3640 10 monclient: build_initial_monmap 2024-09-17T18:37:23.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.837+0000 7f075f3d3640 10 monclient: monmap: 2024-09-17T18:37:23.848 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:23.848 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:37:23.848 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:37:23.848 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:37:23.848 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:23.848 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:23.848 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:37:23.848 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:37:23.848 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:37:23.848 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:23.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.837+0000 7f075f3d3640 1 librados: starting msgr at 2024-09-17T18:37:23.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.837+0000 7f075f3d3640 1 librados: starting objecter 2024-09-17T18:37:23.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.837+0000 7f075f3d3640 1 Processor -- start 2024-09-17T18:37:23.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.837+0000 7f075f3d3640 1 -- start start 2024-09-17T18:37:23.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.837+0000 7f075f3d3640 1 librados: setting wanted keys 2024-09-17T18:37:23.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.837+0000 7f075f3d3640 1 librados: calling monclient init 2024-09-17T18:37:23.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.837+0000 7f075f3d3640 10 monclient: init 2024-09-17T18:37:23.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075f3d3640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:23.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075f3d3640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T18:37:23.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075f3d3640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0758135440 0x7f0758133f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:23.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075f3d3640 10 monclient(hunting): picked mon.noname-b con 0x7f0758135440 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:37:23.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075f3d3640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0758138880 0x7f0758070760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:23.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075f3d3640 10 monclient(hunting): picked mon.noname-c con 0x7f0758138880 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:23.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075f3d3640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f075813dc00 0x7f0758070ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:23.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075f3d3640 10 monclient(hunting): picked mon.noname-a con 0x7f075813dc00 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:37:23.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075f3d3640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:23.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075f3d3640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f075806d6a0 con 0x7f075813dc00 2024-09-17T18:37:23.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075f3d3640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:23.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075f3d3640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f07580a1e60 con 0x7f0758135440 2024-09-17T18:37:23.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075f3d3640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:23.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075f3d3640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f075806d820 con 0x7f0758138880 2024-09-17T18:37:23.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075f3d3640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:23.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075f3d3640 10 monclient(hunting): authenticate will time out at 1847.554233s 2024-09-17T18:37:23.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075c96e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0758138880 0x7f0758070760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:23.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075d970640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f075813dc00 0x7f0758070ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:23.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075c96e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0758138880 0x7f0758070760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:36732/0 (socket says 172.21.15.121:36732) 2024-09-17T18:37:23.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075d970640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f075813dc00 0x7f0758070ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51204/0 (socket says 172.21.15.121:51204) 2024-09-17T18:37:23.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075c96e640 1 -- 172.21.15.121:0/3247830537 learned_addr learned my addr 172.21.15.121:0/3247830537 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:37:23.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075c96e640 10 monclient(hunting): get_auth_request con 0x7f0758138880 auth_method 0 2024-09-17T18:37:23.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075d16f640 1 --2- 172.21.15.121:0/3247830537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0758135440 0x7f0758133f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:23.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075c96e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:23.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075c96e640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:23.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075c96e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:23.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075d970640 10 monclient(hunting): get_auth_request con 0x7f075813dc00 auth_method 0 2024-09-17T18:37:23.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075d970640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:23.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075d970640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:23.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075d970640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:23.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075d16f640 10 monclient(hunting): get_auth_request con 0x7f0758135440 auth_method 0 2024-09-17T18:37:23.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075d16f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:23.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075d16f640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:23.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075d16f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:23.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075c96e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:23.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075c96e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:23.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075c96e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:23.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075d970640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:23.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075d970640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:23.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075d970640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:23.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075d16f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:23.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075d16f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:23.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075d16f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:23.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075d970640 10 monclient(hunting): handle_auth_done global_id 114126 payload 824 2024-09-17T18:37:23.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075d970640 1 -- 172.21.15.121:0/3247830537 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0758138880 msgr2=0x7f0758070760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:23.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075d970640 1 --2- 172.21.15.121:0/3247830537 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0758138880 0x7f0758070760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:23.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075d970640 1 -- 172.21.15.121:0/3247830537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0758135440 msgr2=0x7f0758133f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:23.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075d970640 1 --2- 172.21.15.121:0/3247830537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0758135440 0x7f0758133f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:23.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075d970640 10 monclient: _finish_hunting 0 2024-09-17T18:37:23.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075d970640 1 monclient: found mon.noname-a 2024-09-17T18:37:23.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075d970640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:23.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075d970640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:37:23.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075d970640 1 -- 172.21.15.121:0/3247830537 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0758074350 con 0x7f075813dc00 2024-09-17T18:37:23.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.838+0000 7f075c96e640 1 --2- 172.21.15.121:0/3247830537 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0758138880 0x7f0758070760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:37:23.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.840+0000 7f075d970640 1 -- 172.21.15.121:0/3247830537 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f075813dc00 msgr2=0x7f0758070ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:23.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.840+0000 7f075d970640 1 --2- 172.21.15.121:0/3247830537 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f075813dc00 0x7f0758070ca0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f075000c780 tx=0x7f075000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:23.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.840+0000 7f0749ffb640 10 client.?.objecter ms_handle_connect 0x7f075813dc00 2024-09-17T18:37:23.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.840+0000 7f0749ffb640 10 client.?.objecter resend_mon_ops 2024-09-17T18:37:23.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.841+0000 7f0749ffb640 1 -- 172.21.15.121:0/3247830537 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0750016020 con 0x7f075813dc00 2024-09-17T18:37:23.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.841+0000 7f0749ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:23.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.841+0000 7f0749ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:37:23.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.841+0000 7f0749ffb640 10 monclient: dump: 2024-09-17T18:37:23.853 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:23.853 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:23.853 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.853 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.853 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:23.853 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:23.853 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:23.854 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:23.854 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:23.854 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:23.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.841+0000 7f0749ffb640 10 monclient: _finish_auth 0 2024-09-17T18:37:23.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.841+0000 7f0749ffb640 10 monclient: _check_auth_tickets 2024-09-17T18:37:23.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.841+0000 7f0749ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:23.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.841+0000 7f075f3d3640 5 monclient: authenticate success, global_id 114126 2024-09-17T18:37:23.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.841+0000 7f0749ffb640 1 -- 172.21.15.121:0/3247830537 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f075000ac60 con 0x7f075813dc00 2024-09-17T18:37:23.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.841+0000 7f0749ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:23.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.841+0000 7f0749ffb640 1 -- 172.21.15.121:0/3247830537 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0750012660 con 0x7f075813dc00 2024-09-17T18:37:23.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.841+0000 7f075f3d3640 10 monclient: _renew_subs 2024-09-17T18:37:23.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.841+0000 7f075f3d3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:23.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.841+0000 7f075f3d3640 1 -- 172.21.15.121:0/3247830537 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0758071ff0 con 0x7f075813dc00 2024-09-17T18:37:23.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.841+0000 7f0749ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:23.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.841+0000 7f0749ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:37:23.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.841+0000 7f074b7fe640 4 set_mon_vals no callback set 2024-09-17T18:37:23.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.841+0000 7f0749ffb640 10 monclient: dump: 2024-09-17T18:37:23.855 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:23.855 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:23.855 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.855 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.855 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:23.855 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:23.855 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:23.855 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:23.855 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:23.855 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:23.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.841+0000 7f075f3d3640 10 client.114126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:37:23.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.841+0000 7f075f3d3640 10 monclient: _renew_subs 2024-09-17T18:37:23.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.841+0000 7f075f3d3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:23.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.841+0000 7f075f3d3640 1 -- 172.21.15.121:0/3247830537 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0758072b50 con 0x7f075813dc00 2024-09-17T18:37:23.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.841+0000 7f075f3d3640 1 librados: init done 2024-09-17T18:37:23.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.841+0000 7f0749ffb640 1 -- 172.21.15.121:0/3247830537 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f0750019020 con 0x7f075813dc00 2024-09-17T18:37:23.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.842+0000 7f075f3d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:23.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.842+0000 7f075f3d3640 1 --2- 172.21.15.121:0/3247830537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07581344c0 0x7f07580711e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:23.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.842+0000 7f075f3d3640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:23.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.842+0000 7f075f3d3640 1 -- 172.21.15.121:0/3247830537 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0758073bb0 con 0x7f07581344c0 2024-09-17T18:37:23.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.842+0000 7f0749ffb640 1 --2- 172.21.15.121:0/3247830537 >> v2:172.21.15.173:6812/2423786037 conn(0x7f073c040170 0x7f073c042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:23.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.842+0000 7f0749ffb640 1 -- 172.21.15.121:0/3247830537 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(679..679 src has 247..679) ==== 4685+0+0 (secure 0 0 0) 0x7f0750053340 con 0x7f075813dc00 2024-09-17T18:37:23.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.842+0000 7f0749ffb640 10 client.114126.objecter ms_dispatch 0x7f0758000b90 osd_map(679..679 src has 247..679) 2024-09-17T18:37:23.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.842+0000 7f0749ffb640 3 client.114126.objecter handle_osd_map got epochs [679,679] > 0 2024-09-17T18:37:23.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.842+0000 7f0749ffb640 3 client.114126.objecter handle_osd_map decoding full epoch 679 2024-09-17T18:37:23.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.842+0000 7f0749ffb640 20 client.114126.objecter dump_active .. 0 homeless 2024-09-17T18:37:23.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.842+0000 7f075d16f640 1 --2- 172.21.15.121:0/3247830537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07581344c0 0x7f07580711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:23.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.842+0000 7f075d16f640 10 monclient: get_auth_request con 0x7f07581344c0 auth_method 0 2024-09-17T18:37:23.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.842+0000 7f075d16f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:23.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.842+0000 7f075d16f640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:23.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.842+0000 7f075d16f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:23.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.842+0000 7f075c96e640 1 --2- 172.21.15.121:0/3247830537 >> v2:172.21.15.173:6812/2423786037 conn(0x7f073c040170 0x7f073c042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:23.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.842+0000 7f075c96e640 10 monclient: get_auth_request con 0x7f073c040170 auth_method 0 2024-09-17T18:37:23.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.842+0000 7f075d16f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:23.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.842+0000 7f075d16f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:23.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.842+0000 7f075d16f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:23.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.842+0000 7f075d16f640 10 monclient(hunting): handle_auth_done global_id 84322 payload 824 2024-09-17T18:37:23.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.842+0000 7f075d16f640 1 -- 172.21.15.121:0/3247830537 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f07581358a0 con 0x7f07581344c0 2024-09-17T18:37:23.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.844+0000 7f0749ffb640 10 client.114126.objecter ms_handle_connect 0x7f07581344c0 2024-09-17T18:37:23.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.844+0000 7f075d16f640 1 -- 172.21.15.121:0/3247830537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07581344c0 msgr2=0x7f07580711e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:23.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.844+0000 7f0749ffb640 10 client.114126.objecter resend_mon_ops 2024-09-17T18:37:23.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.844+0000 7f075d16f640 1 --2- 172.21.15.121:0/3247830537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07581344c0 0x7f07580711e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f074c00ea70 tx=0x7f074c00ef30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:23.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.845+0000 7f0749ffb640 10 client.114126.objecter ms_handle_connect 0x7f073c040170 2024-09-17T18:37:23.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.845+0000 7f075c96e640 1 --2- 172.21.15.121:0/3247830537 >> v2:172.21.15.173:6812/2423786037 conn(0x7f073c040170 0x7f073c042630 secure :-1 s=READY pgs=4093 cs=0 l=1 rev1=1 crypto rx=0x7f07580736d0 tx=0x7f075402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:23.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.845+0000 7f0749ffb640 1 -- 172.21.15.121:0/3247830537 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f074c01a070 con 0x7f07581344c0 2024-09-17T18:37:23.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.845+0000 7f0749ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:23.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.845+0000 7f0749ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:37:23.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.845+0000 7f0749ffb640 10 monclient: dump: 2024-09-17T18:37:23.860 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:23.860 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:23.860 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.860 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.860 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:23.860 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:23.861 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:23.861 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:23.861 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:23.861 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:23.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.845+0000 7f0749ffb640 1 -- 172.21.15.121:0/3247830537 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f074c00cc50 con 0x7f07581344c0 2024-09-17T18:37:23.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.845+0000 7f0749ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:23.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.845+0000 7f0749ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:37:23.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.845+0000 7f0749ffb640 1 -- 172.21.15.121:0/3247830537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07581344c0 msgr2=0x7f07580711e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:23.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.845+0000 7f0749ffb640 1 --2- 172.21.15.121:0/3247830537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07581344c0 0x7f07580711e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f074c00ea70 tx=0x7f074c00ef30 comp rx=0 tx=0).stop 2024-09-17T18:37:23.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.845+0000 7f0749ffb640 1 -- 172.21.15.121:0/3247830537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07581344c0 msgr2=0x7f07580711e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:37:23.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.845+0000 7f0749ffb640 1 --2- 172.21.15.121:0/3247830537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07581344c0 0x7f07580711e0 secure :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f074c00ea70 tx=0x7f074c00ef30 comp rx=0 tx=0).stop 2024-09-17T18:37:23.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.851+0000 7f075f3d3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T18:37:23.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.851+0000 7f075f3d3640 1 --2- 172.21.15.121:0/3247830537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0758071720 0x7f0758139470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:23.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.851+0000 7f075f3d3640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:23.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.851+0000 7f075f3d3640 1 -- 172.21.15.121:0/3247830537 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0758071b30 con 0x7f0758071720 2024-09-17T18:37:23.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.851+0000 7f075d16f640 1 --2- 172.21.15.121:0/3247830537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0758071720 0x7f0758139470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:23.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.851+0000 7f075d16f640 10 monclient: get_auth_request con 0x7f0758071720 auth_method 0 2024-09-17T18:37:23.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.851+0000 7f075d16f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:23.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.851+0000 7f075d16f640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:23.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.851+0000 7f075d16f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:23.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.852+0000 7f075d16f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:23.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.852+0000 7f075d16f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:23.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.852+0000 7f075d16f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:23.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.852+0000 7f075d16f640 10 monclient(hunting): handle_auth_done global_id 84325 payload 824 2024-09-17T18:37:23.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.852+0000 7f075d16f640 1 -- 172.21.15.121:0/3247830537 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0758073720 con 0x7f0758071720 2024-09-17T18:37:23.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.854+0000 7f075d16f640 1 -- 172.21.15.121:0/3247830537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0758071720 msgr2=0x7f0758139470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:23.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.854+0000 7f075d16f640 1 --2- 172.21.15.121:0/3247830537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0758071720 0x7f0758139470 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f074c017000 tx=0x7f074c004940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:23.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.854+0000 7f0749ffb640 10 client.114126.objecter ms_handle_connect 0x7f0758071720 2024-09-17T18:37:23.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.854+0000 7f0749ffb640 10 client.114126.objecter resend_mon_ops 2024-09-17T18:37:23.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.854+0000 7f0749ffb640 1 -- 172.21.15.121:0/3247830537 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f074c01a070 con 0x7f0758071720 2024-09-17T18:37:23.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.854+0000 7f0749ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:23.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.854+0000 7f0749ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:37:23.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.854+0000 7f0749ffb640 10 monclient: dump: 2024-09-17T18:37:23.863 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:23.863 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:23.863 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.863 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.863 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:23.863 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:23.863 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:23.863 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:23.863 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:23.863 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:23.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.855+0000 7f0749ffb640 1 -- 172.21.15.121:0/3247830537 <== mon.1 v2:172.21.15.173:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7f074c01b040 con 0x7f0758071720 2024-09-17T18:37:23.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.855+0000 7f0749ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T18:37:23.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.855+0000 7f0749ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:37:23.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.855+0000 7f0749ffb640 1 -- 172.21.15.121:0/3247830537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0758071720 msgr2=0x7f0758139470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:23.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.855+0000 7f0749ffb640 1 --2- 172.21.15.121:0/3247830537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0758071720 0x7f0758139470 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f074c017000 tx=0x7f074c004940 comp rx=0 tx=0).stop 2024-09-17T18:37:23.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.855+0000 7f0749ffb640 1 -- 172.21.15.121:0/3247830537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0758071720 msgr2=0x7f0758139470 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:37:23.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.855+0000 7f0749ffb640 1 --2- 172.21.15.121:0/3247830537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0758071720 0x7f0758139470 secure :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f074c017000 tx=0x7f074c004940 comp rx=0 tx=0).stop 2024-09-17T18:37:23.864 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:37:23.864 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:37:23.864 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:37:23.864 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-17T18:37:23.864 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 66, 2024-09-17T18:37:23.864 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 368952, 2024-09-17T18:37:23.864 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-17T18:37:23.865 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-17T18:37:23.865 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-17T18:37:23.865 INFO:teuthology.orchestra.run.smithi121.stdout: 2 2024-09-17T18:37:23.865 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:37:23.865 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 1, 2024-09-17T18:37:23.865 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:37:23.865 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-17T18:37:23.865 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-17T18:37:23.865 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:37:23.865 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:37:23.865 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:37:23.865 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:37:23.865 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:37:23.865 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:37:23.865 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:37:23.866 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:37:23.866 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:37:23.866 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:37:23.866 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:37:23.866 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:37:23.866 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-17T18:37:23.866 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-17T18:37:23.866 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:37:23.866 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:37:23.866 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:37:23.866 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:37:23.866 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:37:23.866 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:37:23.866 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:37:23.866 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:37:23.867 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:37:23.867 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:37:23.867 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:37:23.867 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:37:23.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:37:23.867 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-17T18:37:23.867 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-17T18:37:23.867 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-17T18:37:23.867 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-17T18:37:23.867 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-17T18:37:23.867 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b", 2024-09-17T18:37:23.867 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:37:23.867 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-17T18:18:20.001795Z", 2024-09-17T18:37:23.867 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-17T18:37:23.867 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-17T18:37:23.868 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 1, 2024-09-17T18:37:23.868 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-17T18:37:23.868 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-17T18:37:23.868 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-17T18:37:23.868 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-17T18:37:23.868 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-17T18:37:23.868 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-17T18:37:23.868 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-17T18:37:23.868 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-17T18:37:23.868 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-17T18:37:23.868 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-17T18:37:23.868 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-17T18:37:23.868 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-17T18:37:23.868 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-17T18:37:23.869 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-17T18:37:23.869 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-17T18:37:23.869 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-17T18:37:23.869 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-17T18:37:23.869 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:37:23.869 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-17T18:37:23.869 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:37:23.869 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-17T18:37:23.869 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:37:23.869 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-17T18:37:23.869 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-17T18:37:23.869 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:37:23.869 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:37:23.869 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:37:23.870 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:37:23.870 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3300", 2024-09-17T18:37:23.870 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:37:23.870 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:37:23.870 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:37:23.870 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:37:23.870 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-17T18:37:23.870 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:37:23.870 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:37:23.870 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:37:23.870 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:37:23.870 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-17T18:37:23.870 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-17T18:37:23.871 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:37:23.871 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:37:23.871 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:37:23.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:37:23.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:37:23.871 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-17T18:37:23.871 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-17T18:37:23.871 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:37:23.871 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:37:23.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:37:23.871 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:37:23.871 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:3300", 2024-09-17T18:37:23.871 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:37:23.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:37:23.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:37:23.872 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:37:23.872 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789", 2024-09-17T18:37:23.872 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:37:23.872 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:37:23.872 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:37:23.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:37:23.872 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.173:6789/0", 2024-09-17T18:37:23.872 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.173:6789/0", 2024-09-17T18:37:23.872 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:37:23.872 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:37:23.872 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:37:23.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:37:23.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:37:23.873 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-17T18:37:23.873 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-17T18:37:23.873 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-17T18:37:23.873 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-17T18:37:23.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:37:23.873 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-09-17T18:37:23.873 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:3301", 2024-09-17T18:37:23.873 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:37:23.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:37:23.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:37:23.873 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-17T18:37:23.873 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-17T18:37:23.873 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-17T18:37:23.873 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:37:23.873 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:37:23.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:37:23.874 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-17T18:37:23.874 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-17T18:37:23.874 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-17T18:37:23.874 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-17T18:37:23.874 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-17T18:37:23.874 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:37:23.874 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:37:23.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:37:23.874 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-17T18:37:23.874 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-17T18:37:23.874 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:37:23.874 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:37:23.874 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:37:23.875 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:37:23.875 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:37:23.875 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:37:23.875 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-17T18:37:23.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:37:23.875 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:37:23.875 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:37:23.875 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 3 2024-09-17T18:37:23.875 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:37:23.875 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-17T18:37:23.875 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": [ 2024-09-17T18:37:23.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-17T18:37:23.875 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T18:37:23.875 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-17T18:37:23.876 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-17T18:37:23.876 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-17T18:37:23.876 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-17T18:37:23.876 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-17T18:37:23.876 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-17T18:37:23.876 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:37:23.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.855+0000 7f075f3d3640 10 librados: watch_flush enter 2024-09-17T18:37:23.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.855+0000 7f075f3d3640 10 librados: watch_flush exit 2024-09-17T18:37:23.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.855+0000 7f075f3d3640 20 client.114126.objecter shutdown clearing up homeless session... 2024-09-17T18:37:23.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.855+0000 7f075f3d3640 10 client.114126.objecter successfully canceled tick 2024-09-17T18:37:23.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.855+0000 7f075f3d3640 1 -- 172.21.15.121:0/3247830537 >> v2:172.21.15.173:6812/2423786037 conn(0x7f073c040170 msgr2=0x7f073c042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:23.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.855+0000 7f075f3d3640 1 --2- 172.21.15.121:0/3247830537 >> v2:172.21.15.173:6812/2423786037 conn(0x7f073c040170 0x7f073c042630 secure :-1 s=READY pgs=4093 cs=0 l=1 rev1=1 crypto rx=0x7f07580736d0 tx=0x7f075402e000 comp rx=0 tx=0).stop 2024-09-17T18:37:23.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.855+0000 7f075f3d3640 10 monclient: shutdown 2024-09-17T18:37:23.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.855+0000 7f075f3d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:23.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.855+0000 7f075f3d3640 1 -- 172.21.15.121:0/3247830537 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f075813dc00 msgr2=0x7f0758070ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:23.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.855+0000 7f075d970640 1 -- 172.21.15.121:0/3247830537 reap_dead start 2024-09-17T18:37:23.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.855+0000 7f075f3d3640 1 --2- 172.21.15.121:0/3247830537 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f075813dc00 0x7f0758070ca0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f075000c780 tx=0x7f075000cc40 comp rx=0 tx=0).stop 2024-09-17T18:37:23.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.855+0000 7f075f3d3640 1 -- 172.21.15.121:0/3247830537 shutdown_connections 2024-09-17T18:37:23.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.855+0000 7f075f3d3640 1 --2- 172.21.15.121:0/3247830537 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f075813dc00 0x7f0758070ca0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:23.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.855+0000 7f075f3d3640 1 -- 172.21.15.121:0/3247830537 >> 172.21.15.121:0/3247830537 conn(0x7f07580a49c0 msgr2=0x7f07580a5e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:23.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.855+0000 7f075f3d3640 1 -- 172.21.15.121:0/3247830537 shutdown_connections 2024-09-17T18:37:23.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.855+0000 7f075f3d3640 1 -- 172.21.15.121:0/3247830537 wait complete. 2024-09-17T18:37:23.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.855+0000 7f075f3d3640 1 librados: shutdown 2024-09-17T18:37:23.879 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T18:37:23.879 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-17T18:37:23.879 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-17T18:37:23.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.974+0000 7f644a2bf640 10 monclient: get_monmap_and_config 2024-09-17T18:37:23.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.974+0000 7f644a2bf640 10 monclient: build_initial_monmap 2024-09-17T18:37:23.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.974+0000 7f644a2bf640 10 monclient: monmap: 2024-09-17T18:37:23.976 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:23.976 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:23.976 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:37:23.974615+0000 2024-09-17T18:37:23.976 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:37:23.974615+0000 2024-09-17T18:37:23.976 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:23.976 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:23.976 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:37:23.976 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:37:23.976 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:37:23.976 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:37:23.977 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:37:23.977 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:23.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.974+0000 7f644a2bf640 1 Processor -- start 2024-09-17T18:37:23.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.974+0000 7f644a2bf640 1 -- start start 2024-09-17T18:37:23.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.974+0000 7f644a2bf640 10 monclient: init 2024-09-17T18:37:23.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.975+0000 7f644a2bf640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:23.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.975+0000 7f644a2bf640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-17T18:37:23.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.975+0000 7f644a2bf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f64441386b0 addr v1:172.21.15.121:6789/0 2024-09-17T18:37:23.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.975+0000 7f644a2bf640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f644406ec00 0x7f6444138a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:23.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.975+0000 7f644a2bf640 10 monclient(hunting): picked mon.noname-a con 0x7f644406ec00 addr v2:172.21.15.121:3300/0 2024-09-17T18:37:23.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.975+0000 7f644a2bf640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f6444138fd0 0x7f644413fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:23.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.975+0000 7f644a2bf640 10 monclient(hunting): picked mon.noname-c con 0x7f6444138fd0 addr v2:172.21.15.173:3300/0 2024-09-17T18:37:23.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.975+0000 7f644a2bf640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64440a5570 con 0x7f64441386b0 2024-09-17T18:37:23.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.975+0000 7f644a2bf640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:23.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.975+0000 7f644a2bf640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f644406f830 con 0x7f644406ec00 2024-09-17T18:37:23.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.975+0000 7f644a2bf640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:23.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.975+0000 7f644a2bf640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f644406fb10 con 0x7f6444138fd0 2024-09-17T18:37:23.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.975+0000 7f644a2bf640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:23.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.975+0000 7f644a2bf640 10 monclient(hunting): authenticate will time out at 1847.691313s 2024-09-17T18:37:23.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.975+0000 7f64437fe640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f644406ec00 0x7f6444138a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:23.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.975+0000 7f6443fff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f64441386b0 0x7f64440707d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:47330/0 (socket says 172.21.15.121:47330) 2024-09-17T18:37:23.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.975+0000 7f6443fff640 1 -- 172.21.15.121:0/3121660070 learned_addr learned my addr 172.21.15.121:0/3121660070 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:37:23.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.975+0000 7f64437fe640 10 monclient(hunting): get_auth_request con 0x7f644406ec00 auth_method 0 2024-09-17T18:37:23.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.975+0000 7f64437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:23.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.975+0000 7f64437fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:23.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.975+0000 7f64437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:23.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.975+0000 7f64437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:23.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.975+0000 7f64437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:23.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.975+0000 7f64437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:23.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.975+0000 7f64437fe640 10 monclient(hunting): handle_auth_done global_id 114129 payload 274 2024-09-17T18:37:23.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.975+0000 7f64437fe640 1 -- 172.21.15.121:0/3121660070 >> v2:172.21.15.173:3300/0 conn(0x7f6444138fd0 msgr2=0x7f644413fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:23.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.975+0000 7f644885c640 1 --2- 172.21.15.121:0/3121660070 >> v2:172.21.15.173:3300/0 conn(0x7f6444138fd0 0x7f644413fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:23.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.975+0000 7f64437fe640 1 --2- 172.21.15.121:0/3121660070 >> v2:172.21.15.173:3300/0 conn(0x7f6444138fd0 0x7f644413fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:23.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.975+0000 7f64437fe640 1 -- 172.21.15.121:0/3121660070 >> v1:172.21.15.121:6789/0 conn(0x7f64441386b0 legacy=0x7f64440707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:23.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.975+0000 7f64437fe640 10 monclient: _finish_hunting 0 2024-09-17T18:37:23.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.975+0000 7f64437fe640 1 monclient: found mon.noname-a 2024-09-17T18:37:23.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.975+0000 7f64437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:23.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.975+0000 7f64437fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:37:23.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.975+0000 7f64437fe640 1 -- 172.21.15.121:0/3121660070 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6444139460 con 0x7f644406ec00 2024-09-17T18:37:23.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.975+0000 7f644885c640 10 monclient: get_auth_request con 0x7f6444138fd0 auth_method 0 2024-09-17T18:37:23.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.975+0000 7f644885c640 1 --2- 172.21.15.121:0/3121660070 >> v2:172.21.15.173:3300/0 conn(0x7f6444138fd0 0x7f644413fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T18:37:23.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.978+0000 7f64437fe640 1 -- 172.21.15.121:0/3121660070 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f644406ec00 msgr2=0x7f6444138a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:23.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.978+0000 7f64437fe640 1 --2- 172.21.15.121:0/3121660070 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f644406ec00 0x7f6444138a90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f64380097a0 tx=0x7f643802a010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7bfa8cd70cb61b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:23.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.978+0000 7f6442ffd640 1 -- 172.21.15.121:0/3121660070 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6438037070 con 0x7f644406ec00 2024-09-17T18:37:23.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.978+0000 7f6442ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:23.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.978+0000 7f6442ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:37:23.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.978+0000 7f6442ffd640 10 monclient: dump: 2024-09-17T18:37:23.989 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:23.989 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:23.989 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.989 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.989 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:23.989 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:23.989 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:23.989 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:23.989 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:23.989 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:23.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.978+0000 7f6442ffd640 10 monclient: _finish_auth 0 2024-09-17T18:37:23.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.978+0000 7f6442ffd640 10 monclient: _check_auth_tickets 2024-09-17T18:37:23.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.978+0000 7f6442ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:23.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.978+0000 7f6442ffd640 1 -- 172.21.15.121:0/3121660070 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6438032640 con 0x7f644406ec00 2024-09-17T18:37:23.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.978+0000 7f644a2bf640 5 monclient: authenticate success, global_id 114129 2024-09-17T18:37:23.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.978+0000 7f644a2bf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:37:23.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.978+0000 7f6442ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:23.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.978+0000 7f644a2bf640 10 monclient: get_monmap_and_config success 2024-09-17T18:37:23.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.978+0000 7f6442ffd640 1 -- 172.21.15.121:0/3121660070 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6438032a10 con 0x7f644406ec00 2024-09-17T18:37:23.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.978+0000 7f644a2bf640 4 set_mon_vals no callback set 2024-09-17T18:37:23.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.978+0000 7f6442ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:23.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.978+0000 7f6442ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:37:23.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.978+0000 7f6442ffd640 10 monclient: dump: 2024-09-17T18:37:23.990 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:23.990 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:23.991 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.991 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.991 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:23.991 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:23.991 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:23.991 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:23.991 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:23.991 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:23.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.979+0000 7f644a2bf640 10 monclient: shutdown 2024-09-17T18:37:23.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.979+0000 7f644a2bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:23.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.979+0000 7f644a2bf640 1 -- 172.21.15.121:0/3121660070 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f644406ec00 msgr2=0x7f6444138a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:23.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.979+0000 7f644a2bf640 1 --2- 172.21.15.121:0/3121660070 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f644406ec00 0x7f6444138a90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f64380097a0 tx=0x7f643802a010 comp rx=0 tx=0).stop 2024-09-17T18:37:23.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.979+0000 7f644a2bf640 1 -- 172.21.15.121:0/3121660070 shutdown_connections 2024-09-17T18:37:23.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.979+0000 7f644a2bf640 1 --2- 172.21.15.121:0/3121660070 >> v2:172.21.15.173:3300/0 conn(0x7f6444138fd0 0x7f644413fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:23.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.979+0000 7f644a2bf640 1 --2- 172.21.15.121:0/3121660070 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f644406ec00 0x7f6444138a90 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:23.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.979+0000 7f644a2bf640 1 -- 172.21.15.121:0/3121660070 >> 172.21.15.121:0/3121660070 conn(0x7f64440a91e0 msgr2=0x7f64440a95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:23.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.979+0000 7f644a2bf640 1 -- 172.21.15.121:0/3121660070 shutdown_connections 2024-09-17T18:37:23.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.979+0000 7f644a2bf640 1 -- 172.21.15.121:0/3121660070 wait complete. 2024-09-17T18:37:23.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.979+0000 7f644a2bf640 10 monclient: build_initial_monmap 2024-09-17T18:37:23.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.979+0000 7f644a2bf640 10 monclient: monmap: 2024-09-17T18:37:23.992 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:23.992 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:37:23.992 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:37:23.992 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:37:23.992 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:23.992 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:23.992 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:37:23.992 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:37:23.993 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:37:23.993 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:23.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.979+0000 7f644a2bf640 1 librados: starting msgr at 2024-09-17T18:37:23.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.979+0000 7f644a2bf640 1 librados: starting objecter 2024-09-17T18:37:23.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.980+0000 7f644a2bf640 1 Processor -- start 2024-09-17T18:37:23.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.980+0000 7f644a2bf640 1 -- start start 2024-09-17T18:37:23.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.980+0000 7f644a2bf640 1 librados: setting wanted keys 2024-09-17T18:37:23.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.980+0000 7f644a2bf640 1 librados: calling monclient init 2024-09-17T18:37:23.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.980+0000 7f644a2bf640 10 monclient: init 2024-09-17T18:37:23.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.980+0000 7f644a2bf640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:23.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.980+0000 7f644a2bf640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:37:23.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.980+0000 7f644a2bf640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f644406ec00 0x7f64440a2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:23.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.980+0000 7f644a2bf640 10 monclient(hunting): picked mon.noname-a con 0x7f644406ec00 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:37:23.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.980+0000 7f644a2bf640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64441386b0 0x7f64441cd680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:23.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.980+0000 7f644a2bf640 10 monclient(hunting): picked mon.noname-b con 0x7f64441386b0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:37:23.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.980+0000 7f644a2bf640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6444138fd0 0x7f64441cdbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:23.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.980+0000 7f644a2bf640 10 monclient(hunting): picked mon.noname-c con 0x7f6444138fd0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:23.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.980+0000 7f644a2bf640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:23.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.980+0000 7f644a2bf640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f64441404c0 con 0x7f644406ec00 2024-09-17T18:37:23.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.980+0000 7f644a2bf640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:23.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.980+0000 7f644a2bf640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6444140340 con 0x7f64441386b0 2024-09-17T18:37:23.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.980+0000 7f644a2bf640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:23.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.980+0000 7f644a2bf640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6444140640 con 0x7f6444138fd0 2024-09-17T18:37:23.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.980+0000 7f644a2bf640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:23.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.980+0000 7f644a2bf640 10 monclient(hunting): authenticate will time out at 1847.696726s 2024-09-17T18:37:23.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.980+0000 7f6443fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f644406ec00 0x7f64440a2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:23.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.980+0000 7f6443fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f644406ec00 0x7f64440a2ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51218/0 (socket says 172.21.15.121:51218) 2024-09-17T18:37:23.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.980+0000 7f6443fff640 1 -- 172.21.15.121:0/1902745303 learned_addr learned my addr 172.21.15.121:0/1902745303 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:37:23.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.980+0000 7f6443fff640 10 monclient(hunting): get_auth_request con 0x7f644406ec00 auth_method 0 2024-09-17T18:37:23.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.980+0000 7f6443fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:23.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.980+0000 7f6443fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:23.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.980+0000 7f6443fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:23.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.980+0000 7f644885c640 1 --2- 172.21.15.121:0/1902745303 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6444138fd0 0x7f64441cdbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:23.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.980+0000 7f644885c640 10 monclient(hunting): get_auth_request con 0x7f6444138fd0 auth_method 0 2024-09-17T18:37:23.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.980+0000 7f644885c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:23.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.980+0000 7f644885c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:23.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.980+0000 7f644885c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:23.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.980+0000 7f6443fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:23.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.980+0000 7f6443fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:23.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.980+0000 7f6443fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:23.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.980+0000 7f644885c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:23.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.980+0000 7f644885c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:23.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.980+0000 7f644885c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:23.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.980+0000 7f64437fe640 1 --2- 172.21.15.121:0/1902745303 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64441386b0 0x7f64441cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:23.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.981+0000 7f64437fe640 10 monclient(hunting): get_auth_request con 0x7f64441386b0 auth_method 0 2024-09-17T18:37:23.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.981+0000 7f64437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:23.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.981+0000 7f64437fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:23.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.981+0000 7f64437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:23.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.981+0000 7f6443fff640 10 monclient(hunting): handle_auth_done global_id 114132 payload 824 2024-09-17T18:37:23.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.981+0000 7f6443fff640 1 -- 172.21.15.121:0/1902745303 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6444138fd0 msgr2=0x7f64441cdbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:23.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.981+0000 7f6443fff640 1 --2- 172.21.15.121:0/1902745303 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6444138fd0 0x7f64441cdbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:23.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.981+0000 7f6443fff640 1 -- 172.21.15.121:0/1902745303 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64441386b0 msgr2=0x7f64441cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:23.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.981+0000 7f6443fff640 1 --2- 172.21.15.121:0/1902745303 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64441386b0 0x7f64441cd680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:23.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.981+0000 7f6443fff640 10 monclient: _finish_hunting 0 2024-09-17T18:37:23.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.981+0000 7f6443fff640 1 monclient: found mon.noname-a 2024-09-17T18:37:23.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.981+0000 7f6443fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:23.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.981+0000 7f6443fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:37:23.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.981+0000 7f6443fff640 1 -- 172.21.15.121:0/1902745303 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64440a0fe0 con 0x7f644406ec00 2024-09-17T18:37:23.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.981+0000 7f644885c640 1 --2- 172.21.15.121:0/1902745303 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6444138fd0 0x7f64441cdbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:37:23.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.983+0000 7f6443fff640 1 -- 172.21.15.121:0/1902745303 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f644406ec00 msgr2=0x7f64440a2ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:23.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.983+0000 7f6443fff640 1 --2- 172.21.15.121:0/1902745303 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f644406ec00 0x7f64440a2ee0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f643400c460 tx=0x7f643400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:23.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.983+0000 7f6440ff9640 10 client.?.objecter ms_handle_connect 0x7f644406ec00 2024-09-17T18:37:23.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.983+0000 7f6440ff9640 10 client.?.objecter resend_mon_ops 2024-09-17T18:37:23.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.983+0000 7f6440ff9640 1 -- 172.21.15.121:0/1902745303 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6434007070 con 0x7f644406ec00 2024-09-17T18:37:23.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.983+0000 7f6440ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:23.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.983+0000 7f6440ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:37:23.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.983+0000 7f6440ff9640 10 monclient: dump: 2024-09-17T18:37:23.997 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:23.997 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:23.997 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.997 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.997 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:23.998 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:23.998 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:23.998 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:23.998 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:23.998 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:23.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.983+0000 7f6440ff9640 10 monclient: _finish_auth 0 2024-09-17T18:37:23.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.983+0000 7f6440ff9640 10 monclient: _check_auth_tickets 2024-09-17T18:37:23.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.983+0000 7f6440ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:23.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.983+0000 7f6440ff9640 1 -- 172.21.15.121:0/1902745303 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6434007230 con 0x7f644406ec00 2024-09-17T18:37:23.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.983+0000 7f644a2bf640 5 monclient: authenticate success, global_id 114132 2024-09-17T18:37:23.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.983+0000 7f6440ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:23.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.983+0000 7f6440ff9640 1 -- 172.21.15.121:0/1902745303 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6434007620 con 0x7f644406ec00 2024-09-17T18:37:23.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.983+0000 7f644a2bf640 10 monclient: _renew_subs 2024-09-17T18:37:23.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.983+0000 7f644a2bf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:23.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.983+0000 7f644a2bf640 1 -- 172.21.15.121:0/1902745303 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f644413fcb0 con 0x7f644406ec00 2024-09-17T18:37:23.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.983+0000 7f6440ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:23.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.983+0000 7f6440ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:37:23.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.983+0000 7f6440ff9640 10 monclient: dump: 2024-09-17T18:37:23.999 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:23.999 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:23.999 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.999 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:23.999 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:23.999 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:23.999 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:23.999 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:23.999 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:23.999 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:23.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.983+0000 7f64427fc640 4 set_mon_vals no callback set 2024-09-17T18:37:24.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.983+0000 7f644a2bf640 10 client.114132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:37:24.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.983+0000 7f644a2bf640 10 monclient: _renew_subs 2024-09-17T18:37:24.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.983+0000 7f644a2bf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:24.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.983+0000 7f644a2bf640 1 -- 172.21.15.121:0/1902745303 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64440a2750 con 0x7f644406ec00 2024-09-17T18:37:24.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.983+0000 7f644a2bf640 1 librados: init done 2024-09-17T18:37:24.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.984+0000 7f6440ff9640 1 -- 172.21.15.121:0/1902745303 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f643401b400 con 0x7f644406ec00 2024-09-17T18:37:24.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.984+0000 7f644a2bf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:24.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.984+0000 7f644a2bf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:24.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.984+0000 7f644a2bf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:24.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.984+0000 7f644a2bf640 1 -- 172.21.15.121:0/1902745303 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6444138a90 con 0x7f644406ec00 2024-09-17T18:37:24.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.984+0000 7f6440ff9640 1 --2- 172.21.15.121:0/1902745303 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6430040170 0x7f6430042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:24.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.984+0000 7f6440ff9640 1 -- 172.21.15.121:0/1902745303 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(679..679 src has 247..679) ==== 4685+0+0 (secure 0 0 0) 0x7f6434053120 con 0x7f644406ec00 2024-09-17T18:37:24.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.984+0000 7f6440ff9640 10 client.114132.objecter ms_dispatch 0x7f6444000b90 osd_map(679..679 src has 247..679) 2024-09-17T18:37:24.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.984+0000 7f6440ff9640 3 client.114132.objecter handle_osd_map got epochs [679,679] > 0 2024-09-17T18:37:24.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.984+0000 7f6440ff9640 3 client.114132.objecter handle_osd_map decoding full epoch 679 2024-09-17T18:37:24.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.984+0000 7f6440ff9640 20 client.114132.objecter dump_active .. 0 homeless 2024-09-17T18:37:24.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.984+0000 7f64437fe640 1 --2- 172.21.15.121:0/1902745303 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6430040170 0x7f6430042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:24.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.984+0000 7f64437fe640 10 monclient: get_auth_request con 0x7f6430040170 auth_method 0 2024-09-17T18:37:24.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.987+0000 7f6440ff9640 10 client.114132.objecter ms_handle_connect 0x7f6430040170 2024-09-17T18:37:24.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.987+0000 7f64437fe640 1 --2- 172.21.15.121:0/1902745303 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6430040170 0x7f6430042630 secure :-1 s=READY pgs=4094 cs=0 l=1 rev1=1 crypto rx=0x7f6444074e20 tx=0x7f643802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:24.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.989+0000 7f6440ff9640 1 -- 172.21.15.121:0/1902745303 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6434051b80 con 0x7f644406ec00 2024-09-17T18:37:24.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.989+0000 7f6440ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:24.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:23.989+0000 7f6440ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:37:24.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:24.228+0000 7f644a2bf640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-17T18:37:24.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:24.228+0000 7f644a2bf640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-17T18:37:24.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:24.228+0000 7f644a2bf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:24.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:24.228+0000 7f644a2bf640 1 -- 172.21.15.121:0/1902745303 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f644406cd40 con 0x7f644406ec00 2024-09-17T18:37:24.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:24.229+0000 7f6440ff9640 1 -- 172.21.15.121:0/1902745303 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f643401b8d0 con 0x7f644406ec00 2024-09-17T18:37:24.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:24.229+0000 7f6440ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-17T18:37:24.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:24.229+0000 7f6440ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:37:24.232 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.230+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.232 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.230+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=2,health=12,kv=14,logm=51} crc {auth=2039214749,config=2586702516,health=3244156202,kv=2790417503,logm=4062412163}) 2024-09-17T18:37:24.232 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.230+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=19,config=1,health=12,kv=13,logm=55} crc {auth=1556859204,config=1747058506,health=3244156202,kv=3939032606,logm=4120329791}) 2024-09-17T18:37:24.232 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.230+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.232 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.230+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=2,health=12,kv=14,logm=51} crc {auth=2039214749,config=2586702516,health=3244156202,kv=2790417503,logm=4062412163}) 2024-09-17T18:37:24.232 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.230+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=19,config=2,health=12,kv=13,logm=54} crc {auth=3889604470,config=2586702516,health=3244156202,kv=1445218477,logm=255300182}) 2024-09-17T18:37:24.233 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.231+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.234 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.231+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3991617845}) 2024-09-17T18:37:24.234 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.231+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=358907163}) 2024-09-17T18:37:24.234 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.231+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.234 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.231+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3991617845}) 2024-09-17T18:37:24.234 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.231+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3155746400}) 2024-09-17T18:37:24.234 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.232+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.234 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.232+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3886716479}) 2024-09-17T18:37:24.234 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.232+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=914800392}) 2024-09-17T18:37:24.234 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.232+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.234 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.232+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3886716479}) 2024-09-17T18:37:24.234 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.232+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3070164288}) 2024-09-17T18:37:24.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:24.232+0000 7f644a2bf640 10 librados: watch_flush enter 2024-09-17T18:37:24.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:24.232+0000 7f644a2bf640 10 librados: watch_flush exit 2024-09-17T18:37:24.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:24.232+0000 7f644a2bf640 20 client.114132.objecter shutdown clearing up homeless session... 2024-09-17T18:37:24.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:24.232+0000 7f644a2bf640 10 client.114132.objecter successfully canceled tick 2024-09-17T18:37:24.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:24.232+0000 7f644a2bf640 1 -- 172.21.15.121:0/1902745303 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6430040170 msgr2=0x7f6430042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:24.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:24.232+0000 7f644a2bf640 1 --2- 172.21.15.121:0/1902745303 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6430040170 0x7f6430042630 secure :-1 s=READY pgs=4094 cs=0 l=1 rev1=1 crypto rx=0x7f6444074e20 tx=0x7f643802e000 comp rx=0 tx=0).stop 2024-09-17T18:37:24.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:24.232+0000 7f644a2bf640 10 monclient: shutdown 2024-09-17T18:37:24.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:24.232+0000 7f644a2bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:24.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:24.232+0000 7f644a2bf640 1 -- 172.21.15.121:0/1902745303 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f644406ec00 msgr2=0x7f64440a2ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:24.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:24.232+0000 7f644a2bf640 1 --2- 172.21.15.121:0/1902745303 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f644406ec00 0x7f64440a2ee0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f643400c460 tx=0x7f643400c920 comp rx=0 tx=0).stop 2024-09-17T18:37:24.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:24.232+0000 7f644a2bf640 1 -- 172.21.15.121:0/1902745303 shutdown_connections 2024-09-17T18:37:24.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:24.232+0000 7f644a2bf640 1 --2- 172.21.15.121:0/1902745303 >> v2:172.21.15.173:6812/2423786037 conn(0x7f6430040170 0x7f6430042630 unknown :-1 s=CLOSED pgs=4094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:24.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:24.232+0000 7f644a2bf640 1 --2- 172.21.15.121:0/1902745303 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64441386b0 0x7f64441cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:24.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:24.232+0000 7f644a2bf640 1 --2- 172.21.15.121:0/1902745303 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6444138fd0 0x7f64441cdbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:24.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:24.232+0000 7f644a2bf640 1 --2- 172.21.15.121:0/1902745303 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f644406ec00 0x7f64440a2ee0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:24.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:24.232+0000 7f644a2bf640 1 -- 172.21.15.121:0/1902745303 >> 172.21.15.121:0/1902745303 conn(0x7f64440a91e0 msgr2=0x7f6444132040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:24.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:24.232+0000 7f644a2bf640 1 -- 172.21.15.121:0/1902745303 shutdown_connections 2024-09-17T18:37:24.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:24.232+0000 7f644a2bf640 1 -- 172.21.15.121:0/1902745303 wait complete. 2024-09-17T18:37:24.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:24.232+0000 7f644a2bf640 1 librados: shutdown 2024-09-17T18:37:24.237 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.233+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.237 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.233+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3058110626}) 2024-09-17T18:37:24.237 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.233+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2900010635}) 2024-09-17T18:37:24.237 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.233+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.238 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.233+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3058110626}) 2024-09-17T18:37:24.238 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.233+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3543310517}) 2024-09-17T18:37:24.245 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-17T18:37:24.246 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.243+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.247 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.243+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=886418969}) 2024-09-17T18:37:24.247 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.243+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=307439632}) 2024-09-17T18:37:24.247 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.243+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.247 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.243+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=886418969}) 2024-09-17T18:37:24.247 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.243+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1370084697}) 2024-09-17T18:37:24.247 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.244+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.247 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.244+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3383360888}) 2024-09-17T18:37:24.247 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.244+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4067288010}) 2024-09-17T18:37:24.247 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.244+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.247 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.244+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3383360888}) 2024-09-17T18:37:24.247 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.244+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3810637294}) 2024-09-17T18:37:24.247 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.244+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.247 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.244+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2409247459}) 2024-09-17T18:37:24.248 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.244+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=286653612}) 2024-09-17T18:37:24.248 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.244+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.248 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.244+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2409247459}) 2024-09-17T18:37:24.248 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.244+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1305097014}) 2024-09-17T18:37:24.249 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.245+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.249 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.245+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2602235214}) 2024-09-17T18:37:24.249 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.245+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3829051356}) 2024-09-17T18:37:24.249 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.245+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.249 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.245+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2602235214}) 2024-09-17T18:37:24.249 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.245+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1076415968}) 2024-09-17T18:37:24.249 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.246+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.249 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.246+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2125946631}) 2024-09-17T18:37:24.250 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.246+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1543367949}) 2024-09-17T18:37:24.250 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.246+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.250 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.246+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2125946631}) 2024-09-17T18:37:24.250 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.246+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=645845795}) 2024-09-17T18:37:24.255 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.253+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.255 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.253+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3185325813}) 2024-09-17T18:37:24.255 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.253+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3266511477}) 2024-09-17T18:37:24.255 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.253+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.256 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.253+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3185325813}) 2024-09-17T18:37:24.256 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.253+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3943886024}) 2024-09-17T18:37:24.256 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.254+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.256 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.254+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1016984824}) 2024-09-17T18:37:24.257 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.254+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1452842523}) 2024-09-17T18:37:24.257 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.254+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.257 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.254+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1016984824}) 2024-09-17T18:37:24.257 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.254+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3204884025}) 2024-09-17T18:37:24.636 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.634+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.637 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.634+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3479920661}) 2024-09-17T18:37:24.637 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.634+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=49440819}) 2024-09-17T18:37:24.637 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.634+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.637 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.634+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3479920661}) 2024-09-17T18:37:24.637 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.634+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3369985333}) 2024-09-17T18:37:24.717 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.715+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.718 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.715+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1568884297}) 2024-09-17T18:37:24.718 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.715+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=83283687}) 2024-09-17T18:37:24.718 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.716+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.718 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.716+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1568884297}) 2024-09-17T18:37:24.718 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.716+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1439811840}) 2024-09-17T18:37:24.719 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.717+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.719 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.717+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2490937920}) 2024-09-17T18:37:24.719 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.717+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=819508224}) 2024-09-17T18:37:24.719 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.717+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.719 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.717+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2490937920}) 2024-09-17T18:37:24.719 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.717+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1239156291}) 2024-09-17T18:37:24.720 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.718+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.720 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.718+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3872871022}) 2024-09-17T18:37:24.720 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.718+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2927746610}) 2024-09-17T18:37:24.720 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.718+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.720 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.718+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3872871022}) 2024-09-17T18:37:24.720 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.718+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=525961820}) 2024-09-17T18:37:24.721 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.719+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.721 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.719+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=311072049}) 2024-09-17T18:37:24.721 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.719+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=18550843}) 2024-09-17T18:37:24.721 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.719+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.721 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.719+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=311072049}) 2024-09-17T18:37:24.721 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.719+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2114616312}) 2024-09-17T18:37:24.722 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.720+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.722 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.720+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3577460512}) 2024-09-17T18:37:24.723 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.720+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1356800162}) 2024-09-17T18:37:24.723 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.720+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.723 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.720+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3577460512}) 2024-09-17T18:37:24.723 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.720+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=226629630}) 2024-09-17T18:37:24.723 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.720+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.723 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.720+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1057279303}) 2024-09-17T18:37:24.723 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.720+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3009009787}) 2024-09-17T18:37:24.723 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.720+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.723 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.720+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1057279303}) 2024-09-17T18:37:24.723 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.720+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2243346619}) 2024-09-17T18:37:24.725 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.721+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.725 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.721+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1337797201}) 2024-09-17T18:37:24.725 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.721+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3844861626}) 2024-09-17T18:37:24.725 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.721+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.725 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.721+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1337797201}) 2024-09-17T18:37:24.725 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.721+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2847928597}) 2024-09-17T18:37:24.725 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.722+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.725 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.722+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2519120908}) 2024-09-17T18:37:24.725 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.722+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=980120400}) 2024-09-17T18:37:24.725 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.722+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.725 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.722+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2519120908}) 2024-09-17T18:37:24.725 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.722+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2805503263}) 2024-09-17T18:37:24.725 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.723+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.726 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.723+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1594859448}) 2024-09-17T18:37:24.726 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.723+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1217576885}) 2024-09-17T18:37:24.726 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.723+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.726 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.723+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1594859448}) 2024-09-17T18:37:24.726 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.723+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2457455128}) 2024-09-17T18:37:24.727 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.723+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.727 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.723+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=789586865}) 2024-09-17T18:37:24.727 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.723+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1821751327}) 2024-09-17T18:37:24.727 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.723+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.727 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.723+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=789586865}) 2024-09-17T18:37:24.727 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.723+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3183956673}) 2024-09-17T18:37:24.863 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.861+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.863 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.861+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1642853246}) 2024-09-17T18:37:24.863 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.861+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1534820743}) 2024-09-17T18:37:24.864 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.861+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.864 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.861+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1642853246}) 2024-09-17T18:37:24.864 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.861+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1218715145}) 2024-09-17T18:37:24.869 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.867+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.869 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.867+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3510082794}) 2024-09-17T18:37:24.869 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.867+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=670034228}) 2024-09-17T18:37:24.869 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.867+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:24.869 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.867+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3510082794}) 2024-09-17T18:37:24.869 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:24.867+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=759058396}) 2024-09-17T18:37:25.246 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T18:37:25.247 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T18:37:25.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.348+0000 7f915495d640 10 monclient: get_monmap_and_config 2024-09-17T18:37:25.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.348+0000 7f915495d640 10 monclient: build_initial_monmap 2024-09-17T18:37:25.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.348+0000 7f915495d640 10 monclient: monmap: 2024-09-17T18:37:25.350 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:25.350 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:25.350 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:37:25.348802+0000 2024-09-17T18:37:25.350 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:37:25.348802+0000 2024-09-17T18:37:25.350 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:25.350 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:25.350 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:37:25.350 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:37:25.350 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:37:25.351 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:37:25.351 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:37:25.351 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:25.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.348+0000 7f915495d640 1 Processor -- start 2024-09-17T18:37:25.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.348+0000 7f915495d640 1 -- start start 2024-09-17T18:37:25.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.348+0000 7f915495d640 10 monclient: init 2024-09-17T18:37:25.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.349+0000 7f915495d640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:25.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.349+0000 7f915495d640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-17T18:37:25.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.349+0000 7f915495d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f91500a5c00 addr v1:172.21.15.121:6789/0 2024-09-17T18:37:25.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.349+0000 7f915495d640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f91500a5fe0 0x7f91500a63f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:25.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.349+0000 7f915495d640 10 monclient(hunting): picked mon.noname-c con 0x7f91500a5fe0 addr v2:172.21.15.173:3300/0 2024-09-17T18:37:25.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.349+0000 7f915495d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f915013dc00 addr v1:172.21.15.173:6789/0 2024-09-17T18:37:25.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.349+0000 7f915495d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91500a8a80 con 0x7f91500a5c00 2024-09-17T18:37:25.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.349+0000 7f915495d640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91500a74d0 con 0x7f915013dc00 2024-09-17T18:37:25.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.349+0000 7f915495d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:25.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.349+0000 7f915495d640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f915009cb90 con 0x7f91500a5fe0 2024-09-17T18:37:25.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.349+0000 7f915495d640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:25.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.349+0000 7f915495d640 10 monclient(hunting): authenticate will time out at 1849.065510s 2024-09-17T18:37:25.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.349+0000 7f914e59c640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f91500a5c00 0x7f915009d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:47344/0 (socket says 172.21.15.121:47344) 2024-09-17T18:37:25.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.349+0000 7f914e59c640 1 -- 172.21.15.121:0/2361467877 learned_addr learned my addr 172.21.15.121:0/2361467877 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:37:25.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.349+0000 7f914dd9b640 1 --2- 172.21.15.121:0/2361467877 >> v2:172.21.15.173:3300/0 conn(0x7f91500a5fe0 0x7f91500a63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:25.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.349+0000 7f914dd9b640 10 monclient(hunting): get_auth_request con 0x7f91500a5fe0 auth_method 0 2024-09-17T18:37:25.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.349+0000 7f914dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:25.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.349+0000 7f914dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:25.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.349+0000 7f914dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:25.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.349+0000 7f914dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:25.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.349+0000 7f914dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:25.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.349+0000 7f914dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:25.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.350+0000 7f914dd9b640 10 monclient(hunting): handle_auth_done global_id 84331 payload 274 2024-09-17T18:37:25.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.350+0000 7f914dd9b640 1 -- 172.21.15.121:0/2361467877 >> v1:172.21.15.173:6789/0 conn(0x7f915013dc00 legacy=0x7f915013fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:25.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.350+0000 7f914dd9b640 1 -- 172.21.15.121:0/2361467877 >> v1:172.21.15.121:6789/0 conn(0x7f91500a5c00 legacy=0x7f915009d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:25.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.350+0000 7f914dd9b640 10 monclient: _finish_hunting 0 2024-09-17T18:37:25.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.350+0000 7f914dd9b640 1 monclient: found mon.noname-c 2024-09-17T18:37:25.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.350+0000 7f914dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:25.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.350+0000 7f914dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.173:3300/0 2024-09-17T18:37:25.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.350+0000 7f914dd9b640 1 -- 172.21.15.121:0/2361467877 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f915009ce70 con 0x7f91500a5fe0 2024-09-17T18:37:25.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.353+0000 7f914dd9b640 1 -- 172.21.15.121:0/2361467877 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91500a5fe0 msgr2=0x7f91500a63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:25.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.353+0000 7f914dd9b640 1 --2- 172.21.15.121:0/2361467877 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91500a5fe0 0x7f91500a63f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f91440097a0 tx=0x7f9144029fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30115d85c7e14f57 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:25.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.353+0000 7f914d59a640 1 -- 172.21.15.121:0/2361467877 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9144037070 con 0x7f91500a5fe0 2024-09-17T18:37:25.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.353+0000 7f914d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:25.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.353+0000 7f914d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:37:25.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.353+0000 7f914d59a640 10 monclient: dump: 2024-09-17T18:37:25.362 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:25.362 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:25.362 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:25.362 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:25.362 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:25.362 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:25.362 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:25.362 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:25.362 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:25.362 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:25.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.353+0000 7f914d59a640 10 monclient: _finish_auth 0 2024-09-17T18:37:25.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.353+0000 7f914d59a640 10 monclient: _check_auth_tickets 2024-09-17T18:37:25.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.353+0000 7f914d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:25.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.353+0000 7f914d59a640 1 -- 172.21.15.121:0/2361467877 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9144032640 con 0x7f91500a5fe0 2024-09-17T18:37:25.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.353+0000 7f914d59a640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:25.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.353+0000 7f914d59a640 1 -- 172.21.15.121:0/2361467877 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9144032a10 con 0x7f91500a5fe0 2024-09-17T18:37:25.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.353+0000 7f915495d640 5 monclient: authenticate success, global_id 84331 2024-09-17T18:37:25.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.353+0000 7f915495d640 10 monclient: get_monmap_and_config success 2024-09-17T18:37:25.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.353+0000 7f915495d640 4 set_mon_vals no callback set 2024-09-17T18:37:25.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.353+0000 7f914d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:25.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.353+0000 7f914d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:37:25.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.353+0000 7f914d59a640 10 monclient: dump: 2024-09-17T18:37:25.363 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:25.363 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:25.363 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:25.363 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:25.364 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:25.364 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:25.364 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:25.364 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:25.364 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:25.364 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:25.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.353+0000 7f915495d640 10 monclient: shutdown 2024-09-17T18:37:25.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.353+0000 7f915495d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:25.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.353+0000 7f915495d640 1 -- 172.21.15.121:0/2361467877 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91500a5fe0 msgr2=0x7f91500a63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:25.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.353+0000 7f915495d640 1 --2- 172.21.15.121:0/2361467877 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91500a5fe0 0x7f91500a63f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f91440097a0 tx=0x7f9144029fd0 comp rx=0 tx=0).stop 2024-09-17T18:37:25.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.353+0000 7f915495d640 1 -- 172.21.15.121:0/2361467877 shutdown_connections 2024-09-17T18:37:25.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.353+0000 7f915495d640 1 --2- 172.21.15.121:0/2361467877 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91500a5fe0 0x7f91500a63f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:25.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.353+0000 7f915495d640 1 -- 172.21.15.121:0/2361467877 >> 172.21.15.121:0/2361467877 conn(0x7f91500ab9b0 msgr2=0x7f91500a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:25.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.353+0000 7f915495d640 1 -- 172.21.15.121:0/2361467877 shutdown_connections 2024-09-17T18:37:25.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.354+0000 7f915495d640 1 -- 172.21.15.121:0/2361467877 wait complete. 2024-09-17T18:37:25.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.354+0000 7f915495d640 10 monclient: build_initial_monmap 2024-09-17T18:37:25.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.354+0000 7f915495d640 10 monclient: monmap: 2024-09-17T18:37:25.365 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:25.365 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:37:25.365 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:37:25.365 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:37:25.365 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:25.365 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:25.365 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:37:25.365 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:37:25.365 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:37:25.365 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:25.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.354+0000 7f915495d640 1 librados: starting msgr at 2024-09-17T18:37:25.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.354+0000 7f915495d640 1 librados: starting objecter 2024-09-17T18:37:25.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f915495d640 1 Processor -- start 2024-09-17T18:37:25.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f915495d640 1 -- start start 2024-09-17T18:37:25.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f915495d640 1 librados: setting wanted keys 2024-09-17T18:37:25.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f915495d640 1 librados: calling monclient init 2024-09-17T18:37:25.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f915495d640 10 monclient: init 2024-09-17T18:37:25.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f915495d640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:25.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f915495d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T18:37:25.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f915495d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91500a5c00 0x7f915013cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:25.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f915495d640 10 monclient(hunting): picked mon.noname-b con 0x7f91500a5c00 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:37:25.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f915495d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91500a5fe0 0x7f91501cde40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:25.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f915495d640 10 monclient(hunting): picked mon.noname-a con 0x7f91500a5fe0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:37:25.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f915495d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f915013dc00 0x7f915013a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:25.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f915495d640 10 monclient(hunting): picked mon.noname-c con 0x7f915013dc00 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:25.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f915495d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:25.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f915495d640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9150070550 con 0x7f91500a5fe0 2024-09-17T18:37:25.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f915495d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:25.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f915495d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f91500703d0 con 0x7f91500a5c00 2024-09-17T18:37:25.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f915495d640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:25.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f915495d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9150141710 con 0x7f915013dc00 2024-09-17T18:37:25.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f915495d640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:25.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f915495d640 10 monclient(hunting): authenticate will time out at 1849.071608s 2024-09-17T18:37:25.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f914dd9b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91500a5fe0 0x7f91501cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:25.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f914dd9b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91500a5fe0 0x7f91501cde40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51230/0 (socket says 172.21.15.121:51230) 2024-09-17T18:37:25.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f914dd9b640 1 -- 172.21.15.121:0/4042021852 learned_addr learned my addr 172.21.15.121:0/4042021852 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:37:25.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f914dd9b640 10 monclient(hunting): get_auth_request con 0x7f91500a5fe0 auth_method 0 2024-09-17T18:37:25.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f914dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:25.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f914dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:25.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f914dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:25.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f914ed9d640 1 --2- 172.21.15.121:0/4042021852 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f915013dc00 0x7f915013a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:25.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f914ed9d640 10 monclient(hunting): get_auth_request con 0x7f915013dc00 auth_method 0 2024-09-17T18:37:25.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f914ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:25.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f914ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:25.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f914ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:25.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f914dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:25.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f914dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:25.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f914dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:25.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f914ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:25.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f914ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:25.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f914ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:25.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f914e59c640 1 --2- 172.21.15.121:0/4042021852 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91500a5c00 0x7f915013cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:25.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f914e59c640 10 monclient(hunting): get_auth_request con 0x7f91500a5c00 auth_method 0 2024-09-17T18:37:25.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f914e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:25.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f914e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:25.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f914e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:25.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.355+0000 7f914ed9d640 10 monclient(hunting): handle_auth_done global_id 64589 payload 824 2024-09-17T18:37:25.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.356+0000 7f914ed9d640 1 -- 172.21.15.121:0/4042021852 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91500a5c00 msgr2=0x7f915013cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:25.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.356+0000 7f914ed9d640 1 --2- 172.21.15.121:0/4042021852 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91500a5c00 0x7f915013cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:25.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.356+0000 7f914ed9d640 1 -- 172.21.15.121:0/4042021852 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91500a5fe0 msgr2=0x7f91501cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:25.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.356+0000 7f914ed9d640 1 --2- 172.21.15.121:0/4042021852 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91500a5fe0 0x7f91501cde40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:25.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.356+0000 7f914ed9d640 10 monclient: _finish_hunting 0 2024-09-17T18:37:25.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.356+0000 7f914ed9d640 1 monclient: found mon.noname-c 2024-09-17T18:37:25.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.356+0000 7f914ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:25.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.356+0000 7f914ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:37:25.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.356+0000 7f914ed9d640 1 -- 172.21.15.121:0/4042021852 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f915013d760 con 0x7f915013dc00 2024-09-17T18:37:25.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.356+0000 7f914dd9b640 1 --2- 172.21.15.121:0/4042021852 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91500a5fe0 0x7f91501cde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:37:25.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.358+0000 7f914ed9d640 1 -- 172.21.15.121:0/4042021852 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f915013dc00 msgr2=0x7f915013a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:25.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.358+0000 7f914ed9d640 1 --2- 172.21.15.121:0/4042021852 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f915013dc00 0x7f915013a7b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f914800c750 tx=0x7f914800cc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:25.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.358+0000 7f913effd640 10 client.?.objecter ms_handle_connect 0x7f915013dc00 2024-09-17T18:37:25.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.358+0000 7f913effd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:37:25.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.358+0000 7f913effd640 1 -- 172.21.15.121:0/4042021852 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9148016020 con 0x7f915013dc00 2024-09-17T18:37:25.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.358+0000 7f913effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:25.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.358+0000 7f913effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:37:25.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.358+0000 7f913effd640 10 monclient: dump: 2024-09-17T18:37:25.370 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:25.370 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:25.370 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:25.370 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:25.370 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:25.370 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:25.370 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:25.370 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:25.370 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:25.370 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:25.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.358+0000 7f913effd640 10 monclient: _finish_auth 0 2024-09-17T18:37:25.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.358+0000 7f913effd640 10 monclient: _check_auth_tickets 2024-09-17T18:37:25.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.358+0000 7f913effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:25.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.358+0000 7f915495d640 5 monclient: authenticate success, global_id 64589 2024-09-17T18:37:25.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.358+0000 7f915495d640 10 monclient: _renew_subs 2024-09-17T18:37:25.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.358+0000 7f913effd640 1 -- 172.21.15.121:0/4042021852 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f914800aa00 con 0x7f915013dc00 2024-09-17T18:37:25.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.358+0000 7f915495d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:37:25.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.358+0000 7f915495d640 1 -- 172.21.15.121:0/4042021852 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9150121980 con 0x7f915013dc00 2024-09-17T18:37:25.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.358+0000 7f913effd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:25.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.358+0000 7f913effd640 1 -- 172.21.15.121:0/4042021852 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914800adf0 con 0x7f915013dc00 2024-09-17T18:37:25.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.358+0000 7f913effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:25.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.358+0000 7f913effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:37:25.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.358+0000 7f913effd640 10 monclient: dump: 2024-09-17T18:37:25.371 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:25.372 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:25.372 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:25.372 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:25.372 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:25.372 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:25.372 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:25.372 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:25.372 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:25.372 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:25.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.358+0000 7f913ffff640 4 set_mon_vals no callback set 2024-09-17T18:37:25.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.358+0000 7f915495d640 10 client.64589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:37:25.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.358+0000 7f915495d640 10 monclient: _renew_subs 2024-09-17T18:37:25.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.358+0000 7f915495d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:37:25.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.358+0000 7f915495d640 1 -- 172.21.15.121:0/4042021852 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9150138ed0 con 0x7f915013dc00 2024-09-17T18:37:25.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.358+0000 7f915495d640 1 librados: init done 2024-09-17T18:37:25.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.359+0000 7f913effd640 1 -- 172.21.15.121:0/4042021852 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f9148019020 con 0x7f915013dc00 2024-09-17T18:37:25.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.359+0000 7f915495d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:25.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.359+0000 7f915495d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:25.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.359+0000 7f915495d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:37:25.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.359+0000 7f915495d640 1 -- 172.21.15.121:0/4042021852 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91500a63f0 con 0x7f915013dc00 2024-09-17T18:37:25.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.359+0000 7f913effd640 1 --2- 172.21.15.121:0/4042021852 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9130040170 0x7f9130042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:25.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.359+0000 7f913effd640 1 -- 172.21.15.121:0/4042021852 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(679..679 src has 247..679) ==== 4685+0+0 (secure 0 0 0) 0x7f91480532e0 con 0x7f915013dc00 2024-09-17T18:37:25.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.359+0000 7f913effd640 10 client.64589.objecter ms_dispatch 0x7f9150000b90 osd_map(679..679 src has 247..679) 2024-09-17T18:37:25.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.359+0000 7f913effd640 3 client.64589.objecter handle_osd_map got epochs [679,679] > 0 2024-09-17T18:37:25.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.359+0000 7f913effd640 3 client.64589.objecter handle_osd_map decoding full epoch 679 2024-09-17T18:37:25.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.359+0000 7f913effd640 20 client.64589.objecter dump_active .. 0 homeless 2024-09-17T18:37:25.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.359+0000 7f914e59c640 1 --2- 172.21.15.121:0/4042021852 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9130040170 0x7f9130042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:25.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.359+0000 7f914e59c640 10 monclient: get_auth_request con 0x7f9130040170 auth_method 0 2024-09-17T18:37:25.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.362+0000 7f914e59c640 1 --2- 172.21.15.121:0/4042021852 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9130040170 0x7f9130042630 secure :-1 s=READY pgs=4095 cs=0 l=1 rev1=1 crypto rx=0x7f91501218b0 tx=0x7f9140009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:25.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.362+0000 7f913effd640 10 client.64589.objecter ms_handle_connect 0x7f9130040170 2024-09-17T18:37:25.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.364+0000 7f913effd640 1 -- 172.21.15.121:0/4042021852 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9148020030 con 0x7f915013dc00 2024-09-17T18:37:25.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.364+0000 7f913effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:25.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.364+0000 7f913effd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:37:25.545 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.544+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.545 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.544+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3181515757}) 2024-09-17T18:37:25.545 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.544+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=10134032}) 2024-09-17T18:37:25.546 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.544+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.546 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.544+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3181515757}) 2024-09-17T18:37:25.546 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.544+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=604905237}) 2024-09-17T18:37:25.546 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.545+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.546 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.545+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3340488759}) 2024-09-17T18:37:25.546 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.545+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3718284084}) 2024-09-17T18:37:25.547 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.545+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.547 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.545+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3340488759}) 2024-09-17T18:37:25.547 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.545+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=262463}) 2024-09-17T18:37:25.547 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.545+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.547 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.545+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2765333914}) 2024-09-17T18:37:25.547 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.545+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1070560486}) 2024-09-17T18:37:25.547 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.545+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.548 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.545+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2765333914}) 2024-09-17T18:37:25.548 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.546+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=344051695}) 2024-09-17T18:37:25.557 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.555+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.557 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.555+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2447088313}) 2024-09-17T18:37:25.557 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.555+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1243196683}) 2024-09-17T18:37:25.557 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.555+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.557 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.555+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2447088313}) 2024-09-17T18:37:25.557 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.555+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=470357245}) 2024-09-17T18:37:25.558 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.556+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.558 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.556+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1027903075}) 2024-09-17T18:37:25.558 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.556+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2236057408}) 2024-09-17T18:37:25.558 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.556+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.558 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.556+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1027903075}) 2024-09-17T18:37:25.558 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.556+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3200649020}) 2024-09-17T18:37:25.560 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.557+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.560 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.557+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4153840640}) 2024-09-17T18:37:25.560 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.557+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3189712812}) 2024-09-17T18:37:25.560 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.557+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.560 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.557+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4153840640}) 2024-09-17T18:37:25.560 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.557+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3578985532}) 2024-09-17T18:37:25.560 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.558+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.560 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.558+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1565695345}) 2024-09-17T18:37:25.560 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.558+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2791845469}) 2024-09-17T18:37:25.560 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.558+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.560 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.558+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1565695345}) 2024-09-17T18:37:25.560 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.558+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1897807499}) 2024-09-17T18:37:25.561 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.559+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.562 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.559+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=490144691}) 2024-09-17T18:37:25.562 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.559+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2622047788}) 2024-09-17T18:37:25.562 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.559+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.562 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.559+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=490144691}) 2024-09-17T18:37:25.562 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.559+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=411312123}) 2024-09-17T18:37:25.562 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.559+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.562 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.559+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3371905805}) 2024-09-17T18:37:25.562 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.559+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2796166126}) 2024-09-17T18:37:25.562 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.559+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.562 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.559+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3371905805}) 2024-09-17T18:37:25.562 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.560+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=825948380}) 2024-09-17T18:37:25.563 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.560+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.564 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.560+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=835906803}) 2024-09-17T18:37:25.564 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.560+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3525771902}) 2024-09-17T18:37:25.564 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.560+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.564 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.560+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=835906803}) 2024-09-17T18:37:25.564 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.560+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2068272004}) 2024-09-17T18:37:25.564 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.561+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.564 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.561+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3998039135}) 2024-09-17T18:37:25.564 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.561+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=397489940}) 2024-09-17T18:37:25.564 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.561+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.564 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.561+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3998039135}) 2024-09-17T18:37:25.564 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.561+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4043596656}) 2024-09-17T18:37:25.566 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.562+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.566 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.562+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3642153292}) 2024-09-17T18:37:25.566 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.562+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3416080771}) 2024-09-17T18:37:25.566 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.562+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.566 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.562+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3642153292}) 2024-09-17T18:37:25.566 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.562+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1570828357}) 2024-09-17T18:37:25.566 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.563+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.566 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.563+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4211278736}) 2024-09-17T18:37:25.566 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.563+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2930862001}) 2024-09-17T18:37:25.566 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.563+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.566 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.563+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4211278736}) 2024-09-17T18:37:25.567 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.563+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1262164246}) 2024-09-17T18:37:25.567 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.564+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.567 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.564+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1597952878}) 2024-09-17T18:37:25.567 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.564+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1176854965}) 2024-09-17T18:37:25.567 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.564+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.567 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.564+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1597952878}) 2024-09-17T18:37:25.567 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.564+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1207535481}) 2024-09-17T18:37:25.569 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.565+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.569 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.565+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=458896487}) 2024-09-17T18:37:25.569 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.565+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=395854960}) 2024-09-17T18:37:25.569 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.565+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.569 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.565+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=458896487}) 2024-09-17T18:37:25.569 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.565+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=929797736}) 2024-09-17T18:37:25.569 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.566+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.569 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.566+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1691649614}) 2024-09-17T18:37:25.569 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.566+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2188716943}) 2024-09-17T18:37:25.569 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.566+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.570 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.566+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1691649614}) 2024-09-17T18:37:25.570 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.566+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=498440250}) 2024-09-17T18:37:25.570 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.567+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.570 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.567+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2141695836}) 2024-09-17T18:37:25.570 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.567+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=544873268}) 2024-09-17T18:37:25.570 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.567+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.570 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.567+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2141695836}) 2024-09-17T18:37:25.570 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.567+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2687260152}) 2024-09-17T18:37:25.572 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.568+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.572 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.568+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1460577590}) 2024-09-17T18:37:25.572 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.568+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3389834387}) 2024-09-17T18:37:25.572 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.568+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.572 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.568+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1460577590}) 2024-09-17T18:37:25.573 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.568+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=961569263}) 2024-09-17T18:37:25.573 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.568+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.573 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.568+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3382855858}) 2024-09-17T18:37:25.573 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.568+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3968109138}) 2024-09-17T18:37:25.573 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.568+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.573 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.568+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3382855858}) 2024-09-17T18:37:25.573 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.568+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2757750411}) 2024-09-17T18:37:25.573 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.569+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.573 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.569+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=150358452}) 2024-09-17T18:37:25.573 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.569+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=972444046}) 2024-09-17T18:37:25.573 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.569+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.573 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.569+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=150358452}) 2024-09-17T18:37:25.573 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.569+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=977806650}) 2024-09-17T18:37:25.574 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.570+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.574 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.570+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=532326999}) 2024-09-17T18:37:25.574 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.570+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2417726378}) 2024-09-17T18:37:25.574 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.570+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.574 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.570+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=532326999}) 2024-09-17T18:37:25.574 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.570+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1594103930}) 2024-09-17T18:37:25.575 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.571+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.576 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.571+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3989953165}) 2024-09-17T18:37:25.576 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.571+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4157545972}) 2024-09-17T18:37:25.576 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.571+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.576 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.571+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3989953165}) 2024-09-17T18:37:25.576 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.571+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1762829017}) 2024-09-17T18:37:25.576 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.572+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.576 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.572+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=245673590}) 2024-09-17T18:37:25.576 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.572+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3503198842}) 2024-09-17T18:37:25.576 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.572+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.576 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.572+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=245673590}) 2024-09-17T18:37:25.576 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.572+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1360515134}) 2024-09-17T18:37:25.576 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.573+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.576 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.573+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=486374447}) 2024-09-17T18:37:25.577 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.573+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4099920121}) 2024-09-17T18:37:25.577 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.573+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.577 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.573+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=486374447}) 2024-09-17T18:37:25.577 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.573+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3573709633}) 2024-09-17T18:37:25.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.614+0000 7f915495d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T18:37:25.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.615+0000 7f915495d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T18:37:25.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.615+0000 7f915495d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:37:25.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.615+0000 7f915495d640 1 -- 172.21.15.121:0/4042021852 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f915006cd40 con 0x7f915013dc00 2024-09-17T18:37:25.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.615+0000 7f913effd640 1 -- 172.21.15.121:0/4042021852 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f9148053080 con 0x7f915013dc00 2024-09-17T18:37:25.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.615+0000 7f913effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T18:37:25.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.615+0000 7f913effd640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:37:25.617 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.615+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.617 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.615+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=47663571}) 2024-09-17T18:37:25.618 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":66,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","modified":"2024-09-17T18:18:20.001795Z","created":"2024-09-17T18:18:20.001795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T18:37:25.618 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.615+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1619220782}) 2024-09-17T18:37:25.618 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.615+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.618 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.615+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=47663571}) 2024-09-17T18:37:25.618 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.615+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4287640795}) 2024-09-17T18:37:25.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.617+0000 7f915495d640 10 librados: watch_flush enter 2024-09-17T18:37:25.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.617+0000 7f915495d640 10 librados: watch_flush exit 2024-09-17T18:37:25.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.617+0000 7f915495d640 20 client.64589.objecter shutdown clearing up homeless session... 2024-09-17T18:37:25.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.617+0000 7f915495d640 10 client.64589.objecter successfully canceled tick 2024-09-17T18:37:25.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.617+0000 7f915495d640 1 -- 172.21.15.121:0/4042021852 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9130040170 msgr2=0x7f9130042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:25.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.617+0000 7f915495d640 1 --2- 172.21.15.121:0/4042021852 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9130040170 0x7f9130042630 secure :-1 s=READY pgs=4095 cs=0 l=1 rev1=1 crypto rx=0x7f91501218b0 tx=0x7f9140009040 comp rx=0 tx=0).stop 2024-09-17T18:37:25.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.617+0000 7f915495d640 10 monclient: shutdown 2024-09-17T18:37:25.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.617+0000 7f915495d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:25.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.617+0000 7f915495d640 1 -- 172.21.15.121:0/4042021852 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f915013dc00 msgr2=0x7f915013a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:25.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.617+0000 7f915495d640 1 --2- 172.21.15.121:0/4042021852 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f915013dc00 0x7f915013a7b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f914800c750 tx=0x7f914800cc10 comp rx=0 tx=0).stop 2024-09-17T18:37:25.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.617+0000 7f915495d640 1 -- 172.21.15.121:0/4042021852 shutdown_connections 2024-09-17T18:37:25.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.617+0000 7f915495d640 1 --2- 172.21.15.121:0/4042021852 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9130040170 0x7f9130042630 unknown :-1 s=CLOSED pgs=4095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:25.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.617+0000 7f915495d640 1 --2- 172.21.15.121:0/4042021852 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f915013dc00 0x7f915013a7b0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:25.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.617+0000 7f915495d640 1 --2- 172.21.15.121:0/4042021852 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91500a5fe0 0x7f91501cde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:25.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.617+0000 7f915495d640 1 --2- 172.21.15.121:0/4042021852 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91500a5c00 0x7f915013cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:25.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.617+0000 7f915495d640 1 -- 172.21.15.121:0/4042021852 >> 172.21.15.121:0/4042021852 conn(0x7f91500ab9b0 msgr2=0x7f91500a4660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:25.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.617+0000 7f915495d640 1 -- 172.21.15.121:0/4042021852 shutdown_connections 2024-09-17T18:37:25.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.617+0000 7f915495d640 1 -- 172.21.15.121:0/4042021852 wait complete. 2024-09-17T18:37:25.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.617+0000 7f915495d640 1 librados: shutdown 2024-09-17T18:37:25.629 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T18:37:25.629 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-17T18:37:25.640 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-17T18:37:25.640 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T18:37:25.641 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.634+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.642 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.634+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1421046617}) 2024-09-17T18:37:25.642 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.634+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2082986772}) 2024-09-17T18:37:25.642 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.634+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.642 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.634+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1421046617}) 2024-09-17T18:37:25.642 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.634+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2653775866}) 2024-09-17T18:37:25.642 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.635+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.642 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.635+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3879198615}) 2024-09-17T18:37:25.642 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.635+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1333659643}) 2024-09-17T18:37:25.642 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.635+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.642 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.635+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3879198615}) 2024-09-17T18:37:25.642 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.635+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2676299004}) 2024-09-17T18:37:25.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.737+0000 7fcc0b3ad640 10 monclient: get_monmap_and_config 2024-09-17T18:37:25.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.737+0000 7fcc0b3ad640 10 monclient: build_initial_monmap 2024-09-17T18:37:25.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.737+0000 7fcc0b3ad640 10 monclient: monmap: 2024-09-17T18:37:25.739 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:25.739 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:25.739 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:37:25.737784+0000 2024-09-17T18:37:25.739 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:37:25.737784+0000 2024-09-17T18:37:25.739 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:25.739 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:25.739 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:37:25.739 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:37:25.739 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:37:25.739 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:37:25.739 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:37:25.739 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:25.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.737+0000 7fcc0b3ad640 1 Processor -- start 2024-09-17T18:37:25.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.737+0000 7fcc0b3ad640 1 -- start start 2024-09-17T18:37:25.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.737+0000 7fcc0b3ad640 10 monclient: init 2024-09-17T18:37:25.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.738+0000 7fcc0b3ad640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:25.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.738+0000 7fcc0b3ad640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-17T18:37:25.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.738+0000 7fcc0b3ad640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fcc040a5c00 0x7fcc0409d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:25.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.738+0000 7fcc0b3ad640 10 monclient(hunting): picked mon.noname-c con 0x7fcc040a5c00 addr v2:172.21.15.173:3300/0 2024-09-17T18:37:25.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.738+0000 7fcc0b3ad640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcc040a5fe0 addr v1:172.21.15.121:6789/0 2024-09-17T18:37:25.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.738+0000 7fcc0b3ad640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc0413dc00 0x7fcc0413fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:25.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.738+0000 7fcc0b3ad640 10 monclient(hunting): picked mon.noname-b con 0x7fcc0413dc00 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:25.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.738+0000 7fcc0b3ad640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc040a8a80 con 0x7fcc040a5fe0 2024-09-17T18:37:25.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.738+0000 7fcc0b3ad640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:25.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.738+0000 7fcc0b3ad640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fcc040a7110 con 0x7fcc040a5c00 2024-09-17T18:37:25.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.738+0000 7fcc0b3ad640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:25.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.738+0000 7fcc0b3ad640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcc0409de00 con 0x7fcc0413dc00 2024-09-17T18:37:25.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.738+0000 7fcc0b3ad640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:25.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.738+0000 7fcc0b3ad640 10 monclient(hunting): authenticate will time out at 1849.454517s 2024-09-17T18:37:25.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.738+0000 7fcc0994a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc0413dc00 0x7fcc0413fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:25.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.738+0000 7fcc08948640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fcc040a5fe0 0x7fcc040a63f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:47356/0 (socket says 172.21.15.121:47356) 2024-09-17T18:37:25.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.738+0000 7fcc08948640 1 -- 172.21.15.121:0/1308913609 learned_addr learned my addr 172.21.15.121:0/1308913609 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:37:25.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.738+0000 7fcc0994a640 10 monclient(hunting): get_auth_request con 0x7fcc0413dc00 auth_method 0 2024-09-17T18:37:25.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.738+0000 7fcc0994a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:25.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.738+0000 7fcc0994a640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:25.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.738+0000 7fcc0994a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:25.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.738+0000 7fcc0994a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:25.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.738+0000 7fcc0994a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:25.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.738+0000 7fcc0994a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:25.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.738+0000 7fcc09149640 1 --2- 172.21.15.121:0/1308913609 >> v2:172.21.15.173:3300/0 conn(0x7fcc040a5c00 0x7fcc0409d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:25.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.738+0000 7fcc09149640 10 monclient(hunting): get_auth_request con 0x7fcc040a5c00 auth_method 0 2024-09-17T18:37:25.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.738+0000 7fcc09149640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:25.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.738+0000 7fcc09149640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:25.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.738+0000 7fcc09149640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:25.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.738+0000 7fcc0994a640 10 monclient(hunting): handle_auth_done global_id 64592 payload 274 2024-09-17T18:37:25.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.738+0000 7fcc0994a640 1 -- 172.21.15.121:0/1308913609 >> v2:172.21.15.173:3300/0 conn(0x7fcc040a5c00 msgr2=0x7fcc0409d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:25.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.738+0000 7fcc0994a640 1 --2- 172.21.15.121:0/1308913609 >> v2:172.21.15.173:3300/0 conn(0x7fcc040a5c00 0x7fcc0409d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:25.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.738+0000 7fcc0994a640 1 -- 172.21.15.121:0/1308913609 >> v1:172.21.15.121:6789/0 conn(0x7fcc040a5fe0 legacy=0x7fcc040a63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:25.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.738+0000 7fcc0994a640 10 monclient: _finish_hunting 0 2024-09-17T18:37:25.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.738+0000 7fcc0994a640 1 monclient: found mon.noname-b 2024-09-17T18:37:25.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.738+0000 7fcc0994a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:25.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.738+0000 7fcc0994a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:37:25.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.738+0000 7fcc0994a640 1 -- 172.21.15.121:0/1308913609 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc040a74d0 con 0x7fcc0413dc00 2024-09-17T18:37:25.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.741+0000 7fcc0994a640 1 -- 172.21.15.121:0/1308913609 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc0413dc00 msgr2=0x7fcc0413fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:25.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.741+0000 7fcc0994a640 1 --2- 172.21.15.121:0/1308913609 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc0413dc00 0x7fcc0413fff0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc009d90 tx=0x7fcbfc02a490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14edfe03d5c99be server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:25.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.741+0000 7fcbfbfff640 1 -- 172.21.15.121:0/1308913609 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbfc007950 con 0x7fcc0413dc00 2024-09-17T18:37:25.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.741+0000 7fcbfbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:25.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.741+0000 7fcbfbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:37:25.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.741+0000 7fcbfbfff640 10 monclient: dump: 2024-09-17T18:37:25.750 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:25.750 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:25.750 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:25.750 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:25.750 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:25.750 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:25.750 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:25.750 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:25.751 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:25.751 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:25.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.741+0000 7fcbfbfff640 10 monclient: _finish_auth 0 2024-09-17T18:37:25.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.741+0000 7fcbfbfff640 10 monclient: _check_auth_tickets 2024-09-17T18:37:25.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.741+0000 7fcbfbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:25.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.741+0000 7fcc0b3ad640 5 monclient: authenticate success, global_id 64592 2024-09-17T18:37:25.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.741+0000 7fcc0b3ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:37:25.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.741+0000 7fcbfbfff640 1 -- 172.21.15.121:0/1308913609 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbfc007af0 con 0x7fcc0413dc00 2024-09-17T18:37:25.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.741+0000 7fcbfbfff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:25.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.741+0000 7fcc0b3ad640 10 monclient: get_monmap_and_config success 2024-09-17T18:37:25.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.741+0000 7fcc0b3ad640 4 set_mon_vals no callback set 2024-09-17T18:37:25.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.741+0000 7fcbfbfff640 1 -- 172.21.15.121:0/1308913609 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbfc031400 con 0x7fcc0413dc00 2024-09-17T18:37:25.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.741+0000 7fcbfbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:25.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.741+0000 7fcbfbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:37:25.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.741+0000 7fcbfbfff640 10 monclient: dump: 2024-09-17T18:37:25.752 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:25.752 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:25.752 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:25.752 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:25.752 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:25.752 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:25.752 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:25.752 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:25.752 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:25.752 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:25.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.742+0000 7fcc0b3ad640 10 monclient: shutdown 2024-09-17T18:37:25.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.742+0000 7fcc0b3ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:25.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.742+0000 7fcc0b3ad640 1 -- 172.21.15.121:0/1308913609 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc0413dc00 msgr2=0x7fcc0413fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:25.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.742+0000 7fcc0b3ad640 1 --2- 172.21.15.121:0/1308913609 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc0413dc00 0x7fcc0413fff0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc009d90 tx=0x7fcbfc02a490 comp rx=0 tx=0).stop 2024-09-17T18:37:25.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.742+0000 7fcc0b3ad640 1 -- 172.21.15.121:0/1308913609 shutdown_connections 2024-09-17T18:37:25.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.742+0000 7fcc0b3ad640 1 --2- 172.21.15.121:0/1308913609 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc0413dc00 0x7fcc0413fff0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:25.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.742+0000 7fcc0b3ad640 1 --2- 172.21.15.121:0/1308913609 >> v2:172.21.15.173:3300/0 conn(0x7fcc040a5c00 0x7fcc0409d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:25.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.742+0000 7fcc0b3ad640 1 -- 172.21.15.121:0/1308913609 >> 172.21.15.121:0/1308913609 conn(0x7fcc040ab9b0 msgr2=0x7fcc040a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:25.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.742+0000 7fcc0b3ad640 1 -- 172.21.15.121:0/1308913609 shutdown_connections 2024-09-17T18:37:25.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.742+0000 7fcc0b3ad640 1 -- 172.21.15.121:0/1308913609 wait complete. 2024-09-17T18:37:25.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.743+0000 7fcc0b3ad640 10 monclient: build_initial_monmap 2024-09-17T18:37:25.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.743+0000 7fcc0b3ad640 10 monclient: monmap: 2024-09-17T18:37:25.753 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:25.753 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:37:25.753 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:37:25.753 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:37:25.753 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:25.753 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:25.753 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:37:25.754 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:37:25.754 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:37:25.754 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:25.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.743+0000 7fcc0b3ad640 1 librados: starting msgr at 2024-09-17T18:37:25.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.743+0000 7fcc0b3ad640 1 librados: starting objecter 2024-09-17T18:37:25.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.743+0000 7fcc0b3ad640 1 Processor -- start 2024-09-17T18:37:25.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.743+0000 7fcc0b3ad640 1 -- start start 2024-09-17T18:37:25.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.743+0000 7fcc0b3ad640 1 librados: setting wanted keys 2024-09-17T18:37:25.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.743+0000 7fcc0b3ad640 1 librados: calling monclient init 2024-09-17T18:37:25.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.743+0000 7fcc0b3ad640 10 monclient: init 2024-09-17T18:37:25.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.743+0000 7fcc0b3ad640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:25.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.743+0000 7fcc0b3ad640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T18:37:25.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.743+0000 7fcc0b3ad640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcc040a5c00 0x7fcc04121dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:25.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.743+0000 7fcc0b3ad640 10 monclient(hunting): picked mon.noname-b con 0x7fcc040a5c00 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:37:25.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.743+0000 7fcc0b3ad640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc040a5fe0 0x7fcc0413c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:25.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.743+0000 7fcc0b3ad640 10 monclient(hunting): picked mon.noname-a con 0x7fcc040a5fe0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:37:25.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.743+0000 7fcc0b3ad640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc0413dc00 0x7fcc04138c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:25.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.743+0000 7fcc0b3ad640 10 monclient(hunting): picked mon.noname-c con 0x7fcc0413dc00 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:25.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.743+0000 7fcc0b3ad640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:25.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.743+0000 7fcc0b3ad640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcc040703d0 con 0x7fcc040a5fe0 2024-09-17T18:37:25.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.743+0000 7fcc0b3ad640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:25.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.743+0000 7fcc0b3ad640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcc04070550 con 0x7fcc040a5c00 2024-09-17T18:37:25.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.743+0000 7fcc0b3ad640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:25.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.743+0000 7fcc0b3ad640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcc04140530 con 0x7fcc0413dc00 2024-09-17T18:37:25.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.743+0000 7fcc0b3ad640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:25.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.743+0000 7fcc0b3ad640 10 monclient(hunting): authenticate will time out at 1849.459848s 2024-09-17T18:37:25.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.743+0000 7fcc08948640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc040a5fe0 0x7fcc0413c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:25.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.743+0000 7fcc0994a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc0413dc00 0x7fcc04138c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:25.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.743+0000 7fcc08948640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc040a5fe0 0x7fcc0413c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51248/0 (socket says 172.21.15.121:51248) 2024-09-17T18:37:25.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.743+0000 7fcc08948640 1 -- 172.21.15.121:0/1975317554 learned_addr learned my addr 172.21.15.121:0/1975317554 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:37:25.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.743+0000 7fcc0994a640 10 monclient(hunting): get_auth_request con 0x7fcc0413dc00 auth_method 0 2024-09-17T18:37:25.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.743+0000 7fcc0994a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:25.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.743+0000 7fcc0994a640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:25.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.743+0000 7fcc0994a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:25.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.743+0000 7fcc08948640 10 monclient(hunting): get_auth_request con 0x7fcc040a5fe0 auth_method 0 2024-09-17T18:37:25.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.743+0000 7fcc08948640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:25.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.743+0000 7fcc08948640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:25.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.743+0000 7fcc08948640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:25.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.744+0000 7fcc08948640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:25.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.744+0000 7fcc08948640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:25.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.744+0000 7fcc08948640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:25.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.744+0000 7fcc0994a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:25.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.744+0000 7fcc0994a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:25.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.744+0000 7fcc0994a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:25.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.744+0000 7fcc08948640 10 monclient(hunting): handle_auth_done global_id 114138 payload 824 2024-09-17T18:37:25.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.744+0000 7fcc08948640 1 -- 172.21.15.121:0/1975317554 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc0413dc00 msgr2=0x7fcc04138c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:25.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.744+0000 7fcc08948640 1 --2- 172.21.15.121:0/1975317554 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc0413dc00 0x7fcc04138c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:25.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.744+0000 7fcc08948640 1 -- 172.21.15.121:0/1975317554 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcc040a5c00 msgr2=0x7fcc04121dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:25.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.744+0000 7fcc09149640 1 --2- 172.21.15.121:0/1975317554 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcc040a5c00 0x7fcc04121dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:25.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.744+0000 7fcc08948640 1 --2- 172.21.15.121:0/1975317554 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcc040a5c00 0x7fcc04121dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:25.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.744+0000 7fcc08948640 10 monclient: _finish_hunting 0 2024-09-17T18:37:25.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.744+0000 7fcc08948640 1 monclient: found mon.noname-a 2024-09-17T18:37:25.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.744+0000 7fcc08948640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:25.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.744+0000 7fcc08948640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:37:25.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.744+0000 7fcc08948640 1 -- 172.21.15.121:0/1975317554 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc0413d620 con 0x7fcc040a5fe0 2024-09-17T18:37:25.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.744+0000 7fcc0994a640 1 --2- 172.21.15.121:0/1975317554 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc0413dc00 0x7fcc04138c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:37:25.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.746+0000 7fcc08948640 1 -- 172.21.15.121:0/1975317554 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc040a5fe0 msgr2=0x7fcc0413c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:25.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.746+0000 7fcc08948640 1 --2- 172.21.15.121:0/1975317554 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc040a5fe0 0x7fcc0413c020 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcbf400c460 tx=0x7fcbf400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:25.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.746+0000 7fcbf9ffb640 10 client.?.objecter ms_handle_connect 0x7fcc040a5fe0 2024-09-17T18:37:25.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.746+0000 7fcbf9ffb640 10 client.?.objecter resend_mon_ops 2024-09-17T18:37:25.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.746+0000 7fcbf9ffb640 1 -- 172.21.15.121:0/1975317554 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf4017070 con 0x7fcc040a5fe0 2024-09-17T18:37:25.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.746+0000 7fcbf9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:25.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.746+0000 7fcbf9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:37:25.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.746+0000 7fcbf9ffb640 10 monclient: dump: 2024-09-17T18:37:25.758 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:25.758 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:25.758 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:25.758 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:25.758 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:25.758 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:25.758 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:25.758 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:25.758 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:25.758 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:25.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.746+0000 7fcbf9ffb640 10 monclient: _finish_auth 0 2024-09-17T18:37:25.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.746+0000 7fcbf9ffb640 10 monclient: _check_auth_tickets 2024-09-17T18:37:25.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.746+0000 7fcbf9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:25.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.746+0000 7fcbf9ffb640 1 -- 172.21.15.121:0/1975317554 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbf40070d0 con 0x7fcc040a5fe0 2024-09-17T18:37:25.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.746+0000 7fcc0b3ad640 5 monclient: authenticate success, global_id 114138 2024-09-17T18:37:25.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.746+0000 7fcbf9ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:25.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.746+0000 7fcbf9ffb640 1 -- 172.21.15.121:0/1975317554 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf40074c0 con 0x7fcc040a5fe0 2024-09-17T18:37:25.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.746+0000 7fcbf9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:25.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.746+0000 7fcbf9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:37:25.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.746+0000 7fcbf9ffb640 10 monclient: dump: 2024-09-17T18:37:25.759 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:25.759 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:25.759 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:25.759 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:25.759 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:25.759 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:25.759 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:25.759 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:25.760 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:25.760 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:25.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.746+0000 7fcc0b3ad640 10 monclient: _renew_subs 2024-09-17T18:37:25.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.746+0000 7fcc0b3ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:25.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.746+0000 7fcc0b3ad640 1 -- 172.21.15.121:0/1975317554 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc040a23f0 con 0x7fcc040a5fe0 2024-09-17T18:37:25.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.746+0000 7fcbfb7fe640 4 set_mon_vals no callback set 2024-09-17T18:37:25.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.746+0000 7fcc0b3ad640 10 client.114138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:37:25.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.746+0000 7fcc0b3ad640 10 monclient: _renew_subs 2024-09-17T18:37:25.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.746+0000 7fcc0b3ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:25.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.746+0000 7fcc0b3ad640 1 -- 172.21.15.121:0/1975317554 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc04139180 con 0x7fcc040a5fe0 2024-09-17T18:37:25.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.746+0000 7fcc0b3ad640 1 librados: init done 2024-09-17T18:37:25.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.747+0000 7fcbf9ffb640 1 -- 172.21.15.121:0/1975317554 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fcbf401a070 con 0x7fcc040a5fe0 2024-09-17T18:37:25.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.747+0000 7fcbf9ffb640 1 --2- 172.21.15.121:0/1975317554 >> v2:172.21.15.173:6812/2423786037 conn(0x7fcbe8040170 0x7fcbe8042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:25.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.747+0000 7fcbf9ffb640 1 -- 172.21.15.121:0/1975317554 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(679..679 src has 247..679) ==== 4685+0+0 (secure 0 0 0) 0x7fcbf4052970 con 0x7fcc040a5fe0 2024-09-17T18:37:25.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.747+0000 7fcc0b3ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:25.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.747+0000 7fcbf9ffb640 10 client.114138.objecter ms_dispatch 0x7fcc04000b90 osd_map(679..679 src has 247..679) 2024-09-17T18:37:25.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.747+0000 7fcbf9ffb640 3 client.114138.objecter handle_osd_map got epochs [679,679] > 0 2024-09-17T18:37:25.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.747+0000 7fcbf9ffb640 3 client.114138.objecter handle_osd_map decoding full epoch 679 2024-09-17T18:37:25.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.747+0000 7fcc0b3ad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:25.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.747+0000 7fcc0b3ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:25.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.747+0000 7fcc0b3ad640 1 -- 172.21.15.121:0/1975317554 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc040a1580 con 0x7fcc040a5fe0 2024-09-17T18:37:25.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.747+0000 7fcbf9ffb640 20 client.114138.objecter dump_active .. 0 homeless 2024-09-17T18:37:25.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.747+0000 7fcc09149640 1 --2- 172.21.15.121:0/1975317554 >> v2:172.21.15.173:6812/2423786037 conn(0x7fcbe8040170 0x7fcbe8042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:25.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.747+0000 7fcc09149640 10 monclient: get_auth_request con 0x7fcbe8040170 auth_method 0 2024-09-17T18:37:25.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.750+0000 7fcbf9ffb640 10 client.114138.objecter ms_handle_connect 0x7fcbe8040170 2024-09-17T18:37:25.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.750+0000 7fcc09149640 1 --2- 172.21.15.121:0/1975317554 >> v2:172.21.15.173:6812/2423786037 conn(0x7fcbe8040170 0x7fcbe8042630 secure :-1 s=READY pgs=4096 cs=0 l=1 rev1=1 crypto rx=0x7fcc00006fd0 tx=0x7fcc00008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:25.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.752+0000 7fcbf9ffb640 1 -- 172.21.15.121:0/1975317554 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fcc040a1580 con 0x7fcc040a5fe0 2024-09-17T18:37:25.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.752+0000 7fcbf9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:25.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.752+0000 7fcbf9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:37:25.865 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.863+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.865 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.863+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1837471971}) 2024-09-17T18:37:25.865 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.863+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2354358006}) 2024-09-17T18:37:25.865 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.863+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.865 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.863+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1837471971}) 2024-09-17T18:37:25.865 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.863+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=799263054}) 2024-09-17T18:37:25.867 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.866+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.868 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.866+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3468049761}) 2024-09-17T18:37:25.868 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.866+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2067174096}) 2024-09-17T18:37:25.868 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.866+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.868 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.866+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3468049761}) 2024-09-17T18:37:25.868 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.866+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2781679124}) 2024-09-17T18:37:25.873 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.871+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.873 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.871+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3878800113}) 2024-09-17T18:37:25.873 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.871+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=976586419}) 2024-09-17T18:37:25.873 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.871+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:25.873 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.871+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3878800113}) 2024-09-17T18:37:25.873 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:25.871+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2285212542}) 2024-09-17T18:37:25.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.926+0000 7fcc0b3ad640 1 -- 172.21.15.121:0/1975317554 --> v2:172.21.15.173:6812/2423786037 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcc040aa790 con 0x7fcbe8040170 2024-09-17T18:37:25.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.929+0000 7fcbf9ffb640 1 -- 172.21.15.121:0/1975317554 <== mgr.4105 v2:172.21.15.173:6812/2423786037 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114139 (secure 0 0 0) 0x7fcc040aa790 con 0x7fcbe8040170 2024-09-17T18:37:25.932 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:37:25.932 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-09-17T18:37:25.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.931+0000 7fcc0b3ad640 10 librados: watch_flush enter 2024-09-17T18:37:25.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.931+0000 7fcc0b3ad640 10 librados: watch_flush exit 2024-09-17T18:37:25.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.931+0000 7fcc0b3ad640 20 client.114138.objecter shutdown clearing up homeless session... 2024-09-17T18:37:25.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.931+0000 7fcc0b3ad640 10 client.114138.objecter successfully canceled tick 2024-09-17T18:37:25.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.931+0000 7fcc0b3ad640 1 -- 172.21.15.121:0/1975317554 >> v2:172.21.15.173:6812/2423786037 conn(0x7fcbe8040170 msgr2=0x7fcbe8042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:25.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.931+0000 7fcc0b3ad640 1 --2- 172.21.15.121:0/1975317554 >> v2:172.21.15.173:6812/2423786037 conn(0x7fcbe8040170 0x7fcbe8042630 secure :-1 s=READY pgs=4096 cs=0 l=1 rev1=1 crypto rx=0x7fcc00006fd0 tx=0x7fcc00008040 comp rx=0 tx=0).stop 2024-09-17T18:37:25.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.931+0000 7fcc0b3ad640 10 monclient: shutdown 2024-09-17T18:37:25.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.931+0000 7fcc0b3ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:25.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.931+0000 7fcc0b3ad640 1 -- 172.21.15.121:0/1975317554 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc040a5fe0 msgr2=0x7fcc0413c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:25.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.931+0000 7fcc0b3ad640 1 --2- 172.21.15.121:0/1975317554 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc040a5fe0 0x7fcc0413c020 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcbf400c460 tx=0x7fcbf400c920 comp rx=0 tx=0).stop 2024-09-17T18:37:25.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.931+0000 7fcc0b3ad640 1 -- 172.21.15.121:0/1975317554 shutdown_connections 2024-09-17T18:37:25.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.931+0000 7fcc0b3ad640 1 --2- 172.21.15.121:0/1975317554 >> v2:172.21.15.173:6812/2423786037 conn(0x7fcbe8040170 0x7fcbe8042630 unknown :-1 s=CLOSED pgs=4096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:25.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.931+0000 7fcc0b3ad640 1 --2- 172.21.15.121:0/1975317554 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc0413dc00 0x7fcc04138c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:25.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.931+0000 7fcc0b3ad640 1 --2- 172.21.15.121:0/1975317554 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc040a5fe0 0x7fcc0413c020 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:25.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.931+0000 7fcc0b3ad640 1 --2- 172.21.15.121:0/1975317554 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcc040a5c00 0x7fcc04121dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:25.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.931+0000 7fcc0b3ad640 1 -- 172.21.15.121:0/1975317554 >> 172.21.15.121:0/1975317554 conn(0x7fcc040ab9b0 msgr2=0x7fcc040a0e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:25.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.931+0000 7fcc0b3ad640 1 -- 172.21.15.121:0/1975317554 shutdown_connections 2024-09-17T18:37:25.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.931+0000 7fcc0b3ad640 1 -- 172.21.15.121:0/1975317554 wait complete. 2024-09-17T18:37:25.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:25.932+0000 7fcc0b3ad640 1 librados: shutdown 2024-09-17T18:37:25.944 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":1227,"stamp":"2024-09-17T18:37:25.778399+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":459824,"kb_used_data":39448,"kb_used_omap":9,"kb_used_meta":420214,"kb_avail":565771216,"statfs":{"total":579820584960,"available":579349725184,"internally_reserved":0,"allocated":40394752,"data_stored":18330092,"data_compressed":59128,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":430299809},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003191"},"pg_stats":[{"pgid":"5.15","version":"0'0","reported_seq":1279,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.583221+0000","last_change":"2024-09-17T18:20:02.310686+0000","last_active":"2024-09-17T18:36:25.583221+0000","last_peered":"2024-09-17T18:36:25.583221+0000","last_clean":"2024-09-17T18:36:25.583221+0000","last_became_active":"2024-09-17T18:20:02.310239+0000","last_became_peered":"2024-09-17T18:20:02.310239+0000","last_unstale":"2024-09-17T18:36:25.583221+0000","last_undegraded":"2024-09-17T18:36:25.583221+0000","last_fullsized":"2024-09-17T18:36:25.583221+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:52:11.931160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1298,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.717134+0000","last_change":"2024-09-17T18:20:02.326174+0000","last_active":"2024-09-17T18:36:25.717134+0000","last_peered":"2024-09-17T18:36:25.717134+0000","last_clean":"2024-09-17T18:36:25.717134+0000","last_became_active":"2024-09-17T18:20:02.322561+0000","last_became_peered":"2024-09-17T18:20:02.322561+0000","last_unstale":"2024-09-17T18:36:25.717134+0000","last_undegraded":"2024-09-17T18:36:25.717134+0000","last_fullsized":"2024-09-17T18:36:25.717134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:29:09.470382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1296,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.735731+0000","last_change":"2024-09-17T18:20:02.250996+0000","last_active":"2024-09-17T18:36:26.735731+0000","last_peered":"2024-09-17T18:36:26.735731+0000","last_clean":"2024-09-17T18:36:26.735731+0000","last_became_active":"2024-09-17T18:20:02.250675+0000","last_became_peered":"2024-09-17T18:20:02.250675+0000","last_unstale":"2024-09-17T18:36:26.735731+0000","last_undegraded":"2024-09-17T18:36:26.735731+0000","last_fullsized":"2024-09-17T18:36:26.735731+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:17:16.240715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1295,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.433900+0000","last_change":"2024-09-17T18:20:02.342701+0000","last_active":"2024-09-17T18:36:25.433900+0000","last_peered":"2024-09-17T18:36:25.433900+0000","last_clean":"2024-09-17T18:36:25.433900+0000","last_became_active":"2024-09-17T18:20:02.342457+0000","last_became_peered":"2024-09-17T18:20:02.342457+0000","last_unstale":"2024-09-17T18:36:25.433900+0000","last_undegraded":"2024-09-17T18:36:25.433900+0000","last_fullsized":"2024-09-17T18:36:25.433900+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:57:23.288202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1280,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.583101+0000","last_change":"2024-09-17T18:20:02.251851+0000","last_active":"2024-09-17T18:36:25.583101+0000","last_peered":"2024-09-17T18:36:25.583101+0000","last_clean":"2024-09-17T18:36:25.583101+0000","last_became_active":"2024-09-17T18:20:02.249100+0000","last_became_peered":"2024-09-17T18:20:02.249100+0000","last_unstale":"2024-09-17T18:36:25.583101+0000","last_undegraded":"2024-09-17T18:36:25.583101+0000","last_fullsized":"2024-09-17T18:36:25.583101+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:00:45.271045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1297,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.735456+0000","last_change":"2024-09-17T18:20:02.173218+0000","last_active":"2024-09-17T18:36:26.735456+0000","last_peered":"2024-09-17T18:36:26.735456+0000","last_clean":"2024-09-17T18:36:26.735456+0000","last_became_active":"2024-09-17T18:20:02.172083+0000","last_became_peered":"2024-09-17T18:20:02.172083+0000","last_unstale":"2024-09-17T18:36:26.735456+0000","last_undegraded":"2024-09-17T18:36:26.735456+0000","last_fullsized":"2024-09-17T18:36:26.735456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T21:54:16.156188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1296,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.735377+0000","last_change":"2024-09-17T18:20:02.244905+0000","last_active":"2024-09-17T18:36:26.735377+0000","last_peered":"2024-09-17T18:36:26.735377+0000","last_clean":"2024-09-17T18:36:26.735377+0000","last_became_active":"2024-09-17T18:20:02.244673+0000","last_became_peered":"2024-09-17T18:20:02.244673+0000","last_unstale":"2024-09-17T18:36:26.735377+0000","last_undegraded":"2024-09-17T18:36:26.735377+0000","last_fullsized":"2024-09-17T18:36:26.735377+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T21:29:37.865312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1297,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.735570+0000","last_change":"2024-09-17T18:20:02.316100+0000","last_active":"2024-09-17T18:36:26.735570+0000","last_peered":"2024-09-17T18:36:26.735570+0000","last_clean":"2024-09-17T18:36:26.735570+0000","last_became_active":"2024-09-17T18:20:02.315070+0000","last_became_peered":"2024-09-17T18:20:02.315070+0000","last_unstale":"2024-09-17T18:36:26.735570+0000","last_undegraded":"2024-09-17T18:36:26.735570+0000","last_fullsized":"2024-09-17T18:36:26.735570+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:45:14.819013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1296,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.735378+0000","last_change":"2024-09-17T18:20:02.348899+0000","last_active":"2024-09-17T18:36:26.735378+0000","last_peered":"2024-09-17T18:36:26.735378+0000","last_clean":"2024-09-17T18:36:26.735378+0000","last_became_active":"2024-09-17T18:20:02.348022+0000","last_became_peered":"2024-09-17T18:20:02.348022+0000","last_unstale":"2024-09-17T18:36:26.735378+0000","last_undegraded":"2024-09-17T18:36:26.735378+0000","last_fullsized":"2024-09-17T18:36:26.735378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:02:14.991712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1282,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:27.012026+0000","last_change":"2024-09-17T18:20:02.331615+0000","last_active":"2024-09-17T18:36:27.012026+0000","last_peered":"2024-09-17T18:36:27.012026+0000","last_clean":"2024-09-17T18:36:27.012026+0000","last_became_active":"2024-09-17T18:20:02.328118+0000","last_became_peered":"2024-09-17T18:20:02.328118+0000","last_unstale":"2024-09-17T18:36:27.012026+0000","last_undegraded":"2024-09-17T18:36:27.012026+0000","last_fullsized":"2024-09-17T18:36:27.012026+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:39:54.243953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1279,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.583300+0000","last_change":"2024-09-17T18:20:02.248888+0000","last_active":"2024-09-17T18:36:25.583300+0000","last_peered":"2024-09-17T18:36:25.583300+0000","last_clean":"2024-09-17T18:36:25.583300+0000","last_became_active":"2024-09-17T18:20:02.242391+0000","last_became_peered":"2024-09-17T18:20:02.242391+0000","last_unstale":"2024-09-17T18:36:25.583300+0000","last_undegraded":"2024-09-17T18:36:25.583300+0000","last_fullsized":"2024-09-17T18:36:25.583300+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:02:05.721872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1294,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.433853+0000","last_change":"2024-09-17T18:20:02.249495+0000","last_active":"2024-09-17T18:36:25.433853+0000","last_peered":"2024-09-17T18:36:25.433853+0000","last_clean":"2024-09-17T18:36:25.433853+0000","last_became_active":"2024-09-17T18:20:02.248819+0000","last_became_peered":"2024-09-17T18:20:02.248819+0000","last_unstale":"2024-09-17T18:36:25.433853+0000","last_undegraded":"2024-09-17T18:36:25.433853+0000","last_fullsized":"2024-09-17T18:36:25.433853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T20:11:51.090905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1276,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.898945+0000","last_change":"2024-09-17T18:20:02.273998+0000","last_active":"2024-09-17T18:36:26.898945+0000","last_peered":"2024-09-17T18:36:26.898945+0000","last_clean":"2024-09-17T18:36:26.898945+0000","last_became_active":"2024-09-17T18:20:02.259327+0000","last_became_peered":"2024-09-17T18:20:02.259327+0000","last_unstale":"2024-09-17T18:36:26.898945+0000","last_undegraded":"2024-09-17T18:36:26.898945+0000","last_fullsized":"2024-09-17T18:36:26.898945+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:55:51.961001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":1397,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.433765+0000","last_change":"2024-09-17T18:19:00.245854+0000","last_active":"2024-09-17T18:36:25.433765+0000","last_peered":"2024-09-17T18:36:25.433765+0000","last_clean":"2024-09-17T18:36:25.433765+0000","last_became_active":"2024-09-17T18:19:00.244903+0000","last_became_peered":"2024-09-17T18:19:00.244903+0000","last_unstale":"2024-09-17T18:36:25.433765+0000","last_undegraded":"2024-09-17T18:36:25.433765+0000","last_fullsized":"2024-09-17T18:36:25.433765+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:55:38.336150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":1296,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.583163+0000","last_change":"2024-09-17T18:19:01.532053+0000","last_active":"2024-09-17T18:36:25.583163+0000","last_peered":"2024-09-17T18:36:25.583163+0000","last_clean":"2024-09-17T18:36:25.583163+0000","last_became_active":"2024-09-17T18:18:59.526797+0000","last_became_peered":"2024-09-17T18:18:59.526797+0000","last_unstale":"2024-09-17T18:36:25.583163+0000","last_undegraded":"2024-09-17T18:36:25.583163+0000","last_fullsized":"2024-09-17T18:36:25.583163+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:19:28.196155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000425759,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1278,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.583305+0000","last_change":"2024-09-17T18:20:02.311785+0000","last_active":"2024-09-17T18:36:25.583305+0000","last_peered":"2024-09-17T18:36:25.583305+0000","last_clean":"2024-09-17T18:36:25.583305+0000","last_became_active":"2024-09-17T18:20:02.310633+0000","last_became_peered":"2024-09-17T18:20:02.310633+0000","last_unstale":"2024-09-17T18:36:25.583305+0000","last_undegraded":"2024-09-17T18:36:25.583305+0000","last_fullsized":"2024-09-17T18:36:25.583305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:08:06.293083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1312,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.433657+0000","last_change":"2024-09-17T18:19:01.527425+0000","last_active":"2024-09-17T18:36:25.433657+0000","last_peered":"2024-09-17T18:36:25.433657+0000","last_clean":"2024-09-17T18:36:25.433657+0000","last_became_active":"2024-09-17T18:18:59.526855+0000","last_became_peered":"2024-09-17T18:18:59.526855+0000","last_unstale":"2024-09-17T18:36:25.433657+0000","last_undegraded":"2024-09-17T18:36:25.433657+0000","last_fullsized":"2024-09-17T18:36:25.433657+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:05:38.754621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032532500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1297,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.735460+0000","last_change":"2024-09-17T18:20:02.260633+0000","last_active":"2024-09-17T18:36:26.735460+0000","last_peered":"2024-09-17T18:36:26.735460+0000","last_clean":"2024-09-17T18:36:26.735460+0000","last_became_active":"2024-09-17T18:20:02.260389+0000","last_became_peered":"2024-09-17T18:20:02.260389+0000","last_unstale":"2024-09-17T18:36:26.735460+0000","last_undegraded":"2024-09-17T18:36:26.735460+0000","last_fullsized":"2024-09-17T18:36:26.735460+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:20:12.888792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1312,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.735449+0000","last_change":"2024-09-17T18:19:01.527848+0000","last_active":"2024-09-17T18:36:26.735449+0000","last_peered":"2024-09-17T18:36:26.735449+0000","last_clean":"2024-09-17T18:36:26.735449+0000","last_became_active":"2024-09-17T18:18:59.529390+0000","last_became_peered":"2024-09-17T18:18:59.529390+0000","last_unstale":"2024-09-17T18:36:26.735449+0000","last_undegraded":"2024-09-17T18:36:26.735449+0000","last_fullsized":"2024-09-17T18:36:26.735449+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:40:45.376247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065986199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1299,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.717264+0000","last_change":"2024-09-17T18:20:02.260849+0000","last_active":"2024-09-17T18:36:25.717264+0000","last_peered":"2024-09-17T18:36:25.717264+0000","last_clean":"2024-09-17T18:36:25.717264+0000","last_became_active":"2024-09-17T18:20:02.251405+0000","last_became_peered":"2024-09-17T18:20:02.251405+0000","last_unstale":"2024-09-17T18:36:25.717264+0000","last_undegraded":"2024-09-17T18:36:25.717264+0000","last_fullsized":"2024-09-17T18:36:25.717264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:03:03.725725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":1297,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.583147+0000","last_change":"2024-09-17T18:19:01.532623+0000","last_active":"2024-09-17T18:36:25.583147+0000","last_peered":"2024-09-17T18:36:25.583147+0000","last_clean":"2024-09-17T18:36:25.583147+0000","last_became_active":"2024-09-17T18:18:59.526791+0000","last_became_peered":"2024-09-17T18:18:59.526791+0000","last_unstale":"2024-09-17T18:36:25.583147+0000","last_undegraded":"2024-09-17T18:36:25.583147+0000","last_fullsized":"2024-09-17T18:36:25.583147+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T21:46:24.217486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010088289999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1299,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.717202+0000","last_change":"2024-09-17T18:20:02.274215+0000","last_active":"2024-09-17T18:36:25.717202+0000","last_peered":"2024-09-17T18:36:25.717202+0000","last_clean":"2024-09-17T18:36:25.717202+0000","last_became_active":"2024-09-17T18:20:02.268644+0000","last_became_peered":"2024-09-17T18:20:02.268644+0000","last_unstale":"2024-09-17T18:36:25.717202+0000","last_undegraded":"2024-09-17T18:36:25.717202+0000","last_fullsized":"2024-09-17T18:36:25.717202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:12:55.005158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1298,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:27.012101+0000","last_change":"2024-09-17T18:19:01.527784+0000","last_active":"2024-09-17T18:36:27.012101+0000","last_peered":"2024-09-17T18:36:27.012101+0000","last_clean":"2024-09-17T18:36:27.012101+0000","last_became_active":"2024-09-17T18:19:00.244613+0000","last_became_peered":"2024-09-17T18:19:00.244613+0000","last_unstale":"2024-09-17T18:36:27.012101+0000","last_undegraded":"2024-09-17T18:36:27.012101+0000","last_fullsized":"2024-09-17T18:36:27.012101+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:31:24.599660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063646500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1280,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.583127+0000","last_change":"2024-09-17T18:20:02.254779+0000","last_active":"2024-09-17T18:36:25.583127+0000","last_peered":"2024-09-17T18:36:25.583127+0000","last_clean":"2024-09-17T18:36:25.583127+0000","last_became_active":"2024-09-17T18:20:02.254259+0000","last_became_peered":"2024-09-17T18:20:02.254259+0000","last_unstale":"2024-09-17T18:36:25.583127+0000","last_undegraded":"2024-09-17T18:36:25.583127+0000","last_fullsized":"2024-09-17T18:36:25.583127+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:02:19.820701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1311,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.433670+0000","last_change":"2024-09-17T18:19:01.527563+0000","last_active":"2024-09-17T18:36:25.433670+0000","last_peered":"2024-09-17T18:36:25.433670+0000","last_clean":"2024-09-17T18:36:25.433670+0000","last_became_active":"2024-09-17T18:19:00.244851+0000","last_became_peered":"2024-09-17T18:19:00.244851+0000","last_unstale":"2024-09-17T18:36:25.433670+0000","last_undegraded":"2024-09-17T18:36:25.433670+0000","last_fullsized":"2024-09-17T18:36:25.433670+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:19:57.036107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041282100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"19'7","reported_seq":1309,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.433672+0000","last_change":"2024-09-17T18:20:02.253575+0000","last_active":"2024-09-17T18:36:25.433672+0000","last_peered":"2024-09-17T18:36:25.433672+0000","last_clean":"2024-09-17T18:36:25.433672+0000","last_became_active":"2024-09-17T18:20:02.253328+0000","last_became_peered":"2024-09-17T18:20:02.253328+0000","last_unstale":"2024-09-17T18:36:25.433672+0000","last_undegraded":"2024-09-17T18:36:25.433672+0000","last_fullsized":"2024-09-17T18:36:25.433672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:35:00.211682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1298,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:27.012077+0000","last_change":"2024-09-17T18:19:01.527858+0000","last_active":"2024-09-17T18:36:27.012077+0000","last_peered":"2024-09-17T18:36:27.012077+0000","last_clean":"2024-09-17T18:36:27.012077+0000","last_became_active":"2024-09-17T18:18:59.525295+0000","last_became_peered":"2024-09-17T18:18:59.525295+0000","last_unstale":"2024-09-17T18:36:27.012077+0000","last_undegraded":"2024-09-17T18:36:27.012077+0000","last_fullsized":"2024-09-17T18:36:27.012077+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:41:21.181263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062121300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1297,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.735549+0000","last_change":"2024-09-17T18:20:02.234786+0000","last_active":"2024-09-17T18:36:26.735549+0000","last_peered":"2024-09-17T18:36:26.735549+0000","last_clean":"2024-09-17T18:36:26.735549+0000","last_became_active":"2024-09-17T18:20:02.234414+0000","last_became_peered":"2024-09-17T18:20:02.234414+0000","last_unstale":"2024-09-17T18:36:26.735549+0000","last_undegraded":"2024-09-17T18:36:26.735549+0000","last_fullsized":"2024-09-17T18:36:26.735549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:07:37.564071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1311,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.433654+0000","last_change":"2024-09-17T18:19:01.527658+0000","last_active":"2024-09-17T18:36:25.433654+0000","last_peered":"2024-09-17T18:36:25.433654+0000","last_clean":"2024-09-17T18:36:25.433654+0000","last_became_active":"2024-09-17T18:19:00.056752+0000","last_became_peered":"2024-09-17T18:19:00.056752+0000","last_unstale":"2024-09-17T18:36:25.433654+0000","last_undegraded":"2024-09-17T18:36:25.433654+0000","last_fullsized":"2024-09-17T18:36:25.433654+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:57:47.271964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049856399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1282,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:27.011781+0000","last_change":"2024-09-17T18:20:02.227335+0000","last_active":"2024-09-17T18:36:27.011781+0000","last_peered":"2024-09-17T18:36:27.011781+0000","last_clean":"2024-09-17T18:36:27.011781+0000","last_became_active":"2024-09-17T18:20:02.226503+0000","last_became_peered":"2024-09-17T18:20:02.226503+0000","last_unstale":"2024-09-17T18:36:27.011781+0000","last_undegraded":"2024-09-17T18:36:27.011781+0000","last_fullsized":"2024-09-17T18:36:27.011781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T03:43:28.520222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1282,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:27.011779+0000","last_change":"2024-09-17T18:20:02.313761+0000","last_active":"2024-09-17T18:36:27.011779+0000","last_peered":"2024-09-17T18:36:27.011779+0000","last_clean":"2024-09-17T18:36:27.011779+0000","last_became_active":"2024-09-17T18:20:02.311023+0000","last_became_peered":"2024-09-17T18:20:02.311023+0000","last_unstale":"2024-09-17T18:36:27.011779+0000","last_undegraded":"2024-09-17T18:36:27.011779+0000","last_fullsized":"2024-09-17T18:36:27.011779+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:18:02.124093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1275,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.898650+0000","last_change":"2024-09-17T18:20:02.351223+0000","last_active":"2024-09-17T18:36:26.898650+0000","last_peered":"2024-09-17T18:36:26.898650+0000","last_clean":"2024-09-17T18:36:26.898650+0000","last_became_active":"2024-09-17T18:20:02.339489+0000","last_became_peered":"2024-09-17T18:20:02.339489+0000","last_unstale":"2024-09-17T18:36:26.898650+0000","last_undegraded":"2024-09-17T18:36:26.898650+0000","last_fullsized":"2024-09-17T18:36:26.898650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:23:22.992560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1295,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.433542+0000","last_change":"2024-09-17T18:20:02.287570+0000","last_active":"2024-09-17T18:36:25.433542+0000","last_peered":"2024-09-17T18:36:25.433542+0000","last_clean":"2024-09-17T18:36:25.433542+0000","last_became_active":"2024-09-17T18:20:02.286560+0000","last_became_peered":"2024-09-17T18:20:02.286560+0000","last_unstale":"2024-09-17T18:36:25.433542+0000","last_undegraded":"2024-09-17T18:36:25.433542+0000","last_fullsized":"2024-09-17T18:36:25.433542+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:54:48.894903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1299,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.717249+0000","last_change":"2024-09-17T18:20:02.280897+0000","last_active":"2024-09-17T18:36:25.717249+0000","last_peered":"2024-09-17T18:36:25.717249+0000","last_clean":"2024-09-17T18:36:25.717249+0000","last_became_active":"2024-09-17T18:20:02.270521+0000","last_became_peered":"2024-09-17T18:20:02.270521+0000","last_unstale":"2024-09-17T18:36:25.717249+0000","last_undegraded":"2024-09-17T18:36:25.717249+0000","last_fullsized":"2024-09-17T18:36:25.717249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:40:16.489769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1296,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.433793+0000","last_change":"2024-09-17T18:20:02.294208+0000","last_active":"2024-09-17T18:36:25.433793+0000","last_peered":"2024-09-17T18:36:25.433793+0000","last_clean":"2024-09-17T18:36:25.433793+0000","last_became_active":"2024-09-17T18:20:02.293587+0000","last_became_peered":"2024-09-17T18:20:02.293587+0000","last_unstale":"2024-09-17T18:36:25.433793+0000","last_undegraded":"2024-09-17T18:36:25.433793+0000","last_fullsized":"2024-09-17T18:36:25.433793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:06:11.621162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1274,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.898883+0000","last_change":"2024-09-17T18:20:02.275664+0000","last_active":"2024-09-17T18:36:26.898883+0000","last_peered":"2024-09-17T18:36:26.898883+0000","last_clean":"2024-09-17T18:36:26.898883+0000","last_became_active":"2024-09-17T18:20:02.259616+0000","last_became_peered":"2024-09-17T18:20:02.259616+0000","last_unstale":"2024-09-17T18:36:26.898883+0000","last_undegraded":"2024-09-17T18:36:26.898883+0000","last_fullsized":"2024-09-17T18:36:26.898883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T21:08:28.298068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1281,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:27.012129+0000","last_change":"2024-09-17T18:20:02.314014+0000","last_active":"2024-09-17T18:36:27.012129+0000","last_peered":"2024-09-17T18:36:27.012129+0000","last_clean":"2024-09-17T18:36:27.012129+0000","last_became_active":"2024-09-17T18:20:02.313525+0000","last_became_peered":"2024-09-17T18:20:02.313525+0000","last_unstale":"2024-09-17T18:36:27.012129+0000","last_undegraded":"2024-09-17T18:36:27.012129+0000","last_fullsized":"2024-09-17T18:36:27.012129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:02:30.324064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1274,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.898897+0000","last_change":"2024-09-17T18:20:02.306541+0000","last_active":"2024-09-17T18:36:26.898897+0000","last_peered":"2024-09-17T18:36:26.898897+0000","last_clean":"2024-09-17T18:36:26.898897+0000","last_became_active":"2024-09-17T18:20:02.292188+0000","last_became_peered":"2024-09-17T18:20:02.292188+0000","last_unstale":"2024-09-17T18:36:26.898897+0000","last_undegraded":"2024-09-17T18:36:26.898897+0000","last_fullsized":"2024-09-17T18:36:26.898897+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:10:44.904234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1278,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.583267+0000","last_change":"2024-09-17T18:20:02.256567+0000","last_active":"2024-09-17T18:36:25.583267+0000","last_peered":"2024-09-17T18:36:25.583267+0000","last_clean":"2024-09-17T18:36:25.583267+0000","last_became_active":"2024-09-17T18:20:02.254813+0000","last_became_peered":"2024-09-17T18:20:02.254813+0000","last_unstale":"2024-09-17T18:36:25.583267+0000","last_undegraded":"2024-09-17T18:36:25.583267+0000","last_fullsized":"2024-09-17T18:36:25.583267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:20:59.122518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1275,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.898721+0000","last_change":"2024-09-17T18:20:02.327177+0000","last_active":"2024-09-17T18:36:26.898721+0000","last_peered":"2024-09-17T18:36:26.898721+0000","last_clean":"2024-09-17T18:36:26.898721+0000","last_became_active":"2024-09-17T18:20:02.299834+0000","last_became_peered":"2024-09-17T18:20:02.299834+0000","last_unstale":"2024-09-17T18:36:26.898721+0000","last_undegraded":"2024-09-17T18:36:26.898721+0000","last_fullsized":"2024-09-17T18:36:26.898721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:14:12.763217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1282,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:27.011862+0000","last_change":"2024-09-17T18:20:02.230651+0000","last_active":"2024-09-17T18:36:27.011862+0000","last_peered":"2024-09-17T18:36:27.011862+0000","last_clean":"2024-09-17T18:36:27.011862+0000","last_became_active":"2024-09-17T18:20:02.228502+0000","last_became_peered":"2024-09-17T18:20:02.228502+0000","last_unstale":"2024-09-17T18:36:27.011862+0000","last_undegraded":"2024-09-17T18:36:27.011862+0000","last_fullsized":"2024-09-17T18:36:27.011862+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:36:19.108865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59128,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607778,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69680,"kb_used_data":6552,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94302160,"statfs":{"total":96636764160,"available":96565411840,"internally_reserved":0,"allocated":6709248,"data_stored":2878698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607777,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78008,"kb_used_data":6556,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94293832,"statfs":{"total":96636764160,"available":96556883968,"internally_reserved":0,"allocated":6713344,"data_stored":2878717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640479,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82168,"kb_used_data":6556,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289672,"statfs":{"total":96636764160,"available":96552624128,"internally_reserved":0,"allocated":6713344,"data_stored":2878717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640481,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82224,"kb_used_data":6616,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289616,"statfs":{"total":96636764160,"available":96552566784,"internally_reserved":0,"allocated":6774784,"data_stored":3407631,"data_compressed":29564,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640480,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73836,"kb_used_data":6552,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94298004,"statfs":{"total":96636764160,"available":96561156096,"internally_reserved":0,"allocated":6709248,"data_stored":2878698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640482,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73908,"kb_used_data":6616,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94297932,"statfs":{"total":96636764160,"available":96561082368,"internally_reserved":0,"allocated":6774784,"data_stored":3407631,"data_compressed":29564,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29564,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29564,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T18:37:25.946 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T18:37:26.008 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.006+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.008 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.006+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1709440650}) 2024-09-17T18:37:26.008 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.006+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2408373614}) 2024-09-17T18:37:26.008 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.006+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.008 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.006+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1709440650}) 2024-09-17T18:37:26.009 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.006+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3062617920}) 2024-09-17T18:37:26.009 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.007+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.009 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.007+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=906676839}) 2024-09-17T18:37:26.009 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.007+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=322415341}) 2024-09-17T18:37:26.009 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.007+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.009 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.007+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=906676839}) 2024-09-17T18:37:26.009 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.007+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3332040666}) 2024-09-17T18:37:26.010 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.008+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.010 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.008+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3206684952}) 2024-09-17T18:37:26.010 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.008+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3165443457}) 2024-09-17T18:37:26.010 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.008+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.010 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.008+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3206684952}) 2024-09-17T18:37:26.010 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.008+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4074385199}) 2024-09-17T18:37:26.011 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.009+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.011 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.009+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2271454939}) 2024-09-17T18:37:26.011 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.009+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1149700344}) 2024-09-17T18:37:26.011 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.009+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.011 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.009+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2271454939}) 2024-09-17T18:37:26.011 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.009+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3209099574}) 2024-09-17T18:37:26.012 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.010+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.012 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.010+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1464827804}) 2024-09-17T18:37:26.012 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.010+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=691154699}) 2024-09-17T18:37:26.012 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.010+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.012 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.010+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1464827804}) 2024-09-17T18:37:26.012 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.010+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3289836490}) 2024-09-17T18:37:26.013 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.011+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.014 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.011+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=118872815}) 2024-09-17T18:37:26.014 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.011+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2023049429}) 2024-09-17T18:37:26.014 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.011+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.014 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.011+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=118872815}) 2024-09-17T18:37:26.014 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.011+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3277941145}) 2024-09-17T18:37:26.014 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.012+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.014 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.012+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=290342418}) 2024-09-17T18:37:26.014 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.012+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2909907839}) 2024-09-17T18:37:26.014 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.012+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.014 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.012+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=290342418}) 2024-09-17T18:37:26.014 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.012+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4063515000}) 2024-09-17T18:37:26.015 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.013+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.015 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.013+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1942531576}) 2024-09-17T18:37:26.015 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.013+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2547083387}) 2024-09-17T18:37:26.015 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.013+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.015 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.013+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1942531576}) 2024-09-17T18:37:26.015 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.013+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1260737385}) 2024-09-17T18:37:26.016 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.013+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.016 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.013+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2144310373}) 2024-09-17T18:37:26.016 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.013+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3379060077}) 2024-09-17T18:37:26.016 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.013+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.016 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.013+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2144310373}) 2024-09-17T18:37:26.016 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.013+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=407352424}) 2024-09-17T18:37:26.017 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.014+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.017 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.014+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4111129045}) 2024-09-17T18:37:26.017 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.014+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=286619076}) 2024-09-17T18:37:26.017 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.014+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.017 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.014+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4111129045}) 2024-09-17T18:37:26.017 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.014+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2464565539}) 2024-09-17T18:37:26.017 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.015+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.017 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.015+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2206473735}) 2024-09-17T18:37:26.017 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.015+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3755593550}) 2024-09-17T18:37:26.018 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.015+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.018 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.015+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2206473735}) 2024-09-17T18:37:26.018 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.015+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4150740822}) 2024-09-17T18:37:26.018 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.016+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.018 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.016+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=52845938}) 2024-09-17T18:37:26.019 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.016+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2772552911}) 2024-09-17T18:37:26.019 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.016+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.019 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.016+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=52845938}) 2024-09-17T18:37:26.019 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.016+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1236291416}) 2024-09-17T18:37:26.021 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.017+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.021 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.017+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=427140809}) 2024-09-17T18:37:26.021 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.017+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=789572684}) 2024-09-17T18:37:26.021 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.017+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.021 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.017+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=427140809}) 2024-09-17T18:37:26.021 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.017+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1284356961}) 2024-09-17T18:37:26.021 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.018+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.021 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.018+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3954695044}) 2024-09-17T18:37:26.021 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.018+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=574244414}) 2024-09-17T18:37:26.021 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.018+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.021 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.018+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3954695044}) 2024-09-17T18:37:26.021 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.018+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2374709538}) 2024-09-17T18:37:26.022 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.018+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.022 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.018+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1198622429}) 2024-09-17T18:37:26.022 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.018+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=549056755}) 2024-09-17T18:37:26.022 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.018+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.022 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.018+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1198622429}) 2024-09-17T18:37:26.022 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.018+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=567283292}) 2024-09-17T18:37:26.022 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.019+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.022 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.019+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3040223858}) 2024-09-17T18:37:26.022 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.019+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2632174277}) 2024-09-17T18:37:26.022 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.019+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.022 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.019+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3040223858}) 2024-09-17T18:37:26.022 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.019+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2819538667}) 2024-09-17T18:37:26.024 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.019+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.024 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.019+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1034221839}) 2024-09-17T18:37:26.025 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.019+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=320178305}) 2024-09-17T18:37:26.025 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.019+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.025 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.019+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1034221839}) 2024-09-17T18:37:26.025 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.019+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=99,mdsmap=1} crc {logm=708449875,mdsmap=833889183}) 2024-09-17T18:37:26.025 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.021+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.025 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.021+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=1,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=15,mon_config_key=17,monmap=3,nvmeofgw=2,osd_metadata=4,osd_pg_creating=1,osd_snap=36,osdmap=10} crc {logm=591699915,mdsmap=761013555,mgr=802028765,mgr_command_descs=1957073259,mgr_metadata=402068389,mgrstat=1568083665,mon_config_key=1582663684,monmap=2277444362,nvmeofgw=1596038039,osd_metadata=3168578409,osd_pg_creating=3478035569,osd_snap=1997112994,osdmap=1565553672}) 2024-09-17T18:37:26.025 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.021+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=1,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=15,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=35,osdmap=6} crc {logm=591699915,mdsmap=761013554,mgr=3361498112,mgr_command_descs=1957073259,mgr_metadata=402068389,mgrstat=1568083665,mon_config_key=1582663684,monmap=2277444362,nvmeofgw=598337686,osd_metadata=121115626,osd_pg_creating=3478035569,osd_snap=737004388,osdmap=4052619344}) 2024-09-17T18:37:26.025 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.021+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.025 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.021+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=1,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=15,mon_config_key=17,monmap=3,nvmeofgw=2,osd_metadata=4,osd_pg_creating=1,osd_snap=36,osdmap=10} crc {logm=591699915,mdsmap=761013555,mgr=802028765,mgr_command_descs=1957073259,mgr_metadata=402068389,mgrstat=1568083665,mon_config_key=1582663684,monmap=2277444362,nvmeofgw=1596038039,osd_metadata=3168578409,osd_pg_creating=3478035569,osd_snap=1997112994,osdmap=1565553672}) 2024-09-17T18:37:26.025 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.021+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=1,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=15,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=35,osdmap=7} crc {logm=591699915,mdsmap=761013554,mgr=3361498112,mgr_command_descs=1957073259,mgr_metadata=402068389,mgrstat=1568083665,mon_config_key=1408710985,monmap=2277444362,nvmeofgw=598337686,osd_metadata=121115626,osd_pg_creating=3478035569,osd_snap=4104447125,osdmap=4195373814}) 2024-09-17T18:37:26.025 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.021+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.025 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.021+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=813184936}) 2024-09-17T18:37:26.025 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.021+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3237698155}) 2024-09-17T18:37:26.026 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.021+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.026 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.021+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=813184936}) 2024-09-17T18:37:26.026 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.021+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3856245129}) 2024-09-17T18:37:26.026 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.023+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.026 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.023+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2546323121}) 2024-09-17T18:37:26.026 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.023+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1005923347}) 2024-09-17T18:37:26.026 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.023+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.026 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.023+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2546323121}) 2024-09-17T18:37:26.026 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.023+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2279731926}) 2024-09-17T18:37:26.027 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.023+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.027 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.023+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2878801095}) 2024-09-17T18:37:26.027 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.023+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1736388651}) 2024-09-17T18:37:26.027 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.023+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.027 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.023+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2878801095}) 2024-09-17T18:37:26.028 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.023+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3931840915}) 2024-09-17T18:37:26.028 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.024+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.028 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.024+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4212530575}) 2024-09-17T18:37:26.028 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.024+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4125044120}) 2024-09-17T18:37:26.028 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.024+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.028 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.024+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4212530575}) 2024-09-17T18:37:26.028 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.024+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1642976559}) 2024-09-17T18:37:26.028 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.026+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.029 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.026+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3046157465}) 2024-09-17T18:37:26.029 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.026+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1868861874}) 2024-09-17T18:37:26.029 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.026+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.029 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.026+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3046157465}) 2024-09-17T18:37:26.029 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.026+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1291232822}) 2024-09-17T18:37:26.030 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.028+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.030 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.028+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1061985430}) 2024-09-17T18:37:26.030 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.028+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1500698817}) 2024-09-17T18:37:26.030 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.028+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.030 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.028+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1061985430}) 2024-09-17T18:37:26.030 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.028+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2194202886}) 2024-09-17T18:37:26.034 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.033+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.034 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.033+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=24} crc {osdmap=2408940579}) 2024-09-17T18:37:26.035 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.033+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=25} crc {osdmap=1377489005}) 2024-09-17T18:37:26.035 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.033+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T18:37:26.035 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.033+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=24} crc {osdmap=2408940579}) 2024-09-17T18:37:26.035 INFO:tasks.ceph.mon.a.smithi121.stderr:2024-09-17T18:37:26.033+0000 7f8d7d951640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=24} crc {osdmap=1017595425}) 2024-09-17T18:37:26.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.044+0000 7f683c32b640 10 monclient: get_monmap_and_config 2024-09-17T18:37:26.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.044+0000 7f683c32b640 10 monclient: build_initial_monmap 2024-09-17T18:37:26.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.044+0000 7f683c32b640 10 monclient: monmap: 2024-09-17T18:37:26.046 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:26.046 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:26.046 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:37:26.045003+0000 2024-09-17T18:37:26.046 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:37:26.045003+0000 2024-09-17T18:37:26.046 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:26.046 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:26.046 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:37:26.046 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:37:26.046 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:37:26.047 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:37:26.047 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:37:26.047 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:26.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.044+0000 7f683c32b640 1 Processor -- start 2024-09-17T18:37:26.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.045+0000 7f683c32b640 1 -- start start 2024-09-17T18:37:26.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.045+0000 7f683c32b640 10 monclient: init 2024-09-17T18:37:26.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.045+0000 7f683c32b640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:26.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.045+0000 7f683c32b640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-17T18:37:26.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.045+0000 7f683c32b640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f68340a0a70 0x7f68341348f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:26.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.045+0000 7f683c32b640 10 monclient(hunting): picked mon.noname-a con 0x7f68340a0a70 addr v2:172.21.15.121:3300/0 2024-09-17T18:37:26.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.045+0000 7f683c32b640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f68340a0e50 0x7f68340a22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:26.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.045+0000 7f683c32b640 10 monclient(hunting): picked mon.noname-c con 0x7f68340a0e50 addr v2:172.21.15.173:3300/0 2024-09-17T18:37:26.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.045+0000 7f683c32b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68340a1260 0x7f68340a2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:26.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.045+0000 7f683c32b640 10 monclient(hunting): picked mon.noname-b con 0x7f68340a1260 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:26.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.045+0000 7f683c32b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:26.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.045+0000 7f683c32b640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f68340a48b0 con 0x7f68340a0a70 2024-09-17T18:37:26.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.045+0000 7f683c32b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:26.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.045+0000 7f683c32b640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f683409ef80 con 0x7f68340a0e50 2024-09-17T18:37:26.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.045+0000 7f683c32b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:26.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.045+0000 7f683c32b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f683409f5b0 con 0x7f68340a1260 2024-09-17T18:37:26.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.045+0000 7f683c32b640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:26.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.045+0000 7f683c32b640 10 monclient(hunting): authenticate will time out at 1849.761661s 2024-09-17T18:37:26.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.045+0000 7f683a0c7640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f68340a0a70 0x7f68341348f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:26.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.045+0000 7f683a8c8640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68340a1260 0x7f68340a2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:26.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.045+0000 7f683a8c8640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68340a1260 0x7f68340a2820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:36778/0 (socket says 172.21.15.121:36778) 2024-09-17T18:37:26.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.045+0000 7f683a8c8640 1 -- 172.21.15.121:0/682719703 learned_addr learned my addr 172.21.15.121:0/682719703 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:37:26.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.045+0000 7f683a0c7640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f68340a0a70 0x7f68341348f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51262/0 (socket says 172.21.15.121:51262) 2024-09-17T18:37:26.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.045+0000 7f683a8c8640 10 monclient(hunting): get_auth_request con 0x7f68340a1260 auth_method 0 2024-09-17T18:37:26.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.045+0000 7f683a8c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:26.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.045+0000 7f683a8c8640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:26.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.045+0000 7f683a8c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:26.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.045+0000 7f683a0c7640 10 monclient(hunting): get_auth_request con 0x7f68340a0a70 auth_method 0 2024-09-17T18:37:26.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.045+0000 7f683a0c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:26.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.045+0000 7f683a0c7640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:26.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.045+0000 7f683a0c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:26.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.045+0000 7f683a0c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:26.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.045+0000 7f683a0c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:26.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.045+0000 7f683a0c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:26.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.045+0000 7f683a8c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:26.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.045+0000 7f683a8c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:26.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.045+0000 7f683a8c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:26.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.046+0000 7f68398c6640 1 --2- 172.21.15.121:0/682719703 >> v2:172.21.15.173:3300/0 conn(0x7f68340a0e50 0x7f68340a22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:26.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.046+0000 7f683a0c7640 10 monclient(hunting): handle_auth_done global_id 114141 payload 274 2024-09-17T18:37:26.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.046+0000 7f683a0c7640 1 -- 172.21.15.121:0/682719703 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68340a1260 msgr2=0x7f68340a2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:26.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.046+0000 7f683a0c7640 1 --2- 172.21.15.121:0/682719703 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68340a1260 0x7f68340a2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:26.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.046+0000 7f683a0c7640 1 -- 172.21.15.121:0/682719703 >> v2:172.21.15.173:3300/0 conn(0x7f68340a0e50 msgr2=0x7f68340a22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:26.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.046+0000 7f683a0c7640 1 --2- 172.21.15.121:0/682719703 >> v2:172.21.15.173:3300/0 conn(0x7f68340a0e50 0x7f68340a22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:26.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.046+0000 7f683a0c7640 10 monclient: _finish_hunting 0 2024-09-17T18:37:26.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.046+0000 7f683a0c7640 1 monclient: found mon.noname-a 2024-09-17T18:37:26.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.046+0000 7f683a0c7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:26.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.046+0000 7f683a0c7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:37:26.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.046+0000 7f683a0c7640 1 -- 172.21.15.121:0/682719703 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f683409eca0 con 0x7f68340a0a70 2024-09-17T18:37:26.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.046+0000 7f68398c6640 10 monclient: get_auth_request con 0x7f68340a0e50 auth_method 0 2024-09-17T18:37:26.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.046+0000 7f683a8c8640 1 --2- 172.21.15.121:0/682719703 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68340a1260 0x7f68340a2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:37:26.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.046+0000 7f68398c6640 1 --2- 172.21.15.121:0/682719703 >> v2:172.21.15.173:3300/0 conn(0x7f68340a0e50 0x7f68340a22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T18:37:26.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.048+0000 7f683a0c7640 1 -- 172.21.15.121:0/682719703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68340a0a70 msgr2=0x7f68341348f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:26.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.048+0000 7f683a0c7640 1 --2- 172.21.15.121:0/682719703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68340a0a70 0x7f68341348f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f68340a34c0 tx=0x7f683002c700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c96949e91944f770 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:26.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.048+0000 7f683a0c7640 1 -- 172.21.15.121:0/682719703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68340a0a70 msgr2=0x7f68341348f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-17T18:37:26.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.048+0000 7f683a0c7640 1 -- 172.21.15.121:0/682719703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68340a0a70 msgr2=0x7f68341348f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:37:26.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.048+0000 7f68390c5640 1 -- 172.21.15.121:0/682719703 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6830031920 con 0x7f68340a0a70 2024-09-17T18:37:26.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.048+0000 7f68390c5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:26.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.048+0000 7f683a0c7640 1 --2- 172.21.15.121:0/682719703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68340a0a70 0x7f68341348f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f68340a34c0 tx=0x7f683002c700 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:37:26.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.048+0000 7f683a0c7640 1 --2- 172.21.15.121:0/682719703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68340a0a70 0x7f68341348f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f68340a34c0 tx=0x7f683002c700 comp rx=0 tx=0).stop 2024-09-17T18:37:26.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.048+0000 7f68390c5640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:37:26.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.048+0000 7f68390c5640 10 monclient: dump: 2024-09-17T18:37:26.062 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:26.062 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:26.062 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:26.062 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:26.062 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:26.062 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:26.062 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:26.062 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:26.062 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:26.062 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:26.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.048+0000 7f68390c5640 10 monclient: _finish_auth 0 2024-09-17T18:37:26.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.048+0000 7f68390c5640 10 monclient: _check_auth_tickets 2024-09-17T18:37:26.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.048+0000 7f68390c5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:26.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f68390c5640 10 monclient: ms_handle_reset current mon [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:37:26.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f68390c5640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:26.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f68390c5640 1 -- 172.21.15.121:0/682719703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68340a0a70 msgr2=0x7f68341348f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:37:26.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f68390c5640 1 --2- 172.21.15.121:0/682719703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68340a0a70 0x7f68341348f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:26.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f68390c5640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T18:37:26.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f68390c5640 1 --2- 172.21.15.121:0/682719703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6820003750 0x7f6820005b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:26.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f68390c5640 10 monclient(hunting): picked mon.a con 0x7f6820003750 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:37:26.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f68390c5640 1 --2- 172.21.15.121:0/682719703 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68340a1260 0x7f68200083f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:26.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f68390c5640 10 monclient(hunting): picked mon.c con 0x7f68340a1260 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:26.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f68390c5640 1 --2- 172.21.15.121:0/682719703 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6820008b90 0x7f682000b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:26.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f68390c5640 10 monclient(hunting): picked mon.b con 0x7f6820008b90 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:37:26.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f68390c5640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:26.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f68390c5640 1 -- 172.21.15.121:0/682719703 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f682000b900 con 0x7f6820003750 2024-09-17T18:37:26.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f68390c5640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:26.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f68390c5640 1 -- 172.21.15.121:0/682719703 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f682000bab0 con 0x7f6820008b90 2024-09-17T18:37:26.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f68390c5640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:26.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f68390c5640 1 -- 172.21.15.121:0/682719703 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f682000bcb0 con 0x7f68340a1260 2024-09-17T18:37:26.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f68390c5640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:26.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f68390c5640 10 monclient(hunting): ms_handle_reset hunted mon [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:37:26.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f68398c6640 1 --2- 172.21.15.121:0/682719703 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68340a1260 0x7f68200083f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:26.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f683a0c7640 1 --2- 172.21.15.121:0/682719703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6820003750 0x7f6820005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:26.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f68398c6640 10 monclient(hunting): get_auth_request con 0x7f68340a1260 auth_method 0 2024-09-17T18:37:26.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f68398c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:26.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f68398c6640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:26.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f68398c6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:37:26.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f683a0c7640 10 monclient(hunting): get_auth_request con 0x7f6820003750 auth_method 0 2024-09-17T18:37:26.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f683a0c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:26.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f683a0c7640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:26.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f683a0c7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:37:26.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f68398c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:26.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f68398c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:26.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f68398c6640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:37:26.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f683a0c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:26.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f683a0c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:26.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f683a0c7640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:37:26.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f68398c6640 10 monclient(hunting): handle_auth_done global_id 114141 payload 293 2024-09-17T18:37:26.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f68398c6640 1 -- 172.21.15.121:0/682719703 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6820008b90 msgr2=0x7f682000b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:26.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f68398c6640 1 --2- 172.21.15.121:0/682719703 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6820008b90 0x7f682000b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:26.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f68398c6640 1 -- 172.21.15.121:0/682719703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6820003750 msgr2=0x7f6820005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:26.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f68398c6640 1 --2- 172.21.15.121:0/682719703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6820003750 0x7f6820005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:26.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f68398c6640 10 monclient: _finish_hunting 0 2024-09-17T18:37:26.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f68398c6640 1 monclient: found mon.c 2024-09-17T18:37:26.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f68398c6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:26.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f68398c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:37:26.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f68398c6640 1 -- 172.21.15.121:0/682719703 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=2+}) -- 0x7f682000bf70 con 0x7f68340a1260 2024-09-17T18:37:26.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f68398c6640 10 monclient: _finish_auth 0 2024-09-17T18:37:26.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f68398c6640 10 monclient: _check_auth_tickets 2024-09-17T18:37:26.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f68398c6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:26.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f683c32b640 5 monclient: authenticate success, global_id 114141 2024-09-17T18:37:26.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f683c32b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:37:26.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.049+0000 7f683a0c7640 1 --2- 172.21.15.121:0/682719703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6820003750 0x7f6820005b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:37:26.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.051+0000 7f68398c6640 1 -- 172.21.15.121:0/682719703 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68340a1260 msgr2=0x7f68200083f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:26.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.051+0000 7f68398c6640 1 --2- 172.21.15.121:0/682719703 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68340a1260 0x7f68200083f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f682400aa80 tx=0x7f683000b3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b9b5d997bcfff6f server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:26.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.051+0000 7f68390c5640 1 -- 172.21.15.121:0/682719703 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6824015070 con 0x7f68340a1260 2024-09-17T18:37:26.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.051+0000 7f68390c5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:26.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.051+0000 7f68390c5640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:37:26.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.051+0000 7f68390c5640 10 monclient: dump: 2024-09-17T18:37:26.067 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:26.067 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:26.068 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:26.068 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:26.068 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:26.068 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:26.068 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:26.068 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:26.068 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:26.068 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:26.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.051+0000 7f68390c5640 1 -- 172.21.15.121:0/682719703 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f682400cc60 con 0x7f68340a1260 2024-09-17T18:37:26.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.051+0000 7f683c32b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:37:26.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.051+0000 7f68390c5640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:26.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.051+0000 7f683c32b640 10 monclient: get_monmap_and_config success 2024-09-17T18:37:26.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.051+0000 7f683c32b640 4 set_mon_vals no callback set 2024-09-17T18:37:26.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.052+0000 7f683c32b640 10 monclient: shutdown 2024-09-17T18:37:26.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.052+0000 7f683c32b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:26.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.052+0000 7f683c32b640 1 -- 172.21.15.121:0/682719703 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68340a1260 msgr2=0x7f68200083f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:26.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.052+0000 7f683c32b640 1 --2- 172.21.15.121:0/682719703 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68340a1260 0x7f68200083f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f682400aa80 tx=0x7f683000b3a0 comp rx=0 tx=0).stop 2024-09-17T18:37:26.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.052+0000 7f683a8c8640 1 -- 172.21.15.121:0/682719703 reap_dead start 2024-09-17T18:37:26.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.052+0000 7f683c32b640 1 -- 172.21.15.121:0/682719703 shutdown_connections 2024-09-17T18:37:26.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.052+0000 7f683c32b640 1 --2- 172.21.15.121:0/682719703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68340a0a70 0x7f68341348f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:26.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.052+0000 7f683c32b640 1 -- 172.21.15.121:0/682719703 >> 172.21.15.121:0/682719703 conn(0x7f68340a3b50 msgr2=0x7f68340a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:26.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.052+0000 7f683c32b640 1 -- 172.21.15.121:0/682719703 shutdown_connections 2024-09-17T18:37:26.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.052+0000 7f683c32b640 1 -- 172.21.15.121:0/682719703 wait complete. 2024-09-17T18:37:26.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.053+0000 7f683c32b640 10 monclient: build_initial_monmap 2024-09-17T18:37:26.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.053+0000 7f683c32b640 10 monclient: monmap: 2024-09-17T18:37:26.069 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:26.069 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:37:26.070 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:37:26.070 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:37:26.070 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:26.070 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:26.070 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:37:26.070 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:37:26.070 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:37:26.070 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:26.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.053+0000 7f683c32b640 1 librados: starting msgr at 2024-09-17T18:37:26.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.053+0000 7f683c32b640 1 librados: starting objecter 2024-09-17T18:37:26.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.053+0000 7f683c32b640 1 Processor -- start 2024-09-17T18:37:26.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.053+0000 7f683c32b640 1 -- start start 2024-09-17T18:37:26.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.053+0000 7f683c32b640 1 librados: setting wanted keys 2024-09-17T18:37:26.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.053+0000 7f683c32b640 1 librados: calling monclient init 2024-09-17T18:37:26.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.053+0000 7f683c32b640 10 monclient: init 2024-09-17T18:37:26.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.053+0000 7f683c32b640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:26.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.053+0000 7f683c32b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T18:37:26.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.053+0000 7f683c32b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f68340a0a70 0x7f683411fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:26.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.053+0000 7f683c32b640 10 monclient(hunting): picked mon.noname-b con 0x7f68340a0a70 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:37:26.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.053+0000 7f683c32b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68340a1260 0x7f6834120380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:26.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.053+0000 7f683c32b640 10 monclient(hunting): picked mon.noname-a con 0x7f68340a1260 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:37:26.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.053+0000 7f683c32b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68341208c0 0x7f68341cf7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:26.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.053+0000 7f683c32b640 10 monclient(hunting): picked mon.noname-c con 0x7f68341208c0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:26.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.053+0000 7f683c32b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:26.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.053+0000 7f683c32b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f683409de60 con 0x7f68340a1260 2024-09-17T18:37:26.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.053+0000 7f683c32b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:26.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.053+0000 7f683c32b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f683409dce0 con 0x7f68340a0a70 2024-09-17T18:37:26.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.053+0000 7f683c32b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:26.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.053+0000 7f683c32b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f68340a1a50 con 0x7f68341208c0 2024-09-17T18:37:26.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.053+0000 7f683c32b640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:26.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.053+0000 7f683c32b640 10 monclient(hunting): authenticate will time out at 1849.769879s 2024-09-17T18:37:26.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.053+0000 7f68398c6640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68340a1260 0x7f6834120380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:26.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.053+0000 7f683a8c8640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68341208c0 0x7f68341cf7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:26.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.053+0000 7f68398c6640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68340a1260 0x7f6834120380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51286/0 (socket says 172.21.15.121:51286) 2024-09-17T18:37:26.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.053+0000 7f68398c6640 1 -- 172.21.15.121:0/3383871338 learned_addr learned my addr 172.21.15.121:0/3383871338 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:37:26.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.053+0000 7f68398c6640 10 monclient(hunting): get_auth_request con 0x7f68340a1260 auth_method 0 2024-09-17T18:37:26.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.053+0000 7f68398c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:26.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.053+0000 7f68398c6640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:26.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.053+0000 7f68398c6640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:26.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.053+0000 7f683a8c8640 10 monclient(hunting): get_auth_request con 0x7f68341208c0 auth_method 0 2024-09-17T18:37:26.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.053+0000 7f683a8c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:26.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.053+0000 7f683a8c8640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:26.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.053+0000 7f683a8c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:26.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.054+0000 7f68398c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:26.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.054+0000 7f68398c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:26.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.054+0000 7f68398c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:26.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.054+0000 7f683a8c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:26.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.054+0000 7f683a8c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:26.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.054+0000 7f683a8c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:26.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.054+0000 7f68398c6640 10 monclient(hunting): handle_auth_done global_id 114144 payload 824 2024-09-17T18:37:26.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.054+0000 7f683a0c7640 1 --2- 172.21.15.121:0/3383871338 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f68340a0a70 0x7f683411fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:26.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.054+0000 7f68398c6640 1 -- 172.21.15.121:0/3383871338 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68341208c0 msgr2=0x7f68341cf7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:26.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.054+0000 7f68398c6640 1 --2- 172.21.15.121:0/3383871338 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68341208c0 0x7f68341cf7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:26.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.054+0000 7f68398c6640 1 -- 172.21.15.121:0/3383871338 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f68340a0a70 msgr2=0x7f683411fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:26.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.054+0000 7f68398c6640 1 --2- 172.21.15.121:0/3383871338 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f68340a0a70 0x7f683411fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:26.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.054+0000 7f68398c6640 10 monclient: _finish_hunting 0 2024-09-17T18:37:26.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.054+0000 7f68398c6640 1 monclient: found mon.noname-a 2024-09-17T18:37:26.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.054+0000 7f68398c6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:26.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.054+0000 7f68398c6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:37:26.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.054+0000 7f68398c6640 1 -- 172.21.15.121:0/3383871338 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6834123b70 con 0x7f68340a1260 2024-09-17T18:37:26.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.056+0000 7f68398c6640 1 -- 172.21.15.121:0/3383871338 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68340a1260 msgr2=0x7f6834120380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:26.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.056+0000 7f68398c6640 1 --2- 172.21.15.121:0/3383871338 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68340a1260 0x7f6834120380 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f682000be80 tx=0x7f6824004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:26.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.056+0000 7f682affd640 10 client.?.objecter ms_handle_connect 0x7f68340a1260 2024-09-17T18:37:26.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.056+0000 7f682affd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:37:26.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.056+0000 7f682affd640 1 -- 172.21.15.121:0/3383871338 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f682402a070 con 0x7f68340a1260 2024-09-17T18:37:26.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.056+0000 7f682affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:26.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.056+0000 7f682affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:37:26.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.056+0000 7f682affd640 10 monclient: dump: 2024-09-17T18:37:26.074 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:26.074 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:26.074 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:26.074 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:26.074 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:26.075 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:26.075 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:26.075 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:26.075 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:26.075 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:26.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.056+0000 7f682affd640 10 monclient: _finish_auth 0 2024-09-17T18:37:26.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.056+0000 7f682affd640 10 monclient: _check_auth_tickets 2024-09-17T18:37:26.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.056+0000 7f682affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:26.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.056+0000 7f683c32b640 5 monclient: authenticate success, global_id 114144 2024-09-17T18:37:26.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.056+0000 7f682affd640 1 -- 172.21.15.121:0/3383871338 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f682401bbc0 con 0x7f68340a1260 2024-09-17T18:37:26.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.056+0000 7f682affd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:26.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.056+0000 7f682affd640 1 -- 172.21.15.121:0/3383871338 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6824015040 con 0x7f68340a1260 2024-09-17T18:37:26.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.056+0000 7f683c32b640 10 monclient: _renew_subs 2024-09-17T18:37:26.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.056+0000 7f683c32b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:26.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.056+0000 7f683c32b640 1 -- 172.21.15.121:0/3383871338 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f683409e240 con 0x7f68340a1260 2024-09-17T18:37:26.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.056+0000 7f682affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:26.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.056+0000 7f682affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:37:26.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.056+0000 7f682affd640 10 monclient: dump: 2024-09-17T18:37:26.076 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:26.076 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:26.076 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:26.076 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:26.076 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:26.076 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:26.076 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:26.076 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:26.076 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:26.076 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:26.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.056+0000 7f68388c4640 4 set_mon_vals no callback set 2024-09-17T18:37:26.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.056+0000 7f683c32b640 10 client.114144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:37:26.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.056+0000 7f683c32b640 10 monclient: _renew_subs 2024-09-17T18:37:26.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.056+0000 7f683c32b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:26.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.056+0000 7f683c32b640 1 -- 172.21.15.121:0/3383871338 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6834121810 con 0x7f68340a1260 2024-09-17T18:37:26.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.056+0000 7f683c32b640 1 librados: init done 2024-09-17T18:37:26.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.057+0000 7f683c32b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:26.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.057+0000 7f682affd640 1 -- 172.21.15.121:0/3383871338 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f682402d070 con 0x7f68340a1260 2024-09-17T18:37:26.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.057+0000 7f683c32b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:26.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.057+0000 7f683c32b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:26.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.057+0000 7f683c32b640 1 -- 172.21.15.121:0/3383871338 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68341348f0 con 0x7f68340a1260 2024-09-17T18:37:26.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.057+0000 7f682affd640 1 --2- 172.21.15.121:0/3383871338 >> v2:172.21.15.173:6812/2423786037 conn(0x7f68180401a0 0x7f6818042660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:26.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.057+0000 7f682affd640 1 -- 172.21.15.121:0/3383871338 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(679..679 src has 247..679) ==== 4685+0+0 (secure 0 0 0) 0x7f68240547a0 con 0x7f68340a1260 2024-09-17T18:37:26.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.057+0000 7f682affd640 10 client.114144.objecter ms_dispatch 0x7f6834000b90 osd_map(679..679 src has 247..679) 2024-09-17T18:37:26.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.057+0000 7f682affd640 3 client.114144.objecter handle_osd_map got epochs [679,679] > 0 2024-09-17T18:37:26.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.057+0000 7f682affd640 3 client.114144.objecter handle_osd_map decoding full epoch 679 2024-09-17T18:37:26.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.057+0000 7f682affd640 20 client.114144.objecter dump_active .. 0 homeless 2024-09-17T18:37:26.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.058+0000 7f683a0c7640 1 --2- 172.21.15.121:0/3383871338 >> v2:172.21.15.173:6812/2423786037 conn(0x7f68180401a0 0x7f6818042660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:26.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.058+0000 7f683a0c7640 10 monclient: get_auth_request con 0x7f68180401a0 auth_method 0 2024-09-17T18:37:26.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.060+0000 7f683a0c7640 1 --2- 172.21.15.121:0/3383871338 >> v2:172.21.15.173:6812/2423786037 conn(0x7f68180401a0 0x7f6818042660 secure :-1 s=READY pgs=4097 cs=0 l=1 rev1=1 crypto rx=0x7f683002c740 tx=0x7f6830031000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:26.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.060+0000 7f682affd640 10 client.114144.objecter ms_handle_connect 0x7f68180401a0 2024-09-17T18:37:26.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.062+0000 7f682affd640 1 -- 172.21.15.121:0/3383871338 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6824020070 con 0x7f68340a1260 2024-09-17T18:37:26.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.062+0000 7f682affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:26.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.062+0000 7f682affd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:37:26.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.233+0000 7f683c32b640 1 -- 172.21.15.121:0/3383871338 --> v2:172.21.15.173:6812/2423786037 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f68340a5490 con 0x7f68180401a0 2024-09-17T18:37:26.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.236+0000 7f682affd640 1 -- 172.21.15.121:0/3383871338 <== mgr.4105 v2:172.21.15.173:6812/2423786037 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114139 (secure 0 0 0) 0x7f68340a5490 con 0x7f68180401a0 2024-09-17T18:37:26.239 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:37:26.239 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-09-17T18:37:26.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.238+0000 7f683c32b640 10 librados: watch_flush enter 2024-09-17T18:37:26.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.238+0000 7f683c32b640 10 librados: watch_flush exit 2024-09-17T18:37:26.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.238+0000 7f683c32b640 20 client.114144.objecter shutdown clearing up homeless session... 2024-09-17T18:37:26.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.238+0000 7f683c32b640 10 client.114144.objecter successfully canceled tick 2024-09-17T18:37:26.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.239+0000 7f683c32b640 1 -- 172.21.15.121:0/3383871338 >> v2:172.21.15.173:6812/2423786037 conn(0x7f68180401a0 msgr2=0x7f6818042660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:26.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.239+0000 7f683c32b640 1 --2- 172.21.15.121:0/3383871338 >> v2:172.21.15.173:6812/2423786037 conn(0x7f68180401a0 0x7f6818042660 secure :-1 s=READY pgs=4097 cs=0 l=1 rev1=1 crypto rx=0x7f683002c740 tx=0x7f6830031000 comp rx=0 tx=0).stop 2024-09-17T18:37:26.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.239+0000 7f683c32b640 10 monclient: shutdown 2024-09-17T18:37:26.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.239+0000 7f683c32b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:26.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.239+0000 7f683c32b640 1 -- 172.21.15.121:0/3383871338 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68340a1260 msgr2=0x7f6834120380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:26.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.239+0000 7f683c32b640 1 --2- 172.21.15.121:0/3383871338 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68340a1260 0x7f6834120380 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f682000be80 tx=0x7f6824004730 comp rx=0 tx=0).stop 2024-09-17T18:37:26.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.239+0000 7f683c32b640 1 -- 172.21.15.121:0/3383871338 shutdown_connections 2024-09-17T18:37:26.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.239+0000 7f683c32b640 1 --2- 172.21.15.121:0/3383871338 >> v2:172.21.15.173:6812/2423786037 conn(0x7f68180401a0 0x7f6818042660 unknown :-1 s=CLOSED pgs=4097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:26.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.239+0000 7f683c32b640 1 --2- 172.21.15.121:0/3383871338 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68341208c0 0x7f68341cf7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:26.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.239+0000 7f683c32b640 1 --2- 172.21.15.121:0/3383871338 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68340a1260 0x7f6834120380 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:26.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.239+0000 7f683c32b640 1 --2- 172.21.15.121:0/3383871338 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f68340a0a70 0x7f683411fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:26.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.239+0000 7f683c32b640 1 -- 172.21.15.121:0/3383871338 >> 172.21.15.121:0/3383871338 conn(0x7f68340a3b50 msgr2=0x7f6834073ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:26.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.239+0000 7f683c32b640 1 -- 172.21.15.121:0/3383871338 shutdown_connections 2024-09-17T18:37:26.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.239+0000 7f683c32b640 1 -- 172.21.15.121:0/3383871338 wait complete. 2024-09-17T18:37:26.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.239+0000 7f683c32b640 1 librados: shutdown 2024-09-17T18:37:26.251 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":1227,"stamp":"2024-09-17T18:37:25.778399+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":459824,"kb_used_data":39448,"kb_used_omap":9,"kb_used_meta":420214,"kb_avail":565771216,"statfs":{"total":579820584960,"available":579349725184,"internally_reserved":0,"allocated":40394752,"data_stored":18330092,"data_compressed":59128,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":430299809},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003191"},"pg_stats":[{"pgid":"5.15","version":"0'0","reported_seq":1279,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.583221+0000","last_change":"2024-09-17T18:20:02.310686+0000","last_active":"2024-09-17T18:36:25.583221+0000","last_peered":"2024-09-17T18:36:25.583221+0000","last_clean":"2024-09-17T18:36:25.583221+0000","last_became_active":"2024-09-17T18:20:02.310239+0000","last_became_peered":"2024-09-17T18:20:02.310239+0000","last_unstale":"2024-09-17T18:36:25.583221+0000","last_undegraded":"2024-09-17T18:36:25.583221+0000","last_fullsized":"2024-09-17T18:36:25.583221+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:52:11.931160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1298,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.717134+0000","last_change":"2024-09-17T18:20:02.326174+0000","last_active":"2024-09-17T18:36:25.717134+0000","last_peered":"2024-09-17T18:36:25.717134+0000","last_clean":"2024-09-17T18:36:25.717134+0000","last_became_active":"2024-09-17T18:20:02.322561+0000","last_became_peered":"2024-09-17T18:20:02.322561+0000","last_unstale":"2024-09-17T18:36:25.717134+0000","last_undegraded":"2024-09-17T18:36:25.717134+0000","last_fullsized":"2024-09-17T18:36:25.717134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:29:09.470382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1296,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.735731+0000","last_change":"2024-09-17T18:20:02.250996+0000","last_active":"2024-09-17T18:36:26.735731+0000","last_peered":"2024-09-17T18:36:26.735731+0000","last_clean":"2024-09-17T18:36:26.735731+0000","last_became_active":"2024-09-17T18:20:02.250675+0000","last_became_peered":"2024-09-17T18:20:02.250675+0000","last_unstale":"2024-09-17T18:36:26.735731+0000","last_undegraded":"2024-09-17T18:36:26.735731+0000","last_fullsized":"2024-09-17T18:36:26.735731+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:17:16.240715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1295,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.433900+0000","last_change":"2024-09-17T18:20:02.342701+0000","last_active":"2024-09-17T18:36:25.433900+0000","last_peered":"2024-09-17T18:36:25.433900+0000","last_clean":"2024-09-17T18:36:25.433900+0000","last_became_active":"2024-09-17T18:20:02.342457+0000","last_became_peered":"2024-09-17T18:20:02.342457+0000","last_unstale":"2024-09-17T18:36:25.433900+0000","last_undegraded":"2024-09-17T18:36:25.433900+0000","last_fullsized":"2024-09-17T18:36:25.433900+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:57:23.288202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1280,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.583101+0000","last_change":"2024-09-17T18:20:02.251851+0000","last_active":"2024-09-17T18:36:25.583101+0000","last_peered":"2024-09-17T18:36:25.583101+0000","last_clean":"2024-09-17T18:36:25.583101+0000","last_became_active":"2024-09-17T18:20:02.249100+0000","last_became_peered":"2024-09-17T18:20:02.249100+0000","last_unstale":"2024-09-17T18:36:25.583101+0000","last_undegraded":"2024-09-17T18:36:25.583101+0000","last_fullsized":"2024-09-17T18:36:25.583101+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:00:45.271045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1297,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.735456+0000","last_change":"2024-09-17T18:20:02.173218+0000","last_active":"2024-09-17T18:36:26.735456+0000","last_peered":"2024-09-17T18:36:26.735456+0000","last_clean":"2024-09-17T18:36:26.735456+0000","last_became_active":"2024-09-17T18:20:02.172083+0000","last_became_peered":"2024-09-17T18:20:02.172083+0000","last_unstale":"2024-09-17T18:36:26.735456+0000","last_undegraded":"2024-09-17T18:36:26.735456+0000","last_fullsized":"2024-09-17T18:36:26.735456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T21:54:16.156188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1296,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.735377+0000","last_change":"2024-09-17T18:20:02.244905+0000","last_active":"2024-09-17T18:36:26.735377+0000","last_peered":"2024-09-17T18:36:26.735377+0000","last_clean":"2024-09-17T18:36:26.735377+0000","last_became_active":"2024-09-17T18:20:02.244673+0000","last_became_peered":"2024-09-17T18:20:02.244673+0000","last_unstale":"2024-09-17T18:36:26.735377+0000","last_undegraded":"2024-09-17T18:36:26.735377+0000","last_fullsized":"2024-09-17T18:36:26.735377+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T21:29:37.865312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1297,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.735570+0000","last_change":"2024-09-17T18:20:02.316100+0000","last_active":"2024-09-17T18:36:26.735570+0000","last_peered":"2024-09-17T18:36:26.735570+0000","last_clean":"2024-09-17T18:36:26.735570+0000","last_became_active":"2024-09-17T18:20:02.315070+0000","last_became_peered":"2024-09-17T18:20:02.315070+0000","last_unstale":"2024-09-17T18:36:26.735570+0000","last_undegraded":"2024-09-17T18:36:26.735570+0000","last_fullsized":"2024-09-17T18:36:26.735570+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:45:14.819013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1296,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.735378+0000","last_change":"2024-09-17T18:20:02.348899+0000","last_active":"2024-09-17T18:36:26.735378+0000","last_peered":"2024-09-17T18:36:26.735378+0000","last_clean":"2024-09-17T18:36:26.735378+0000","last_became_active":"2024-09-17T18:20:02.348022+0000","last_became_peered":"2024-09-17T18:20:02.348022+0000","last_unstale":"2024-09-17T18:36:26.735378+0000","last_undegraded":"2024-09-17T18:36:26.735378+0000","last_fullsized":"2024-09-17T18:36:26.735378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:02:14.991712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1282,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:27.012026+0000","last_change":"2024-09-17T18:20:02.331615+0000","last_active":"2024-09-17T18:36:27.012026+0000","last_peered":"2024-09-17T18:36:27.012026+0000","last_clean":"2024-09-17T18:36:27.012026+0000","last_became_active":"2024-09-17T18:20:02.328118+0000","last_became_peered":"2024-09-17T18:20:02.328118+0000","last_unstale":"2024-09-17T18:36:27.012026+0000","last_undegraded":"2024-09-17T18:36:27.012026+0000","last_fullsized":"2024-09-17T18:36:27.012026+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:39:54.243953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1279,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.583300+0000","last_change":"2024-09-17T18:20:02.248888+0000","last_active":"2024-09-17T18:36:25.583300+0000","last_peered":"2024-09-17T18:36:25.583300+0000","last_clean":"2024-09-17T18:36:25.583300+0000","last_became_active":"2024-09-17T18:20:02.242391+0000","last_became_peered":"2024-09-17T18:20:02.242391+0000","last_unstale":"2024-09-17T18:36:25.583300+0000","last_undegraded":"2024-09-17T18:36:25.583300+0000","last_fullsized":"2024-09-17T18:36:25.583300+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:02:05.721872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1294,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.433853+0000","last_change":"2024-09-17T18:20:02.249495+0000","last_active":"2024-09-17T18:36:25.433853+0000","last_peered":"2024-09-17T18:36:25.433853+0000","last_clean":"2024-09-17T18:36:25.433853+0000","last_became_active":"2024-09-17T18:20:02.248819+0000","last_became_peered":"2024-09-17T18:20:02.248819+0000","last_unstale":"2024-09-17T18:36:25.433853+0000","last_undegraded":"2024-09-17T18:36:25.433853+0000","last_fullsized":"2024-09-17T18:36:25.433853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T20:11:51.090905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1276,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.898945+0000","last_change":"2024-09-17T18:20:02.273998+0000","last_active":"2024-09-17T18:36:26.898945+0000","last_peered":"2024-09-17T18:36:26.898945+0000","last_clean":"2024-09-17T18:36:26.898945+0000","last_became_active":"2024-09-17T18:20:02.259327+0000","last_became_peered":"2024-09-17T18:20:02.259327+0000","last_unstale":"2024-09-17T18:36:26.898945+0000","last_undegraded":"2024-09-17T18:36:26.898945+0000","last_fullsized":"2024-09-17T18:36:26.898945+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:55:51.961001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":1397,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.433765+0000","last_change":"2024-09-17T18:19:00.245854+0000","last_active":"2024-09-17T18:36:25.433765+0000","last_peered":"2024-09-17T18:36:25.433765+0000","last_clean":"2024-09-17T18:36:25.433765+0000","last_became_active":"2024-09-17T18:19:00.244903+0000","last_became_peered":"2024-09-17T18:19:00.244903+0000","last_unstale":"2024-09-17T18:36:25.433765+0000","last_undegraded":"2024-09-17T18:36:25.433765+0000","last_fullsized":"2024-09-17T18:36:25.433765+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:55:38.336150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":1296,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.583163+0000","last_change":"2024-09-17T18:19:01.532053+0000","last_active":"2024-09-17T18:36:25.583163+0000","last_peered":"2024-09-17T18:36:25.583163+0000","last_clean":"2024-09-17T18:36:25.583163+0000","last_became_active":"2024-09-17T18:18:59.526797+0000","last_became_peered":"2024-09-17T18:18:59.526797+0000","last_unstale":"2024-09-17T18:36:25.583163+0000","last_undegraded":"2024-09-17T18:36:25.583163+0000","last_fullsized":"2024-09-17T18:36:25.583163+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:19:28.196155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000425759,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1278,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.583305+0000","last_change":"2024-09-17T18:20:02.311785+0000","last_active":"2024-09-17T18:36:25.583305+0000","last_peered":"2024-09-17T18:36:25.583305+0000","last_clean":"2024-09-17T18:36:25.583305+0000","last_became_active":"2024-09-17T18:20:02.310633+0000","last_became_peered":"2024-09-17T18:20:02.310633+0000","last_unstale":"2024-09-17T18:36:25.583305+0000","last_undegraded":"2024-09-17T18:36:25.583305+0000","last_fullsized":"2024-09-17T18:36:25.583305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:08:06.293083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1312,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.433657+0000","last_change":"2024-09-17T18:19:01.527425+0000","last_active":"2024-09-17T18:36:25.433657+0000","last_peered":"2024-09-17T18:36:25.433657+0000","last_clean":"2024-09-17T18:36:25.433657+0000","last_became_active":"2024-09-17T18:18:59.526855+0000","last_became_peered":"2024-09-17T18:18:59.526855+0000","last_unstale":"2024-09-17T18:36:25.433657+0000","last_undegraded":"2024-09-17T18:36:25.433657+0000","last_fullsized":"2024-09-17T18:36:25.433657+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:05:38.754621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032532500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1297,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.735460+0000","last_change":"2024-09-17T18:20:02.260633+0000","last_active":"2024-09-17T18:36:26.735460+0000","last_peered":"2024-09-17T18:36:26.735460+0000","last_clean":"2024-09-17T18:36:26.735460+0000","last_became_active":"2024-09-17T18:20:02.260389+0000","last_became_peered":"2024-09-17T18:20:02.260389+0000","last_unstale":"2024-09-17T18:36:26.735460+0000","last_undegraded":"2024-09-17T18:36:26.735460+0000","last_fullsized":"2024-09-17T18:36:26.735460+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:20:12.888792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1312,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.735449+0000","last_change":"2024-09-17T18:19:01.527848+0000","last_active":"2024-09-17T18:36:26.735449+0000","last_peered":"2024-09-17T18:36:26.735449+0000","last_clean":"2024-09-17T18:36:26.735449+0000","last_became_active":"2024-09-17T18:18:59.529390+0000","last_became_peered":"2024-09-17T18:18:59.529390+0000","last_unstale":"2024-09-17T18:36:26.735449+0000","last_undegraded":"2024-09-17T18:36:26.735449+0000","last_fullsized":"2024-09-17T18:36:26.735449+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:40:45.376247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065986199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1299,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.717264+0000","last_change":"2024-09-17T18:20:02.260849+0000","last_active":"2024-09-17T18:36:25.717264+0000","last_peered":"2024-09-17T18:36:25.717264+0000","last_clean":"2024-09-17T18:36:25.717264+0000","last_became_active":"2024-09-17T18:20:02.251405+0000","last_became_peered":"2024-09-17T18:20:02.251405+0000","last_unstale":"2024-09-17T18:36:25.717264+0000","last_undegraded":"2024-09-17T18:36:25.717264+0000","last_fullsized":"2024-09-17T18:36:25.717264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:03:03.725725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":1297,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.583147+0000","last_change":"2024-09-17T18:19:01.532623+0000","last_active":"2024-09-17T18:36:25.583147+0000","last_peered":"2024-09-17T18:36:25.583147+0000","last_clean":"2024-09-17T18:36:25.583147+0000","last_became_active":"2024-09-17T18:18:59.526791+0000","last_became_peered":"2024-09-17T18:18:59.526791+0000","last_unstale":"2024-09-17T18:36:25.583147+0000","last_undegraded":"2024-09-17T18:36:25.583147+0000","last_fullsized":"2024-09-17T18:36:25.583147+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T21:46:24.217486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010088289999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1299,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.717202+0000","last_change":"2024-09-17T18:20:02.274215+0000","last_active":"2024-09-17T18:36:25.717202+0000","last_peered":"2024-09-17T18:36:25.717202+0000","last_clean":"2024-09-17T18:36:25.717202+0000","last_became_active":"2024-09-17T18:20:02.268644+0000","last_became_peered":"2024-09-17T18:20:02.268644+0000","last_unstale":"2024-09-17T18:36:25.717202+0000","last_undegraded":"2024-09-17T18:36:25.717202+0000","last_fullsized":"2024-09-17T18:36:25.717202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:12:55.005158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1298,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:27.012101+0000","last_change":"2024-09-17T18:19:01.527784+0000","last_active":"2024-09-17T18:36:27.012101+0000","last_peered":"2024-09-17T18:36:27.012101+0000","last_clean":"2024-09-17T18:36:27.012101+0000","last_became_active":"2024-09-17T18:19:00.244613+0000","last_became_peered":"2024-09-17T18:19:00.244613+0000","last_unstale":"2024-09-17T18:36:27.012101+0000","last_undegraded":"2024-09-17T18:36:27.012101+0000","last_fullsized":"2024-09-17T18:36:27.012101+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:31:24.599660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063646500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1280,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.583127+0000","last_change":"2024-09-17T18:20:02.254779+0000","last_active":"2024-09-17T18:36:25.583127+0000","last_peered":"2024-09-17T18:36:25.583127+0000","last_clean":"2024-09-17T18:36:25.583127+0000","last_became_active":"2024-09-17T18:20:02.254259+0000","last_became_peered":"2024-09-17T18:20:02.254259+0000","last_unstale":"2024-09-17T18:36:25.583127+0000","last_undegraded":"2024-09-17T18:36:25.583127+0000","last_fullsized":"2024-09-17T18:36:25.583127+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:02:19.820701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1311,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.433670+0000","last_change":"2024-09-17T18:19:01.527563+0000","last_active":"2024-09-17T18:36:25.433670+0000","last_peered":"2024-09-17T18:36:25.433670+0000","last_clean":"2024-09-17T18:36:25.433670+0000","last_became_active":"2024-09-17T18:19:00.244851+0000","last_became_peered":"2024-09-17T18:19:00.244851+0000","last_unstale":"2024-09-17T18:36:25.433670+0000","last_undegraded":"2024-09-17T18:36:25.433670+0000","last_fullsized":"2024-09-17T18:36:25.433670+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:19:57.036107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041282100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"19'7","reported_seq":1309,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.433672+0000","last_change":"2024-09-17T18:20:02.253575+0000","last_active":"2024-09-17T18:36:25.433672+0000","last_peered":"2024-09-17T18:36:25.433672+0000","last_clean":"2024-09-17T18:36:25.433672+0000","last_became_active":"2024-09-17T18:20:02.253328+0000","last_became_peered":"2024-09-17T18:20:02.253328+0000","last_unstale":"2024-09-17T18:36:25.433672+0000","last_undegraded":"2024-09-17T18:36:25.433672+0000","last_fullsized":"2024-09-17T18:36:25.433672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:35:00.211682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1298,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:27.012077+0000","last_change":"2024-09-17T18:19:01.527858+0000","last_active":"2024-09-17T18:36:27.012077+0000","last_peered":"2024-09-17T18:36:27.012077+0000","last_clean":"2024-09-17T18:36:27.012077+0000","last_became_active":"2024-09-17T18:18:59.525295+0000","last_became_peered":"2024-09-17T18:18:59.525295+0000","last_unstale":"2024-09-17T18:36:27.012077+0000","last_undegraded":"2024-09-17T18:36:27.012077+0000","last_fullsized":"2024-09-17T18:36:27.012077+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:41:21.181263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062121300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1297,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.735549+0000","last_change":"2024-09-17T18:20:02.234786+0000","last_active":"2024-09-17T18:36:26.735549+0000","last_peered":"2024-09-17T18:36:26.735549+0000","last_clean":"2024-09-17T18:36:26.735549+0000","last_became_active":"2024-09-17T18:20:02.234414+0000","last_became_peered":"2024-09-17T18:20:02.234414+0000","last_unstale":"2024-09-17T18:36:26.735549+0000","last_undegraded":"2024-09-17T18:36:26.735549+0000","last_fullsized":"2024-09-17T18:36:26.735549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:07:37.564071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1311,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.433654+0000","last_change":"2024-09-17T18:19:01.527658+0000","last_active":"2024-09-17T18:36:25.433654+0000","last_peered":"2024-09-17T18:36:25.433654+0000","last_clean":"2024-09-17T18:36:25.433654+0000","last_became_active":"2024-09-17T18:19:00.056752+0000","last_became_peered":"2024-09-17T18:19:00.056752+0000","last_unstale":"2024-09-17T18:36:25.433654+0000","last_undegraded":"2024-09-17T18:36:25.433654+0000","last_fullsized":"2024-09-17T18:36:25.433654+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:57:47.271964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049856399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1282,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:27.011781+0000","last_change":"2024-09-17T18:20:02.227335+0000","last_active":"2024-09-17T18:36:27.011781+0000","last_peered":"2024-09-17T18:36:27.011781+0000","last_clean":"2024-09-17T18:36:27.011781+0000","last_became_active":"2024-09-17T18:20:02.226503+0000","last_became_peered":"2024-09-17T18:20:02.226503+0000","last_unstale":"2024-09-17T18:36:27.011781+0000","last_undegraded":"2024-09-17T18:36:27.011781+0000","last_fullsized":"2024-09-17T18:36:27.011781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T03:43:28.520222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1282,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:27.011779+0000","last_change":"2024-09-17T18:20:02.313761+0000","last_active":"2024-09-17T18:36:27.011779+0000","last_peered":"2024-09-17T18:36:27.011779+0000","last_clean":"2024-09-17T18:36:27.011779+0000","last_became_active":"2024-09-17T18:20:02.311023+0000","last_became_peered":"2024-09-17T18:20:02.311023+0000","last_unstale":"2024-09-17T18:36:27.011779+0000","last_undegraded":"2024-09-17T18:36:27.011779+0000","last_fullsized":"2024-09-17T18:36:27.011779+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:18:02.124093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1275,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.898650+0000","last_change":"2024-09-17T18:20:02.351223+0000","last_active":"2024-09-17T18:36:26.898650+0000","last_peered":"2024-09-17T18:36:26.898650+0000","last_clean":"2024-09-17T18:36:26.898650+0000","last_became_active":"2024-09-17T18:20:02.339489+0000","last_became_peered":"2024-09-17T18:20:02.339489+0000","last_unstale":"2024-09-17T18:36:26.898650+0000","last_undegraded":"2024-09-17T18:36:26.898650+0000","last_fullsized":"2024-09-17T18:36:26.898650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:23:22.992560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1295,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.433542+0000","last_change":"2024-09-17T18:20:02.287570+0000","last_active":"2024-09-17T18:36:25.433542+0000","last_peered":"2024-09-17T18:36:25.433542+0000","last_clean":"2024-09-17T18:36:25.433542+0000","last_became_active":"2024-09-17T18:20:02.286560+0000","last_became_peered":"2024-09-17T18:20:02.286560+0000","last_unstale":"2024-09-17T18:36:25.433542+0000","last_undegraded":"2024-09-17T18:36:25.433542+0000","last_fullsized":"2024-09-17T18:36:25.433542+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:54:48.894903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1299,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.717249+0000","last_change":"2024-09-17T18:20:02.280897+0000","last_active":"2024-09-17T18:36:25.717249+0000","last_peered":"2024-09-17T18:36:25.717249+0000","last_clean":"2024-09-17T18:36:25.717249+0000","last_became_active":"2024-09-17T18:20:02.270521+0000","last_became_peered":"2024-09-17T18:20:02.270521+0000","last_unstale":"2024-09-17T18:36:25.717249+0000","last_undegraded":"2024-09-17T18:36:25.717249+0000","last_fullsized":"2024-09-17T18:36:25.717249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:40:16.489769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1296,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.433793+0000","last_change":"2024-09-17T18:20:02.294208+0000","last_active":"2024-09-17T18:36:25.433793+0000","last_peered":"2024-09-17T18:36:25.433793+0000","last_clean":"2024-09-17T18:36:25.433793+0000","last_became_active":"2024-09-17T18:20:02.293587+0000","last_became_peered":"2024-09-17T18:20:02.293587+0000","last_unstale":"2024-09-17T18:36:25.433793+0000","last_undegraded":"2024-09-17T18:36:25.433793+0000","last_fullsized":"2024-09-17T18:36:25.433793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:06:11.621162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1274,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.898883+0000","last_change":"2024-09-17T18:20:02.275664+0000","last_active":"2024-09-17T18:36:26.898883+0000","last_peered":"2024-09-17T18:36:26.898883+0000","last_clean":"2024-09-17T18:36:26.898883+0000","last_became_active":"2024-09-17T18:20:02.259616+0000","last_became_peered":"2024-09-17T18:20:02.259616+0000","last_unstale":"2024-09-17T18:36:26.898883+0000","last_undegraded":"2024-09-17T18:36:26.898883+0000","last_fullsized":"2024-09-17T18:36:26.898883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T21:08:28.298068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1281,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:27.012129+0000","last_change":"2024-09-17T18:20:02.314014+0000","last_active":"2024-09-17T18:36:27.012129+0000","last_peered":"2024-09-17T18:36:27.012129+0000","last_clean":"2024-09-17T18:36:27.012129+0000","last_became_active":"2024-09-17T18:20:02.313525+0000","last_became_peered":"2024-09-17T18:20:02.313525+0000","last_unstale":"2024-09-17T18:36:27.012129+0000","last_undegraded":"2024-09-17T18:36:27.012129+0000","last_fullsized":"2024-09-17T18:36:27.012129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:02:30.324064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1274,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.898897+0000","last_change":"2024-09-17T18:20:02.306541+0000","last_active":"2024-09-17T18:36:26.898897+0000","last_peered":"2024-09-17T18:36:26.898897+0000","last_clean":"2024-09-17T18:36:26.898897+0000","last_became_active":"2024-09-17T18:20:02.292188+0000","last_became_peered":"2024-09-17T18:20:02.292188+0000","last_unstale":"2024-09-17T18:36:26.898897+0000","last_undegraded":"2024-09-17T18:36:26.898897+0000","last_fullsized":"2024-09-17T18:36:26.898897+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:10:44.904234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1278,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.583267+0000","last_change":"2024-09-17T18:20:02.256567+0000","last_active":"2024-09-17T18:36:25.583267+0000","last_peered":"2024-09-17T18:36:25.583267+0000","last_clean":"2024-09-17T18:36:25.583267+0000","last_became_active":"2024-09-17T18:20:02.254813+0000","last_became_peered":"2024-09-17T18:20:02.254813+0000","last_unstale":"2024-09-17T18:36:25.583267+0000","last_undegraded":"2024-09-17T18:36:25.583267+0000","last_fullsized":"2024-09-17T18:36:25.583267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:20:59.122518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1275,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.898721+0000","last_change":"2024-09-17T18:20:02.327177+0000","last_active":"2024-09-17T18:36:26.898721+0000","last_peered":"2024-09-17T18:36:26.898721+0000","last_clean":"2024-09-17T18:36:26.898721+0000","last_became_active":"2024-09-17T18:20:02.299834+0000","last_became_peered":"2024-09-17T18:20:02.299834+0000","last_unstale":"2024-09-17T18:36:26.898721+0000","last_undegraded":"2024-09-17T18:36:26.898721+0000","last_fullsized":"2024-09-17T18:36:26.898721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:14:12.763217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1282,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:27.011862+0000","last_change":"2024-09-17T18:20:02.230651+0000","last_active":"2024-09-17T18:36:27.011862+0000","last_peered":"2024-09-17T18:36:27.011862+0000","last_clean":"2024-09-17T18:36:27.011862+0000","last_became_active":"2024-09-17T18:20:02.228502+0000","last_became_peered":"2024-09-17T18:20:02.228502+0000","last_unstale":"2024-09-17T18:36:27.011862+0000","last_undegraded":"2024-09-17T18:36:27.011862+0000","last_fullsized":"2024-09-17T18:36:27.011862+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:36:19.108865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59128,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607778,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69680,"kb_used_data":6552,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94302160,"statfs":{"total":96636764160,"available":96565411840,"internally_reserved":0,"allocated":6709248,"data_stored":2878698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607777,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78008,"kb_used_data":6556,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94293832,"statfs":{"total":96636764160,"available":96556883968,"internally_reserved":0,"allocated":6713344,"data_stored":2878717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640479,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82168,"kb_used_data":6556,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289672,"statfs":{"total":96636764160,"available":96552624128,"internally_reserved":0,"allocated":6713344,"data_stored":2878717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640481,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82224,"kb_used_data":6616,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289616,"statfs":{"total":96636764160,"available":96552566784,"internally_reserved":0,"allocated":6774784,"data_stored":3407631,"data_compressed":29564,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640480,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73836,"kb_used_data":6552,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94298004,"statfs":{"total":96636764160,"available":96561156096,"internally_reserved":0,"allocated":6709248,"data_stored":2878698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640482,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73908,"kb_used_data":6616,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94297932,"statfs":{"total":96636764160,"available":96561082368,"internally_reserved":0,"allocated":6774784,"data_stored":3407631,"data_compressed":29564,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29564,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29564,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T18:37:26.252 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-17T18:37:26.253 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T18:37:26.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.347+0000 7f088e67f640 10 monclient: get_monmap_and_config 2024-09-17T18:37:26.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.347+0000 7f088e67f640 10 monclient: build_initial_monmap 2024-09-17T18:37:26.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.347+0000 7f088e67f640 10 monclient: monmap: 2024-09-17T18:37:26.349 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:26.349 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:26.349 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:37:26.348243+0000 2024-09-17T18:37:26.349 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:37:26.348243+0000 2024-09-17T18:37:26.349 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:26.350 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:26.350 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:37:26.350 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:37:26.350 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:37:26.350 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:37:26.350 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:37:26.350 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:26.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.348+0000 7f088e67f640 1 Processor -- start 2024-09-17T18:37:26.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.348+0000 7f088e67f640 1 -- start start 2024-09-17T18:37:26.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.348+0000 7f088e67f640 10 monclient: init 2024-09-17T18:37:26.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.348+0000 7f088e67f640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:26.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.348+0000 7f088e67f640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-17T18:37:26.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.348+0000 7f088e67f640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f08880a33f0 0x7f088809e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:26.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.348+0000 7f088e67f640 10 monclient(hunting): picked mon.noname-c con 0x7f08880a33f0 addr v2:172.21.15.173:3300/0 2024-09-17T18:37:26.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.348+0000 7f088e67f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f088809da70 addr v1:172.21.15.173:6789/0 2024-09-17T18:37:26.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.348+0000 7f088e67f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f088813a850 0x7f088813cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:26.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.348+0000 7f088e67f640 10 monclient(hunting): picked mon.noname-b con 0x7f088813a850 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:26.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.348+0000 7f088e67f640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0888074b40 con 0x7f088809da70 2024-09-17T18:37:26.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.348+0000 7f088e67f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:26.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.348+0000 7f088e67f640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f0888074550 con 0x7f08880a33f0 2024-09-17T18:37:26.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.348+0000 7f088e67f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:26.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.348+0000 7f088e67f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f08880a37d0 con 0x7f088813a850 2024-09-17T18:37:26.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.348+0000 7f088e67f640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:26.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.348+0000 7f088e67f640 10 monclient(hunting): authenticate will time out at 1850.064961s 2024-09-17T18:37:26.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.349+0000 7f088cc1c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f088813a850 0x7f088813cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:26.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.349+0000 7f088cc1c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f088813a850 0x7f088813cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:36798/0 (socket says 172.21.15.121:36798) 2024-09-17T18:37:26.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.349+0000 7f088cc1c640 1 -- 172.21.15.121:0/645773706 learned_addr learned my addr 172.21.15.121:0/645773706 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:37:26.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.349+0000 7f088cc1c640 10 monclient(hunting): get_auth_request con 0x7f088813a850 auth_method 0 2024-09-17T18:37:26.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.349+0000 7f088cc1c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:26.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.349+0000 7f088cc1c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:26.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.349+0000 7f088cc1c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:26.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.349+0000 7f088cc1c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:26.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.349+0000 7f088cc1c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:26.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.349+0000 7f088cc1c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:26.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.349+0000 7f0887fff640 1 --2- 172.21.15.121:0/645773706 >> v2:172.21.15.173:3300/0 conn(0x7f08880a33f0 0x7f088809e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:26.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.349+0000 7f0887fff640 10 monclient(hunting): get_auth_request con 0x7f08880a33f0 auth_method 0 2024-09-17T18:37:26.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.349+0000 7f0887fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:26.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.349+0000 7f0887fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:26.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.349+0000 7f0887fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:26.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.349+0000 7f088cc1c640 10 monclient(hunting): handle_auth_done global_id 64604 payload 274 2024-09-17T18:37:26.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.349+0000 7f088cc1c640 1 -- 172.21.15.121:0/645773706 >> v2:172.21.15.173:3300/0 conn(0x7f08880a33f0 msgr2=0x7f088809e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:26.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.349+0000 7f088cc1c640 1 --2- 172.21.15.121:0/645773706 >> v2:172.21.15.173:3300/0 conn(0x7f08880a33f0 0x7f088809e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:26.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.349+0000 7f088cc1c640 1 -- 172.21.15.121:0/645773706 >> v1:172.21.15.173:6789/0 conn(0x7f088809da70 legacy=0x7f088809ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:26.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.349+0000 7f088cc1c640 10 monclient: _finish_hunting 0 2024-09-17T18:37:26.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.349+0000 7f088cc1c640 1 monclient: found mon.noname-b 2024-09-17T18:37:26.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.349+0000 7f088cc1c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:26.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.349+0000 7f088cc1c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:37:26.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.349+0000 7f088cc1c640 1 -- 172.21.15.121:0/645773706 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08880a13c0 con 0x7f088813a850 2024-09-17T18:37:26.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.352+0000 7f088cc1c640 1 -- 172.21.15.121:0/645773706 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f088813a850 msgr2=0x7f088813cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:26.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.352+0000 7f088cc1c640 1 --2- 172.21.15.121:0/645773706 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f088813a850 0x7f088813cc40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0880009e50 tx=0x7f088002a560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b965d0ee315b3bce server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:26.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.352+0000 7f0886ffd640 1 -- 172.21.15.121:0/645773706 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0880036070 con 0x7f088813a850 2024-09-17T18:37:26.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.352+0000 7f0886ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:26.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.352+0000 7f0886ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:37:26.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.352+0000 7f0886ffd640 10 monclient: dump: 2024-09-17T18:37:26.361 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:26.361 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:26.361 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:26.361 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:26.361 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:26.361 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:26.361 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:26.361 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:26.361 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:26.361 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:26.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.352+0000 7f0886ffd640 10 monclient: _finish_auth 0 2024-09-17T18:37:26.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.352+0000 7f0886ffd640 10 monclient: _check_auth_tickets 2024-09-17T18:37:26.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.352+0000 7f0886ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:26.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.352+0000 7f088e67f640 5 monclient: authenticate success, global_id 64604 2024-09-17T18:37:26.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.352+0000 7f0886ffd640 1 -- 172.21.15.121:0/645773706 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0880007800 con 0x7f088813a850 2024-09-17T18:37:26.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.352+0000 7f088e67f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:37:26.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.352+0000 7f0886ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:26.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.352+0000 7f088e67f640 10 monclient: get_monmap_and_config success 2024-09-17T18:37:26.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.352+0000 7f088e67f640 4 set_mon_vals no callback set 2024-09-17T18:37:26.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.352+0000 7f0886ffd640 1 -- 172.21.15.121:0/645773706 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0880007bd0 con 0x7f088813a850 2024-09-17T18:37:26.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.352+0000 7f0886ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:26.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.352+0000 7f0886ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:37:26.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.352+0000 7f0886ffd640 10 monclient: dump: 2024-09-17T18:37:26.362 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:26.362 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:26.362 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:26.362 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:26.363 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:26.363 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:26.363 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:26.363 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:26.363 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:26.363 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:26.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.352+0000 7f088e67f640 10 monclient: shutdown 2024-09-17T18:37:26.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.352+0000 7f088e67f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:26.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.352+0000 7f088e67f640 1 -- 172.21.15.121:0/645773706 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f088813a850 msgr2=0x7f088813cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:26.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.352+0000 7f088e67f640 1 --2- 172.21.15.121:0/645773706 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f088813a850 0x7f088813cc40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0880009e50 tx=0x7f088002a560 comp rx=0 tx=0).stop 2024-09-17T18:37:26.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.352+0000 7f088e67f640 1 -- 172.21.15.121:0/645773706 shutdown_connections 2024-09-17T18:37:26.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.352+0000 7f088e67f640 1 --2- 172.21.15.121:0/645773706 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f088813a850 0x7f088813cc40 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:26.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.352+0000 7f088e67f640 1 --2- 172.21.15.121:0/645773706 >> v2:172.21.15.173:3300/0 conn(0x7f08880a33f0 0x7f088809e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:26.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.352+0000 7f088e67f640 1 -- 172.21.15.121:0/645773706 >> 172.21.15.121:0/645773706 conn(0x7f08880a3b50 msgr2=0x7f08880a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:26.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.352+0000 7f088e67f640 1 -- 172.21.15.121:0/645773706 shutdown_connections 2024-09-17T18:37:26.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.352+0000 7f088e67f640 1 -- 172.21.15.121:0/645773706 wait complete. 2024-09-17T18:37:26.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.353+0000 7f088e67f640 10 monclient: build_initial_monmap 2024-09-17T18:37:26.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.353+0000 7f088e67f640 10 monclient: monmap: 2024-09-17T18:37:26.364 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:26.364 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:37:26.364 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:37:26.364 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:37:26.364 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:26.364 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:26.364 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:37:26.364 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:37:26.364 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:37:26.364 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:26.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.353+0000 7f088e67f640 1 librados: starting msgr at 2024-09-17T18:37:26.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.353+0000 7f088e67f640 1 librados: starting objecter 2024-09-17T18:37:26.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.353+0000 7f088e67f640 1 Processor -- start 2024-09-17T18:37:26.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.353+0000 7f088e67f640 1 -- start start 2024-09-17T18:37:26.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.353+0000 7f088e67f640 1 librados: setting wanted keys 2024-09-17T18:37:26.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.353+0000 7f088e67f640 1 librados: calling monclient init 2024-09-17T18:37:26.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.353+0000 7f088e67f640 10 monclient: init 2024-09-17T18:37:26.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f088e67f640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:26.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f088e67f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T18:37:26.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f088e67f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08880a33f0 0x7f08880a22e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:26.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f088e67f640 10 monclient(hunting): picked mon.noname-c con 0x7f08880a33f0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:26.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f088e67f640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f088813a850 0x7f0888136be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:26.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f088e67f640 10 monclient(hunting): picked mon.noname-a con 0x7f088813a850 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:37:26.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f088e67f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0888120d80 0x7f088811f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:26.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f088e67f640 10 monclient(hunting): picked mon.noname-b con 0x7f0888120d80 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:37:26.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f088e67f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:26.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f088e67f640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f088809f320 con 0x7f088813a850 2024-09-17T18:37:26.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f088e67f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:26.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f088e67f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f088809f1a0 con 0x7f0888120d80 2024-09-17T18:37:26.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f088e67f640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:26.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f088e67f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f088813d180 con 0x7f08880a33f0 2024-09-17T18:37:26.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f088e67f640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:26.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f088e67f640 10 monclient(hunting): authenticate will time out at 1850.070443s 2024-09-17T18:37:26.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f0887fff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08880a33f0 0x7f08880a22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:26.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f0887fff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08880a33f0 0x7f08880a22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:36812/0 (socket says 172.21.15.121:36812) 2024-09-17T18:37:26.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f08877fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f088813a850 0x7f0888136be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:26.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f0887fff640 1 -- 172.21.15.121:0/782417362 learned_addr learned my addr 172.21.15.121:0/782417362 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:37:26.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f0887fff640 10 monclient(hunting): get_auth_request con 0x7f08880a33f0 auth_method 0 2024-09-17T18:37:26.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f0887fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:26.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f0887fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:26.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f0887fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:26.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f08877fe640 10 monclient(hunting): get_auth_request con 0x7f088813a850 auth_method 0 2024-09-17T18:37:26.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f08877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:26.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f08877fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:26.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f08877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:26.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f0887fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:26.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f0887fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:26.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f0887fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:26.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f08877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:26.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f08877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:26.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f088cc1c640 1 --2- 172.21.15.121:0/782417362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0888120d80 0x7f088811f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:26.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f08877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:26.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f088cc1c640 10 monclient(hunting): get_auth_request con 0x7f0888120d80 auth_method 0 2024-09-17T18:37:26.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f088cc1c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:26.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f088cc1c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:26.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f088cc1c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:26.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f088cc1c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:26.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f088cc1c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:26.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f088cc1c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:26.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f0887fff640 10 monclient(hunting): handle_auth_done global_id 64607 payload 824 2024-09-17T18:37:26.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f0887fff640 1 -- 172.21.15.121:0/782417362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0888120d80 msgr2=0x7f088811f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:26.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f0887fff640 1 --2- 172.21.15.121:0/782417362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0888120d80 0x7f088811f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:26.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f0887fff640 1 -- 172.21.15.121:0/782417362 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f088813a850 msgr2=0x7f0888136be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:26.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f0887fff640 1 --2- 172.21.15.121:0/782417362 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f088813a850 0x7f0888136be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:26.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f0887fff640 10 monclient: _finish_hunting 0 2024-09-17T18:37:26.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f0887fff640 1 monclient: found mon.noname-c 2024-09-17T18:37:26.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f0887fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:26.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f0887fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:37:26.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f0887fff640 1 -- 172.21.15.121:0/782417362 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08880a2d50 con 0x7f08880a33f0 2024-09-17T18:37:26.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.354+0000 7f08877fe640 1 --2- 172.21.15.121:0/782417362 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f088813a850 0x7f0888136be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:37:26.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.357+0000 7f0887fff640 1 -- 172.21.15.121:0/782417362 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08880a33f0 msgr2=0x7f08880a22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:26.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.357+0000 7f0887fff640 1 --2- 172.21.15.121:0/782417362 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08880a33f0 0x7f08880a22e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f087800d800 tx=0x7f087800dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:26.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.357+0000 7f0884ff9640 10 client.?.objecter ms_handle_connect 0x7f08880a33f0 2024-09-17T18:37:26.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.357+0000 7f0884ff9640 10 client.?.objecter resend_mon_ops 2024-09-17T18:37:26.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.357+0000 7f0884ff9640 1 -- 172.21.15.121:0/782417362 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087800b8d0 con 0x7f08880a33f0 2024-09-17T18:37:26.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.357+0000 7f0884ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:26.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.357+0000 7f0884ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:37:26.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.357+0000 7f0884ff9640 10 monclient: dump: 2024-09-17T18:37:26.369 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:26.369 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:26.369 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:26.369 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:26.369 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:26.369 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:26.369 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:26.370 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:26.370 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:26.370 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:26.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.357+0000 7f0884ff9640 10 monclient: _finish_auth 0 2024-09-17T18:37:26.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.357+0000 7f0884ff9640 10 monclient: _check_auth_tickets 2024-09-17T18:37:26.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.357+0000 7f0884ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:26.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.357+0000 7f088e67f640 5 monclient: authenticate success, global_id 64607 2024-09-17T18:37:26.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.357+0000 7f0884ff9640 1 -- 172.21.15.121:0/782417362 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0878019070 con 0x7f08880a33f0 2024-09-17T18:37:26.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.357+0000 7f0884ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:26.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.357+0000 7f0884ff9640 1 -- 172.21.15.121:0/782417362 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087800bc90 con 0x7f08880a33f0 2024-09-17T18:37:26.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.357+0000 7f088e67f640 10 monclient: _renew_subs 2024-09-17T18:37:26.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.357+0000 7f088e67f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:37:26.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.357+0000 7f088e67f640 1 -- 172.21.15.121:0/782417362 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08880a64b0 con 0x7f08880a33f0 2024-09-17T18:37:26.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.357+0000 7f08867fc640 4 set_mon_vals no callback set 2024-09-17T18:37:26.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.357+0000 7f0884ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:26.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.357+0000 7f0884ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:37:26.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.357+0000 7f0884ff9640 10 monclient: dump: 2024-09-17T18:37:26.371 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:26.371 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:26.371 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:26.371 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:26.371 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:26.371 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:26.371 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:26.371 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:26.371 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:26.371 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:26.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.357+0000 7f088e67f640 10 client.64607.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:37:26.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.357+0000 7f088e67f640 10 monclient: _renew_subs 2024-09-17T18:37:26.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.357+0000 7f088e67f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:37:26.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.357+0000 7f088e67f640 1 -- 172.21.15.121:0/782417362 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0888121ce0 con 0x7f08880a33f0 2024-09-17T18:37:26.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.357+0000 7f088e67f640 1 librados: init done 2024-09-17T18:37:26.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.358+0000 7f0884ff9640 1 -- 172.21.15.121:0/782417362 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f087801e460 con 0x7f08880a33f0 2024-09-17T18:37:26.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.358+0000 7f088e67f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:26.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.358+0000 7f088e67f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:26.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.358+0000 7f088e67f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:37:26.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.358+0000 7f088e67f640 1 -- 172.21.15.121:0/782417362 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f088809e550 con 0x7f08880a33f0 2024-09-17T18:37:26.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.358+0000 7f0884ff9640 1 --2- 172.21.15.121:0/782417362 >> v2:172.21.15.173:6812/2423786037 conn(0x7f0868040170 0x7f0868042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:26.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.358+0000 7f0884ff9640 1 -- 172.21.15.121:0/782417362 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(679..679 src has 247..679) ==== 4685+0+0 (secure 0 0 0) 0x7f0878053910 con 0x7f08880a33f0 2024-09-17T18:37:26.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.358+0000 7f0884ff9640 10 client.64607.objecter ms_dispatch 0x7f0888000b90 osd_map(679..679 src has 247..679) 2024-09-17T18:37:26.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.358+0000 7f0884ff9640 3 client.64607.objecter handle_osd_map got epochs [679,679] > 0 2024-09-17T18:37:26.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.358+0000 7f0884ff9640 3 client.64607.objecter handle_osd_map decoding full epoch 679 2024-09-17T18:37:26.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.358+0000 7f0884ff9640 20 client.64607.objecter dump_active .. 0 homeless 2024-09-17T18:37:26.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.358+0000 7f08877fe640 1 --2- 172.21.15.121:0/782417362 >> v2:172.21.15.173:6812/2423786037 conn(0x7f0868040170 0x7f0868042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:26.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.358+0000 7f08877fe640 10 monclient: get_auth_request con 0x7f0868040170 auth_method 0 2024-09-17T18:37:26.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.361+0000 7f08877fe640 1 --2- 172.21.15.121:0/782417362 >> v2:172.21.15.173:6812/2423786037 conn(0x7f0868040170 0x7f0868042630 secure :-1 s=READY pgs=4098 cs=0 l=1 rev1=1 crypto rx=0x7f08880a62d0 tx=0x7f0870009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:26.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.361+0000 7f0884ff9640 10 client.64607.objecter ms_handle_connect 0x7f0868040170 2024-09-17T18:37:26.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.363+0000 7f0884ff9640 1 -- 172.21.15.121:0/782417362 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0878052370 con 0x7f08880a33f0 2024-09-17T18:37:26.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.363+0000 7f0884ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:26.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.363+0000 7f0884ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:37:26.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.551+0000 7f088e67f640 1 -- 172.21.15.121:0/782417362 --> v2:172.21.15.173:6812/2423786037 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0888122150 con 0x7f0868040170 2024-09-17T18:37:26.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.553+0000 7f0884ff9640 1 -- 172.21.15.121:0/782417362 <== mgr.4105 v2:172.21.15.173:6812/2423786037 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114139 (secure 0 0 0) 0x7f0870009e10 con 0x7f0868040170 2024-09-17T18:37:26.556 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:37:26.556 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-09-17T18:37:26.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.555+0000 7f088e67f640 10 librados: watch_flush enter 2024-09-17T18:37:26.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.555+0000 7f088e67f640 10 librados: watch_flush exit 2024-09-17T18:37:26.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.555+0000 7f088e67f640 20 client.64607.objecter shutdown clearing up homeless session... 2024-09-17T18:37:26.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.555+0000 7f088e67f640 10 client.64607.objecter successfully canceled tick 2024-09-17T18:37:26.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.555+0000 7f088e67f640 1 -- 172.21.15.121:0/782417362 >> v2:172.21.15.173:6812/2423786037 conn(0x7f0868040170 msgr2=0x7f0868042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:26.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.555+0000 7f088e67f640 1 --2- 172.21.15.121:0/782417362 >> v2:172.21.15.173:6812/2423786037 conn(0x7f0868040170 0x7f0868042630 secure :-1 s=READY pgs=4098 cs=0 l=1 rev1=1 crypto rx=0x7f08880a62d0 tx=0x7f0870009060 comp rx=0 tx=0).stop 2024-09-17T18:37:26.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.555+0000 7f088e67f640 10 monclient: shutdown 2024-09-17T18:37:26.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.555+0000 7f088e67f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:26.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.555+0000 7f088e67f640 1 -- 172.21.15.121:0/782417362 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08880a33f0 msgr2=0x7f08880a22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:26.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.555+0000 7f088e67f640 1 --2- 172.21.15.121:0/782417362 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08880a33f0 0x7f08880a22e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f087800d800 tx=0x7f087800dcc0 comp rx=0 tx=0).stop 2024-09-17T18:37:26.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.556+0000 7f088e67f640 1 -- 172.21.15.121:0/782417362 shutdown_connections 2024-09-17T18:37:26.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.556+0000 7f088e67f640 1 --2- 172.21.15.121:0/782417362 >> v2:172.21.15.173:6812/2423786037 conn(0x7f0868040170 0x7f0868042630 unknown :-1 s=CLOSED pgs=4098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:26.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.556+0000 7f088e67f640 1 --2- 172.21.15.121:0/782417362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0888120d80 0x7f088811f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:26.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.556+0000 7f088e67f640 1 --2- 172.21.15.121:0/782417362 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f088813a850 0x7f0888136be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:26.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.556+0000 7f088e67f640 1 --2- 172.21.15.121:0/782417362 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08880a33f0 0x7f08880a22e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:26.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.556+0000 7f088e67f640 1 -- 172.21.15.121:0/782417362 >> 172.21.15.121:0/782417362 conn(0x7f08880a3b50 msgr2=0x7f0888139a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:26.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.556+0000 7f088e67f640 1 -- 172.21.15.121:0/782417362 shutdown_connections 2024-09-17T18:37:26.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.556+0000 7f088e67f640 1 -- 172.21.15.121:0/782417362 wait complete. 2024-09-17T18:37:26.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.556+0000 7f088e67f640 1 librados: shutdown 2024-09-17T18:37:26.568 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":1227,"stamp":"2024-09-17T18:37:25.778399+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":459824,"kb_used_data":39448,"kb_used_omap":9,"kb_used_meta":420214,"kb_avail":565771216,"statfs":{"total":579820584960,"available":579349725184,"internally_reserved":0,"allocated":40394752,"data_stored":18330092,"data_compressed":59128,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":430299809},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003191"},"pg_stats":[{"pgid":"5.15","version":"0'0","reported_seq":1279,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.583221+0000","last_change":"2024-09-17T18:20:02.310686+0000","last_active":"2024-09-17T18:36:25.583221+0000","last_peered":"2024-09-17T18:36:25.583221+0000","last_clean":"2024-09-17T18:36:25.583221+0000","last_became_active":"2024-09-17T18:20:02.310239+0000","last_became_peered":"2024-09-17T18:20:02.310239+0000","last_unstale":"2024-09-17T18:36:25.583221+0000","last_undegraded":"2024-09-17T18:36:25.583221+0000","last_fullsized":"2024-09-17T18:36:25.583221+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:52:11.931160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1298,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.717134+0000","last_change":"2024-09-17T18:20:02.326174+0000","last_active":"2024-09-17T18:36:25.717134+0000","last_peered":"2024-09-17T18:36:25.717134+0000","last_clean":"2024-09-17T18:36:25.717134+0000","last_became_active":"2024-09-17T18:20:02.322561+0000","last_became_peered":"2024-09-17T18:20:02.322561+0000","last_unstale":"2024-09-17T18:36:25.717134+0000","last_undegraded":"2024-09-17T18:36:25.717134+0000","last_fullsized":"2024-09-17T18:36:25.717134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:29:09.470382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1296,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.735731+0000","last_change":"2024-09-17T18:20:02.250996+0000","last_active":"2024-09-17T18:36:26.735731+0000","last_peered":"2024-09-17T18:36:26.735731+0000","last_clean":"2024-09-17T18:36:26.735731+0000","last_became_active":"2024-09-17T18:20:02.250675+0000","last_became_peered":"2024-09-17T18:20:02.250675+0000","last_unstale":"2024-09-17T18:36:26.735731+0000","last_undegraded":"2024-09-17T18:36:26.735731+0000","last_fullsized":"2024-09-17T18:36:26.735731+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:17:16.240715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1295,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.433900+0000","last_change":"2024-09-17T18:20:02.342701+0000","last_active":"2024-09-17T18:36:25.433900+0000","last_peered":"2024-09-17T18:36:25.433900+0000","last_clean":"2024-09-17T18:36:25.433900+0000","last_became_active":"2024-09-17T18:20:02.342457+0000","last_became_peered":"2024-09-17T18:20:02.342457+0000","last_unstale":"2024-09-17T18:36:25.433900+0000","last_undegraded":"2024-09-17T18:36:25.433900+0000","last_fullsized":"2024-09-17T18:36:25.433900+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:57:23.288202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1280,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.583101+0000","last_change":"2024-09-17T18:20:02.251851+0000","last_active":"2024-09-17T18:36:25.583101+0000","last_peered":"2024-09-17T18:36:25.583101+0000","last_clean":"2024-09-17T18:36:25.583101+0000","last_became_active":"2024-09-17T18:20:02.249100+0000","last_became_peered":"2024-09-17T18:20:02.249100+0000","last_unstale":"2024-09-17T18:36:25.583101+0000","last_undegraded":"2024-09-17T18:36:25.583101+0000","last_fullsized":"2024-09-17T18:36:25.583101+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:00:45.271045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1297,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.735456+0000","last_change":"2024-09-17T18:20:02.173218+0000","last_active":"2024-09-17T18:36:26.735456+0000","last_peered":"2024-09-17T18:36:26.735456+0000","last_clean":"2024-09-17T18:36:26.735456+0000","last_became_active":"2024-09-17T18:20:02.172083+0000","last_became_peered":"2024-09-17T18:20:02.172083+0000","last_unstale":"2024-09-17T18:36:26.735456+0000","last_undegraded":"2024-09-17T18:36:26.735456+0000","last_fullsized":"2024-09-17T18:36:26.735456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T21:54:16.156188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1296,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.735377+0000","last_change":"2024-09-17T18:20:02.244905+0000","last_active":"2024-09-17T18:36:26.735377+0000","last_peered":"2024-09-17T18:36:26.735377+0000","last_clean":"2024-09-17T18:36:26.735377+0000","last_became_active":"2024-09-17T18:20:02.244673+0000","last_became_peered":"2024-09-17T18:20:02.244673+0000","last_unstale":"2024-09-17T18:36:26.735377+0000","last_undegraded":"2024-09-17T18:36:26.735377+0000","last_fullsized":"2024-09-17T18:36:26.735377+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T21:29:37.865312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1297,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.735570+0000","last_change":"2024-09-17T18:20:02.316100+0000","last_active":"2024-09-17T18:36:26.735570+0000","last_peered":"2024-09-17T18:36:26.735570+0000","last_clean":"2024-09-17T18:36:26.735570+0000","last_became_active":"2024-09-17T18:20:02.315070+0000","last_became_peered":"2024-09-17T18:20:02.315070+0000","last_unstale":"2024-09-17T18:36:26.735570+0000","last_undegraded":"2024-09-17T18:36:26.735570+0000","last_fullsized":"2024-09-17T18:36:26.735570+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:45:14.819013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1296,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.735378+0000","last_change":"2024-09-17T18:20:02.348899+0000","last_active":"2024-09-17T18:36:26.735378+0000","last_peered":"2024-09-17T18:36:26.735378+0000","last_clean":"2024-09-17T18:36:26.735378+0000","last_became_active":"2024-09-17T18:20:02.348022+0000","last_became_peered":"2024-09-17T18:20:02.348022+0000","last_unstale":"2024-09-17T18:36:26.735378+0000","last_undegraded":"2024-09-17T18:36:26.735378+0000","last_fullsized":"2024-09-17T18:36:26.735378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:02:14.991712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1282,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:27.012026+0000","last_change":"2024-09-17T18:20:02.331615+0000","last_active":"2024-09-17T18:36:27.012026+0000","last_peered":"2024-09-17T18:36:27.012026+0000","last_clean":"2024-09-17T18:36:27.012026+0000","last_became_active":"2024-09-17T18:20:02.328118+0000","last_became_peered":"2024-09-17T18:20:02.328118+0000","last_unstale":"2024-09-17T18:36:27.012026+0000","last_undegraded":"2024-09-17T18:36:27.012026+0000","last_fullsized":"2024-09-17T18:36:27.012026+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:39:54.243953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1279,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.583300+0000","last_change":"2024-09-17T18:20:02.248888+0000","last_active":"2024-09-17T18:36:25.583300+0000","last_peered":"2024-09-17T18:36:25.583300+0000","last_clean":"2024-09-17T18:36:25.583300+0000","last_became_active":"2024-09-17T18:20:02.242391+0000","last_became_peered":"2024-09-17T18:20:02.242391+0000","last_unstale":"2024-09-17T18:36:25.583300+0000","last_undegraded":"2024-09-17T18:36:25.583300+0000","last_fullsized":"2024-09-17T18:36:25.583300+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:02:05.721872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1294,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.433853+0000","last_change":"2024-09-17T18:20:02.249495+0000","last_active":"2024-09-17T18:36:25.433853+0000","last_peered":"2024-09-17T18:36:25.433853+0000","last_clean":"2024-09-17T18:36:25.433853+0000","last_became_active":"2024-09-17T18:20:02.248819+0000","last_became_peered":"2024-09-17T18:20:02.248819+0000","last_unstale":"2024-09-17T18:36:25.433853+0000","last_undegraded":"2024-09-17T18:36:25.433853+0000","last_fullsized":"2024-09-17T18:36:25.433853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T20:11:51.090905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1276,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.898945+0000","last_change":"2024-09-17T18:20:02.273998+0000","last_active":"2024-09-17T18:36:26.898945+0000","last_peered":"2024-09-17T18:36:26.898945+0000","last_clean":"2024-09-17T18:36:26.898945+0000","last_became_active":"2024-09-17T18:20:02.259327+0000","last_became_peered":"2024-09-17T18:20:02.259327+0000","last_unstale":"2024-09-17T18:36:26.898945+0000","last_undegraded":"2024-09-17T18:36:26.898945+0000","last_fullsized":"2024-09-17T18:36:26.898945+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:55:51.961001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":1397,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.433765+0000","last_change":"2024-09-17T18:19:00.245854+0000","last_active":"2024-09-17T18:36:25.433765+0000","last_peered":"2024-09-17T18:36:25.433765+0000","last_clean":"2024-09-17T18:36:25.433765+0000","last_became_active":"2024-09-17T18:19:00.244903+0000","last_became_peered":"2024-09-17T18:19:00.244903+0000","last_unstale":"2024-09-17T18:36:25.433765+0000","last_undegraded":"2024-09-17T18:36:25.433765+0000","last_fullsized":"2024-09-17T18:36:25.433765+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:55:38.336150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":1296,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.583163+0000","last_change":"2024-09-17T18:19:01.532053+0000","last_active":"2024-09-17T18:36:25.583163+0000","last_peered":"2024-09-17T18:36:25.583163+0000","last_clean":"2024-09-17T18:36:25.583163+0000","last_became_active":"2024-09-17T18:18:59.526797+0000","last_became_peered":"2024-09-17T18:18:59.526797+0000","last_unstale":"2024-09-17T18:36:25.583163+0000","last_undegraded":"2024-09-17T18:36:25.583163+0000","last_fullsized":"2024-09-17T18:36:25.583163+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:19:28.196155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000425759,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1278,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.583305+0000","last_change":"2024-09-17T18:20:02.311785+0000","last_active":"2024-09-17T18:36:25.583305+0000","last_peered":"2024-09-17T18:36:25.583305+0000","last_clean":"2024-09-17T18:36:25.583305+0000","last_became_active":"2024-09-17T18:20:02.310633+0000","last_became_peered":"2024-09-17T18:20:02.310633+0000","last_unstale":"2024-09-17T18:36:25.583305+0000","last_undegraded":"2024-09-17T18:36:25.583305+0000","last_fullsized":"2024-09-17T18:36:25.583305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:08:06.293083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1312,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.433657+0000","last_change":"2024-09-17T18:19:01.527425+0000","last_active":"2024-09-17T18:36:25.433657+0000","last_peered":"2024-09-17T18:36:25.433657+0000","last_clean":"2024-09-17T18:36:25.433657+0000","last_became_active":"2024-09-17T18:18:59.526855+0000","last_became_peered":"2024-09-17T18:18:59.526855+0000","last_unstale":"2024-09-17T18:36:25.433657+0000","last_undegraded":"2024-09-17T18:36:25.433657+0000","last_fullsized":"2024-09-17T18:36:25.433657+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:05:38.754621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032532500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1297,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.735460+0000","last_change":"2024-09-17T18:20:02.260633+0000","last_active":"2024-09-17T18:36:26.735460+0000","last_peered":"2024-09-17T18:36:26.735460+0000","last_clean":"2024-09-17T18:36:26.735460+0000","last_became_active":"2024-09-17T18:20:02.260389+0000","last_became_peered":"2024-09-17T18:20:02.260389+0000","last_unstale":"2024-09-17T18:36:26.735460+0000","last_undegraded":"2024-09-17T18:36:26.735460+0000","last_fullsized":"2024-09-17T18:36:26.735460+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:20:12.888792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1312,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.735449+0000","last_change":"2024-09-17T18:19:01.527848+0000","last_active":"2024-09-17T18:36:26.735449+0000","last_peered":"2024-09-17T18:36:26.735449+0000","last_clean":"2024-09-17T18:36:26.735449+0000","last_became_active":"2024-09-17T18:18:59.529390+0000","last_became_peered":"2024-09-17T18:18:59.529390+0000","last_unstale":"2024-09-17T18:36:26.735449+0000","last_undegraded":"2024-09-17T18:36:26.735449+0000","last_fullsized":"2024-09-17T18:36:26.735449+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:40:45.376247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065986199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1299,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.717264+0000","last_change":"2024-09-17T18:20:02.260849+0000","last_active":"2024-09-17T18:36:25.717264+0000","last_peered":"2024-09-17T18:36:25.717264+0000","last_clean":"2024-09-17T18:36:25.717264+0000","last_became_active":"2024-09-17T18:20:02.251405+0000","last_became_peered":"2024-09-17T18:20:02.251405+0000","last_unstale":"2024-09-17T18:36:25.717264+0000","last_undegraded":"2024-09-17T18:36:25.717264+0000","last_fullsized":"2024-09-17T18:36:25.717264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:03:03.725725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":1297,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.583147+0000","last_change":"2024-09-17T18:19:01.532623+0000","last_active":"2024-09-17T18:36:25.583147+0000","last_peered":"2024-09-17T18:36:25.583147+0000","last_clean":"2024-09-17T18:36:25.583147+0000","last_became_active":"2024-09-17T18:18:59.526791+0000","last_became_peered":"2024-09-17T18:18:59.526791+0000","last_unstale":"2024-09-17T18:36:25.583147+0000","last_undegraded":"2024-09-17T18:36:25.583147+0000","last_fullsized":"2024-09-17T18:36:25.583147+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T21:46:24.217486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010088289999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1299,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.717202+0000","last_change":"2024-09-17T18:20:02.274215+0000","last_active":"2024-09-17T18:36:25.717202+0000","last_peered":"2024-09-17T18:36:25.717202+0000","last_clean":"2024-09-17T18:36:25.717202+0000","last_became_active":"2024-09-17T18:20:02.268644+0000","last_became_peered":"2024-09-17T18:20:02.268644+0000","last_unstale":"2024-09-17T18:36:25.717202+0000","last_undegraded":"2024-09-17T18:36:25.717202+0000","last_fullsized":"2024-09-17T18:36:25.717202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:12:55.005158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1298,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:27.012101+0000","last_change":"2024-09-17T18:19:01.527784+0000","last_active":"2024-09-17T18:36:27.012101+0000","last_peered":"2024-09-17T18:36:27.012101+0000","last_clean":"2024-09-17T18:36:27.012101+0000","last_became_active":"2024-09-17T18:19:00.244613+0000","last_became_peered":"2024-09-17T18:19:00.244613+0000","last_unstale":"2024-09-17T18:36:27.012101+0000","last_undegraded":"2024-09-17T18:36:27.012101+0000","last_fullsized":"2024-09-17T18:36:27.012101+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:31:24.599660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063646500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1280,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.583127+0000","last_change":"2024-09-17T18:20:02.254779+0000","last_active":"2024-09-17T18:36:25.583127+0000","last_peered":"2024-09-17T18:36:25.583127+0000","last_clean":"2024-09-17T18:36:25.583127+0000","last_became_active":"2024-09-17T18:20:02.254259+0000","last_became_peered":"2024-09-17T18:20:02.254259+0000","last_unstale":"2024-09-17T18:36:25.583127+0000","last_undegraded":"2024-09-17T18:36:25.583127+0000","last_fullsized":"2024-09-17T18:36:25.583127+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:02:19.820701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1311,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.433670+0000","last_change":"2024-09-17T18:19:01.527563+0000","last_active":"2024-09-17T18:36:25.433670+0000","last_peered":"2024-09-17T18:36:25.433670+0000","last_clean":"2024-09-17T18:36:25.433670+0000","last_became_active":"2024-09-17T18:19:00.244851+0000","last_became_peered":"2024-09-17T18:19:00.244851+0000","last_unstale":"2024-09-17T18:36:25.433670+0000","last_undegraded":"2024-09-17T18:36:25.433670+0000","last_fullsized":"2024-09-17T18:36:25.433670+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:19:57.036107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041282100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"19'7","reported_seq":1309,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.433672+0000","last_change":"2024-09-17T18:20:02.253575+0000","last_active":"2024-09-17T18:36:25.433672+0000","last_peered":"2024-09-17T18:36:25.433672+0000","last_clean":"2024-09-17T18:36:25.433672+0000","last_became_active":"2024-09-17T18:20:02.253328+0000","last_became_peered":"2024-09-17T18:20:02.253328+0000","last_unstale":"2024-09-17T18:36:25.433672+0000","last_undegraded":"2024-09-17T18:36:25.433672+0000","last_fullsized":"2024-09-17T18:36:25.433672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:35:00.211682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1298,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:27.012077+0000","last_change":"2024-09-17T18:19:01.527858+0000","last_active":"2024-09-17T18:36:27.012077+0000","last_peered":"2024-09-17T18:36:27.012077+0000","last_clean":"2024-09-17T18:36:27.012077+0000","last_became_active":"2024-09-17T18:18:59.525295+0000","last_became_peered":"2024-09-17T18:18:59.525295+0000","last_unstale":"2024-09-17T18:36:27.012077+0000","last_undegraded":"2024-09-17T18:36:27.012077+0000","last_fullsized":"2024-09-17T18:36:27.012077+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:41:21.181263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062121300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1297,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.735549+0000","last_change":"2024-09-17T18:20:02.234786+0000","last_active":"2024-09-17T18:36:26.735549+0000","last_peered":"2024-09-17T18:36:26.735549+0000","last_clean":"2024-09-17T18:36:26.735549+0000","last_became_active":"2024-09-17T18:20:02.234414+0000","last_became_peered":"2024-09-17T18:20:02.234414+0000","last_unstale":"2024-09-17T18:36:26.735549+0000","last_undegraded":"2024-09-17T18:36:26.735549+0000","last_fullsized":"2024-09-17T18:36:26.735549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:07:37.564071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1311,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.433654+0000","last_change":"2024-09-17T18:19:01.527658+0000","last_active":"2024-09-17T18:36:25.433654+0000","last_peered":"2024-09-17T18:36:25.433654+0000","last_clean":"2024-09-17T18:36:25.433654+0000","last_became_active":"2024-09-17T18:19:00.056752+0000","last_became_peered":"2024-09-17T18:19:00.056752+0000","last_unstale":"2024-09-17T18:36:25.433654+0000","last_undegraded":"2024-09-17T18:36:25.433654+0000","last_fullsized":"2024-09-17T18:36:25.433654+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:57:47.271964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049856399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1282,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:27.011781+0000","last_change":"2024-09-17T18:20:02.227335+0000","last_active":"2024-09-17T18:36:27.011781+0000","last_peered":"2024-09-17T18:36:27.011781+0000","last_clean":"2024-09-17T18:36:27.011781+0000","last_became_active":"2024-09-17T18:20:02.226503+0000","last_became_peered":"2024-09-17T18:20:02.226503+0000","last_unstale":"2024-09-17T18:36:27.011781+0000","last_undegraded":"2024-09-17T18:36:27.011781+0000","last_fullsized":"2024-09-17T18:36:27.011781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T03:43:28.520222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1282,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:27.011779+0000","last_change":"2024-09-17T18:20:02.313761+0000","last_active":"2024-09-17T18:36:27.011779+0000","last_peered":"2024-09-17T18:36:27.011779+0000","last_clean":"2024-09-17T18:36:27.011779+0000","last_became_active":"2024-09-17T18:20:02.311023+0000","last_became_peered":"2024-09-17T18:20:02.311023+0000","last_unstale":"2024-09-17T18:36:27.011779+0000","last_undegraded":"2024-09-17T18:36:27.011779+0000","last_fullsized":"2024-09-17T18:36:27.011779+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:18:02.124093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1275,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.898650+0000","last_change":"2024-09-17T18:20:02.351223+0000","last_active":"2024-09-17T18:36:26.898650+0000","last_peered":"2024-09-17T18:36:26.898650+0000","last_clean":"2024-09-17T18:36:26.898650+0000","last_became_active":"2024-09-17T18:20:02.339489+0000","last_became_peered":"2024-09-17T18:20:02.339489+0000","last_unstale":"2024-09-17T18:36:26.898650+0000","last_undegraded":"2024-09-17T18:36:26.898650+0000","last_fullsized":"2024-09-17T18:36:26.898650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:23:22.992560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1295,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.433542+0000","last_change":"2024-09-17T18:20:02.287570+0000","last_active":"2024-09-17T18:36:25.433542+0000","last_peered":"2024-09-17T18:36:25.433542+0000","last_clean":"2024-09-17T18:36:25.433542+0000","last_became_active":"2024-09-17T18:20:02.286560+0000","last_became_peered":"2024-09-17T18:20:02.286560+0000","last_unstale":"2024-09-17T18:36:25.433542+0000","last_undegraded":"2024-09-17T18:36:25.433542+0000","last_fullsized":"2024-09-17T18:36:25.433542+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:54:48.894903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1299,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.717249+0000","last_change":"2024-09-17T18:20:02.280897+0000","last_active":"2024-09-17T18:36:25.717249+0000","last_peered":"2024-09-17T18:36:25.717249+0000","last_clean":"2024-09-17T18:36:25.717249+0000","last_became_active":"2024-09-17T18:20:02.270521+0000","last_became_peered":"2024-09-17T18:20:02.270521+0000","last_unstale":"2024-09-17T18:36:25.717249+0000","last_undegraded":"2024-09-17T18:36:25.717249+0000","last_fullsized":"2024-09-17T18:36:25.717249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:40:16.489769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1296,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.433793+0000","last_change":"2024-09-17T18:20:02.294208+0000","last_active":"2024-09-17T18:36:25.433793+0000","last_peered":"2024-09-17T18:36:25.433793+0000","last_clean":"2024-09-17T18:36:25.433793+0000","last_became_active":"2024-09-17T18:20:02.293587+0000","last_became_peered":"2024-09-17T18:20:02.293587+0000","last_unstale":"2024-09-17T18:36:25.433793+0000","last_undegraded":"2024-09-17T18:36:25.433793+0000","last_fullsized":"2024-09-17T18:36:25.433793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:06:11.621162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1274,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.898883+0000","last_change":"2024-09-17T18:20:02.275664+0000","last_active":"2024-09-17T18:36:26.898883+0000","last_peered":"2024-09-17T18:36:26.898883+0000","last_clean":"2024-09-17T18:36:26.898883+0000","last_became_active":"2024-09-17T18:20:02.259616+0000","last_became_peered":"2024-09-17T18:20:02.259616+0000","last_unstale":"2024-09-17T18:36:26.898883+0000","last_undegraded":"2024-09-17T18:36:26.898883+0000","last_fullsized":"2024-09-17T18:36:26.898883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T21:08:28.298068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1281,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:27.012129+0000","last_change":"2024-09-17T18:20:02.314014+0000","last_active":"2024-09-17T18:36:27.012129+0000","last_peered":"2024-09-17T18:36:27.012129+0000","last_clean":"2024-09-17T18:36:27.012129+0000","last_became_active":"2024-09-17T18:20:02.313525+0000","last_became_peered":"2024-09-17T18:20:02.313525+0000","last_unstale":"2024-09-17T18:36:27.012129+0000","last_undegraded":"2024-09-17T18:36:27.012129+0000","last_fullsized":"2024-09-17T18:36:27.012129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:02:30.324064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1274,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.898897+0000","last_change":"2024-09-17T18:20:02.306541+0000","last_active":"2024-09-17T18:36:26.898897+0000","last_peered":"2024-09-17T18:36:26.898897+0000","last_clean":"2024-09-17T18:36:26.898897+0000","last_became_active":"2024-09-17T18:20:02.292188+0000","last_became_peered":"2024-09-17T18:20:02.292188+0000","last_unstale":"2024-09-17T18:36:26.898897+0000","last_undegraded":"2024-09-17T18:36:26.898897+0000","last_fullsized":"2024-09-17T18:36:26.898897+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:10:44.904234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1278,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.583267+0000","last_change":"2024-09-17T18:20:02.256567+0000","last_active":"2024-09-17T18:36:25.583267+0000","last_peered":"2024-09-17T18:36:25.583267+0000","last_clean":"2024-09-17T18:36:25.583267+0000","last_became_active":"2024-09-17T18:20:02.254813+0000","last_became_peered":"2024-09-17T18:20:02.254813+0000","last_unstale":"2024-09-17T18:36:25.583267+0000","last_undegraded":"2024-09-17T18:36:25.583267+0000","last_fullsized":"2024-09-17T18:36:25.583267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:20:59.122518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1275,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.898721+0000","last_change":"2024-09-17T18:20:02.327177+0000","last_active":"2024-09-17T18:36:26.898721+0000","last_peered":"2024-09-17T18:36:26.898721+0000","last_clean":"2024-09-17T18:36:26.898721+0000","last_became_active":"2024-09-17T18:20:02.299834+0000","last_became_peered":"2024-09-17T18:20:02.299834+0000","last_unstale":"2024-09-17T18:36:26.898721+0000","last_undegraded":"2024-09-17T18:36:26.898721+0000","last_fullsized":"2024-09-17T18:36:26.898721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:14:12.763217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1282,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:27.011862+0000","last_change":"2024-09-17T18:20:02.230651+0000","last_active":"2024-09-17T18:36:27.011862+0000","last_peered":"2024-09-17T18:36:27.011862+0000","last_clean":"2024-09-17T18:36:27.011862+0000","last_became_active":"2024-09-17T18:20:02.228502+0000","last_became_peered":"2024-09-17T18:20:02.228502+0000","last_unstale":"2024-09-17T18:36:27.011862+0000","last_undegraded":"2024-09-17T18:36:27.011862+0000","last_fullsized":"2024-09-17T18:36:27.011862+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:36:19.108865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59128,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607778,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69680,"kb_used_data":6552,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94302160,"statfs":{"total":96636764160,"available":96565411840,"internally_reserved":0,"allocated":6709248,"data_stored":2878698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607777,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78008,"kb_used_data":6556,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94293832,"statfs":{"total":96636764160,"available":96556883968,"internally_reserved":0,"allocated":6713344,"data_stored":2878717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640479,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82168,"kb_used_data":6556,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289672,"statfs":{"total":96636764160,"available":96552624128,"internally_reserved":0,"allocated":6713344,"data_stored":2878717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640481,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82224,"kb_used_data":6616,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289616,"statfs":{"total":96636764160,"available":96552566784,"internally_reserved":0,"allocated":6774784,"data_stored":3407631,"data_compressed":29564,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640480,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73836,"kb_used_data":6552,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94298004,"statfs":{"total":96636764160,"available":96561156096,"internally_reserved":0,"allocated":6709248,"data_stored":2878698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640482,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73908,"kb_used_data":6616,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94297932,"statfs":{"total":96636764160,"available":96561082368,"internally_reserved":0,"allocated":6774784,"data_stored":3407631,"data_compressed":29564,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29564,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29564,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T18:37:26.569 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T18:37:26.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.664+0000 7fd2df7f4640 10 monclient: get_monmap_and_config 2024-09-17T18:37:26.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.664+0000 7fd2df7f4640 10 monclient: build_initial_monmap 2024-09-17T18:37:26.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.664+0000 7fd2df7f4640 10 monclient: monmap: 2024-09-17T18:37:26.666 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:26.666 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:26.666 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:37:26.665111+0000 2024-09-17T18:37:26.666 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:37:26.665111+0000 2024-09-17T18:37:26.666 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:26.666 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:26.666 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:37:26.667 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:37:26.667 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:37:26.667 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:37:26.667 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:37:26.667 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:26.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.665+0000 7fd2df7f4640 1 Processor -- start 2024-09-17T18:37:26.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.665+0000 7fd2df7f4640 1 -- start start 2024-09-17T18:37:26.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.665+0000 7fd2df7f4640 10 monclient: init 2024-09-17T18:37:26.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.665+0000 7fd2df7f4640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:26.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.665+0000 7fd2df7f4640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-17T18:37:26.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.665+0000 7fd2df7f4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd2d80a33f0 addr v1:172.21.15.121:6789/0 2024-09-17T18:37:26.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.665+0000 7fd2df7f4640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd2d809da70 0x7fd2d809ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:26.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.665+0000 7fd2df7f4640 10 monclient(hunting): picked mon.noname-a con 0x7fd2d809da70 addr v2:172.21.15.121:3300/0 2024-09-17T18:37:26.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.665+0000 7fd2df7f4640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2d813a850 0x7fd2d813cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:26.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.665+0000 7fd2df7f4640 10 monclient(hunting): picked mon.noname-b con 0x7fd2d813a850 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:26.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.665+0000 7fd2df7f4640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2d8074b40 con 0x7fd2d80a33f0 2024-09-17T18:37:26.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.665+0000 7fd2df7f4640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:26.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.665+0000 7fd2df7f4640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fd2d8074550 con 0x7fd2d809da70 2024-09-17T18:37:26.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.665+0000 7fd2df7f4640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:26.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.665+0000 7fd2df7f4640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd2d80a37d0 con 0x7fd2d813a850 2024-09-17T18:37:26.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.665+0000 7fd2df7f4640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:26.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.665+0000 7fd2df7f4640 10 monclient(hunting): authenticate will time out at 1850.381765s 2024-09-17T18:37:26.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.665+0000 7fd2dcd8f640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd2d809da70 0x7fd2d809ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:26.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.665+0000 7fd2ddd91640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2d813a850 0x7fd2d813cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:26.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.665+0000 7fd2dd590640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fd2d80a33f0 0x7fd2d809e550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:47362/0 (socket says 172.21.15.121:47362) 2024-09-17T18:37:26.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.665+0000 7fd2dd590640 1 -- 172.21.15.121:0/4027347270 learned_addr learned my addr 172.21.15.121:0/4027347270 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:37:26.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.665+0000 7fd2ddd91640 10 monclient(hunting): get_auth_request con 0x7fd2d813a850 auth_method 0 2024-09-17T18:37:26.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.665+0000 7fd2ddd91640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:26.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.665+0000 7fd2ddd91640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:26.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.665+0000 7fd2ddd91640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:26.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.665+0000 7fd2dcd8f640 10 monclient(hunting): get_auth_request con 0x7fd2d809da70 auth_method 0 2024-09-17T18:37:26.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.665+0000 7fd2dcd8f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:26.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.665+0000 7fd2dcd8f640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:26.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.665+0000 7fd2dcd8f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:26.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.666+0000 7fd2ddd91640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:26.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.666+0000 7fd2ddd91640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:26.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.666+0000 7fd2ddd91640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:26.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.666+0000 7fd2dcd8f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:26.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.666+0000 7fd2dcd8f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:26.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.666+0000 7fd2dcd8f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:26.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.666+0000 7fd2ddd91640 10 monclient(hunting): handle_auth_done global_id 64610 payload 274 2024-09-17T18:37:26.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.666+0000 7fd2ddd91640 1 -- 172.21.15.121:0/4027347270 >> v2:172.21.15.121:3300/0 conn(0x7fd2d809da70 msgr2=0x7fd2d809ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:26.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.666+0000 7fd2ddd91640 1 --2- 172.21.15.121:0/4027347270 >> v2:172.21.15.121:3300/0 conn(0x7fd2d809da70 0x7fd2d809ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:26.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.666+0000 7fd2ddd91640 1 -- 172.21.15.121:0/4027347270 >> v1:172.21.15.121:6789/0 conn(0x7fd2d80a33f0 legacy=0x7fd2d809e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:26.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.666+0000 7fd2ddd91640 10 monclient: _finish_hunting 0 2024-09-17T18:37:26.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.666+0000 7fd2ddd91640 1 monclient: found mon.noname-b 2024-09-17T18:37:26.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.666+0000 7fd2ddd91640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:26.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.666+0000 7fd2ddd91640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:37:26.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.666+0000 7fd2ddd91640 1 -- 172.21.15.121:0/4027347270 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2d80a13c0 con 0x7fd2d813a850 2024-09-17T18:37:26.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.668+0000 7fd2ddd91640 1 -- 172.21.15.121:0/4027347270 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2d813a850 msgr2=0x7fd2d813cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:26.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.668+0000 7fd2ddd91640 1 --2- 172.21.15.121:0/4027347270 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2d813a850 0x7fd2d813cc40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8009d90 tx=0x7fd2c802a490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1efd372b886dd829 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:26.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.669+0000 7fd2d7fff640 1 -- 172.21.15.121:0/4027347270 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c8007950 con 0x7fd2d813a850 2024-09-17T18:37:26.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.669+0000 7fd2d7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:26.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.669+0000 7fd2d7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:37:26.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.669+0000 7fd2d7fff640 10 monclient: dump: 2024-09-17T18:37:26.678 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:26.678 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:26.678 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:26.678 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:26.678 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:26.678 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:26.678 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:26.678 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:26.679 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:26.679 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:26.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.669+0000 7fd2d7fff640 10 monclient: _finish_auth 0 2024-09-17T18:37:26.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.669+0000 7fd2d7fff640 10 monclient: _check_auth_tickets 2024-09-17T18:37:26.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.669+0000 7fd2d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:26.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.669+0000 7fd2df7f4640 5 monclient: authenticate success, global_id 64610 2024-09-17T18:37:26.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.669+0000 7fd2d7fff640 1 -- 172.21.15.121:0/4027347270 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2c8007af0 con 0x7fd2d813a850 2024-09-17T18:37:26.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.669+0000 7fd2df7f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:37:26.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.669+0000 7fd2d7fff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:26.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.669+0000 7fd2df7f4640 10 monclient: get_monmap_and_config success 2024-09-17T18:37:26.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.669+0000 7fd2df7f4640 4 set_mon_vals no callback set 2024-09-17T18:37:26.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.669+0000 7fd2d7fff640 1 -- 172.21.15.121:0/4027347270 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c8031400 con 0x7fd2d813a850 2024-09-17T18:37:26.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.669+0000 7fd2d7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:26.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.669+0000 7fd2d7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:37:26.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.669+0000 7fd2d7fff640 10 monclient: dump: 2024-09-17T18:37:26.680 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:26.680 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:26.680 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:26.680 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:26.680 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:26.680 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:26.680 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:26.680 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:26.680 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:26.680 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:26.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.669+0000 7fd2df7f4640 10 monclient: shutdown 2024-09-17T18:37:26.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.669+0000 7fd2df7f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:26.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.669+0000 7fd2df7f4640 1 -- 172.21.15.121:0/4027347270 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2d813a850 msgr2=0x7fd2d813cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:26.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.669+0000 7fd2df7f4640 1 --2- 172.21.15.121:0/4027347270 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2d813a850 0x7fd2d813cc40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8009d90 tx=0x7fd2c802a490 comp rx=0 tx=0).stop 2024-09-17T18:37:26.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.669+0000 7fd2df7f4640 1 -- 172.21.15.121:0/4027347270 shutdown_connections 2024-09-17T18:37:26.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.669+0000 7fd2df7f4640 1 --2- 172.21.15.121:0/4027347270 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2d813a850 0x7fd2d813cc40 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:26.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.669+0000 7fd2df7f4640 1 --2- 172.21.15.121:0/4027347270 >> v2:172.21.15.121:3300/0 conn(0x7fd2d809da70 0x7fd2d809ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:26.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.669+0000 7fd2df7f4640 1 -- 172.21.15.121:0/4027347270 >> 172.21.15.121:0/4027347270 conn(0x7fd2d80a3b50 msgr2=0x7fd2d80a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:26.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.669+0000 7fd2df7f4640 1 -- 172.21.15.121:0/4027347270 shutdown_connections 2024-09-17T18:37:26.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.669+0000 7fd2df7f4640 1 -- 172.21.15.121:0/4027347270 wait complete. 2024-09-17T18:37:26.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.670+0000 7fd2df7f4640 10 monclient: build_initial_monmap 2024-09-17T18:37:26.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.670+0000 7fd2df7f4640 10 monclient: monmap: 2024-09-17T18:37:26.681 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:26.681 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:37:26.681 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:37:26.681 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:37:26.681 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:26.681 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:26.681 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:37:26.682 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:37:26.682 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:37:26.682 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:26.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.670+0000 7fd2df7f4640 1 librados: starting msgr at 2024-09-17T18:37:26.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.670+0000 7fd2df7f4640 1 librados: starting objecter 2024-09-17T18:37:26.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.670+0000 7fd2df7f4640 1 Processor -- start 2024-09-17T18:37:26.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.670+0000 7fd2df7f4640 1 -- start start 2024-09-17T18:37:26.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.670+0000 7fd2df7f4640 1 librados: setting wanted keys 2024-09-17T18:37:26.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.670+0000 7fd2df7f4640 1 librados: calling monclient init 2024-09-17T18:37:26.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.670+0000 7fd2df7f4640 10 monclient: init 2024-09-17T18:37:26.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2df7f4640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:26.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2df7f4640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T18:37:26.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2df7f4640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd2d80a33f0 0x7fd2d80a22e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:26.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2df7f4640 10 monclient(hunting): picked mon.noname-b con 0x7fd2d80a33f0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:37:26.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2df7f4640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2d813a850 0x7fd2d8136c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:26.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2df7f4640 10 monclient(hunting): picked mon.noname-c con 0x7fd2d813a850 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:26.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2df7f4640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2d8137170 0x7fd2d81cb540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:26.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2df7f4640 10 monclient(hunting): picked mon.noname-a con 0x7fd2d8137170 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:37:26.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2df7f4640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:26.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2df7f4640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d809f320 con 0x7fd2d8137170 2024-09-17T18:37:26.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2df7f4640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:26.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2df7f4640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d809f1a0 con 0x7fd2d80a33f0 2024-09-17T18:37:26.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2df7f4640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:26.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2df7f4640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd2d813d180 con 0x7fd2d813a850 2024-09-17T18:37:26.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2df7f4640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:26.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2df7f4640 10 monclient(hunting): authenticate will time out at 1850.387104s 2024-09-17T18:37:26.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2ddd91640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2d8137170 0x7fd2d81cb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:26.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2dcd8f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2d813a850 0x7fd2d8136c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:26.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2ddd91640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2d8137170 0x7fd2d81cb540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51306/0 (socket says 172.21.15.121:51306) 2024-09-17T18:37:26.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2dcd8f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2d813a850 0x7fd2d8136c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:36832/0 (socket says 172.21.15.121:36832) 2024-09-17T18:37:26.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2ddd91640 1 -- 172.21.15.121:0/3210047602 learned_addr learned my addr 172.21.15.121:0/3210047602 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:37:26.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2ddd91640 10 monclient(hunting): get_auth_request con 0x7fd2d8137170 auth_method 0 2024-09-17T18:37:26.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2ddd91640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:26.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2ddd91640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:26.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2ddd91640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:26.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2dcd8f640 10 monclient(hunting): get_auth_request con 0x7fd2d813a850 auth_method 0 2024-09-17T18:37:26.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2dcd8f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:26.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2dcd8f640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:26.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2dcd8f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:26.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2dcd8f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:26.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2dcd8f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:26.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2dcd8f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:26.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2ddd91640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:26.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2ddd91640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:26.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2ddd91640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:26.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2dcd8f640 10 monclient(hunting): handle_auth_done global_id 64613 payload 824 2024-09-17T18:37:26.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2dcd8f640 1 -- 172.21.15.121:0/3210047602 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd2d80a33f0 msgr2=0x7fd2d80a22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:26.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2dcd8f640 1 --2- 172.21.15.121:0/3210047602 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd2d80a33f0 0x7fd2d80a22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:26.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2dcd8f640 1 -- 172.21.15.121:0/3210047602 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2d8137170 msgr2=0x7fd2d81cb540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:26.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2dcd8f640 1 --2- 172.21.15.121:0/3210047602 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2d8137170 0x7fd2d81cb540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:26.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2dcd8f640 10 monclient: _finish_hunting 0 2024-09-17T18:37:26.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2dcd8f640 1 monclient: found mon.noname-c 2024-09-17T18:37:26.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2dcd8f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:26.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2dcd8f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:37:26.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2dcd8f640 1 -- 172.21.15.121:0/3210047602 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2d81225f0 con 0x7fd2d813a850 2024-09-17T18:37:26.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.671+0000 7fd2ddd91640 1 --2- 172.21.15.121:0/3210047602 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2d8137170 0x7fd2d81cb540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:37:26.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.673+0000 7fd2dcd8f640 1 -- 172.21.15.121:0/3210047602 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2d813a850 msgr2=0x7fd2d8136c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:26.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.673+0000 7fd2dcd8f640 1 --2- 172.21.15.121:0/3210047602 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2d813a850 0x7fd2d8136c30 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc00e840 tx=0x7fd2cc00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:26.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.673+0000 7fd2d5ffb640 10 client.?.objecter ms_handle_connect 0x7fd2d813a850 2024-09-17T18:37:26.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.673+0000 7fd2d5ffb640 10 client.?.objecter resend_mon_ops 2024-09-17T18:37:26.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.673+0000 7fd2d5ffb640 1 -- 172.21.15.121:0/3210047602 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2cc017070 con 0x7fd2d813a850 2024-09-17T18:37:26.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.673+0000 7fd2d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:26.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.673+0000 7fd2d5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:37:26.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.673+0000 7fd2d5ffb640 10 monclient: dump: 2024-09-17T18:37:26.686 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:26.686 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:26.686 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:26.686 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:26.686 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:26.686 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:26.686 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:26.686 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:26.686 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:26.686 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:26.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.673+0000 7fd2d5ffb640 10 monclient: _finish_auth 0 2024-09-17T18:37:26.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.673+0000 7fd2d5ffb640 10 monclient: _check_auth_tickets 2024-09-17T18:37:26.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.673+0000 7fd2d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:26.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.673+0000 7fd2d5ffb640 1 -- 172.21.15.121:0/3210047602 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2cc00c940 con 0x7fd2d813a850 2024-09-17T18:37:26.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.673+0000 7fd2df7f4640 5 monclient: authenticate success, global_id 64613 2024-09-17T18:37:26.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.673+0000 7fd2df7f4640 10 monclient: _renew_subs 2024-09-17T18:37:26.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.673+0000 7fd2df7f4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:37:26.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.673+0000 7fd2df7f4640 1 -- 172.21.15.121:0/3210047602 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2d80a6420 con 0x7fd2d813a850 2024-09-17T18:37:26.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.673+0000 7fd2d5ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:26.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.673+0000 7fd2d5ffb640 1 -- 172.21.15.121:0/3210047602 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2cc00cd10 con 0x7fd2d813a850 2024-09-17T18:37:26.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.673+0000 7fd2d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:26.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.673+0000 7fd2d5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:37:26.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.673+0000 7fd2d5ffb640 10 monclient: dump: 2024-09-17T18:37:26.687 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:26.687 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:26.687 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:26.687 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:26.687 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:26.687 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:26.688 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:26.688 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:26.688 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:26.688 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:26.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.673+0000 7fd2d77fe640 4 set_mon_vals no callback set 2024-09-17T18:37:26.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.674+0000 7fd2df7f4640 10 client.64613.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:37:26.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.674+0000 7fd2df7f4640 10 monclient: _renew_subs 2024-09-17T18:37:26.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.674+0000 7fd2df7f4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:37:26.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.674+0000 7fd2df7f4640 1 -- 172.21.15.121:0/3210047602 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2d8121010 con 0x7fd2d813a850 2024-09-17T18:37:26.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.674+0000 7fd2df7f4640 1 librados: init done 2024-09-17T18:37:26.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.675+0000 7fd2df7f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:26.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.675+0000 7fd2df7f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:26.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.675+0000 7fd2df7f4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:37:26.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.675+0000 7fd2d5ffb640 1 -- 172.21.15.121:0/3210047602 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fd2cc01a070 con 0x7fd2d813a850 2024-09-17T18:37:26.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.675+0000 7fd2df7f4640 1 -- 172.21.15.121:0/3210047602 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2d809e550 con 0x7fd2d813a850 2024-09-17T18:37:26.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.675+0000 7fd2d5ffb640 1 --2- 172.21.15.121:0/3210047602 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd2c40401c0 0x7fd2c4042680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:26.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.675+0000 7fd2d5ffb640 1 -- 172.21.15.121:0/3210047602 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(679..679 src has 247..679) ==== 4685+0+0 (secure 0 0 0) 0x7fd2cc053230 con 0x7fd2d813a850 2024-09-17T18:37:26.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.675+0000 7fd2d5ffb640 10 client.64613.objecter ms_dispatch 0x7fd2d8000b90 osd_map(679..679 src has 247..679) 2024-09-17T18:37:26.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.675+0000 7fd2d5ffb640 3 client.64613.objecter handle_osd_map got epochs [679,679] > 0 2024-09-17T18:37:26.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.675+0000 7fd2d5ffb640 3 client.64613.objecter handle_osd_map decoding full epoch 679 2024-09-17T18:37:26.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.675+0000 7fd2d5ffb640 20 client.64613.objecter dump_active .. 0 homeless 2024-09-17T18:37:26.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.675+0000 7fd2dd590640 1 --2- 172.21.15.121:0/3210047602 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd2c40401c0 0x7fd2c4042680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:26.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.675+0000 7fd2dd590640 10 monclient: get_auth_request con 0x7fd2c40401c0 auth_method 0 2024-09-17T18:37:26.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.678+0000 7fd2dd590640 1 --2- 172.21.15.121:0/3210047602 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd2c40401c0 0x7fd2c4042680 secure :-1 s=READY pgs=4099 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0007fd0 tx=0x7fd2d0007aa0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:26.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.678+0000 7fd2d5ffb640 10 client.64613.objecter ms_handle_connect 0x7fd2c40401c0 2024-09-17T18:37:26.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.679+0000 7fd2d5ffb640 1 -- 172.21.15.121:0/3210047602 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd2cc051c90 con 0x7fd2d813a850 2024-09-17T18:37:26.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.679+0000 7fd2d5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:26.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.679+0000 7fd2d5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:37:26.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.853+0000 7fd2df7f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-17T18:37:26.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.853+0000 7fd2df7f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T18:37:26.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.853+0000 7fd2df7f4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:37:26.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.853+0000 7fd2df7f4640 1 -- 172.21.15.121:0/3210047602 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd2d813c750 con 0x7fd2d813a850 2024-09-17T18:37:26.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.853+0000 7fd2d5ffb640 1 -- 172.21.15.121:0/3210047602 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v679) ==== 74+0+14052 (secure 0 0 0) 0x7fd2cc013c20 con 0x7fd2d813a850 2024-09-17T18:37:26.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.853+0000 7fd2d5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T18:37:26.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.853+0000 7fd2d5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:37:26.856 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:37:26.856 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":679,"fsid":"a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b","created":"2024-09-17T18:18:47.237753+0000","modified":"2024-09-17T18:36:25.428956+0000","last_up_change":"2024-09-17T18:18:57.511756+0000","last_in_change":"2024-09-17T18:18:49.165275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":101,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":340,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T18:18:57.521369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-17T18:18:58.173538+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-09-17T18:20:00.703375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7bf586d1-05ac-4c7e-8b88-313f07a76801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":674,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":1526257129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6801","nonce":1526257129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":1526257129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":1526257129}]},"public_addr":"172.21.15.121:6800/1526257129","cluster_addr":"172.21.15.121:6801/1526257129","heartbeat_back_addr":"172.21.15.121:6803/1526257129","heartbeat_front_addr":"172.21.15.121:6802/1526257129","state":["exists","up"]},{"osd":1,"uuid":"e26c6163-1eda-4200-82e1-57b752fd8a46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":674,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":196419920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6809","nonce":196419920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":196419920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":196419920}]},"public_addr":"172.21.15.121:6808/196419920","cluster_addr":"172.21.15.121:6809/196419920","heartbeat_back_addr":"172.21.15.121:6811/196419920","heartbeat_front_addr":"172.21.15.121:6810/196419920","state":["exists","up"]},{"osd":2,"uuid":"6fb45798-21d4-449e-9f68-7210d984e036","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":674,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2545987420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6805","nonce":2545987420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":2545987420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2545987420}]},"public_addr":"172.21.15.121:6804/2545987420","cluster_addr":"172.21.15.121:6805/2545987420","heartbeat_back_addr":"172.21.15.121:6807/2545987420","heartbeat_front_addr":"172.21.15.121:6806/2545987420","state":["exists","up"]},{"osd":3,"uuid":"3de6630d-4fb7-4f64-b4f9-e5c304b1be63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":674,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":3493964597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6801","nonce":3493964597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6803","nonce":3493964597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":3493964597}]},"public_addr":"172.21.15.173:6800/3493964597","cluster_addr":"172.21.15.173:6801/3493964597","heartbeat_back_addr":"172.21.15.173:6803/3493964597","heartbeat_front_addr":"172.21.15.173:6802/3493964597","state":["exists","up"]},{"osd":4,"uuid":"cfa74383-b7c9-4ecd-8258-3792fbccc237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":674,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":1961801317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6805","nonce":1961801317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6807","nonce":1961801317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":1961801317}]},"public_addr":"172.21.15.173:6804/1961801317","cluster_addr":"172.21.15.173:6805/1961801317","heartbeat_back_addr":"172.21.15.173:6807/1961801317","heartbeat_front_addr":"172.21.15.173:6806/1961801317","state":["exists","up"]},{"osd":5,"uuid":"8bdb7768-a603-4f14-9650-47b503621215","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":674,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1801998802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6809","nonce":1801998802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6811","nonce":1801998802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1801998802}]},"public_addr":"172.21.15.173:6808/1801998802","cluster_addr":"172.21.15.173:6809/1801998802","heartbeat_back_addr":"172.21.15.173:6811/1801998802","heartbeat_front_addr":"172.21.15.173:6810/1801998802","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T18:18:55.231353+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T18:18:56.130097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T18:18:56.242909+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T18:18:55.454088+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T18:18:55.965469+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T18:18:55.389354+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":12,"snaps":[{"begin":3,"length":4}]},{"pool":13,"snaps":[{"begin":1,"length":4}]},{"pool":31,"snaps":[{"begin":1,"length":1}]},{"pool":70,"snaps":[{"begin":1,"length":1}]},{"pool":92,"snaps":[{"begin":7,"length":2}]},{"pool":105,"snaps":[{"begin":1,"length":1}]},{"pool":124,"snaps":[{"begin":1,"length":2}]},{"pool":139,"snaps":[{"begin":3,"length":1}]},{"pool":145,"snaps":[{"begin":1,"length":4}]},{"pool":149,"snaps":[{"begin":2,"length":1}]},{"pool":154,"snaps":[{"begin":2,"length":2}]},{"pool":163,"snaps":[{"begin":2,"length":3}]},{"pool":168,"snaps":[{"begin":3,"length":4}]},{"pool":171,"snaps":[{"begin":2,"length":4}]},{"pool":173,"snaps":[{"begin":2,"length":2}]},{"pool":182,"snaps":[{"begin":3,"length":1}]},{"pool":184,"snaps":[{"begin":2,"length":1}]},{"pool":189,"snaps":[{"begin":1,"length":4}]},{"pool":194,"snaps":[{"begin":7,"length":1}]},{"pool":204,"snaps":[{"begin":4,"length":4}]},{"pool":223,"snaps":[{"begin":8,"length":1}]},{"pool":234,"snaps":[{"begin":10,"length":1}]},{"pool":240,"snaps":[{"begin":11,"length":2}]},{"pool":255,"snaps":[{"begin":14,"length":1}]},{"pool":321,"snaps":[{"begin":2,"length":1}]},{"pool":322,"snaps":[{"begin":3,"length":1}]},{"pool":325,"snaps":[{"begin":4,"length":1}]},{"pool":329,"snaps":[{"begin":6,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T18:37:26.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.855+0000 7fd2df7f4640 10 librados: watch_flush enter 2024-09-17T18:37:26.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.856+0000 7fd2df7f4640 10 librados: watch_flush exit 2024-09-17T18:37:26.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.856+0000 7fd2df7f4640 20 client.64613.objecter shutdown clearing up homeless session... 2024-09-17T18:37:26.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.856+0000 7fd2df7f4640 10 client.64613.objecter successfully canceled tick 2024-09-17T18:37:26.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.856+0000 7fd2df7f4640 1 -- 172.21.15.121:0/3210047602 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd2c40401c0 msgr2=0x7fd2c4042680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:26.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.856+0000 7fd2df7f4640 1 --2- 172.21.15.121:0/3210047602 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd2c40401c0 0x7fd2c4042680 secure :-1 s=READY pgs=4099 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0007fd0 tx=0x7fd2d0007aa0 comp rx=0 tx=0).stop 2024-09-17T18:37:26.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.856+0000 7fd2df7f4640 10 monclient: shutdown 2024-09-17T18:37:26.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.856+0000 7fd2df7f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:26.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.856+0000 7fd2df7f4640 1 -- 172.21.15.121:0/3210047602 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2d813a850 msgr2=0x7fd2d8136c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:26.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.856+0000 7fd2df7f4640 1 --2- 172.21.15.121:0/3210047602 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2d813a850 0x7fd2d8136c30 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc00e840 tx=0x7fd2cc00ed00 comp rx=0 tx=0).stop 2024-09-17T18:37:26.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.856+0000 7fd2df7f4640 1 -- 172.21.15.121:0/3210047602 shutdown_connections 2024-09-17T18:37:26.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.856+0000 7fd2df7f4640 1 --2- 172.21.15.121:0/3210047602 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd2c40401c0 0x7fd2c4042680 unknown :-1 s=CLOSED pgs=4099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:26.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.856+0000 7fd2df7f4640 1 --2- 172.21.15.121:0/3210047602 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2d8137170 0x7fd2d81cb540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:26.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.856+0000 7fd2df7f4640 1 --2- 172.21.15.121:0/3210047602 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2d813a850 0x7fd2d8136c30 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:26.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.856+0000 7fd2df7f4640 1 --2- 172.21.15.121:0/3210047602 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd2d80a33f0 0x7fd2d80a22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:26.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.856+0000 7fd2df7f4640 1 -- 172.21.15.121:0/3210047602 >> 172.21.15.121:0/3210047602 conn(0x7fd2d80a3b50 msgr2=0x7fd2d813b140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:26.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.856+0000 7fd2df7f4640 1 -- 172.21.15.121:0/3210047602 shutdown_connections 2024-09-17T18:37:26.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.856+0000 7fd2df7f4640 1 -- 172.21.15.121:0/3210047602 wait complete. 2024-09-17T18:37:26.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:26.856+0000 7fd2df7f4640 1 librados: shutdown 2024-09-17T18:37:27.869 INFO:tasks.ceph:Scrubbing osd.0 2024-09-17T18:37:27.870 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-17T18:37:27.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.973+0000 7f879c7e2640 10 monclient: get_monmap_and_config 2024-09-17T18:37:27.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.973+0000 7f879c7e2640 10 monclient: build_initial_monmap 2024-09-17T18:37:27.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.973+0000 7f879c7e2640 10 monclient: monmap: 2024-09-17T18:37:27.976 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:27.976 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:27.976 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:37:27.974356+0000 2024-09-17T18:37:27.976 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:37:27.974356+0000 2024-09-17T18:37:27.976 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:27.976 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:27.976 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:37:27.976 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:37:27.976 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:37:27.976 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:37:27.977 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:37:27.977 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:27.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.974+0000 7f879c7e2640 1 Processor -- start 2024-09-17T18:37:27.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.974+0000 7f879c7e2640 1 -- start start 2024-09-17T18:37:27.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.974+0000 7f879c7e2640 10 monclient: init 2024-09-17T18:37:27.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.974+0000 7f879c7e2640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:27.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.974+0000 7f879c7e2640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-17T18:37:27.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.974+0000 7f879c7e2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f879409e470 addr v1:172.21.15.173:6789/0 2024-09-17T18:37:27.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.974+0000 7f879c7e2640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87940a1270 0x7f879413a770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:27.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.974+0000 7f879c7e2640 10 monclient(hunting): picked mon.noname-b con 0x7f87940a1270 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:27.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.974+0000 7f879c7e2640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f879409ef60 0x7f879413ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:27.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.974+0000 7f879c7e2640 10 monclient(hunting): picked mon.noname-a con 0x7f879409ef60 addr v2:172.21.15.121:3300/0 2024-09-17T18:37:27.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.975+0000 7f879c7e2640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8794074ab0 con 0x7f879409e470 2024-09-17T18:37:27.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.975+0000 7f879c7e2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:27.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.975+0000 7f879c7e2640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f87940a0a40 con 0x7f879409ef60 2024-09-17T18:37:27.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.975+0000 7f879c7e2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:27.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.975+0000 7f879c7e2640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f87940a36a0 con 0x7f87940a1270 2024-09-17T18:37:27.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.975+0000 7f879c7e2640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:27.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.975+0000 7f879c7e2640 10 monclient(hunting): authenticate will time out at 1851.691030s 2024-09-17T18:37:27.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.975+0000 7f879ad7f640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f879409ef60 0x7f879413ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:27.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.975+0000 7f8799d7d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87940a1270 0x7f879413a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:27.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.975+0000 7f879ad7f640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f879409ef60 0x7f879413ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51318/0 (socket says 172.21.15.121:51318) 2024-09-17T18:37:27.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.975+0000 7f879ad7f640 1 -- 172.21.15.121:0/3640639149 learned_addr learned my addr 172.21.15.121:0/3640639149 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:37:27.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.975+0000 7f879ad7f640 10 monclient(hunting): get_auth_request con 0x7f879409ef60 auth_method 0 2024-09-17T18:37:27.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.975+0000 7f879ad7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:27.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.975+0000 7f879ad7f640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:27.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.975+0000 7f879ad7f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:27.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.975+0000 7f8799d7d640 10 monclient(hunting): get_auth_request con 0x7f87940a1270 auth_method 0 2024-09-17T18:37:27.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.975+0000 7f8799d7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:27.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.975+0000 7f8799d7d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:27.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.975+0000 7f8799d7d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:27.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.975+0000 7f879ad7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:27.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.975+0000 7f879ad7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:27.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.975+0000 7f879ad7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:27.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.975+0000 7f8799d7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:27.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.975+0000 7f8799d7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:27.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.975+0000 7f8799d7d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:27.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.975+0000 7f879ad7f640 10 monclient(hunting): handle_auth_done global_id 114156 payload 274 2024-09-17T18:37:27.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.975+0000 7f879ad7f640 1 -- 172.21.15.121:0/3640639149 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87940a1270 msgr2=0x7f879413a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:27.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.975+0000 7f879ad7f640 1 --2- 172.21.15.121:0/3640639149 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87940a1270 0x7f879413a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:27.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.975+0000 7f879ad7f640 1 -- 172.21.15.121:0/3640639149 >> v1:172.21.15.173:6789/0 conn(0x7f879409e470 legacy=0x7f879409e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:27.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.975+0000 7f879ad7f640 10 monclient: _finish_hunting 0 2024-09-17T18:37:27.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.975+0000 7f879ad7f640 1 monclient: found mon.noname-a 2024-09-17T18:37:27.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.975+0000 7f879ad7f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:27.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.975+0000 7f879ad7f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:37:27.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.975+0000 7f879ad7f640 1 -- 172.21.15.121:0/3640639149 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f879409f520 con 0x7f879409ef60 2024-09-17T18:37:27.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.975+0000 7f8799d7d640 1 --2- 172.21.15.121:0/3640639149 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87940a1270 0x7f879413a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:37:27.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.978+0000 7f879ad7f640 1 -- 172.21.15.121:0/3640639149 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f879409ef60 msgr2=0x7f879413ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:27.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.978+0000 7f879ad7f640 1 --2- 172.21.15.121:0/3640639149 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f879409ef60 0x7f879413ccc0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f878c00d150 tx=0x7f878c02c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16247dbf92e5b038 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:27.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.978+0000 7f879957c640 1 -- 172.21.15.121:0/3640639149 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f878c038070 con 0x7f879409ef60 2024-09-17T18:37:27.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.978+0000 7f879957c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:27.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.978+0000 7f879957c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:37:27.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.978+0000 7f879957c640 10 monclient: dump: 2024-09-17T18:37:27.988 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:27.988 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:27.988 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:27.988 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:27.988 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:27.988 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:27.988 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:27.988 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:27.988 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:27.988 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:27.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.978+0000 7f879957c640 10 monclient: _finish_auth 0 2024-09-17T18:37:27.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.978+0000 7f879957c640 10 monclient: _check_auth_tickets 2024-09-17T18:37:27.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.978+0000 7f879957c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:27.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.978+0000 7f879c7e2640 5 monclient: authenticate success, global_id 114156 2024-09-17T18:37:27.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.978+0000 7f879957c640 1 -- 172.21.15.121:0/3640639149 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f878c009800 con 0x7f879409ef60 2024-09-17T18:37:27.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.978+0000 7f879957c640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:27.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.978+0000 7f879c7e2640 10 monclient: get_monmap_and_config success 2024-09-17T18:37:27.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.978+0000 7f879c7e2640 4 set_mon_vals no callback set 2024-09-17T18:37:27.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.978+0000 7f879957c640 1 -- 172.21.15.121:0/3640639149 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f878c009bd0 con 0x7f879409ef60 2024-09-17T18:37:27.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.978+0000 7f879957c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:27.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.978+0000 7f879957c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:37:27.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.978+0000 7f879957c640 10 monclient: dump: 2024-09-17T18:37:27.989 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:27.989 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:27.989 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:27.989 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:27.989 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:27.989 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:27.989 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:27.990 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:27.990 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:27.990 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:27.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.978+0000 7f879c7e2640 10 monclient: shutdown 2024-09-17T18:37:27.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.978+0000 7f879c7e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:27.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.978+0000 7f879c7e2640 1 -- 172.21.15.121:0/3640639149 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f879409ef60 msgr2=0x7f879413ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:27.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.978+0000 7f879c7e2640 1 --2- 172.21.15.121:0/3640639149 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f879409ef60 0x7f879413ccc0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f878c00d150 tx=0x7f878c02c760 comp rx=0 tx=0).stop 2024-09-17T18:37:27.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.978+0000 7f879c7e2640 1 -- 172.21.15.121:0/3640639149 shutdown_connections 2024-09-17T18:37:27.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.978+0000 7f879c7e2640 1 --2- 172.21.15.121:0/3640639149 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f879409ef60 0x7f879413ccc0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:27.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.978+0000 7f879c7e2640 1 --2- 172.21.15.121:0/3640639149 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87940a1270 0x7f879413a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:27.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.978+0000 7f879c7e2640 1 -- 172.21.15.121:0/3640639149 >> 172.21.15.121:0/3640639149 conn(0x7f87940a3a70 msgr2=0x7f87940a3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:27.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.979+0000 7f879c7e2640 1 -- 172.21.15.121:0/3640639149 shutdown_connections 2024-09-17T18:37:27.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.979+0000 7f879c7e2640 1 -- 172.21.15.121:0/3640639149 wait complete. 2024-09-17T18:37:27.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.979+0000 7f879c7e2640 10 monclient: build_initial_monmap 2024-09-17T18:37:27.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.979+0000 7f879c7e2640 10 monclient: monmap: 2024-09-17T18:37:27.991 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:27.991 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:37:27.991 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:37:27.991 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:37:27.991 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:27.991 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:27.991 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:37:27.991 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:37:27.991 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:37:27.991 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:27.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.979+0000 7f879c7e2640 1 librados: starting msgr at 2024-09-17T18:37:27.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.979+0000 7f879c7e2640 1 librados: starting objecter 2024-09-17T18:37:27.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.979+0000 7f879c7e2640 1 Processor -- start 2024-09-17T18:37:27.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.979+0000 7f879c7e2640 1 -- start start 2024-09-17T18:37:27.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.979+0000 7f879c7e2640 1 librados: setting wanted keys 2024-09-17T18:37:27.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.979+0000 7f879c7e2640 1 librados: calling monclient init 2024-09-17T18:37:27.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.979+0000 7f879c7e2640 10 monclient: init 2024-09-17T18:37:27.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f879c7e2640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:27.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f879c7e2640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T18:37:27.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f879c7e2640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f879409e470 0x7f87941b8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:27.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f879c7e2640 10 monclient(hunting): picked mon.noname-a con 0x7f879409e470 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:37:27.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f879c7e2640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f879409ef60 0x7f87941a37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:27.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f879c7e2640 10 monclient(hunting): picked mon.noname-c con 0x7f879409ef60 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:27.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f879c7e2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87941a3d00 0x7f87941a41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:27.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f879c7e2640 10 monclient(hunting): picked mon.noname-b con 0x7f87941a3d00 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:37:27.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f879c7e2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:27.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f879c7e2640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f87940a1af0 con 0x7f879409e470 2024-09-17T18:37:27.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f879c7e2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:27.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f879c7e2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f87940a1970 con 0x7f87941a3d00 2024-09-17T18:37:27.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f879c7e2640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:27.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f879c7e2640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f87940a2200 con 0x7f879409ef60 2024-09-17T18:37:27.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f879c7e2640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:27.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f879c7e2640 10 monclient(hunting): authenticate will time out at 1851.696483s 2024-09-17T18:37:27.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f8799d7d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f879409ef60 0x7f87941a37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:27.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f879a57e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f879409e470 0x7f87941b8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:27.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f8799d7d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f879409ef60 0x7f87941a37c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:36842/0 (socket says 172.21.15.121:36842) 2024-09-17T18:37:27.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f879a57e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f879409e470 0x7f87941b8f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51320/0 (socket says 172.21.15.121:51320) 2024-09-17T18:37:27.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f8799d7d640 1 -- 172.21.15.121:0/2429134734 learned_addr learned my addr 172.21.15.121:0/2429134734 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:37:27.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f8799d7d640 10 monclient(hunting): get_auth_request con 0x7f879409ef60 auth_method 0 2024-09-17T18:37:27.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f8799d7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:27.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f8799d7d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:27.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f8799d7d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:27.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f879a57e640 10 monclient(hunting): get_auth_request con 0x7f879409e470 auth_method 0 2024-09-17T18:37:27.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f879a57e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:27.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f879a57e640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:27.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f879a57e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:27.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f8799d7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:27.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f8799d7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:27.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f8799d7d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:27.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f879a57e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:27.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f879a57e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:27.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f879a57e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:27.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f8799d7d640 10 monclient(hunting): handle_auth_done global_id 64619 payload 824 2024-09-17T18:37:27.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f8799d7d640 1 -- 172.21.15.121:0/2429134734 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87941a3d00 msgr2=0x7f87941a41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:27.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f8799d7d640 1 --2- 172.21.15.121:0/2429134734 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87941a3d00 0x7f87941a41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:27.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f8799d7d640 1 -- 172.21.15.121:0/2429134734 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f879409e470 msgr2=0x7f87941b8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:27.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f8799d7d640 1 --2- 172.21.15.121:0/2429134734 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f879409e470 0x7f87941b8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:27.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f8799d7d640 10 monclient: _finish_hunting 0 2024-09-17T18:37:27.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f8799d7d640 1 monclient: found mon.noname-c 2024-09-17T18:37:27.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f8799d7d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:27.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f8799d7d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:37:27.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f8799d7d640 1 -- 172.21.15.121:0/2429134734 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87941b8310 con 0x7f879409ef60 2024-09-17T18:37:27.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.980+0000 7f879a57e640 1 --2- 172.21.15.121:0/2429134734 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f879409e470 0x7f87941b8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:37:27.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.983+0000 7f8799d7d640 1 -- 172.21.15.121:0/2429134734 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f879409ef60 msgr2=0x7f87941a37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:27.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.983+0000 7f8799d7d640 1 --2- 172.21.15.121:0/2429134734 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f879409ef60 0x7f87941a37c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f878400eb30 tx=0x7f8784009e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:27.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.983+0000 7f878affd640 10 client.?.objecter ms_handle_connect 0x7f879409ef60 2024-09-17T18:37:27.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.983+0000 7f878affd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:37:27.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.983+0000 7f878affd640 1 -- 172.21.15.121:0/2429134734 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8784017070 con 0x7f879409ef60 2024-09-17T18:37:27.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.983+0000 7f878affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:27.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.983+0000 7f878affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:37:27.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.983+0000 7f878affd640 10 monclient: dump: 2024-09-17T18:37:27.995 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:27.995 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:27.995 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:27.996 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:27.996 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:27.996 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:27.996 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:27.996 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:27.996 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:27.996 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:27.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.983+0000 7f878affd640 10 monclient: _finish_auth 0 2024-09-17T18:37:27.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.983+0000 7f878affd640 10 monclient: _check_auth_tickets 2024-09-17T18:37:27.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.983+0000 7f878affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:27.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.983+0000 7f878affd640 1 -- 172.21.15.121:0/2429134734 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f878400cc10 con 0x7f879409ef60 2024-09-17T18:37:27.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.983+0000 7f879c7e2640 5 monclient: authenticate success, global_id 64619 2024-09-17T18:37:27.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.983+0000 7f879c7e2640 10 monclient: _renew_subs 2024-09-17T18:37:27.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.983+0000 7f879c7e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:37:27.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.983+0000 7f879c7e2640 1 -- 172.21.15.121:0/2429134734 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87941a6770 con 0x7f879409ef60 2024-09-17T18:37:27.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.983+0000 7f878affd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:27.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.983+0000 7f878affd640 1 -- 172.21.15.121:0/2429134734 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8784013470 con 0x7f879409ef60 2024-09-17T18:37:27.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.983+0000 7f878affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:27.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.983+0000 7f878affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:37:27.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.983+0000 7f878affd640 10 monclient: dump: 2024-09-17T18:37:27.997 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:27.997 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:27.997 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:27.997 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:27.997 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:27.997 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:27.998 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:27.998 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:27.998 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:27.998 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:27.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.983+0000 7f8798d7b640 4 set_mon_vals no callback set 2024-09-17T18:37:27.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.983+0000 7f879c7e2640 10 client.64619.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:37:27.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.983+0000 7f879c7e2640 10 monclient: _renew_subs 2024-09-17T18:37:27.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.983+0000 7f879c7e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:37:27.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.983+0000 7f879c7e2640 1 -- 172.21.15.121:0/2429134734 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f87941a5280 con 0x7f879409ef60 2024-09-17T18:37:27.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.983+0000 7f879c7e2640 1 librados: init done 2024-09-17T18:37:27.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.984+0000 7f878affd640 1 -- 172.21.15.121:0/2429134734 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f878401a070 con 0x7f879409ef60 2024-09-17T18:37:27.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.984+0000 7f879c7e2640 10 client.64619.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:27.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.984+0000 7f879c7e2640 15 client.64619.objecter _session_command_op_assign -1 1 2024-09-17T18:37:27.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.984+0000 7f879c7e2640 20 client.64619.objecter _get_session osd=-1 returning homeless 2024-09-17T18:37:27.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.984+0000 7f879c7e2640 10 client.64619.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:37:27.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.984+0000 7f879c7e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:37:27.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.984+0000 7f879c7e2640 1 -- 172.21.15.121:0/2429134734 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f879409e850 con 0x7f879409ef60 2024-09-17T18:37:27.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.984+0000 7f878affd640 1 --2- 172.21.15.121:0/2429134734 >> v2:172.21.15.173:6812/2423786037 conn(0x7f8778040170 0x7f8778042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:27.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.984+0000 7f878affd640 1 -- 172.21.15.121:0/2429134734 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(679..679 src has 247..679) ==== 4685+0+0 (secure 0 0 0) 0x7f8784053400 con 0x7f879409ef60 2024-09-17T18:37:27.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.984+0000 7f878affd640 10 client.64619.objecter ms_dispatch 0x7f8794000b90 osd_map(679..679 src has 247..679) 2024-09-17T18:37:27.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.984+0000 7f878affd640 3 client.64619.objecter handle_osd_map got epochs [679,679] > 0 2024-09-17T18:37:27.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.984+0000 7f878affd640 3 client.64619.objecter handle_osd_map decoding full epoch 679 2024-09-17T18:37:27.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.984+0000 7f878affd640 10 client.64619.objecter checking command 1 2024-09-17T18:37:27.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.984+0000 7f878affd640 1 --2- 172.21.15.121:0/2429134734 >> v2:172.21.15.121:6800/1526257129 conn(0x7f8778047630 0x7f8778049ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:27.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.984+0000 7f878affd640 20 client.64619.objecter _get_session s=0x7f8778046f80 osd=0 3 2024-09-17T18:37:27.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.984+0000 7f878affd640 20 client.64619.objecter put_session s=0x7f8778046f80 osd=0 3 2024-09-17T18:37:27.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.984+0000 7f878affd640 15 client.64619.objecter _session_command_op_remove -1 1 2024-09-17T18:37:27.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.984+0000 7f878affd640 20 client.64619.objecter _get_session s=0x7f8778046f80 osd=0 3 2024-09-17T18:37:27.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.984+0000 7f878affd640 20 client.64619.objecter get_session s=0x7f8778046f80 osd=0 3 2024-09-17T18:37:27.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.984+0000 7f878affd640 15 client.64619.objecter _session_command_op_assign 0 1 2024-09-17T18:37:27.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.984+0000 7f878affd640 20 client.64619.objecter put_session s=0x7f8778046f80 osd=0 4 2024-09-17T18:37:27.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.984+0000 7f878affd640 10 client.64619.objecter _send_command 1 2024-09-17T18:37:28.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.984+0000 7f878affd640 1 -- 172.21.15.121:0/2429134734 --> v2:172.21.15.121:6800/1526257129 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f87840093e0 con 0x7f8778047630 2024-09-17T18:37:28.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.984+0000 7f878affd640 20 client.64619.objecter dump_active .. 0 homeless 2024-09-17T18:37:28.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.984+0000 7f879a57e640 1 --2- 172.21.15.121:0/2429134734 >> v2:172.21.15.173:6812/2423786037 conn(0x7f8778040170 0x7f8778042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:28.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.984+0000 7f879a57e640 10 monclient: get_auth_request con 0x7f8778040170 auth_method 0 2024-09-17T18:37:28.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.984+0000 7f879ad7f640 1 --2- 172.21.15.121:0/2429134734 >> v2:172.21.15.121:6800/1526257129 conn(0x7f8778047630 0x7f8778049ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:28.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.984+0000 7f879ad7f640 10 monclient: get_auth_request con 0x7f8778047630 auth_method 0 2024-09-17T18:37:28.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.987+0000 7f878affd640 10 client.64619.objecter ms_handle_connect 0x7f8778040170 2024-09-17T18:37:28.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.987+0000 7f879a57e640 1 --2- 172.21.15.121:0/2429134734 >> v2:172.21.15.173:6812/2423786037 conn(0x7f8778040170 0x7f8778042630 secure :-1 s=READY pgs=4100 cs=0 l=1 rev1=1 crypto rx=0x7f8790004320 tx=0x7f8790007e30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:28.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.989+0000 7f879ad7f640 1 --2- 172.21.15.121:0/2429134734 >> v2:172.21.15.121:6800/1526257129 conn(0x7f8778047630 0x7f8778049ad0 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:28.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.989+0000 7f878affd640 10 client.64619.objecter ms_handle_connect 0x7f8778047630 2024-09-17T18:37:28.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.989+0000 7f878affd640 1 -- 172.21.15.121:0/2429134734 <== osd.0 v2:172.21.15.121:6800/1526257129 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f87840093e0 con 0x7f8778047630 2024-09-17T18:37:28.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.989+0000 7f878affd640 10 client.64619.objecter ms_dispatch 0x7f8794000b90 command_reply(tid 1: 0 ) 2024-09-17T18:37:28.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.989+0000 7f878affd640 10 client.64619.objecter _finish_command 1 = system:0 2024-09-17T18:37:28.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.989+0000 7f878affd640 20 client.64619.objecter put_session s=0x7f8778046f80 osd=0 4 2024-09-17T18:37:28.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:27.989+0000 7f878affd640 15 client.64619.objecter _session_command_op_remove 0 1 2024-09-17T18:37:28.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.001+0000 7f879c7e2640 10 client.64619.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-17T18:37:28.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.001+0000 7f879c7e2640 15 client.64619.objecter _session_command_op_assign -1 2 2024-09-17T18:37:28.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.001+0000 7f879c7e2640 20 client.64619.objecter _get_session s=0x7f8778046f80 osd=0 3 2024-09-17T18:37:28.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.001+0000 7f879c7e2640 20 client.64619.objecter put_session s=0x7f8778046f80 osd=0 3 2024-09-17T18:37:28.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.001+0000 7f879c7e2640 20 client.64619.objecter _get_session s=0x7f8778046f80 osd=0 3 2024-09-17T18:37:28.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.001+0000 7f879c7e2640 15 client.64619.objecter _session_command_op_remove -1 2 2024-09-17T18:37:28.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.001+0000 7f879c7e2640 20 client.64619.objecter get_session s=0x7f8778046f80 osd=0 3 2024-09-17T18:37:28.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.001+0000 7f879c7e2640 15 client.64619.objecter _session_command_op_assign 0 2 2024-09-17T18:37:28.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.001+0000 7f879c7e2640 20 client.64619.objecter put_session s=0x7f8778046f80 osd=0 4 2024-09-17T18:37:28.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.001+0000 7f879c7e2640 10 client.64619.objecter _send_command 2 2024-09-17T18:37:28.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.001+0000 7f879c7e2640 1 -- 172.21.15.121:0/2429134734 --> v2:172.21.15.121:6800/1526257129 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f87941a71b0 con 0x7f8778047630 2024-09-17T18:37:28.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.002+0000 7f878affd640 1 -- 172.21.15.121:0/2429134734 <== osd.0 v2:172.21.15.121:6800/1526257129 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f878c039040 con 0x7f8778047630 2024-09-17T18:37:28.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.002+0000 7f878affd640 10 client.64619.objecter ms_dispatch 0x7f8794000b90 command_reply(tid 2: 0 ) 2024-09-17T18:37:28.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.002+0000 7f878affd640 10 client.64619.objecter _finish_command 2 = system:0 2024-09-17T18:37:28.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.002+0000 7f878affd640 20 client.64619.objecter put_session s=0x7f8778046f80 osd=0 4 2024-09-17T18:37:28.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.002+0000 7f878affd640 15 client.64619.objecter _session_command_op_remove 0 2 2024-09-17T18:37:28.005 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:37:28.005 INFO:teuthology.orchestra.run.smithi121.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T18:37:28.005 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:37:28.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.002+0000 7f879c7e2640 10 librados: watch_flush enter 2024-09-17T18:37:28.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.002+0000 7f879c7e2640 10 librados: watch_flush exit 2024-09-17T18:37:28.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.002+0000 7f879c7e2640 10 client.64619.objecter close_session for osd.0 2024-09-17T18:37:28.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.002+0000 7f879c7e2640 1 -- 172.21.15.121:0/2429134734 >> v2:172.21.15.121:6800/1526257129 conn(0x7f8778047630 msgr2=0x7f8778049ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:28.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.002+0000 7f879c7e2640 1 --2- 172.21.15.121:0/2429134734 >> v2:172.21.15.121:6800/1526257129 conn(0x7f8778047630 0x7f8778049ad0 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:28.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.002+0000 7f879c7e2640 20 client.64619.objecter put_session s=0x7f8778046f80 osd=0 1 2024-09-17T18:37:28.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.002+0000 7f879c7e2640 20 client.64619.objecter shutdown clearing up homeless session... 2024-09-17T18:37:28.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.002+0000 7f879c7e2640 10 client.64619.objecter successfully canceled tick 2024-09-17T18:37:28.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.002+0000 7f879c7e2640 1 -- 172.21.15.121:0/2429134734 >> v2:172.21.15.173:6812/2423786037 conn(0x7f8778040170 msgr2=0x7f8778042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:28.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.002+0000 7f879c7e2640 1 --2- 172.21.15.121:0/2429134734 >> v2:172.21.15.173:6812/2423786037 conn(0x7f8778040170 0x7f8778042630 secure :-1 s=READY pgs=4100 cs=0 l=1 rev1=1 crypto rx=0x7f8790004320 tx=0x7f8790007e30 comp rx=0 tx=0).stop 2024-09-17T18:37:28.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.002+0000 7f879c7e2640 10 monclient: shutdown 2024-09-17T18:37:28.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.002+0000 7f879c7e2640 20 monclient: shutdown canceling and discarding version request 1 2024-09-17T18:37:28.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.002+0000 7f879c7e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:28.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.002+0000 7f879c7e2640 1 -- 172.21.15.121:0/2429134734 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f879409ef60 msgr2=0x7f87941a37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:28.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.002+0000 7f879c7e2640 1 --2- 172.21.15.121:0/2429134734 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f879409ef60 0x7f87941a37c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f878400eb30 tx=0x7f8784009e00 comp rx=0 tx=0).stop 2024-09-17T18:37:28.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.002+0000 7f879ad7f640 1 -- 172.21.15.121:0/2429134734 reap_dead start 2024-09-17T18:37:28.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.002+0000 7f879c7e2640 1 -- 172.21.15.121:0/2429134734 shutdown_connections 2024-09-17T18:37:28.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.002+0000 7f879c7e2640 1 -- 172.21.15.121:0/2429134734 >> 172.21.15.121:0/2429134734 conn(0x7f87940a3a70 msgr2=0x7f87940a4e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:28.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.002+0000 7f879c7e2640 1 -- 172.21.15.121:0/2429134734 shutdown_connections 2024-09-17T18:37:28.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.002+0000 7f879c7e2640 1 -- 172.21.15.121:0/2429134734 wait complete. 2024-09-17T18:37:28.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.002+0000 7f879c7e2640 1 librados: shutdown 2024-09-17T18:37:28.014 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-17T18:37:28.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.109+0000 7fd1654b1640 10 monclient: get_monmap_and_config 2024-09-17T18:37:28.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.109+0000 7fd1654b1640 10 monclient: build_initial_monmap 2024-09-17T18:37:28.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.109+0000 7fd1654b1640 10 monclient: monmap: 2024-09-17T18:37:28.111 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:28.111 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:28.111 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:37:28.110047+0000 2024-09-17T18:37:28.111 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:37:28.110047+0000 2024-09-17T18:37:28.111 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:28.111 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:28.111 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:37:28.111 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:37:28.112 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:37:28.112 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:37:28.112 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:37:28.112 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:28.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.110+0000 7fd1654b1640 1 Processor -- start 2024-09-17T18:37:28.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.110+0000 7fd1654b1640 1 -- start start 2024-09-17T18:37:28.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.110+0000 7fd1654b1640 10 monclient: init 2024-09-17T18:37:28.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.110+0000 7fd1654b1640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:28.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.110+0000 7fd1654b1640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-17T18:37:28.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.110+0000 7fd1654b1640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd1600a5b90 0x7fd1600a5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:28.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.110+0000 7fd1654b1640 10 monclient(hunting): picked mon.noname-a con 0x7fd1600a5b90 addr v2:172.21.15.121:3300/0 2024-09-17T18:37:28.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.110+0000 7fd1654b1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd16009d830 addr v1:172.21.15.173:6789/0 2024-09-17T18:37:28.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.110+0000 7fd1654b1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd1600a7080 addr v1:172.21.15.121:6789/0 2024-09-17T18:37:28.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.110+0000 7fd1654b1640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd160139770 con 0x7fd1600a7080 2024-09-17T18:37:28.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.110+0000 7fd1654b1640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1600a8690 con 0x7fd16009d830 2024-09-17T18:37:28.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.110+0000 7fd1654b1640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:28.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.110+0000 7fd1654b1640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fd16009cde0 con 0x7fd1600a5b90 2024-09-17T18:37:28.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.110+0000 7fd1654b1640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:28.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.110+0000 7fd1654b1640 10 monclient(hunting): authenticate will time out at 1851.826859s 2024-09-17T18:37:28.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.110+0000 7fd15effd640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd1600a5b90 0x7fd1600a5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:28.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.110+0000 7fd15effd640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd1600a5b90 0x7fd1600a5f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51330/0 (socket says 172.21.15.121:51330) 2024-09-17T18:37:28.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.110+0000 7fd15f7fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fd1600a7080 0x7fd160139060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:47368/0 (socket says 172.21.15.121:47368) 2024-09-17T18:37:28.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.110+0000 7fd15effd640 1 -- 172.21.15.121:0/1311353358 learned_addr learned my addr 172.21.15.121:0/1311353358 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:37:28.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.110+0000 7fd15effd640 10 monclient(hunting): get_auth_request con 0x7fd1600a5b90 auth_method 0 2024-09-17T18:37:28.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.110+0000 7fd15effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:28.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.110+0000 7fd15effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:28.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.110+0000 7fd15effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:28.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.111+0000 7fd15effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:28.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.111+0000 7fd15effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:28.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.111+0000 7fd15effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:28.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.111+0000 7fd15effd640 10 monclient(hunting): handle_auth_done global_id 114162 payload 274 2024-09-17T18:37:28.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.111+0000 7fd15effd640 1 -- 172.21.15.121:0/1311353358 >> v1:172.21.15.173:6789/0 conn(0x7fd16009d830 legacy=0x7fd160138950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:28.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.111+0000 7fd15effd640 1 -- 172.21.15.121:0/1311353358 >> v1:172.21.15.121:6789/0 conn(0x7fd1600a7080 legacy=0x7fd160139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:28.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.111+0000 7fd15effd640 10 monclient: _finish_hunting 0 2024-09-17T18:37:28.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.111+0000 7fd15effd640 1 monclient: found mon.noname-a 2024-09-17T18:37:28.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.111+0000 7fd15effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:28.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.111+0000 7fd15effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:37:28.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.111+0000 7fd15effd640 1 -- 172.21.15.121:0/1311353358 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd16009cb00 con 0x7fd1600a5b90 2024-09-17T18:37:28.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.114+0000 7fd15effd640 1 -- 172.21.15.121:0/1311353358 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1600a5b90 msgr2=0x7fd1600a5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:28.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.114+0000 7fd15effd640 1 --2- 172.21.15.121:0/1311353358 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1600a5b90 0x7fd1600a5f70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd16006ed40 tx=0x7fd15002a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2a6248f28be82c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:28.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.114+0000 7fd15e7fc640 1 -- 172.21.15.121:0/1311353358 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd150037070 con 0x7fd1600a5b90 2024-09-17T18:37:28.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.114+0000 7fd15e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:28.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.114+0000 7fd15e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:37:28.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.114+0000 7fd15e7fc640 10 monclient: dump: 2024-09-17T18:37:28.123 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:28.123 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:28.123 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:28.123 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:28.123 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:28.123 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:28.123 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:28.123 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:28.123 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:28.123 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:28.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.114+0000 7fd15e7fc640 10 monclient: _finish_auth 0 2024-09-17T18:37:28.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.114+0000 7fd15e7fc640 10 monclient: _check_auth_tickets 2024-09-17T18:37:28.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.114+0000 7fd15e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:28.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.114+0000 7fd15e7fc640 1 -- 172.21.15.121:0/1311353358 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd150032640 con 0x7fd1600a5b90 2024-09-17T18:37:28.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.114+0000 7fd1654b1640 5 monclient: authenticate success, global_id 114162 2024-09-17T18:37:28.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.114+0000 7fd1654b1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:37:28.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.114+0000 7fd15e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:28.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.114+0000 7fd1654b1640 10 monclient: get_monmap_and_config success 2024-09-17T18:37:28.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.114+0000 7fd15e7fc640 1 -- 172.21.15.121:0/1311353358 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd150032a10 con 0x7fd1600a5b90 2024-09-17T18:37:28.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.114+0000 7fd15e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:28.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.114+0000 7fd1654b1640 4 set_mon_vals no callback set 2024-09-17T18:37:28.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.114+0000 7fd15e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:37:28.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.114+0000 7fd15e7fc640 10 monclient: dump: 2024-09-17T18:37:28.124 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:28.124 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:28.124 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:28.124 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:28.124 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:28.124 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:28.124 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:28.125 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:28.125 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:28.125 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:28.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.114+0000 7fd1654b1640 10 monclient: shutdown 2024-09-17T18:37:28.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.114+0000 7fd1654b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:28.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.114+0000 7fd1654b1640 1 -- 172.21.15.121:0/1311353358 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1600a5b90 msgr2=0x7fd1600a5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:28.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.114+0000 7fd1654b1640 1 --2- 172.21.15.121:0/1311353358 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1600a5b90 0x7fd1600a5f70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd16006ed40 tx=0x7fd15002a060 comp rx=0 tx=0).stop 2024-09-17T18:37:28.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.115+0000 7fd1654b1640 1 -- 172.21.15.121:0/1311353358 shutdown_connections 2024-09-17T18:37:28.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.115+0000 7fd1654b1640 1 --2- 172.21.15.121:0/1311353358 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1600a5b90 0x7fd1600a5f70 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:28.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.115+0000 7fd1654b1640 1 -- 172.21.15.121:0/1311353358 >> 172.21.15.121:0/1311353358 conn(0x7fd1600ab880 msgr2=0x7fd1600a7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:28.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.115+0000 7fd1654b1640 1 -- 172.21.15.121:0/1311353358 shutdown_connections 2024-09-17T18:37:28.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.115+0000 7fd1654b1640 1 -- 172.21.15.121:0/1311353358 wait complete. 2024-09-17T18:37:28.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.115+0000 7fd1654b1640 10 monclient: build_initial_monmap 2024-09-17T18:37:28.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.115+0000 7fd1654b1640 10 monclient: monmap: 2024-09-17T18:37:28.125 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:28.125 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:37:28.125 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:37:28.126 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:37:28.126 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:28.126 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:28.126 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:37:28.126 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:37:28.126 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:37:28.126 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:28.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.115+0000 7fd1654b1640 1 librados: starting msgr at 2024-09-17T18:37:28.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.115+0000 7fd1654b1640 1 librados: starting objecter 2024-09-17T18:37:28.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.115+0000 7fd1654b1640 1 Processor -- start 2024-09-17T18:37:28.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd1654b1640 1 -- start start 2024-09-17T18:37:28.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd1654b1640 1 librados: setting wanted keys 2024-09-17T18:37:28.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd1654b1640 1 librados: calling monclient init 2024-09-17T18:37:28.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd1654b1640 10 monclient: init 2024-09-17T18:37:28.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd1654b1640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:28.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd1654b1640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:37:28.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd1654b1640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd16009d830 0x7fd1601cdc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:28.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd1654b1640 10 monclient(hunting): picked mon.noname-a con 0x7fd16009d830 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:37:28.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd1654b1640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1600a5b90 0x7fd1601ce1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:28.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd1654b1640 10 monclient(hunting): picked mon.noname-b con 0x7fd1600a5b90 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:37:28.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd1654b1640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1600a7080 0x7fd1601254e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:28.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd1654b1640 10 monclient(hunting): picked mon.noname-c con 0x7fd1600a7080 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:28.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd1654b1640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:28.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd1654b1640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd160070450 con 0x7fd16009d830 2024-09-17T18:37:28.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd1654b1640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:28.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd1654b1640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd1600702d0 con 0x7fd1600a5b90 2024-09-17T18:37:28.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd1654b1640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:28.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd1654b1640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd1600a64b0 con 0x7fd1600a7080 2024-09-17T18:37:28.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd1654b1640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:28.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd1654b1640 10 monclient(hunting): authenticate will time out at 1851.832489s 2024-09-17T18:37:28.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd15effd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd16009d830 0x7fd1601cdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:28.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd15effd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd16009d830 0x7fd1601cdc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51342/0 (socket says 172.21.15.121:51342) 2024-09-17T18:37:28.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd15effd640 1 -- 172.21.15.121:0/2301926612 learned_addr learned my addr 172.21.15.121:0/2301926612 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:37:28.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd15effd640 10 monclient(hunting): get_auth_request con 0x7fd16009d830 auth_method 0 2024-09-17T18:37:28.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd15effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:28.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd15effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:28.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd15effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:28.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd15f7fe640 1 --2- 172.21.15.121:0/2301926612 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1600a7080 0x7fd1601254e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:28.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd15f7fe640 10 monclient(hunting): get_auth_request con 0x7fd1600a7080 auth_method 0 2024-09-17T18:37:28.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd15f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:28.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd15f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:28.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd15f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:28.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd15effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:28.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd15effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:28.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd15effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:28.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd15f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:28.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd15f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:28.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd15f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:28.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd157fff640 1 --2- 172.21.15.121:0/2301926612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1600a5b90 0x7fd1601ce1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:28.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd157fff640 10 monclient(hunting): get_auth_request con 0x7fd1600a5b90 auth_method 0 2024-09-17T18:37:28.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd157fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:28.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd157fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:28.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd157fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:28.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd15effd640 10 monclient(hunting): handle_auth_done global_id 114165 payload 824 2024-09-17T18:37:28.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd15effd640 1 -- 172.21.15.121:0/2301926612 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1600a7080 msgr2=0x7fd1601254e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:28.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd15effd640 1 --2- 172.21.15.121:0/2301926612 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1600a7080 0x7fd1601254e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:28.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd15effd640 1 -- 172.21.15.121:0/2301926612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1600a5b90 msgr2=0x7fd1601ce1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:28.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd15effd640 1 --2- 172.21.15.121:0/2301926612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1600a5b90 0x7fd1601ce1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:28.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd15effd640 10 monclient: _finish_hunting 0 2024-09-17T18:37:28.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd15effd640 1 monclient: found mon.noname-a 2024-09-17T18:37:28.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd15effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:28.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd15effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:37:28.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd15effd640 1 -- 172.21.15.121:0/2301926612 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd160126fa0 con 0x7fd16009d830 2024-09-17T18:37:28.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.116+0000 7fd15f7fe640 1 --2- 172.21.15.121:0/2301926612 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1600a7080 0x7fd1601254e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:37:28.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.119+0000 7fd15effd640 1 -- 172.21.15.121:0/2301926612 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd16009d830 msgr2=0x7fd1601cdc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:28.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.119+0000 7fd15effd640 1 --2- 172.21.15.121:0/2301926612 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd16009d830 0x7fd1601cdc70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd150000c00 tx=0x7fd1500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:28.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.119+0000 7fd1577fe640 10 client.?.objecter ms_handle_connect 0x7fd16009d830 2024-09-17T18:37:28.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.119+0000 7fd1577fe640 10 client.?.objecter resend_mon_ops 2024-09-17T18:37:28.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.119+0000 7fd1577fe640 1 -- 172.21.15.121:0/2301926612 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd150041070 con 0x7fd16009d830 2024-09-17T18:37:28.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.119+0000 7fd1577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:28.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.119+0000 7fd1577fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:37:28.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.119+0000 7fd1577fe640 10 monclient: dump: 2024-09-17T18:37:28.130 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:28.130 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:28.130 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:28.130 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:28.130 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:28.130 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:28.131 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:28.131 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:28.131 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:28.131 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:28.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.119+0000 7fd1577fe640 10 monclient: _finish_auth 0 2024-09-17T18:37:28.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.119+0000 7fd1577fe640 10 monclient: _check_auth_tickets 2024-09-17T18:37:28.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.119+0000 7fd1577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:28.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.119+0000 7fd1577fe640 1 -- 172.21.15.121:0/2301926612 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd15003c880 con 0x7fd16009d830 2024-09-17T18:37:28.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.119+0000 7fd1654b1640 5 monclient: authenticate success, global_id 114165 2024-09-17T18:37:28.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.119+0000 7fd1654b1640 10 monclient: _renew_subs 2024-09-17T18:37:28.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.119+0000 7fd1654b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:28.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.119+0000 7fd1654b1640 1 -- 172.21.15.121:0/2301926612 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd160128d40 con 0x7fd16009d830 2024-09-17T18:37:28.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.119+0000 7fd1577fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:28.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.119+0000 7fd1577fe640 1 -- 172.21.15.121:0/2301926612 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd150037040 con 0x7fd16009d830 2024-09-17T18:37:28.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.119+0000 7fd1577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:28.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.119+0000 7fd1577fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:37:28.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.119+0000 7fd1577fe640 10 monclient: dump: 2024-09-17T18:37:28.132 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:28.132 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:28.132 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:28.132 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:28.132 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:28.132 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:28.132 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:28.132 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:28.132 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:28.132 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:28.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.119+0000 7fd1654b1640 10 client.114165.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:37:28.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.119+0000 7fd1654b1640 10 monclient: _renew_subs 2024-09-17T18:37:28.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.119+0000 7fd1654b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:28.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.119+0000 7fd15dffb640 4 set_mon_vals no callback set 2024-09-17T18:37:28.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.119+0000 7fd1654b1640 1 -- 172.21.15.121:0/2301926612 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1601388c0 con 0x7fd16009d830 2024-09-17T18:37:28.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.119+0000 7fd1654b1640 1 librados: init done 2024-09-17T18:37:28.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.120+0000 7fd1577fe640 1 -- 172.21.15.121:0/2301926612 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fd150044070 con 0x7fd16009d830 2024-09-17T18:37:28.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.120+0000 7fd1654b1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:28.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.120+0000 7fd1654b1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:28.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.120+0000 7fd1654b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:28.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.120+0000 7fd1654b1640 1 -- 172.21.15.121:0/2301926612 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1600a6920 con 0x7fd16009d830 2024-09-17T18:37:28.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.120+0000 7fd1577fe640 1 --2- 172.21.15.121:0/2301926612 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd140040170 0x7fd140042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:28.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.120+0000 7fd1577fe640 1 -- 172.21.15.121:0/2301926612 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(679..679 src has 247..679) ==== 4685+0+0 (secure 0 0 0) 0x7fd150073cf0 con 0x7fd16009d830 2024-09-17T18:37:28.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.120+0000 7fd1577fe640 10 client.114165.objecter ms_dispatch 0x7fd160000b90 osd_map(679..679 src has 247..679) 2024-09-17T18:37:28.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.120+0000 7fd1577fe640 3 client.114165.objecter handle_osd_map got epochs [679,679] > 0 2024-09-17T18:37:28.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.120+0000 7fd1577fe640 3 client.114165.objecter handle_osd_map decoding full epoch 679 2024-09-17T18:37:28.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.120+0000 7fd1577fe640 20 client.114165.objecter dump_active .. 0 homeless 2024-09-17T18:37:28.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.120+0000 7fd157fff640 1 --2- 172.21.15.121:0/2301926612 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd140040170 0x7fd140042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:28.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.120+0000 7fd157fff640 10 monclient: get_auth_request con 0x7fd140040170 auth_method 0 2024-09-17T18:37:28.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.123+0000 7fd1577fe640 10 client.114165.objecter ms_handle_connect 0x7fd140040170 2024-09-17T18:37:28.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.123+0000 7fd157fff640 1 --2- 172.21.15.121:0/2301926612 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd140040170 0x7fd140042630 secure :-1 s=READY pgs=4101 cs=0 l=1 rev1=1 crypto rx=0x7fd160128990 tx=0x7fd148009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:28.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.124+0000 7fd1577fe640 1 -- 172.21.15.121:0/2301926612 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd150079020 con 0x7fd16009d830 2024-09-17T18:37:28.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.124+0000 7fd1577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:28.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.124+0000 7fd1577fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:37:28.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.297+0000 7fd1654b1640 1 -- 172.21.15.121:0/2301926612 --> v2:172.21.15.173:6812/2423786037 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7fd160129260 con 0x7fd140040170 2024-09-17T18:37:28.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.297+0000 7fd1577fe640 1 -- 172.21.15.121:0/2301926612 <== mgr.4105 v2:172.21.15.173:6812/2423786037 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fd160129260 con 0x7fd140040170 2024-09-17T18:37:28.299 INFO:teuthology.orchestra.run.smithi121.stderr:instructed osd(s) 0 to deep-scrub 2024-09-17T18:37:28.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.299+0000 7fd1654b1640 10 librados: watch_flush enter 2024-09-17T18:37:28.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.299+0000 7fd1654b1640 10 librados: watch_flush exit 2024-09-17T18:37:28.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.299+0000 7fd1654b1640 20 client.114165.objecter shutdown clearing up homeless session... 2024-09-17T18:37:28.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.299+0000 7fd1654b1640 10 client.114165.objecter successfully canceled tick 2024-09-17T18:37:28.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.299+0000 7fd1654b1640 1 -- 172.21.15.121:0/2301926612 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd140040170 msgr2=0x7fd140042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:28.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.299+0000 7fd1654b1640 1 --2- 172.21.15.121:0/2301926612 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd140040170 0x7fd140042630 secure :-1 s=READY pgs=4101 cs=0 l=1 rev1=1 crypto rx=0x7fd160128990 tx=0x7fd148009040 comp rx=0 tx=0).stop 2024-09-17T18:37:28.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.299+0000 7fd1654b1640 10 monclient: shutdown 2024-09-17T18:37:28.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.299+0000 7fd1654b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:28.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.299+0000 7fd1654b1640 1 -- 172.21.15.121:0/2301926612 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd16009d830 msgr2=0x7fd1601cdc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:28.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.299+0000 7fd1654b1640 1 --2- 172.21.15.121:0/2301926612 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd16009d830 0x7fd1601cdc70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd150000c00 tx=0x7fd1500046c0 comp rx=0 tx=0).stop 2024-09-17T18:37:28.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.300+0000 7fd1654b1640 1 -- 172.21.15.121:0/2301926612 shutdown_connections 2024-09-17T18:37:28.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.300+0000 7fd1654b1640 1 --2- 172.21.15.121:0/2301926612 >> v2:172.21.15.173:6812/2423786037 conn(0x7fd140040170 0x7fd140042630 unknown :-1 s=CLOSED pgs=4101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:28.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.300+0000 7fd1654b1640 1 --2- 172.21.15.121:0/2301926612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1600a5b90 0x7fd1601ce1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:28.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.300+0000 7fd1654b1640 1 --2- 172.21.15.121:0/2301926612 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1600a7080 0x7fd1601254e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:28.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.300+0000 7fd1654b1640 1 --2- 172.21.15.121:0/2301926612 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd16009d830 0x7fd1601cdc70 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:28.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.300+0000 7fd1654b1640 1 -- 172.21.15.121:0/2301926612 >> 172.21.15.121:0/2301926612 conn(0x7fd1600ab880 msgr2=0x7fd1600a32b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:28.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.300+0000 7fd1654b1640 1 -- 172.21.15.121:0/2301926612 shutdown_connections 2024-09-17T18:37:28.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.300+0000 7fd1654b1640 1 -- 172.21.15.121:0/2301926612 wait complete. 2024-09-17T18:37:28.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.300+0000 7fd1654b1640 1 librados: shutdown 2024-09-17T18:37:28.312 INFO:tasks.ceph:Scrubbing osd.1 2024-09-17T18:37:28.312 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-17T18:37:28.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.410+0000 7fb9cbdbb640 10 monclient: get_monmap_and_config 2024-09-17T18:37:28.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.410+0000 7fb9cbdbb640 10 monclient: build_initial_monmap 2024-09-17T18:37:28.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.410+0000 7fb9cbdbb640 10 monclient: monmap: 2024-09-17T18:37:28.413 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:28.413 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:28.413 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:37:28.411500+0000 2024-09-17T18:37:28.413 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:37:28.411500+0000 2024-09-17T18:37:28.413 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:28.413 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:28.413 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:37:28.413 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:37:28.413 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:37:28.413 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:37:28.413 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:37:28.414 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:28.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.411+0000 7fb9cbdbb640 1 Processor -- start 2024-09-17T18:37:28.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.411+0000 7fb9cbdbb640 1 -- start start 2024-09-17T18:37:28.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.411+0000 7fb9cbdbb640 10 monclient: init 2024-09-17T18:37:28.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.412+0000 7fb9cbdbb640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:28.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.412+0000 7fb9cbdbb640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-17T18:37:28.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.412+0000 7fb9cbdbb640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fb9c4077670 0x7fb9c40a2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:28.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.412+0000 7fb9cbdbb640 10 monclient(hunting): picked mon.noname-c con 0x7fb9c4077670 addr v2:172.21.15.173:3300/0 2024-09-17T18:37:28.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.412+0000 7fb9cbdbb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb9c40a0a70 addr v1:172.21.15.121:6789/0 2024-09-17T18:37:28.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.412+0000 7fb9cbdbb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb9c4139ac0 addr v1:172.21.15.173:6789/0 2024-09-17T18:37:28.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.412+0000 7fb9cbdbb640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9c409eca0 con 0x7fb9c40a0a70 2024-09-17T18:37:28.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.412+0000 7fb9cbdbb640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9c40a3740 con 0x7fb9c4139ac0 2024-09-17T18:37:28.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.412+0000 7fb9cbdbb640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:28.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.412+0000 7fb9cbdbb640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fb9c4077a50 con 0x7fb9c4077670 2024-09-17T18:37:28.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.412+0000 7fb9cbdbb640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:28.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.412+0000 7fb9cbdbb640 10 monclient(hunting): authenticate will time out at 1852.128248s 2024-09-17T18:37:28.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.412+0000 7fb9c9356640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fb9c40a0a70 0x7fb9c40a0e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:47376/0 (socket says 172.21.15.121:47376) 2024-09-17T18:37:28.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.412+0000 7fb9c9356640 1 -- 172.21.15.121:0/1402098427 learned_addr learned my addr 172.21.15.121:0/1402098427 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:37:28.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.412+0000 7fb9c9b57640 1 --2- 172.21.15.121:0/1402098427 >> v2:172.21.15.173:3300/0 conn(0x7fb9c4077670 0x7fb9c40a2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:28.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.412+0000 7fb9c9b57640 10 monclient(hunting): get_auth_request con 0x7fb9c4077670 auth_method 0 2024-09-17T18:37:28.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.412+0000 7fb9c9b57640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:28.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.412+0000 7fb9c9b57640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:28.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.412+0000 7fb9c9b57640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:28.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.412+0000 7fb9c9b57640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:28.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.412+0000 7fb9c9b57640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:28.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.412+0000 7fb9c9b57640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:28.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.412+0000 7fb9c9b57640 10 monclient(hunting): handle_auth_done global_id 84349 payload 274 2024-09-17T18:37:28.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.412+0000 7fb9c9b57640 1 -- 172.21.15.121:0/1402098427 >> v1:172.21.15.173:6789/0 conn(0x7fb9c4139ac0 legacy=0x7fb9c413beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:28.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.412+0000 7fb9c9b57640 1 -- 172.21.15.121:0/1402098427 >> v1:172.21.15.121:6789/0 conn(0x7fb9c40a0a70 legacy=0x7fb9c40a0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:28.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.412+0000 7fb9c9b57640 10 monclient: _finish_hunting 0 2024-09-17T18:37:28.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.412+0000 7fb9c9b57640 1 monclient: found mon.noname-c 2024-09-17T18:37:28.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.412+0000 7fb9c9b57640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:28.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.413+0000 7fb9c9b57640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.173:3300/0 2024-09-17T18:37:28.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.413+0000 7fb9c9b57640 1 -- 172.21.15.121:0/1402098427 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9c409ef80 con 0x7fb9c4077670 2024-09-17T18:37:28.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.415+0000 7fb9c9b57640 1 -- 172.21.15.121:0/1402098427 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9c4077670 msgr2=0x7fb9c40a2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:28.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.415+0000 7fb9c9b57640 1 --2- 172.21.15.121:0/1402098427 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9c4077670 0x7fb9c40a2200 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb9c00097d0 tx=0x7fb9c002a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=851d963f1e71ff4c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:28.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.416+0000 7fb9c8b55640 1 -- 172.21.15.121:0/1402098427 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c0037070 con 0x7fb9c4077670 2024-09-17T18:37:28.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.416+0000 7fb9c8b55640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:28.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.416+0000 7fb9c8b55640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:37:28.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.416+0000 7fb9c8b55640 10 monclient: dump: 2024-09-17T18:37:28.425 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:28.425 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:28.425 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:28.425 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:28.425 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:28.425 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:28.425 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:28.425 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:28.425 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:28.425 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:28.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.416+0000 7fb9c8b55640 10 monclient: _finish_auth 0 2024-09-17T18:37:28.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.416+0000 7fb9c8b55640 10 monclient: _check_auth_tickets 2024-09-17T18:37:28.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.416+0000 7fb9c8b55640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:28.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.416+0000 7fb9c8b55640 1 -- 172.21.15.121:0/1402098427 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9c0032440 con 0x7fb9c4077670 2024-09-17T18:37:28.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.416+0000 7fb9c8b55640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:28.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.416+0000 7fb9c8b55640 1 -- 172.21.15.121:0/1402098427 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c0032810 con 0x7fb9c4077670 2024-09-17T18:37:28.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.416+0000 7fb9c8b55640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:28.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.416+0000 7fb9c8b55640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:37:28.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.416+0000 7fb9c8b55640 10 monclient: dump: 2024-09-17T18:37:28.426 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:28.426 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:28.426 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:28.426 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:28.426 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:28.426 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:28.426 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:28.426 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:28.427 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:28.427 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:28.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.416+0000 7fb9cbdbb640 5 monclient: authenticate success, global_id 84349 2024-09-17T18:37:28.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.416+0000 7fb9cbdbb640 10 monclient: get_monmap_and_config success 2024-09-17T18:37:28.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.416+0000 7fb9cbdbb640 4 set_mon_vals no callback set 2024-09-17T18:37:28.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.416+0000 7fb9cbdbb640 10 monclient: shutdown 2024-09-17T18:37:28.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.416+0000 7fb9cbdbb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:28.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.416+0000 7fb9cbdbb640 1 -- 172.21.15.121:0/1402098427 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9c4077670 msgr2=0x7fb9c40a2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:28.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.416+0000 7fb9cbdbb640 1 --2- 172.21.15.121:0/1402098427 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9c4077670 0x7fb9c40a2200 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb9c00097d0 tx=0x7fb9c002a000 comp rx=0 tx=0).stop 2024-09-17T18:37:28.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.416+0000 7fb9cbdbb640 1 -- 172.21.15.121:0/1402098427 shutdown_connections 2024-09-17T18:37:28.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.416+0000 7fb9cbdbb640 1 --2- 172.21.15.121:0/1402098427 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9c4077670 0x7fb9c40a2200 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:28.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.416+0000 7fb9cbdbb640 1 -- 172.21.15.121:0/1402098427 >> 172.21.15.121:0/1402098427 conn(0x7fb9c40a3a70 msgr2=0x7fb9c40a3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:28.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.417+0000 7fb9cbdbb640 1 -- 172.21.15.121:0/1402098427 shutdown_connections 2024-09-17T18:37:28.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.417+0000 7fb9cbdbb640 1 -- 172.21.15.121:0/1402098427 wait complete. 2024-09-17T18:37:28.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.417+0000 7fb9cbdbb640 10 monclient: build_initial_monmap 2024-09-17T18:37:28.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.417+0000 7fb9cbdbb640 10 monclient: monmap: 2024-09-17T18:37:28.428 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:28.428 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:37:28.428 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:37:28.428 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:37:28.428 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:28.428 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:28.428 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:37:28.428 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:37:28.428 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:37:28.428 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:28.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.417+0000 7fb9cbdbb640 1 librados: starting msgr at 2024-09-17T18:37:28.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.417+0000 7fb9cbdbb640 1 librados: starting objecter 2024-09-17T18:37:28.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.417+0000 7fb9cbdbb640 1 Processor -- start 2024-09-17T18:37:28.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.417+0000 7fb9cbdbb640 1 -- start start 2024-09-17T18:37:28.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.417+0000 7fb9cbdbb640 1 librados: setting wanted keys 2024-09-17T18:37:28.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.417+0000 7fb9cbdbb640 1 librados: calling monclient init 2024-09-17T18:37:28.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.417+0000 7fb9cbdbb640 10 monclient: init 2024-09-17T18:37:28.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9cbdbb640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:28.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9cbdbb640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T18:37:28.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9cbdbb640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9c4077670 0x7fb9c411e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:28.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9cbdbb640 10 monclient(hunting): picked mon.noname-c con 0x7fb9c4077670 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:28.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9cbdbb640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb9c40a0a70 0x7fb9c409ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:28.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9cbdbb640 10 monclient(hunting): picked mon.noname-a con 0x7fb9c40a0a70 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:37:28.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9cbdbb640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9c4139ac0 0x7fb9c409cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:28.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9cbdbb640 10 monclient(hunting): picked mon.noname-b con 0x7fb9c4139ac0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:37:28.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9cbdbb640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:28.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9cbdbb640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c413d5d0 con 0x7fb9c40a0a70 2024-09-17T18:37:28.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9cbdbb640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:28.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9cbdbb640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c413d750 con 0x7fb9c4139ac0 2024-09-17T18:37:28.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9cbdbb640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:28.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9cbdbb640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb9c413d8d0 con 0x7fb9c4077670 2024-09-17T18:37:28.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9cbdbb640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:28.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9cbdbb640 10 monclient(hunting): authenticate will time out at 1852.134311s 2024-09-17T18:37:28.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9c9356640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb9c40a0a70 0x7fb9c409ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:28.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9c9b57640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9c4077670 0x7fb9c411e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:28.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9c9356640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb9c40a0a70 0x7fb9c409ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51358/0 (socket says 172.21.15.121:51358) 2024-09-17T18:37:28.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9c9356640 1 -- 172.21.15.121:0/3450499268 learned_addr learned my addr 172.21.15.121:0/3450499268 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:37:28.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9c9356640 10 monclient(hunting): get_auth_request con 0x7fb9c40a0a70 auth_method 0 2024-09-17T18:37:28.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9c9356640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:28.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9c9356640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:28.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9c9356640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:28.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9c9b57640 10 monclient(hunting): get_auth_request con 0x7fb9c4077670 auth_method 0 2024-09-17T18:37:28.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9c9b57640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:28.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9c9b57640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:28.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9c9b57640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:28.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9c9356640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:28.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9c9356640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:28.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9c9356640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:28.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9c9b57640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:28.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9c9b57640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:28.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9c9b57640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:28.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9ca358640 1 --2- 172.21.15.121:0/3450499268 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9c4139ac0 0x7fb9c409cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:28.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9ca358640 10 monclient(hunting): get_auth_request con 0x7fb9c4139ac0 auth_method 0 2024-09-17T18:37:28.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9ca358640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:28.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9ca358640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:28.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9ca358640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:28.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9ca358640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:28.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9ca358640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:28.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9ca358640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:28.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9c9356640 10 monclient(hunting): handle_auth_done global_id 114168 payload 824 2024-09-17T18:37:28.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9c9356640 1 -- 172.21.15.121:0/3450499268 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9c4077670 msgr2=0x7fb9c411e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:28.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9c9356640 1 --2- 172.21.15.121:0/3450499268 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9c4077670 0x7fb9c411e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:28.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9c9356640 1 -- 172.21.15.121:0/3450499268 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9c4139ac0 msgr2=0x7fb9c409cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:28.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9c9356640 1 --2- 172.21.15.121:0/3450499268 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9c4139ac0 0x7fb9c409cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:28.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9c9356640 10 monclient: _finish_hunting 0 2024-09-17T18:37:28.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9c9356640 1 monclient: found mon.noname-a 2024-09-17T18:37:28.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9c9356640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:28.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9c9356640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:37:28.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.418+0000 7fb9c9356640 1 -- 172.21.15.121:0/3450499268 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9c411ecb0 con 0x7fb9c40a0a70 2024-09-17T18:37:28.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.420+0000 7fb9c9356640 1 -- 172.21.15.121:0/3450499268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb9c40a0a70 msgr2=0x7fb9c409ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:28.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.420+0000 7fb9c9356640 1 --2- 172.21.15.121:0/3450499268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb9c40a0a70 0x7fb9c409ca60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb9b400c460 tx=0x7fb9b400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:28.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.420+0000 7fb9ba7fc640 10 client.?.objecter ms_handle_connect 0x7fb9c40a0a70 2024-09-17T18:37:28.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.420+0000 7fb9ba7fc640 10 client.?.objecter resend_mon_ops 2024-09-17T18:37:28.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.421+0000 7fb9ba7fc640 1 -- 172.21.15.121:0/3450499268 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b4017070 con 0x7fb9c40a0a70 2024-09-17T18:37:28.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.421+0000 7fb9ba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:28.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.421+0000 7fb9ba7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:37:28.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.421+0000 7fb9ba7fc640 10 monclient: dump: 2024-09-17T18:37:28.433 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:28.433 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:28.433 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:28.433 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:28.433 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:28.433 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:28.433 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:28.433 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:28.433 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:28.433 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:28.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.421+0000 7fb9ba7fc640 10 monclient: _finish_auth 0 2024-09-17T18:37:28.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.421+0000 7fb9ba7fc640 10 monclient: _check_auth_tickets 2024-09-17T18:37:28.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.421+0000 7fb9ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:28.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.421+0000 7fb9cbdbb640 5 monclient: authenticate success, global_id 114168 2024-09-17T18:37:28.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.421+0000 7fb9ba7fc640 1 -- 172.21.15.121:0/3450499268 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9b40070d0 con 0x7fb9c40a0a70 2024-09-17T18:37:28.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.421+0000 7fb9cbdbb640 10 monclient: _renew_subs 2024-09-17T18:37:28.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.421+0000 7fb9cbdbb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:28.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.421+0000 7fb9cbdbb640 1 -- 172.21.15.121:0/3450499268 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9c411fb20 con 0x7fb9c40a0a70 2024-09-17T18:37:28.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.421+0000 7fb9ba7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:28.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.421+0000 7fb9ba7fc640 1 -- 172.21.15.121:0/3450499268 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b40074c0 con 0x7fb9c40a0a70 2024-09-17T18:37:28.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.421+0000 7fb9ba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:28.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.421+0000 7fb9ba7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:37:28.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.421+0000 7fb9ba7fc640 10 monclient: dump: 2024-09-17T18:37:28.434 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:28.435 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:28.435 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:28.435 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:28.435 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:28.435 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:28.435 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:28.435 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:28.435 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:28.435 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:28.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.421+0000 7fb9cbdbb640 10 client.114168.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:37:28.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.421+0000 7fb9bbfff640 4 set_mon_vals no callback set 2024-09-17T18:37:28.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.421+0000 7fb9cbdbb640 10 monclient: _renew_subs 2024-09-17T18:37:28.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.421+0000 7fb9cbdbb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:28.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.421+0000 7fb9cbdbb640 1 -- 172.21.15.121:0/3450499268 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9c4135640 con 0x7fb9c40a0a70 2024-09-17T18:37:28.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.421+0000 7fb9cbdbb640 1 librados: init done 2024-09-17T18:37:28.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.421+0000 7fb9cbdbb640 10 client.114168.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:28.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.421+0000 7fb9cbdbb640 15 client.114168.objecter _session_command_op_assign -1 1 2024-09-17T18:37:28.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.421+0000 7fb9cbdbb640 20 client.114168.objecter _get_session osd=-1 returning homeless 2024-09-17T18:37:28.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.421+0000 7fb9cbdbb640 10 client.114168.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:37:28.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.421+0000 7fb9cbdbb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:28.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.421+0000 7fb9cbdbb640 1 -- 172.21.15.121:0/3450499268 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb9c40a2280 con 0x7fb9c40a0a70 2024-09-17T18:37:28.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.422+0000 7fb9ba7fc640 1 -- 172.21.15.121:0/3450499268 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fb9b401a070 con 0x7fb9c40a0a70 2024-09-17T18:37:28.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.422+0000 7fb9ba7fc640 1 --2- 172.21.15.121:0/3450499268 >> v2:172.21.15.173:6812/2423786037 conn(0x7fb9a80401a0 0x7fb9a8042660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:28.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.422+0000 7fb9ba7fc640 1 -- 172.21.15.121:0/3450499268 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(679..679 src has 247..679) ==== 4685+0+0 (secure 0 0 0) 0x7fb9b4052960 con 0x7fb9c40a0a70 2024-09-17T18:37:28.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.422+0000 7fb9ba7fc640 10 client.114168.objecter ms_dispatch 0x7fb9c4000b90 osd_map(679..679 src has 247..679) 2024-09-17T18:37:28.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.422+0000 7fb9ba7fc640 3 client.114168.objecter handle_osd_map got epochs [679,679] > 0 2024-09-17T18:37:28.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.422+0000 7fb9ba7fc640 3 client.114168.objecter handle_osd_map decoding full epoch 679 2024-09-17T18:37:28.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.422+0000 7fb9ba7fc640 10 client.114168.objecter checking command 1 2024-09-17T18:37:28.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.422+0000 7fb9ba7fc640 1 --2- 172.21.15.121:0/3450499268 >> v2:172.21.15.121:6808/196419920 conn(0x7fb9a8047660 0x7fb9a8049b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:28.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.422+0000 7fb9ba7fc640 20 client.114168.objecter _get_session s=0x7fb9a8046fb0 osd=1 3 2024-09-17T18:37:28.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.422+0000 7fb9ba7fc640 20 client.114168.objecter put_session s=0x7fb9a8046fb0 osd=1 3 2024-09-17T18:37:28.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.422+0000 7fb9ba7fc640 15 client.114168.objecter _session_command_op_remove -1 1 2024-09-17T18:37:28.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.422+0000 7fb9ba7fc640 20 client.114168.objecter _get_session s=0x7fb9a8046fb0 osd=1 3 2024-09-17T18:37:28.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.422+0000 7fb9ba7fc640 20 client.114168.objecter get_session s=0x7fb9a8046fb0 osd=1 3 2024-09-17T18:37:28.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.422+0000 7fb9ba7fc640 15 client.114168.objecter _session_command_op_assign 1 1 2024-09-17T18:37:28.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.422+0000 7fb9ba7fc640 20 client.114168.objecter put_session s=0x7fb9a8046fb0 osd=1 4 2024-09-17T18:37:28.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.422+0000 7fb9ba7fc640 10 client.114168.objecter _send_command 1 2024-09-17T18:37:28.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.422+0000 7fb9ba7fc640 1 -- 172.21.15.121:0/3450499268 --> v2:172.21.15.121:6808/196419920 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb9b400ae50 con 0x7fb9a8047660 2024-09-17T18:37:28.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.422+0000 7fb9ba7fc640 20 client.114168.objecter dump_active .. 0 homeless 2024-09-17T18:37:28.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.422+0000 7fb9ba7fc640 1 -- 172.21.15.121:0/3450499268 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_get_version_reply(handle=1 version=679) ==== 24+0+0 (secure 0 0 0) 0x7fb9b40572a0 con 0x7fb9c40a0a70 2024-09-17T18:37:28.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.422+0000 7fb9ba7fc640 10 monclient: handle_get_version_reply finishing 1 version 679 2024-09-17T18:37:28.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.422+0000 7fb9c9b57640 1 --2- 172.21.15.121:0/3450499268 >> v2:172.21.15.173:6812/2423786037 conn(0x7fb9a80401a0 0x7fb9a8042660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:28.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.422+0000 7fb9c9b57640 10 monclient: get_auth_request con 0x7fb9a80401a0 auth_method 0 2024-09-17T18:37:28.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.422+0000 7fb9ca358640 1 --2- 172.21.15.121:0/3450499268 >> v2:172.21.15.121:6808/196419920 conn(0x7fb9a8047660 0x7fb9a8049b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:28.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.422+0000 7fb9ca358640 10 monclient: get_auth_request con 0x7fb9a8047660 auth_method 0 2024-09-17T18:37:28.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.425+0000 7fb9ba7fc640 10 client.114168.objecter ms_handle_connect 0x7fb9a80401a0 2024-09-17T18:37:28.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.425+0000 7fb9c9b57640 1 --2- 172.21.15.121:0/3450499268 >> v2:172.21.15.173:6812/2423786037 conn(0x7fb9a80401a0 0x7fb9a8042660 secure :-1 s=READY pgs=4102 cs=0 l=1 rev1=1 crypto rx=0x7fb9c00097a0 tx=0x7fb9c002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:28.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.427+0000 7fb9ba7fc640 10 client.114168.objecter ms_handle_connect 0x7fb9a8047660 2024-09-17T18:37:28.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.427+0000 7fb9ca358640 1 --2- 172.21.15.121:0/3450499268 >> v2:172.21.15.121:6808/196419920 conn(0x7fb9a8047660 0x7fb9a8049b00 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:28.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.427+0000 7fb9ba7fc640 1 -- 172.21.15.121:0/3450499268 <== osd.1 v2:172.21.15.121:6808/196419920 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb9b400ae50 con 0x7fb9a8047660 2024-09-17T18:37:28.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.427+0000 7fb9ba7fc640 10 client.114168.objecter ms_dispatch 0x7fb9c4000b90 command_reply(tid 1: 0 ) 2024-09-17T18:37:28.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.427+0000 7fb9ba7fc640 10 client.114168.objecter _finish_command 1 = system:0 2024-09-17T18:37:28.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.427+0000 7fb9ba7fc640 20 client.114168.objecter put_session s=0x7fb9a8046fb0 osd=1 4 2024-09-17T18:37:28.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.427+0000 7fb9ba7fc640 15 client.114168.objecter _session_command_op_remove 1 1 2024-09-17T18:37:28.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.439+0000 7fb9cbdbb640 10 client.114168.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-17T18:37:28.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.439+0000 7fb9cbdbb640 15 client.114168.objecter _session_command_op_assign -1 2 2024-09-17T18:37:28.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.439+0000 7fb9cbdbb640 20 client.114168.objecter _get_session s=0x7fb9a8046fb0 osd=1 3 2024-09-17T18:37:28.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.439+0000 7fb9cbdbb640 20 client.114168.objecter put_session s=0x7fb9a8046fb0 osd=1 3 2024-09-17T18:37:28.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.439+0000 7fb9cbdbb640 20 client.114168.objecter _get_session s=0x7fb9a8046fb0 osd=1 3 2024-09-17T18:37:28.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.439+0000 7fb9cbdbb640 15 client.114168.objecter _session_command_op_remove -1 2 2024-09-17T18:37:28.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.439+0000 7fb9cbdbb640 20 client.114168.objecter get_session s=0x7fb9a8046fb0 osd=1 3 2024-09-17T18:37:28.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.439+0000 7fb9cbdbb640 15 client.114168.objecter _session_command_op_assign 1 2 2024-09-17T18:37:28.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.439+0000 7fb9cbdbb640 20 client.114168.objecter put_session s=0x7fb9a8046fb0 osd=1 4 2024-09-17T18:37:28.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.439+0000 7fb9cbdbb640 10 client.114168.objecter _send_command 2 2024-09-17T18:37:28.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.440+0000 7fb9cbdbb640 1 -- 172.21.15.121:0/3450499268 --> v2:172.21.15.121:6808/196419920 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fb9c411d7c0 con 0x7fb9a8047660 2024-09-17T18:37:28.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.440+0000 7fb9ba7fc640 1 -- 172.21.15.121:0/3450499268 <== osd.1 v2:172.21.15.121:6808/196419920 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fb9c411d7c0 con 0x7fb9a8047660 2024-09-17T18:37:28.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.440+0000 7fb9ba7fc640 10 client.114168.objecter ms_dispatch 0x7fb9c4000b90 command_reply(tid 2: 0 ) 2024-09-17T18:37:28.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.440+0000 7fb9ba7fc640 10 client.114168.objecter _finish_command 2 = system:0 2024-09-17T18:37:28.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.440+0000 7fb9ba7fc640 20 client.114168.objecter put_session s=0x7fb9a8046fb0 osd=1 4 2024-09-17T18:37:28.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.440+0000 7fb9ba7fc640 15 client.114168.objecter _session_command_op_remove 1 2 2024-09-17T18:37:28.443 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:37:28.443 INFO:teuthology.orchestra.run.smithi121.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T18:37:28.443 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:37:28.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.440+0000 7fb9cbdbb640 10 librados: watch_flush enter 2024-09-17T18:37:28.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.440+0000 7fb9cbdbb640 10 librados: watch_flush exit 2024-09-17T18:37:28.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.440+0000 7fb9cbdbb640 10 client.114168.objecter close_session for osd.1 2024-09-17T18:37:28.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.440+0000 7fb9cbdbb640 1 -- 172.21.15.121:0/3450499268 >> v2:172.21.15.121:6808/196419920 conn(0x7fb9a8047660 msgr2=0x7fb9a8049b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:28.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.440+0000 7fb9cbdbb640 1 --2- 172.21.15.121:0/3450499268 >> v2:172.21.15.121:6808/196419920 conn(0x7fb9a8047660 0x7fb9a8049b00 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:28.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.440+0000 7fb9cbdbb640 20 client.114168.objecter put_session s=0x7fb9a8046fb0 osd=1 1 2024-09-17T18:37:28.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.440+0000 7fb9cbdbb640 20 client.114168.objecter shutdown clearing up homeless session... 2024-09-17T18:37:28.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.440+0000 7fb9cbdbb640 10 client.114168.objecter successfully canceled tick 2024-09-17T18:37:28.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.440+0000 7fb9cbdbb640 1 -- 172.21.15.121:0/3450499268 >> v2:172.21.15.173:6812/2423786037 conn(0x7fb9a80401a0 msgr2=0x7fb9a8042660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:28.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.440+0000 7fb9cbdbb640 1 --2- 172.21.15.121:0/3450499268 >> v2:172.21.15.173:6812/2423786037 conn(0x7fb9a80401a0 0x7fb9a8042660 secure :-1 s=READY pgs=4102 cs=0 l=1 rev1=1 crypto rx=0x7fb9c00097a0 tx=0x7fb9c002e000 comp rx=0 tx=0).stop 2024-09-17T18:37:28.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.440+0000 7fb9cbdbb640 10 monclient: shutdown 2024-09-17T18:37:28.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.440+0000 7fb9cbdbb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:28.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.440+0000 7fb9cbdbb640 1 -- 172.21.15.121:0/3450499268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb9c40a0a70 msgr2=0x7fb9c409ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:28.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.440+0000 7fb9cbdbb640 1 --2- 172.21.15.121:0/3450499268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb9c40a0a70 0x7fb9c409ca60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb9b400c460 tx=0x7fb9b400c920 comp rx=0 tx=0).stop 2024-09-17T18:37:28.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.440+0000 7fb9ca358640 1 -- 172.21.15.121:0/3450499268 reap_dead start 2024-09-17T18:37:28.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.440+0000 7fb9cbdbb640 1 -- 172.21.15.121:0/3450499268 shutdown_connections 2024-09-17T18:37:28.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.440+0000 7fb9cbdbb640 1 -- 172.21.15.121:0/3450499268 >> 172.21.15.121:0/3450499268 conn(0x7fb9c40a3a70 msgr2=0x7fb9c413bf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:28.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.440+0000 7fb9cbdbb640 1 -- 172.21.15.121:0/3450499268 shutdown_connections 2024-09-17T18:37:28.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.440+0000 7fb9cbdbb640 1 -- 172.21.15.121:0/3450499268 wait complete. 2024-09-17T18:37:28.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.440+0000 7fb9cbdbb640 1 librados: shutdown 2024-09-17T18:37:28.452 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-17T18:37:28.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.548+0000 7f3554bec640 10 monclient: get_monmap_and_config 2024-09-17T18:37:28.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.549+0000 7f3554bec640 10 monclient: build_initial_monmap 2024-09-17T18:37:28.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.549+0000 7f3554bec640 10 monclient: monmap: 2024-09-17T18:37:28.551 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:28.551 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:28.551 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:37:28.549608+0000 2024-09-17T18:37:28.551 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:37:28.549608+0000 2024-09-17T18:37:28.551 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:28.551 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:28.551 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:37:28.551 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:37:28.551 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:37:28.551 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:37:28.551 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:37:28.551 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:28.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.549+0000 7f3554bec640 1 Processor -- start 2024-09-17T18:37:28.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.549+0000 7f3554bec640 1 -- start start 2024-09-17T18:37:28.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.549+0000 7f3554bec640 10 monclient: init 2024-09-17T18:37:28.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.550+0000 7f3554bec640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:28.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.550+0000 7f3554bec640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-17T18:37:28.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.550+0000 7f3554bec640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3550073900 addr v1:172.21.15.121:6789/0 2024-09-17T18:37:28.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.550+0000 7f3554bec640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f355013b6f0 0x7f355013dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:28.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.550+0000 7f3554bec640 10 monclient(hunting): picked mon.noname-a con 0x7f355013b6f0 addr v2:172.21.15.121:3300/0 2024-09-17T18:37:28.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.550+0000 7f3554bec640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f355013e020 addr v1:172.21.15.173:6789/0 2024-09-17T18:37:28.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.550+0000 7f3554bec640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35501388b0 con 0x7f3550073900 2024-09-17T18:37:28.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.550+0000 7f3554bec640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35500701a0 con 0x7f355013e020 2024-09-17T18:37:28.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.550+0000 7f3554bec640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:28.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.550+0000 7f3554bec640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f355006fbe0 con 0x7f355013b6f0 2024-09-17T18:37:28.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.550+0000 7f3554bec640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:28.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.550+0000 7f3554bec640 10 monclient(hunting): authenticate will time out at 1852.266366s 2024-09-17T18:37:28.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.550+0000 7f354dd9b640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f355013b6f0 0x7f355013dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:28.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.550+0000 7f354e59c640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f3550073900 0x7f3550073ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:47390/0 (socket says 172.21.15.121:47390) 2024-09-17T18:37:28.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.550+0000 7f354dd9b640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f355013b6f0 0x7f355013dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51384/0 (socket says 172.21.15.121:51384) 2024-09-17T18:37:28.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.550+0000 7f354e59c640 1 -- 172.21.15.121:0/3491433200 learned_addr learned my addr 172.21.15.121:0/3491433200 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:37:28.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.550+0000 7f354dd9b640 10 monclient(hunting): get_auth_request con 0x7f355013b6f0 auth_method 0 2024-09-17T18:37:28.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.550+0000 7f354dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:28.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.550+0000 7f354dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:28.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.550+0000 7f354dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:28.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.552+0000 7f354e59c640 1 -- 172.21.15.121:0/3491433200 >> v1:172.21.15.121:6789/0 conn(0x7f3550073900 legacy=0x7f3550073ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:28.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.552+0000 7f354dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:28.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.552+0000 7f354dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:28.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.552+0000 7f354dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:28.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.552+0000 7f354dd9b640 10 monclient(hunting): handle_auth_done global_id 114171 payload 274 2024-09-17T18:37:28.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.552+0000 7f354dd9b640 1 -- 172.21.15.121:0/3491433200 >> v1:172.21.15.173:6789/0 conn(0x7f355013e020 legacy=0x7f3550140410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:28.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.552+0000 7f354dd9b640 1 -- 172.21.15.121:0/3491433200 >> v1:172.21.15.121:6789/0 conn(0x7f3550073900 legacy=0x7f3550073ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:28.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.552+0000 7f354dd9b640 10 monclient: _finish_hunting 0 2024-09-17T18:37:28.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.552+0000 7f354dd9b640 1 monclient: found mon.noname-a 2024-09-17T18:37:28.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.552+0000 7f354dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:28.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.552+0000 7f354dd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:37:28.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.552+0000 7f354dd9b640 1 -- 172.21.15.121:0/3491433200 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3550077290 con 0x7f355013b6f0 2024-09-17T18:37:28.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.555+0000 7f354dd9b640 1 -- 172.21.15.121:0/3491433200 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f355013b6f0 msgr2=0x7f355013dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:28.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.555+0000 7f354dd9b640 1 --2- 172.21.15.121:0/3491433200 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f355013b6f0 0x7f355013dae0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3544009820 tx=0x7f354402a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=598eaa6664711797 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:28.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.555+0000 7f354d59a640 1 -- 172.21.15.121:0/3491433200 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3544037070 con 0x7f355013b6f0 2024-09-17T18:37:28.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.555+0000 7f354d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:28.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.555+0000 7f354d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:37:28.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.555+0000 7f354d59a640 10 monclient: dump: 2024-09-17T18:37:28.558 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:28.558 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:28.558 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:28.558 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:28.558 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:28.559 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:28.559 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:28.559 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:28.559 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:28.559 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:28.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.555+0000 7f354d59a640 10 monclient: _finish_auth 0 2024-09-17T18:37:28.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.555+0000 7f354d59a640 10 monclient: _check_auth_tickets 2024-09-17T18:37:28.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.555+0000 7f354d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:28.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.555+0000 7f354d59a640 1 -- 172.21.15.121:0/3491433200 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3544032640 con 0x7f355013b6f0 2024-09-17T18:37:28.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.555+0000 7f3554bec640 5 monclient: authenticate success, global_id 114171 2024-09-17T18:37:28.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.555+0000 7f3554bec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:37:28.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.555+0000 7f354d59a640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:28.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.555+0000 7f3554bec640 10 monclient: get_monmap_and_config success 2024-09-17T18:37:28.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.555+0000 7f354d59a640 1 -- 172.21.15.121:0/3491433200 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3544032a10 con 0x7f355013b6f0 2024-09-17T18:37:28.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.555+0000 7f354d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:28.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.555+0000 7f3554bec640 4 set_mon_vals no callback set 2024-09-17T18:37:28.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.555+0000 7f354d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:37:28.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.555+0000 7f354d59a640 10 monclient: dump: 2024-09-17T18:37:28.560 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:28.560 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:28.560 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:28.560 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:28.560 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:28.560 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:28.560 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:28.560 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:28.560 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:28.560 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:28.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.556+0000 7f3554bec640 10 monclient: shutdown 2024-09-17T18:37:28.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.556+0000 7f3554bec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:28.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.556+0000 7f3554bec640 1 -- 172.21.15.121:0/3491433200 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f355013b6f0 msgr2=0x7f355013dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:28.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.556+0000 7f3554bec640 1 --2- 172.21.15.121:0/3491433200 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f355013b6f0 0x7f355013dae0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3544009820 tx=0x7f354402a100 comp rx=0 tx=0).stop 2024-09-17T18:37:28.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.556+0000 7f3554bec640 1 -- 172.21.15.121:0/3491433200 shutdown_connections 2024-09-17T18:37:28.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.556+0000 7f3554bec640 1 --2- 172.21.15.121:0/3491433200 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f355013b6f0 0x7f355013dae0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:28.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.556+0000 7f3554bec640 1 -- 172.21.15.121:0/3491433200 >> 172.21.15.121:0/3491433200 conn(0x7f35500a49c0 msgr2=0x7f35500768d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:28.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.556+0000 7f3554bec640 1 -- 172.21.15.121:0/3491433200 shutdown_connections 2024-09-17T18:37:28.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.556+0000 7f3554bec640 1 -- 172.21.15.121:0/3491433200 wait complete. 2024-09-17T18:37:28.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.557+0000 7f3554bec640 10 monclient: build_initial_monmap 2024-09-17T18:37:28.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.557+0000 7f3554bec640 10 monclient: monmap: 2024-09-17T18:37:28.568 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:28.568 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:37:28.568 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:37:28.568 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:37:28.568 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:28.569 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:28.569 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:37:28.569 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:37:28.569 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:37:28.569 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:28.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.557+0000 7f3554bec640 1 librados: starting msgr at 2024-09-17T18:37:28.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.557+0000 7f3554bec640 1 librados: starting objecter 2024-09-17T18:37:28.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.557+0000 7f3554bec640 1 Processor -- start 2024-09-17T18:37:28.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.557+0000 7f3554bec640 1 -- start start 2024-09-17T18:37:28.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.557+0000 7f3554bec640 1 librados: setting wanted keys 2024-09-17T18:37:28.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.557+0000 7f3554bec640 1 librados: calling monclient init 2024-09-17T18:37:28.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.557+0000 7f3554bec640 10 monclient: init 2024-09-17T18:37:28.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.557+0000 7f3554bec640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:28.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.557+0000 7f3554bec640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T18:37:28.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.557+0000 7f3554bec640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3550073900 0x7f35501ca1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:28.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.557+0000 7f3554bec640 10 monclient(hunting): picked mon.noname-c con 0x7f3550073900 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:28.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.557+0000 7f3554bec640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f355013b6f0 0x7f3550135b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:28.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.557+0000 7f3554bec640 10 monclient(hunting): picked mon.noname-b con 0x7f355013b6f0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:37:28.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.557+0000 7f3554bec640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f355013e020 0x7f3550136080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:28.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.557+0000 7f3554bec640 10 monclient(hunting): picked mon.noname-a con 0x7f355013e020 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:37:28.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.557+0000 7f3554bec640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:28.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.557+0000 7f3554bec640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3550077770 con 0x7f355013e020 2024-09-17T18:37:28.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.557+0000 7f3554bec640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:28.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.557+0000 7f3554bec640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f35500775f0 con 0x7f355013b6f0 2024-09-17T18:37:28.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.557+0000 7f3554bec640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:28.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.557+0000 7f3554bec640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3550074450 con 0x7f3550073900 2024-09-17T18:37:28.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.557+0000 7f3554bec640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:28.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.557+0000 7f3554bec640 10 monclient(hunting): authenticate will time out at 1852.273949s 2024-09-17T18:37:28.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.558+0000 7f354e59c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3550073900 0x7f35501ca1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:28.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.558+0000 7f354ed9d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f355013e020 0x7f3550136080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:28.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.558+0000 7f354e59c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3550073900 0x7f35501ca1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:36878/0 (socket says 172.21.15.121:36878) 2024-09-17T18:37:28.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.558+0000 7f354e59c640 1 -- 172.21.15.121:0/2069300875 learned_addr learned my addr 172.21.15.121:0/2069300875 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:37:28.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.558+0000 7f354e59c640 10 monclient(hunting): get_auth_request con 0x7f3550073900 auth_method 0 2024-09-17T18:37:28.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.558+0000 7f354e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:28.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.558+0000 7f354e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:28.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.558+0000 7f354e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:28.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.558+0000 7f354ed9d640 10 monclient(hunting): get_auth_request con 0x7f355013e020 auth_method 0 2024-09-17T18:37:28.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.558+0000 7f354ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:28.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.558+0000 7f354ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:28.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.558+0000 7f354ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:28.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.558+0000 7f354e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:28.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.558+0000 7f354e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:28.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.558+0000 7f354e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:28.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.558+0000 7f354ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:28.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.558+0000 7f354ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:28.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.558+0000 7f354ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:28.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.558+0000 7f354dd9b640 1 --2- 172.21.15.121:0/2069300875 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f355013b6f0 0x7f3550135b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:28.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.558+0000 7f354ed9d640 10 monclient(hunting): handle_auth_done global_id 114177 payload 824 2024-09-17T18:37:28.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.558+0000 7f354ed9d640 1 -- 172.21.15.121:0/2069300875 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3550073900 msgr2=0x7f35501ca1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:28.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.558+0000 7f354ed9d640 1 --2- 172.21.15.121:0/2069300875 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3550073900 0x7f35501ca1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:28.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.558+0000 7f354ed9d640 1 -- 172.21.15.121:0/2069300875 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f355013b6f0 msgr2=0x7f3550135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:28.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.558+0000 7f354ed9d640 1 --2- 172.21.15.121:0/2069300875 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f355013b6f0 0x7f3550135b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:28.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.558+0000 7f354ed9d640 10 monclient: _finish_hunting 0 2024-09-17T18:37:28.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.558+0000 7f354ed9d640 1 monclient: found mon.noname-a 2024-09-17T18:37:28.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.558+0000 7f354ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:28.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.558+0000 7f354ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:37:28.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.558+0000 7f354ed9d640 1 -- 172.21.15.121:0/2069300875 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3550121610 con 0x7f355013e020 2024-09-17T18:37:28.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.558+0000 7f354e59c640 1 --2- 172.21.15.121:0/2069300875 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3550073900 0x7f35501ca1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:37:28.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.560+0000 7f354ed9d640 1 -- 172.21.15.121:0/2069300875 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f355013e020 msgr2=0x7f3550136080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:28.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.560+0000 7f354ed9d640 1 --2- 172.21.15.121:0/2069300875 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f355013e020 0x7f3550136080 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f354800c780 tx=0x7f354800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:28.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.560+0000 7f353effd640 10 client.?.objecter ms_handle_connect 0x7f355013e020 2024-09-17T18:37:28.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.560+0000 7f353effd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:37:28.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.560+0000 7f353effd640 1 -- 172.21.15.121:0/2069300875 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3548016020 con 0x7f355013e020 2024-09-17T18:37:28.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.560+0000 7f353effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:28.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.560+0000 7f353effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:37:28.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.560+0000 7f353effd640 10 monclient: dump: 2024-09-17T18:37:28.573 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:28.573 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:28.573 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:28.573 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:28.573 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:28.573 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:28.573 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:28.574 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:28.574 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:28.574 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:28.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.560+0000 7f353effd640 10 monclient: _finish_auth 0 2024-09-17T18:37:28.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.560+0000 7f353effd640 10 monclient: _check_auth_tickets 2024-09-17T18:37:28.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.560+0000 7f353effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:28.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.560+0000 7f353effd640 1 -- 172.21.15.121:0/2069300875 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f354800ac60 con 0x7f355013e020 2024-09-17T18:37:28.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.560+0000 7f3554bec640 5 monclient: authenticate success, global_id 114177 2024-09-17T18:37:28.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.560+0000 7f3554bec640 10 monclient: _renew_subs 2024-09-17T18:37:28.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.560+0000 7f3554bec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:28.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.560+0000 7f3554bec640 1 -- 172.21.15.121:0/2069300875 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3550074c70 con 0x7f355013e020 2024-09-17T18:37:28.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.560+0000 7f353effd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:28.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.560+0000 7f353effd640 1 -- 172.21.15.121:0/2069300875 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3548012660 con 0x7f355013e020 2024-09-17T18:37:28.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.560+0000 7f353effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:28.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.560+0000 7f353effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:37:28.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.560+0000 7f353effd640 10 monclient: dump: 2024-09-17T18:37:28.575 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:28.575 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:28.575 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:28.575 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:28.575 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:28.575 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:28.575 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:28.575 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:28.575 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:28.575 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:28.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.560+0000 7f3554bec640 10 client.114177.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:37:28.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.560+0000 7f354cd99640 4 set_mon_vals no callback set 2024-09-17T18:37:28.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.560+0000 7f3554bec640 10 monclient: _renew_subs 2024-09-17T18:37:28.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.560+0000 7f3554bec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:28.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.560+0000 7f3554bec640 1 -- 172.21.15.121:0/2069300875 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35501225f0 con 0x7f355013e020 2024-09-17T18:37:28.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.560+0000 7f3554bec640 1 librados: init done 2024-09-17T18:37:28.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.561+0000 7f353effd640 1 -- 172.21.15.121:0/2069300875 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f3548019020 con 0x7f355013e020 2024-09-17T18:37:28.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.561+0000 7f3554bec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:28.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.561+0000 7f3554bec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:28.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.561+0000 7f3554bec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:28.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.561+0000 7f3554bec640 1 -- 172.21.15.121:0/2069300875 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3550073ce0 con 0x7f355013e020 2024-09-17T18:37:28.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.561+0000 7f354ed9d640 1 -- 172.21.15.121:0/2069300875 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f355013e020 msgr2=0x7f3550136080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-17T18:37:28.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.561+0000 7f354ed9d640 1 -- 172.21.15.121:0/2069300875 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f355013e020 msgr2=0x7f3550136080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:37:28.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.561+0000 7f354ed9d640 1 --2- 172.21.15.121:0/2069300875 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f355013e020 0x7f3550136080 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f354800c780 tx=0x7f354800cc40 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:37:28.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.561+0000 7f354ed9d640 1 --2- 172.21.15.121:0/2069300875 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f355013e020 0x7f3550136080 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f354800c780 tx=0x7f354800cc40 comp rx=0 tx=0).stop 2024-09-17T18:37:28.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.561+0000 7f353effd640 1 --2- 172.21.15.121:0/2069300875 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3530040170 0x7f3530042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:28.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.561+0000 7f353effd640 10 monclient: ms_handle_reset current mon [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:37:28.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.561+0000 7f353effd640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:28.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.561+0000 7f353effd640 1 -- 172.21.15.121:0/2069300875 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f355013e020 msgr2=0x7f3550136080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:37:28.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.561+0000 7f353effd640 1 --2- 172.21.15.121:0/2069300875 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f355013e020 0x7f3550136080 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:28.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.561+0000 7f353effd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T18:37:28.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.561+0000 7f353effd640 1 --2- 172.21.15.121:0/2069300875 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3550073900 0x7f3530044d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:28.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.561+0000 7f353effd640 10 monclient(hunting): picked mon.c con 0x7f3550073900 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:28.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.561+0000 7f353effd640 1 --2- 172.21.15.121:0/2069300875 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f355013b6f0 0x7f3530047610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:28.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.561+0000 7f353effd640 10 monclient(hunting): picked mon.b con 0x7f355013b6f0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:37:28.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.561+0000 7f353effd640 1 --2- 172.21.15.121:0/2069300875 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3530047f90 0x7f353004a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:28.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.561+0000 7f353effd640 10 monclient(hunting): picked mon.a con 0x7f3530047f90 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:37:28.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.561+0000 7f353effd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:28.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.561+0000 7f353effd640 1 -- 172.21.15.121:0/2069300875 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f353004aed0 con 0x7f3530047f90 2024-09-17T18:37:28.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f353effd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:28.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f353effd640 1 -- 172.21.15.121:0/2069300875 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f353004b050 con 0x7f355013b6f0 2024-09-17T18:37:28.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f353effd640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:28.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f353effd640 1 -- 172.21.15.121:0/2069300875 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f353004b1d0 con 0x7f3550073900 2024-09-17T18:37:28.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f353effd640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:28.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f354e59c640 1 --2- 172.21.15.121:0/2069300875 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3530047f90 0x7f353004a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:28.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f354dd9b640 1 --2- 172.21.15.121:0/2069300875 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3550073900 0x7f3530044d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:28.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f354e59c640 10 monclient(hunting): get_auth_request con 0x7f3530047f90 auth_method 0 2024-09-17T18:37:28.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f354e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:28.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f354e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:28.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f354e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:37:28.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f354dd9b640 10 monclient(hunting): get_auth_request con 0x7f3550073900 auth_method 0 2024-09-17T18:37:28.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f354dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:28.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f354dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:28.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f354dd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:37:28.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f354e59c640 1 --2- 172.21.15.121:0/2069300875 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3530040170 0x7f3530042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:28.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f354e59c640 10 monclient(hunting): get_auth_request con 0x7f3530040170 auth_method 0 2024-09-17T18:37:28.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f354ed9d640 1 --2- 172.21.15.121:0/2069300875 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f355013b6f0 0x7f3530047610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:28.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f354e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:28.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f354e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:28.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f354e59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:37:28.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f354dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:28.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f354dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:28.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f354dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:37:28.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f354ed9d640 10 monclient(hunting): get_auth_request con 0x7f355013b6f0 auth_method 0 2024-09-17T18:37:28.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f354ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:28.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f354ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:28.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f354ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:37:28.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f354e59c640 10 monclient(hunting): handle_auth_done global_id 114177 payload 293 2024-09-17T18:37:28.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f354e59c640 1 -- 172.21.15.121:0/2069300875 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3550073900 msgr2=0x7f3530044d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:28.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f354e59c640 1 --2- 172.21.15.121:0/2069300875 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3550073900 0x7f3530044d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:28.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f354e59c640 1 -- 172.21.15.121:0/2069300875 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f355013b6f0 msgr2=0x7f3530047610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:28.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f354e59c640 1 --2- 172.21.15.121:0/2069300875 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f355013b6f0 0x7f3530047610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:28.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f354e59c640 10 monclient: _finish_hunting 0 2024-09-17T18:37:28.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f354e59c640 1 monclient: found mon.a 2024-09-17T18:37:28.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f354e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:28.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f354e59c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:28.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f354e59c640 1 -- 172.21.15.121:0/2069300875 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7f353004b3d0 con 0x7f3530047f90 2024-09-17T18:37:28.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f354e59c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:28.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f354e59c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:28.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f354e59c640 1 -- 172.21.15.121:0/2069300875 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35501388b0 con 0x7f3530047f90 2024-09-17T18:37:28.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f354e59c640 10 monclient: _finish_auth 0 2024-09-17T18:37:28.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f354e59c640 10 monclient: _check_auth_tickets 2024-09-17T18:37:28.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f354e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:28.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.562+0000 7f354ed9d640 1 --2- 172.21.15.121:0/2069300875 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f355013b6f0 0x7f3530047610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T18:37:28.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.564+0000 7f354e59c640 1 -- 172.21.15.121:0/2069300875 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3530047f90 msgr2=0x7f353004a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:28.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.564+0000 7f353effd640 10 client.114177.objecter ms_handle_connect 0x7f3530047f90 2024-09-17T18:37:28.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.564+0000 7f353effd640 10 client.114177.objecter resend_mon_ops 2024-09-17T18:37:28.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.564+0000 7f354e59c640 1 --2- 172.21.15.121:0/2069300875 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3530047f90 0x7f353004a420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3540016fb0 tx=0x7f3548003d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:28.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.564+0000 7f354e59c640 1 --2- 172.21.15.121:0/2069300875 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3530040170 0x7f3530042630 secure :-1 s=READY pgs=4103 cs=0 l=1 rev1=1 crypto rx=0x7f3540016af0 tx=0x7f3540009070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:28.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.564+0000 7f353effd640 10 client.114177.objecter ms_handle_connect 0x7f3530040170 2024-09-17T18:37:28.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.566+0000 7f353effd640 1 -- 172.21.15.121:0/2069300875 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3540010040 con 0x7f3530047f90 2024-09-17T18:37:28.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.566+0000 7f353effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:28.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.566+0000 7f353effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:37:28.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.566+0000 7f353effd640 10 monclient: dump: 2024-09-17T18:37:28.581 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:28.581 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:28.581 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:28.581 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:28.581 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:28.581 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:28.581 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:28.581 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:28.581 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:28.582 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:28.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.566+0000 7f353effd640 1 -- 172.21.15.121:0/2069300875 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f354001d070 con 0x7f3530047f90 2024-09-17T18:37:28.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.566+0000 7f353effd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:28.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.566+0000 7f353ffff640 4 set_mon_vals no callback set 2024-09-17T18:37:28.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.567+0000 7f353effd640 1 -- 172.21.15.121:0/2069300875 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f35400047e0 con 0x7f3530047f90 2024-09-17T18:37:28.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.567+0000 7f353effd640 1 -- 172.21.15.121:0/2069300875 <== mon.0 v2:172.21.15.121:3300/0 4 ==== osd_map(679..679 src has 247..679) ==== 4685+0+0 (secure 0 0 0) 0x7f35400552c0 con 0x7f3530047f90 2024-09-17T18:37:28.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.567+0000 7f353effd640 10 client.114177.objecter ms_dispatch 0x7f3550000b90 osd_map(679..679 src has 247..679) 2024-09-17T18:37:28.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.567+0000 7f353effd640 3 client.114177.objecter handle_osd_map got epochs [679,679] > 0 2024-09-17T18:37:28.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.567+0000 7f353effd640 3 client.114177.objecter handle_osd_map decoding full epoch 679 2024-09-17T18:37:28.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.567+0000 7f353effd640 20 client.114177.objecter dump_active .. 0 homeless 2024-09-17T18:37:28.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.570+0000 7f353effd640 1 -- 172.21.15.121:0/2069300875 <== mon.0 v2:172.21.15.121:3300/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3540007a60 con 0x7f3530047f90 2024-09-17T18:37:28.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.570+0000 7f353effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:28.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.570+0000 7f353effd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:37:28.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.747+0000 7f3554bec640 1 -- 172.21.15.121:0/2069300875 --> v2:172.21.15.173:6812/2423786037 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f355006cd40 con 0x7f3530040170 2024-09-17T18:37:28.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.748+0000 7f353effd640 1 -- 172.21.15.121:0/2069300875 <== mgr.4105 v2:172.21.15.173:6812/2423786037 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f355006cd40 con 0x7f3530040170 2024-09-17T18:37:28.750 INFO:teuthology.orchestra.run.smithi121.stderr:instructed osd(s) 1 to deep-scrub 2024-09-17T18:37:28.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.750+0000 7f3554bec640 10 librados: watch_flush enter 2024-09-17T18:37:28.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.750+0000 7f3554bec640 10 librados: watch_flush exit 2024-09-17T18:37:28.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.750+0000 7f3554bec640 20 client.114177.objecter shutdown clearing up homeless session... 2024-09-17T18:37:28.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.750+0000 7f3554bec640 10 client.114177.objecter successfully canceled tick 2024-09-17T18:37:28.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.750+0000 7f3554bec640 1 -- 172.21.15.121:0/2069300875 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3530040170 msgr2=0x7f3530042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:28.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.750+0000 7f3554bec640 1 --2- 172.21.15.121:0/2069300875 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3530040170 0x7f3530042630 secure :-1 s=READY pgs=4103 cs=0 l=1 rev1=1 crypto rx=0x7f3540016af0 tx=0x7f3540009070 comp rx=0 tx=0).stop 2024-09-17T18:37:28.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.750+0000 7f3554bec640 10 monclient: shutdown 2024-09-17T18:37:28.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.750+0000 7f3554bec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:28.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.750+0000 7f3554bec640 1 -- 172.21.15.121:0/2069300875 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3530047f90 msgr2=0x7f353004a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:28.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.750+0000 7f3554bec640 1 --2- 172.21.15.121:0/2069300875 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3530047f90 0x7f353004a420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3540016fb0 tx=0x7f3548003d10 comp rx=0 tx=0).stop 2024-09-17T18:37:28.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.750+0000 7f3554bec640 1 -- 172.21.15.121:0/2069300875 shutdown_connections 2024-09-17T18:37:28.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.750+0000 7f3554bec640 1 --2- 172.21.15.121:0/2069300875 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f355013b6f0 0x7f3530047610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:28.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.750+0000 7f3554bec640 1 --2- 172.21.15.121:0/2069300875 >> v2:172.21.15.173:6812/2423786037 conn(0x7f3530040170 0x7f3530042630 unknown :-1 s=CLOSED pgs=4103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:28.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.750+0000 7f3554bec640 1 --2- 172.21.15.121:0/2069300875 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3530047f90 0x7f353004a420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:28.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.750+0000 7f3554bec640 1 --2- 172.21.15.121:0/2069300875 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3550073900 0x7f3530044d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:28.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.750+0000 7f3554bec640 1 -- 172.21.15.121:0/2069300875 >> 172.21.15.121:0/2069300875 conn(0x7f35500a49c0 msgr2=0x7f3550136c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:28.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.750+0000 7f3554bec640 1 -- 172.21.15.121:0/2069300875 shutdown_connections 2024-09-17T18:37:28.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.750+0000 7f3554bec640 1 -- 172.21.15.121:0/2069300875 wait complete. 2024-09-17T18:37:28.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.750+0000 7f3554bec640 1 librados: shutdown 2024-09-17T18:37:28.762 INFO:tasks.ceph:Scrubbing osd.2 2024-09-17T18:37:28.763 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-17T18:37:28.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.861+0000 7f97634c6640 10 monclient: get_monmap_and_config 2024-09-17T18:37:28.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.861+0000 7f97634c6640 10 monclient: build_initial_monmap 2024-09-17T18:37:28.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.861+0000 7f97634c6640 10 monclient: monmap: 2024-09-17T18:37:28.863 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:28.863 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:28.863 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:37:28.862170+0000 2024-09-17T18:37:28.863 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:37:28.862170+0000 2024-09-17T18:37:28.863 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:28.863 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:28.863 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:37:28.864 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:37:28.864 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:37:28.864 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:37:28.864 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:37:28.864 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:28.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.862+0000 7f97634c6640 1 Processor -- start 2024-09-17T18:37:28.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.862+0000 7f97634c6640 1 -- start start 2024-09-17T18:37:28.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.862+0000 7f97634c6640 10 monclient: init 2024-09-17T18:37:28.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.862+0000 7f97634c6640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:28.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.862+0000 7f97634c6640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-17T18:37:28.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.862+0000 7f97634c6640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f975c136670 0x7f975c1360a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:28.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.862+0000 7f97634c6640 10 monclient(hunting): picked mon.noname-c con 0x7f975c136670 addr v2:172.21.15.173:3300/0 2024-09-17T18:37:28.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.862+0000 7f97634c6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f975c136a50 addr v1:172.21.15.121:6789/0 2024-09-17T18:37:28.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.862+0000 7f97634c6640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f975c0a2080 0x7f975c13e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:28.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.862+0000 7f97634c6640 10 monclient(hunting): picked mon.noname-a con 0x7f975c0a2080 addr v2:172.21.15.121:3300/0 2024-09-17T18:37:28.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.862+0000 7f97634c6640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f975c0a2fe0 con 0x7f975c136a50 2024-09-17T18:37:28.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.862+0000 7f97634c6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:28.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.862+0000 7f97634c6640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f975c0a32c0 con 0x7f975c0a2080 2024-09-17T18:37:28.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.862+0000 7f97634c6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:28.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.862+0000 7f97634c6640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f975c0a2a20 con 0x7f975c136670 2024-09-17T18:37:28.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.862+0000 7f97634c6640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:28.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.862+0000 7f97634c6640 10 monclient(hunting): authenticate will time out at 1852.578915s 2024-09-17T18:37:28.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.863+0000 7f9761a63640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f975c0a2080 0x7f975c13e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:28.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.863+0000 7f9761a63640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f975c0a2080 0x7f975c13e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51424/0 (socket says 172.21.15.121:51424) 2024-09-17T18:37:28.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.863+0000 7f9761a63640 1 -- 172.21.15.121:0/1046505593 learned_addr learned my addr 172.21.15.121:0/1046505593 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:37:28.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.863+0000 7f9761a63640 10 monclient(hunting): get_auth_request con 0x7f975c0a2080 auth_method 0 2024-09-17T18:37:28.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.863+0000 7f9761a63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:28.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.863+0000 7f9761a63640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:28.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.863+0000 7f9761a63640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:28.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.863+0000 7f9761262640 1 --2- 172.21.15.121:0/1046505593 >> v2:172.21.15.173:3300/0 conn(0x7f975c136670 0x7f975c1360a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:28.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.863+0000 7f9761262640 10 monclient(hunting): get_auth_request con 0x7f975c136670 auth_method 0 2024-09-17T18:37:28.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.863+0000 7f9761262640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:28.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.863+0000 7f9761262640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:28.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.863+0000 7f9761262640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:28.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.863+0000 7f9761262640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:28.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.863+0000 7f9761262640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:28.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.863+0000 7f9761262640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:28.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.863+0000 7f9761262640 10 monclient(hunting): handle_auth_done global_id 84355 payload 274 2024-09-17T18:37:28.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.863+0000 7f9761262640 1 -- 172.21.15.121:0/1046505593 >> v2:172.21.15.121:3300/0 conn(0x7f975c0a2080 msgr2=0x7f975c13e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:28.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.863+0000 7f9761262640 1 --2- 172.21.15.121:0/1046505593 >> v2:172.21.15.121:3300/0 conn(0x7f975c0a2080 0x7f975c13e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:28.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.863+0000 7f9761262640 1 -- 172.21.15.121:0/1046505593 >> v1:172.21.15.121:6789/0 conn(0x7f975c136a50 legacy=0x7f975c0a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:28.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.863+0000 7f9761262640 10 monclient: _finish_hunting 0 2024-09-17T18:37:28.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.863+0000 7f9761262640 1 monclient: found mon.noname-c 2024-09-17T18:37:28.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.863+0000 7f9761262640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:28.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.863+0000 7f9761262640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.173:3300/0 2024-09-17T18:37:28.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.863+0000 7f9761262640 1 -- 172.21.15.121:0/1046505593 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f975c0a2d00 con 0x7f975c136670 2024-09-17T18:37:28.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.866+0000 7f9761262640 1 -- 172.21.15.121:0/1046505593 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f975c136670 msgr2=0x7f975c1360a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:28.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.866+0000 7f9761262640 1 --2- 172.21.15.121:0/1046505593 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f975c136670 0x7f975c1360a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f975c06e340 tx=0x7f975802a080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93b2504e70a5c53c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:28.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.866+0000 7f9753fff640 1 -- 172.21.15.121:0/1046505593 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9758037070 con 0x7f975c136670 2024-09-17T18:37:28.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.866+0000 7f9753fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:28.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.866+0000 7f9753fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:37:28.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.866+0000 7f9753fff640 10 monclient: dump: 2024-09-17T18:37:28.875 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:28.876 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:28.876 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:28.876 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:28.876 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:28.876 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:28.876 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:28.876 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:28.876 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:28.876 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:28.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.866+0000 7f9753fff640 10 monclient: _finish_auth 0 2024-09-17T18:37:28.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.866+0000 7f9753fff640 10 monclient: _check_auth_tickets 2024-09-17T18:37:28.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.866+0000 7f9753fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:28.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.866+0000 7f9753fff640 1 -- 172.21.15.121:0/1046505593 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9758032440 con 0x7f975c136670 2024-09-17T18:37:28.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.866+0000 7f97634c6640 5 monclient: authenticate success, global_id 84355 2024-09-17T18:37:28.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.866+0000 7f97634c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:37:28.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.866+0000 7f9753fff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:28.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.866+0000 7f97634c6640 10 monclient: get_monmap_and_config success 2024-09-17T18:37:28.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.866+0000 7f9753fff640 1 -- 172.21.15.121:0/1046505593 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97580327f0 con 0x7f975c136670 2024-09-17T18:37:28.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.866+0000 7f97634c6640 4 set_mon_vals no callback set 2024-09-17T18:37:28.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.866+0000 7f9753fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:28.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.866+0000 7f9753fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:37:28.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.866+0000 7f9753fff640 10 monclient: dump: 2024-09-17T18:37:28.877 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:28.877 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:28.877 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:28.877 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:28.877 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:28.877 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:28.877 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:28.878 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:28.878 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:28.878 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:28.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.867+0000 7f97634c6640 10 monclient: shutdown 2024-09-17T18:37:28.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.867+0000 7f97634c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:28.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.867+0000 7f97634c6640 1 -- 172.21.15.121:0/1046505593 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f975c136670 msgr2=0x7f975c1360a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:28.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.867+0000 7f97634c6640 1 --2- 172.21.15.121:0/1046505593 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f975c136670 0x7f975c1360a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f975c06e340 tx=0x7f975802a080 comp rx=0 tx=0).stop 2024-09-17T18:37:28.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.867+0000 7f97634c6640 1 -- 172.21.15.121:0/1046505593 shutdown_connections 2024-09-17T18:37:28.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.867+0000 7f97634c6640 1 --2- 172.21.15.121:0/1046505593 >> v2:172.21.15.121:3300/0 conn(0x7f975c0a2080 0x7f975c13e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:28.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.867+0000 7f97634c6640 1 --2- 172.21.15.121:0/1046505593 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f975c136670 0x7f975c1360a0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:28.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.867+0000 7f97634c6640 1 -- 172.21.15.121:0/1046505593 >> 172.21.15.121:0/1046505593 conn(0x7f975c0a3a70 msgr2=0x7f975c0a3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:28.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.867+0000 7f97634c6640 1 -- 172.21.15.121:0/1046505593 shutdown_connections 2024-09-17T18:37:28.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.867+0000 7f97634c6640 1 -- 172.21.15.121:0/1046505593 wait complete. 2024-09-17T18:37:28.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.867+0000 7f97634c6640 10 monclient: build_initial_monmap 2024-09-17T18:37:28.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.867+0000 7f97634c6640 10 monclient: monmap: 2024-09-17T18:37:28.879 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:28.879 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:37:28.879 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:37:28.879 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:37:28.879 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:28.879 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:28.879 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:37:28.879 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:37:28.879 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:37:28.879 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:28.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.867+0000 7f97634c6640 1 librados: starting msgr at 2024-09-17T18:37:28.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.867+0000 7f97634c6640 1 librados: starting objecter 2024-09-17T18:37:28.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.868+0000 7f97634c6640 1 Processor -- start 2024-09-17T18:37:28.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.868+0000 7f97634c6640 1 -- start start 2024-09-17T18:37:28.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.868+0000 7f97634c6640 1 librados: setting wanted keys 2024-09-17T18:37:28.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.868+0000 7f97634c6640 1 librados: calling monclient init 2024-09-17T18:37:28.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.868+0000 7f97634c6640 10 monclient: init 2024-09-17T18:37:28.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.868+0000 7f97634c6640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:28.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.868+0000 7f97634c6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T18:37:28.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.868+0000 7f97634c6640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f975c0a2080 0x7f975c1cb760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:28.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.868+0000 7f97634c6640 10 monclient(hunting): picked mon.noname-c con 0x7f975c0a2080 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:28.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.868+0000 7f97634c6640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f975c136670 0x7f975c11d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:28.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.868+0000 7f97634c6640 10 monclient(hunting): picked mon.noname-a con 0x7f975c136670 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:37:28.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.868+0000 7f97634c6640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f975c136a50 0x7f975c11db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:28.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.868+0000 7f97634c6640 10 monclient(hunting): picked mon.noname-b con 0x7f975c136a50 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:37:28.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.868+0000 7f97634c6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:28.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.868+0000 7f97634c6640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f975c077a60 con 0x7f975c136670 2024-09-17T18:37:28.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.868+0000 7f97634c6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:28.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.868+0000 7f97634c6640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f975c0778e0 con 0x7f975c136a50 2024-09-17T18:37:28.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.868+0000 7f97634c6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:28.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.868+0000 7f97634c6640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f975c136e50 con 0x7f975c0a2080 2024-09-17T18:37:28.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.868+0000 7f97634c6640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:28.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.868+0000 7f97634c6640 10 monclient(hunting): authenticate will time out at 1852.584666s 2024-09-17T18:37:28.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.868+0000 7f9761262640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f975c0a2080 0x7f975c1cb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:28.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.868+0000 7f9760a61640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f975c136670 0x7f975c11d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:28.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.868+0000 7f9761262640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f975c0a2080 0x7f975c1cb760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:36898/0 (socket says 172.21.15.121:36898) 2024-09-17T18:37:28.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.868+0000 7f9760a61640 10 monclient(hunting): get_auth_request con 0x7f975c136670 auth_method 0 2024-09-17T18:37:28.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.868+0000 7f9761262640 1 -- 172.21.15.121:0/2623400584 learned_addr learned my addr 172.21.15.121:0/2623400584 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:37:28.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.868+0000 7f9760a61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:28.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.868+0000 7f9760a61640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:28.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.868+0000 7f9760a61640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:28.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.868+0000 7f9761262640 10 monclient(hunting): get_auth_request con 0x7f975c0a2080 auth_method 0 2024-09-17T18:37:28.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.868+0000 7f9761262640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:28.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.868+0000 7f9761262640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:28.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.868+0000 7f9761262640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:28.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.868+0000 7f9760a61640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:28.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.868+0000 7f9760a61640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:28.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.868+0000 7f9760a61640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:28.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.868+0000 7f9761262640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:28.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.868+0000 7f9761262640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:28.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.868+0000 7f9761262640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:28.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.869+0000 7f9761262640 10 monclient(hunting): handle_auth_done global_id 64631 payload 824 2024-09-17T18:37:28.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.869+0000 7f9761a63640 1 --2- 172.21.15.121:0/2623400584 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f975c136a50 0x7f975c11db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:28.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.869+0000 7f9761262640 1 -- 172.21.15.121:0/2623400584 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f975c136a50 msgr2=0x7f975c11db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:28.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.869+0000 7f9761262640 1 --2- 172.21.15.121:0/2623400584 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f975c136a50 0x7f975c11db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:28.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.869+0000 7f9761262640 1 -- 172.21.15.121:0/2623400584 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f975c136670 msgr2=0x7f975c11d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:28.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.869+0000 7f9761262640 1 --2- 172.21.15.121:0/2623400584 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f975c136670 0x7f975c11d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:28.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.869+0000 7f9761262640 10 monclient: _finish_hunting 0 2024-09-17T18:37:28.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.869+0000 7f9761262640 1 monclient: found mon.noname-c 2024-09-17T18:37:28.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.869+0000 7f9761262640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:28.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.869+0000 7f9761262640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:37:28.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.869+0000 7f9761262640 1 -- 172.21.15.121:0/2623400584 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f975c11ed00 con 0x7f975c0a2080 2024-09-17T18:37:28.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.869+0000 7f9761a63640 10 monclient: get_auth_request con 0x7f975c136a50 auth_method 0 2024-09-17T18:37:28.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.869+0000 7f9760a61640 1 --2- 172.21.15.121:0/2623400584 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f975c136670 0x7f975c11d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:37:28.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.869+0000 7f9761a63640 1 --2- 172.21.15.121:0/2623400584 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f975c136a50 0x7f975c11db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T18:37:28.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.871+0000 7f9761262640 1 -- 172.21.15.121:0/2623400584 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f975c0a2080 msgr2=0x7f975c1cb760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:28.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.871+0000 7f9761262640 1 --2- 172.21.15.121:0/2623400584 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f975c0a2080 0x7f975c1cb760 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f9758000c00 tx=0x7f97580046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:28.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.871+0000 7f9751ffb640 10 client.?.objecter ms_handle_connect 0x7f975c0a2080 2024-09-17T18:37:28.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.871+0000 7f9751ffb640 10 client.?.objecter resend_mon_ops 2024-09-17T18:37:28.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.871+0000 7f9751ffb640 1 -- 172.21.15.121:0/2623400584 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9758041020 con 0x7f975c0a2080 2024-09-17T18:37:28.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.871+0000 7f9751ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:28.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.871+0000 7f9751ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:37:28.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.871+0000 7f9751ffb640 10 monclient: dump: 2024-09-17T18:37:28.884 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:28.884 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:28.884 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:28.884 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:28.884 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:28.884 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:28.884 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:28.884 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:28.884 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:28.884 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:28.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.871+0000 7f9751ffb640 10 monclient: _finish_auth 0 2024-09-17T18:37:28.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.871+0000 7f9751ffb640 10 monclient: _check_auth_tickets 2024-09-17T18:37:28.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.871+0000 7f9751ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:28.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.871+0000 7f9751ffb640 1 -- 172.21.15.121:0/2623400584 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f975803d910 con 0x7f975c0a2080 2024-09-17T18:37:28.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.871+0000 7f97634c6640 5 monclient: authenticate success, global_id 64631 2024-09-17T18:37:28.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.871+0000 7f97634c6640 10 monclient: _renew_subs 2024-09-17T18:37:28.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.871+0000 7f97634c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:37:28.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.871+0000 7f97634c6640 1 -- 172.21.15.121:0/2623400584 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f975c1207c0 con 0x7f975c0a2080 2024-09-17T18:37:28.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.871+0000 7f9751ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:28.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.871+0000 7f9751ffb640 1 -- 172.21.15.121:0/2623400584 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9758037040 con 0x7f975c0a2080 2024-09-17T18:37:28.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.871+0000 7f9751ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:28.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.871+0000 7f9751ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:37:28.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.871+0000 7f9751ffb640 10 monclient: dump: 2024-09-17T18:37:28.885 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:28.885 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:28.886 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:28.886 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:28.886 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:28.886 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:28.886 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:28.886 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:28.886 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:28.886 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:28.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.871+0000 7f97537fe640 4 set_mon_vals no callback set 2024-09-17T18:37:28.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.871+0000 7f97634c6640 10 client.64631.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:37:28.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.871+0000 7f97634c6640 10 monclient: _renew_subs 2024-09-17T18:37:28.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.871+0000 7f97634c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:37:28.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.871+0000 7f97634c6640 1 -- 172.21.15.121:0/2623400584 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f975c1cbca0 con 0x7f975c0a2080 2024-09-17T18:37:28.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.871+0000 7f97634c6640 1 librados: init done 2024-09-17T18:37:28.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.872+0000 7f97634c6640 10 client.64631.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:28.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.872+0000 7f97634c6640 15 client.64631.objecter _session_command_op_assign -1 1 2024-09-17T18:37:28.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.872+0000 7f97634c6640 20 client.64631.objecter _get_session osd=-1 returning homeless 2024-09-17T18:37:28.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.872+0000 7f97634c6640 10 client.64631.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:37:28.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.872+0000 7f97634c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:37:28.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.872+0000 7f97634c6640 1 -- 172.21.15.121:0/2623400584 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f975c0a24b0 con 0x7f975c0a2080 2024-09-17T18:37:28.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.872+0000 7f9751ffb640 1 -- 172.21.15.121:0/2623400584 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f9758044020 con 0x7f975c0a2080 2024-09-17T18:37:28.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.872+0000 7f9751ffb640 1 --2- 172.21.15.121:0/2623400584 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9740040170 0x7f9740042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:28.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.872+0000 7f9751ffb640 1 -- 172.21.15.121:0/2623400584 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(679..679 src has 247..679) ==== 4685+0+0 (secure 0 0 0) 0x7f9758073a70 con 0x7f975c0a2080 2024-09-17T18:37:28.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.872+0000 7f9751ffb640 10 client.64631.objecter ms_dispatch 0x7f975c000b90 osd_map(679..679 src has 247..679) 2024-09-17T18:37:28.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.872+0000 7f9751ffb640 3 client.64631.objecter handle_osd_map got epochs [679,679] > 0 2024-09-17T18:37:28.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.872+0000 7f9751ffb640 3 client.64631.objecter handle_osd_map decoding full epoch 679 2024-09-17T18:37:28.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.873+0000 7f9751ffb640 10 client.64631.objecter checking command 1 2024-09-17T18:37:28.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.873+0000 7f9751ffb640 1 --2- 172.21.15.121:0/2623400584 >> v2:172.21.15.121:6804/2545987420 conn(0x7f9740047630 0x7f9740049ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:28.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.873+0000 7f9751ffb640 20 client.64631.objecter _get_session s=0x7f9740046f80 osd=2 3 2024-09-17T18:37:28.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.873+0000 7f9751ffb640 20 client.64631.objecter put_session s=0x7f9740046f80 osd=2 3 2024-09-17T18:37:28.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.873+0000 7f9751ffb640 15 client.64631.objecter _session_command_op_remove -1 1 2024-09-17T18:37:28.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.873+0000 7f9751ffb640 20 client.64631.objecter _get_session s=0x7f9740046f80 osd=2 3 2024-09-17T18:37:28.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.873+0000 7f9751ffb640 20 client.64631.objecter get_session s=0x7f9740046f80 osd=2 3 2024-09-17T18:37:28.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.873+0000 7f9751ffb640 15 client.64631.objecter _session_command_op_assign 2 1 2024-09-17T18:37:28.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.873+0000 7f9751ffb640 20 client.64631.objecter put_session s=0x7f9740046f80 osd=2 4 2024-09-17T18:37:28.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.873+0000 7f9751ffb640 10 client.64631.objecter _send_command 1 2024-09-17T18:37:28.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.873+0000 7f9751ffb640 1 -- 172.21.15.121:0/2623400584 --> v2:172.21.15.121:6804/2545987420 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9758008eb0 con 0x7f9740047630 2024-09-17T18:37:28.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.873+0000 7f9751ffb640 20 client.64631.objecter dump_active .. 0 homeless 2024-09-17T18:37:28.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.873+0000 7f9751ffb640 1 -- 172.21.15.121:0/2623400584 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_get_version_reply(handle=1 version=679) ==== 24+0+0 (secure 0 0 0) 0x7f97580782a0 con 0x7f975c0a2080 2024-09-17T18:37:28.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.873+0000 7f9751ffb640 10 monclient: handle_get_version_reply finishing 1 version 679 2024-09-17T18:37:28.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.873+0000 7f9761a63640 1 --2- 172.21.15.121:0/2623400584 >> v2:172.21.15.121:6804/2545987420 conn(0x7f9740047630 0x7f9740049ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:28.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.873+0000 7f9761a63640 10 monclient: get_auth_request con 0x7f9740047630 auth_method 0 2024-09-17T18:37:28.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.873+0000 7f9760a61640 1 --2- 172.21.15.121:0/2623400584 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9740040170 0x7f9740042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:28.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.873+0000 7f9760a61640 10 monclient: get_auth_request con 0x7f9740040170 auth_method 0 2024-09-17T18:37:28.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.875+0000 7f9751ffb640 10 client.64631.objecter ms_handle_connect 0x7f9740040170 2024-09-17T18:37:28.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.875+0000 7f9760a61640 1 --2- 172.21.15.121:0/2623400584 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9740040170 0x7f9740042630 secure :-1 s=READY pgs=4104 cs=0 l=1 rev1=1 crypto rx=0x7f975c11f320 tx=0x7f974c007e30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:28.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.877+0000 7f9761a63640 1 --2- 172.21.15.121:0/2623400584 >> v2:172.21.15.121:6804/2545987420 conn(0x7f9740047630 0x7f9740049ad0 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:28.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.877+0000 7f9751ffb640 10 client.64631.objecter ms_handle_connect 0x7f9740047630 2024-09-17T18:37:28.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.878+0000 7f9751ffb640 1 -- 172.21.15.121:0/2623400584 <== osd.2 v2:172.21.15.121:6804/2545987420 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9758008eb0 con 0x7f9740047630 2024-09-17T18:37:28.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.878+0000 7f9751ffb640 10 client.64631.objecter ms_dispatch 0x7f975c000b90 command_reply(tid 1: 0 ) 2024-09-17T18:37:28.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.878+0000 7f9751ffb640 10 client.64631.objecter _finish_command 1 = system:0 2024-09-17T18:37:28.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.878+0000 7f9751ffb640 20 client.64631.objecter put_session s=0x7f9740046f80 osd=2 4 2024-09-17T18:37:28.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.878+0000 7f9751ffb640 15 client.64631.objecter _session_command_op_remove 2 1 2024-09-17T18:37:28.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.890+0000 7f97634c6640 10 client.64631.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-17T18:37:28.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.890+0000 7f97634c6640 15 client.64631.objecter _session_command_op_assign -1 2 2024-09-17T18:37:28.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.890+0000 7f97634c6640 20 client.64631.objecter _get_session s=0x7f9740046f80 osd=2 3 2024-09-17T18:37:28.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.890+0000 7f97634c6640 20 client.64631.objecter put_session s=0x7f9740046f80 osd=2 3 2024-09-17T18:37:28.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.890+0000 7f97634c6640 20 client.64631.objecter _get_session s=0x7f9740046f80 osd=2 3 2024-09-17T18:37:28.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.890+0000 7f97634c6640 15 client.64631.objecter _session_command_op_remove -1 2 2024-09-17T18:37:28.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.890+0000 7f97634c6640 20 client.64631.objecter get_session s=0x7f9740046f80 osd=2 3 2024-09-17T18:37:28.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.890+0000 7f97634c6640 15 client.64631.objecter _session_command_op_assign 2 2 2024-09-17T18:37:28.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.890+0000 7f97634c6640 20 client.64631.objecter put_session s=0x7f9740046f80 osd=2 4 2024-09-17T18:37:28.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.890+0000 7f97634c6640 10 client.64631.objecter _send_command 2 2024-09-17T18:37:28.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.890+0000 7f97634c6640 1 -- 172.21.15.121:0/2623400584 --> v2:172.21.15.121:6804/2545987420 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f975c121a40 con 0x7f9740047630 2024-09-17T18:37:28.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.890+0000 7f9751ffb640 1 -- 172.21.15.121:0/2623400584 <== osd.2 v2:172.21.15.121:6804/2545987420 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f975c121a40 con 0x7f9740047630 2024-09-17T18:37:28.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.890+0000 7f9751ffb640 10 client.64631.objecter ms_dispatch 0x7f975c000b90 command_reply(tid 2: 0 ) 2024-09-17T18:37:28.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.890+0000 7f9751ffb640 10 client.64631.objecter _finish_command 2 = system:0 2024-09-17T18:37:28.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.890+0000 7f9751ffb640 20 client.64631.objecter put_session s=0x7f9740046f80 osd=2 4 2024-09-17T18:37:28.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.890+0000 7f9751ffb640 15 client.64631.objecter _session_command_op_remove 2 2 2024-09-17T18:37:28.894 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:37:28.894 INFO:teuthology.orchestra.run.smithi121.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T18:37:28.894 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:37:28.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.891+0000 7f97634c6640 10 librados: watch_flush enter 2024-09-17T18:37:28.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.891+0000 7f97634c6640 10 librados: watch_flush exit 2024-09-17T18:37:28.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.891+0000 7f97634c6640 10 client.64631.objecter close_session for osd.2 2024-09-17T18:37:28.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.891+0000 7f97634c6640 1 -- 172.21.15.121:0/2623400584 >> v2:172.21.15.121:6804/2545987420 conn(0x7f9740047630 msgr2=0x7f9740049ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:28.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.891+0000 7f97634c6640 1 --2- 172.21.15.121:0/2623400584 >> v2:172.21.15.121:6804/2545987420 conn(0x7f9740047630 0x7f9740049ad0 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:28.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.891+0000 7f97634c6640 20 client.64631.objecter put_session s=0x7f9740046f80 osd=2 1 2024-09-17T18:37:28.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.891+0000 7f97634c6640 20 client.64631.objecter shutdown clearing up homeless session... 2024-09-17T18:37:28.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.891+0000 7f97634c6640 10 client.64631.objecter successfully canceled tick 2024-09-17T18:37:28.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.891+0000 7f97634c6640 1 -- 172.21.15.121:0/2623400584 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9740040170 msgr2=0x7f9740042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:28.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.891+0000 7f97634c6640 1 --2- 172.21.15.121:0/2623400584 >> v2:172.21.15.173:6812/2423786037 conn(0x7f9740040170 0x7f9740042630 secure :-1 s=READY pgs=4104 cs=0 l=1 rev1=1 crypto rx=0x7f975c11f320 tx=0x7f974c007e30 comp rx=0 tx=0).stop 2024-09-17T18:37:28.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.891+0000 7f97634c6640 10 monclient: shutdown 2024-09-17T18:37:28.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.891+0000 7f97634c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:28.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.891+0000 7f97634c6640 1 -- 172.21.15.121:0/2623400584 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f975c0a2080 msgr2=0x7f975c1cb760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:28.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.891+0000 7f97634c6640 1 --2- 172.21.15.121:0/2623400584 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f975c0a2080 0x7f975c1cb760 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f9758000c00 tx=0x7f97580046c0 comp rx=0 tx=0).stop 2024-09-17T18:37:28.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.891+0000 7f9761a63640 1 -- 172.21.15.121:0/2623400584 reap_dead start 2024-09-17T18:37:28.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.891+0000 7f97634c6640 1 -- 172.21.15.121:0/2623400584 shutdown_connections 2024-09-17T18:37:28.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.891+0000 7f97634c6640 1 -- 172.21.15.121:0/2623400584 >> 172.21.15.121:0/2623400584 conn(0x7f975c0a3a70 msgr2=0x7f975c131440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:28.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.891+0000 7f97634c6640 1 -- 172.21.15.121:0/2623400584 shutdown_connections 2024-09-17T18:37:28.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.891+0000 7f97634c6640 1 -- 172.21.15.121:0/2623400584 wait complete. 2024-09-17T18:37:28.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.891+0000 7f97634c6640 1 librados: shutdown 2024-09-17T18:37:28.902 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-17T18:37:28.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.998+0000 7fae90db4640 10 monclient: get_monmap_and_config 2024-09-17T18:37:29.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.998+0000 7fae90db4640 10 monclient: build_initial_monmap 2024-09-17T18:37:29.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.998+0000 7fae90db4640 10 monclient: monmap: 2024-09-17T18:37:29.000 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:29.000 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:29.000 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:37:28.998766+0000 2024-09-17T18:37:29.000 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:37:28.998766+0000 2024-09-17T18:37:29.000 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:29.000 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:29.000 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:37:29.000 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:37:29.000 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:37:29.000 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:37:29.000 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:37:29.000 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:29.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.998+0000 7fae90db4640 1 Processor -- start 2024-09-17T18:37:29.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.998+0000 7fae90db4640 1 -- start start 2024-09-17T18:37:29.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.998+0000 7fae90db4640 10 monclient: init 2024-09-17T18:37:29.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.999+0000 7fae90db4640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:29.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.999+0000 7fae90db4640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-17T18:37:29.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.999+0000 7fae90db4640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fae8c09e550 0x7fae8c09e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:29.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.999+0000 7fae90db4640 10 monclient(hunting): picked mon.noname-a con 0x7fae8c09e550 addr v2:172.21.15.121:3300/0 2024-09-17T18:37:29.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.999+0000 7fae90db4640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fae8c09ee70 0x7fae8c139840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:29.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.999+0000 7fae90db4640 10 monclient(hunting): picked mon.noname-b con 0x7fae8c09ee70 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:29.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.999+0000 7fae90db4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fae8c09f2c0 addr v1:172.21.15.121:6789/0 2024-09-17T18:37:29.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.999+0000 7fae90db4640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae8c0a4460 con 0x7fae8c09f2c0 2024-09-17T18:37:29.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.999+0000 7fae90db4640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:29.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.999+0000 7fae90db4640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fae8c0a4950 con 0x7fae8c09e550 2024-09-17T18:37:29.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.999+0000 7fae90db4640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:29.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.999+0000 7fae90db4640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fae8c0a2b60 con 0x7fae8c09ee70 2024-09-17T18:37:29.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.999+0000 7fae90db4640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:29.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.999+0000 7fae90db4640 10 monclient(hunting): authenticate will time out at 1852.715558s 2024-09-17T18:37:29.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.999+0000 7fae89d9b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fae8c09ee70 0x7fae8c139840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:29.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.999+0000 7fae8a59c640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fae8c09e550 0x7fae8c09e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:29.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.999+0000 7fae89d9b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fae8c09ee70 0x7fae8c139840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:36904/0 (socket says 172.21.15.121:36904) 2024-09-17T18:37:29.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.999+0000 7fae8a59c640 10 monclient(hunting): get_auth_request con 0x7fae8c09e550 auth_method 0 2024-09-17T18:37:29.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.999+0000 7fae89d9b640 1 -- 172.21.15.121:0/3581122382 learned_addr learned my addr 172.21.15.121:0/3581122382 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:37:29.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.999+0000 7fae8a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:29.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.999+0000 7fae8a59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:29.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.999+0000 7fae8a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:29.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.999+0000 7fae89d9b640 10 monclient(hunting): get_auth_request con 0x7fae8c09ee70 auth_method 0 2024-09-17T18:37:29.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.999+0000 7fae89d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:29.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.999+0000 7fae89d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:29.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.999+0000 7fae89d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:29.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.999+0000 7fae89d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:29.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.999+0000 7fae89d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:29.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.999+0000 7fae89d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:29.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.999+0000 7fae8a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:29.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.999+0000 7fae8a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:29.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.999+0000 7fae8a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:29.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.999+0000 7fae89d9b640 10 monclient(hunting): handle_auth_done global_id 64634 payload 274 2024-09-17T18:37:29.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.999+0000 7fae89d9b640 1 -- 172.21.15.121:0/3581122382 >> v2:172.21.15.121:3300/0 conn(0x7fae8c09e550 msgr2=0x7fae8c09e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:29.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.999+0000 7fae89d9b640 1 --2- 172.21.15.121:0/3581122382 >> v2:172.21.15.121:3300/0 conn(0x7fae8c09e550 0x7fae8c09e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:29.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.999+0000 7fae89d9b640 1 -- 172.21.15.121:0/3581122382 >> v1:172.21.15.121:6789/0 conn(0x7fae8c09f2c0 legacy=0x7fae8c13bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:29.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.999+0000 7fae89d9b640 10 monclient: _finish_hunting 0 2024-09-17T18:37:29.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.999+0000 7fae89d9b640 1 monclient: found mon.noname-b 2024-09-17T18:37:29.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.999+0000 7fae89d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:29.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.999+0000 7fae89d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:37:29.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:28.999+0000 7fae89d9b640 1 -- 172.21.15.121:0/3581122382 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae8c09f840 con 0x7fae8c09ee70 2024-09-17T18:37:29.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.002+0000 7fae89d9b640 1 -- 172.21.15.121:0/3581122382 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fae8c09ee70 msgr2=0x7fae8c139840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:29.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.002+0000 7fae89d9b640 1 --2- 172.21.15.121:0/3581122382 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fae8c09ee70 0x7fae8c139840 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fae8c09e0a0 tx=0x7fae8002a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e21ffc65bb745e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:29.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.002+0000 7fae8959a640 1 -- 172.21.15.121:0/3581122382 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae80037070 con 0x7fae8c09ee70 2024-09-17T18:37:29.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.002+0000 7fae8959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:29.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.002+0000 7fae8959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:37:29.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.002+0000 7fae8959a640 10 monclient: dump: 2024-09-17T18:37:29.012 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:29.012 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:29.012 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:29.012 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:29.012 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:29.012 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:29.012 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:29.012 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:29.012 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:29.012 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:29.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.002+0000 7fae8959a640 10 monclient: _finish_auth 0 2024-09-17T18:37:29.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.002+0000 7fae8959a640 10 monclient: _check_auth_tickets 2024-09-17T18:37:29.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.002+0000 7fae8959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:29.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.002+0000 7fae8959a640 1 -- 172.21.15.121:0/3581122382 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae80032440 con 0x7fae8c09ee70 2024-09-17T18:37:29.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.002+0000 7fae90db4640 5 monclient: authenticate success, global_id 64634 2024-09-17T18:37:29.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.002+0000 7fae90db4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:37:29.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.002+0000 7fae8959a640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:29.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.002+0000 7fae90db4640 10 monclient: get_monmap_and_config success 2024-09-17T18:37:29.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.002+0000 7fae8959a640 1 -- 172.21.15.121:0/3581122382 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae80032810 con 0x7fae8c09ee70 2024-09-17T18:37:29.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.002+0000 7fae8959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:29.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.002+0000 7fae90db4640 4 set_mon_vals no callback set 2024-09-17T18:37:29.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.002+0000 7fae8959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:37:29.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.002+0000 7fae8959a640 10 monclient: dump: 2024-09-17T18:37:29.013 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:29.013 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:29.013 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:29.013 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:29.013 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:29.014 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:29.014 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:29.014 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:29.014 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:29.014 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:29.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.003+0000 7fae90db4640 10 monclient: shutdown 2024-09-17T18:37:29.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.003+0000 7fae90db4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:29.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.003+0000 7fae90db4640 1 -- 172.21.15.121:0/3581122382 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fae8c09ee70 msgr2=0x7fae8c139840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:29.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.003+0000 7fae90db4640 1 --2- 172.21.15.121:0/3581122382 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fae8c09ee70 0x7fae8c139840 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fae8c09e0a0 tx=0x7fae8002a060 comp rx=0 tx=0).stop 2024-09-17T18:37:29.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.003+0000 7fae90db4640 1 -- 172.21.15.121:0/3581122382 shutdown_connections 2024-09-17T18:37:29.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.003+0000 7fae90db4640 1 --2- 172.21.15.121:0/3581122382 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fae8c09ee70 0x7fae8c139840 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:29.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.003+0000 7fae90db4640 1 --2- 172.21.15.121:0/3581122382 >> v2:172.21.15.121:3300/0 conn(0x7fae8c09e550 0x7fae8c09e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:29.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.003+0000 7fae90db4640 1 -- 172.21.15.121:0/3581122382 >> 172.21.15.121:0/3581122382 conn(0x7fae8c0a7830 msgr2=0x7fae8c0a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:29.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.003+0000 7fae90db4640 1 -- 172.21.15.121:0/3581122382 shutdown_connections 2024-09-17T18:37:29.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.003+0000 7fae90db4640 1 -- 172.21.15.121:0/3581122382 wait complete. 2024-09-17T18:37:29.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.004+0000 7fae90db4640 10 monclient: build_initial_monmap 2024-09-17T18:37:29.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.004+0000 7fae90db4640 10 monclient: monmap: 2024-09-17T18:37:29.015 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:29.015 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:37:29.015 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:37:29.015 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:37:29.015 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:29.015 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:29.015 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:37:29.015 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:37:29.015 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:37:29.015 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:29.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.004+0000 7fae90db4640 1 librados: starting msgr at 2024-09-17T18:37:29.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.004+0000 7fae90db4640 1 librados: starting objecter 2024-09-17T18:37:29.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.004+0000 7fae90db4640 1 Processor -- start 2024-09-17T18:37:29.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.004+0000 7fae90db4640 1 -- start start 2024-09-17T18:37:29.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.004+0000 7fae90db4640 1 librados: setting wanted keys 2024-09-17T18:37:29.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.004+0000 7fae90db4640 1 librados: calling monclient init 2024-09-17T18:37:29.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.004+0000 7fae90db4640 10 monclient: init 2024-09-17T18:37:29.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae90db4640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:29.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae90db4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T18:37:29.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae90db4640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fae8c09e550 0x7fae8c13c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:29.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae90db4640 10 monclient(hunting): picked mon.noname-c con 0x7fae8c09e550 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:29.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae90db4640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae8c09ee70 0x7fae8c13c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:29.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae90db4640 10 monclient(hunting): picked mon.noname-a con 0x7fae8c09ee70 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:37:29.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae90db4640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae8c09f2c0 0x7fae8c125070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:29.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae90db4640 10 monclient(hunting): picked mon.noname-b con 0x7fae8c09f2c0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:37:29.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae90db4640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:29.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae90db4640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fae8c09de60 con 0x7fae8c09ee70 2024-09-17T18:37:29.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae90db4640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:29.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae90db4640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fae8c09dce0 con 0x7fae8c09f2c0 2024-09-17T18:37:29.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae90db4640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:29.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae90db4640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fae8c0747e0 con 0x7fae8c09e550 2024-09-17T18:37:29.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae90db4640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:29.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae90db4640 10 monclient(hunting): authenticate will time out at 1852.721167s 2024-09-17T18:37:29.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae8a59c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fae8c09e550 0x7fae8c13c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:29.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae89d9b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae8c09ee70 0x7fae8c13c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:29.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae89d9b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae8c09ee70 0x7fae8c13c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51448/0 (socket says 172.21.15.121:51448) 2024-09-17T18:37:29.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae8a59c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fae8c09e550 0x7fae8c13c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:36916/0 (socket says 172.21.15.121:36916) 2024-09-17T18:37:29.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae89d9b640 1 -- 172.21.15.121:0/4207556803 learned_addr learned my addr 172.21.15.121:0/4207556803 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:37:29.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae89d9b640 10 monclient(hunting): get_auth_request con 0x7fae8c09ee70 auth_method 0 2024-09-17T18:37:29.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae89d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:29.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae89d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:29.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae89d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:29.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae8a59c640 10 monclient(hunting): get_auth_request con 0x7fae8c09e550 auth_method 0 2024-09-17T18:37:29.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae8a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:29.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae8a59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:29.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae8a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:29.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae89d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:29.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae89d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:29.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae89d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:29.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae8a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:29.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae8a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:29.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae8a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:29.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae8a59c640 10 monclient(hunting): handle_auth_done global_id 64637 payload 824 2024-09-17T18:37:29.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae8a59c640 1 -- 172.21.15.121:0/4207556803 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae8c09f2c0 msgr2=0x7fae8c125070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:29.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae8a59c640 1 --2- 172.21.15.121:0/4207556803 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae8c09f2c0 0x7fae8c125070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:29.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae8a59c640 1 -- 172.21.15.121:0/4207556803 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae8c09ee70 msgr2=0x7fae8c13c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:29.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae8a59c640 1 --2- 172.21.15.121:0/4207556803 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae8c09ee70 0x7fae8c13c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:29.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae8a59c640 10 monclient: _finish_hunting 0 2024-09-17T18:37:29.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae8a59c640 1 monclient: found mon.noname-c 2024-09-17T18:37:29.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae8a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:29.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae8a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:37:29.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae8a59c640 1 -- 172.21.15.121:0/4207556803 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae8c127db0 con 0x7fae8c09e550 2024-09-17T18:37:29.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.005+0000 7fae89d9b640 1 --2- 172.21.15.121:0/4207556803 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae8c09ee70 0x7fae8c13c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:37:29.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.007+0000 7fae8a59c640 1 -- 172.21.15.121:0/4207556803 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fae8c09e550 msgr2=0x7fae8c13c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:29.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.007+0000 7fae8a59c640 1 --2- 172.21.15.121:0/4207556803 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fae8c09e550 0x7fae8c13c1f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fae7c00e850 tx=0x7fae7c00ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:29.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.007+0000 7fae7affd640 10 client.?.objecter ms_handle_connect 0x7fae8c09e550 2024-09-17T18:37:29.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.007+0000 7fae7affd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:37:29.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.007+0000 7fae7affd640 1 -- 172.21.15.121:0/4207556803 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae7c017070 con 0x7fae8c09e550 2024-09-17T18:37:29.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.007+0000 7fae7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:29.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.007+0000 7fae7affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:37:29.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.007+0000 7fae7affd640 10 monclient: dump: 2024-09-17T18:37:29.020 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:29.020 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:29.020 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:29.020 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:29.020 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:29.020 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:29.020 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:29.020 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:29.020 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:29.020 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:29.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.007+0000 7fae7affd640 10 monclient: _finish_auth 0 2024-09-17T18:37:29.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.007+0000 7fae7affd640 10 monclient: _check_auth_tickets 2024-09-17T18:37:29.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.007+0000 7fae7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:29.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.007+0000 7fae90db4640 5 monclient: authenticate success, global_id 64637 2024-09-17T18:37:29.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.007+0000 7fae7affd640 1 -- 172.21.15.121:0/4207556803 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae7c00cb50 con 0x7fae8c09e550 2024-09-17T18:37:29.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.007+0000 7fae90db4640 10 monclient: _renew_subs 2024-09-17T18:37:29.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.007+0000 7fae90db4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:37:29.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.007+0000 7fae90db4640 1 -- 172.21.15.121:0/4207556803 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae8c0a2530 con 0x7fae8c09e550 2024-09-17T18:37:29.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.007+0000 7fae7affd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:29.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.007+0000 7fae7affd640 1 -- 172.21.15.121:0/4207556803 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae7c013470 con 0x7fae8c09e550 2024-09-17T18:37:29.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.007+0000 7fae7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:29.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.007+0000 7fae7affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:37:29.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.007+0000 7fae7affd640 10 monclient: dump: 2024-09-17T18:37:29.024 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:29.024 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:29.025 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:29.025 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:29.025 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:29.025 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:29.025 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:29.025 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:29.025 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:29.025 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:29.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.007+0000 7fae90db4640 10 client.64637.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:37:29.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.007+0000 7fae90db4640 10 monclient: _renew_subs 2024-09-17T18:37:29.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.007+0000 7fae90db4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:37:29.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.007+0000 7fae90db4640 1 -- 172.21.15.121:0/4207556803 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fae8c1277f0 con 0x7fae8c09e550 2024-09-17T18:37:29.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.007+0000 7fae90db4640 1 librados: init done 2024-09-17T18:37:29.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.007+0000 7fae88d99640 4 set_mon_vals no callback set 2024-09-17T18:37:29.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.008+0000 7fae90db4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:29.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.008+0000 7fae90db4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:29.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.008+0000 7fae90db4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:37:29.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.008+0000 7fae90db4640 1 -- 172.21.15.121:0/4207556803 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae8c09e930 con 0x7fae8c09e550 2024-09-17T18:37:29.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.008+0000 7fae7affd640 1 -- 172.21.15.121:0/4207556803 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fae7c01a070 con 0x7fae8c09e550 2024-09-17T18:37:29.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.009+0000 7fae7affd640 1 --2- 172.21.15.121:0/4207556803 >> v2:172.21.15.173:6812/2423786037 conn(0x7fae6c0401a0 0x7fae6c042660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:29.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.009+0000 7fae7affd640 1 -- 172.21.15.121:0/4207556803 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(679..679 src has 247..679) ==== 4685+0+0 (secure 0 0 0) 0x7fae7c053480 con 0x7fae8c09e550 2024-09-17T18:37:29.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.009+0000 7fae7affd640 10 client.64637.objecter ms_dispatch 0x7fae8c000b90 osd_map(679..679 src has 247..679) 2024-09-17T18:37:29.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.009+0000 7fae7affd640 3 client.64637.objecter handle_osd_map got epochs [679,679] > 0 2024-09-17T18:37:29.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.009+0000 7fae7affd640 3 client.64637.objecter handle_osd_map decoding full epoch 679 2024-09-17T18:37:29.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.009+0000 7fae7affd640 20 client.64637.objecter dump_active .. 0 homeless 2024-09-17T18:37:29.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.009+0000 7fae89d9b640 1 --2- 172.21.15.121:0/4207556803 >> v2:172.21.15.173:6812/2423786037 conn(0x7fae6c0401a0 0x7fae6c042660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:29.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.009+0000 7fae89d9b640 10 monclient: get_auth_request con 0x7fae6c0401a0 auth_method 0 2024-09-17T18:37:29.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.012+0000 7fae7affd640 10 client.64637.objecter ms_handle_connect 0x7fae6c0401a0 2024-09-17T18:37:29.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.012+0000 7fae89d9b640 1 --2- 172.21.15.121:0/4207556803 >> v2:172.21.15.173:6812/2423786037 conn(0x7fae6c0401a0 0x7fae6c042660 secure :-1 s=READY pgs=4105 cs=0 l=1 rev1=1 crypto rx=0x7fae8002a0a0 tx=0x7fae8002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:29.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.013+0000 7fae7affd640 1 -- 172.21.15.121:0/4207556803 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fae7c023d00 con 0x7fae8c09e550 2024-09-17T18:37:29.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.013+0000 7fae7affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:29.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.013+0000 7fae7affd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:37:29.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.186+0000 7fae90db4640 1 -- 172.21.15.121:0/4207556803 --> v2:172.21.15.173:6812/2423786037 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7fae8c13ccd0 con 0x7fae6c0401a0 2024-09-17T18:37:29.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.187+0000 7fae7affd640 1 -- 172.21.15.121:0/4207556803 <== mgr.4105 v2:172.21.15.173:6812/2423786037 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fae8c13ccd0 con 0x7fae6c0401a0 2024-09-17T18:37:29.189 INFO:teuthology.orchestra.run.smithi121.stderr:instructed osd(s) 2 to deep-scrub 2024-09-17T18:37:29.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.189+0000 7fae90db4640 10 librados: watch_flush enter 2024-09-17T18:37:29.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.189+0000 7fae90db4640 10 librados: watch_flush exit 2024-09-17T18:37:29.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.189+0000 7fae90db4640 20 client.64637.objecter shutdown clearing up homeless session... 2024-09-17T18:37:29.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.189+0000 7fae90db4640 10 client.64637.objecter successfully canceled tick 2024-09-17T18:37:29.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.189+0000 7fae90db4640 1 -- 172.21.15.121:0/4207556803 >> v2:172.21.15.173:6812/2423786037 conn(0x7fae6c0401a0 msgr2=0x7fae6c042660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:29.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.189+0000 7fae90db4640 1 --2- 172.21.15.121:0/4207556803 >> v2:172.21.15.173:6812/2423786037 conn(0x7fae6c0401a0 0x7fae6c042660 secure :-1 s=READY pgs=4105 cs=0 l=1 rev1=1 crypto rx=0x7fae8002a0a0 tx=0x7fae8002e000 comp rx=0 tx=0).stop 2024-09-17T18:37:29.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.189+0000 7fae90db4640 10 monclient: shutdown 2024-09-17T18:37:29.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.189+0000 7fae90db4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:29.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.189+0000 7fae90db4640 1 -- 172.21.15.121:0/4207556803 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fae8c09e550 msgr2=0x7fae8c13c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:29.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.189+0000 7fae90db4640 1 --2- 172.21.15.121:0/4207556803 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fae8c09e550 0x7fae8c13c1f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fae7c00e850 tx=0x7fae7c00ed10 comp rx=0 tx=0).stop 2024-09-17T18:37:29.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.190+0000 7fae90db4640 1 -- 172.21.15.121:0/4207556803 shutdown_connections 2024-09-17T18:37:29.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.190+0000 7fae90db4640 1 --2- 172.21.15.121:0/4207556803 >> v2:172.21.15.173:6812/2423786037 conn(0x7fae6c0401a0 0x7fae6c042660 unknown :-1 s=CLOSED pgs=4105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:29.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.190+0000 7fae90db4640 1 --2- 172.21.15.121:0/4207556803 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae8c09f2c0 0x7fae8c125070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:29.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.190+0000 7fae90db4640 1 --2- 172.21.15.121:0/4207556803 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae8c09ee70 0x7fae8c13c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:29.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.190+0000 7fae90db4640 1 --2- 172.21.15.121:0/4207556803 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fae8c09e550 0x7fae8c13c1f0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:29.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.190+0000 7fae90db4640 1 -- 172.21.15.121:0/4207556803 >> 172.21.15.121:0/4207556803 conn(0x7fae8c0a7830 msgr2=0x7fae8c072d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:29.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.190+0000 7fae90db4640 1 -- 172.21.15.121:0/4207556803 shutdown_connections 2024-09-17T18:37:29.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.190+0000 7fae90db4640 1 -- 172.21.15.121:0/4207556803 wait complete. 2024-09-17T18:37:29.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.190+0000 7fae90db4640 1 librados: shutdown 2024-09-17T18:37:29.202 INFO:tasks.ceph:Scrubbing osd.3 2024-09-17T18:37:29.202 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-17T18:37:29.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.301+0000 7fbf3fbf8640 10 monclient: get_monmap_and_config 2024-09-17T18:37:29.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.301+0000 7fbf3fbf8640 10 monclient: build_initial_monmap 2024-09-17T18:37:29.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.301+0000 7fbf3fbf8640 10 monclient: monmap: 2024-09-17T18:37:29.303 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:29.303 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:29.303 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:37:29.302269+0000 2024-09-17T18:37:29.303 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:37:29.302269+0000 2024-09-17T18:37:29.304 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:29.304 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:29.304 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:37:29.304 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:37:29.304 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:37:29.304 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:37:29.304 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:37:29.304 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:29.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.302+0000 7fbf3fbf8640 1 Processor -- start 2024-09-17T18:37:29.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.302+0000 7fbf3fbf8640 1 -- start start 2024-09-17T18:37:29.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.302+0000 7fbf3fbf8640 10 monclient: init 2024-09-17T18:37:29.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.302+0000 7fbf3fbf8640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:29.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.302+0000 7fbf3fbf8640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-17T18:37:29.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.303+0000 7fbf3fbf8640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fbf38136670 0x7fbf381360a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:29.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.303+0000 7fbf3fbf8640 10 monclient(hunting): picked mon.noname-c con 0x7fbf38136670 addr v2:172.21.15.173:3300/0 2024-09-17T18:37:29.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.303+0000 7fbf3fbf8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbf38136a50 addr v1:172.21.15.121:6789/0 2024-09-17T18:37:29.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.303+0000 7fbf3fbf8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbf380a2080 addr v1:172.21.15.173:6789/0 2024-09-17T18:37:29.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.303+0000 7fbf3fbf8640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf380a2fe0 con 0x7fbf38136a50 2024-09-17T18:37:29.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.303+0000 7fbf3fbf8640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf380a2a20 con 0x7fbf380a2080 2024-09-17T18:37:29.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.303+0000 7fbf3fbf8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:29.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.303+0000 7fbf3fbf8640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fbf380a32c0 con 0x7fbf38136670 2024-09-17T18:37:29.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.303+0000 7fbf3fbf8640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:29.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.303+0000 7fbf3fbf8640 10 monclient(hunting): authenticate will time out at 1853.019124s 2024-09-17T18:37:29.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.303+0000 7fbf3d193640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fbf38136a50 0x7fbf380a1970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:52750/0 (socket says 172.21.15.121:52750) 2024-09-17T18:37:29.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.303+0000 7fbf3d193640 1 -- 172.21.15.121:0/2614922671 learned_addr learned my addr 172.21.15.121:0/2614922671 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:37:29.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.303+0000 7fbf3d994640 1 --2- 172.21.15.121:0/2614922671 >> v2:172.21.15.173:3300/0 conn(0x7fbf38136670 0x7fbf381360a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:29.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.303+0000 7fbf3d994640 10 monclient(hunting): get_auth_request con 0x7fbf38136670 auth_method 0 2024-09-17T18:37:29.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.303+0000 7fbf3d994640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:29.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.303+0000 7fbf3d994640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:29.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.303+0000 7fbf3d994640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:29.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.303+0000 7fbf3d994640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:29.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.303+0000 7fbf3d994640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:29.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.303+0000 7fbf3d994640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:29.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.303+0000 7fbf3d994640 10 monclient(hunting): handle_auth_done global_id 84358 payload 274 2024-09-17T18:37:29.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.303+0000 7fbf3d994640 1 -- 172.21.15.121:0/2614922671 >> v1:172.21.15.173:6789/0 conn(0x7fbf380a2080 legacy=0x7fbf3813e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:29.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.303+0000 7fbf3d994640 1 -- 172.21.15.121:0/2614922671 >> v1:172.21.15.121:6789/0 conn(0x7fbf38136a50 legacy=0x7fbf380a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:29.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.303+0000 7fbf3d994640 10 monclient: _finish_hunting 0 2024-09-17T18:37:29.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.303+0000 7fbf3d994640 1 monclient: found mon.noname-c 2024-09-17T18:37:29.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.303+0000 7fbf3d994640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:29.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.303+0000 7fbf3d994640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.173:3300/0 2024-09-17T18:37:29.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.303+0000 7fbf3d994640 1 -- 172.21.15.121:0/2614922671 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf380a2d00 con 0x7fbf38136670 2024-09-17T18:37:29.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.306+0000 7fbf3d994640 1 -- 172.21.15.121:0/2614922671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf38136670 msgr2=0x7fbf381360a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:29.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.306+0000 7fbf3d994640 1 --2- 172.21.15.121:0/2614922671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf38136670 0x7fbf381360a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fbf380a3440 tx=0x7fbf3402a040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=acd689583e4e79ea server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:29.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.307+0000 7fbf3c992640 1 -- 172.21.15.121:0/2614922671 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf34037070 con 0x7fbf38136670 2024-09-17T18:37:29.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.307+0000 7fbf3c992640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:29.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.307+0000 7fbf3c992640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:37:29.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.307+0000 7fbf3c992640 10 monclient: dump: 2024-09-17T18:37:29.313 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:29.313 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:29.313 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:29.313 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:29.313 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:29.314 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:29.314 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:29.314 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:29.314 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:29.314 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:29.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.307+0000 7fbf3c992640 10 monclient: _finish_auth 0 2024-09-17T18:37:29.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.307+0000 7fbf3c992640 10 monclient: _check_auth_tickets 2024-09-17T18:37:29.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.307+0000 7fbf3c992640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:29.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.307+0000 7fbf3c992640 1 -- 172.21.15.121:0/2614922671 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf34032640 con 0x7fbf38136670 2024-09-17T18:37:29.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.307+0000 7fbf3fbf8640 5 monclient: authenticate success, global_id 84358 2024-09-17T18:37:29.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.307+0000 7fbf3fbf8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:37:29.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.307+0000 7fbf3c992640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:29.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.307+0000 7fbf3fbf8640 10 monclient: get_monmap_and_config success 2024-09-17T18:37:29.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.307+0000 7fbf3fbf8640 4 set_mon_vals no callback set 2024-09-17T18:37:29.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.307+0000 7fbf3c992640 1 -- 172.21.15.121:0/2614922671 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf34032a10 con 0x7fbf38136670 2024-09-17T18:37:29.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.307+0000 7fbf3c992640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:29.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.307+0000 7fbf3c992640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:37:29.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.307+0000 7fbf3c992640 10 monclient: dump: 2024-09-17T18:37:29.315 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:29.315 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:29.315 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:29.315 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:29.315 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:29.315 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:29.315 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:29.315 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:29.315 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:29.315 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:29.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.307+0000 7fbf3fbf8640 10 monclient: shutdown 2024-09-17T18:37:29.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.307+0000 7fbf3fbf8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:29.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.307+0000 7fbf3fbf8640 1 -- 172.21.15.121:0/2614922671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf38136670 msgr2=0x7fbf381360a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:29.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.307+0000 7fbf3fbf8640 1 --2- 172.21.15.121:0/2614922671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf38136670 0x7fbf381360a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fbf380a3440 tx=0x7fbf3402a040 comp rx=0 tx=0).stop 2024-09-17T18:37:29.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.307+0000 7fbf3fbf8640 1 -- 172.21.15.121:0/2614922671 shutdown_connections 2024-09-17T18:37:29.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.307+0000 7fbf3fbf8640 1 --2- 172.21.15.121:0/2614922671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf38136670 0x7fbf381360a0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:29.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.307+0000 7fbf3fbf8640 1 -- 172.21.15.121:0/2614922671 >> 172.21.15.121:0/2614922671 conn(0x7fbf380a3a70 msgr2=0x7fbf380a3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:29.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.308+0000 7fbf3fbf8640 1 -- 172.21.15.121:0/2614922671 shutdown_connections 2024-09-17T18:37:29.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.308+0000 7fbf3fbf8640 1 -- 172.21.15.121:0/2614922671 wait complete. 2024-09-17T18:37:29.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.308+0000 7fbf3fbf8640 10 monclient: build_initial_monmap 2024-09-17T18:37:29.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.308+0000 7fbf3fbf8640 10 monclient: monmap: 2024-09-17T18:37:29.316 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:29.316 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:37:29.316 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:37:29.316 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:37:29.316 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:29.316 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:29.317 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:37:29.317 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:37:29.317 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:37:29.317 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:29.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.308+0000 7fbf3fbf8640 1 librados: starting msgr at 2024-09-17T18:37:29.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.308+0000 7fbf3fbf8640 1 librados: starting objecter 2024-09-17T18:37:29.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.308+0000 7fbf3fbf8640 1 Processor -- start 2024-09-17T18:37:29.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3fbf8640 1 -- start start 2024-09-17T18:37:29.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3fbf8640 1 librados: setting wanted keys 2024-09-17T18:37:29.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3fbf8640 1 librados: calling monclient init 2024-09-17T18:37:29.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3fbf8640 10 monclient: init 2024-09-17T18:37:29.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3fbf8640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:29.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3fbf8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T18:37:29.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3fbf8640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbf380a2080 0x7fbf38073880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:29.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3fbf8640 10 monclient(hunting): picked mon.noname-c con 0x7fbf380a2080 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:29.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3fbf8640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbf38136670 0x7fbf3809e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:29.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3fbf8640 10 monclient(hunting): picked mon.noname-a con 0x7fbf38136670 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:37:29.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3fbf8640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf38136a50 0x7fbf3809e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:29.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3fbf8640 10 monclient(hunting): picked mon.noname-b con 0x7fbf38136a50 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:37:29.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3fbf8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:29.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3fbf8640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fbf38077a60 con 0x7fbf38136670 2024-09-17T18:37:29.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3fbf8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:29.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3fbf8640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbf380778e0 con 0x7fbf38136a50 2024-09-17T18:37:29.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3fbf8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:29.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3fbf8640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbf38136e50 con 0x7fbf380a2080 2024-09-17T18:37:29.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3fbf8640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:29.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3fbf8640 10 monclient(hunting): authenticate will time out at 1853.025523s 2024-09-17T18:37:29.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3d994640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbf380a2080 0x7fbf38073880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:29.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3d193640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbf38136670 0x7fbf3809e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:29.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3d994640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbf380a2080 0x7fbf38073880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:58112/0 (socket says 172.21.15.121:58112) 2024-09-17T18:37:29.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3d994640 1 -- 172.21.15.121:0/4234925 learned_addr learned my addr 172.21.15.121:0/4234925 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:37:29.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3d994640 10 monclient(hunting): get_auth_request con 0x7fbf380a2080 auth_method 0 2024-09-17T18:37:29.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3d994640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:29.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3d994640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:29.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3d994640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:29.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3d193640 1 --2- 172.21.15.121:0/4234925 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbf38136670 0x7fbf3809e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38470/0 (socket says 172.21.15.121:38470) 2024-09-17T18:37:29.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3d193640 10 monclient(hunting): get_auth_request con 0x7fbf38136670 auth_method 0 2024-09-17T18:37:29.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3d193640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:29.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3d193640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:29.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3d193640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:29.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3d994640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:29.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3d994640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:29.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3d994640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:29.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3d193640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:29.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3d193640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:29.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3d193640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:29.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3e195640 1 --2- 172.21.15.121:0/4234925 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf38136a50 0x7fbf3809e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:29.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3e195640 10 monclient(hunting): get_auth_request con 0x7fbf38136a50 auth_method 0 2024-09-17T18:37:29.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3e195640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:29.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3e195640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:29.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3e195640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:29.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3e195640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:29.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3e195640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:29.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3e195640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:29.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3d994640 10 monclient(hunting): handle_auth_done global_id 64640 payload 824 2024-09-17T18:37:29.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3d994640 1 -- 172.21.15.121:0/4234925 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf38136a50 msgr2=0x7fbf3809e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:29.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3d994640 1 --2- 172.21.15.121:0/4234925 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf38136a50 0x7fbf3809e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:29.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3d994640 1 -- 172.21.15.121:0/4234925 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbf38136670 msgr2=0x7fbf3809e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:29.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3d994640 1 --2- 172.21.15.121:0/4234925 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbf38136670 0x7fbf3809e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:29.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3d994640 10 monclient: _finish_hunting 0 2024-09-17T18:37:29.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3d994640 1 monclient: found mon.noname-c 2024-09-17T18:37:29.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3d994640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:29.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3d994640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:37:29.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.309+0000 7fbf3d994640 1 -- 172.21.15.121:0/4234925 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf38071e60 con 0x7fbf380a2080 2024-09-17T18:37:29.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.310+0000 7fbf3d193640 1 --2- 172.21.15.121:0/4234925 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbf38136670 0x7fbf3809e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:37:29.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.312+0000 7fbf3d994640 1 -- 172.21.15.121:0/4234925 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbf380a2080 msgr2=0x7fbf38073880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:29.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.312+0000 7fbf3d994640 1 --2- 172.21.15.121:0/4234925 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbf380a2080 0x7fbf38073880 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fbf34009890 tx=0x7fbf340046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:29.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.312+0000 7fbf2e7fc640 10 client.?.objecter ms_handle_connect 0x7fbf380a2080 2024-09-17T18:37:29.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.312+0000 7fbf2e7fc640 10 client.?.objecter resend_mon_ops 2024-09-17T18:37:29.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.312+0000 7fbf2e7fc640 1 -- 172.21.15.121:0/4234925 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf34042020 con 0x7fbf380a2080 2024-09-17T18:37:29.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.312+0000 7fbf2e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:29.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.312+0000 7fbf2e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:37:29.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.312+0000 7fbf2e7fc640 10 monclient: dump: 2024-09-17T18:37:29.324 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:29.324 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:29.324 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:29.324 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:29.324 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:29.324 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:29.324 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:29.325 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:29.325 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:29.325 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:29.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.312+0000 7fbf2e7fc640 10 monclient: _finish_auth 0 2024-09-17T18:37:29.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.312+0000 7fbf2e7fc640 10 monclient: _check_auth_tickets 2024-09-17T18:37:29.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.312+0000 7fbf2e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:29.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.312+0000 7fbf2e7fc640 1 -- 172.21.15.121:0/4234925 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf34030800 con 0x7fbf380a2080 2024-09-17T18:37:29.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.312+0000 7fbf2e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:29.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.312+0000 7fbf2e7fc640 1 -- 172.21.15.121:0/4234925 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf34037040 con 0x7fbf380a2080 2024-09-17T18:37:29.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.312+0000 7fbf3fbf8640 5 monclient: authenticate success, global_id 64640 2024-09-17T18:37:29.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.312+0000 7fbf3fbf8640 10 monclient: _renew_subs 2024-09-17T18:37:29.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.312+0000 7fbf3fbf8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:37:29.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.312+0000 7fbf3fbf8640 1 -- 172.21.15.121:0/4234925 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf38072140 con 0x7fbf380a2080 2024-09-17T18:37:29.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.312+0000 7fbf2ffff640 4 set_mon_vals no callback set 2024-09-17T18:37:29.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.312+0000 7fbf2e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:29.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.312+0000 7fbf2e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:37:29.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.312+0000 7fbf2e7fc640 10 monclient: dump: 2024-09-17T18:37:29.326 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:29.326 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:29.326 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:29.326 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:29.326 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:29.326 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:29.326 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:29.326 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:29.327 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:29.327 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:29.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.312+0000 7fbf3fbf8640 10 client.64640.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:37:29.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.312+0000 7fbf3fbf8640 10 monclient: _renew_subs 2024-09-17T18:37:29.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.312+0000 7fbf3fbf8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:37:29.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.312+0000 7fbf3fbf8640 1 -- 172.21.15.121:0/4234925 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf380a0350 con 0x7fbf380a2080 2024-09-17T18:37:29.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.312+0000 7fbf3fbf8640 1 librados: init done 2024-09-17T18:37:29.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.313+0000 7fbf3fbf8640 10 client.64640.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:29.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.313+0000 7fbf3fbf8640 15 client.64640.objecter _session_command_op_assign -1 1 2024-09-17T18:37:29.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.313+0000 7fbf3fbf8640 20 client.64640.objecter _get_session osd=-1 returning homeless 2024-09-17T18:37:29.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.313+0000 7fbf3fbf8640 10 client.64640.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:37:29.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.313+0000 7fbf3fbf8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:37:29.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.313+0000 7fbf3fbf8640 1 -- 172.21.15.121:0/4234925 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbf380a24b0 con 0x7fbf380a2080 2024-09-17T18:37:29.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.313+0000 7fbf2e7fc640 1 -- 172.21.15.121:0/4234925 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fbf34045020 con 0x7fbf380a2080 2024-09-17T18:37:29.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.313+0000 7fbf2e7fc640 1 --2- 172.21.15.121:0/4234925 >> v2:172.21.15.173:6812/2423786037 conn(0x7fbf1c040170 0x7fbf1c042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:29.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.313+0000 7fbf2e7fc640 1 -- 172.21.15.121:0/4234925 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(679..679 src has 247..679) ==== 4685+0+0 (secure 0 0 0) 0x7fbf34073a90 con 0x7fbf380a2080 2024-09-17T18:37:29.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.313+0000 7fbf2e7fc640 10 client.64640.objecter ms_dispatch 0x7fbf38000b90 osd_map(679..679 src has 247..679) 2024-09-17T18:37:29.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.313+0000 7fbf2e7fc640 3 client.64640.objecter handle_osd_map got epochs [679,679] > 0 2024-09-17T18:37:29.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.313+0000 7fbf2e7fc640 3 client.64640.objecter handle_osd_map decoding full epoch 679 2024-09-17T18:37:29.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.313+0000 7fbf2e7fc640 10 client.64640.objecter checking command 1 2024-09-17T18:37:29.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.313+0000 7fbf2e7fc640 1 --2- 172.21.15.121:0/4234925 >> v2:172.21.15.173:6800/3493964597 conn(0x7fbf1c047630 0x7fbf1c049ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:29.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.313+0000 7fbf2e7fc640 20 client.64640.objecter _get_session s=0x7fbf1c046f80 osd=3 3 2024-09-17T18:37:29.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.313+0000 7fbf2e7fc640 20 client.64640.objecter put_session s=0x7fbf1c046f80 osd=3 3 2024-09-17T18:37:29.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.313+0000 7fbf2e7fc640 15 client.64640.objecter _session_command_op_remove -1 1 2024-09-17T18:37:29.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.313+0000 7fbf2e7fc640 20 client.64640.objecter _get_session s=0x7fbf1c046f80 osd=3 3 2024-09-17T18:37:29.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.313+0000 7fbf2e7fc640 20 client.64640.objecter get_session s=0x7fbf1c046f80 osd=3 3 2024-09-17T18:37:29.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.313+0000 7fbf2e7fc640 15 client.64640.objecter _session_command_op_assign 3 1 2024-09-17T18:37:29.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.313+0000 7fbf2e7fc640 20 client.64640.objecter put_session s=0x7fbf1c046f80 osd=3 4 2024-09-17T18:37:29.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.313+0000 7fbf2e7fc640 10 client.64640.objecter _send_command 1 2024-09-17T18:37:29.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.313+0000 7fbf2e7fc640 1 -- 172.21.15.121:0/4234925 --> v2:172.21.15.173:6800/3493964597 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbf34008eb0 con 0x7fbf1c047630 2024-09-17T18:37:29.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.313+0000 7fbf2e7fc640 20 client.64640.objecter dump_active .. 0 homeless 2024-09-17T18:37:29.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.313+0000 7fbf2e7fc640 1 -- 172.21.15.121:0/4234925 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_get_version_reply(handle=1 version=679) ==== 24+0+0 (secure 0 0 0) 0x7fbf340782a0 con 0x7fbf380a2080 2024-09-17T18:37:29.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.313+0000 7fbf2e7fc640 10 monclient: handle_get_version_reply finishing 1 version 679 2024-09-17T18:37:29.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.313+0000 7fbf3d193640 1 --2- 172.21.15.121:0/4234925 >> v2:172.21.15.173:6812/2423786037 conn(0x7fbf1c040170 0x7fbf1c042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:29.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.314+0000 7fbf3d193640 10 monclient: get_auth_request con 0x7fbf1c040170 auth_method 0 2024-09-17T18:37:29.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.314+0000 7fbf3e195640 1 --2- 172.21.15.121:0/4234925 >> v2:172.21.15.173:6800/3493964597 conn(0x7fbf1c047630 0x7fbf1c049ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:29.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.314+0000 7fbf3e195640 10 monclient: get_auth_request con 0x7fbf1c047630 auth_method 0 2024-09-17T18:37:29.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.316+0000 7fbf2e7fc640 10 client.64640.objecter ms_handle_connect 0x7fbf1c040170 2024-09-17T18:37:29.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.316+0000 7fbf3d193640 1 --2- 172.21.15.121:0/4234925 >> v2:172.21.15.173:6812/2423786037 conn(0x7fbf1c040170 0x7fbf1c042630 secure :-1 s=READY pgs=4106 cs=0 l=1 rev1=1 crypto rx=0x7fbf28007c90 tx=0x7fbf28007be0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:29.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.318+0000 7fbf3e195640 1 --2- 172.21.15.121:0/4234925 >> v2:172.21.15.173:6800/3493964597 conn(0x7fbf1c047630 0x7fbf1c049ad0 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:29.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.318+0000 7fbf2e7fc640 10 client.64640.objecter ms_handle_connect 0x7fbf1c047630 2024-09-17T18:37:29.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.319+0000 7fbf2e7fc640 1 -- 172.21.15.121:0/4234925 <== osd.3 v2:172.21.15.173:6800/3493964597 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbf34008eb0 con 0x7fbf1c047630 2024-09-17T18:37:29.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.319+0000 7fbf2e7fc640 10 client.64640.objecter ms_dispatch 0x7fbf38000b90 command_reply(tid 1: 0 ) 2024-09-17T18:37:29.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.319+0000 7fbf2e7fc640 10 client.64640.objecter _finish_command 1 = system:0 2024-09-17T18:37:29.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.319+0000 7fbf2e7fc640 20 client.64640.objecter put_session s=0x7fbf1c046f80 osd=3 4 2024-09-17T18:37:29.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.319+0000 7fbf2e7fc640 15 client.64640.objecter _session_command_op_remove 3 1 2024-09-17T18:37:29.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.331+0000 7fbf3fbf8640 10 client.64640.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-17T18:37:29.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.331+0000 7fbf3fbf8640 15 client.64640.objecter _session_command_op_assign -1 2 2024-09-17T18:37:29.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.331+0000 7fbf3fbf8640 20 client.64640.objecter _get_session s=0x7fbf1c046f80 osd=3 3 2024-09-17T18:37:29.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.331+0000 7fbf3fbf8640 20 client.64640.objecter put_session s=0x7fbf1c046f80 osd=3 3 2024-09-17T18:37:29.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.331+0000 7fbf3fbf8640 20 client.64640.objecter _get_session s=0x7fbf1c046f80 osd=3 3 2024-09-17T18:37:29.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.331+0000 7fbf3fbf8640 15 client.64640.objecter _session_command_op_remove -1 2 2024-09-17T18:37:29.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.331+0000 7fbf3fbf8640 20 client.64640.objecter get_session s=0x7fbf1c046f80 osd=3 3 2024-09-17T18:37:29.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.331+0000 7fbf3fbf8640 15 client.64640.objecter _session_command_op_assign 3 2 2024-09-17T18:37:29.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.331+0000 7fbf3fbf8640 20 client.64640.objecter put_session s=0x7fbf1c046f80 osd=3 4 2024-09-17T18:37:29.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.331+0000 7fbf3fbf8640 10 client.64640.objecter _send_command 2 2024-09-17T18:37:29.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.331+0000 7fbf3fbf8640 1 -- 172.21.15.121:0/4234925 --> v2:172.21.15.173:6800/3493964597 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fbf38073f90 con 0x7fbf1c047630 2024-09-17T18:37:29.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.332+0000 7fbf2e7fc640 1 -- 172.21.15.121:0/4234925 <== osd.3 v2:172.21.15.173:6800/3493964597 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fbf38073f90 con 0x7fbf1c047630 2024-09-17T18:37:29.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.332+0000 7fbf2e7fc640 10 client.64640.objecter ms_dispatch 0x7fbf38000b90 command_reply(tid 2: 0 ) 2024-09-17T18:37:29.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.332+0000 7fbf2e7fc640 10 client.64640.objecter _finish_command 2 = system:0 2024-09-17T18:37:29.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.332+0000 7fbf2e7fc640 20 client.64640.objecter put_session s=0x7fbf1c046f80 osd=3 4 2024-09-17T18:37:29.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.332+0000 7fbf2e7fc640 15 client.64640.objecter _session_command_op_remove 3 2 2024-09-17T18:37:29.335 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:37:29.335 INFO:teuthology.orchestra.run.smithi121.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T18:37:29.335 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:37:29.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.332+0000 7fbf3fbf8640 10 librados: watch_flush enter 2024-09-17T18:37:29.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.332+0000 7fbf3fbf8640 10 librados: watch_flush exit 2024-09-17T18:37:29.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.332+0000 7fbf3fbf8640 10 client.64640.objecter close_session for osd.3 2024-09-17T18:37:29.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.332+0000 7fbf3fbf8640 1 -- 172.21.15.121:0/4234925 >> v2:172.21.15.173:6800/3493964597 conn(0x7fbf1c047630 msgr2=0x7fbf1c049ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:29.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.332+0000 7fbf3fbf8640 1 --2- 172.21.15.121:0/4234925 >> v2:172.21.15.173:6800/3493964597 conn(0x7fbf1c047630 0x7fbf1c049ad0 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:29.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.332+0000 7fbf3fbf8640 20 client.64640.objecter put_session s=0x7fbf1c046f80 osd=3 1 2024-09-17T18:37:29.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.332+0000 7fbf3fbf8640 20 client.64640.objecter shutdown clearing up homeless session... 2024-09-17T18:37:29.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.332+0000 7fbf3fbf8640 10 client.64640.objecter successfully canceled tick 2024-09-17T18:37:29.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.332+0000 7fbf3fbf8640 1 -- 172.21.15.121:0/4234925 >> v2:172.21.15.173:6812/2423786037 conn(0x7fbf1c040170 msgr2=0x7fbf1c042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:29.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.332+0000 7fbf3fbf8640 1 --2- 172.21.15.121:0/4234925 >> v2:172.21.15.173:6812/2423786037 conn(0x7fbf1c040170 0x7fbf1c042630 secure :-1 s=READY pgs=4106 cs=0 l=1 rev1=1 crypto rx=0x7fbf28007c90 tx=0x7fbf28007be0 comp rx=0 tx=0).stop 2024-09-17T18:37:29.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.332+0000 7fbf3fbf8640 10 monclient: shutdown 2024-09-17T18:37:29.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.332+0000 7fbf3fbf8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:29.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.332+0000 7fbf3fbf8640 1 -- 172.21.15.121:0/4234925 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbf380a2080 msgr2=0x7fbf38073880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:29.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.332+0000 7fbf3fbf8640 1 --2- 172.21.15.121:0/4234925 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbf380a2080 0x7fbf38073880 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fbf34009890 tx=0x7fbf340046c0 comp rx=0 tx=0).stop 2024-09-17T18:37:29.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.332+0000 7fbf3e195640 1 -- 172.21.15.121:0/4234925 reap_dead start 2024-09-17T18:37:29.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.332+0000 7fbf3fbf8640 1 -- 172.21.15.121:0/4234925 shutdown_connections 2024-09-17T18:37:29.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.332+0000 7fbf3fbf8640 1 -- 172.21.15.121:0/4234925 >> 172.21.15.121:0/4234925 conn(0x7fbf380a3a70 msgr2=0x7fbf38131240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:29.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.332+0000 7fbf3fbf8640 1 -- 172.21.15.121:0/4234925 shutdown_connections 2024-09-17T18:37:29.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.332+0000 7fbf3fbf8640 1 -- 172.21.15.121:0/4234925 wait complete. 2024-09-17T18:37:29.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.332+0000 7fbf3fbf8640 1 librados: shutdown 2024-09-17T18:37:29.344 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-17T18:37:29.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.440+0000 7f70ac42a640 10 monclient: get_monmap_and_config 2024-09-17T18:37:29.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.440+0000 7f70ac42a640 10 monclient: build_initial_monmap 2024-09-17T18:37:29.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.440+0000 7f70ac42a640 10 monclient: monmap: 2024-09-17T18:37:29.442 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:29.442 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:29.442 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:37:29.441132+0000 2024-09-17T18:37:29.442 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:37:29.441132+0000 2024-09-17T18:37:29.442 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:29.442 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:29.442 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:37:29.442 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:37:29.443 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:37:29.443 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:37:29.443 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:37:29.443 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:29.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.441+0000 7f70ac42a640 1 Processor -- start 2024-09-17T18:37:29.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.441+0000 7f70ac42a640 1 -- start start 2024-09-17T18:37:29.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.441+0000 7f70ac42a640 10 monclient: init 2024-09-17T18:37:29.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.441+0000 7f70ac42a640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:29.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.441+0000 7f70ac42a640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-17T18:37:29.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.441+0000 7f70ac42a640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f70a409e550 0x7f70a409e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:29.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.441+0000 7f70ac42a640 10 monclient(hunting): picked mon.noname-a con 0x7f70a409e550 addr v2:172.21.15.121:3300/0 2024-09-17T18:37:29.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.441+0000 7f70ac42a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70a409ee70 0x7f70a4139840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:29.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.441+0000 7f70ac42a640 10 monclient(hunting): picked mon.noname-b con 0x7f70a409ee70 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:29.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.441+0000 7f70ac42a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f70a409f2c0 addr v1:172.21.15.121:6789/0 2024-09-17T18:37:29.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.441+0000 7f70ac42a640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70a40a4460 con 0x7f70a409f2c0 2024-09-17T18:37:29.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.441+0000 7f70ac42a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:29.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.441+0000 7f70ac42a640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f70a40a4950 con 0x7f70a409e550 2024-09-17T18:37:29.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.441+0000 7f70ac42a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:29.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.441+0000 7f70ac42a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f70a40a2b60 con 0x7f70a409ee70 2024-09-17T18:37:29.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.441+0000 7f70ac42a640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:29.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.441+0000 7f70ac42a640 10 monclient(hunting): authenticate will time out at 1853.157958s 2024-09-17T18:37:29.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.441+0000 7f70aa1c6640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f70a409e550 0x7f70a409e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:29.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.442+0000 7f70a99c5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70a409ee70 0x7f70a4139840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:29.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.442+0000 7f70aa1c6640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f70a409e550 0x7f70a409e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38484/0 (socket says 172.21.15.121:38484) 2024-09-17T18:37:29.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.442+0000 7f70aa1c6640 1 -- 172.21.15.121:0/858194909 learned_addr learned my addr 172.21.15.121:0/858194909 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:37:29.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.442+0000 7f70aa1c6640 10 monclient(hunting): get_auth_request con 0x7f70a409e550 auth_method 0 2024-09-17T18:37:29.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.442+0000 7f70aa1c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:29.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.442+0000 7f70aa1c6640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:29.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.442+0000 7f70aa1c6640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:29.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.442+0000 7f70a99c5640 10 monclient(hunting): get_auth_request con 0x7f70a409ee70 auth_method 0 2024-09-17T18:37:29.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.442+0000 7f70a99c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:29.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.442+0000 7f70a99c5640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:29.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.442+0000 7f70a99c5640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:29.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.442+0000 7f70aa1c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:29.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.442+0000 7f70aa1c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:29.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.442+0000 7f70aa1c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:29.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.442+0000 7f70a99c5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:29.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.442+0000 7f70a99c5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:29.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.442+0000 7f70a99c5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:29.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.442+0000 7f70a99c5640 10 monclient(hunting): handle_auth_done global_id 64643 payload 274 2024-09-17T18:37:29.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.442+0000 7f70a99c5640 1 -- 172.21.15.121:0/858194909 >> v2:172.21.15.121:3300/0 conn(0x7f70a409e550 msgr2=0x7f70a409e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:29.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.442+0000 7f70a99c5640 1 --2- 172.21.15.121:0/858194909 >> v2:172.21.15.121:3300/0 conn(0x7f70a409e550 0x7f70a409e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:29.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.442+0000 7f70a99c5640 1 -- 172.21.15.121:0/858194909 >> v1:172.21.15.121:6789/0 conn(0x7f70a409f2c0 legacy=0x7f70a413bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:29.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.442+0000 7f70a99c5640 10 monclient: _finish_hunting 0 2024-09-17T18:37:29.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.442+0000 7f70a99c5640 1 monclient: found mon.noname-b 2024-09-17T18:37:29.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.442+0000 7f70a99c5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:29.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.442+0000 7f70a99c5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:37:29.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.442+0000 7f70a99c5640 1 -- 172.21.15.121:0/858194909 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70a409f840 con 0x7f70a409ee70 2024-09-17T18:37:29.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.442+0000 7f70a99c5640 1 -- 172.21.15.121:0/858194909 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70a409ee70 msgr2=0x7f70a4139840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T18:37:29.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.442+0000 7f70a99c5640 1 -- 172.21.15.121:0/858194909 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70a409ee70 msgr2=0x7f70a4139840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T18:37:29.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.442+0000 7f70a99c5640 1 --2- 172.21.15.121:0/858194909 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70a409ee70 0x7f70a4139840 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7f70a409e0a0 tx=0x7f7094029fb0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:37:29.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.442+0000 7f70a99c5640 1 --2- 172.21.15.121:0/858194909 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70a409ee70 0x7f70a4139840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:37:29.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.643+0000 7f70a99c5640 1 --2- 172.21.15.121:0/858194909 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70a409ee70 0x7f70a4139840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:29.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.643+0000 7f70a99c5640 10 monclient: get_auth_request con 0x7f70a409ee70 auth_method 0 2024-09-17T18:37:29.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.643+0000 7f70a99c5640 0 --2- 172.21.15.121:0/858194909 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70a409ee70 0x7f70a4139840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-17T18:37:29.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.643+0000 7f70a99c5640 1 --2- 172.21.15.121:0/858194909 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70a409ee70 0x7f70a4139840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:29.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.643+0000 7f70a91c4640 10 monclient: ms_handle_reset current mon [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:29.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.643+0000 7f70a91c4640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:29.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.643+0000 7f70a91c4640 1 -- 172.21.15.121:0/858194909 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70a409ee70 msgr2=0x7f70a4139840 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-09-17T18:37:29.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.643+0000 7f70a91c4640 1 --2- 172.21.15.121:0/858194909 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70a409ee70 0x7f70a4139840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:29.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.643+0000 7f70a91c4640 10 monclient: _add_conns ranks=[4,1] 2024-09-17T18:37:29.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.643+0000 7f70a91c4640 1 --2- 172.21.15.121:0/858194909 >> v2:172.21.15.173:3300/0 conn(0x7f7090003500 0x7f7090005980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:29.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.643+0000 7f70a91c4640 10 monclient(hunting): picked mon.noname-c con 0x7f7090003500 addr v2:172.21.15.173:3300/0 2024-09-17T18:37:29.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.643+0000 7f70a91c4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f70900062b0 addr v1:172.21.15.173:6789/0 2024-09-17T18:37:29.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.643+0000 7f70a91c4640 1 -- 172.21.15.121:0/858194909 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f709000a260 con 0x7f70900062b0 2024-09-17T18:37:29.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.643+0000 7f70a91c4640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:29.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.643+0000 7f70a91c4640 1 -- 172.21.15.121:0/858194909 --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f709000b460 con 0x7f7090003500 2024-09-17T18:37:29.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.643+0000 7f70a91c4640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:29.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.643+0000 7f70a91c4640 10 monclient(hunting): ms_handle_reset stray mon [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:29.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.643+0000 7f70aa1c6640 1 --2- 172.21.15.121:0/858194909 >> v2:172.21.15.173:3300/0 conn(0x7f7090003500 0x7f7090005980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:29.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.643+0000 7f70aa1c6640 10 monclient(hunting): get_auth_request con 0x7f7090003500 auth_method 0 2024-09-17T18:37:29.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.643+0000 7f70aa1c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:29.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.643+0000 7f70aa1c6640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:29.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.643+0000 7f70aa1c6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:37:29.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.644+0000 7f70aa1c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:29.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.644+0000 7f70aa1c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:29.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.644+0000 7f70aa1c6640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:37:29.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.644+0000 7f70aa1c6640 10 monclient(hunting): handle_auth_done global_id 64643 payload 293 2024-09-17T18:37:29.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.644+0000 7f70aa1c6640 1 -- 172.21.15.121:0/858194909 >> v1:172.21.15.173:6789/0 conn(0x7f70900062b0 legacy=0x7f7090008770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:29.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.644+0000 7f70aa1c6640 10 monclient: _finish_hunting 0 2024-09-17T18:37:29.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.644+0000 7f70aa1c6640 1 monclient: found mon.noname-c 2024-09-17T18:37:29.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.644+0000 7f70aa1c6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:29.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.644+0000 7f70aa1c6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.173:3300/0 2024-09-17T18:37:29.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.644+0000 7f70aa1c6640 1 -- 172.21.15.121:0/858194909 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f709000b720 con 0x7f7090003500 2024-09-17T18:37:29.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.646+0000 7f70aa1c6640 1 -- 172.21.15.121:0/858194909 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7090003500 msgr2=0x7f7090005980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:29.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.646+0000 7f70aa1c6640 1 --2- 172.21.15.121:0/858194909 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7090003500 0x7f7090005980 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f70a000be00 tx=0x7f7094009190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7297a26f37f5c085 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:29.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.646+0000 7f70a91c4640 1 -- 172.21.15.121:0/858194909 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a001e070 con 0x7f7090003500 2024-09-17T18:37:29.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.646+0000 7f70a91c4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:29.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.646+0000 7f70a91c4640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:37:29.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.646+0000 7f70a91c4640 10 monclient: dump: 2024-09-17T18:37:29.652 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:29.652 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:29.652 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:29.652 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:29.652 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:29.652 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:29.652 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:29.652 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:29.653 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:29.653 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:29.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.646+0000 7f70a91c4640 10 monclient: _finish_auth 0 2024-09-17T18:37:29.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.646+0000 7f70a91c4640 10 monclient: _check_auth_tickets 2024-09-17T18:37:29.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.646+0000 7f70a91c4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:29.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.646+0000 7f70a91c4640 1 -- 172.21.15.121:0/858194909 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70a0011860 con 0x7f7090003500 2024-09-17T18:37:29.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.646+0000 7f70ac42a640 5 monclient: authenticate success, global_id 64643 2024-09-17T18:37:29.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.646+0000 7f70ac42a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:37:29.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.646+0000 7f70a91c4640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:29.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.646+0000 7f70ac42a640 10 monclient: get_monmap_and_config success 2024-09-17T18:37:29.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.646+0000 7f70ac42a640 4 set_mon_vals no callback set 2024-09-17T18:37:29.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.646+0000 7f70a91c4640 1 -- 172.21.15.121:0/858194909 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a000ba80 con 0x7f7090003500 2024-09-17T18:37:29.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.646+0000 7f70a91c4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:29.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.646+0000 7f70a91c4640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:37:29.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.646+0000 7f70a91c4640 10 monclient: dump: 2024-09-17T18:37:29.654 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:29.654 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:29.654 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:29.654 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:29.654 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:29.654 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:29.654 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:29.654 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:29.654 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:29.654 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:29.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.647+0000 7f70ac42a640 10 monclient: shutdown 2024-09-17T18:37:29.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.647+0000 7f70ac42a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:29.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.647+0000 7f70ac42a640 1 -- 172.21.15.121:0/858194909 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7090003500 msgr2=0x7f7090005980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:29.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.647+0000 7f70ac42a640 1 --2- 172.21.15.121:0/858194909 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7090003500 0x7f7090005980 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f70a000be00 tx=0x7f7094009190 comp rx=0 tx=0).stop 2024-09-17T18:37:29.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.647+0000 7f70aa9c7640 1 -- 172.21.15.121:0/858194909 reap_dead start 2024-09-17T18:37:29.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.647+0000 7f70ac42a640 1 -- 172.21.15.121:0/858194909 shutdown_connections 2024-09-17T18:37:29.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.647+0000 7f70ac42a640 1 --2- 172.21.15.121:0/858194909 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7090003500 0x7f7090005980 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:29.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.647+0000 7f70ac42a640 1 -- 172.21.15.121:0/858194909 >> 172.21.15.121:0/858194909 conn(0x7f70a40a7830 msgr2=0x7f70a40a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:29.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.647+0000 7f70ac42a640 1 -- 172.21.15.121:0/858194909 shutdown_connections 2024-09-17T18:37:29.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.647+0000 7f70ac42a640 1 -- 172.21.15.121:0/858194909 wait complete. 2024-09-17T18:37:29.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.648+0000 7f70ac42a640 10 monclient: build_initial_monmap 2024-09-17T18:37:29.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.648+0000 7f70ac42a640 10 monclient: monmap: 2024-09-17T18:37:29.655 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:29.655 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:37:29.655 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:37:29.655 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:37:29.655 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:29.655 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:29.655 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:37:29.655 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:37:29.656 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:37:29.656 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:29.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.648+0000 7f70ac42a640 1 librados: starting msgr at 2024-09-17T18:37:29.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.648+0000 7f70ac42a640 1 librados: starting objecter 2024-09-17T18:37:29.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.648+0000 7f70ac42a640 1 Processor -- start 2024-09-17T18:37:29.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.648+0000 7f70ac42a640 1 -- start start 2024-09-17T18:37:29.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.648+0000 7f70ac42a640 1 librados: setting wanted keys 2024-09-17T18:37:29.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.648+0000 7f70ac42a640 1 librados: calling monclient init 2024-09-17T18:37:29.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.648+0000 7f70ac42a640 10 monclient: init 2024-09-17T18:37:29.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70ac42a640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:29.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70ac42a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T18:37:29.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70ac42a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7090003500 0x7f70a41a9300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:29.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70ac42a640 10 monclient(hunting): picked mon.noname-b con 0x7f7090003500 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:37:29.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70ac42a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70a41c96a0 0x7f70a41c3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:29.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70ac42a640 10 monclient(hunting): picked mon.noname-a con 0x7f70a41c96a0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:37:29.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70ac42a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70a41c5c80 0x7f70a41c6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:29.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70ac42a640 10 monclient(hunting): picked mon.noname-c con 0x7f70a41c5c80 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:29.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70ac42a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:29.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70ac42a640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f70a409de60 con 0x7f70a41c96a0 2024-09-17T18:37:29.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70ac42a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:29.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70ac42a640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f70a409dce0 con 0x7f7090003500 2024-09-17T18:37:29.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70ac42a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:29.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70ac42a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f70a40747e0 con 0x7f70a41c5c80 2024-09-17T18:37:29.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70ac42a640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:29.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70ac42a640 10 monclient(hunting): authenticate will time out at 1853.365072s 2024-09-17T18:37:29.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70aa9c7640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70a41c5c80 0x7f70a41c6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:29.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70a99c5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70a41c96a0 0x7f70a41c3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:29.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70aa9c7640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70a41c5c80 0x7f70a41c6130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:58144/0 (socket says 172.21.15.121:58144) 2024-09-17T18:37:29.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70a99c5640 10 monclient(hunting): get_auth_request con 0x7f70a41c96a0 auth_method 0 2024-09-17T18:37:29.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70aa9c7640 1 -- 172.21.15.121:0/2598179436 learned_addr learned my addr 172.21.15.121:0/2598179436 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:37:29.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70a99c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:29.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70a99c5640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:29.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70a99c5640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:29.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70aa9c7640 10 monclient(hunting): get_auth_request con 0x7f70a41c5c80 auth_method 0 2024-09-17T18:37:29.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70aa9c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:29.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70aa9c7640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:29.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70aa9c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:29.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70a99c5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:29.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70a99c5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:29.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70aa1c6640 1 --2- 172.21.15.121:0/2598179436 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7090003500 0x7f70a41a9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:29.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70a99c5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:29.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70aa9c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:29.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70aa9c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:29.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70aa9c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:29.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70aa1c6640 10 monclient(hunting): get_auth_request con 0x7f7090003500 auth_method 0 2024-09-17T18:37:29.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70aa1c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:29.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70aa1c6640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:29.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70aa1c6640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:29.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70aa1c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:29.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70aa1c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:29.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70aa1c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:29.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70a99c5640 10 monclient(hunting): handle_auth_done global_id 114198 payload 824 2024-09-17T18:37:29.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70a99c5640 1 -- 172.21.15.121:0/2598179436 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70a41c5c80 msgr2=0x7f70a41c6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:29.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70a99c5640 1 --2- 172.21.15.121:0/2598179436 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70a41c5c80 0x7f70a41c6130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:29.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70a99c5640 1 -- 172.21.15.121:0/2598179436 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7090003500 msgr2=0x7f70a41a9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:29.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70a99c5640 1 --2- 172.21.15.121:0/2598179436 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7090003500 0x7f70a41a9300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:29.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70a99c5640 10 monclient: _finish_hunting 0 2024-09-17T18:37:29.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70a99c5640 1 monclient: found mon.noname-a 2024-09-17T18:37:29.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70a99c5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:29.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70a99c5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:37:29.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.649+0000 7f70a99c5640 1 -- 172.21.15.121:0/2598179436 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70a41a9c90 con 0x7f70a41c96a0 2024-09-17T18:37:29.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.651+0000 7f70a99c5640 1 -- 172.21.15.121:0/2598179436 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70a41c96a0 msgr2=0x7f70a41c3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:29.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.651+0000 7f70a99c5640 1 --2- 172.21.15.121:0/2598179436 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70a41c96a0 0x7f70a41c3180 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f709402a5a0 tx=0x7f70940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:29.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.651+0000 7f709affd640 10 client.?.objecter ms_handle_connect 0x7f70a41c96a0 2024-09-17T18:37:29.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.651+0000 7f709affd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:37:29.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.651+0000 7f709affd640 1 -- 172.21.15.121:0/2598179436 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709403b070 con 0x7f70a41c96a0 2024-09-17T18:37:29.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.651+0000 7f709affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:29.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.651+0000 7f709affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:37:29.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.651+0000 7f709affd640 10 monclient: dump: 2024-09-17T18:37:29.662 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:29.662 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:29.662 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:29.662 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:29.662 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:29.663 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:29.663 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:29.663 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:29.663 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:29.663 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:29.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.651+0000 7f709affd640 10 monclient: _finish_auth 0 2024-09-17T18:37:29.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.651+0000 7f709affd640 10 monclient: _check_auth_tickets 2024-09-17T18:37:29.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.651+0000 7f709affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:29.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.651+0000 7f70ac42a640 5 monclient: authenticate success, global_id 114198 2024-09-17T18:37:29.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.651+0000 7f709affd640 1 -- 172.21.15.121:0/2598179436 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7094035890 con 0x7f70a41c96a0 2024-09-17T18:37:29.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.651+0000 7f70ac42a640 10 monclient: _renew_subs 2024-09-17T18:37:29.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.651+0000 7f70ac42a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:29.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.651+0000 7f70ac42a640 1 -- 172.21.15.121:0/2598179436 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70a41a87a0 con 0x7f70a41c96a0 2024-09-17T18:37:29.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.651+0000 7f709affd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:29.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.651+0000 7f709affd640 1 -- 172.21.15.121:0/2598179436 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7094035c40 con 0x7f70a41c96a0 2024-09-17T18:37:29.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.651+0000 7f709affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:29.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.651+0000 7f709affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:37:29.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.651+0000 7f709affd640 10 monclient: dump: 2024-09-17T18:37:29.664 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:29.664 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:29.664 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:29.664 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:29.664 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:29.664 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:29.664 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:29.664 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:29.664 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:29.664 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:29.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.651+0000 7f70a89c3640 4 set_mon_vals no callback set 2024-09-17T18:37:29.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.651+0000 7f70ac42a640 10 client.114198.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:37:29.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.651+0000 7f70ac42a640 10 monclient: _renew_subs 2024-09-17T18:37:29.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.651+0000 7f70ac42a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:29.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.651+0000 7f70ac42a640 1 -- 172.21.15.121:0/2598179436 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70a41aa260 con 0x7f70a41c96a0 2024-09-17T18:37:29.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.651+0000 7f70ac42a640 1 librados: init done 2024-09-17T18:37:29.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.652+0000 7f709affd640 1 -- 172.21.15.121:0/2598179436 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f709403e070 con 0x7f70a41c96a0 2024-09-17T18:37:29.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.652+0000 7f70ac42a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:29.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.652+0000 7f70ac42a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:29.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.652+0000 7f70ac42a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:29.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.652+0000 7f70ac42a640 1 -- 172.21.15.121:0/2598179436 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70a409e930 con 0x7f70a41c96a0 2024-09-17T18:37:29.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.652+0000 7f709affd640 1 --2- 172.21.15.121:0/2598179436 >> v2:172.21.15.173:6812/2423786037 conn(0x7f70880401a0 0x7f7088042660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:29.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.653+0000 7f709affd640 1 -- 172.21.15.121:0/2598179436 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(679..679 src has 247..679) ==== 4685+0+0 (secure 0 0 0) 0x7f7094073140 con 0x7f70a41c96a0 2024-09-17T18:37:29.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.653+0000 7f709affd640 10 client.114198.objecter ms_dispatch 0x7f70a4000b90 osd_map(679..679 src has 247..679) 2024-09-17T18:37:29.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.653+0000 7f709affd640 3 client.114198.objecter handle_osd_map got epochs [679,679] > 0 2024-09-17T18:37:29.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.653+0000 7f709affd640 3 client.114198.objecter handle_osd_map decoding full epoch 679 2024-09-17T18:37:29.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.653+0000 7f709affd640 20 client.114198.objecter dump_active .. 0 homeless 2024-09-17T18:37:29.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.653+0000 7f70aa1c6640 1 --2- 172.21.15.121:0/2598179436 >> v2:172.21.15.173:6812/2423786037 conn(0x7f70880401a0 0x7f7088042660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:29.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.653+0000 7f70aa1c6640 10 monclient: get_auth_request con 0x7f70880401a0 auth_method 0 2024-09-17T18:37:29.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.655+0000 7f709affd640 10 client.114198.objecter ms_handle_connect 0x7f70880401a0 2024-09-17T18:37:29.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.655+0000 7f70aa1c6640 1 --2- 172.21.15.121:0/2598179436 >> v2:172.21.15.173:6812/2423786037 conn(0x7f70880401a0 0x7f7088042660 secure :-1 s=READY pgs=4107 cs=0 l=1 rev1=1 crypto rx=0x7f70a0000c00 tx=0x7f70a0015000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:29.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.657+0000 7f709affd640 1 -- 172.21.15.121:0/2598179436 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7094045070 con 0x7f70a41c96a0 2024-09-17T18:37:29.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.657+0000 7f709affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:29.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.657+0000 7f709affd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:37:29.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.831+0000 7f70ac42a640 1 -- 172.21.15.121:0/2598179436 --> v2:172.21.15.173:6812/2423786037 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f70a406cd40 con 0x7f70880401a0 2024-09-17T18:37:29.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.832+0000 7f709affd640 1 -- 172.21.15.121:0/2598179436 <== mgr.4105 v2:172.21.15.173:6812/2423786037 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f70a406cd40 con 0x7f70880401a0 2024-09-17T18:37:29.834 INFO:teuthology.orchestra.run.smithi121.stderr:instructed osd(s) 3 to deep-scrub 2024-09-17T18:37:29.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.834+0000 7f70ac42a640 10 librados: watch_flush enter 2024-09-17T18:37:29.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.834+0000 7f70ac42a640 10 librados: watch_flush exit 2024-09-17T18:37:29.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.834+0000 7f70ac42a640 20 client.114198.objecter shutdown clearing up homeless session... 2024-09-17T18:37:29.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.834+0000 7f70ac42a640 10 client.114198.objecter successfully canceled tick 2024-09-17T18:37:29.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.834+0000 7f70ac42a640 1 -- 172.21.15.121:0/2598179436 >> v2:172.21.15.173:6812/2423786037 conn(0x7f70880401a0 msgr2=0x7f7088042660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:29.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.834+0000 7f70ac42a640 1 --2- 172.21.15.121:0/2598179436 >> v2:172.21.15.173:6812/2423786037 conn(0x7f70880401a0 0x7f7088042660 secure :-1 s=READY pgs=4107 cs=0 l=1 rev1=1 crypto rx=0x7f70a0000c00 tx=0x7f70a0015000 comp rx=0 tx=0).stop 2024-09-17T18:37:29.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.834+0000 7f70ac42a640 10 monclient: shutdown 2024-09-17T18:37:29.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.834+0000 7f70ac42a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:29.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.834+0000 7f70ac42a640 1 -- 172.21.15.121:0/2598179436 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70a41c96a0 msgr2=0x7f70a41c3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:29.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.834+0000 7f70ac42a640 1 --2- 172.21.15.121:0/2598179436 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70a41c96a0 0x7f70a41c3180 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f709402a5a0 tx=0x7f70940046c0 comp rx=0 tx=0).stop 2024-09-17T18:37:29.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.834+0000 7f70ac42a640 1 -- 172.21.15.121:0/2598179436 shutdown_connections 2024-09-17T18:37:29.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.834+0000 7f70ac42a640 1 --2- 172.21.15.121:0/2598179436 >> v2:172.21.15.173:6812/2423786037 conn(0x7f70880401a0 0x7f7088042660 unknown :-1 s=CLOSED pgs=4107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:29.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.834+0000 7f70ac42a640 1 --2- 172.21.15.121:0/2598179436 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70a41c5c80 0x7f70a41c6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:29.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.834+0000 7f70ac42a640 1 --2- 172.21.15.121:0/2598179436 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70a41c96a0 0x7f70a41c3180 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:29.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.834+0000 7f70ac42a640 1 --2- 172.21.15.121:0/2598179436 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7090003500 0x7f70a41a9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:29.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.834+0000 7f70ac42a640 1 -- 172.21.15.121:0/2598179436 >> 172.21.15.121:0/2598179436 conn(0x7f70a40a7830 msgr2=0x7f70a4073b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:29.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.834+0000 7f70ac42a640 1 -- 172.21.15.121:0/2598179436 shutdown_connections 2024-09-17T18:37:29.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.834+0000 7f70ac42a640 1 -- 172.21.15.121:0/2598179436 wait complete. 2024-09-17T18:37:29.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.834+0000 7f70ac42a640 1 librados: shutdown 2024-09-17T18:37:29.847 INFO:tasks.ceph:Scrubbing osd.4 2024-09-17T18:37:29.847 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-17T18:37:29.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.945+0000 7f67636f6640 10 monclient: get_monmap_and_config 2024-09-17T18:37:29.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.945+0000 7f67636f6640 10 monclient: build_initial_monmap 2024-09-17T18:37:29.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.945+0000 7f67636f6640 10 monclient: monmap: 2024-09-17T18:37:29.947 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:29.947 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:29.947 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:37:29.946146+0000 2024-09-17T18:37:29.947 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:37:29.946146+0000 2024-09-17T18:37:29.947 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:29.947 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:29.947 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:37:29.947 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:37:29.947 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:37:29.948 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:37:29.948 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:37:29.948 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:29.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.946+0000 7f67636f6640 1 Processor -- start 2024-09-17T18:37:29.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.946+0000 7f67636f6640 1 -- start start 2024-09-17T18:37:29.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.946+0000 7f67636f6640 10 monclient: init 2024-09-17T18:37:29.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.946+0000 7f67636f6640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:29.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.946+0000 7f67636f6640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-17T18:37:29.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.946+0000 7f67636f6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f675c077670 addr v1:172.21.15.173:6789/0 2024-09-17T18:37:29.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.946+0000 7f67636f6640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f675c0a0a70 0x7f675c0a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:29.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.946+0000 7f67636f6640 10 monclient(hunting): picked mon.noname-a con 0x7f675c0a0a70 addr v2:172.21.15.121:3300/0 2024-09-17T18:37:29.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.946+0000 7f67636f6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f675c139ac0 addr v1:172.21.15.121:6789/0 2024-09-17T18:37:29.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.946+0000 7f67636f6640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f675c0a2910 con 0x7f675c139ac0 2024-09-17T18:37:29.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.946+0000 7f67636f6640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f675c09eca0 con 0x7f675c077670 2024-09-17T18:37:29.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.946+0000 7f67636f6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:29.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.946+0000 7f67636f6640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f675c077a50 con 0x7f675c0a0a70 2024-09-17T18:37:29.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.946+0000 7f67636f6640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:29.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.946+0000 7f67636f6640 10 monclient(hunting): authenticate will time out at 1853.662769s 2024-09-17T18:37:29.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.946+0000 7f6760c91640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f675c0a0a70 0x7f675c0a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:29.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.946+0000 7f6760c91640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f675c0a0a70 0x7f675c0a0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38510/0 (socket says 172.21.15.121:38510) 2024-09-17T18:37:29.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.946+0000 7f6760c91640 1 -- 172.21.15.121:0/1388337612 learned_addr learned my addr 172.21.15.121:0/1388337612 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:37:29.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.946+0000 7f6761c93640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f675c139ac0 0x7f675c13beb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:52774/0 (socket says 172.21.15.121:52774) 2024-09-17T18:37:29.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.946+0000 7f6760c91640 10 monclient(hunting): get_auth_request con 0x7f675c0a0a70 auth_method 0 2024-09-17T18:37:29.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.946+0000 7f6760c91640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:29.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.946+0000 7f6760c91640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:29.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.946+0000 7f6760c91640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:29.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.946+0000 7f6760c91640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:29.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.946+0000 7f6760c91640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:29.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.947+0000 7f6760c91640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:29.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.949+0000 7f6761c93640 1 -- 172.21.15.121:0/1388337612 >> v1:172.21.15.121:6789/0 conn(0x7f675c139ac0 legacy=0x7f675c13beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:29.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.949+0000 7f6760c91640 10 monclient(hunting): handle_auth_done global_id 114201 payload 274 2024-09-17T18:37:29.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.949+0000 7f6760c91640 1 -- 172.21.15.121:0/1388337612 >> v1:172.21.15.173:6789/0 conn(0x7f675c077670 legacy=0x7f675c0a2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:29.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.949+0000 7f6760c91640 1 -- 172.21.15.121:0/1388337612 >> v1:172.21.15.121:6789/0 conn(0x7f675c139ac0 legacy=0x7f675c13beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:29.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.949+0000 7f6760c91640 10 monclient: _finish_hunting 0 2024-09-17T18:37:29.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.949+0000 7f6760c91640 1 monclient: found mon.noname-a 2024-09-17T18:37:29.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.949+0000 7f6760c91640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:29.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.949+0000 7f6760c91640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:37:29.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.949+0000 7f6760c91640 1 -- 172.21.15.121:0/1388337612 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f675c0a3740 con 0x7f675c0a0a70 2024-09-17T18:37:29.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.951+0000 7f6760c91640 1 -- 172.21.15.121:0/1388337612 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f675c0a0a70 msgr2=0x7f675c0a0e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:29.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.951+0000 7f6760c91640 1 --2- 172.21.15.121:0/1388337612 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f675c0a0a70 0x7f675c0a0e80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6750009f00 tx=0x7f675002b7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b43353845adc3e server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:29.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.952+0000 7f675bfff640 1 -- 172.21.15.121:0/1388337612 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6750037070 con 0x7f675c0a0a70 2024-09-17T18:37:29.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.952+0000 7f675bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:29.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.952+0000 7f675bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:37:29.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.952+0000 7f675bfff640 10 monclient: dump: 2024-09-17T18:37:29.955 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:29.955 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:29.955 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:29.955 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:29.955 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:29.956 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:29.956 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:29.956 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:29.956 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:29.956 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:29.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.952+0000 7f675bfff640 10 monclient: _finish_auth 0 2024-09-17T18:37:29.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.952+0000 7f675bfff640 10 monclient: _check_auth_tickets 2024-09-17T18:37:29.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.952+0000 7f675bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:29.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.952+0000 7f67636f6640 5 monclient: authenticate success, global_id 114201 2024-09-17T18:37:29.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.952+0000 7f675bfff640 1 -- 172.21.15.121:0/1388337612 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6750007800 con 0x7f675c0a0a70 2024-09-17T18:37:29.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.952+0000 7f67636f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:37:29.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.952+0000 7f675bfff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:29.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.952+0000 7f67636f6640 10 monclient: get_monmap_and_config success 2024-09-17T18:37:29.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.952+0000 7f67636f6640 4 set_mon_vals no callback set 2024-09-17T18:37:29.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.952+0000 7f675bfff640 1 -- 172.21.15.121:0/1388337612 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6750007bd0 con 0x7f675c0a0a70 2024-09-17T18:37:29.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.952+0000 7f675bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:29.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.952+0000 7f675bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:37:29.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.952+0000 7f675bfff640 10 monclient: dump: 2024-09-17T18:37:29.957 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:29.957 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:29.957 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:29.957 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:29.957 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:29.957 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:29.957 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:29.957 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:29.957 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:29.957 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:29.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.952+0000 7f67636f6640 10 monclient: shutdown 2024-09-17T18:37:29.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.952+0000 7f67636f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:29.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.952+0000 7f67636f6640 1 -- 172.21.15.121:0/1388337612 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f675c0a0a70 msgr2=0x7f675c0a0e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:29.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.952+0000 7f67636f6640 1 --2- 172.21.15.121:0/1388337612 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f675c0a0a70 0x7f675c0a0e80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6750009f00 tx=0x7f675002b7d0 comp rx=0 tx=0).stop 2024-09-17T18:37:29.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.952+0000 7f67636f6640 1 -- 172.21.15.121:0/1388337612 shutdown_connections 2024-09-17T18:37:29.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.952+0000 7f67636f6640 1 --2- 172.21.15.121:0/1388337612 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f675c0a0a70 0x7f675c0a0e80 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:29.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.952+0000 7f67636f6640 1 -- 172.21.15.121:0/1388337612 >> 172.21.15.121:0/1388337612 conn(0x7f675c0a3a70 msgr2=0x7f675c0a3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:29.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.952+0000 7f67636f6640 1 -- 172.21.15.121:0/1388337612 shutdown_connections 2024-09-17T18:37:29.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.952+0000 7f67636f6640 1 -- 172.21.15.121:0/1388337612 wait complete. 2024-09-17T18:37:29.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.953+0000 7f67636f6640 10 monclient: build_initial_monmap 2024-09-17T18:37:29.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.953+0000 7f67636f6640 10 monclient: monmap: 2024-09-17T18:37:29.958 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:29.958 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:37:29.958 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:37:29.958 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:37:29.958 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:29.958 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:29.959 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:37:29.959 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:37:29.959 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:37:29.959 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:29.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.953+0000 7f67636f6640 1 librados: starting msgr at 2024-09-17T18:37:29.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.953+0000 7f67636f6640 1 librados: starting objecter 2024-09-17T18:37:29.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.953+0000 7f67636f6640 1 Processor -- start 2024-09-17T18:37:29.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.953+0000 7f67636f6640 1 -- start start 2024-09-17T18:37:29.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.953+0000 7f67636f6640 1 librados: setting wanted keys 2024-09-17T18:37:29.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.953+0000 7f67636f6640 1 librados: calling monclient init 2024-09-17T18:37:29.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.953+0000 7f67636f6640 10 monclient: init 2024-09-17T18:37:29.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f67636f6640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:29.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f67636f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:37:29.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f67636f6640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f675c077670 0x7f675c1cdf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:29.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f67636f6640 10 monclient(hunting): picked mon.noname-a con 0x7f675c077670 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:37:29.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f67636f6640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f675c0a0a70 0x7f675c1ce450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:29.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f67636f6640 10 monclient(hunting): picked mon.noname-b con 0x7f675c0a0a70 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:37:29.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f67636f6640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f675c139ac0 0x7f675c072950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:29.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f67636f6640 10 monclient(hunting): picked mon.noname-c con 0x7f675c139ac0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:29.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f67636f6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:29.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f67636f6640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f675c0a1af0 con 0x7f675c077670 2024-09-17T18:37:29.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f67636f6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:29.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f67636f6640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f675c0a1970 con 0x7f675c0a0a70 2024-09-17T18:37:29.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f67636f6640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:29.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f67636f6640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f675c0a1c70 con 0x7f675c139ac0 2024-09-17T18:37:29.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f67636f6640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:29.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f67636f6640 10 monclient(hunting): authenticate will time out at 1853.670125s 2024-09-17T18:37:29.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f6761492640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f675c077670 0x7f675c1cdf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:29.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f6761492640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f675c077670 0x7f675c1cdf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38526/0 (socket says 172.21.15.121:38526) 2024-09-17T18:37:29.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f6761492640 1 -- 172.21.15.121:0/2259729449 learned_addr learned my addr 172.21.15.121:0/2259729449 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:37:29.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f6761492640 10 monclient(hunting): get_auth_request con 0x7f675c077670 auth_method 0 2024-09-17T18:37:29.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f6761492640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:29.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f6761492640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:29.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f6761492640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:29.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f6761c93640 1 --2- 172.21.15.121:0/2259729449 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f675c139ac0 0x7f675c072950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:29.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f6761c93640 10 monclient(hunting): get_auth_request con 0x7f675c139ac0 auth_method 0 2024-09-17T18:37:29.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f6761c93640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:29.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f6761c93640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:29.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f6761c93640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:29.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f6761492640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:29.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f6761492640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:29.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f6761492640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:29.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f6761c93640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:29.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f6761c93640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:29.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f6761c93640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:29.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f6761492640 10 monclient(hunting): handle_auth_done global_id 114204 payload 824 2024-09-17T18:37:29.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f6760c91640 1 --2- 172.21.15.121:0/2259729449 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f675c0a0a70 0x7f675c1ce450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:29.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f6761492640 1 -- 172.21.15.121:0/2259729449 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f675c139ac0 msgr2=0x7f675c072950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:29.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f6761492640 1 --2- 172.21.15.121:0/2259729449 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f675c139ac0 0x7f675c072950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:29.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f6761492640 1 -- 172.21.15.121:0/2259729449 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f675c0a0a70 msgr2=0x7f675c1ce450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:29.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f6761492640 1 --2- 172.21.15.121:0/2259729449 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f675c0a0a70 0x7f675c1ce450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:29.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f6761492640 10 monclient: _finish_hunting 0 2024-09-17T18:37:29.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f6761492640 1 monclient: found mon.noname-a 2024-09-17T18:37:29.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f6761492640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:29.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f6761492640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:37:29.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f6761492640 1 -- 172.21.15.121:0/2259729449 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f675c11e3b0 con 0x7f675c077670 2024-09-17T18:37:29.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f6760c91640 10 monclient: get_auth_request con 0x7f675c0a0a70 auth_method 0 2024-09-17T18:37:29.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.954+0000 7f6760c91640 1 --2- 172.21.15.121:0/2259729449 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f675c0a0a70 0x7f675c1ce450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T18:37:29.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.956+0000 7f6761492640 1 -- 172.21.15.121:0/2259729449 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f675c077670 msgr2=0x7f675c1cdf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:29.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.956+0000 7f6761492640 1 --2- 172.21.15.121:0/2259729449 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f675c077670 0x7f675c1cdf10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f675400c430 tx=0x7f675400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:29.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.956+0000 7f6759ffb640 10 client.?.objecter ms_handle_connect 0x7f675c077670 2024-09-17T18:37:29.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.956+0000 7f6759ffb640 10 client.?.objecter resend_mon_ops 2024-09-17T18:37:29.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.956+0000 7f6759ffb640 1 -- 172.21.15.121:0/2259729449 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67540119a0 con 0x7f675c077670 2024-09-17T18:37:29.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.956+0000 7f6759ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:29.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.956+0000 7f6759ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:37:29.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.956+0000 7f6759ffb640 10 monclient: dump: 2024-09-17T18:37:29.967 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:29.967 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:29.967 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:29.967 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:29.967 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:29.967 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:29.967 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:29.967 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:29.967 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:29.967 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:29.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.956+0000 7f6759ffb640 10 monclient: _finish_auth 0 2024-09-17T18:37:29.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.956+0000 7f6759ffb640 10 monclient: _check_auth_tickets 2024-09-17T18:37:29.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.956+0000 7f6759ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:29.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.956+0000 7f6759ffb640 1 -- 172.21.15.121:0/2259729449 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6754017070 con 0x7f675c077670 2024-09-17T18:37:29.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.956+0000 7f67636f6640 5 monclient: authenticate success, global_id 114204 2024-09-17T18:37:29.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.956+0000 7f67636f6640 10 monclient: _renew_subs 2024-09-17T18:37:29.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.956+0000 7f67636f6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:29.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.956+0000 7f67636f6640 1 -- 172.21.15.121:0/2259729449 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f675c123be0 con 0x7f675c077670 2024-09-17T18:37:29.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.956+0000 7f6759ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:29.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.956+0000 7f6759ffb640 1 -- 172.21.15.121:0/2259729449 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6754011d80 con 0x7f675c077670 2024-09-17T18:37:29.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.956+0000 7f6759ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:29.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.956+0000 7f6759ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:37:29.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.956+0000 7f6759ffb640 10 monclient: dump: 2024-09-17T18:37:29.968 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:29.968 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:29.968 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:29.969 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:29.969 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:29.969 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:29.969 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:29.969 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:29.969 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:29.969 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:29.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.957+0000 7f67636f6640 10 client.114204.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:37:29.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.956+0000 7f675b7fe640 4 set_mon_vals no callback set 2024-09-17T18:37:29.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.957+0000 7f67636f6640 10 monclient: _renew_subs 2024-09-17T18:37:29.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.957+0000 7f67636f6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:29.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.957+0000 7f67636f6640 1 -- 172.21.15.121:0/2259729449 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f675c09e160 con 0x7f675c077670 2024-09-17T18:37:29.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.957+0000 7f67636f6640 1 librados: init done 2024-09-17T18:37:29.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.957+0000 7f6759ffb640 1 -- 172.21.15.121:0/2259729449 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f675401c460 con 0x7f675c077670 2024-09-17T18:37:29.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.957+0000 7f67636f6640 10 client.114204.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:29.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.957+0000 7f67636f6640 15 client.114204.objecter _session_command_op_assign -1 1 2024-09-17T18:37:29.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.957+0000 7f67636f6640 20 client.114204.objecter _get_session osd=-1 returning homeless 2024-09-17T18:37:29.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.957+0000 7f67636f6640 10 client.114204.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:37:29.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.957+0000 7f67636f6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:29.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.957+0000 7f67636f6640 1 -- 172.21.15.121:0/2259729449 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f675c0a0e80 con 0x7f675c077670 2024-09-17T18:37:29.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.957+0000 7f6759ffb640 1 --2- 172.21.15.121:0/2259729449 >> v2:172.21.15.173:6812/2423786037 conn(0x7f67480401a0 0x7f6748042660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:29.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.958+0000 7f6759ffb640 1 -- 172.21.15.121:0/2259729449 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(679..679 src has 247..679) ==== 4685+0+0 (secure 0 0 0) 0x7f67540532f0 con 0x7f675c077670 2024-09-17T18:37:29.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.958+0000 7f6759ffb640 10 client.114204.objecter ms_dispatch 0x7f675c000b90 osd_map(679..679 src has 247..679) 2024-09-17T18:37:29.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.958+0000 7f6759ffb640 3 client.114204.objecter handle_osd_map got epochs [679,679] > 0 2024-09-17T18:37:29.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.958+0000 7f6759ffb640 3 client.114204.objecter handle_osd_map decoding full epoch 679 2024-09-17T18:37:29.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.958+0000 7f6759ffb640 10 client.114204.objecter checking command 1 2024-09-17T18:37:29.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.958+0000 7f6759ffb640 1 --2- 172.21.15.121:0/2259729449 >> v2:172.21.15.173:6804/1961801317 conn(0x7f6748047660 0x7f6748049b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:29.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.958+0000 7f6759ffb640 20 client.114204.objecter _get_session s=0x7f6748046fb0 osd=4 3 2024-09-17T18:37:29.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.958+0000 7f6759ffb640 20 client.114204.objecter put_session s=0x7f6748046fb0 osd=4 3 2024-09-17T18:37:29.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.958+0000 7f6759ffb640 15 client.114204.objecter _session_command_op_remove -1 1 2024-09-17T18:37:29.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.958+0000 7f6759ffb640 20 client.114204.objecter _get_session s=0x7f6748046fb0 osd=4 3 2024-09-17T18:37:29.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.958+0000 7f6759ffb640 20 client.114204.objecter get_session s=0x7f6748046fb0 osd=4 3 2024-09-17T18:37:29.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.958+0000 7f6759ffb640 15 client.114204.objecter _session_command_op_assign 4 1 2024-09-17T18:37:29.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.958+0000 7f6759ffb640 20 client.114204.objecter put_session s=0x7f6748046fb0 osd=4 4 2024-09-17T18:37:29.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.958+0000 7f6759ffb640 10 client.114204.objecter _send_command 1 2024-09-17T18:37:29.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.958+0000 7f6759ffb640 1 -- 172.21.15.121:0/2259729449 --> v2:172.21.15.173:6804/1961801317 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6754013ec0 con 0x7f6748047660 2024-09-17T18:37:29.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.958+0000 7f6759ffb640 20 client.114204.objecter dump_active .. 0 homeless 2024-09-17T18:37:29.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.958+0000 7f6759ffb640 1 -- 172.21.15.121:0/2259729449 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_get_version_reply(handle=1 version=679) ==== 24+0+0 (secure 0 0 0) 0x7f675405a050 con 0x7f675c077670 2024-09-17T18:37:29.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.958+0000 7f6759ffb640 10 monclient: handle_get_version_reply finishing 1 version 679 2024-09-17T18:37:29.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.958+0000 7f6760c91640 1 --2- 172.21.15.121:0/2259729449 >> v2:172.21.15.173:6812/2423786037 conn(0x7f67480401a0 0x7f6748042660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:29.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.958+0000 7f6760c91640 10 monclient: get_auth_request con 0x7f67480401a0 auth_method 0 2024-09-17T18:37:29.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.958+0000 7f6761c93640 1 --2- 172.21.15.121:0/2259729449 >> v2:172.21.15.173:6804/1961801317 conn(0x7f6748047660 0x7f6748049b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:29.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.958+0000 7f6761c93640 10 monclient: get_auth_request con 0x7f6748047660 auth_method 0 2024-09-17T18:37:29.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.960+0000 7f6760c91640 1 --2- 172.21.15.121:0/2259729449 >> v2:172.21.15.173:6812/2423786037 conn(0x7f67480401a0 0x7f6748042660 secure :-1 s=READY pgs=4108 cs=0 l=1 rev1=1 crypto rx=0x7f675c09df20 tx=0x7f675000d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:29.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.960+0000 7f6759ffb640 10 client.114204.objecter ms_handle_connect 0x7f67480401a0 2024-09-17T18:37:29.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.963+0000 7f6759ffb640 10 client.114204.objecter ms_handle_connect 0x7f6748047660 2024-09-17T18:37:29.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.963+0000 7f6761c93640 1 --2- 172.21.15.121:0/2259729449 >> v2:172.21.15.173:6804/1961801317 conn(0x7f6748047660 0x7f6748049b00 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:29.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.964+0000 7f6759ffb640 1 -- 172.21.15.121:0/2259729449 <== osd.4 v2:172.21.15.173:6804/1961801317 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6754013ec0 con 0x7f6748047660 2024-09-17T18:37:29.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.964+0000 7f6759ffb640 10 client.114204.objecter ms_dispatch 0x7f675c000b90 command_reply(tid 1: 0 ) 2024-09-17T18:37:29.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.964+0000 7f6759ffb640 10 client.114204.objecter _finish_command 1 = system:0 2024-09-17T18:37:29.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.964+0000 7f6759ffb640 20 client.114204.objecter put_session s=0x7f6748046fb0 osd=4 4 2024-09-17T18:37:29.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.964+0000 7f6759ffb640 15 client.114204.objecter _session_command_op_remove 4 1 2024-09-17T18:37:29.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.976+0000 7f67636f6640 10 client.114204.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-17T18:37:29.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.976+0000 7f67636f6640 15 client.114204.objecter _session_command_op_assign -1 2 2024-09-17T18:37:29.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.976+0000 7f67636f6640 20 client.114204.objecter _get_session s=0x7f6748046fb0 osd=4 3 2024-09-17T18:37:29.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.976+0000 7f67636f6640 20 client.114204.objecter put_session s=0x7f6748046fb0 osd=4 3 2024-09-17T18:37:29.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.976+0000 7f67636f6640 20 client.114204.objecter _get_session s=0x7f6748046fb0 osd=4 3 2024-09-17T18:37:29.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.976+0000 7f67636f6640 15 client.114204.objecter _session_command_op_remove -1 2 2024-09-17T18:37:29.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.976+0000 7f67636f6640 20 client.114204.objecter get_session s=0x7f6748046fb0 osd=4 3 2024-09-17T18:37:29.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.976+0000 7f67636f6640 15 client.114204.objecter _session_command_op_assign 4 2 2024-09-17T18:37:29.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.976+0000 7f67636f6640 20 client.114204.objecter put_session s=0x7f6748046fb0 osd=4 4 2024-09-17T18:37:29.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.976+0000 7f67636f6640 10 client.114204.objecter _send_command 2 2024-09-17T18:37:29.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.976+0000 7f67636f6640 1 -- 172.21.15.121:0/2259729449 --> v2:172.21.15.173:6804/1961801317 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f675c11eb40 con 0x7f6748047660 2024-09-17T18:37:29.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.976+0000 7f6759ffb640 1 -- 172.21.15.121:0/2259729449 <== osd.4 v2:172.21.15.173:6804/1961801317 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f675c11eb40 con 0x7f6748047660 2024-09-17T18:37:29.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.976+0000 7f6759ffb640 10 client.114204.objecter ms_dispatch 0x7f675c000b90 command_reply(tid 2: 0 ) 2024-09-17T18:37:29.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.976+0000 7f6759ffb640 10 client.114204.objecter _finish_command 2 = system:0 2024-09-17T18:37:29.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.976+0000 7f6759ffb640 20 client.114204.objecter put_session s=0x7f6748046fb0 osd=4 4 2024-09-17T18:37:29.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.976+0000 7f6759ffb640 15 client.114204.objecter _session_command_op_remove 4 2 2024-09-17T18:37:29.979 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:37:29.979 INFO:teuthology.orchestra.run.smithi121.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T18:37:29.979 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:37:29.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.976+0000 7f67636f6640 10 librados: watch_flush enter 2024-09-17T18:37:29.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.976+0000 7f67636f6640 10 librados: watch_flush exit 2024-09-17T18:37:29.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.976+0000 7f67636f6640 10 client.114204.objecter close_session for osd.4 2024-09-17T18:37:29.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.976+0000 7f67636f6640 1 -- 172.21.15.121:0/2259729449 >> v2:172.21.15.173:6804/1961801317 conn(0x7f6748047660 msgr2=0x7f6748049b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:29.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.976+0000 7f67636f6640 1 --2- 172.21.15.121:0/2259729449 >> v2:172.21.15.173:6804/1961801317 conn(0x7f6748047660 0x7f6748049b00 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:29.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.976+0000 7f67636f6640 20 client.114204.objecter put_session s=0x7f6748046fb0 osd=4 1 2024-09-17T18:37:29.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.976+0000 7f67636f6640 20 client.114204.objecter shutdown clearing up homeless session... 2024-09-17T18:37:29.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.976+0000 7f67636f6640 10 client.114204.objecter successfully canceled tick 2024-09-17T18:37:29.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.976+0000 7f67636f6640 1 -- 172.21.15.121:0/2259729449 >> v2:172.21.15.173:6812/2423786037 conn(0x7f67480401a0 msgr2=0x7f6748042660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:29.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.976+0000 7f67636f6640 1 --2- 172.21.15.121:0/2259729449 >> v2:172.21.15.173:6812/2423786037 conn(0x7f67480401a0 0x7f6748042660 secure :-1 s=READY pgs=4108 cs=0 l=1 rev1=1 crypto rx=0x7f675c09df20 tx=0x7f675000d040 comp rx=0 tx=0).stop 2024-09-17T18:37:29.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.976+0000 7f67636f6640 10 monclient: shutdown 2024-09-17T18:37:29.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.976+0000 7f67636f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:29.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.976+0000 7f67636f6640 1 -- 172.21.15.121:0/2259729449 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f675c077670 msgr2=0x7f675c1cdf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:29.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.976+0000 7f67636f6640 1 --2- 172.21.15.121:0/2259729449 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f675c077670 0x7f675c1cdf10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f675400c430 tx=0x7f675400c8f0 comp rx=0 tx=0).stop 2024-09-17T18:37:29.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.976+0000 7f6761c93640 1 -- 172.21.15.121:0/2259729449 reap_dead start 2024-09-17T18:37:29.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.977+0000 7f67636f6640 1 -- 172.21.15.121:0/2259729449 shutdown_connections 2024-09-17T18:37:29.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.977+0000 7f67636f6640 1 -- 172.21.15.121:0/2259729449 >> 172.21.15.121:0/2259729449 conn(0x7f675c0a3a70 msgr2=0x7f675c134970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:29.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.977+0000 7f67636f6640 1 -- 172.21.15.121:0/2259729449 shutdown_connections 2024-09-17T18:37:29.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.977+0000 7f67636f6640 1 -- 172.21.15.121:0/2259729449 wait complete. 2024-09-17T18:37:29.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:29.977+0000 7f67636f6640 1 librados: shutdown 2024-09-17T18:37:29.988 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-17T18:37:30.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.084+0000 7f46efcb8640 10 monclient: get_monmap_and_config 2024-09-17T18:37:30.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.084+0000 7f46efcb8640 10 monclient: build_initial_monmap 2024-09-17T18:37:30.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.084+0000 7f46efcb8640 10 monclient: monmap: 2024-09-17T18:37:30.086 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:30.086 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:30.086 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:37:30.084664+0000 2024-09-17T18:37:30.086 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:37:30.084664+0000 2024-09-17T18:37:30.086 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:30.086 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:30.086 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:37:30.086 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:37:30.086 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:37:30.086 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:37:30.086 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:37:30.086 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:30.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.084+0000 7f46efcb8640 1 Processor -- start 2024-09-17T18:37:30.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.084+0000 7f46efcb8640 1 -- start start 2024-09-17T18:37:30.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.084+0000 7f46efcb8640 10 monclient: init 2024-09-17T18:37:30.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.085+0000 7f46efcb8640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:30.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.085+0000 7f46efcb8640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-17T18:37:30.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.085+0000 7f46efcb8640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46e80700f0 0x7f46e8134e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:30.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.085+0000 7f46efcb8640 10 monclient(hunting): picked mon.noname-b con 0x7f46e80700f0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:30.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.085+0000 7f46efcb8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f46e809e810 addr v1:172.21.15.121:6789/0 2024-09-17T18:37:30.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.085+0000 7f46efcb8640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f46e809ec30 0x7f46e8138a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:30.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.085+0000 7f46efcb8640 10 monclient(hunting): picked mon.noname-c con 0x7f46e809ec30 addr v2:172.21.15.173:3300/0 2024-09-17T18:37:30.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.085+0000 7f46efcb8640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46e809ca60 con 0x7f46e809e810 2024-09-17T18:37:30.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.085+0000 7f46efcb8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:30.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.085+0000 7f46efcb8640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f46e80704d0 con 0x7f46e809ec30 2024-09-17T18:37:30.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.085+0000 7f46efcb8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:30.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.085+0000 7f46efcb8640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f46e809cd20 con 0x7f46e80700f0 2024-09-17T18:37:30.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.085+0000 7f46efcb8640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:30.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.085+0000 7f46efcb8640 10 monclient(hunting): authenticate will time out at 1853.801531s 2024-09-17T18:37:30.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.085+0000 7f46eda54640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46e80700f0 0x7f46e8134e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:30.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.085+0000 7f46eda54640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46e80700f0 0x7f46e8134e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:58154/0 (socket says 172.21.15.121:58154) 2024-09-17T18:37:30.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.085+0000 7f46eda54640 1 -- 172.21.15.121:0/73212464 learned_addr learned my addr 172.21.15.121:0/73212464 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:37:30.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.085+0000 7f46eda54640 10 monclient(hunting): get_auth_request con 0x7f46e80700f0 auth_method 0 2024-09-17T18:37:30.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.085+0000 7f46eda54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:30.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.085+0000 7f46eda54640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:30.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.085+0000 7f46eda54640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:30.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.085+0000 7f46ed253640 1 --1- 172.21.15.121:0/73212464 >> v1:172.21.15.121:6789/0 conn(0x7f46e809e810 0x7f46e8138360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:52790/0 (socket says 172.21.15.121:52790) 2024-09-17T18:37:30.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.085+0000 7f46eda54640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:30.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.085+0000 7f46eda54640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:30.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.085+0000 7f46eda54640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:30.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.085+0000 7f46ee255640 1 --2- 172.21.15.121:0/73212464 >> v2:172.21.15.173:3300/0 conn(0x7f46e809ec30 0x7f46e8138a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:30.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.085+0000 7f46ee255640 10 monclient(hunting): get_auth_request con 0x7f46e809ec30 auth_method 0 2024-09-17T18:37:30.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.085+0000 7f46ee255640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:30.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.085+0000 7f46ee255640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:30.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.085+0000 7f46ee255640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:30.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.085+0000 7f46eda54640 10 monclient(hunting): handle_auth_done global_id 64652 payload 274 2024-09-17T18:37:30.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.085+0000 7f46eda54640 1 -- 172.21.15.121:0/73212464 >> v2:172.21.15.173:3300/0 conn(0x7f46e809ec30 msgr2=0x7f46e8138a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:30.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.085+0000 7f46eda54640 1 --2- 172.21.15.121:0/73212464 >> v2:172.21.15.173:3300/0 conn(0x7f46e809ec30 0x7f46e8138a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:30.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.085+0000 7f46eda54640 1 -- 172.21.15.121:0/73212464 >> v1:172.21.15.121:6789/0 conn(0x7f46e809e810 legacy=0x7f46e8138360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:30.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.085+0000 7f46eda54640 10 monclient: _finish_hunting 0 2024-09-17T18:37:30.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.085+0000 7f46eda54640 1 monclient: found mon.noname-b 2024-09-17T18:37:30.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.085+0000 7f46eda54640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:30.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.085+0000 7f46eda54640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:37:30.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.085+0000 7f46eda54640 1 -- 172.21.15.121:0/73212464 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46e809d000 con 0x7f46e80700f0 2024-09-17T18:37:30.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.088+0000 7f46eda54640 1 -- 172.21.15.121:0/73212464 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46e80700f0 msgr2=0x7f46e8134e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:30.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.088+0000 7f46eda54640 1 --2- 172.21.15.121:0/73212464 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46e80700f0 0x7f46e8134e50 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f46d8009e50 tx=0x7f46d802a5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b86cd97102d6930 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:30.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.088+0000 7f46eca52640 1 -- 172.21.15.121:0/73212464 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d8036070 con 0x7f46e80700f0 2024-09-17T18:37:30.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.088+0000 7f46eca52640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:30.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.088+0000 7f46eca52640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:37:30.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.088+0000 7f46eca52640 10 monclient: dump: 2024-09-17T18:37:30.097 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:30.098 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:30.098 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:30.098 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:30.098 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:30.098 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:30.098 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:30.098 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:30.098 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:30.098 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:30.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.088+0000 7f46eca52640 10 monclient: _finish_auth 0 2024-09-17T18:37:30.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.088+0000 7f46eca52640 10 monclient: _check_auth_tickets 2024-09-17T18:37:30.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.088+0000 7f46eca52640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:30.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.088+0000 7f46efcb8640 5 monclient: authenticate success, global_id 64652 2024-09-17T18:37:30.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.088+0000 7f46efcb8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:37:30.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.088+0000 7f46eca52640 1 -- 172.21.15.121:0/73212464 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46d8007800 con 0x7f46e80700f0 2024-09-17T18:37:30.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.088+0000 7f46eca52640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:30.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.088+0000 7f46efcb8640 10 monclient: get_monmap_and_config success 2024-09-17T18:37:30.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.088+0000 7f46efcb8640 4 set_mon_vals no callback set 2024-09-17T18:37:30.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.088+0000 7f46eca52640 1 -- 172.21.15.121:0/73212464 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d8007bd0 con 0x7f46e80700f0 2024-09-17T18:37:30.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.088+0000 7f46eca52640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:30.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.088+0000 7f46eca52640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:37:30.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.088+0000 7f46eca52640 10 monclient: dump: 2024-09-17T18:37:30.099 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:30.099 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:30.099 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:30.099 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:30.099 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:30.099 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:30.099 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:30.099 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:30.099 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:30.099 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:30.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.089+0000 7f46efcb8640 10 monclient: shutdown 2024-09-17T18:37:30.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.089+0000 7f46efcb8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:30.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.089+0000 7f46efcb8640 1 -- 172.21.15.121:0/73212464 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46e80700f0 msgr2=0x7f46e8134e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:30.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.089+0000 7f46efcb8640 1 --2- 172.21.15.121:0/73212464 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46e80700f0 0x7f46e8134e50 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f46d8009e50 tx=0x7f46d802a5f0 comp rx=0 tx=0).stop 2024-09-17T18:37:30.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.089+0000 7f46efcb8640 1 -- 172.21.15.121:0/73212464 shutdown_connections 2024-09-17T18:37:30.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.089+0000 7f46efcb8640 1 --2- 172.21.15.121:0/73212464 >> v2:172.21.15.173:3300/0 conn(0x7f46e809ec30 0x7f46e8138a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:30.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.089+0000 7f46efcb8640 1 --2- 172.21.15.121:0/73212464 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46e80700f0 0x7f46e8134e50 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:30.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.089+0000 7f46efcb8640 1 -- 172.21.15.121:0/73212464 >> 172.21.15.121:0/73212464 conn(0x7f46e80ab830 msgr2=0x7f46e80a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:30.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.089+0000 7f46efcb8640 1 -- 172.21.15.121:0/73212464 shutdown_connections 2024-09-17T18:37:30.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.089+0000 7f46efcb8640 1 -- 172.21.15.121:0/73212464 wait complete. 2024-09-17T18:37:30.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.089+0000 7f46efcb8640 10 monclient: build_initial_monmap 2024-09-17T18:37:30.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.089+0000 7f46efcb8640 10 monclient: monmap: 2024-09-17T18:37:30.100 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:30.101 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:37:30.101 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:37:30.101 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:37:30.101 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:30.101 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:30.101 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:37:30.101 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:37:30.101 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:37:30.101 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:30.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.090+0000 7f46efcb8640 1 librados: starting msgr at 2024-09-17T18:37:30.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.090+0000 7f46efcb8640 1 librados: starting objecter 2024-09-17T18:37:30.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.090+0000 7f46efcb8640 1 Processor -- start 2024-09-17T18:37:30.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.090+0000 7f46efcb8640 1 -- start start 2024-09-17T18:37:30.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.090+0000 7f46efcb8640 1 librados: setting wanted keys 2024-09-17T18:37:30.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.090+0000 7f46efcb8640 1 librados: calling monclient init 2024-09-17T18:37:30.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.090+0000 7f46efcb8640 10 monclient: init 2024-09-17T18:37:30.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.090+0000 7f46efcb8640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:30.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.090+0000 7f46efcb8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T18:37:30.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.090+0000 7f46efcb8640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46e80700f0 0x7f46e8131bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:30.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.090+0000 7f46efcb8640 10 monclient(hunting): picked mon.noname-b con 0x7f46e80700f0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:37:30.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.090+0000 7f46efcb8640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46e809e810 0x7f46e8132100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:30.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.090+0000 7f46efcb8640 10 monclient(hunting): picked mon.noname-a con 0x7f46e809e810 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:37:30.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.090+0000 7f46efcb8640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46e809ec30 0x7f46e80a3430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:30.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.090+0000 7f46efcb8640 10 monclient(hunting): picked mon.noname-c con 0x7f46e809ec30 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:30.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.090+0000 7f46efcb8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:30.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.090+0000 7f46efcb8640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f46e8135550 con 0x7f46e809e810 2024-09-17T18:37:30.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.090+0000 7f46efcb8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:30.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.090+0000 7f46efcb8640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f46e81353d0 con 0x7f46e80700f0 2024-09-17T18:37:30.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.090+0000 7f46efcb8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:30.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.090+0000 7f46efcb8640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f46e8138fb0 con 0x7f46e809ec30 2024-09-17T18:37:30.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.090+0000 7f46efcb8640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:30.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.090+0000 7f46efcb8640 10 monclient(hunting): authenticate will time out at 1853.806762s 2024-09-17T18:37:30.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.090+0000 7f46ed253640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46e809e810 0x7f46e8132100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:30.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.090+0000 7f46ee255640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46e809ec30 0x7f46e80a3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:30.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.090+0000 7f46ed253640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46e809e810 0x7f46e8132100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38528/0 (socket says 172.21.15.121:38528) 2024-09-17T18:37:30.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.090+0000 7f46ed253640 1 -- 172.21.15.121:0/1345491025 learned_addr learned my addr 172.21.15.121:0/1345491025 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:37:30.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.090+0000 7f46ed253640 10 monclient(hunting): get_auth_request con 0x7f46e809e810 auth_method 0 2024-09-17T18:37:30.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.090+0000 7f46ed253640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:30.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.090+0000 7f46ed253640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:30.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.090+0000 7f46ed253640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:30.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.090+0000 7f46ee255640 10 monclient(hunting): get_auth_request con 0x7f46e809ec30 auth_method 0 2024-09-17T18:37:30.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.090+0000 7f46ee255640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:30.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.090+0000 7f46ee255640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:30.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.090+0000 7f46ee255640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:30.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.090+0000 7f46ed253640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:30.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.090+0000 7f46ed253640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:30.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.090+0000 7f46ed253640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:30.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.090+0000 7f46ee255640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:30.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.090+0000 7f46ee255640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:30.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.090+0000 7f46ee255640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:30.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.091+0000 7f46eda54640 1 --2- 172.21.15.121:0/1345491025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46e80700f0 0x7f46e8131bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:30.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.091+0000 7f46eda54640 10 monclient(hunting): get_auth_request con 0x7f46e80700f0 auth_method 0 2024-09-17T18:37:30.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.091+0000 7f46eda54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:30.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.091+0000 7f46eda54640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:30.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.091+0000 7f46eda54640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:30.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.091+0000 7f46ed253640 10 monclient(hunting): handle_auth_done global_id 114207 payload 824 2024-09-17T18:37:30.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.091+0000 7f46ed253640 1 -- 172.21.15.121:0/1345491025 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46e809ec30 msgr2=0x7f46e80a3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:30.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.091+0000 7f46ed253640 1 --2- 172.21.15.121:0/1345491025 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46e809ec30 0x7f46e80a3430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:30.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.091+0000 7f46ed253640 1 -- 172.21.15.121:0/1345491025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46e80700f0 msgr2=0x7f46e8131bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:30.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.091+0000 7f46ed253640 1 --2- 172.21.15.121:0/1345491025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46e80700f0 0x7f46e8131bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:30.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.091+0000 7f46ed253640 10 monclient: _finish_hunting 0 2024-09-17T18:37:30.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.091+0000 7f46ed253640 1 monclient: found mon.noname-a 2024-09-17T18:37:30.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.091+0000 7f46ed253640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:30.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.091+0000 7f46ed253640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:37:30.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.091+0000 7f46ed253640 1 -- 172.21.15.121:0/1345491025 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46e813ad20 con 0x7f46e809e810 2024-09-17T18:37:30.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.091+0000 7f46ee255640 1 --2- 172.21.15.121:0/1345491025 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46e809ec30 0x7f46e80a3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:37:30.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.093+0000 7f46ed253640 1 -- 172.21.15.121:0/1345491025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46e809e810 msgr2=0x7f46e8132100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:30.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.093+0000 7f46ed253640 1 --2- 172.21.15.121:0/1345491025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46e809e810 0x7f46e8132100 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f46dc00c450 tx=0x7f46dc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:30.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.093+0000 7f46e67fc640 10 client.?.objecter ms_handle_connect 0x7f46e809e810 2024-09-17T18:37:30.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.093+0000 7f46e67fc640 10 client.?.objecter resend_mon_ops 2024-09-17T18:37:30.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.093+0000 7f46e67fc640 1 -- 172.21.15.121:0/1345491025 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46dc017070 con 0x7f46e809e810 2024-09-17T18:37:30.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.093+0000 7f46e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:30.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.093+0000 7f46e67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:37:30.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.093+0000 7f46e67fc640 10 monclient: dump: 2024-09-17T18:37:30.105 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:30.106 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:30.106 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:30.106 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:30.106 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:30.106 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:30.106 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:30.106 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:30.106 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:30.106 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:30.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.093+0000 7f46e67fc640 10 monclient: _finish_auth 0 2024-09-17T18:37:30.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.093+0000 7f46e67fc640 10 monclient: _check_auth_tickets 2024-09-17T18:37:30.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.093+0000 7f46e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:30.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.093+0000 7f46e67fc640 1 -- 172.21.15.121:0/1345491025 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46dc005810 con 0x7f46e809e810 2024-09-17T18:37:30.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.093+0000 7f46efcb8640 5 monclient: authenticate success, global_id 114207 2024-09-17T18:37:30.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.093+0000 7f46efcb8640 10 monclient: _renew_subs 2024-09-17T18:37:30.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.093+0000 7f46efcb8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:30.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.093+0000 7f46efcb8640 1 -- 172.21.15.121:0/1345491025 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46e813b000 con 0x7f46e809e810 2024-09-17T18:37:30.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.093+0000 7f46e67fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:30.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.093+0000 7f46e67fc640 1 -- 172.21.15.121:0/1345491025 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46dc007690 con 0x7f46e809e810 2024-09-17T18:37:30.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.093+0000 7f46e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:30.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.093+0000 7f46e67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:37:30.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.093+0000 7f46e67fc640 10 monclient: dump: 2024-09-17T18:37:30.107 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:30.107 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:30.107 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:30.107 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:30.107 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:30.107 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:30.107 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:30.107 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:30.107 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:30.107 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:30.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.093+0000 7f46efcb8640 10 client.114207.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:37:30.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.093+0000 7f46efcb8640 10 monclient: _renew_subs 2024-09-17T18:37:30.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.093+0000 7f46efcb8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:30.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.093+0000 7f46e7fff640 4 set_mon_vals no callback set 2024-09-17T18:37:30.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.093+0000 7f46efcb8640 1 -- 172.21.15.121:0/1345491025 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46e813fef0 con 0x7f46e809e810 2024-09-17T18:37:30.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.093+0000 7f46efcb8640 1 librados: init done 2024-09-17T18:37:30.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.094+0000 7f46e67fc640 1 -- 172.21.15.121:0/1345491025 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f46dc01a070 con 0x7f46e809e810 2024-09-17T18:37:30.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.094+0000 7f46efcb8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:30.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.094+0000 7f46efcb8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:30.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.094+0000 7f46efcb8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:30.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.094+0000 7f46efcb8640 1 -- 172.21.15.121:0/1345491025 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46e8134e50 con 0x7f46e809e810 2024-09-17T18:37:30.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.094+0000 7f46e67fc640 1 --2- 172.21.15.121:0/1345491025 >> v2:172.21.15.173:6812/2423786037 conn(0x7f46d4040170 0x7f46d4042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:30.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.094+0000 7f46e67fc640 1 -- 172.21.15.121:0/1345491025 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(679..679 src has 247..679) ==== 4685+0+0 (secure 0 0 0) 0x7f46dc0528a0 con 0x7f46e809e810 2024-09-17T18:37:30.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.094+0000 7f46e67fc640 10 client.114207.objecter ms_dispatch 0x7f46e8000b90 osd_map(679..679 src has 247..679) 2024-09-17T18:37:30.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.094+0000 7f46e67fc640 3 client.114207.objecter handle_osd_map got epochs [679,679] > 0 2024-09-17T18:37:30.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.094+0000 7f46e67fc640 3 client.114207.objecter handle_osd_map decoding full epoch 679 2024-09-17T18:37:30.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.094+0000 7f46e67fc640 20 client.114207.objecter dump_active .. 0 homeless 2024-09-17T18:37:30.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.094+0000 7f46eda54640 1 --2- 172.21.15.121:0/1345491025 >> v2:172.21.15.173:6812/2423786037 conn(0x7f46d4040170 0x7f46d4042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:30.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.094+0000 7f46eda54640 10 monclient: get_auth_request con 0x7f46d4040170 auth_method 0 2024-09-17T18:37:30.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.097+0000 7f46eda54640 1 --2- 172.21.15.121:0/1345491025 >> v2:172.21.15.173:6812/2423786037 conn(0x7f46d4040170 0x7f46d4042630 secure :-1 s=READY pgs=4109 cs=0 l=1 rev1=1 crypto rx=0x7f46d8033000 tx=0x7f46d8009000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:30.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.097+0000 7f46e67fc640 10 client.114207.objecter ms_handle_connect 0x7f46d4040170 2024-09-17T18:37:30.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.098+0000 7f46e67fc640 1 -- 172.21.15.121:0/1345491025 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f46dc005e20 con 0x7f46e809e810 2024-09-17T18:37:30.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.098+0000 7f46e67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:30.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.099+0000 7f46e67fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:37:30.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.274+0000 7f46efcb8640 1 -- 172.21.15.121:0/1345491025 --> v2:172.21.15.173:6812/2423786037 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f46e8132640 con 0x7f46d4040170 2024-09-17T18:37:30.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.274+0000 7f46e67fc640 1 -- 172.21.15.121:0/1345491025 <== mgr.4105 v2:172.21.15.173:6812/2423786037 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f46e8132640 con 0x7f46d4040170 2024-09-17T18:37:30.276 INFO:teuthology.orchestra.run.smithi121.stderr:instructed osd(s) 4 to deep-scrub 2024-09-17T18:37:30.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.276+0000 7f46efcb8640 10 librados: watch_flush enter 2024-09-17T18:37:30.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.276+0000 7f46efcb8640 10 librados: watch_flush exit 2024-09-17T18:37:30.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.276+0000 7f46efcb8640 20 client.114207.objecter shutdown clearing up homeless session... 2024-09-17T18:37:30.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.277+0000 7f46efcb8640 10 client.114207.objecter successfully canceled tick 2024-09-17T18:37:30.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.277+0000 7f46efcb8640 1 -- 172.21.15.121:0/1345491025 >> v2:172.21.15.173:6812/2423786037 conn(0x7f46d4040170 msgr2=0x7f46d4042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:30.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.277+0000 7f46efcb8640 1 --2- 172.21.15.121:0/1345491025 >> v2:172.21.15.173:6812/2423786037 conn(0x7f46d4040170 0x7f46d4042630 secure :-1 s=READY pgs=4109 cs=0 l=1 rev1=1 crypto rx=0x7f46d8033000 tx=0x7f46d8009000 comp rx=0 tx=0).stop 2024-09-17T18:37:30.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.277+0000 7f46efcb8640 10 monclient: shutdown 2024-09-17T18:37:30.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.277+0000 7f46efcb8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:30.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.277+0000 7f46efcb8640 1 -- 172.21.15.121:0/1345491025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46e809e810 msgr2=0x7f46e8132100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:30.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.277+0000 7f46efcb8640 1 --2- 172.21.15.121:0/1345491025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46e809e810 0x7f46e8132100 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f46dc00c450 tx=0x7f46dc00c910 comp rx=0 tx=0).stop 2024-09-17T18:37:30.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.277+0000 7f46efcb8640 1 -- 172.21.15.121:0/1345491025 shutdown_connections 2024-09-17T18:37:30.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.277+0000 7f46efcb8640 1 --2- 172.21.15.121:0/1345491025 >> v2:172.21.15.173:6812/2423786037 conn(0x7f46d4040170 0x7f46d4042630 unknown :-1 s=CLOSED pgs=4109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:30.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.277+0000 7f46efcb8640 1 --2- 172.21.15.121:0/1345491025 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46e809ec30 0x7f46e80a3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:30.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.277+0000 7f46efcb8640 1 --2- 172.21.15.121:0/1345491025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46e809e810 0x7f46e8132100 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:30.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.277+0000 7f46efcb8640 1 --2- 172.21.15.121:0/1345491025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46e80700f0 0x7f46e8131bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:30.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.277+0000 7f46efcb8640 1 -- 172.21.15.121:0/1345491025 >> 172.21.15.121:0/1345491025 conn(0x7f46e80ab830 msgr2=0x7f46e81335e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:30.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.277+0000 7f46efcb8640 1 -- 172.21.15.121:0/1345491025 shutdown_connections 2024-09-17T18:37:30.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.277+0000 7f46efcb8640 1 -- 172.21.15.121:0/1345491025 wait complete. 2024-09-17T18:37:30.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.277+0000 7f46efcb8640 1 librados: shutdown 2024-09-17T18:37:30.290 INFO:tasks.ceph:Scrubbing osd.5 2024-09-17T18:37:30.290 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-17T18:37:30.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.389+0000 7fdf96f3a640 10 monclient: get_monmap_and_config 2024-09-17T18:37:30.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.389+0000 7fdf96f3a640 10 monclient: build_initial_monmap 2024-09-17T18:37:30.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.389+0000 7fdf96f3a640 10 monclient: monmap: 2024-09-17T18:37:30.391 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:30.391 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:30.391 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:37:30.390159+0000 2024-09-17T18:37:30.391 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:37:30.390159+0000 2024-09-17T18:37:30.391 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:30.391 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:30.391 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:37:30.391 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:37:30.391 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:37:30.392 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:37:30.392 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:37:30.392 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:30.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.390+0000 7fdf96f3a640 1 Processor -- start 2024-09-17T18:37:30.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.390+0000 7fdf96f3a640 1 -- start start 2024-09-17T18:37:30.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.390+0000 7fdf96f3a640 10 monclient: init 2024-09-17T18:37:30.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.390+0000 7fdf96f3a640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:30.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.390+0000 7fdf96f3a640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-17T18:37:30.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.390+0000 7fdf96f3a640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fdf90077670 0x7fdf901364c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:30.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.390+0000 7fdf96f3a640 10 monclient(hunting): picked mon.noname-a con 0x7fdf90077670 addr v2:172.21.15.121:3300/0 2024-09-17T18:37:30.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.390+0000 7fdf96f3a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdf90136a00 0x7fdf900a1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:30.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.390+0000 7fdf96f3a640 10 monclient(hunting): picked mon.noname-b con 0x7fdf90136a00 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:30.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.390+0000 7fdf96f3a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdf90136de0 addr v1:172.21.15.121:6789/0 2024-09-17T18:37:30.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.390+0000 7fdf96f3a640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf900a2a20 con 0x7fdf90136de0 2024-09-17T18:37:30.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.390+0000 7fdf96f3a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:30.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.390+0000 7fdf96f3a640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fdf90077a50 con 0x7fdf90077670 2024-09-17T18:37:30.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.390+0000 7fdf96f3a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:30.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.390+0000 7fdf96f3a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fdf900a2cc0 con 0x7fdf90136a00 2024-09-17T18:37:30.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.390+0000 7fdf96f3a640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:30.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.390+0000 7fdf96f3a640 10 monclient(hunting): authenticate will time out at 1854.106961s 2024-09-17T18:37:30.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.391+0000 7fdf8ffff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdf90136a00 0x7fdf900a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:30.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.391+0000 7fdf94cd6640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fdf90077670 0x7fdf901364c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:30.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.391+0000 7fdf8ffff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdf90136a00 0x7fdf900a1970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:58176/0 (socket says 172.21.15.121:58176) 2024-09-17T18:37:30.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.391+0000 7fdf94cd6640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fdf90077670 0x7fdf901364c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38540/0 (socket says 172.21.15.121:38540) 2024-09-17T18:37:30.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.391+0000 7fdf8ffff640 1 -- 172.21.15.121:0/3956515762 learned_addr learned my addr 172.21.15.121:0/3956515762 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:37:30.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.391+0000 7fdf8ffff640 10 monclient(hunting): get_auth_request con 0x7fdf90136a00 auth_method 0 2024-09-17T18:37:30.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.391+0000 7fdf8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:30.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.391+0000 7fdf8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:30.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.391+0000 7fdf8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:30.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.391+0000 7fdf94cd6640 10 monclient(hunting): get_auth_request con 0x7fdf90077670 auth_method 0 2024-09-17T18:37:30.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.391+0000 7fdf94cd6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:30.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.391+0000 7fdf94cd6640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:30.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.391+0000 7fdf94cd6640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:30.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.391+0000 7fdf8ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:30.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.391+0000 7fdf8ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:30.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.391+0000 7fdf8ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:30.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.391+0000 7fdf8ffff640 10 monclient(hunting): handle_auth_done global_id 64658 payload 274 2024-09-17T18:37:30.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.391+0000 7fdf8ffff640 1 -- 172.21.15.121:0/3956515762 >> v2:172.21.15.121:3300/0 conn(0x7fdf90077670 msgr2=0x7fdf901364c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:30.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.391+0000 7fdf8ffff640 1 --2- 172.21.15.121:0/3956515762 >> v2:172.21.15.121:3300/0 conn(0x7fdf90077670 0x7fdf901364c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:30.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.391+0000 7fdf8ffff640 1 -- 172.21.15.121:0/3956515762 >> v1:172.21.15.121:6789/0 conn(0x7fdf90136de0 legacy=0x7fdf900a1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:30.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.391+0000 7fdf8ffff640 10 monclient: _finish_hunting 0 2024-09-17T18:37:30.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.391+0000 7fdf8ffff640 1 monclient: found mon.noname-b 2024-09-17T18:37:30.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.391+0000 7fdf8ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:30.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.391+0000 7fdf8ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:37:30.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.391+0000 7fdf8ffff640 1 -- 172.21.15.121:0/3956515762 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf900a2fa0 con 0x7fdf90136a00 2024-09-17T18:37:30.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.394+0000 7fdf8ffff640 1 -- 172.21.15.121:0/3956515762 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdf90136a00 msgr2=0x7fdf900a1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:30.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.394+0000 7fdf8ffff640 1 --2- 172.21.15.121:0/3956515762 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdf90136a00 0x7fdf900a1970 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c009870 tx=0x7fdf7c02a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69ad9492df35e023 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:30.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.394+0000 7fdf8f7fe640 1 -- 172.21.15.121:0/3956515762 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf7c037070 con 0x7fdf90136a00 2024-09-17T18:37:30.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.394+0000 7fdf8f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:30.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.394+0000 7fdf8f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:37:30.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.394+0000 7fdf8f7fe640 10 monclient: dump: 2024-09-17T18:37:30.403 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:30.403 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:30.403 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:30.403 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:30.404 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:30.404 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:30.404 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:30.404 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:30.404 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:30.404 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:30.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.394+0000 7fdf8f7fe640 10 monclient: _finish_auth 0 2024-09-17T18:37:30.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.394+0000 7fdf8f7fe640 10 monclient: _check_auth_tickets 2024-09-17T18:37:30.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.394+0000 7fdf8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:30.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.394+0000 7fdf8f7fe640 1 -- 172.21.15.121:0/3956515762 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf7c032440 con 0x7fdf90136a00 2024-09-17T18:37:30.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.394+0000 7fdf8f7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:30.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.394+0000 7fdf8f7fe640 1 -- 172.21.15.121:0/3956515762 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf7c032810 con 0x7fdf90136a00 2024-09-17T18:37:30.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.394+0000 7fdf96f3a640 5 monclient: authenticate success, global_id 64658 2024-09-17T18:37:30.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.394+0000 7fdf96f3a640 10 monclient: get_monmap_and_config success 2024-09-17T18:37:30.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.394+0000 7fdf96f3a640 4 set_mon_vals no callback set 2024-09-17T18:37:30.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.394+0000 7fdf8f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:30.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.394+0000 7fdf8f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:37:30.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.394+0000 7fdf8f7fe640 10 monclient: dump: 2024-09-17T18:37:30.405 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:30.405 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:30.405 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:30.405 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:30.405 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:30.405 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:30.405 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:30.405 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:30.405 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:30.405 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:30.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.394+0000 7fdf96f3a640 10 monclient: shutdown 2024-09-17T18:37:30.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.394+0000 7fdf96f3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:30.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.394+0000 7fdf96f3a640 1 -- 172.21.15.121:0/3956515762 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdf90136a00 msgr2=0x7fdf900a1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:30.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.394+0000 7fdf96f3a640 1 --2- 172.21.15.121:0/3956515762 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdf90136a00 0x7fdf900a1970 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c009870 tx=0x7fdf7c02a0a0 comp rx=0 tx=0).stop 2024-09-17T18:37:30.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.394+0000 7fdf96f3a640 1 -- 172.21.15.121:0/3956515762 shutdown_connections 2024-09-17T18:37:30.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.394+0000 7fdf96f3a640 1 --2- 172.21.15.121:0/3956515762 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdf90136a00 0x7fdf900a1970 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:30.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.394+0000 7fdf96f3a640 1 --2- 172.21.15.121:0/3956515762 >> v2:172.21.15.121:3300/0 conn(0x7fdf90077670 0x7fdf901364c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:30.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.394+0000 7fdf96f3a640 1 -- 172.21.15.121:0/3956515762 >> 172.21.15.121:0/3956515762 conn(0x7fdf900a3a70 msgr2=0x7fdf900a3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:30.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.394+0000 7fdf96f3a640 1 -- 172.21.15.121:0/3956515762 shutdown_connections 2024-09-17T18:37:30.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.394+0000 7fdf96f3a640 1 -- 172.21.15.121:0/3956515762 wait complete. 2024-09-17T18:37:30.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.395+0000 7fdf96f3a640 10 monclient: build_initial_monmap 2024-09-17T18:37:30.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.395+0000 7fdf96f3a640 10 monclient: monmap: 2024-09-17T18:37:30.406 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:30.406 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:37:30.406 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:37:30.406 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:37:30.406 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:30.406 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:30.407 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:37:30.407 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:37:30.407 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:37:30.407 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:30.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.395+0000 7fdf96f3a640 1 librados: starting msgr at 2024-09-17T18:37:30.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.395+0000 7fdf96f3a640 1 librados: starting objecter 2024-09-17T18:37:30.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.395+0000 7fdf96f3a640 1 Processor -- start 2024-09-17T18:37:30.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.395+0000 7fdf96f3a640 1 -- start start 2024-09-17T18:37:30.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.395+0000 7fdf96f3a640 1 librados: setting wanted keys 2024-09-17T18:37:30.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.395+0000 7fdf96f3a640 1 librados: calling monclient init 2024-09-17T18:37:30.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.395+0000 7fdf96f3a640 10 monclient: init 2024-09-17T18:37:30.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf96f3a640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:30.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf96f3a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:37:30.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf96f3a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf90077670 0x7fdf90073c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:30.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf96f3a640 10 monclient(hunting): picked mon.noname-a con 0x7fdf90077670 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:37:30.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf96f3a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf90136a00 0x7fdf9009e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:30.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf96f3a640 10 monclient(hunting): picked mon.noname-b con 0x7fdf90136a00 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:37:30.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf96f3a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdf90136de0 0x7fdf9009e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:30.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf96f3a640 10 monclient(hunting): picked mon.noname-c con 0x7fdf90136de0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:30.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf96f3a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:30.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf96f3a640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fdf900a2740 con 0x7fdf90077670 2024-09-17T18:37:30.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf96f3a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:30.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf96f3a640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdf900a25c0 con 0x7fdf90136a00 2024-09-17T18:37:30.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf96f3a640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:30.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf96f3a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fdf9013e4b0 con 0x7fdf90136de0 2024-09-17T18:37:30.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf96f3a640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:30.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf96f3a640 10 monclient(hunting): authenticate will time out at 1854.112545s 2024-09-17T18:37:30.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf94cd6640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf90077670 0x7fdf90073c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:30.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf954d7640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdf90136de0 0x7fdf9009e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:30.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf94cd6640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf90077670 0x7fdf90073c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38548/0 (socket says 172.21.15.121:38548) 2024-09-17T18:37:30.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf94cd6640 1 -- 172.21.15.121:0/200394840 learned_addr learned my addr 172.21.15.121:0/200394840 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:37:30.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf94cd6640 10 monclient(hunting): get_auth_request con 0x7fdf90077670 auth_method 0 2024-09-17T18:37:30.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf94cd6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:30.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf94cd6640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:30.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf94cd6640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:30.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf954d7640 10 monclient(hunting): get_auth_request con 0x7fdf90136de0 auth_method 0 2024-09-17T18:37:30.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf954d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:30.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf954d7640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:30.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf954d7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:30.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf94cd6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:30.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf94cd6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:30.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf94cd6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:30.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf954d7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:30.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf954d7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:30.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf954d7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:30.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf8ffff640 1 --2- 172.21.15.121:0/200394840 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf90136a00 0x7fdf9009e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:30.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf8ffff640 10 monclient(hunting): get_auth_request con 0x7fdf90136a00 auth_method 0 2024-09-17T18:37:30.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:30.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:30.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:30.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf94cd6640 10 monclient(hunting): handle_auth_done global_id 114213 payload 824 2024-09-17T18:37:30.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf94cd6640 1 -- 172.21.15.121:0/200394840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdf90136de0 msgr2=0x7fdf9009e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:30.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf94cd6640 1 --2- 172.21.15.121:0/200394840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdf90136de0 0x7fdf9009e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:30.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf94cd6640 1 -- 172.21.15.121:0/200394840 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf90136a00 msgr2=0x7fdf9009e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:30.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf94cd6640 1 --2- 172.21.15.121:0/200394840 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf90136a00 0x7fdf9009e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:30.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf94cd6640 10 monclient: _finish_hunting 0 2024-09-17T18:37:30.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf94cd6640 1 monclient: found mon.noname-a 2024-09-17T18:37:30.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf94cd6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:30.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf94cd6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:37:30.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf94cd6640 1 -- 172.21.15.121:0/200394840 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf90135de0 con 0x7fdf90077670 2024-09-17T18:37:30.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.396+0000 7fdf954d7640 1 --2- 172.21.15.121:0/200394840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdf90136de0 0x7fdf9009e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:37:30.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.399+0000 7fdf94cd6640 1 -- 172.21.15.121:0/200394840 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf90077670 msgr2=0x7fdf90073c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:30.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.399+0000 7fdf94cd6640 1 --2- 172.21.15.121:0/200394840 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf90077670 0x7fdf90073c10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdf8400d5f0 tx=0x7fdf8400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:30.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.399+0000 7fdf8d7fa640 10 client.?.objecter ms_handle_connect 0x7fdf90077670 2024-09-17T18:37:30.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.399+0000 7fdf8d7fa640 10 client.?.objecter resend_mon_ops 2024-09-17T18:37:30.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.399+0000 7fdf8d7fa640 1 -- 172.21.15.121:0/200394840 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf84012800 con 0x7fdf90077670 2024-09-17T18:37:30.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.399+0000 7fdf8d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:30.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.399+0000 7fdf8d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:37:30.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.399+0000 7fdf8d7fa640 10 monclient: dump: 2024-09-17T18:37:30.411 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:30.411 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:30.411 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:30.411 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:30.412 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:30.412 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:30.412 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:30.412 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:30.412 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:30.412 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:30.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.399+0000 7fdf8d7fa640 10 monclient: _finish_auth 0 2024-09-17T18:37:30.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.399+0000 7fdf8d7fa640 10 monclient: _check_auth_tickets 2024-09-17T18:37:30.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.399+0000 7fdf8d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:30.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.399+0000 7fdf8d7fa640 1 -- 172.21.15.121:0/200394840 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf84019070 con 0x7fdf90077670 2024-09-17T18:37:30.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.399+0000 7fdf96f3a640 5 monclient: authenticate success, global_id 114213 2024-09-17T18:37:30.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.399+0000 7fdf96f3a640 10 monclient: _renew_subs 2024-09-17T18:37:30.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.399+0000 7fdf96f3a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:30.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.399+0000 7fdf96f3a640 1 -- 172.21.15.121:0/200394840 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf90074150 con 0x7fdf90077670 2024-09-17T18:37:30.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.399+0000 7fdf8d7fa640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:30.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.399+0000 7fdf8d7fa640 1 -- 172.21.15.121:0/200394840 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf84012be0 con 0x7fdf90077670 2024-09-17T18:37:30.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.399+0000 7fdf8d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:30.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.399+0000 7fdf8d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:37:30.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.399+0000 7fdf8d7fa640 10 monclient: dump: 2024-09-17T18:37:30.413 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:30.413 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:30.413 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:30.413 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:30.413 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:30.413 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:30.413 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:30.413 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:30.413 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:30.413 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:30.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.399+0000 7fdf8effd640 4 set_mon_vals no callback set 2024-09-17T18:37:30.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.399+0000 7fdf96f3a640 10 client.114213.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:37:30.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.399+0000 7fdf96f3a640 10 monclient: _renew_subs 2024-09-17T18:37:30.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.399+0000 7fdf96f3a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:30.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.399+0000 7fdf96f3a640 1 -- 172.21.15.121:0/200394840 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf9013d140 con 0x7fdf90077670 2024-09-17T18:37:30.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.399+0000 7fdf96f3a640 1 librados: init done 2024-09-17T18:37:30.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.400+0000 7fdf96f3a640 10 client.114213.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:30.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.400+0000 7fdf96f3a640 15 client.114213.objecter _session_command_op_assign -1 1 2024-09-17T18:37:30.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.400+0000 7fdf96f3a640 20 client.114213.objecter _get_session osd=-1 returning homeless 2024-09-17T18:37:30.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.400+0000 7fdf96f3a640 10 client.114213.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:37:30.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.400+0000 7fdf96f3a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:30.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.400+0000 7fdf96f3a640 1 -- 172.21.15.121:0/200394840 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdf9009e160 con 0x7fdf90077670 2024-09-17T18:37:30.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.400+0000 7fdf8d7fa640 1 -- 172.21.15.121:0/200394840 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fdf8401e460 con 0x7fdf90077670 2024-09-17T18:37:30.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.400+0000 7fdf8d7fa640 1 --2- 172.21.15.121:0/200394840 >> v2:172.21.15.173:6812/2423786037 conn(0x7fdf74040170 0x7fdf74042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:30.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.400+0000 7fdf8d7fa640 1 -- 172.21.15.121:0/200394840 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(679..679 src has 247..679) ==== 4685+0+0 (secure 0 0 0) 0x7fdf840538a0 con 0x7fdf90077670 2024-09-17T18:37:30.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.400+0000 7fdf8d7fa640 10 client.114213.objecter ms_dispatch 0x7fdf90000b90 osd_map(679..679 src has 247..679) 2024-09-17T18:37:30.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.400+0000 7fdf8d7fa640 3 client.114213.objecter handle_osd_map got epochs [679,679] > 0 2024-09-17T18:37:30.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.400+0000 7fdf8d7fa640 3 client.114213.objecter handle_osd_map decoding full epoch 679 2024-09-17T18:37:30.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.400+0000 7fdf8d7fa640 10 client.114213.objecter checking command 1 2024-09-17T18:37:30.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.400+0000 7fdf8d7fa640 1 --2- 172.21.15.121:0/200394840 >> v2:172.21.15.173:6808/1801998802 conn(0x7fdf74047630 0x7fdf74049ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:30.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.400+0000 7fdf8d7fa640 20 client.114213.objecter _get_session s=0x7fdf74046f80 osd=5 3 2024-09-17T18:37:30.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.400+0000 7fdf8d7fa640 20 client.114213.objecter put_session s=0x7fdf74046f80 osd=5 3 2024-09-17T18:37:30.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.400+0000 7fdf8d7fa640 15 client.114213.objecter _session_command_op_remove -1 1 2024-09-17T18:37:30.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.400+0000 7fdf8d7fa640 20 client.114213.objecter _get_session s=0x7fdf74046f80 osd=5 3 2024-09-17T18:37:30.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.400+0000 7fdf8d7fa640 20 client.114213.objecter get_session s=0x7fdf74046f80 osd=5 3 2024-09-17T18:37:30.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.400+0000 7fdf8d7fa640 15 client.114213.objecter _session_command_op_assign 5 1 2024-09-17T18:37:30.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.400+0000 7fdf8d7fa640 20 client.114213.objecter put_session s=0x7fdf74046f80 osd=5 4 2024-09-17T18:37:30.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.400+0000 7fdf8d7fa640 10 client.114213.objecter _send_command 1 2024-09-17T18:37:30.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.400+0000 7fdf8d7fa640 1 -- 172.21.15.121:0/200394840 --> v2:172.21.15.173:6808/1801998802 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdf84022ec0 con 0x7fdf74047630 2024-09-17T18:37:30.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.400+0000 7fdf8d7fa640 20 client.114213.objecter dump_active .. 0 homeless 2024-09-17T18:37:30.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.400+0000 7fdf8d7fa640 1 -- 172.21.15.121:0/200394840 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_get_version_reply(handle=1 version=679) ==== 24+0+0 (secure 0 0 0) 0x7fdf840582a0 con 0x7fdf90077670 2024-09-17T18:37:30.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.400+0000 7fdf8d7fa640 10 monclient: handle_get_version_reply finishing 1 version 679 2024-09-17T18:37:30.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.400+0000 7fdf8ffff640 1 --2- 172.21.15.121:0/200394840 >> v2:172.21.15.173:6812/2423786037 conn(0x7fdf74040170 0x7fdf74042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:30.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.400+0000 7fdf8ffff640 10 monclient: get_auth_request con 0x7fdf74040170 auth_method 0 2024-09-17T18:37:30.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.400+0000 7fdf954d7640 1 --2- 172.21.15.121:0/200394840 >> v2:172.21.15.173:6808/1801998802 conn(0x7fdf74047630 0x7fdf74049ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:30.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.400+0000 7fdf954d7640 10 monclient: get_auth_request con 0x7fdf74047630 auth_method 0 2024-09-17T18:37:30.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.403+0000 7fdf8d7fa640 10 client.114213.objecter ms_handle_connect 0x7fdf74040170 2024-09-17T18:37:30.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.403+0000 7fdf8ffff640 1 --2- 172.21.15.121:0/200394840 >> v2:172.21.15.173:6812/2423786037 conn(0x7fdf74040170 0x7fdf74042630 secure :-1 s=READY pgs=4110 cs=0 l=1 rev1=1 crypto rx=0x7fdf9006f130 tx=0x7fdf7c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:30.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.405+0000 7fdf954d7640 1 --2- 172.21.15.121:0/200394840 >> v2:172.21.15.173:6808/1801998802 conn(0x7fdf74047630 0x7fdf74049ad0 crc :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:30.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.405+0000 7fdf8d7fa640 10 client.114213.objecter ms_handle_connect 0x7fdf74047630 2024-09-17T18:37:30.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.406+0000 7fdf8d7fa640 1 -- 172.21.15.121:0/200394840 <== osd.5 v2:172.21.15.173:6808/1801998802 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdf84022ec0 con 0x7fdf74047630 2024-09-17T18:37:30.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.406+0000 7fdf8d7fa640 10 client.114213.objecter ms_dispatch 0x7fdf90000b90 command_reply(tid 1: 0 ) 2024-09-17T18:37:30.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.406+0000 7fdf8d7fa640 10 client.114213.objecter _finish_command 1 = system:0 2024-09-17T18:37:30.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.406+0000 7fdf8d7fa640 20 client.114213.objecter put_session s=0x7fdf74046f80 osd=5 4 2024-09-17T18:37:30.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.406+0000 7fdf8d7fa640 15 client.114213.objecter _session_command_op_remove 5 1 2024-09-17T18:37:30.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.418+0000 7fdf96f3a640 10 client.114213.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-17T18:37:30.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.418+0000 7fdf96f3a640 15 client.114213.objecter _session_command_op_assign -1 2 2024-09-17T18:37:30.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.418+0000 7fdf96f3a640 20 client.114213.objecter _get_session s=0x7fdf74046f80 osd=5 3 2024-09-17T18:37:30.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.418+0000 7fdf96f3a640 20 client.114213.objecter put_session s=0x7fdf74046f80 osd=5 3 2024-09-17T18:37:30.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.418+0000 7fdf96f3a640 20 client.114213.objecter _get_session s=0x7fdf74046f80 osd=5 3 2024-09-17T18:37:30.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.418+0000 7fdf96f3a640 15 client.114213.objecter _session_command_op_remove -1 2 2024-09-17T18:37:30.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.418+0000 7fdf96f3a640 20 client.114213.objecter get_session s=0x7fdf74046f80 osd=5 3 2024-09-17T18:37:30.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.418+0000 7fdf96f3a640 15 client.114213.objecter _session_command_op_assign 5 2 2024-09-17T18:37:30.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.418+0000 7fdf96f3a640 20 client.114213.objecter put_session s=0x7fdf74046f80 osd=5 4 2024-09-17T18:37:30.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.418+0000 7fdf96f3a640 10 client.114213.objecter _send_command 2 2024-09-17T18:37:30.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.418+0000 7fdf96f3a640 1 -- 172.21.15.121:0/200394840 --> v2:172.21.15.173:6808/1801998802 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fdf901364c0 con 0x7fdf74047630 2024-09-17T18:37:30.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.419+0000 7fdf8d7fa640 1 -- 172.21.15.121:0/200394840 <== osd.5 v2:172.21.15.173:6808/1801998802 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fdf901364c0 con 0x7fdf74047630 2024-09-17T18:37:30.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.419+0000 7fdf8d7fa640 10 client.114213.objecter ms_dispatch 0x7fdf90000b90 command_reply(tid 2: 0 ) 2024-09-17T18:37:30.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.419+0000 7fdf8d7fa640 10 client.114213.objecter _finish_command 2 = system:0 2024-09-17T18:37:30.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.419+0000 7fdf8d7fa640 20 client.114213.objecter put_session s=0x7fdf74046f80 osd=5 4 2024-09-17T18:37:30.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.419+0000 7fdf8d7fa640 15 client.114213.objecter _session_command_op_remove 5 2 2024-09-17T18:37:30.421 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-17T18:37:30.422 INFO:teuthology.orchestra.run.smithi121.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T18:37:30.422 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-17T18:37:30.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.419+0000 7fdf96f3a640 10 librados: watch_flush enter 2024-09-17T18:37:30.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.419+0000 7fdf96f3a640 10 librados: watch_flush exit 2024-09-17T18:37:30.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.419+0000 7fdf96f3a640 10 client.114213.objecter close_session for osd.5 2024-09-17T18:37:30.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.419+0000 7fdf96f3a640 1 -- 172.21.15.121:0/200394840 >> v2:172.21.15.173:6808/1801998802 conn(0x7fdf74047630 msgr2=0x7fdf74049ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:30.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.419+0000 7fdf96f3a640 1 --2- 172.21.15.121:0/200394840 >> v2:172.21.15.173:6808/1801998802 conn(0x7fdf74047630 0x7fdf74049ad0 crc :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:30.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.419+0000 7fdf96f3a640 20 client.114213.objecter put_session s=0x7fdf74046f80 osd=5 1 2024-09-17T18:37:30.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.419+0000 7fdf96f3a640 20 client.114213.objecter shutdown clearing up homeless session... 2024-09-17T18:37:30.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.419+0000 7fdf96f3a640 10 client.114213.objecter successfully canceled tick 2024-09-17T18:37:30.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.419+0000 7fdf96f3a640 1 -- 172.21.15.121:0/200394840 >> v2:172.21.15.173:6812/2423786037 conn(0x7fdf74040170 msgr2=0x7fdf74042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:30.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.419+0000 7fdf96f3a640 1 --2- 172.21.15.121:0/200394840 >> v2:172.21.15.173:6812/2423786037 conn(0x7fdf74040170 0x7fdf74042630 secure :-1 s=READY pgs=4110 cs=0 l=1 rev1=1 crypto rx=0x7fdf9006f130 tx=0x7fdf7c02e000 comp rx=0 tx=0).stop 2024-09-17T18:37:30.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.419+0000 7fdf96f3a640 10 monclient: shutdown 2024-09-17T18:37:30.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.419+0000 7fdf96f3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:30.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.419+0000 7fdf96f3a640 1 -- 172.21.15.121:0/200394840 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf90077670 msgr2=0x7fdf90073c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:30.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.419+0000 7fdf96f3a640 1 --2- 172.21.15.121:0/200394840 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf90077670 0x7fdf90073c10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdf8400d5f0 tx=0x7fdf8400dab0 comp rx=0 tx=0).stop 2024-09-17T18:37:30.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.419+0000 7fdf954d7640 1 -- 172.21.15.121:0/200394840 reap_dead start 2024-09-17T18:37:30.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.419+0000 7fdf96f3a640 1 -- 172.21.15.121:0/200394840 shutdown_connections 2024-09-17T18:37:30.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.419+0000 7fdf96f3a640 1 -- 172.21.15.121:0/200394840 >> 172.21.15.121:0/200394840 conn(0x7fdf900a3a70 msgr2=0x7fdf90132fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:30.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.419+0000 7fdf96f3a640 1 -- 172.21.15.121:0/200394840 shutdown_connections 2024-09-17T18:37:30.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.419+0000 7fdf96f3a640 1 -- 172.21.15.121:0/200394840 wait complete. 2024-09-17T18:37:30.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.419+0000 7fdf96f3a640 1 librados: shutdown 2024-09-17T18:37:30.431 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-17T18:37:30.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.529+0000 7f0929324640 10 monclient: get_monmap_and_config 2024-09-17T18:37:30.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.529+0000 7f0929324640 10 monclient: build_initial_monmap 2024-09-17T18:37:30.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.529+0000 7f0929324640 10 monclient: monmap: 2024-09-17T18:37:30.531 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:30.531 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:30.531 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:37:30.530247+0000 2024-09-17T18:37:30.531 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:37:30.530247+0000 2024-09-17T18:37:30.532 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:30.532 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:30.532 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:37:30.532 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:37:30.532 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:37:30.532 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:37:30.532 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:37:30.532 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:30.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.530+0000 7f0929324640 1 Processor -- start 2024-09-17T18:37:30.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.530+0000 7f0929324640 1 -- start start 2024-09-17T18:37:30.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.530+0000 7f0929324640 10 monclient: init 2024-09-17T18:37:30.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.530+0000 7f0929324640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:30.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.530+0000 7f0929324640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-17T18:37:30.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.530+0000 7f0929324640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09240a5c00 0x7f092409d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:30.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.530+0000 7f0929324640 10 monclient(hunting): picked mon.noname-b con 0x7f09240a5c00 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:30.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.530+0000 7f0929324640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f09240a71f0 addr v1:172.21.15.121:6789/0 2024-09-17T18:37:30.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.530+0000 7f0929324640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f0924070000 0x7f092413fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:30.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.530+0000 7f0929324640 10 monclient(hunting): picked mon.noname-c con 0x7f0924070000 addr v2:172.21.15.173:3300/0 2024-09-17T18:37:30.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.530+0000 7f0929324640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f092409cbd0 con 0x7f09240a71f0 2024-09-17T18:37:30.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.530+0000 7f0929324640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:30.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.530+0000 7f0929324640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7f092409d190 con 0x7f0924070000 2024-09-17T18:37:30.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.530+0000 7f0929324640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:30.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.530+0000 7f0929324640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f092409ceb0 con 0x7f09240a5c00 2024-09-17T18:37:30.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.530+0000 7f0929324640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:30.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.530+0000 7f0929324640 10 monclient(hunting): authenticate will time out at 1854.246771s 2024-09-17T18:37:30.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.530+0000 7f091a7fc640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f09240a71f0 0x7f09240a5fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:52808/0 (socket says 172.21.15.121:52808) 2024-09-17T18:37:30.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.530+0000 7f0922ffd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09240a5c00 0x7f092409d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:30.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.530+0000 7f091a7fc640 1 -- 172.21.15.121:0/771329751 learned_addr learned my addr 172.21.15.121:0/771329751 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:37:30.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.530+0000 7f0922ffd640 10 monclient(hunting): get_auth_request con 0x7f09240a5c00 auth_method 0 2024-09-17T18:37:30.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.530+0000 7f0922ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:30.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.530+0000 7f0922ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:30.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.530+0000 7f0922ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:30.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.530+0000 7f0922ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:30.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.530+0000 7f0922ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:30.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.531+0000 7f0922ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:30.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.531+0000 7f0922ffd640 10 monclient(hunting): handle_auth_done global_id 64664 payload 274 2024-09-17T18:37:30.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.531+0000 7f0922ffd640 1 -- 172.21.15.121:0/771329751 >> v2:172.21.15.173:3300/0 conn(0x7f0924070000 msgr2=0x7f092413fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:30.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.531+0000 7f09237fe640 1 --2- 172.21.15.121:0/771329751 >> v2:172.21.15.173:3300/0 conn(0x7f0924070000 0x7f092413fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:30.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.531+0000 7f0922ffd640 1 --2- 172.21.15.121:0/771329751 >> v2:172.21.15.173:3300/0 conn(0x7f0924070000 0x7f092413fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:30.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.531+0000 7f0922ffd640 1 -- 172.21.15.121:0/771329751 >> v1:172.21.15.121:6789/0 conn(0x7f09240a71f0 legacy=0x7f09240a5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:30.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.531+0000 7f0922ffd640 10 monclient: _finish_hunting 0 2024-09-17T18:37:30.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.531+0000 7f0922ffd640 1 monclient: found mon.noname-b 2024-09-17T18:37:30.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.531+0000 7f0922ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:30.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.531+0000 7f0922ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:37:30.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.531+0000 7f0922ffd640 1 -- 172.21.15.121:0/771329751 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09240a66f0 con 0x7f09240a5c00 2024-09-17T18:37:30.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.533+0000 7f0922ffd640 1 -- 172.21.15.121:0/771329751 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09240a5c00 msgr2=0x7f092409d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:30.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.533+0000 7f0922ffd640 1 --2- 172.21.15.121:0/771329751 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09240a5c00 0x7f092409d890 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f09140097a0 tx=0x7f0914029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=780982fd10c91781 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:30.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.534+0000 7f09227fc640 1 -- 172.21.15.121:0/771329751 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0914037070 con 0x7f09240a5c00 2024-09-17T18:37:30.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.534+0000 7f09227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:30.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.534+0000 7f09227fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:37:30.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.534+0000 7f09227fc640 10 monclient: dump: 2024-09-17T18:37:30.544 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:30.544 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:30.544 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:30.544 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:30.544 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:30.544 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:30.544 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:30.544 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:30.544 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:30.544 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:30.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.534+0000 7f09227fc640 10 monclient: _finish_auth 0 2024-09-17T18:37:30.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.534+0000 7f09227fc640 10 monclient: _check_auth_tickets 2024-09-17T18:37:30.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.534+0000 7f09227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:30.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.534+0000 7f09227fc640 1 -- 172.21.15.121:0/771329751 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0914032640 con 0x7f09240a5c00 2024-09-17T18:37:30.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.534+0000 7f0929324640 5 monclient: authenticate success, global_id 64664 2024-09-17T18:37:30.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.534+0000 7f0929324640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:37:30.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.534+0000 7f09227fc640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:30.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.534+0000 7f0929324640 10 monclient: get_monmap_and_config success 2024-09-17T18:37:30.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.534+0000 7f09227fc640 1 -- 172.21.15.121:0/771329751 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0914032a10 con 0x7f09240a5c00 2024-09-17T18:37:30.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.534+0000 7f0929324640 4 set_mon_vals no callback set 2024-09-17T18:37:30.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.534+0000 7f09227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:30.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.534+0000 7f09227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:37:30.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.534+0000 7f09227fc640 10 monclient: dump: 2024-09-17T18:37:30.545 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:30.545 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:30.545 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:30.546 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:30.546 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:30.546 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:30.546 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:30.546 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:30.546 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:30.546 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:30.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.534+0000 7f0929324640 10 monclient: shutdown 2024-09-17T18:37:30.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.534+0000 7f0929324640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:30.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.534+0000 7f0929324640 1 -- 172.21.15.121:0/771329751 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09240a5c00 msgr2=0x7f092409d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:30.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.534+0000 7f0929324640 1 --2- 172.21.15.121:0/771329751 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09240a5c00 0x7f092409d890 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f09140097a0 tx=0x7f0914029fd0 comp rx=0 tx=0).stop 2024-09-17T18:37:30.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.534+0000 7f0929324640 1 -- 172.21.15.121:0/771329751 shutdown_connections 2024-09-17T18:37:30.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.534+0000 7f0929324640 1 --2- 172.21.15.121:0/771329751 >> v2:172.21.15.173:3300/0 conn(0x7f0924070000 0x7f092413fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:30.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.534+0000 7f0929324640 1 --2- 172.21.15.121:0/771329751 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09240a5c00 0x7f092409d890 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:30.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.534+0000 7f0929324640 1 -- 172.21.15.121:0/771329751 >> 172.21.15.121:0/771329751 conn(0x7f09240ab620 msgr2=0x7f09240aba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:30.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.534+0000 7f0929324640 1 -- 172.21.15.121:0/771329751 shutdown_connections 2024-09-17T18:37:30.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.534+0000 7f0929324640 1 -- 172.21.15.121:0/771329751 wait complete. 2024-09-17T18:37:30.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.535+0000 7f0929324640 10 monclient: build_initial_monmap 2024-09-17T18:37:30.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.535+0000 7f0929324640 10 monclient: monmap: 2024-09-17T18:37:30.547 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:30.547 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:37:30.547 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:37:30.547 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:37:30.547 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:30.547 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:30.547 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:37:30.547 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:37:30.547 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:37:30.548 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:30.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.535+0000 7f0929324640 1 librados: starting msgr at 2024-09-17T18:37:30.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.535+0000 7f0929324640 1 librados: starting objecter 2024-09-17T18:37:30.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.535+0000 7f0929324640 1 Processor -- start 2024-09-17T18:37:30.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.535+0000 7f0929324640 1 -- start start 2024-09-17T18:37:30.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.535+0000 7f0929324640 1 librados: setting wanted keys 2024-09-17T18:37:30.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.535+0000 7f0929324640 1 librados: calling monclient init 2024-09-17T18:37:30.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.535+0000 7f0929324640 10 monclient: init 2024-09-17T18:37:30.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f0929324640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:30.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f0929324640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T18:37:30.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f0929324640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0924070000 0x7f0924121dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:30.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f0929324640 10 monclient(hunting): picked mon.noname-a con 0x7f0924070000 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:37:30.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f0929324640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09240a5c00 0x7f092413e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:30.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f0929324640 10 monclient(hunting): picked mon.noname-c con 0x7f09240a5c00 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:30.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f0929324640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09240a71f0 0x7f0924138c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:30.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f0929324640 10 monclient(hunting): picked mon.noname-b con 0x7f09240a71f0 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:37:30.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f0929324640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:30.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f0929324640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0924140360 con 0x7f0924070000 2024-09-17T18:37:30.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f0929324640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:30.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f0929324640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f09240a6a60 con 0x7f09240a71f0 2024-09-17T18:37:30.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f0929324640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:30.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f0929324640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f09241404e0 con 0x7f09240a5c00 2024-09-17T18:37:30.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f0929324640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:30.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f0929324640 10 monclient(hunting): authenticate will time out at 1854.252161s 2024-09-17T18:37:30.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f0922ffd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0924070000 0x7f0924121dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:30.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f091a7fc640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09240a5c00 0x7f092413e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:30.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f0922ffd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0924070000 0x7f0924121dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38558/0 (socket says 172.21.15.121:38558) 2024-09-17T18:37:30.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f091a7fc640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09240a5c00 0x7f092413e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:58198/0 (socket says 172.21.15.121:58198) 2024-09-17T18:37:30.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f0922ffd640 1 -- 172.21.15.121:0/3246477965 learned_addr learned my addr 172.21.15.121:0/3246477965 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:37:30.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f0922ffd640 10 monclient(hunting): get_auth_request con 0x7f0924070000 auth_method 0 2024-09-17T18:37:30.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f0922ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:30.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f0922ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:30.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f0922ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:30.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f091a7fc640 10 monclient(hunting): get_auth_request con 0x7f09240a5c00 auth_method 0 2024-09-17T18:37:30.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f091a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:30.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f091a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:30.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f091a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:30.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f0922ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:30.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f0922ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:30.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f0922ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:30.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f091a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:30.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f091a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:30.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f091a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:30.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f09237fe640 1 --2- 172.21.15.121:0/3246477965 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09240a71f0 0x7f0924138c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:30.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f09237fe640 10 monclient(hunting): get_auth_request con 0x7f09240a71f0 auth_method 0 2024-09-17T18:37:30.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f09237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:30.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f09237fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:30.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f09237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:30.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f091a7fc640 10 monclient(hunting): handle_auth_done global_id 64667 payload 824 2024-09-17T18:37:30.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f091a7fc640 1 -- 172.21.15.121:0/3246477965 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09240a71f0 msgr2=0x7f0924138c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:30.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f091a7fc640 1 --2- 172.21.15.121:0/3246477965 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09240a71f0 0x7f0924138c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:30.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f091a7fc640 1 -- 172.21.15.121:0/3246477965 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0924070000 msgr2=0x7f0924121dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:30.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f091a7fc640 1 --2- 172.21.15.121:0/3246477965 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0924070000 0x7f0924121dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:30.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f091a7fc640 10 monclient: _finish_hunting 0 2024-09-17T18:37:30.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f091a7fc640 1 monclient: found mon.noname-c 2024-09-17T18:37:30.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f091a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:30.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f091a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:37:30.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f091a7fc640 1 -- 172.21.15.121:0/3246477965 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f092413f860 con 0x7f09240a5c00 2024-09-17T18:37:30.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f0922ffd640 1 --2- 172.21.15.121:0/3246477965 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0924070000 0x7f0924121dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:37:30.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.536+0000 7f09237fe640 1 --2- 172.21.15.121:0/3246477965 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09240a71f0 0x7f0924138c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T18:37:30.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.538+0000 7f091a7fc640 1 -- 172.21.15.121:0/3246477965 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09240a5c00 msgr2=0x7f092413e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:30.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.538+0000 7f091a7fc640 1 --2- 172.21.15.121:0/3246477965 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09240a5c00 0x7f092413e320 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f09240a23c0 tx=0x7f090c00c870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:30.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.538+0000 7f091bfff640 10 client.?.objecter ms_handle_connect 0x7f09240a5c00 2024-09-17T18:37:30.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.538+0000 7f091bfff640 10 client.?.objecter resend_mon_ops 2024-09-17T18:37:30.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.538+0000 7f091bfff640 1 -- 172.21.15.121:0/3246477965 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f090c017070 con 0x7f09240a5c00 2024-09-17T18:37:30.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.538+0000 7f091bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:30.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.538+0000 7f091bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:37:30.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.538+0000 7f091bfff640 10 monclient: dump: 2024-09-17T18:37:30.552 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:30.552 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:30.553 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:30.553 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:30.553 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:30.553 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:30.553 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:30.553 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:30.553 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:30.553 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:30.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.538+0000 7f091bfff640 10 monclient: _finish_auth 0 2024-09-17T18:37:30.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.538+0000 7f091bfff640 10 monclient: _check_auth_tickets 2024-09-17T18:37:30.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.538+0000 7f091bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:30.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.538+0000 7f091bfff640 1 -- 172.21.15.121:0/3246477965 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f090c0070d0 con 0x7f09240a5c00 2024-09-17T18:37:30.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.538+0000 7f0929324640 5 monclient: authenticate success, global_id 64667 2024-09-17T18:37:30.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.538+0000 7f091bfff640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:30.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.538+0000 7f091bfff640 1 -- 172.21.15.121:0/3246477965 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f090c0074c0 con 0x7f09240a5c00 2024-09-17T18:37:30.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.538+0000 7f091bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:30.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.538+0000 7f091bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:37:30.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.538+0000 7f091bfff640 10 monclient: dump: 2024-09-17T18:37:30.554 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:30.554 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:30.554 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:30.554 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:30.554 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:30.554 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:30.554 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:30.554 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:30.554 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:30.554 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:30.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.538+0000 7f0929324640 10 monclient: _renew_subs 2024-09-17T18:37:30.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.538+0000 7f0929324640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:37:30.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.538+0000 7f0929324640 1 -- 172.21.15.121:0/3246477965 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09240aa640 con 0x7f09240a5c00 2024-09-17T18:37:30.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.538+0000 7f0921ffb640 4 set_mon_vals no callback set 2024-09-17T18:37:30.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.539+0000 7f0929324640 10 client.64667.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:37:30.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.539+0000 7f0929324640 10 monclient: _renew_subs 2024-09-17T18:37:30.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.539+0000 7f0929324640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:37:30.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.539+0000 7f0929324640 1 -- 172.21.15.121:0/3246477965 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0924122d30 con 0x7f09240a5c00 2024-09-17T18:37:30.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.539+0000 7f0929324640 1 librados: init done 2024-09-17T18:37:30.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.539+0000 7f0929324640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:30.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.539+0000 7f0929324640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:30.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.539+0000 7f0929324640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:37:30.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.539+0000 7f091bfff640 1 -- 172.21.15.121:0/3246477965 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f090c01a070 con 0x7f09240a5c00 2024-09-17T18:37:30.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.539+0000 7f0929324640 1 -- 172.21.15.121:0/3246477965 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f092409d890 con 0x7f09240a5c00 2024-09-17T18:37:30.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.540+0000 7f091bfff640 1 --2- 172.21.15.121:0/3246477965 >> v2:172.21.15.173:6812/2423786037 conn(0x7f0904040170 0x7f0904042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:30.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.540+0000 7f091bfff640 1 -- 172.21.15.121:0/3246477965 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(679..679 src has 247..679) ==== 4685+0+0 (secure 0 0 0) 0x7f090c052890 con 0x7f09240a5c00 2024-09-17T18:37:30.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.540+0000 7f091bfff640 10 client.64667.objecter ms_dispatch 0x7f0924000b90 osd_map(679..679 src has 247..679) 2024-09-17T18:37:30.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.540+0000 7f091bfff640 3 client.64667.objecter handle_osd_map got epochs [679,679] > 0 2024-09-17T18:37:30.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.540+0000 7f091bfff640 3 client.64667.objecter handle_osd_map decoding full epoch 679 2024-09-17T18:37:30.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.540+0000 7f091bfff640 20 client.64667.objecter dump_active .. 0 homeless 2024-09-17T18:37:30.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.540+0000 7f0922ffd640 1 --2- 172.21.15.121:0/3246477965 >> v2:172.21.15.173:6812/2423786037 conn(0x7f0904040170 0x7f0904042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:30.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.540+0000 7f0922ffd640 10 monclient: get_auth_request con 0x7f0904040170 auth_method 0 2024-09-17T18:37:30.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.543+0000 7f091bfff640 10 client.64667.objecter ms_handle_connect 0x7f0904040170 2024-09-17T18:37:30.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.543+0000 7f0922ffd640 1 --2- 172.21.15.121:0/3246477965 >> v2:172.21.15.173:6812/2423786037 conn(0x7f0904040170 0x7f0904042630 secure :-1 s=READY pgs=4111 cs=0 l=1 rev1=1 crypto rx=0x7f091402a010 tx=0x7f091402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:30.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.544+0000 7f091bfff640 1 -- 172.21.15.121:0/3246477965 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f090c0076e0 con 0x7f09240a5c00 2024-09-17T18:37:30.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.544+0000 7f091bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:30.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.544+0000 7f091bfff640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:37:30.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.719+0000 7f0929324640 1 -- 172.21.15.121:0/3246477965 --> v2:172.21.15.173:6812/2423786037 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f09240a23f0 con 0x7f0904040170 2024-09-17T18:37:30.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.720+0000 7f091bfff640 1 -- 172.21.15.121:0/3246477965 <== mgr.4105 v2:172.21.15.173:6812/2423786037 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f09240a23f0 con 0x7f0904040170 2024-09-17T18:37:30.722 INFO:teuthology.orchestra.run.smithi121.stderr:instructed osd(s) 5 to deep-scrub 2024-09-17T18:37:30.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.721+0000 7f0929324640 10 librados: watch_flush enter 2024-09-17T18:37:30.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.721+0000 7f0929324640 10 librados: watch_flush exit 2024-09-17T18:37:30.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.721+0000 7f0929324640 20 client.64667.objecter shutdown clearing up homeless session... 2024-09-17T18:37:30.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.721+0000 7f0929324640 10 client.64667.objecter successfully canceled tick 2024-09-17T18:37:30.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.722+0000 7f0929324640 1 -- 172.21.15.121:0/3246477965 >> v2:172.21.15.173:6812/2423786037 conn(0x7f0904040170 msgr2=0x7f0904042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:30.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.722+0000 7f0929324640 1 --2- 172.21.15.121:0/3246477965 >> v2:172.21.15.173:6812/2423786037 conn(0x7f0904040170 0x7f0904042630 secure :-1 s=READY pgs=4111 cs=0 l=1 rev1=1 crypto rx=0x7f091402a010 tx=0x7f091402e000 comp rx=0 tx=0).stop 2024-09-17T18:37:30.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.722+0000 7f0929324640 10 monclient: shutdown 2024-09-17T18:37:30.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.722+0000 7f0929324640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:30.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.722+0000 7f0929324640 1 -- 172.21.15.121:0/3246477965 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09240a5c00 msgr2=0x7f092413e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:30.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.722+0000 7f0929324640 1 --2- 172.21.15.121:0/3246477965 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09240a5c00 0x7f092413e320 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f09240a23c0 tx=0x7f090c00c870 comp rx=0 tx=0).stop 2024-09-17T18:37:30.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.722+0000 7f0929324640 1 -- 172.21.15.121:0/3246477965 shutdown_connections 2024-09-17T18:37:30.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.722+0000 7f0929324640 1 --2- 172.21.15.121:0/3246477965 >> v2:172.21.15.173:6812/2423786037 conn(0x7f0904040170 0x7f0904042630 unknown :-1 s=CLOSED pgs=4111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:30.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.722+0000 7f0929324640 1 --2- 172.21.15.121:0/3246477965 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09240a71f0 0x7f0924138c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:30.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.722+0000 7f0929324640 1 --2- 172.21.15.121:0/3246477965 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09240a5c00 0x7f092413e320 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:30.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.722+0000 7f0929324640 1 --2- 172.21.15.121:0/3246477965 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0924070000 0x7f0924121dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:30.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.722+0000 7f0929324640 1 -- 172.21.15.121:0/3246477965 >> 172.21.15.121:0/3246477965 conn(0x7f09240ab620 msgr2=0x7f09240a47f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:30.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.722+0000 7f0929324640 1 -- 172.21.15.121:0/3246477965 shutdown_connections 2024-09-17T18:37:30.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.722+0000 7f0929324640 1 -- 172.21.15.121:0/3246477965 wait complete. 2024-09-17T18:37:30.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.722+0000 7f0929324640 1 librados: shutdown 2024-09-17T18:37:30.734 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T18:37:30.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.829+0000 7fc7e0f0e640 10 monclient: get_monmap_and_config 2024-09-17T18:37:30.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.829+0000 7fc7e0f0e640 10 monclient: build_initial_monmap 2024-09-17T18:37:30.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.829+0000 7fc7e0f0e640 10 monclient: monmap: 2024-09-17T18:37:30.831 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:30.831 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:30.831 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:37:30.829998+0000 2024-09-17T18:37:30.831 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:37:30.829998+0000 2024-09-17T18:37:30.831 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:30.831 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:30.831 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:37:30.831 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:37:30.832 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:37:30.832 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:37:30.832 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:37:30.832 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:30.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.830+0000 7fc7e0f0e640 1 Processor -- start 2024-09-17T18:37:30.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.830+0000 7fc7e0f0e640 1 -- start start 2024-09-17T18:37:30.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.830+0000 7fc7e0f0e640 10 monclient: init 2024-09-17T18:37:30.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.830+0000 7fc7e0f0e640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:30.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.830+0000 7fc7e0f0e640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-17T18:37:30.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.830+0000 7fc7e0f0e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc7dc0a33f0 addr v1:172.21.15.173:6789/0 2024-09-17T18:37:30.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.830+0000 7fc7e0f0e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc7dc09da70 addr v1:172.21.15.121:6789/0 2024-09-17T18:37:30.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.830+0000 7fc7e0f0e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7dc13b860 0x7fc7dc13dc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:30.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.830+0000 7fc7e0f0e640 10 monclient(hunting): picked mon.noname-b con 0x7fc7dc13b860 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:30.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.830+0000 7fc7e0f0e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7dc074b40 con 0x7fc7dc09da70 2024-09-17T18:37:30.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.830+0000 7fc7e0f0e640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7dc09f370 con 0x7fc7dc0a33f0 2024-09-17T18:37:30.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.830+0000 7fc7e0f0e640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:30.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.830+0000 7fc7e0f0e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc7dc074550 con 0x7fc7dc13b860 2024-09-17T18:37:30.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.830+0000 7fc7e0f0e640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:30.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.830+0000 7fc7e0f0e640 10 monclient(hunting): authenticate will time out at 1854.546640s 2024-09-17T18:37:30.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.830+0000 7fc7dad9d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7dc13b860 0x7fc7dc13dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:30.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.830+0000 7fc7d3fff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fc7dc09da70 0x7fc7dc09ec60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:52812/0 (socket says 172.21.15.121:52812) 2024-09-17T18:37:30.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.830+0000 7fc7d3fff640 1 -- 172.21.15.121:0/3533740561 learned_addr learned my addr 172.21.15.121:0/3533740561 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:37:30.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.830+0000 7fc7dad9d640 10 monclient(hunting): get_auth_request con 0x7fc7dc13b860 auth_method 0 2024-09-17T18:37:30.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.830+0000 7fc7dad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:30.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.830+0000 7fc7dad9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:30.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.830+0000 7fc7dad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:30.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.830+0000 7fc7dad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:30.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.830+0000 7fc7dad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:30.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.830+0000 7fc7dad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:30.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.830+0000 7fc7dad9d640 10 monclient(hunting): handle_auth_done global_id 64670 payload 274 2024-09-17T18:37:30.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.830+0000 7fc7dad9d640 1 -- 172.21.15.121:0/3533740561 >> v1:172.21.15.173:6789/0 conn(0x7fc7dc0a33f0 legacy=0x7fc7dc09e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:30.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.830+0000 7fc7dad9d640 1 -- 172.21.15.121:0/3533740561 >> v1:172.21.15.121:6789/0 conn(0x7fc7dc09da70 legacy=0x7fc7dc09ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:30.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.830+0000 7fc7dad9d640 10 monclient: _finish_hunting 0 2024-09-17T18:37:30.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.830+0000 7fc7dad9d640 1 monclient: found mon.noname-b 2024-09-17T18:37:30.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.830+0000 7fc7dad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:30.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.830+0000 7fc7dad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:37:30.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.831+0000 7fc7dad9d640 1 -- 172.21.15.121:0/3533740561 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7dc0a13c0 con 0x7fc7dc13b860 2024-09-17T18:37:30.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.833+0000 7fc7dad9d640 1 -- 172.21.15.121:0/3533740561 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7dc13b860 msgr2=0x7fc7dc13dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:30.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.833+0000 7fc7dad9d640 1 --2- 172.21.15.121:0/3533740561 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7dc13b860 0x7fc7dc13dc50 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc7d4009dc0 tx=0x7fc7d402a480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77a18811bd544e29 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:30.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.833+0000 7fc7d9d9b640 1 -- 172.21.15.121:0/3533740561 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d4007a30 con 0x7fc7dc13b860 2024-09-17T18:37:30.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.833+0000 7fc7d9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:30.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.833+0000 7fc7d9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:37:30.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.833+0000 7fc7d9d9b640 10 monclient: dump: 2024-09-17T18:37:30.843 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:30.843 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:30.843 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:30.844 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:30.844 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:30.844 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:30.844 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:30.844 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:30.844 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:30.844 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:30.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.833+0000 7fc7d9d9b640 10 monclient: _finish_auth 0 2024-09-17T18:37:30.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.833+0000 7fc7d9d9b640 10 monclient: _check_auth_tickets 2024-09-17T18:37:30.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.833+0000 7fc7d9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:30.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.833+0000 7fc7d9d9b640 1 -- 172.21.15.121:0/3533740561 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7d4007bd0 con 0x7fc7dc13b860 2024-09-17T18:37:30.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.833+0000 7fc7e0f0e640 5 monclient: authenticate success, global_id 64670 2024-09-17T18:37:30.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.833+0000 7fc7e0f0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:37:30.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.833+0000 7fc7d9d9b640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:30.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.833+0000 7fc7e0f0e640 10 monclient: get_monmap_and_config success 2024-09-17T18:37:30.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.833+0000 7fc7d9d9b640 1 -- 172.21.15.121:0/3533740561 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d4031400 con 0x7fc7dc13b860 2024-09-17T18:37:30.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.833+0000 7fc7e0f0e640 4 set_mon_vals no callback set 2024-09-17T18:37:30.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.833+0000 7fc7d9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:30.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.833+0000 7fc7d9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:37:30.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.833+0000 7fc7d9d9b640 10 monclient: dump: 2024-09-17T18:37:30.845 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:30.845 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:30.845 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:30.846 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:30.846 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:30.846 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:30.846 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:30.846 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:30.846 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:30.846 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:30.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.834+0000 7fc7e0f0e640 10 monclient: shutdown 2024-09-17T18:37:30.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.834+0000 7fc7e0f0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:30.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.834+0000 7fc7e0f0e640 1 -- 172.21.15.121:0/3533740561 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7dc13b860 msgr2=0x7fc7dc13dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:30.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.834+0000 7fc7e0f0e640 1 --2- 172.21.15.121:0/3533740561 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7dc13b860 0x7fc7dc13dc50 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc7d4009dc0 tx=0x7fc7d402a480 comp rx=0 tx=0).stop 2024-09-17T18:37:30.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.834+0000 7fc7e0f0e640 1 -- 172.21.15.121:0/3533740561 shutdown_connections 2024-09-17T18:37:30.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.834+0000 7fc7e0f0e640 1 --2- 172.21.15.121:0/3533740561 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7dc13b860 0x7fc7dc13dc50 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:30.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.834+0000 7fc7e0f0e640 1 -- 172.21.15.121:0/3533740561 >> 172.21.15.121:0/3533740561 conn(0x7fc7dc0a3b50 msgr2=0x7fc7dc0a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:30.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.834+0000 7fc7e0f0e640 1 -- 172.21.15.121:0/3533740561 shutdown_connections 2024-09-17T18:37:30.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.834+0000 7fc7e0f0e640 1 -- 172.21.15.121:0/3533740561 wait complete. 2024-09-17T18:37:30.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.835+0000 7fc7e0f0e640 10 monclient: build_initial_monmap 2024-09-17T18:37:30.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.835+0000 7fc7e0f0e640 10 monclient: monmap: 2024-09-17T18:37:30.847 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:30.847 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:37:30.847 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:37:30.847 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:37:30.847 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:30.847 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:30.847 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:37:30.847 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:37:30.847 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:37:30.848 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:30.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.835+0000 7fc7e0f0e640 1 librados: starting msgr at 2024-09-17T18:37:30.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.835+0000 7fc7e0f0e640 1 librados: starting objecter 2024-09-17T18:37:30.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.835+0000 7fc7e0f0e640 1 Processor -- start 2024-09-17T18:37:30.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.835+0000 7fc7e0f0e640 1 -- start start 2024-09-17T18:37:30.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.835+0000 7fc7e0f0e640 1 librados: setting wanted keys 2024-09-17T18:37:30.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.835+0000 7fc7e0f0e640 1 librados: calling monclient init 2024-09-17T18:37:30.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.835+0000 7fc7e0f0e640 10 monclient: init 2024-09-17T18:37:30.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.835+0000 7fc7e0f0e640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:30.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.835+0000 7fc7e0f0e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:37:30.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7e0f0e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7dc0a33f0 0x7fc7dc0a2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:30.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7e0f0e640 10 monclient(hunting): picked mon.noname-a con 0x7fc7dc0a33f0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:37:30.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7e0f0e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7dc13b860 0x7fc7dc136df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:30.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7e0f0e640 10 monclient(hunting): picked mon.noname-b con 0x7fc7dc13b860 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:37:30.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7e0f0e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7dc137330 0x7fc7dc072b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:30.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7e0f0e640 10 monclient(hunting): picked mon.noname-c con 0x7fc7dc137330 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:30.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7e0f0e640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:30.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7e0f0e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc7dc13e190 con 0x7fc7dc0a33f0 2024-09-17T18:37:30.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7e0f0e640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:30.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7e0f0e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc7dc09f7b0 con 0x7fc7dc13b860 2024-09-17T18:37:30.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7e0f0e640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:30.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7e0f0e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc7dc13e310 con 0x7fc7dc137330 2024-09-17T18:37:30.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7e0f0e640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:30.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7e0f0e640 10 monclient(hunting): authenticate will time out at 1854.552073s 2024-09-17T18:37:30.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7da59c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7dc0a33f0 0x7fc7dc0a2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:30.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7dad9d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7dc137330 0x7fc7dc072b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:30.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7da59c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7dc0a33f0 0x7fc7dc0a2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38570/0 (socket says 172.21.15.121:38570) 2024-09-17T18:37:30.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7da59c640 1 -- 172.21.15.121:0/1223180838 learned_addr learned my addr 172.21.15.121:0/1223180838 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:37:30.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7da59c640 10 monclient(hunting): get_auth_request con 0x7fc7dc0a33f0 auth_method 0 2024-09-17T18:37:30.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7da59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:30.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7da59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:30.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7da59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:30.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7dad9d640 10 monclient(hunting): get_auth_request con 0x7fc7dc137330 auth_method 0 2024-09-17T18:37:30.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7dad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:30.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7dad9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:30.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7dad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:30.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7da59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:30.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7da59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:30.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7da59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:30.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7dad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:30.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7dad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:30.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7dad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:30.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7d3fff640 1 --2- 172.21.15.121:0/1223180838 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7dc13b860 0x7fc7dc136df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:30.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7d3fff640 10 monclient(hunting): get_auth_request con 0x7fc7dc13b860 auth_method 0 2024-09-17T18:37:30.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:30.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:30.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:30.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7da59c640 10 monclient(hunting): handle_auth_done global_id 114219 payload 824 2024-09-17T18:37:30.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7da59c640 1 -- 172.21.15.121:0/1223180838 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7dc137330 msgr2=0x7fc7dc072b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:30.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7da59c640 1 --2- 172.21.15.121:0/1223180838 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7dc137330 0x7fc7dc072b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:30.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7da59c640 1 -- 172.21.15.121:0/1223180838 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7dc13b860 msgr2=0x7fc7dc136df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:30.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7da59c640 1 --2- 172.21.15.121:0/1223180838 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7dc13b860 0x7fc7dc136df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:30.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7da59c640 10 monclient: _finish_hunting 0 2024-09-17T18:37:30.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7da59c640 1 monclient: found mon.noname-a 2024-09-17T18:37:30.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7da59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:30.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7da59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:37:30.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7da59c640 1 -- 172.21.15.121:0/1223180838 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7dc09e240 con 0x7fc7dc0a33f0 2024-09-17T18:37:30.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7d3fff640 1 --2- 172.21.15.121:0/1223180838 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7dc13b860 0x7fc7dc136df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T18:37:30.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.836+0000 7fc7dad9d640 1 --2- 172.21.15.121:0/1223180838 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7dc137330 0x7fc7dc072b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:37:30.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.838+0000 7fc7da59c640 1 -- 172.21.15.121:0/1223180838 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7dc0a33f0 msgr2=0x7fc7dc0a2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:30.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.838+0000 7fc7da59c640 1 --2- 172.21.15.121:0/1223180838 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7dc0a33f0 0x7fc7dc0a2310 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc00c430 tx=0x7fc7cc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:30.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.838+0000 7fc7d2ffd640 10 client.?.objecter ms_handle_connect 0x7fc7dc0a33f0 2024-09-17T18:37:30.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.838+0000 7fc7d2ffd640 10 client.?.objecter resend_mon_ops 2024-09-17T18:37:30.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.838+0000 7fc7d2ffd640 1 -- 172.21.15.121:0/1223180838 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc0119a0 con 0x7fc7dc0a33f0 2024-09-17T18:37:30.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.838+0000 7fc7d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:30.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.838+0000 7fc7d2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:37:30.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.838+0000 7fc7d2ffd640 10 monclient: dump: 2024-09-17T18:37:30.853 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:30.853 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:30.853 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:30.853 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:30.853 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:30.853 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:30.854 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:30.854 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:30.854 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:30.854 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:30.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.838+0000 7fc7d2ffd640 10 monclient: _finish_auth 0 2024-09-17T18:37:30.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.838+0000 7fc7d2ffd640 10 monclient: _check_auth_tickets 2024-09-17T18:37:30.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.838+0000 7fc7d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:30.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.838+0000 7fc7d2ffd640 1 -- 172.21.15.121:0/1223180838 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7cc017070 con 0x7fc7dc0a33f0 2024-09-17T18:37:30.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.838+0000 7fc7e0f0e640 5 monclient: authenticate success, global_id 114219 2024-09-17T18:37:30.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.838+0000 7fc7e0f0e640 10 monclient: _renew_subs 2024-09-17T18:37:30.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.838+0000 7fc7e0f0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:30.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.838+0000 7fc7e0f0e640 1 -- 172.21.15.121:0/1223180838 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7dc11f2c0 con 0x7fc7dc0a33f0 2024-09-17T18:37:30.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.838+0000 7fc7d2ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:30.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.838+0000 7fc7d2ffd640 1 -- 172.21.15.121:0/1223180838 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc011d80 con 0x7fc7dc0a33f0 2024-09-17T18:37:30.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.838+0000 7fc7d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:30.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.838+0000 7fc7d2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:37:30.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.838+0000 7fc7d2ffd640 10 monclient: dump: 2024-09-17T18:37:30.855 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:30.855 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:30.855 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:30.855 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:30.855 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:30.855 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:30.855 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:30.856 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:30.856 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:30.856 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:30.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.838+0000 7fc7d959a640 4 set_mon_vals no callback set 2024-09-17T18:37:30.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.838+0000 7fc7e0f0e640 10 client.114219.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:37:30.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.838+0000 7fc7e0f0e640 10 monclient: _renew_subs 2024-09-17T18:37:30.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.838+0000 7fc7e0f0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:30.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.838+0000 7fc7e0f0e640 1 -- 172.21.15.121:0/1223180838 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7dc11efe0 con 0x7fc7dc0a33f0 2024-09-17T18:37:30.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.838+0000 7fc7e0f0e640 1 librados: init done 2024-09-17T18:37:30.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.839+0000 7fc7d2ffd640 1 -- 172.21.15.121:0/1223180838 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fc7cc01c460 con 0x7fc7dc0a33f0 2024-09-17T18:37:30.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.839+0000 7fc7d2ffd640 1 --2- 172.21.15.121:0/1223180838 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc7bc0401a0 0x7fc7bc042660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:30.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.839+0000 7fc7e0f0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:30.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.839+0000 7fc7e0f0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:30.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.839+0000 7fc7e0f0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:30.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.839+0000 7fc7d2ffd640 1 -- 172.21.15.121:0/1223180838 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(679..679 src has 247..679) ==== 4685+0+0 (secure 0 0 0) 0x7fc7cc053270 con 0x7fc7dc0a33f0 2024-09-17T18:37:30.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.839+0000 7fc7d2ffd640 10 client.114219.objecter ms_dispatch 0x7fc7dc000b90 osd_map(679..679 src has 247..679) 2024-09-17T18:37:30.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.839+0000 7fc7d2ffd640 3 client.114219.objecter handle_osd_map got epochs [679,679] > 0 2024-09-17T18:37:30.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.839+0000 7fc7d2ffd640 3 client.114219.objecter handle_osd_map decoding full epoch 679 2024-09-17T18:37:30.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.839+0000 7fc7e0f0e640 1 -- 172.21.15.121:0/1223180838 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7dc09e550 con 0x7fc7dc0a33f0 2024-09-17T18:37:30.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.840+0000 7fc7d2ffd640 20 client.114219.objecter dump_active .. 0 homeless 2024-09-17T18:37:30.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.840+0000 7fc7d3fff640 1 --2- 172.21.15.121:0/1223180838 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc7bc0401a0 0x7fc7bc042660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:30.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.840+0000 7fc7d3fff640 10 monclient: get_auth_request con 0x7fc7bc0401a0 auth_method 0 2024-09-17T18:37:30.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.842+0000 7fc7d3fff640 1 --2- 172.21.15.121:0/1223180838 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc7bc0401a0 0x7fc7bc042660 secure :-1 s=READY pgs=4112 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc0a2060 tx=0x7fc7c4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:30.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.842+0000 7fc7d2ffd640 10 client.114219.objecter ms_handle_connect 0x7fc7bc0401a0 2024-09-17T18:37:30.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.844+0000 7fc7d2ffd640 1 -- 172.21.15.121:0/1223180838 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc7dc09e240 con 0x7fc7dc0a33f0 2024-09-17T18:37:30.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.844+0000 7fc7d2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:30.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:30.844+0000 7fc7d2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:37:31.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:31.016+0000 7fc7e0f0e640 1 -- 172.21.15.121:0/1223180838 --> v2:172.21.15.173:6812/2423786037 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc7dc11ebb0 con 0x7fc7bc0401a0 2024-09-17T18:37:31.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:31.018+0000 7fc7d2ffd640 1 -- 172.21.15.121:0/1223180838 <== mgr.4105 v2:172.21.15.173:6812/2423786037 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114139 (secure 0 0 0) 0x7fc7c40094c0 con 0x7fc7bc0401a0 2024-09-17T18:37:31.021 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:37:31.022 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-09-17T18:37:31.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:31.021+0000 7fc7e0f0e640 10 librados: watch_flush enter 2024-09-17T18:37:31.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:31.021+0000 7fc7e0f0e640 10 librados: watch_flush exit 2024-09-17T18:37:31.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:31.021+0000 7fc7e0f0e640 20 client.114219.objecter shutdown clearing up homeless session... 2024-09-17T18:37:31.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:31.021+0000 7fc7e0f0e640 10 client.114219.objecter successfully canceled tick 2024-09-17T18:37:31.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:31.021+0000 7fc7e0f0e640 1 -- 172.21.15.121:0/1223180838 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc7bc0401a0 msgr2=0x7fc7bc042660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:31.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:31.021+0000 7fc7e0f0e640 1 --2- 172.21.15.121:0/1223180838 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc7bc0401a0 0x7fc7bc042660 secure :-1 s=READY pgs=4112 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc0a2060 tx=0x7fc7c4008040 comp rx=0 tx=0).stop 2024-09-17T18:37:31.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:31.021+0000 7fc7e0f0e640 10 monclient: shutdown 2024-09-17T18:37:31.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:31.021+0000 7fc7e0f0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:31.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:31.021+0000 7fc7e0f0e640 1 -- 172.21.15.121:0/1223180838 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7dc0a33f0 msgr2=0x7fc7dc0a2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:31.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:31.021+0000 7fc7e0f0e640 1 --2- 172.21.15.121:0/1223180838 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7dc0a33f0 0x7fc7dc0a2310 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc00c430 tx=0x7fc7cc00c8f0 comp rx=0 tx=0).stop 2024-09-17T18:37:31.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:31.021+0000 7fc7e0f0e640 1 -- 172.21.15.121:0/1223180838 shutdown_connections 2024-09-17T18:37:31.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:31.021+0000 7fc7e0f0e640 1 --2- 172.21.15.121:0/1223180838 >> v2:172.21.15.173:6812/2423786037 conn(0x7fc7bc0401a0 0x7fc7bc042660 unknown :-1 s=CLOSED pgs=4112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:31.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:31.021+0000 7fc7e0f0e640 1 --2- 172.21.15.121:0/1223180838 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7dc13b860 0x7fc7dc136df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:31.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:31.021+0000 7fc7e0f0e640 1 --2- 172.21.15.121:0/1223180838 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7dc137330 0x7fc7dc072b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:31.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:31.021+0000 7fc7e0f0e640 1 --2- 172.21.15.121:0/1223180838 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7dc0a33f0 0x7fc7dc0a2310 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:31.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:31.021+0000 7fc7e0f0e640 1 -- 172.21.15.121:0/1223180838 >> 172.21.15.121:0/1223180838 conn(0x7fc7dc0a3b50 msgr2=0x7fc7dc0a4f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:31.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:31.021+0000 7fc7e0f0e640 1 -- 172.21.15.121:0/1223180838 shutdown_connections 2024-09-17T18:37:31.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:31.021+0000 7fc7e0f0e640 1 -- 172.21.15.121:0/1223180838 wait complete. 2024-09-17T18:37:31.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:31.021+0000 7fc7e0f0e640 1 librados: shutdown 2024-09-17T18:37:31.033 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":1229,"stamp":"2024-09-17T18:37:29.779573+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":459824,"kb_used_data":39448,"kb_used_omap":9,"kb_used_meta":420214,"kb_avail":565771216,"statfs":{"total":579820584960,"available":579349725184,"internally_reserved":0,"allocated":40394752,"data_stored":18330092,"data_compressed":59128,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":430299809},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003413"},"pg_stats":[{"pgid":"5.15","version":"0'0","reported_seq":1279,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.583221+0000","last_change":"2024-09-17T18:20:02.310686+0000","last_active":"2024-09-17T18:36:25.583221+0000","last_peered":"2024-09-17T18:36:25.583221+0000","last_clean":"2024-09-17T18:36:25.583221+0000","last_became_active":"2024-09-17T18:20:02.310239+0000","last_became_peered":"2024-09-17T18:20:02.310239+0000","last_unstale":"2024-09-17T18:36:25.583221+0000","last_undegraded":"2024-09-17T18:36:25.583221+0000","last_fullsized":"2024-09-17T18:36:25.583221+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:52:11.931160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1298,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.717134+0000","last_change":"2024-09-17T18:20:02.326174+0000","last_active":"2024-09-17T18:36:25.717134+0000","last_peered":"2024-09-17T18:36:25.717134+0000","last_clean":"2024-09-17T18:36:25.717134+0000","last_became_active":"2024-09-17T18:20:02.322561+0000","last_became_peered":"2024-09-17T18:20:02.322561+0000","last_unstale":"2024-09-17T18:36:25.717134+0000","last_undegraded":"2024-09-17T18:36:25.717134+0000","last_fullsized":"2024-09-17T18:36:25.717134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:29:09.470382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1296,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.735731+0000","last_change":"2024-09-17T18:20:02.250996+0000","last_active":"2024-09-17T18:36:26.735731+0000","last_peered":"2024-09-17T18:36:26.735731+0000","last_clean":"2024-09-17T18:36:26.735731+0000","last_became_active":"2024-09-17T18:20:02.250675+0000","last_became_peered":"2024-09-17T18:20:02.250675+0000","last_unstale":"2024-09-17T18:36:26.735731+0000","last_undegraded":"2024-09-17T18:36:26.735731+0000","last_fullsized":"2024-09-17T18:36:26.735731+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:17:16.240715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1295,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.433900+0000","last_change":"2024-09-17T18:20:02.342701+0000","last_active":"2024-09-17T18:36:25.433900+0000","last_peered":"2024-09-17T18:36:25.433900+0000","last_clean":"2024-09-17T18:36:25.433900+0000","last_became_active":"2024-09-17T18:20:02.342457+0000","last_became_peered":"2024-09-17T18:20:02.342457+0000","last_unstale":"2024-09-17T18:36:25.433900+0000","last_undegraded":"2024-09-17T18:36:25.433900+0000","last_fullsized":"2024-09-17T18:36:25.433900+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:57:23.288202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1280,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.583101+0000","last_change":"2024-09-17T18:20:02.251851+0000","last_active":"2024-09-17T18:36:25.583101+0000","last_peered":"2024-09-17T18:36:25.583101+0000","last_clean":"2024-09-17T18:36:25.583101+0000","last_became_active":"2024-09-17T18:20:02.249100+0000","last_became_peered":"2024-09-17T18:20:02.249100+0000","last_unstale":"2024-09-17T18:36:25.583101+0000","last_undegraded":"2024-09-17T18:36:25.583101+0000","last_fullsized":"2024-09-17T18:36:25.583101+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:00:45.271045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1297,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.735456+0000","last_change":"2024-09-17T18:20:02.173218+0000","last_active":"2024-09-17T18:36:26.735456+0000","last_peered":"2024-09-17T18:36:26.735456+0000","last_clean":"2024-09-17T18:36:26.735456+0000","last_became_active":"2024-09-17T18:20:02.172083+0000","last_became_peered":"2024-09-17T18:20:02.172083+0000","last_unstale":"2024-09-17T18:36:26.735456+0000","last_undegraded":"2024-09-17T18:36:26.735456+0000","last_fullsized":"2024-09-17T18:36:26.735456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T21:54:16.156188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1296,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.735377+0000","last_change":"2024-09-17T18:20:02.244905+0000","last_active":"2024-09-17T18:36:26.735377+0000","last_peered":"2024-09-17T18:36:26.735377+0000","last_clean":"2024-09-17T18:36:26.735377+0000","last_became_active":"2024-09-17T18:20:02.244673+0000","last_became_peered":"2024-09-17T18:20:02.244673+0000","last_unstale":"2024-09-17T18:36:26.735377+0000","last_undegraded":"2024-09-17T18:36:26.735377+0000","last_fullsized":"2024-09-17T18:36:26.735377+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T21:29:37.865312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1297,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.735570+0000","last_change":"2024-09-17T18:20:02.316100+0000","last_active":"2024-09-17T18:36:26.735570+0000","last_peered":"2024-09-17T18:36:26.735570+0000","last_clean":"2024-09-17T18:36:26.735570+0000","last_became_active":"2024-09-17T18:20:02.315070+0000","last_became_peered":"2024-09-17T18:20:02.315070+0000","last_unstale":"2024-09-17T18:36:26.735570+0000","last_undegraded":"2024-09-17T18:36:26.735570+0000","last_fullsized":"2024-09-17T18:36:26.735570+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:45:14.819013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1296,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.735378+0000","last_change":"2024-09-17T18:20:02.348899+0000","last_active":"2024-09-17T18:36:26.735378+0000","last_peered":"2024-09-17T18:36:26.735378+0000","last_clean":"2024-09-17T18:36:26.735378+0000","last_became_active":"2024-09-17T18:20:02.348022+0000","last_became_peered":"2024-09-17T18:20:02.348022+0000","last_unstale":"2024-09-17T18:36:26.735378+0000","last_undegraded":"2024-09-17T18:36:26.735378+0000","last_fullsized":"2024-09-17T18:36:26.735378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:02:14.991712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1282,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:27.012026+0000","last_change":"2024-09-17T18:20:02.331615+0000","last_active":"2024-09-17T18:36:27.012026+0000","last_peered":"2024-09-17T18:36:27.012026+0000","last_clean":"2024-09-17T18:36:27.012026+0000","last_became_active":"2024-09-17T18:20:02.328118+0000","last_became_peered":"2024-09-17T18:20:02.328118+0000","last_unstale":"2024-09-17T18:36:27.012026+0000","last_undegraded":"2024-09-17T18:36:27.012026+0000","last_fullsized":"2024-09-17T18:36:27.012026+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:39:54.243953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1279,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.583300+0000","last_change":"2024-09-17T18:20:02.248888+0000","last_active":"2024-09-17T18:36:25.583300+0000","last_peered":"2024-09-17T18:36:25.583300+0000","last_clean":"2024-09-17T18:36:25.583300+0000","last_became_active":"2024-09-17T18:20:02.242391+0000","last_became_peered":"2024-09-17T18:20:02.242391+0000","last_unstale":"2024-09-17T18:36:25.583300+0000","last_undegraded":"2024-09-17T18:36:25.583300+0000","last_fullsized":"2024-09-17T18:36:25.583300+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:02:05.721872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1294,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.433853+0000","last_change":"2024-09-17T18:20:02.249495+0000","last_active":"2024-09-17T18:36:25.433853+0000","last_peered":"2024-09-17T18:36:25.433853+0000","last_clean":"2024-09-17T18:36:25.433853+0000","last_became_active":"2024-09-17T18:20:02.248819+0000","last_became_peered":"2024-09-17T18:20:02.248819+0000","last_unstale":"2024-09-17T18:36:25.433853+0000","last_undegraded":"2024-09-17T18:36:25.433853+0000","last_fullsized":"2024-09-17T18:36:25.433853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T20:11:51.090905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1276,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.898945+0000","last_change":"2024-09-17T18:20:02.273998+0000","last_active":"2024-09-17T18:36:26.898945+0000","last_peered":"2024-09-17T18:36:26.898945+0000","last_clean":"2024-09-17T18:36:26.898945+0000","last_became_active":"2024-09-17T18:20:02.259327+0000","last_became_peered":"2024-09-17T18:20:02.259327+0000","last_unstale":"2024-09-17T18:36:26.898945+0000","last_undegraded":"2024-09-17T18:36:26.898945+0000","last_fullsized":"2024-09-17T18:36:26.898945+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:55:51.961001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":1397,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.433765+0000","last_change":"2024-09-17T18:19:00.245854+0000","last_active":"2024-09-17T18:36:25.433765+0000","last_peered":"2024-09-17T18:36:25.433765+0000","last_clean":"2024-09-17T18:36:25.433765+0000","last_became_active":"2024-09-17T18:19:00.244903+0000","last_became_peered":"2024-09-17T18:19:00.244903+0000","last_unstale":"2024-09-17T18:36:25.433765+0000","last_undegraded":"2024-09-17T18:36:25.433765+0000","last_fullsized":"2024-09-17T18:36:25.433765+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:55:38.336150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":1296,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.583163+0000","last_change":"2024-09-17T18:19:01.532053+0000","last_active":"2024-09-17T18:36:25.583163+0000","last_peered":"2024-09-17T18:36:25.583163+0000","last_clean":"2024-09-17T18:36:25.583163+0000","last_became_active":"2024-09-17T18:18:59.526797+0000","last_became_peered":"2024-09-17T18:18:59.526797+0000","last_unstale":"2024-09-17T18:36:25.583163+0000","last_undegraded":"2024-09-17T18:36:25.583163+0000","last_fullsized":"2024-09-17T18:36:25.583163+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:19:28.196155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000425759,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1278,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.583305+0000","last_change":"2024-09-17T18:20:02.311785+0000","last_active":"2024-09-17T18:36:25.583305+0000","last_peered":"2024-09-17T18:36:25.583305+0000","last_clean":"2024-09-17T18:36:25.583305+0000","last_became_active":"2024-09-17T18:20:02.310633+0000","last_became_peered":"2024-09-17T18:20:02.310633+0000","last_unstale":"2024-09-17T18:36:25.583305+0000","last_undegraded":"2024-09-17T18:36:25.583305+0000","last_fullsized":"2024-09-17T18:36:25.583305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:08:06.293083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1312,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.433657+0000","last_change":"2024-09-17T18:19:01.527425+0000","last_active":"2024-09-17T18:36:25.433657+0000","last_peered":"2024-09-17T18:36:25.433657+0000","last_clean":"2024-09-17T18:36:25.433657+0000","last_became_active":"2024-09-17T18:18:59.526855+0000","last_became_peered":"2024-09-17T18:18:59.526855+0000","last_unstale":"2024-09-17T18:36:25.433657+0000","last_undegraded":"2024-09-17T18:36:25.433657+0000","last_fullsized":"2024-09-17T18:36:25.433657+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:05:38.754621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032532500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1297,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.735460+0000","last_change":"2024-09-17T18:20:02.260633+0000","last_active":"2024-09-17T18:36:26.735460+0000","last_peered":"2024-09-17T18:36:26.735460+0000","last_clean":"2024-09-17T18:36:26.735460+0000","last_became_active":"2024-09-17T18:20:02.260389+0000","last_became_peered":"2024-09-17T18:20:02.260389+0000","last_unstale":"2024-09-17T18:36:26.735460+0000","last_undegraded":"2024-09-17T18:36:26.735460+0000","last_fullsized":"2024-09-17T18:36:26.735460+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:20:12.888792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1312,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.735449+0000","last_change":"2024-09-17T18:19:01.527848+0000","last_active":"2024-09-17T18:36:26.735449+0000","last_peered":"2024-09-17T18:36:26.735449+0000","last_clean":"2024-09-17T18:36:26.735449+0000","last_became_active":"2024-09-17T18:18:59.529390+0000","last_became_peered":"2024-09-17T18:18:59.529390+0000","last_unstale":"2024-09-17T18:36:26.735449+0000","last_undegraded":"2024-09-17T18:36:26.735449+0000","last_fullsized":"2024-09-17T18:36:26.735449+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:40:45.376247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065986199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1299,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.717264+0000","last_change":"2024-09-17T18:20:02.260849+0000","last_active":"2024-09-17T18:36:25.717264+0000","last_peered":"2024-09-17T18:36:25.717264+0000","last_clean":"2024-09-17T18:36:25.717264+0000","last_became_active":"2024-09-17T18:20:02.251405+0000","last_became_peered":"2024-09-17T18:20:02.251405+0000","last_unstale":"2024-09-17T18:36:25.717264+0000","last_undegraded":"2024-09-17T18:36:25.717264+0000","last_fullsized":"2024-09-17T18:36:25.717264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:03:03.725725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":1297,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.583147+0000","last_change":"2024-09-17T18:19:01.532623+0000","last_active":"2024-09-17T18:36:25.583147+0000","last_peered":"2024-09-17T18:36:25.583147+0000","last_clean":"2024-09-17T18:36:25.583147+0000","last_became_active":"2024-09-17T18:18:59.526791+0000","last_became_peered":"2024-09-17T18:18:59.526791+0000","last_unstale":"2024-09-17T18:36:25.583147+0000","last_undegraded":"2024-09-17T18:36:25.583147+0000","last_fullsized":"2024-09-17T18:36:25.583147+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T21:46:24.217486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010088289999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1299,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.717202+0000","last_change":"2024-09-17T18:20:02.274215+0000","last_active":"2024-09-17T18:36:25.717202+0000","last_peered":"2024-09-17T18:36:25.717202+0000","last_clean":"2024-09-17T18:36:25.717202+0000","last_became_active":"2024-09-17T18:20:02.268644+0000","last_became_peered":"2024-09-17T18:20:02.268644+0000","last_unstale":"2024-09-17T18:36:25.717202+0000","last_undegraded":"2024-09-17T18:36:25.717202+0000","last_fullsized":"2024-09-17T18:36:25.717202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:12:55.005158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1298,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:27.012101+0000","last_change":"2024-09-17T18:19:01.527784+0000","last_active":"2024-09-17T18:36:27.012101+0000","last_peered":"2024-09-17T18:36:27.012101+0000","last_clean":"2024-09-17T18:36:27.012101+0000","last_became_active":"2024-09-17T18:19:00.244613+0000","last_became_peered":"2024-09-17T18:19:00.244613+0000","last_unstale":"2024-09-17T18:36:27.012101+0000","last_undegraded":"2024-09-17T18:36:27.012101+0000","last_fullsized":"2024-09-17T18:36:27.012101+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:31:24.599660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063646500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1280,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.583127+0000","last_change":"2024-09-17T18:20:02.254779+0000","last_active":"2024-09-17T18:36:25.583127+0000","last_peered":"2024-09-17T18:36:25.583127+0000","last_clean":"2024-09-17T18:36:25.583127+0000","last_became_active":"2024-09-17T18:20:02.254259+0000","last_became_peered":"2024-09-17T18:20:02.254259+0000","last_unstale":"2024-09-17T18:36:25.583127+0000","last_undegraded":"2024-09-17T18:36:25.583127+0000","last_fullsized":"2024-09-17T18:36:25.583127+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:02:19.820701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1311,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.433670+0000","last_change":"2024-09-17T18:19:01.527563+0000","last_active":"2024-09-17T18:36:25.433670+0000","last_peered":"2024-09-17T18:36:25.433670+0000","last_clean":"2024-09-17T18:36:25.433670+0000","last_became_active":"2024-09-17T18:19:00.244851+0000","last_became_peered":"2024-09-17T18:19:00.244851+0000","last_unstale":"2024-09-17T18:36:25.433670+0000","last_undegraded":"2024-09-17T18:36:25.433670+0000","last_fullsized":"2024-09-17T18:36:25.433670+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:19:57.036107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041282100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"19'7","reported_seq":1309,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.433672+0000","last_change":"2024-09-17T18:20:02.253575+0000","last_active":"2024-09-17T18:36:25.433672+0000","last_peered":"2024-09-17T18:36:25.433672+0000","last_clean":"2024-09-17T18:36:25.433672+0000","last_became_active":"2024-09-17T18:20:02.253328+0000","last_became_peered":"2024-09-17T18:20:02.253328+0000","last_unstale":"2024-09-17T18:36:25.433672+0000","last_undegraded":"2024-09-17T18:36:25.433672+0000","last_fullsized":"2024-09-17T18:36:25.433672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:35:00.211682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1298,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:27.012077+0000","last_change":"2024-09-17T18:19:01.527858+0000","last_active":"2024-09-17T18:36:27.012077+0000","last_peered":"2024-09-17T18:36:27.012077+0000","last_clean":"2024-09-17T18:36:27.012077+0000","last_became_active":"2024-09-17T18:18:59.525295+0000","last_became_peered":"2024-09-17T18:18:59.525295+0000","last_unstale":"2024-09-17T18:36:27.012077+0000","last_undegraded":"2024-09-17T18:36:27.012077+0000","last_fullsized":"2024-09-17T18:36:27.012077+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:41:21.181263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062121300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1297,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.735549+0000","last_change":"2024-09-17T18:20:02.234786+0000","last_active":"2024-09-17T18:36:26.735549+0000","last_peered":"2024-09-17T18:36:26.735549+0000","last_clean":"2024-09-17T18:36:26.735549+0000","last_became_active":"2024-09-17T18:20:02.234414+0000","last_became_peered":"2024-09-17T18:20:02.234414+0000","last_unstale":"2024-09-17T18:36:26.735549+0000","last_undegraded":"2024-09-17T18:36:26.735549+0000","last_fullsized":"2024-09-17T18:36:26.735549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:07:37.564071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1311,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.433654+0000","last_change":"2024-09-17T18:19:01.527658+0000","last_active":"2024-09-17T18:36:25.433654+0000","last_peered":"2024-09-17T18:36:25.433654+0000","last_clean":"2024-09-17T18:36:25.433654+0000","last_became_active":"2024-09-17T18:19:00.056752+0000","last_became_peered":"2024-09-17T18:19:00.056752+0000","last_unstale":"2024-09-17T18:36:25.433654+0000","last_undegraded":"2024-09-17T18:36:25.433654+0000","last_fullsized":"2024-09-17T18:36:25.433654+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:18:58.512908+0000","last_clean_scrub_stamp":"2024-09-17T18:18:58.512908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:57:47.271964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049856399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1282,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:27.011781+0000","last_change":"2024-09-17T18:20:02.227335+0000","last_active":"2024-09-17T18:36:27.011781+0000","last_peered":"2024-09-17T18:36:27.011781+0000","last_clean":"2024-09-17T18:36:27.011781+0000","last_became_active":"2024-09-17T18:20:02.226503+0000","last_became_peered":"2024-09-17T18:20:02.226503+0000","last_unstale":"2024-09-17T18:36:27.011781+0000","last_undegraded":"2024-09-17T18:36:27.011781+0000","last_fullsized":"2024-09-17T18:36:27.011781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T03:43:28.520222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1282,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:27.011779+0000","last_change":"2024-09-17T18:20:02.313761+0000","last_active":"2024-09-17T18:36:27.011779+0000","last_peered":"2024-09-17T18:36:27.011779+0000","last_clean":"2024-09-17T18:36:27.011779+0000","last_became_active":"2024-09-17T18:20:02.311023+0000","last_became_peered":"2024-09-17T18:20:02.311023+0000","last_unstale":"2024-09-17T18:36:27.011779+0000","last_undegraded":"2024-09-17T18:36:27.011779+0000","last_fullsized":"2024-09-17T18:36:27.011779+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:18:02.124093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1275,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.898650+0000","last_change":"2024-09-17T18:20:02.351223+0000","last_active":"2024-09-17T18:36:26.898650+0000","last_peered":"2024-09-17T18:36:26.898650+0000","last_clean":"2024-09-17T18:36:26.898650+0000","last_became_active":"2024-09-17T18:20:02.339489+0000","last_became_peered":"2024-09-17T18:20:02.339489+0000","last_unstale":"2024-09-17T18:36:26.898650+0000","last_undegraded":"2024-09-17T18:36:26.898650+0000","last_fullsized":"2024-09-17T18:36:26.898650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:23:22.992560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1295,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.433542+0000","last_change":"2024-09-17T18:20:02.287570+0000","last_active":"2024-09-17T18:36:25.433542+0000","last_peered":"2024-09-17T18:36:25.433542+0000","last_clean":"2024-09-17T18:36:25.433542+0000","last_became_active":"2024-09-17T18:20:02.286560+0000","last_became_peered":"2024-09-17T18:20:02.286560+0000","last_unstale":"2024-09-17T18:36:25.433542+0000","last_undegraded":"2024-09-17T18:36:25.433542+0000","last_fullsized":"2024-09-17T18:36:25.433542+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:54:48.894903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1299,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.717249+0000","last_change":"2024-09-17T18:20:02.280897+0000","last_active":"2024-09-17T18:36:25.717249+0000","last_peered":"2024-09-17T18:36:25.717249+0000","last_clean":"2024-09-17T18:36:25.717249+0000","last_became_active":"2024-09-17T18:20:02.270521+0000","last_became_peered":"2024-09-17T18:20:02.270521+0000","last_unstale":"2024-09-17T18:36:25.717249+0000","last_undegraded":"2024-09-17T18:36:25.717249+0000","last_fullsized":"2024-09-17T18:36:25.717249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:40:16.489769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1296,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.433793+0000","last_change":"2024-09-17T18:20:02.294208+0000","last_active":"2024-09-17T18:36:25.433793+0000","last_peered":"2024-09-17T18:36:25.433793+0000","last_clean":"2024-09-17T18:36:25.433793+0000","last_became_active":"2024-09-17T18:20:02.293587+0000","last_became_peered":"2024-09-17T18:20:02.293587+0000","last_unstale":"2024-09-17T18:36:25.433793+0000","last_undegraded":"2024-09-17T18:36:25.433793+0000","last_fullsized":"2024-09-17T18:36:25.433793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:06:11.621162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1274,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.898883+0000","last_change":"2024-09-17T18:20:02.275664+0000","last_active":"2024-09-17T18:36:26.898883+0000","last_peered":"2024-09-17T18:36:26.898883+0000","last_clean":"2024-09-17T18:36:26.898883+0000","last_became_active":"2024-09-17T18:20:02.259616+0000","last_became_peered":"2024-09-17T18:20:02.259616+0000","last_unstale":"2024-09-17T18:36:26.898883+0000","last_undegraded":"2024-09-17T18:36:26.898883+0000","last_fullsized":"2024-09-17T18:36:26.898883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T21:08:28.298068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1281,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:27.012129+0000","last_change":"2024-09-17T18:20:02.314014+0000","last_active":"2024-09-17T18:36:27.012129+0000","last_peered":"2024-09-17T18:36:27.012129+0000","last_clean":"2024-09-17T18:36:27.012129+0000","last_became_active":"2024-09-17T18:20:02.313525+0000","last_became_peered":"2024-09-17T18:20:02.313525+0000","last_unstale":"2024-09-17T18:36:27.012129+0000","last_undegraded":"2024-09-17T18:36:27.012129+0000","last_fullsized":"2024-09-17T18:36:27.012129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:02:30.324064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1274,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.898897+0000","last_change":"2024-09-17T18:20:02.306541+0000","last_active":"2024-09-17T18:36:26.898897+0000","last_peered":"2024-09-17T18:36:26.898897+0000","last_clean":"2024-09-17T18:36:26.898897+0000","last_became_active":"2024-09-17T18:20:02.292188+0000","last_became_peered":"2024-09-17T18:20:02.292188+0000","last_unstale":"2024-09-17T18:36:26.898897+0000","last_undegraded":"2024-09-17T18:36:26.898897+0000","last_fullsized":"2024-09-17T18:36:26.898897+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:10:44.904234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1278,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:25.583267+0000","last_change":"2024-09-17T18:20:02.256567+0000","last_active":"2024-09-17T18:36:25.583267+0000","last_peered":"2024-09-17T18:36:25.583267+0000","last_clean":"2024-09-17T18:36:25.583267+0000","last_became_active":"2024-09-17T18:20:02.254813+0000","last_became_peered":"2024-09-17T18:20:02.254813+0000","last_unstale":"2024-09-17T18:36:25.583267+0000","last_undegraded":"2024-09-17T18:36:25.583267+0000","last_fullsized":"2024-09-17T18:36:25.583267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:20:59.122518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1275,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:26.898721+0000","last_change":"2024-09-17T18:20:02.327177+0000","last_active":"2024-09-17T18:36:26.898721+0000","last_peered":"2024-09-17T18:36:26.898721+0000","last_clean":"2024-09-17T18:36:26.898721+0000","last_became_active":"2024-09-17T18:20:02.299834+0000","last_became_peered":"2024-09-17T18:20:02.299834+0000","last_unstale":"2024-09-17T18:36:26.898721+0000","last_undegraded":"2024-09-17T18:36:26.898721+0000","last_fullsized":"2024-09-17T18:36:26.898721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:14:12.763217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1282,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:36:27.011862+0000","last_change":"2024-09-17T18:20:02.230651+0000","last_active":"2024-09-17T18:36:27.011862+0000","last_peered":"2024-09-17T18:36:27.011862+0000","last_clean":"2024-09-17T18:36:27.011862+0000","last_became_active":"2024-09-17T18:20:02.228502+0000","last_became_peered":"2024-09-17T18:20:02.228502+0000","last_unstale":"2024-09-17T18:36:27.011862+0000","last_undegraded":"2024-09-17T18:36:27.011862+0000","last_fullsized":"2024-09-17T18:36:27.011862+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:20:01.098792+0000","last_clean_scrub_stamp":"2024-09-17T18:20:01.098792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:36:19.108865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59128,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607779,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69680,"kb_used_data":6552,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94302160,"statfs":{"total":96636764160,"available":96565411840,"internally_reserved":0,"allocated":6709248,"data_stored":2878698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607778,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78008,"kb_used_data":6556,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94293832,"statfs":{"total":96636764160,"available":96556883968,"internally_reserved":0,"allocated":6713344,"data_stored":2878717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640480,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82168,"kb_used_data":6556,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289672,"statfs":{"total":96636764160,"available":96552624128,"internally_reserved":0,"allocated":6713344,"data_stored":2878717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640482,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82224,"kb_used_data":6616,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289616,"statfs":{"total":96636764160,"available":96552566784,"internally_reserved":0,"allocated":6774784,"data_stored":3407631,"data_compressed":29564,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640481,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73836,"kb_used_data":6552,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94298004,"statfs":{"total":96636764160,"available":96561156096,"internally_reserved":0,"allocated":6709248,"data_stored":2878698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640483,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73908,"kb_used_data":6616,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94297932,"statfs":{"total":96636764160,"available":96561082368,"internally_reserved":0,"allocated":6774784,"data_stored":3407631,"data_compressed":29564,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29564,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29564,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T18:37:31.038 INFO:tasks.ceph:pgid 5.15 last_scrub_stamp 2024-09-17T18:20:01.098792+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=20, tm_sec=1, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=37, tm_sec=26, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T18:37:31.038 INFO:tasks.ceph:pgid 5.14 last_scrub_stamp 2024-09-17T18:20:01.098792+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=20, tm_sec=1, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=37, tm_sec=26, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T18:37:31.038 INFO:tasks.ceph:pgid 5.17 last_scrub_stamp 2024-09-17T18:20:01.098792+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=20, tm_sec=1, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=37, tm_sec=26, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T18:37:31.039 INFO:tasks.ceph:pgid 5.16 last_scrub_stamp 2024-09-17T18:20:01.098792+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=20, tm_sec=1, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=37, tm_sec=26, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T18:37:31.039 INFO:tasks.ceph:pgid 5.9 last_scrub_stamp 2024-09-17T18:20:01.098792+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=20, tm_sec=1, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=37, tm_sec=26, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T18:37:31.039 INFO:tasks.ceph:pgid 5.8 last_scrub_stamp 2024-09-17T18:20:01.098792+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=20, tm_sec=1, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=37, tm_sec=26, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T18:37:31.039 INFO:tasks.ceph:pgid 5.b last_scrub_stamp 2024-09-17T18:20:01.098792+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=20, tm_sec=1, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=37, tm_sec=26, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T18:37:31.039 INFO:tasks.ceph:pgid 5.a last_scrub_stamp 2024-09-17T18:20:01.098792+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=20, tm_sec=1, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=37, tm_sec=26, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T18:37:31.039 INFO:tasks.ceph:pgid 5.d last_scrub_stamp 2024-09-17T18:20:01.098792+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=20, tm_sec=1, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=37, tm_sec=26, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T18:37:31.039 INFO:tasks.ceph:pgid 5.c last_scrub_stamp 2024-09-17T18:20:01.098792+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=20, tm_sec=1, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=37, tm_sec=26, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T18:37:31.039 INFO:tasks.ceph:pgid 5.f last_scrub_stamp 2024-09-17T18:20:01.098792+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=20, tm_sec=1, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=37, tm_sec=26, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T18:37:31.040 INFO:tasks.ceph:pgid 5.e last_scrub_stamp 2024-09-17T18:20:01.098792+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=20, tm_sec=1, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=37, tm_sec=26, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T18:37:31.040 INFO:tasks.ceph:pgid 5.4 last_scrub_stamp 2024-09-17T18:20:01.098792+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=20, tm_sec=1, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=37, tm_sec=26, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T18:37:31.040 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-17T18:18:58.512908+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=18, tm_sec=58, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=37, tm_sec=26, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T18:37:31.040 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-17T18:18:58.512908+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=18, tm_sec=58, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=37, tm_sec=26, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T18:37:31.040 INFO:tasks.ceph:pgid 5.7 last_scrub_stamp 2024-09-17T18:20:01.098792+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=20, tm_sec=1, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=37, tm_sec=26, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T18:37:31.040 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-17T18:18:58.512908+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=18, tm_sec=58, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=37, tm_sec=26, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T18:37:31.040 INFO:tasks.ceph:pgid 5.6 last_scrub_stamp 2024-09-17T18:20:01.098792+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=20, tm_sec=1, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=37, tm_sec=26, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T18:37:31.040 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-17T18:18:58.512908+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=18, tm_sec=58, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=37, tm_sec=26, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T18:37:31.040 INFO:tasks.ceph:pgid 5.5 last_scrub_stamp 2024-09-17T18:20:01.098792+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=20, tm_sec=1, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=37, tm_sec=26, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T18:37:31.041 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-17T18:18:58.512908+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=18, tm_sec=58, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=37, tm_sec=26, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T18:37:31.041 INFO:tasks.ceph:pgid 5.3 last_scrub_stamp 2024-09-17T18:20:01.098792+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=20, tm_sec=1, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=37, tm_sec=26, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T18:37:31.041 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-17T18:18:58.512908+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=18, tm_sec=58, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=37, tm_sec=26, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T18:37:31.041 INFO:tasks.ceph:pgid 5.2 last_scrub_stamp 2024-09-17T18:20:01.098792+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=20, tm_sec=1, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=37, tm_sec=26, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T18:37:31.041 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-17T18:18:58.512908+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=18, tm_sec=58, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=37, tm_sec=26, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T18:37:31.041 INFO:tasks.ceph:pgid 5.1 last_scrub_stamp 2024-09-17T18:20:01.098792+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=20, tm_sec=1, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=37, tm_sec=26, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T18:37:31.041 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-17T18:18:58.512908+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=18, tm_sec=58, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=37, tm_sec=26, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T18:37:31.041 INFO:tasks.ceph:pgid 5.0 last_scrub_stamp 2024-09-17T18:20:01.098792+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=20, tm_sec=1, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=37, tm_sec=26, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T18:37:31.041 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-17T18:18:58.512908+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=18, tm_sec=58, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=37, tm_sec=26, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T18:37:31.042 INFO:tasks.ceph:pgid 5.12 last_scrub_stamp 2024-09-17T18:20:01.098792+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=20, tm_sec=1, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=37, tm_sec=26, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T18:37:31.042 INFO:tasks.ceph:pgid 5.13 last_scrub_stamp 2024-09-17T18:20:01.098792+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=20, tm_sec=1, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=37, tm_sec=26, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T18:37:31.042 INFO:tasks.ceph:pgid 5.10 last_scrub_stamp 2024-09-17T18:20:01.098792+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=20, tm_sec=1, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=37, tm_sec=26, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T18:37:31.042 INFO:tasks.ceph:pgid 5.11 last_scrub_stamp 2024-09-17T18:20:01.098792+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=20, tm_sec=1, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=37, tm_sec=26, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T18:37:31.042 INFO:tasks.ceph:pgid 5.1e last_scrub_stamp 2024-09-17T18:20:01.098792+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=20, tm_sec=1, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=37, tm_sec=26, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T18:37:31.042 INFO:tasks.ceph:pgid 5.1f last_scrub_stamp 2024-09-17T18:20:01.098792+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=20, tm_sec=1, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=37, tm_sec=26, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T18:37:31.042 INFO:tasks.ceph:pgid 5.1c last_scrub_stamp 2024-09-17T18:20:01.098792+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=20, tm_sec=1, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=37, tm_sec=26, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T18:37:31.042 INFO:tasks.ceph:pgid 5.1d last_scrub_stamp 2024-09-17T18:20:01.098792+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=20, tm_sec=1, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=37, tm_sec=26, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T18:37:31.043 INFO:tasks.ceph:pgid 5.1a last_scrub_stamp 2024-09-17T18:20:01.098792+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=20, tm_sec=1, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=37, tm_sec=26, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T18:37:31.043 INFO:tasks.ceph:pgid 5.1b last_scrub_stamp 2024-09-17T18:20:01.098792+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=20, tm_sec=1, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=37, tm_sec=26, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T18:37:31.043 INFO:tasks.ceph:pgid 5.18 last_scrub_stamp 2024-09-17T18:20:01.098792+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=20, tm_sec=1, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=37, tm_sec=26, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T18:37:31.043 INFO:tasks.ceph:pgid 5.19 last_scrub_stamp 2024-09-17T18:20:01.098792+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=20, tm_sec=1, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=18, tm_min=37, tm_sec=26, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T18:37:31.043 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-17T18:37:51.044 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T18:37:51.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.151+0000 7f4a4af6b640 10 monclient: get_monmap_and_config 2024-09-17T18:37:51.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.151+0000 7f4a4af6b640 10 monclient: build_initial_monmap 2024-09-17T18:37:51.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.151+0000 7f4a4af6b640 10 monclient: monmap: 2024-09-17T18:37:51.153 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:51.153 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:51.154 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:37:51.152377+0000 2024-09-17T18:37:51.154 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:37:51.152377+0000 2024-09-17T18:37:51.154 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:51.154 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:51.154 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:37:51.154 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:37:51.154 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:37:51.154 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:37:51.154 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:37:51.154 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:51.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.152+0000 7f4a4af6b640 1 Processor -- start 2024-09-17T18:37:51.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.152+0000 7f4a4af6b640 1 -- start start 2024-09-17T18:37:51.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.152+0000 7f4a4af6b640 10 monclient: init 2024-09-17T18:37:51.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.153+0000 7f4a4af6b640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:51.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.153+0000 7f4a4af6b640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-17T18:37:51.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.153+0000 7f4a4af6b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4a440738a0 addr v1:172.21.15.121:6789/0 2024-09-17T18:37:51.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.153+0000 7f4a4af6b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4a44138a40 addr v1:172.21.15.173:6789/0 2024-09-17T18:37:51.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.153+0000 7f4a4af6b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a4413eff0 0x7f4a441413e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:51.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.153+0000 7f4a4af6b640 10 monclient(hunting): picked mon.noname-b con 0x7f4a4413eff0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:51.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.153+0000 7f4a4af6b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a4406fd90 con 0x7f4a440738a0 2024-09-17T18:37:51.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.153+0000 7f4a4af6b640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a44070070 con 0x7f4a44138a40 2024-09-17T18:37:51.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.153+0000 7f4a4af6b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:51.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.153+0000 7f4a4af6b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4a44070350 con 0x7f4a4413eff0 2024-09-17T18:37:51.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.153+0000 7f4a4af6b640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:51.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.153+0000 7f4a4af6b640 10 monclient(hunting): authenticate will time out at 1874.869115s 2024-09-17T18:37:51.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.153+0000 7f4a49508640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a4413eff0 0x7f4a441413e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:51.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.153+0000 7f4a48d07640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f4a440738a0 0x7f4a44073c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:42864/0 (socket says 172.21.15.121:42864) 2024-09-17T18:37:51.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.153+0000 7f4a48d07640 1 -- 172.21.15.121:0/3752076735 learned_addr learned my addr 172.21.15.121:0/3752076735 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:37:51.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.153+0000 7f4a49508640 10 monclient(hunting): get_auth_request con 0x7f4a4413eff0 auth_method 0 2024-09-17T18:37:51.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.153+0000 7f4a49508640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:51.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.153+0000 7f4a49508640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:51.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.153+0000 7f4a49508640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:51.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.153+0000 7f4a49508640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:51.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.153+0000 7f4a49508640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:51.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.153+0000 7f4a49508640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:51.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.153+0000 7f4a49508640 10 monclient(hunting): handle_auth_done global_id 64676 payload 274 2024-09-17T18:37:51.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.153+0000 7f4a49508640 1 -- 172.21.15.121:0/3752076735 >> v1:172.21.15.173:6789/0 conn(0x7f4a44138a40 legacy=0x7f4a4413d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:51.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.153+0000 7f4a49508640 1 -- 172.21.15.121:0/3752076735 >> v1:172.21.15.121:6789/0 conn(0x7f4a440738a0 legacy=0x7f4a44073c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:51.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.153+0000 7f4a49508640 10 monclient: _finish_hunting 0 2024-09-17T18:37:51.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.153+0000 7f4a49508640 1 monclient: found mon.noname-b 2024-09-17T18:37:51.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.153+0000 7f4a49508640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:51.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.153+0000 7f4a49508640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.121:3301/0 2024-09-17T18:37:51.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.153+0000 7f4a49508640 1 -- 172.21.15.121:0/3752076735 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a440a4020 con 0x7f4a4413eff0 2024-09-17T18:37:51.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.156+0000 7f4a49508640 1 -- 172.21.15.121:0/3752076735 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a4413eff0 msgr2=0x7f4a441413e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:51.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.156+0000 7f4a49508640 1 --2- 172.21.15.121:0/3752076735 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a4413eff0 0x7f4a441413e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c009df0 tx=0x7f4a3c02a4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=892383f850ce65e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:51.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.156+0000 7f4a437fe640 1 -- 172.21.15.121:0/3752076735 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c007a30 con 0x7f4a4413eff0 2024-09-17T18:37:51.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.156+0000 7f4a437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:51.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.156+0000 7f4a437fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T18:37:51.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.156+0000 7f4a437fe640 10 monclient: dump: 2024-09-17T18:37:51.165 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:51.165 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:51.165 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:51.165 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:51.165 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:51.165 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:51.165 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:51.165 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:51.166 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:51.166 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:51.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.156+0000 7f4a437fe640 10 monclient: _finish_auth 0 2024-09-17T18:37:51.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.156+0000 7f4a437fe640 10 monclient: _check_auth_tickets 2024-09-17T18:37:51.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.156+0000 7f4a437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:51.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.156+0000 7f4a437fe640 1 -- 172.21.15.121:0/3752076735 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a3c007bd0 con 0x7f4a4413eff0 2024-09-17T18:37:51.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.156+0000 7f4a4af6b640 5 monclient: authenticate success, global_id 64676 2024-09-17T18:37:51.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.156+0000 7f4a4af6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:37:51.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.156+0000 7f4a437fe640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:51.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.156+0000 7f4a4af6b640 10 monclient: get_monmap_and_config success 2024-09-17T18:37:51.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.156+0000 7f4a437fe640 1 -- 172.21.15.121:0/3752076735 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c031400 con 0x7f4a4413eff0 2024-09-17T18:37:51.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.156+0000 7f4a4af6b640 4 set_mon_vals no callback set 2024-09-17T18:37:51.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.156+0000 7f4a437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:51.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.156+0000 7f4a437fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:37:51.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.156+0000 7f4a437fe640 10 monclient: dump: 2024-09-17T18:37:51.167 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:51.167 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:51.167 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:51.167 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:51.167 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:51.167 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:51.167 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:51.167 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:51.167 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:51.167 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:51.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.157+0000 7f4a4af6b640 10 monclient: shutdown 2024-09-17T18:37:51.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.157+0000 7f4a4af6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:51.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.157+0000 7f4a4af6b640 1 -- 172.21.15.121:0/3752076735 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a4413eff0 msgr2=0x7f4a441413e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:51.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.157+0000 7f4a4af6b640 1 --2- 172.21.15.121:0/3752076735 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a4413eff0 0x7f4a441413e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c009df0 tx=0x7f4a3c02a4b0 comp rx=0 tx=0).stop 2024-09-17T18:37:51.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.157+0000 7f4a4af6b640 1 -- 172.21.15.121:0/3752076735 shutdown_connections 2024-09-17T18:37:51.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.157+0000 7f4a4af6b640 1 --2- 172.21.15.121:0/3752076735 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a4413eff0 0x7f4a441413e0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:51.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.157+0000 7f4a4af6b640 1 -- 172.21.15.121:0/3752076735 >> 172.21.15.121:0/3752076735 conn(0x7f4a44076f00 msgr2=0x7f4a44077310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:51.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.157+0000 7f4a4af6b640 1 -- 172.21.15.121:0/3752076735 shutdown_connections 2024-09-17T18:37:51.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.157+0000 7f4a4af6b640 1 -- 172.21.15.121:0/3752076735 wait complete. 2024-09-17T18:37:51.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.157+0000 7f4a4af6b640 10 monclient: build_initial_monmap 2024-09-17T18:37:51.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.157+0000 7f4a4af6b640 10 monclient: monmap: 2024-09-17T18:37:51.168 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:51.168 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:37:51.168 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:37:51.168 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:37:51.168 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:51.168 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:51.168 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:37:51.168 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:37:51.169 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:37:51.169 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:51.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.157+0000 7f4a4af6b640 1 librados: starting msgr at 2024-09-17T18:37:51.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.157+0000 7f4a4af6b640 1 librados: starting objecter 2024-09-17T18:37:51.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.158+0000 7f4a4af6b640 1 Processor -- start 2024-09-17T18:37:51.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.158+0000 7f4a4af6b640 1 -- start start 2024-09-17T18:37:51.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.158+0000 7f4a4af6b640 1 librados: setting wanted keys 2024-09-17T18:37:51.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.158+0000 7f4a4af6b640 1 librados: calling monclient init 2024-09-17T18:37:51.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.158+0000 7f4a4af6b640 10 monclient: init 2024-09-17T18:37:51.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.158+0000 7f4a4af6b640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:51.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.158+0000 7f4a4af6b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T18:37:51.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.158+0000 7f4a4af6b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a440738a0 0x7f4a441cdab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:51.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.158+0000 7f4a4af6b640 10 monclient(hunting): picked mon.noname-c con 0x7f4a440738a0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:51.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.158+0000 7f4a4af6b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a44138a40 0x7f4a4413d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:51.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.158+0000 7f4a4af6b640 10 monclient(hunting): picked mon.noname-b con 0x7f4a44138a40 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:37:51.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.158+0000 7f4a4af6b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a4413eff0 0x7f4a4413df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:51.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.158+0000 7f4a4af6b640 10 monclient(hunting): picked mon.noname-a con 0x7f4a4413eff0 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:37:51.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.158+0000 7f4a4af6b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:51.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.158+0000 7f4a4af6b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4a440a4370 con 0x7f4a4413eff0 2024-09-17T18:37:51.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.158+0000 7f4a4af6b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:51.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.158+0000 7f4a4af6b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4a4406d6a0 con 0x7f4a44138a40 2024-09-17T18:37:51.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.158+0000 7f4a4af6b640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:51.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.158+0000 7f4a4af6b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4a4413ecb0 con 0x7f4a440738a0 2024-09-17T18:37:51.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.158+0000 7f4a4af6b640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:51.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.158+0000 7f4a4af6b640 10 monclient(hunting): authenticate will time out at 1874.874756s 2024-09-17T18:37:51.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.158+0000 7f4a48d07640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a440738a0 0x7f4a441cdab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:51.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.158+0000 7f4a49508640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a4413eff0 0x7f4a4413df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:51.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.158+0000 7f4a49508640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a4413eff0 0x7f4a4413df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41812/0 (socket says 172.21.15.121:41812) 2024-09-17T18:37:51.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.158+0000 7f4a48d07640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a440738a0 0x7f4a441cdab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:34962/0 (socket says 172.21.15.121:34962) 2024-09-17T18:37:51.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.158+0000 7f4a49508640 1 -- 172.21.15.121:0/489072255 learned_addr learned my addr 172.21.15.121:0/489072255 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:37:51.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.158+0000 7f4a49508640 10 monclient(hunting): get_auth_request con 0x7f4a4413eff0 auth_method 0 2024-09-17T18:37:51.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.158+0000 7f4a49508640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:51.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.158+0000 7f4a49508640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:51.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.158+0000 7f4a49508640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:51.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.158+0000 7f4a48d07640 10 monclient(hunting): get_auth_request con 0x7f4a440738a0 auth_method 0 2024-09-17T18:37:51.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.158+0000 7f4a48d07640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:51.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.158+0000 7f4a48d07640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:51.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.158+0000 7f4a48d07640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:51.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.158+0000 7f4a49508640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:51.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.158+0000 7f4a49508640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:51.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.158+0000 7f4a49508640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:51.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.158+0000 7f4a48d07640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:51.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.158+0000 7f4a48d07640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:51.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.159+0000 7f4a48d07640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:51.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.159+0000 7f4a43fff640 1 --2- 172.21.15.121:0/489072255 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a44138a40 0x7f4a4413d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:51.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.159+0000 7f4a43fff640 10 monclient(hunting): get_auth_request con 0x7f4a44138a40 auth_method 0 2024-09-17T18:37:51.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.159+0000 7f4a43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:51.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.159+0000 7f4a43fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:51.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.159+0000 7f4a43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:51.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.159+0000 7f4a49508640 10 monclient(hunting): handle_auth_done global_id 114222 payload 824 2024-09-17T18:37:51.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.159+0000 7f4a49508640 1 -- 172.21.15.121:0/489072255 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a440738a0 msgr2=0x7f4a441cdab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:51.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.159+0000 7f4a49508640 1 --2- 172.21.15.121:0/489072255 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a440738a0 0x7f4a441cdab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:51.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.159+0000 7f4a49508640 1 -- 172.21.15.121:0/489072255 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a44138a40 msgr2=0x7f4a4413d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:51.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.159+0000 7f4a49508640 1 --2- 172.21.15.121:0/489072255 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a44138a40 0x7f4a4413d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:51.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.159+0000 7f4a49508640 10 monclient: _finish_hunting 0 2024-09-17T18:37:51.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.159+0000 7f4a49508640 1 monclient: found mon.noname-a 2024-09-17T18:37:51.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.159+0000 7f4a49508640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:51.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.159+0000 7f4a49508640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.121:3300/0 2024-09-17T18:37:51.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.159+0000 7f4a49508640 1 -- 172.21.15.121:0/489072255 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a4413e450 con 0x7f4a4413eff0 2024-09-17T18:37:51.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.159+0000 7f4a48d07640 1 --2- 172.21.15.121:0/489072255 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a440738a0 0x7f4a441cdab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:37:51.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.161+0000 7f4a49508640 1 -- 172.21.15.121:0/489072255 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a4413eff0 msgr2=0x7f4a4413df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:51.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.161+0000 7f4a49508640 1 --2- 172.21.15.121:0/489072255 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a4413eff0 0x7f4a4413df10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c009ee0 tx=0x7f4a3c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:51.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.161+0000 7f4a417fa640 10 client.?.objecter ms_handle_connect 0x7f4a4413eff0 2024-09-17T18:37:51.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.161+0000 7f4a417fa640 10 client.?.objecter resend_mon_ops 2024-09-17T18:37:51.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.161+0000 7f4a417fa640 1 -- 172.21.15.121:0/489072255 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c040070 con 0x7f4a4413eff0 2024-09-17T18:37:51.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.161+0000 7f4a417fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:51.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.161+0000 7f4a417fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T18:37:51.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.161+0000 7f4a417fa640 10 monclient: dump: 2024-09-17T18:37:51.173 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:51.173 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:51.173 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:51.173 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:51.173 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:51.173 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:51.173 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:51.173 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:51.173 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:51.174 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:51.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.161+0000 7f4a417fa640 10 monclient: _finish_auth 0 2024-09-17T18:37:51.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.161+0000 7f4a417fa640 10 monclient: _check_auth_tickets 2024-09-17T18:37:51.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.161+0000 7f4a417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:51.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.161+0000 7f4a417fa640 1 -- 172.21.15.121:0/489072255 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a3c045460 con 0x7f4a4413eff0 2024-09-17T18:37:51.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.161+0000 7f4a417fa640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:51.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.161+0000 7f4a417fa640 1 -- 172.21.15.121:0/489072255 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c007a30 con 0x7f4a4413eff0 2024-09-17T18:37:51.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.161+0000 7f4a4af6b640 5 monclient: authenticate success, global_id 114222 2024-09-17T18:37:51.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.161+0000 7f4a4af6b640 10 monclient: _renew_subs 2024-09-17T18:37:51.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.161+0000 7f4a4af6b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:51.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.161+0000 7f4a4af6b640 1 -- 172.21.15.121:0/489072255 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a44074c80 con 0x7f4a4413eff0 2024-09-17T18:37:51.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.161+0000 7f4a417fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:51.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.161+0000 7f4a42ffd640 4 set_mon_vals no callback set 2024-09-17T18:37:51.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.161+0000 7f4a417fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:37:51.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.161+0000 7f4a417fa640 10 monclient: dump: 2024-09-17T18:37:51.174 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:51.175 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:51.175 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:51.175 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:51.175 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:51.175 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:51.175 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:51.175 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:51.175 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:51.175 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:51.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.161+0000 7f4a4af6b640 10 client.114222.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:37:51.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.161+0000 7f4a4af6b640 10 monclient: _renew_subs 2024-09-17T18:37:51.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.161+0000 7f4a4af6b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:51.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.161+0000 7f4a4af6b640 1 -- 172.21.15.121:0/489072255 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a44135ed0 con 0x7f4a4413eff0 2024-09-17T18:37:51.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.161+0000 7f4a4af6b640 1 librados: init done 2024-09-17T18:37:51.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.162+0000 7f4a4af6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:51.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.162+0000 7f4a4af6b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:51.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.162+0000 7f4a4af6b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:51.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.162+0000 7f4a417fa640 1 -- 172.21.15.121:0/489072255 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7f4a3c039040 con 0x7f4a4413eff0 2024-09-17T18:37:51.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.162+0000 7f4a4af6b640 1 -- 172.21.15.121:0/489072255 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a44073c80 con 0x7f4a4413eff0 2024-09-17T18:37:51.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.162+0000 7f4a417fa640 1 --2- 172.21.15.121:0/489072255 >> v2:172.21.15.173:6812/2423786037 conn(0x7f4a28040170 0x7f4a28042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:51.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.162+0000 7f4a417fa640 1 -- 172.21.15.121:0/489072255 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(679..679 src has 247..679) ==== 4685+0+0 (secure 0 0 0) 0x7f4a3c074e10 con 0x7f4a4413eff0 2024-09-17T18:37:51.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.162+0000 7f4a417fa640 10 client.114222.objecter ms_dispatch 0x7f4a44000b90 osd_map(679..679 src has 247..679) 2024-09-17T18:37:51.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.162+0000 7f4a417fa640 3 client.114222.objecter handle_osd_map got epochs [679,679] > 0 2024-09-17T18:37:51.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.162+0000 7f4a417fa640 3 client.114222.objecter handle_osd_map decoding full epoch 679 2024-09-17T18:37:51.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.163+0000 7f4a417fa640 20 client.114222.objecter dump_active .. 0 homeless 2024-09-17T18:37:51.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.163+0000 7f4a48d07640 1 --2- 172.21.15.121:0/489072255 >> v2:172.21.15.173:6812/2423786037 conn(0x7f4a28040170 0x7f4a28042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:51.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.163+0000 7f4a48d07640 10 monclient: get_auth_request con 0x7f4a28040170 auth_method 0 2024-09-17T18:37:51.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.165+0000 7f4a48d07640 1 --2- 172.21.15.121:0/489072255 >> v2:172.21.15.173:6812/2423786037 conn(0x7f4a28040170 0x7f4a28042630 secure :-1 s=READY pgs=4113 cs=0 l=1 rev1=1 crypto rx=0x7f4a38007c90 tx=0x7f4a38007be0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:51.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.165+0000 7f4a417fa640 10 client.114222.objecter ms_handle_connect 0x7f4a28040170 2024-09-17T18:37:51.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.167+0000 7f4a417fa640 1 -- 172.21.15.121:0/489072255 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4a3c073870 con 0x7f4a4413eff0 2024-09-17T18:37:51.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.167+0000 7f4a417fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:51.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.167+0000 7f4a417fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:37:51.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.337+0000 7f4a4af6b640 1 -- 172.21.15.121:0/489072255 --> v2:172.21.15.173:6812/2423786037 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4a440a44f0 con 0x7f4a28040170 2024-09-17T18:37:51.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.339+0000 7f4a417fa640 1 -- 172.21.15.121:0/489072255 <== mgr.4105 v2:172.21.15.173:6812/2423786037 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114161 (secure 0 0 0) 0x7f4a440a44f0 con 0x7f4a28040170 2024-09-17T18:37:51.342 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-17T18:37:51.343 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-09-17T18:37:51.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.342+0000 7f4a4af6b640 10 librados: watch_flush enter 2024-09-17T18:37:51.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.342+0000 7f4a4af6b640 10 librados: watch_flush exit 2024-09-17T18:37:51.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.342+0000 7f4a4af6b640 20 client.114222.objecter shutdown clearing up homeless session... 2024-09-17T18:37:51.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.342+0000 7f4a4af6b640 10 client.114222.objecter successfully canceled tick 2024-09-17T18:37:51.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.342+0000 7f4a4af6b640 1 -- 172.21.15.121:0/489072255 >> v2:172.21.15.173:6812/2423786037 conn(0x7f4a28040170 msgr2=0x7f4a28042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:51.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.342+0000 7f4a4af6b640 1 --2- 172.21.15.121:0/489072255 >> v2:172.21.15.173:6812/2423786037 conn(0x7f4a28040170 0x7f4a28042630 secure :-1 s=READY pgs=4113 cs=0 l=1 rev1=1 crypto rx=0x7f4a38007c90 tx=0x7f4a38007be0 comp rx=0 tx=0).stop 2024-09-17T18:37:51.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.342+0000 7f4a4af6b640 10 monclient: shutdown 2024-09-17T18:37:51.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.342+0000 7f4a4af6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:51.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.342+0000 7f4a4af6b640 1 -- 172.21.15.121:0/489072255 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a4413eff0 msgr2=0x7f4a4413df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:51.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.342+0000 7f4a4af6b640 1 --2- 172.21.15.121:0/489072255 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a4413eff0 0x7f4a4413df10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c009ee0 tx=0x7f4a3c004a40 comp rx=0 tx=0).stop 2024-09-17T18:37:51.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.342+0000 7f4a4af6b640 1 -- 172.21.15.121:0/489072255 shutdown_connections 2024-09-17T18:37:51.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.342+0000 7f4a4af6b640 1 --2- 172.21.15.121:0/489072255 >> v2:172.21.15.173:6812/2423786037 conn(0x7f4a28040170 0x7f4a28042630 unknown :-1 s=CLOSED pgs=4113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:51.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.342+0000 7f4a4af6b640 1 --2- 172.21.15.121:0/489072255 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a44138a40 0x7f4a4413d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:51.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.342+0000 7f4a4af6b640 1 --2- 172.21.15.121:0/489072255 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a4413eff0 0x7f4a4413df10 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:51.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.342+0000 7f4a4af6b640 1 --2- 172.21.15.121:0/489072255 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a440738a0 0x7f4a441cdab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:51.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.342+0000 7f4a4af6b640 1 -- 172.21.15.121:0/489072255 >> 172.21.15.121:0/489072255 conn(0x7f4a44076f00 msgr2=0x7f4a440a5e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:51.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.342+0000 7f4a4af6b640 1 -- 172.21.15.121:0/489072255 shutdown_connections 2024-09-17T18:37:51.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.342+0000 7f4a4af6b640 1 -- 172.21.15.121:0/489072255 wait complete. 2024-09-17T18:37:51.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.342+0000 7f4a4af6b640 1 librados: shutdown 2024-09-17T18:37:51.354 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":1239,"stamp":"2024-09-17T18:37:49.785136+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87,"ondisk_log_size":87,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":459768,"kb_used_data":39448,"kb_used_omap":9,"kb_used_meta":420214,"kb_avail":565771272,"statfs":{"total":579820584960,"available":579349782528,"internally_reserved":0,"allocated":40394752,"data_stored":18330092,"data_compressed":59128,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":430299809},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003376"},"pg_stats":[{"pgid":"5.15","version":"0'0","reported_seq":1287,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:37:37.422945+0000","last_change":"2024-09-17T18:37:37.422945+0000","last_active":"2024-09-17T18:37:37.422945+0000","last_peered":"2024-09-17T18:37:37.422945+0000","last_clean":"2024-09-17T18:37:37.422945+0000","last_became_active":"2024-09-17T18:20:02.310239+0000","last_became_peered":"2024-09-17T18:20:02.310239+0000","last_unstale":"2024-09-17T18:37:37.422945+0000","last_undegraded":"2024-09-17T18:37:37.422945+0000","last_fullsized":"2024-09-17T18:37:37.422945+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:37:37.422855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:37:37.422855+0000","last_clean_scrub_stamp":"2024-09-17T18:37:37.422855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T03:24:00.277513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1306,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:37:31.310886+0000","last_change":"2024-09-17T18:37:31.310886+0000","last_active":"2024-09-17T18:37:31.310886+0000","last_peered":"2024-09-17T18:37:31.310886+0000","last_clean":"2024-09-17T18:37:31.310886+0000","last_became_active":"2024-09-17T18:20:02.322561+0000","last_became_peered":"2024-09-17T18:20:02.322561+0000","last_unstale":"2024-09-17T18:37:31.310886+0000","last_undegraded":"2024-09-17T18:37:31.310886+0000","last_fullsized":"2024-09-17T18:37:31.310886+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:37:31.310831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:37:31.310831+0000","last_clean_scrub_stamp":"2024-09-17T18:37:31.310831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:49:39.619194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1304,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:37:35.715757+0000","last_change":"2024-09-17T18:37:35.715757+0000","last_active":"2024-09-17T18:37:35.715757+0000","last_peered":"2024-09-17T18:37:35.715757+0000","last_clean":"2024-09-17T18:37:35.715757+0000","last_became_active":"2024-09-17T18:20:02.250675+0000","last_became_peered":"2024-09-17T18:20:02.250675+0000","last_unstale":"2024-09-17T18:37:35.715757+0000","last_undegraded":"2024-09-17T18:37:35.715757+0000","last_fullsized":"2024-09-17T18:37:35.715757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:37:35.715676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:37:35.715676+0000","last_clean_scrub_stamp":"2024-09-17T18:37:35.715676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:49:40.716469+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1303,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:37:36.297588+0000","last_change":"2024-09-17T18:37:36.297588+0000","last_active":"2024-09-17T18:37:36.297588+0000","last_peered":"2024-09-17T18:37:36.297588+0000","last_clean":"2024-09-17T18:37:36.297588+0000","last_became_active":"2024-09-17T18:20:02.342457+0000","last_became_peered":"2024-09-17T18:20:02.342457+0000","last_unstale":"2024-09-17T18:37:36.297588+0000","last_undegraded":"2024-09-17T18:37:36.297588+0000","last_fullsized":"2024-09-17T18:37:36.297588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:37:36.297468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:37:36.297468+0000","last_clean_scrub_stamp":"2024-09-17T18:37:36.297468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:28:36.032394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1288,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:37:35.353334+0000","last_change":"2024-09-17T18:37:35.353334+0000","last_active":"2024-09-17T18:37:35.353334+0000","last_peered":"2024-09-17T18:37:35.353334+0000","last_clean":"2024-09-17T18:37:35.353334+0000","last_became_active":"2024-09-17T18:20:02.249100+0000","last_became_peered":"2024-09-17T18:20:02.249100+0000","last_unstale":"2024-09-17T18:37:35.353334+0000","last_undegraded":"2024-09-17T18:37:35.353334+0000","last_fullsized":"2024-09-17T18:37:35.353334+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:37:35.353255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:37:35.353255+0000","last_clean_scrub_stamp":"2024-09-17T18:37:35.353255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:16:40.131458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1305,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:37:32.701221+0000","last_change":"2024-09-17T18:37:32.701221+0000","last_active":"2024-09-17T18:37:32.701221+0000","last_peered":"2024-09-17T18:37:32.701221+0000","last_clean":"2024-09-17T18:37:32.701221+0000","last_became_active":"2024-09-17T18:20:02.172083+0000","last_became_peered":"2024-09-17T18:20:02.172083+0000","last_unstale":"2024-09-17T18:37:32.701221+0000","last_undegraded":"2024-09-17T18:37:32.701221+0000","last_fullsized":"2024-09-17T18:37:32.701221+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:37:32.701139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:37:32.701139+0000","last_clean_scrub_stamp":"2024-09-17T18:37:32.701139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:56:01.146966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1304,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:37:34.750530+0000","last_change":"2024-09-17T18:37:34.750530+0000","last_active":"2024-09-17T18:37:34.750530+0000","last_peered":"2024-09-17T18:37:34.750530+0000","last_clean":"2024-09-17T18:37:34.750530+0000","last_became_active":"2024-09-17T18:20:02.244673+0000","last_became_peered":"2024-09-17T18:20:02.244673+0000","last_unstale":"2024-09-17T18:37:34.750530+0000","last_undegraded":"2024-09-17T18:37:34.750530+0000","last_fullsized":"2024-09-17T18:37:34.750530+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:37:34.750447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:37:34.750447+0000","last_clean_scrub_stamp":"2024-09-17T18:37:34.750447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:20:58.235655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1305,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:37:31.749330+0000","last_change":"2024-09-17T18:37:31.749330+0000","last_active":"2024-09-17T18:37:31.749330+0000","last_peered":"2024-09-17T18:37:31.749330+0000","last_clean":"2024-09-17T18:37:31.749330+0000","last_became_active":"2024-09-17T18:20:02.315070+0000","last_became_peered":"2024-09-17T18:20:02.315070+0000","last_unstale":"2024-09-17T18:37:31.749330+0000","last_undegraded":"2024-09-17T18:37:31.749330+0000","last_fullsized":"2024-09-17T18:37:31.749330+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:37:31.749263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:37:31.749263+0000","last_clean_scrub_stamp":"2024-09-17T18:37:31.749263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:38:53.786849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1304,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:37:36.728217+0000","last_change":"2024-09-17T18:37:36.728217+0000","last_active":"2024-09-17T18:37:36.728217+0000","last_peered":"2024-09-17T18:37:36.728217+0000","last_clean":"2024-09-17T18:37:36.728217+0000","last_became_active":"2024-09-17T18:20:02.348022+0000","last_became_peered":"2024-09-17T18:20:02.348022+0000","last_unstale":"2024-09-17T18:37:36.728217+0000","last_undegraded":"2024-09-17T18:37:36.728217+0000","last_fullsized":"2024-09-17T18:37:36.728217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:37:36.728124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:37:36.728124+0000","last_clean_scrub_stamp":"2024-09-17T18:37:36.728124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T21:05:51.767087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1290,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:37:31.396104+0000","last_change":"2024-09-17T18:37:31.396104+0000","last_active":"2024-09-17T18:37:31.396104+0000","last_peered":"2024-09-17T18:37:31.396104+0000","last_clean":"2024-09-17T18:37:31.396104+0000","last_became_active":"2024-09-17T18:20:02.328118+0000","last_became_peered":"2024-09-17T18:20:02.328118+0000","last_unstale":"2024-09-17T18:37:31.396104+0000","last_undegraded":"2024-09-17T18:37:31.396104+0000","last_fullsized":"2024-09-17T18:37:31.396104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:37:31.396005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:37:31.396005+0000","last_clean_scrub_stamp":"2024-09-17T18:37:31.396005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T20:29:16.322143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1287,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:37:36.378706+0000","last_change":"2024-09-17T18:37:36.378706+0000","last_active":"2024-09-17T18:37:36.378706+0000","last_peered":"2024-09-17T18:37:36.378706+0000","last_clean":"2024-09-17T18:37:36.378706+0000","last_became_active":"2024-09-17T18:20:02.242391+0000","last_became_peered":"2024-09-17T18:20:02.242391+0000","last_unstale":"2024-09-17T18:37:36.378706+0000","last_undegraded":"2024-09-17T18:37:36.378706+0000","last_fullsized":"2024-09-17T18:37:36.378706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:37:36.378632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:37:36.378632+0000","last_clean_scrub_stamp":"2024-09-17T18:37:36.378632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T21:17:17.537088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1302,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:37:35.290328+0000","last_change":"2024-09-17T18:37:35.290328+0000","last_active":"2024-09-17T18:37:35.290328+0000","last_peered":"2024-09-17T18:37:35.290328+0000","last_clean":"2024-09-17T18:37:35.290328+0000","last_became_active":"2024-09-17T18:20:02.248819+0000","last_became_peered":"2024-09-17T18:20:02.248819+0000","last_unstale":"2024-09-17T18:37:35.290328+0000","last_undegraded":"2024-09-17T18:37:35.290328+0000","last_fullsized":"2024-09-17T18:37:35.290328+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:37:35.290240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:37:35.290240+0000","last_clean_scrub_stamp":"2024-09-17T18:37:35.290240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:20:09.461172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1284,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:37:32.261085+0000","last_change":"2024-09-17T18:37:32.261085+0000","last_active":"2024-09-17T18:37:32.261085+0000","last_peered":"2024-09-17T18:37:32.261085+0000","last_clean":"2024-09-17T18:37:32.261085+0000","last_became_active":"2024-09-17T18:20:02.259327+0000","last_became_peered":"2024-09-17T18:20:02.259327+0000","last_unstale":"2024-09-17T18:37:32.261085+0000","last_undegraded":"2024-09-17T18:37:32.261085+0000","last_fullsized":"2024-09-17T18:37:32.261085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:37:32.260988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:37:32.260988+0000","last_clean_scrub_stamp":"2024-09-17T18:37:32.260988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:06:33.376390+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"679'74","reported_seq":1407,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:37:32.230476+0000","last_change":"2024-09-17T18:37:32.230476+0000","last_active":"2024-09-17T18:37:32.230476+0000","last_peered":"2024-09-17T18:37:32.230476+0000","last_clean":"2024-09-17T18:37:32.230476+0000","last_became_active":"2024-09-17T18:19:00.244903+0000","last_became_peered":"2024-09-17T18:19:00.244903+0000","last_unstale":"2024-09-17T18:37:32.230476+0000","last_undegraded":"2024-09-17T18:37:32.230476+0000","last_fullsized":"2024-09-17T18:37:32.230476+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"679'74","last_scrub_stamp":"2024-09-17T18:37:32.230396+0000","last_deep_scrub":"679'74","last_deep_scrub_stamp":"2024-09-17T18:37:32.230396+0000","last_clean_scrub_stamp":"2024-09-17T18:37:32.230396+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:18:47.988567+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"679'2","reported_seq":1306,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:37:32.408572+0000","last_change":"2024-09-17T18:37:32.408482+0000","last_active":"2024-09-17T18:37:32.408572+0000","last_peered":"2024-09-17T18:37:32.408572+0000","last_clean":"2024-09-17T18:37:32.408572+0000","last_became_active":"2024-09-17T18:18:59.526797+0000","last_became_peered":"2024-09-17T18:18:59.526797+0000","last_unstale":"2024-09-17T18:37:32.408572+0000","last_undegraded":"2024-09-17T18:37:32.408572+0000","last_fullsized":"2024-09-17T18:37:32.408572+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"679'2","last_scrub_stamp":"2024-09-17T18:37:32.408392+0000","last_deep_scrub":"679'2","last_deep_scrub_stamp":"2024-09-17T18:37:32.408392+0000","last_clean_scrub_stamp":"2024-09-17T18:37:32.408392+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:41:14.608685+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000425759,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1286,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:37:34.373927+0000","last_change":"2024-09-17T18:37:34.373927+0000","last_active":"2024-09-17T18:37:34.373927+0000","last_peered":"2024-09-17T18:37:34.373927+0000","last_clean":"2024-09-17T18:37:34.373927+0000","last_became_active":"2024-09-17T18:20:02.310633+0000","last_became_peered":"2024-09-17T18:20:02.310633+0000","last_unstale":"2024-09-17T18:37:34.373927+0000","last_undegraded":"2024-09-17T18:37:34.373927+0000","last_fullsized":"2024-09-17T18:37:34.373927+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:37:34.373840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:37:34.373840+0000","last_clean_scrub_stamp":"2024-09-17T18:37:34.373840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T20:29:20.812683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1320,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:37:31.184574+0000","last_change":"2024-09-17T18:37:31.184574+0000","last_active":"2024-09-17T18:37:31.184574+0000","last_peered":"2024-09-17T18:37:31.184574+0000","last_clean":"2024-09-17T18:37:31.184574+0000","last_became_active":"2024-09-17T18:18:59.526855+0000","last_became_peered":"2024-09-17T18:18:59.526855+0000","last_unstale":"2024-09-17T18:37:31.184574+0000","last_undegraded":"2024-09-17T18:37:31.184574+0000","last_fullsized":"2024-09-17T18:37:31.184574+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:37:31.184484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:37:31.184484+0000","last_clean_scrub_stamp":"2024-09-17T18:37:31.184484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:17:07.420260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032532500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1305,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:37:30.771234+0000","last_change":"2024-09-17T18:37:30.771234+0000","last_active":"2024-09-17T18:37:30.771234+0000","last_peered":"2024-09-17T18:37:30.771234+0000","last_clean":"2024-09-17T18:37:30.771234+0000","last_became_active":"2024-09-17T18:20:02.260389+0000","last_became_peered":"2024-09-17T18:20:02.260389+0000","last_unstale":"2024-09-17T18:37:30.771234+0000","last_undegraded":"2024-09-17T18:37:30.771234+0000","last_fullsized":"2024-09-17T18:37:30.771234+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:37:30.771146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:37:30.771146+0000","last_clean_scrub_stamp":"2024-09-17T18:37:30.771146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:21:08.390829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1321,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:37:33.733859+0000","last_change":"2024-09-17T18:37:33.733798+0000","last_active":"2024-09-17T18:37:33.733859+0000","last_peered":"2024-09-17T18:37:33.733859+0000","last_clean":"2024-09-17T18:37:33.733859+0000","last_became_active":"2024-09-17T18:18:59.529390+0000","last_became_peered":"2024-09-17T18:18:59.529390+0000","last_unstale":"2024-09-17T18:37:33.733859+0000","last_undegraded":"2024-09-17T18:37:33.733859+0000","last_fullsized":"2024-09-17T18:37:33.733859+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:37:33.733722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:37:33.733722+0000","last_clean_scrub_stamp":"2024-09-17T18:37:33.733722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:06:11.996935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00065986199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1307,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:37:30.285662+0000","last_change":"2024-09-17T18:37:30.285662+0000","last_active":"2024-09-17T18:37:30.285662+0000","last_peered":"2024-09-17T18:37:30.285662+0000","last_clean":"2024-09-17T18:37:30.285662+0000","last_became_active":"2024-09-17T18:20:02.251405+0000","last_became_peered":"2024-09-17T18:20:02.251405+0000","last_unstale":"2024-09-17T18:37:30.285662+0000","last_undegraded":"2024-09-17T18:37:30.285662+0000","last_fullsized":"2024-09-17T18:37:30.285662+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:37:30.285591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:37:30.285591+0000","last_clean_scrub_stamp":"2024-09-17T18:37:30.285591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T03:25:25.137819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"679'3","reported_seq":1307,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:37:31.382468+0000","last_change":"2024-09-17T18:37:31.382376+0000","last_active":"2024-09-17T18:37:31.382468+0000","last_peered":"2024-09-17T18:37:31.382468+0000","last_clean":"2024-09-17T18:37:31.382468+0000","last_became_active":"2024-09-17T18:18:59.526791+0000","last_became_peered":"2024-09-17T18:18:59.526791+0000","last_unstale":"2024-09-17T18:37:31.382468+0000","last_undegraded":"2024-09-17T18:37:31.382468+0000","last_fullsized":"2024-09-17T18:37:31.382468+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"679'3","last_scrub_stamp":"2024-09-17T18:37:31.382281+0000","last_deep_scrub":"679'3","last_deep_scrub_stamp":"2024-09-17T18:37:31.382281+0000","last_clean_scrub_stamp":"2024-09-17T18:37:31.382281+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:21:23.679030+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010088289999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1307,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:37:28.303705+0000","last_change":"2024-09-17T18:37:28.303705+0000","last_active":"2024-09-17T18:37:28.303705+0000","last_peered":"2024-09-17T18:37:28.303705+0000","last_clean":"2024-09-17T18:37:28.303705+0000","last_became_active":"2024-09-17T18:20:02.268644+0000","last_became_peered":"2024-09-17T18:20:02.268644+0000","last_unstale":"2024-09-17T18:37:28.303705+0000","last_undegraded":"2024-09-17T18:37:28.303705+0000","last_fullsized":"2024-09-17T18:37:28.303705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:37:28.303648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:37:28.303648+0000","last_clean_scrub_stamp":"2024-09-17T18:37:28.303648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:17:03.577440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1307,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:37:29.393501+0000","last_change":"2024-09-17T18:37:29.393455+0000","last_active":"2024-09-17T18:37:29.393501+0000","last_peered":"2024-09-17T18:37:29.393501+0000","last_clean":"2024-09-17T18:37:29.393501+0000","last_became_active":"2024-09-17T18:19:00.244613+0000","last_became_peered":"2024-09-17T18:19:00.244613+0000","last_unstale":"2024-09-17T18:37:29.393501+0000","last_undegraded":"2024-09-17T18:37:29.393501+0000","last_fullsized":"2024-09-17T18:37:29.393501+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:37:29.393392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:37:29.393392+0000","last_clean_scrub_stamp":"2024-09-17T18:37:29.393392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:12:19.208607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063646500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1288,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:37:33.398793+0000","last_change":"2024-09-17T18:37:33.398793+0000","last_active":"2024-09-17T18:37:33.398793+0000","last_peered":"2024-09-17T18:37:33.398793+0000","last_clean":"2024-09-17T18:37:33.398793+0000","last_became_active":"2024-09-17T18:20:02.254259+0000","last_became_peered":"2024-09-17T18:20:02.254259+0000","last_unstale":"2024-09-17T18:37:33.398793+0000","last_undegraded":"2024-09-17T18:37:33.398793+0000","last_fullsized":"2024-09-17T18:37:33.398793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:37:33.398701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:37:33.398701+0000","last_clean_scrub_stamp":"2024-09-17T18:37:33.398701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:33:10.968278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1320,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:37:30.223623+0000","last_change":"2024-09-17T18:37:30.223552+0000","last_active":"2024-09-17T18:37:30.223623+0000","last_peered":"2024-09-17T18:37:30.223623+0000","last_clean":"2024-09-17T18:37:30.223623+0000","last_became_active":"2024-09-17T18:19:00.244851+0000","last_became_peered":"2024-09-17T18:19:00.244851+0000","last_unstale":"2024-09-17T18:37:30.223623+0000","last_undegraded":"2024-09-17T18:37:30.223623+0000","last_fullsized":"2024-09-17T18:37:30.223623+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:37:30.223467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:37:30.223467+0000","last_clean_scrub_stamp":"2024-09-17T18:37:30.223467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T21:35:32.346924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041282100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"679'8","reported_seq":1318,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:37:34.242922+0000","last_change":"2024-09-17T18:37:34.242922+0000","last_active":"2024-09-17T18:37:34.242922+0000","last_peered":"2024-09-17T18:37:34.242922+0000","last_clean":"2024-09-17T18:37:34.242922+0000","last_became_active":"2024-09-17T18:20:02.253328+0000","last_became_peered":"2024-09-17T18:20:02.253328+0000","last_unstale":"2024-09-17T18:37:34.242922+0000","last_undegraded":"2024-09-17T18:37:34.242922+0000","last_fullsized":"2024-09-17T18:37:34.242922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"679'8","last_scrub_stamp":"2024-09-17T18:37:34.242831+0000","last_deep_scrub":"679'8","last_deep_scrub_stamp":"2024-09-17T18:37:34.242831+0000","last_clean_scrub_stamp":"2024-09-17T18:37:34.242831+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:03:20.890237+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1306,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:37:30.392643+0000","last_change":"2024-09-17T18:37:30.392643+0000","last_active":"2024-09-17T18:37:30.392643+0000","last_peered":"2024-09-17T18:37:30.392643+0000","last_clean":"2024-09-17T18:37:30.392643+0000","last_became_active":"2024-09-17T18:18:59.525295+0000","last_became_peered":"2024-09-17T18:18:59.525295+0000","last_unstale":"2024-09-17T18:37:30.392643+0000","last_undegraded":"2024-09-17T18:37:30.392643+0000","last_fullsized":"2024-09-17T18:37:30.392643+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:37:30.392539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:37:30.392539+0000","last_clean_scrub_stamp":"2024-09-17T18:37:30.392539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:06:45.063910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062121300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1305,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:37:29.799034+0000","last_change":"2024-09-17T18:37:29.799034+0000","last_active":"2024-09-17T18:37:29.799034+0000","last_peered":"2024-09-17T18:37:29.799034+0000","last_clean":"2024-09-17T18:37:29.799034+0000","last_became_active":"2024-09-17T18:20:02.234414+0000","last_became_peered":"2024-09-17T18:20:02.234414+0000","last_unstale":"2024-09-17T18:37:29.799034+0000","last_undegraded":"2024-09-17T18:37:29.799034+0000","last_fullsized":"2024-09-17T18:37:29.799034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:37:29.798962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:37:29.798962+0000","last_clean_scrub_stamp":"2024-09-17T18:37:29.798962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:51:11.607945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1319,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:37:33.242987+0000","last_change":"2024-09-17T18:37:33.242987+0000","last_active":"2024-09-17T18:37:33.242987+0000","last_peered":"2024-09-17T18:37:33.242987+0000","last_clean":"2024-09-17T18:37:33.242987+0000","last_became_active":"2024-09-17T18:19:00.056752+0000","last_became_peered":"2024-09-17T18:19:00.056752+0000","last_unstale":"2024-09-17T18:37:33.242987+0000","last_undegraded":"2024-09-17T18:37:33.242987+0000","last_fullsized":"2024-09-17T18:37:33.242987+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:37:33.242903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:37:33.242903+0000","last_clean_scrub_stamp":"2024-09-17T18:37:33.242903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:18:09.188290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049856399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1290,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:37:32.356668+0000","last_change":"2024-09-17T18:37:32.356668+0000","last_active":"2024-09-17T18:37:32.356668+0000","last_peered":"2024-09-17T18:37:32.356668+0000","last_clean":"2024-09-17T18:37:32.356668+0000","last_became_active":"2024-09-17T18:20:02.226503+0000","last_became_peered":"2024-09-17T18:20:02.226503+0000","last_unstale":"2024-09-17T18:37:32.356668+0000","last_undegraded":"2024-09-17T18:37:32.356668+0000","last_fullsized":"2024-09-17T18:37:32.356668+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:37:32.356582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:37:32.356582+0000","last_clean_scrub_stamp":"2024-09-17T18:37:32.356582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:55:50.972775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1290,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:37:33.336521+0000","last_change":"2024-09-17T18:37:33.336521+0000","last_active":"2024-09-17T18:37:33.336521+0000","last_peered":"2024-09-17T18:37:33.336521+0000","last_clean":"2024-09-17T18:37:33.336521+0000","last_became_active":"2024-09-17T18:20:02.311023+0000","last_became_peered":"2024-09-17T18:20:02.311023+0000","last_unstale":"2024-09-17T18:37:33.336521+0000","last_undegraded":"2024-09-17T18:37:33.336521+0000","last_fullsized":"2024-09-17T18:37:33.336521+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:37:33.336442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:37:33.336442+0000","last_clean_scrub_stamp":"2024-09-17T18:37:33.336442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:47:46.255415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1283,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:37:31.250902+0000","last_change":"2024-09-17T18:37:31.250902+0000","last_active":"2024-09-17T18:37:31.250902+0000","last_peered":"2024-09-17T18:37:31.250902+0000","last_clean":"2024-09-17T18:37:31.250902+0000","last_became_active":"2024-09-17T18:20:02.339489+0000","last_became_peered":"2024-09-17T18:20:02.339489+0000","last_unstale":"2024-09-17T18:37:31.250902+0000","last_undegraded":"2024-09-17T18:37:31.250902+0000","last_fullsized":"2024-09-17T18:37:31.250902+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:37:31.250815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:37:31.250815+0000","last_clean_scrub_stamp":"2024-09-17T18:37:31.250815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:02:16.966247+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1303,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:37:37.263006+0000","last_change":"2024-09-17T18:37:37.263006+0000","last_active":"2024-09-17T18:37:37.263006+0000","last_peered":"2024-09-17T18:37:37.263006+0000","last_clean":"2024-09-17T18:37:37.263006+0000","last_became_active":"2024-09-17T18:20:02.286560+0000","last_became_peered":"2024-09-17T18:20:02.286560+0000","last_unstale":"2024-09-17T18:37:37.263006+0000","last_undegraded":"2024-09-17T18:37:37.263006+0000","last_fullsized":"2024-09-17T18:37:37.263006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:37:37.262922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:37:37.262922+0000","last_clean_scrub_stamp":"2024-09-17T18:37:37.262922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:59:35.942414+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1307,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:37:29.319907+0000","last_change":"2024-09-17T18:37:29.319907+0000","last_active":"2024-09-17T18:37:29.319907+0000","last_peered":"2024-09-17T18:37:29.319907+0000","last_clean":"2024-09-17T18:37:29.319907+0000","last_became_active":"2024-09-17T18:20:02.270521+0000","last_became_peered":"2024-09-17T18:20:02.270521+0000","last_unstale":"2024-09-17T18:37:29.319907+0000","last_undegraded":"2024-09-17T18:37:29.319907+0000","last_fullsized":"2024-09-17T18:37:29.319907+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:37:29.319846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:37:29.319846+0000","last_clean_scrub_stamp":"2024-09-17T18:37:29.319846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:18:59.825393+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1304,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:37:38.260943+0000","last_change":"2024-09-17T18:37:38.260943+0000","last_active":"2024-09-17T18:37:38.260943+0000","last_peered":"2024-09-17T18:37:38.260943+0000","last_clean":"2024-09-17T18:37:38.260943+0000","last_became_active":"2024-09-17T18:20:02.293587+0000","last_became_peered":"2024-09-17T18:20:02.293587+0000","last_unstale":"2024-09-17T18:37:38.260943+0000","last_undegraded":"2024-09-17T18:37:38.260943+0000","last_fullsized":"2024-09-17T18:37:38.260943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:37:38.260854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:37:38.260854+0000","last_clean_scrub_stamp":"2024-09-17T18:37:38.260854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T21:08:46.941965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1283,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:37:34.236968+0000","last_change":"2024-09-17T18:37:34.236968+0000","last_active":"2024-09-17T18:37:34.236968+0000","last_peered":"2024-09-17T18:37:34.236968+0000","last_clean":"2024-09-17T18:37:34.236968+0000","last_became_active":"2024-09-17T18:20:02.259616+0000","last_became_peered":"2024-09-17T18:20:02.259616+0000","last_unstale":"2024-09-17T18:37:34.236968+0000","last_undegraded":"2024-09-17T18:37:34.236968+0000","last_fullsized":"2024-09-17T18:37:34.236968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:37:34.236884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:37:34.236884+0000","last_clean_scrub_stamp":"2024-09-17T18:37:34.236884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:17:16.894833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1290,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:37:35.343094+0000","last_change":"2024-09-17T18:37:35.343094+0000","last_active":"2024-09-17T18:37:35.343094+0000","last_peered":"2024-09-17T18:37:35.343094+0000","last_clean":"2024-09-17T18:37:35.343094+0000","last_became_active":"2024-09-17T18:20:02.313525+0000","last_became_peered":"2024-09-17T18:20:02.313525+0000","last_unstale":"2024-09-17T18:37:35.343094+0000","last_undegraded":"2024-09-17T18:37:35.343094+0000","last_fullsized":"2024-09-17T18:37:35.343094+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:37:35.342981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:37:35.342981+0000","last_clean_scrub_stamp":"2024-09-17T18:37:35.342981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T03:52:27.160931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1283,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:37:33.258674+0000","last_change":"2024-09-17T18:37:33.258615+0000","last_active":"2024-09-17T18:37:33.258674+0000","last_peered":"2024-09-17T18:37:33.258674+0000","last_clean":"2024-09-17T18:37:33.258674+0000","last_became_active":"2024-09-17T18:20:02.292188+0000","last_became_peered":"2024-09-17T18:20:02.292188+0000","last_unstale":"2024-09-17T18:37:33.258674+0000","last_undegraded":"2024-09-17T18:37:33.258674+0000","last_fullsized":"2024-09-17T18:37:33.258674+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:37:33.258542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:37:33.258542+0000","last_clean_scrub_stamp":"2024-09-17T18:37:33.258542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:43:28.207031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1287,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:37:38.435515+0000","last_change":"2024-09-17T18:37:38.435515+0000","last_active":"2024-09-17T18:37:38.435515+0000","last_peered":"2024-09-17T18:37:38.435515+0000","last_clean":"2024-09-17T18:37:38.435515+0000","last_became_active":"2024-09-17T18:20:02.254813+0000","last_became_peered":"2024-09-17T18:20:02.254813+0000","last_unstale":"2024-09-17T18:37:38.435515+0000","last_undegraded":"2024-09-17T18:37:38.435515+0000","last_fullsized":"2024-09-17T18:37:38.435515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:37:38.435442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:37:38.435442+0000","last_clean_scrub_stamp":"2024-09-17T18:37:38.435442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T03:40:35.750810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1283,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:37:35.223887+0000","last_change":"2024-09-17T18:37:35.223887+0000","last_active":"2024-09-17T18:37:35.223887+0000","last_peered":"2024-09-17T18:37:35.223887+0000","last_clean":"2024-09-17T18:37:35.223887+0000","last_became_active":"2024-09-17T18:20:02.299834+0000","last_became_peered":"2024-09-17T18:20:02.299834+0000","last_unstale":"2024-09-17T18:37:35.223887+0000","last_undegraded":"2024-09-17T18:37:35.223887+0000","last_fullsized":"2024-09-17T18:37:35.223887+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:37:35.223798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:37:35.223798+0000","last_clean_scrub_stamp":"2024-09-17T18:37:35.223798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:38:15.016882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1290,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-17T18:37:34.378679+0000","last_change":"2024-09-17T18:37:34.378679+0000","last_active":"2024-09-17T18:37:34.378679+0000","last_peered":"2024-09-17T18:37:34.378679+0000","last_clean":"2024-09-17T18:37:34.378679+0000","last_became_active":"2024-09-17T18:20:02.228502+0000","last_became_peered":"2024-09-17T18:20:02.228502+0000","last_unstale":"2024-09-17T18:37:34.378679+0000","last_undegraded":"2024-09-17T18:37:34.378679+0000","last_fullsized":"2024-09-17T18:37:34.378679+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T18:37:34.378563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T18:37:34.378563+0000","last_clean_scrub_stamp":"2024-09-17T18:37:34.378563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:17:44.395067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59128,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607783,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69672,"kb_used_data":6552,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94302168,"statfs":{"total":96636764160,"available":96565420032,"internally_reserved":0,"allocated":6709248,"data_stored":2878698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607782,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77996,"kb_used_data":6556,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94293844,"statfs":{"total":96636764160,"available":96556896256,"internally_reserved":0,"allocated":6713344,"data_stored":2878717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640484,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82156,"kb_used_data":6556,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289684,"statfs":{"total":96636764160,"available":96552636416,"internally_reserved":0,"allocated":6713344,"data_stored":2878717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640486,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82216,"kb_used_data":6616,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289624,"statfs":{"total":96636764160,"available":96552574976,"internally_reserved":0,"allocated":6774784,"data_stored":3407631,"data_compressed":29564,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640485,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73832,"kb_used_data":6552,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94298008,"statfs":{"total":96636764160,"available":96561160192,"internally_reserved":0,"allocated":6709248,"data_stored":2878698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640487,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73896,"kb_used_data":6616,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94297944,"statfs":{"total":96636764160,"available":96561094656,"internally_reserved":0,"allocated":6774784,"data_stored":3407631,"data_compressed":29564,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29564,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29564,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T18:37:51.357 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-17T18:37:51.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.455+0000 7fa679df8640 10 monclient: get_monmap_and_config 2024-09-17T18:37:51.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.455+0000 7fa679df8640 10 monclient: build_initial_monmap 2024-09-17T18:37:51.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.455+0000 7fa679df8640 10 monclient: monmap: 2024-09-17T18:37:51.458 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:51.458 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:51.458 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:37:51.456517+0000 2024-09-17T18:37:51.458 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:37:51.456517+0000 2024-09-17T18:37:51.458 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:51.458 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:51.458 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a-legacy 2024-09-17T18:37:51.458 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c-legacy 2024-09-17T18:37:51.458 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-b 2024-09-17T18:37:51.458 INFO:teuthology.orchestra.run.smithi121.stderr:3: v2:172.21.15.121:3300/0 mon.noname-a 2024-09-17T18:37:51.458 INFO:teuthology.orchestra.run.smithi121.stderr:4: v2:172.21.15.173:3300/0 mon.noname-c 2024-09-17T18:37:51.458 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:51.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.456+0000 7fa679df8640 1 Processor -- start 2024-09-17T18:37:51.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.456+0000 7fa679df8640 1 -- start start 2024-09-17T18:37:51.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.456+0000 7fa679df8640 10 monclient: init 2024-09-17T18:37:51.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.457+0000 7fa679df8640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:51.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.457+0000 7fa679df8640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-17T18:37:51.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.457+0000 7fa679df8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa6740a5b90 addr v1:172.21.15.121:6789/0 2024-09-17T18:37:51.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.457+0000 7fa679df8640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fa6740a7090 0x7fa67413def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:51.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.457+0000 7fa679df8640 10 monclient(hunting): picked mon.noname-c con 0x7fa6740a7090 addr v2:172.21.15.173:3300/0 2024-09-17T18:37:51.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.457+0000 7fa679df8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa6740a6680 addr v1:172.21.15.173:6789/0 2024-09-17T18:37:51.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.457+0000 7fa679df8640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6740a6e70 con 0x7fa6740a5b90 2024-09-17T18:37:51.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.457+0000 7fa679df8640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa67409cb00 con 0x7fa6740a6680 2024-09-17T18:37:51.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.457+0000 7fa679df8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:51.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.457+0000 7fa679df8640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 -- 0x7fa67409cde0 con 0x7fa6740a7090 2024-09-17T18:37:51.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.457+0000 7fa679df8640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:51.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.457+0000 7fa679df8640 10 monclient(hunting): authenticate will time out at 1875.173333s 2024-09-17T18:37:51.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.457+0000 7fa6737fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fa6740a5b90 0x7fa6740a5f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:42870/0 (socket says 172.21.15.121:42870) 2024-09-17T18:37:51.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.457+0000 7fa6737fe640 1 -- 172.21.15.121:0/1208760279 learned_addr learned my addr 172.21.15.121:0/1208760279 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-17T18:37:51.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.457+0000 7fa66bfff640 1 --2- 172.21.15.121:0/1208760279 >> v2:172.21.15.173:3300/0 conn(0x7fa6740a7090 0x7fa67413def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:51.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.457+0000 7fa66bfff640 10 monclient(hunting): get_auth_request con 0x7fa6740a7090 auth_method 0 2024-09-17T18:37:51.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.457+0000 7fa66bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:51.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.457+0000 7fa66bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:51.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.457+0000 7fa66bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:51.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.457+0000 7fa66bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:51.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.457+0000 7fa66bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:51.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.457+0000 7fa66bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:51.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.457+0000 7fa66bfff640 10 monclient(hunting): handle_auth_done global_id 84385 payload 274 2024-09-17T18:37:51.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.457+0000 7fa66bfff640 1 -- 172.21.15.121:0/1208760279 >> v1:172.21.15.173:6789/0 conn(0x7fa6740a6680 legacy=0x7fa674140440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:51.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.457+0000 7fa66bfff640 1 -- 172.21.15.121:0/1208760279 >> v1:172.21.15.121:6789/0 conn(0x7fa6740a5b90 legacy=0x7fa6740a5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T18:37:51.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.457+0000 7fa66bfff640 10 monclient: _finish_hunting 0 2024-09-17T18:37:51.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.457+0000 7fa66bfff640 1 monclient: found mon.noname-c 2024-09-17T18:37:51.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.457+0000 7fa66bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:51.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.457+0000 7fa66bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.173:3300/0 2024-09-17T18:37:51.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.457+0000 7fa66bfff640 1 -- 172.21.15.121:0/1208760279 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa67409d0c0 con 0x7fa6740a7090 2024-09-17T18:37:51.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.460+0000 7fa66bfff640 1 -- 172.21.15.121:0/1208760279 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6740a7090 msgr2=0x7fa67413def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:51.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.460+0000 7fa66bfff640 1 --2- 172.21.15.121:0/1208760279 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6740a7090 0x7fa67413def0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa65c0097a0 tx=0x7fa65c029fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f911b1da0d3fbc5 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:51.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.461+0000 7fa672ffd640 1 -- 172.21.15.121:0/1208760279 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65c037070 con 0x7fa6740a7090 2024-09-17T18:37:51.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.461+0000 7fa672ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:51.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.461+0000 7fa672ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:37:51.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.461+0000 7fa672ffd640 10 monclient: dump: 2024-09-17T18:37:51.467 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:51.467 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:51.467 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:51.467 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:51.467 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:51.467 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:51.467 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:51.467 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:51.467 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:51.467 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:51.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.461+0000 7fa672ffd640 10 monclient: _finish_auth 0 2024-09-17T18:37:51.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.461+0000 7fa672ffd640 10 monclient: _check_auth_tickets 2024-09-17T18:37:51.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.461+0000 7fa672ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:51.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.461+0000 7fa672ffd640 1 -- 172.21.15.121:0/1208760279 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa65c032640 con 0x7fa6740a7090 2024-09-17T18:37:51.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.461+0000 7fa679df8640 5 monclient: authenticate success, global_id 84385 2024-09-17T18:37:51.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.461+0000 7fa679df8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T18:37:51.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.461+0000 7fa672ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:51.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.461+0000 7fa679df8640 10 monclient: get_monmap_and_config success 2024-09-17T18:37:51.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.461+0000 7fa672ffd640 1 -- 172.21.15.121:0/1208760279 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65c032a10 con 0x7fa6740a7090 2024-09-17T18:37:51.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.461+0000 7fa679df8640 4 set_mon_vals no callback set 2024-09-17T18:37:51.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.461+0000 7fa672ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:51.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.461+0000 7fa672ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T18:37:51.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.461+0000 7fa672ffd640 10 monclient: dump: 2024-09-17T18:37:51.468 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:51.468 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:51.468 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:51.468 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:51.468 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:51.469 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:51.469 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:51.469 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:51.469 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:51.469 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:51.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.461+0000 7fa679df8640 10 monclient: shutdown 2024-09-17T18:37:51.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.461+0000 7fa679df8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:51.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.461+0000 7fa679df8640 1 -- 172.21.15.121:0/1208760279 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6740a7090 msgr2=0x7fa67413def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:51.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.461+0000 7fa679df8640 1 --2- 172.21.15.121:0/1208760279 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6740a7090 0x7fa67413def0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa65c0097a0 tx=0x7fa65c029fd0 comp rx=0 tx=0).stop 2024-09-17T18:37:51.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.461+0000 7fa679df8640 1 -- 172.21.15.121:0/1208760279 shutdown_connections 2024-09-17T18:37:51.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.461+0000 7fa679df8640 1 --2- 172.21.15.121:0/1208760279 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6740a7090 0x7fa67413def0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:51.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.461+0000 7fa679df8640 1 -- 172.21.15.121:0/1208760279 >> 172.21.15.121:0/1208760279 conn(0x7fa6740a7c90 msgr2=0x7fa6740a80a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:51.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.461+0000 7fa679df8640 1 -- 172.21.15.121:0/1208760279 shutdown_connections 2024-09-17T18:37:51.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.461+0000 7fa679df8640 1 -- 172.21.15.121:0/1208760279 wait complete. 2024-09-17T18:37:51.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.462+0000 7fa679df8640 10 monclient: build_initial_monmap 2024-09-17T18:37:51.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.462+0000 7fa679df8640 10 monclient: monmap: 2024-09-17T18:37:51.470 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-17T18:37:51.470 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T18:37:51.470 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-17T18:37:51.470 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-17T18:37:51.470 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-17T18:37:51.470 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:51.470 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.noname-a 2024-09-17T18:37:51.470 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.noname-b 2024-09-17T18:37:51.470 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.noname-c 2024-09-17T18:37:51.470 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:51.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.462+0000 7fa679df8640 1 librados: starting msgr at 2024-09-17T18:37:51.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.462+0000 7fa679df8640 1 librados: starting objecter 2024-09-17T18:37:51.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.462+0000 7fa679df8640 1 Processor -- start 2024-09-17T18:37:51.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.462+0000 7fa679df8640 1 -- start start 2024-09-17T18:37:51.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.462+0000 7fa679df8640 1 librados: setting wanted keys 2024-09-17T18:37:51.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.462+0000 7fa679df8640 1 librados: calling monclient init 2024-09-17T18:37:51.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.462+0000 7fa679df8640 10 monclient: init 2024-09-17T18:37:51.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa679df8640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:51.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa679df8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T18:37:51.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa679df8640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6740a5b90 0x7fa674135d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:51.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa679df8640 10 monclient(hunting): picked mon.noname-c con 0x7fa6740a5b90 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:51.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa679df8640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6740a6680 0x7fa6741362d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:51.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa679df8640 10 monclient(hunting): picked mon.noname-a con 0x7fa6740a6680 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:37:51.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa679df8640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6740a7090 0x7fa67413b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:51.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa679df8640 10 monclient(hunting): picked mon.noname-b con 0x7fa6740a7090 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:37:51.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa679df8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:51.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa679df8640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa6740a8ba0 con 0x7fa6740a6680 2024-09-17T18:37:51.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa679df8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:51.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa679df8640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa6740a8a20 con 0x7fa6740a7090 2024-09-17T18:37:51.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa679df8640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:51.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa679df8640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa674141b60 con 0x7fa6740a5b90 2024-09-17T18:37:51.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa679df8640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:51.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa679df8640 10 monclient(hunting): authenticate will time out at 1875.179291s 2024-09-17T18:37:51.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa66bfff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6740a6680 0x7fa6741362d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:51.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa6737fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6740a5b90 0x7fa674135d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:51.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa66bfff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6740a6680 0x7fa6741362d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41826/0 (socket says 172.21.15.121:41826) 2024-09-17T18:37:51.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa66bfff640 1 -- 172.21.15.121:0/3824917496 learned_addr learned my addr 172.21.15.121:0/3824917496 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-09-17T18:37:51.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa66bfff640 10 monclient(hunting): get_auth_request con 0x7fa6740a6680 auth_method 0 2024-09-17T18:37:51.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa6737fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6740a5b90 0x7fa674135d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:34978/0 (socket says 172.21.15.121:34978) 2024-09-17T18:37:51.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa66bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:51.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa66bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:51.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa66bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:51.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa6737fe640 10 monclient(hunting): get_auth_request con 0x7fa6740a5b90 auth_method 0 2024-09-17T18:37:51.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa6737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:51.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa6737fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:51.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa6737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:51.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa673fff640 1 --2- 172.21.15.121:0/3824917496 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6740a7090 0x7fa67413b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:51.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa66bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:51.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa66bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:51.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa66bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:51.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa6737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:51.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa6737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:51.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa6737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T18:37:51.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa673fff640 10 monclient(hunting): get_auth_request con 0x7fa6740a7090 auth_method 0 2024-09-17T18:37:51.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa673fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:51.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa673fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:51.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa673fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T18:37:51.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa673fff640 1 -- 172.21.15.121:0/3824917496 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6740a7090 msgr2=0x7fa67413b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send send error: (32) Broken pipe 2024-09-17T18:37:51.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa673fff640 1 --2- 172.21.15.121:0/3824917496 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6740a7090 0x7fa67413b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).write auth request write failed r=-32 ((32) Broken pipe) 2024-09-17T18:37:51.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa673fff640 1 --2- 172.21.15.121:0/3824917496 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6740a7090 0x7fa67413b130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T18:37:51.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa6737fe640 10 monclient(hunting): handle_auth_done global_id 64682 payload 824 2024-09-17T18:37:51.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa6737fe640 1 -- 172.21.15.121:0/3824917496 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6740a7090 msgr2=0x7fa67413b130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T18:37:51.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa6737fe640 1 --2- 172.21.15.121:0/3824917496 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6740a7090 0x7fa67413b130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:51.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa6737fe640 1 -- 172.21.15.121:0/3824917496 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6740a6680 msgr2=0x7fa6741362d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:51.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa6737fe640 1 --2- 172.21.15.121:0/3824917496 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6740a6680 0x7fa6741362d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:51.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa6737fe640 10 monclient: _finish_hunting 0 2024-09-17T18:37:51.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa6737fe640 1 monclient: found mon.noname-c 2024-09-17T18:37:51.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa6737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:51.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa6737fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.121:3301/0 2024-09-17T18:37:51.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa6737fe640 1 -- 172.21.15.121:0/3824917496 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6740aa560 con 0x7fa6740a5b90 2024-09-17T18:37:51.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.463+0000 7fa66bfff640 1 --2- 172.21.15.121:0/3824917496 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6740a6680 0x7fa6741362d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T18:37:51.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.465+0000 7fa6737fe640 1 -- 172.21.15.121:0/3824917496 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6740a5b90 msgr2=0x7fa674135d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:51.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.465+0000 7fa6737fe640 1 --2- 172.21.15.121:0/3824917496 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6740a5b90 0x7fa674135d90 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa66400c410 tx=0x7fa66400c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:51.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.465+0000 7fa670ff9640 10 client.?.objecter ms_handle_connect 0x7fa6740a5b90 2024-09-17T18:37:51.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.465+0000 7fa670ff9640 10 client.?.objecter resend_mon_ops 2024-09-17T18:37:51.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.466+0000 7fa670ff9640 1 -- 172.21.15.121:0/3824917496 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa664017070 con 0x7fa6740a5b90 2024-09-17T18:37:51.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.466+0000 7fa670ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:51.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.466+0000 7fa670ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T18:37:51.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.466+0000 7fa670ff9640 10 monclient: dump: 2024-09-17T18:37:51.477 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:51.477 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:51.477 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:51.477 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:51.477 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:51.477 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:51.477 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:51.477 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:51.477 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:51.478 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:51.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.466+0000 7fa670ff9640 10 monclient: _finish_auth 0 2024-09-17T18:37:51.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.466+0000 7fa670ff9640 10 monclient: _check_auth_tickets 2024-09-17T18:37:51.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.466+0000 7fa670ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:51.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.466+0000 7fa670ff9640 1 -- 172.21.15.121:0/3824917496 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6640070d0 con 0x7fa6740a5b90 2024-09-17T18:37:51.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.466+0000 7fa679df8640 5 monclient: authenticate success, global_id 64682 2024-09-17T18:37:51.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.466+0000 7fa679df8640 10 monclient: _renew_subs 2024-09-17T18:37:51.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.466+0000 7fa679df8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:37:51.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.466+0000 7fa679df8640 1 -- 172.21.15.121:0/3824917496 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6740a2380 con 0x7fa6740a5b90 2024-09-17T18:37:51.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.466+0000 7fa670ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:51.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.466+0000 7fa670ff9640 1 -- 172.21.15.121:0/3824917496 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6640074c0 con 0x7fa6740a5b90 2024-09-17T18:37:51.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.466+0000 7fa670ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:51.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.466+0000 7fa670ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T18:37:51.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.466+0000 7fa670ff9640 10 monclient: dump: 2024-09-17T18:37:51.478 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:51.479 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:51.479 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:51.479 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:51.479 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:51.479 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:51.479 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:51.479 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:51.479 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:51.479 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:51.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.466+0000 7fa6727fc640 4 set_mon_vals no callback set 2024-09-17T18:37:51.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.466+0000 7fa679df8640 10 client.64682.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-17T18:37:51.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.466+0000 7fa679df8640 10 monclient: _renew_subs 2024-09-17T18:37:51.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.466+0000 7fa679df8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:37:51.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.466+0000 7fa679df8640 1 -- 172.21.15.121:0/3824917496 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa67413d750 con 0x7fa6740a5b90 2024-09-17T18:37:51.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.466+0000 7fa679df8640 1 librados: init done 2024-09-17T18:37:51.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.466+0000 7fa670ff9640 1 -- 172.21.15.121:0/3824917496 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fa66401a070 con 0x7fa6740a5b90 2024-09-17T18:37:51.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.466+0000 7fa679df8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:51.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.467+0000 7fa679df8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:51.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.467+0000 7fa679df8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:37:51.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.467+0000 7fa679df8640 1 -- 172.21.15.121:0/3824917496 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6740a5f70 con 0x7fa6740a5b90 2024-09-17T18:37:51.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.467+0000 7fa670ff9640 1 --2- 172.21.15.121:0/3824917496 >> v2:172.21.15.173:6812/2423786037 conn(0x7fa654040170 0x7fa654042630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:51.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.467+0000 7fa670ff9640 1 -- 172.21.15.121:0/3824917496 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(679..679 src has 247..679) ==== 4685+0+0 (secure 0 0 0) 0x7fa6640528c0 con 0x7fa6740a5b90 2024-09-17T18:37:51.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.467+0000 7fa670ff9640 10 client.64682.objecter ms_dispatch 0x7fa674000b90 osd_map(679..679 src has 247..679) 2024-09-17T18:37:51.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.467+0000 7fa670ff9640 3 client.64682.objecter handle_osd_map got epochs [679,679] > 0 2024-09-17T18:37:51.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.467+0000 7fa670ff9640 3 client.64682.objecter handle_osd_map decoding full epoch 679 2024-09-17T18:37:51.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.467+0000 7fa670ff9640 20 client.64682.objecter dump_active .. 0 homeless 2024-09-17T18:37:51.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.467+0000 7fa66bfff640 1 --2- 172.21.15.121:0/3824917496 >> v2:172.21.15.173:6812/2423786037 conn(0x7fa654040170 0x7fa654042630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:51.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.467+0000 7fa66bfff640 10 monclient: get_auth_request con 0x7fa654040170 auth_method 0 2024-09-17T18:37:51.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.469+0000 7fa670ff9640 10 client.64682.objecter ms_handle_connect 0x7fa654040170 2024-09-17T18:37:51.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.469+0000 7fa66bfff640 1 --2- 172.21.15.121:0/3824917496 >> v2:172.21.15.173:6812/2423786037 conn(0x7fa654040170 0x7fa654042630 secure :-1 s=READY pgs=4114 cs=0 l=1 rev1=1 crypto rx=0x7fa65c000c00 tx=0x7fa65c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:51.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.471+0000 7fa670ff9640 1 -- 172.21.15.121:0/3824917496 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fa664051320 con 0x7fa6740a5b90 2024-09-17T18:37:51.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.471+0000 7fa670ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T18:37:51.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.471+0000 7fa670ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T18:37:51.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.644+0000 7fa679df8640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-17T18:37:51.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.644+0000 7fa679df8640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-17T18:37:51.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.644+0000 7fa679df8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.121:3301/0 2024-09-17T18:37:51.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.644+0000 7fa679df8640 1 -- 172.21.15.121:0/3824917496 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7fa6740a6be0 con 0x7fa6740a5b90 2024-09-17T18:37:51.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa6737fe640 1 -- 172.21.15.121:0/3824917496 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6740a5b90 msgr2=0x7fa674135d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T18:37:51.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa6737fe640 1 -- 172.21.15.121:0/3824917496 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6740a5b90 msgr2=0x7fa674135d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T18:37:51.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa6737fe640 1 --2- 172.21.15.121:0/3824917496 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6740a5b90 0x7fa674135d90 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa66400c410 tx=0x7fa66400c8d0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T18:37:51.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa6737fe640 1 --2- 172.21.15.121:0/3824917496 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6740a5b90 0x7fa674135d90 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa66400c410 tx=0x7fa66400c8d0 comp rx=0 tx=0).stop 2024-09-17T18:37:51.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa670ff9640 10 monclient: ms_handle_reset current mon [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:51.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa670ff9640 10 monclient: _reopen_session rank -1 2024-09-17T18:37:51.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa670ff9640 1 -- 172.21.15.121:0/3824917496 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6740a5b90 msgr2=0x7fa674135d90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T18:37:51.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa670ff9640 1 --2- 172.21.15.121:0/3824917496 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6740a5b90 0x7fa674135d90 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:51.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa670ff9640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T18:37:51.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa670ff9640 1 --2- 172.21.15.121:0/3824917496 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6740a6680 0x7fa654049480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:51.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa670ff9640 10 monclient(hunting): picked mon.a con 0x7fa6740a6680 addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-17T18:37:51.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa670ff9640 1 --2- 172.21.15.121:0/3824917496 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6740a7090 0x7fa65404bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:51.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa670ff9640 10 monclient(hunting): picked mon.b con 0x7fa6740a7090 addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T18:37:51.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa670ff9640 1 --2- 172.21.15.121:0/3824917496 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa65404c5e0 0x7fa65404ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T18:37:51.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa670ff9640 10 monclient(hunting): picked mon.c con 0x7fa65404c5e0 addr [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-09-17T18:37:51.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa670ff9640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:51.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa670ff9640 1 -- 172.21.15.121:0/3824917496 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa65404f570 con 0x7fa6740a6680 2024-09-17T18:37:51.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa670ff9640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:51.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa670ff9640 1 -- 172.21.15.121:0/3824917496 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa65404f6f0 con 0x7fa6740a7090 2024-09-17T18:37:51.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa670ff9640 10 monclient(hunting): start opening mon connection 2024-09-17T18:37:51.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa670ff9640 1 -- 172.21.15.121:0/3824917496 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa65404f8c0 con 0x7fa65404c5e0 2024-09-17T18:37:51.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa670ff9640 10 monclient(hunting): _renew_subs 2024-09-17T18:37:51.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa6737fe640 1 --2- 172.21.15.121:0/3824917496 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6740a6680 0x7fa654049480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:51.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa6737fe640 10 monclient(hunting): get_auth_request con 0x7fa6740a6680 auth_method 0 2024-09-17T18:37:51.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa6737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:51.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa6737fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:51.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa6737fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:37:51.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa6737fe640 1 --2- 172.21.15.121:0/3824917496 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa65404c5e0 0x7fa65404ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:51.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa6737fe640 10 monclient(hunting): get_auth_request con 0x7fa65404c5e0 auth_method 0 2024-09-17T18:37:51.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa6737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:51.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa6737fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:51.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa6737fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:37:51.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa6737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:51.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa6737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:51.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa673fff640 1 --2- 172.21.15.121:0/3824917496 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6740a7090 0x7fa65404bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T18:37:51.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa6737fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:37:51.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa673fff640 10 monclient(hunting): get_auth_request con 0x7fa6740a7090 auth_method 0 2024-09-17T18:37:51.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa673fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T18:37:51.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa673fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T18:37:51.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa673fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T18:37:51.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa6737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:51.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa6737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:51.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa6737fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:37:51.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa673fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T18:37:51.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa673fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T18:37:51.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa673fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T18:37:51.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa6737fe640 10 monclient(hunting): handle_auth_done global_id 64682 payload 293 2024-09-17T18:37:51.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa6737fe640 1 -- 172.21.15.121:0/3824917496 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa65404c5e0 msgr2=0x7fa65404ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:51.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa6737fe640 1 --2- 172.21.15.121:0/3824917496 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa65404c5e0 0x7fa65404ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:51.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa6737fe640 1 -- 172.21.15.121:0/3824917496 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6740a7090 msgr2=0x7fa65404bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:51.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa6737fe640 1 --2- 172.21.15.121:0/3824917496 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6740a7090 0x7fa65404bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:51.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa6737fe640 10 monclient: _finish_hunting 0 2024-09-17T18:37:51.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa6737fe640 1 monclient: found mon.a 2024-09-17T18:37:51.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa6737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T18:37:51.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa6737fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:51.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa6737fe640 1 -- 172.21.15.121:0/3824917496 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) -- 0x7fa65404fb20 con 0x7fa6740a6680 2024-09-17T18:37:51.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa6737fe640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-17T18:37:51.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa6737fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.121:3300/0 2024-09-17T18:37:51.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa6737fe640 1 -- 172.21.15.121:0/3824917496 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7fa6740a6be0 con 0x7fa6740a6680 2024-09-17T18:37:51.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa6737fe640 10 monclient: _finish_auth 0 2024-09-17T18:37:51.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa6737fe640 10 monclient: _check_auth_tickets 2024-09-17T18:37:51.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.645+0000 7fa6737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T18:37:51.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.647+0000 7fa670ff9640 10 client.64682.objecter ms_handle_connect 0x7fa6740a6680 2024-09-17T18:37:51.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.647+0000 7fa6737fe640 1 -- 172.21.15.121:0/3824917496 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6740a6680 msgr2=0x7fa654049480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T18:37:51.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.647+0000 7fa670ff9640 10 client.64682.objecter resend_mon_ops 2024-09-17T18:37:51.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.647+0000 7fa6737fe640 1 --2- 172.21.15.121:0/3824917496 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6740a6680 0x7fa654049480 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa664051270 tx=0x7fa66401bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T18:37:51.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.647+0000 7fa670ff9640 1 -- 172.21.15.121:0/3824917496 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa66400e460 con 0x7fa6740a6680 2024-09-17T18:37:51.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.647+0000 7fa670ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T18:37:51.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.647+0000 7fa670ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T18:37:51.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.647+0000 7fa670ff9640 10 monclient: dump: 2024-09-17T18:37:51.655 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-17T18:37:51.655 INFO:teuthology.orchestra.run.smithi121.stderr:fsid a03d5f97-ea9b-4e22-a0d6-b2fe3eb06a9b 2024-09-17T18:37:51.655 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:51.655 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-17T18:18:20.001795+0000 2024-09-17T18:37:51.655 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-17T18:37:51.655 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-17T18:37:51.655 INFO:teuthology.orchestra.run.smithi121.stderr:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-09-17T18:37:51.655 INFO:teuthology.orchestra.run.smithi121.stderr:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-09-17T18:37:51.655 INFO:teuthology.orchestra.run.smithi121.stderr:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-09-17T18:37:51.655 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:51.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.648+0000 7fa670ff9640 1 -- 172.21.15.121:0/3824917496 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa664058080 con 0x7fa6740a6680 2024-09-17T18:37:51.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.648+0000 7fa670ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T18:37:51.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.648+0000 7fa6727fc640 4 set_mon_vals no callback set 2024-09-17T18:37:51.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.648+0000 7fa670ff9640 1 -- 172.21.15.121:0/3824917496 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mgrmap(e 5) ==== 50996+0+0 (secure 0 0 0) 0x7fa664010370 con 0x7fa6740a6680 2024-09-17T18:37:51.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.919+0000 7fa670ff9640 1 -- 172.21.15.121:0/3824917496 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v6)=0 v6) ==== 123+0+0 (secure 0 0 0) 0x7fa664012d80 con 0x7fa6740a6680 2024-09-17T18:37:51.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.919+0000 7fa670ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-17T18:37:51.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.919+0000 7fa670ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T18:37:51.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.919+0000 7fa670ff9640 1 -- 172.21.15.121:0/3824917496 <== mon.0 v2:172.21.15.121:3300/0 5 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fa6640549e0 con 0x7fa6740a6680 2024-09-17T18:37:51.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.919+0000 7fa670ff9640 10 monclient: handle_config config(1 keys) 2024-09-17T18:37:51.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.919+0000 7fa6727fc640 4 set_mon_vals no callback set 2024-09-17T18:37:51.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.919+0000 7fa6727fc640 10 set_mon_vals mon_health_to_clog = false 2024-09-17T18:37:51.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.921+0000 7fa679df8640 10 librados: watch_flush enter 2024-09-17T18:37:51.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.921+0000 7fa679df8640 10 librados: watch_flush exit 2024-09-17T18:37:51.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.921+0000 7fa679df8640 20 client.64682.objecter shutdown clearing up homeless session... 2024-09-17T18:37:51.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.921+0000 7fa679df8640 10 client.64682.objecter successfully canceled tick 2024-09-17T18:37:51.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.922+0000 7fa679df8640 1 -- 172.21.15.121:0/3824917496 >> v2:172.21.15.173:6812/2423786037 conn(0x7fa654040170 msgr2=0x7fa654042630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:51.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.922+0000 7fa679df8640 1 --2- 172.21.15.121:0/3824917496 >> v2:172.21.15.173:6812/2423786037 conn(0x7fa654040170 0x7fa654042630 secure :-1 s=READY pgs=4114 cs=0 l=1 rev1=1 crypto rx=0x7fa65c000c00 tx=0x7fa65c02e000 comp rx=0 tx=0).stop 2024-09-17T18:37:51.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.922+0000 7fa679df8640 10 monclient: shutdown 2024-09-17T18:37:51.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.922+0000 7fa679df8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T18:37:51.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.922+0000 7fa679df8640 1 -- 172.21.15.121:0/3824917496 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6740a6680 msgr2=0x7fa654049480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T18:37:51.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.922+0000 7fa679df8640 1 --2- 172.21.15.121:0/3824917496 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6740a6680 0x7fa654049480 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa664051270 tx=0x7fa66401bd60 comp rx=0 tx=0).stop 2024-09-17T18:37:51.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.922+0000 7fa679df8640 1 -- 172.21.15.121:0/3824917496 shutdown_connections 2024-09-17T18:37:51.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.922+0000 7fa679df8640 1 --2- 172.21.15.121:0/3824917496 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6740a7090 0x7fa65404bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:51.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.922+0000 7fa679df8640 1 --2- 172.21.15.121:0/3824917496 >> v2:172.21.15.173:6812/2423786037 conn(0x7fa654040170 0x7fa654042630 unknown :-1 s=CLOSED pgs=4114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:51.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.922+0000 7fa679df8640 1 --2- 172.21.15.121:0/3824917496 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa65404c5e0 0x7fa65404ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:51.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.922+0000 7fa679df8640 1 --2- 172.21.15.121:0/3824917496 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6740a6680 0x7fa654049480 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T18:37:51.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.922+0000 7fa679df8640 1 -- 172.21.15.121:0/3824917496 >> 172.21.15.121:0/3824917496 conn(0x7fa6740a7c90 msgr2=0x7fa674137820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T18:37:51.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.922+0000 7fa679df8640 1 -- 172.21.15.121:0/3824917496 shutdown_connections 2024-09-17T18:37:51.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.922+0000 7fa679df8640 1 -- 172.21.15.121:0/3824917496 wait complete. 2024-09-17T18:37:51.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-17T18:37:51.922+0000 7fa679df8640 1 librados: shutdown 2024-09-17T18:37:51.934 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-17T18:37:51.934 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-17T18:37:51.934 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-17T18:37:51.935 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T18:37:51.977 INFO:tasks.ceph.osd.0:Stopped 2024-09-17T18:37:51.977 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-17T18:37:51.977 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T18:37:52.014 INFO:tasks.ceph.osd.1:Stopped 2024-09-17T18:37:52.015 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-17T18:37:52.015 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T18:37:52.087 INFO:tasks.ceph.osd.2:Stopped 2024-09-17T18:37:52.088 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-17T18:37:52.088 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T18:37:52.121 INFO:tasks.ceph.osd.3:Stopped 2024-09-17T18:37:52.121 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-17T18:37:52.121 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T18:37:52.193 INFO:tasks.ceph.osd.4:Stopped 2024-09-17T18:37:52.193 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-17T18:37:52.193 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T18:37:52.277 INFO:tasks.ceph.osd.5:Stopped 2024-09-17T18:37:52.277 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-17T18:37:52.277 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-17T18:37:52.277 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T18:37:52.307 INFO:tasks.ceph.mgr.x:Stopped 2024-09-17T18:37:52.307 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-17T18:37:52.307 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-17T18:37:52.307 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T18:37:52.318 INFO:tasks.ceph.mon.a:Stopped 2024-09-17T18:37:52.318 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-17T18:37:52.318 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T18:37:52.373 INFO:tasks.ceph.mon.c:Stopped 2024-09-17T18:37:52.373 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-17T18:37:52.373 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T18:37:52.388 INFO:tasks.ceph.mon.b:Stopped 2024-09-17T18:37:52.388 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-17T18:37:52.388 DEBUG:teuthology.orchestra.run.smithi121:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(PG_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v 'slow request' | head -n 1 2024-09-17T18:37:52.447 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi121.front.sepia.ceph.com 2024-09-17T18:37:52.447 DEBUG:teuthology.orchestra.run.smithi121:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-17T18:37:52.636 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi121.front.sepia.ceph.com 2024-09-17T18:37:52.636 DEBUG:teuthology.orchestra.run.smithi121:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-17T18:37:52.734 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi121.front.sepia.ceph.com 2024-09-17T18:37:52.734 DEBUG:teuthology.orchestra.run.smithi121:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-17T18:37:52.790 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi173.front.sepia.ceph.com 2024-09-17T18:37:52.790 DEBUG:teuthology.orchestra.run.smithi173:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-17T18:37:52.928 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi173.front.sepia.ceph.com 2024-09-17T18:37:52.928 DEBUG:teuthology.orchestra.run.smithi173:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-17T18:37:52.999 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi173.front.sepia.ceph.com 2024-09-17T18:37:52.999 DEBUG:teuthology.orchestra.run.smithi173:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-17T18:37:53.056 INFO:tasks.ceph:Archiving mon data... 2024-09-17T18:37:53.057 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909241/data/mon.a.tgz 2024-09-17T18:37:53.057 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2024-09-17T18:37:53.073 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.I5pioGgDBE 2024-09-17T18:37:53.073 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.I5pioGgDBE 2024-09-17T18:37:53.276 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.I5pioGgDBE 2024-09-17T18:37:53.358 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.I5pioGgDBE is 738KB 2024-09-17T18:37:53.458 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.I5pioGgDBE 2024-09-17T18:37:53.473 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909241/data/mon.c.tgz 2024-09-17T18:37:53.473 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2024-09-17T18:37:53.528 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.3SNzESkp5z 2024-09-17T18:37:53.528 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.3SNzESkp5z 2024-09-17T18:37:53.723 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.3SNzESkp5z 2024-09-17T18:37:53.805 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.3SNzESkp5z is 743KB 2024-09-17T18:37:53.905 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.3SNzESkp5z 2024-09-17T18:37:53.920 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909241/data/mon.b.tgz 2024-09-17T18:37:53.920 DEBUG:teuthology.orchestra.run.smithi173:> mktemp 2024-09-17T18:37:53.939 INFO:teuthology.orchestra.run.smithi173.stdout:/tmp/tmp.xTidoK9KIZ 2024-09-17T18:37:53.939 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.xTidoK9KIZ 2024-09-17T18:37:54.158 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0666 /tmp/tmp.xTidoK9KIZ 2024-09-17T18:37:54.240 DEBUG:teuthology.orchestra.remote:smithi173:/tmp/tmp.xTidoK9KIZ is 742KB 2024-09-17T18:37:54.339 DEBUG:teuthology.orchestra.run.smithi173:> rm -fr /tmp/tmp.xTidoK9KIZ 2024-09-17T18:37:54.354 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-17T18:37:54.354 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-17T18:37:54.357 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-17T18:37:54.435 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-17T18:37:54.473 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-17T18:37:54.473 INFO:tasks.ceph:Archiving crash dumps... 2024-09-17T18:37:54.475 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909241/remote/smithi121/crash 2024-09-17T18:37:54.476 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-17T18:37:54.500 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909241/remote/smithi173/crash 2024-09-17T18:37:54.501 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-17T18:37:54.537 INFO:tasks.ceph:Compressing logs... 2024-09-17T18:37:54.537 DEBUG:teuthology.orchestra.run.smithi121:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-17T18:37:54.545 DEBUG:teuthology.orchestra.run.smithi173:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-17T18:37:54.567 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-17T18:37:54.567 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43365.log 2024-09-17T18:37:54.569 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35540.log 2024-09-17T18:37:54.569 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42902.log 2024-09-17T18:37:54.569 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37699.log 2024-09-17T18:37:54.569 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.43365.log: /var/log/ceph/ceph-client.admin.35540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38578.log 2024-09-17T18:37:54.569 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.42902.log: /var/log/ceph/ceph-client.admin.37699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35019.log 2024-09-17T18:37:54.569 INFO:teuthology.orchestra.run.smithi121.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35540.log.gz 2024-09-17T18:37:54.569 INFO:teuthology.orchestra.run.smithi121.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43365.log.gz 2024-09-17T18:37:54.569 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.38578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41774.log 2024-09-17T18:37:54.570 INFO:teuthology.orchestra.run.smithi121.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.42902.log.gz 2024-09-17T18:37:54.570 INFO:teuthology.orchestra.run.smithi121.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37699.log.gz 2024-09-17T18:37:54.570 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40662.log 2024-09-17T18:37:54.570 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.41774.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.38578.log.gz 2024-09-17T18:37:54.570 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40620.log 2024-09-17T18:37:54.570 INFO:teuthology.orchestra.run.smithi121.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35019.log.gz 2024-09-17T18:37:54.579 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37615.log 2024-09-17T18:37:54.579 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.40662.log: 90.8%/var/log/ceph/ceph-client.admin.40620.log: -- replaced with /var/log/ceph/ceph-client.admin.41774.log.gz 2024-09-17T18:37:54.579 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43449.log 2024-09-17T18:37:54.580 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35251.log 2024-09-17T18:37:54.580 INFO:teuthology.orchestra.run.smithi121.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40662.log.gz 87.5% 2024-09-17T18:37:54.580 INFO:teuthology.orchestra.run.smithi121.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40620.log.gz 2024-09-17T18:37:54.580 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.43449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36180.log 2024-09-17T18:37:54.580 INFO:teuthology.orchestra.run.smithi121.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37615.log.gz 2024-09-17T18:37:54.580 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35444.log 2024-09-17T18:37:54.580 INFO:teuthology.orchestra.run.smithi121.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43449.log.gz 2024-09-17T18:37:54.580 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43575.log 2024-09-17T18:37:54.580 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.36180.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35251.log.gz 2024-09-17T18:37:54.580 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40473.log 2024-09-17T18:37:54.580 INFO:teuthology.orchestra.run.smithi121.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36180.log.gz 2024-09-17T18:37:54.580 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.43575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42171.log 2024-09-17T18:37:54.580 INFO:teuthology.orchestra.run.smithi121.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35444.log.gz 2024-09-17T18:37:54.580 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37913.log 2024-09-17T18:37:54.581 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.40473.log: /var/log/ceph/ceph-client.admin.42171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37159.log 2024-09-17T18:37:54.581 INFO:teuthology.orchestra.run.smithi121.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43575.log.gz 2024-09-17T18:37:54.581 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35018.log 2024-09-17T18:37:54.581 INFO:teuthology.orchestra.run.smithi121.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.40473.log.gz 2024-09-17T18:37:54.581 INFO:teuthology.orchestra.run.smithi121.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42171.log.gz 2024-09-17T18:37:54.581 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37531.log 2024-09-17T18:37:54.581 INFO:teuthology.orchestra.run.smithi121.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37913.log.gz 2024-09-17T18:37:54.581 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39265.log 2024-09-17T18:37:54.581 INFO:teuthology.orchestra.run.smithi121.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37159.log.gz 2024-09-17T18:37:54.581 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35424.log 2024-09-17T18:37:54.581 INFO:teuthology.orchestra.run.smithi121.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35018.log.gz 2024-09-17T18:37:54.581 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.39265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42549.log 2024-09-17T18:37:54.581 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35424.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.37531.log.gz 2024-09-17T18:37:54.581 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41478.log 2024-09-17T18:37:54.581 INFO:teuthology.orchestra.run.smithi121.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39265.log.gz 2024-09-17T18:37:54.582 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.42549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43701.log 2024-09-17T18:37:54.582 INFO:teuthology.orchestra.run.smithi121.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35424.log.gz 2024-09-17T18:37:54.582 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.41478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37657.log 2024-09-17T18:37:54.582 INFO:teuthology.orchestra.run.smithi121.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42549.log.gz 2024-09-17T18:37:54.582 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.43701.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-17T18:37:54.582 INFO:teuthology.orchestra.run.smithi121.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41478.log.gz 2024-09-17T18:37:54.582 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37745.log 2024-09-17T18:37:54.582 INFO:teuthology.orchestra.run.smithi121.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43701.log.gz 2024-09-17T18:37:54.582 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38449.log 2024-09-17T18:37:54.582 INFO:teuthology.orchestra.run.smithi121.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37657.log.gz 2024-09-17T18:37:54.582 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39392.log 2024-09-17T18:37:54.582 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.38449.log: gzip 87.9% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.37745.log.gz -- 2024-09-17T18:37:54.582 INFO:teuthology.orchestra.run.smithi121.stderr: /var/log/ceph/ceph-client.admin.42297.log 2024-09-17T18:37:54.582 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35802.log 2024-09-17T18:37:54.582 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.39392.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38449.log.gz 2024-09-17T18:37:54.583 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.42297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31886.log 2024-09-17T18:37:54.583 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35802.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39392.log.gz 2024-09-17T18:37:54.583 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38704.log 2024-09-17T18:37:54.583 INFO:teuthology.orchestra.run.smithi121.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42297.log.gz 2024-09-17T18:37:54.583 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.31886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43617.log 2024-09-17T18:37:54.583 INFO:teuthology.orchestra.run.smithi121.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35802.log.gz 2024-09-17T18:37:54.583 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.38704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37997.log 2024-09-17T18:37:54.583 INFO:teuthology.orchestra.run.smithi121.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.31886.log.gz 2024-09-17T18:37:54.583 INFO:teuthology.orchestra.run.smithi121.stderr:gzip/var/log/ceph/ceph-client.admin.43617.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40557.log 2024-09-17T18:37:54.583 INFO:teuthology.orchestra.run.smithi121.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38704.log.gz 2024-09-17T18:37:54.583 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41308.log 2024-09-17T18:37:54.583 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.40557.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43617.log.gz 2024-09-17T18:37:54.583 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39084.log 2024-09-17T18:37:54.583 INFO:teuthology.orchestra.run.smithi121.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37997.log.gz 2024-09-17T18:37:54.583 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.41308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38081.log 2024-09-17T18:37:54.583 INFO:teuthology.orchestra.run.smithi121.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.40557.log.gz 2024-09-17T18:37:54.584 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.39084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42423.log 2024-09-17T18:37:54.584 INFO:teuthology.orchestra.run.smithi121.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.41308.log.gz 2024-09-17T18:37:54.584 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.38081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38405.log 2024-09-17T18:37:54.631 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36601.log 2024-09-17T18:37:54.632 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35451.log 2024-09-17T18:37:54.632 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.36601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35008.log 2024-09-17T18:37:54.632 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.35451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35098.log 2024-09-17T18:37:54.632 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.35008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35881.log 2024-09-17T18:37:54.632 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.35098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31980.log 2024-09-17T18:37:54.632 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.35881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36188.log 2024-09-17T18:37:54.632 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.31980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35846.log 2024-09-17T18:37:54.632 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.36188.log: gzip 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35881.log.gz 2024-09-17T18:37:54.632 INFO:teuthology.orchestra.run.smithi173.stderr: -5 88.0% -- replaced with /var/log/ceph/ceph-client.admin.31980.log.gz 2024-09-17T18:37:54.632 INFO:teuthology.orchestra.run.smithi173.stderr: --verbose -- /var/log/ceph/ceph-client.admin.36600.log 2024-09-17T18:37:54.632 INFO:teuthology.orchestra.run.smithi173.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.36601.log.gz 2024-09-17T18:37:54.632 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35650.log 2024-09-17T18:37:54.632 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.35846.log: gzip/var/log/ceph/ceph-client.admin.36600.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.34988.log 2024-09-17T18:37:54.633 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.35650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36884.log 2024-09-17T18:37:54.633 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.34988.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.35098.log.gz 2024-09-17T18:37:54.633 INFO:teuthology.orchestra.run.smithi173.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.35846.log.gz 2024-09-17T18:37:54.633 INFO:teuthology.orchestra.run.smithi173.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.36600.log.gz 2024-09-17T18:37:54.633 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36016.log 2024-09-17T18:37:54.633 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.36884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35135.log 2024-09-17T18:37:54.633 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.36016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36316.log 2024-09-17T18:37:54.633 INFO:teuthology.orchestra.run.smithi173.stderr: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.35451.log.gz 2024-09-17T18:37:54.633 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.35135.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.36016.log.gz 2024-09-17T18:37:54.633 INFO:teuthology.orchestra.run.smithi173.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.36884.log.gz 2024-09-17T18:37:54.633 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35395.log 2024-09-17T18:37:54.633 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.36316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36548.log 2024-09-17T18:37:54.633 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.35395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37082.log 2024-09-17T18:37:54.634 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.36548.log: 90.7% 91.3% -- replaced with /var/log/ceph/ceph-client.admin.36316.log.gz 2024-09-17T18:37:54.634 INFO:teuthology.orchestra.run.smithi173.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36548.log.gz 2024-09-17T18:37:54.634 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.37082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35490.log 2024-09-17T18:37:54.634 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31856.log 2024-09-17T18:37:54.634 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.35490.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.37082.log.gz 2024-09-17T18:37:54.634 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36650.log 2024-09-17T18:37:54.634 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.31856.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.31856.log.gz 2024-09-17T18:37:54.634 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36885.log 2024-09-17T18:37:54.634 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.36650.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.36650.log.gz 2024-09-17T18:37:54.634 INFO:teuthology.orchestra.run.smithi173.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.35490.log.gz 2024-09-17T18:37:54.634 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36333.log 2024-09-17T18:37:54.634 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.36885.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.35395.log.gz 2024-09-17T18:37:54.634 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36386.log 2024-09-17T18:37:54.635 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36367.log 2024-09-17T18:37:54.635 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.36386.log: /var/log/ceph/ceph-client.admin.36333.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.36885.log.gz 2024-09-17T18:37:54.635 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-17T18:37:54.639 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.36367.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.36367.log.gz 2024-09-17T18:37:54.639 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35946.log 2024-09-17T18:37:54.639 INFO:teuthology.orchestra.run.smithi173.stderr: 92.4%/var/log/ceph/ceph-osd.4.log: -- replaced with /var/log/ceph/ceph-client.admin.36188.log.gz 2024-09-17T18:37:54.639 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37083.log 2024-09-17T18:37:54.639 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.35946.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.35946.log.gz 2024-09-17T18:37:54.647 INFO:teuthology.orchestra.run.smithi173.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.36333.log.gz 2024-09-17T18:37:54.647 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35995.log 2024-09-17T18:37:54.647 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.37083.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-17T18:37:54.647 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.35995.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35995.log.gz 2024-09-17T18:37:54.647 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36654.log 2024-09-17T18:37:54.647 INFO:teuthology.orchestra.run.smithi173.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.34988.log.gz 2024-09-17T18:37:54.647 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph.audit.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.37083.log.gz 2024-09-17T18:37:54.661 INFO:teuthology.orchestra.run.smithi173.stderr: 92.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-17T18:37:54.661 INFO:teuthology.orchestra.run.smithi173.stderr:gzip 92.3% -- replaced with /var/log/ceph/ceph-client.admin.35650.log.gz 2024-09-17T18:37:54.661 INFO:teuthology.orchestra.run.smithi173.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.32027.log 2024-09-17T18:37:54.661 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.36654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35480.log 2024-09-17T18:37:54.661 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.32027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36271.log 2024-09-17T18:37:54.661 INFO:teuthology.orchestra.run.smithi173.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36654.log.gz 2024-09-17T18:37:54.661 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.35480.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.32027.log.gz 2024-09-17T18:37:54.661 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37081.log 2024-09-17T18:37:54.661 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.36271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35260.log 2024-09-17T18:37:54.661 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.37081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35903.log 2024-09-17T18:37:54.661 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.35260.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.37081.log.gz 2024-09-17T18:37:54.661 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31816.log 2024-09-17T18:37:54.661 INFO:teuthology.orchestra.run.smithi173.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.35008.log.gz 2024-09-17T18:37:54.661 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.35903.log: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.35480.log.gz 2024-09-17T18:37:54.677 INFO:teuthology.orchestra.run.smithi121.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.39084.log.gz 2024-09-17T18:37:54.677 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.42423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39349.log 2024-09-17T18:37:54.677 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.38405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39042.log 2024-09-17T18:37:54.677 INFO:teuthology.orchestra.run.smithi121.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.38081.log.gz 2024-09-17T18:37:54.677 INFO:teuthology.orchestra.run.smithi121.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.42423.log.gz 2024-09-17T18:37:54.677 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.39349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42255.log 2024-09-17T18:37:54.677 INFO:teuthology.orchestra.run.smithi121.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38405.log.gz 2024-09-17T18:37:54.678 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37201.log 2024-09-17T18:37:54.678 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.39042.log: /var/log/ceph/ceph-client.admin.42255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42860.log 2024-09-17T18:37:54.678 INFO:teuthology.orchestra.run.smithi121.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39349.log.gz 2024-09-17T18:37:54.678 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37029.log 2024-09-17T18:37:54.678 INFO:teuthology.orchestra.run.smithi121.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.39042.log.gz 2024-09-17T18:37:54.678 INFO:teuthology.orchestra.run.smithi121.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42255.log.gz 2024-09-17T18:37:54.678 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.42860.log: gzip -5 --verbose -- 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37201.log.gz 2024-09-17T18:37:54.678 INFO:teuthology.orchestra.run.smithi121.stderr: /var/log/ceph/ceph-client.admin.38536.log 2024-09-17T18:37:54.678 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37029.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42860.log.gz 2024-09-17T18:37:54.678 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42675.log 2024-09-17T18:37:54.678 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27274.log 2024-09-17T18:37:54.678 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.38536.log: /var/log/ceph/ceph-client.admin.42675.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.37029.log.gz 2024-09-17T18:37:54.678 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39434.log 2024-09-17T18:37:54.678 INFO:teuthology.orchestra.run.smithi121.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.38536.log.gz 2024-09-17T18:37:54.678 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph.tmp-client.admin.27274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42465.log 2024-09-17T18:37:54.679 INFO:teuthology.orchestra.run.smithi121.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27274.log.gz 2024-09-17T18:37:54.679 INFO:teuthology.orchestra.run.smithi121.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.42675.log.gz 2024-09-17T18:37:54.679 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.39434.log: /var/log/ceph/ceph-client.admin.42045.log 2024-09-17T18:37:54.679 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.42465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39772.log 2024-09-17T18:37:54.679 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.42045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36390.log 2024-09-17T18:37:54.679 INFO:teuthology.orchestra.run.smithi121.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42465.log.gz 2024-09-17T18:37:54.679 INFO:teuthology.orchestra.run.smithi121.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.39434.log.gz 2024-09-17T18:37:54.679 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42002.log 2024-09-17T18:37:54.679 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.39772.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42045.log.gz 2024-09-17T18:37:54.679 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.36390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43827.log 2024-09-17T18:37:54.679 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.42002.log: gzip -5 --verbose -- 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36390.log.gz 2024-09-17T18:37:54.679 INFO:teuthology.orchestra.run.smithi121.stderr: /var/log/ceph/ceph-client.admin.35015.log 2024-09-17T18:37:54.679 INFO:teuthology.orchestra.run.smithi121.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.39772.log.gz 2024-09-17T18:37:54.679 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.43827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40998.log 2024-09-17T18:37:54.679 INFO:teuthology.orchestra.run.smithi121.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42002.log.gz 2024-09-17T18:37:54.680 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35015.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43827.log.gz 2024-09-17T18:37:54.680 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40050.log 2024-09-17T18:37:54.680 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41521.log 2024-09-17T18:37:54.680 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.40998.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35015.log.gz 2024-09-17T18:37:54.680 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.40050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40221.log 2024-09-17T18:37:54.680 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.41521.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.40998.log.gz 2024-09-17T18:37:54.680 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38169.log 2024-09-17T18:37:54.680 INFO:teuthology.orchestra.run.smithi121.stderr: 89.8%/var/log/ceph/ceph-client.admin.40221.log: -- replaced with /var/log/ceph/ceph-client.admin.40050.log.gz 2024-09-17T18:37:54.680 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38620.log 2024-09-17T18:37:54.680 INFO:teuthology.orchestra.run.smithi121.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41521.log.gz 2024-09-17T18:37:54.680 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.38169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40746.log 2024-09-17T18:37:54.680 INFO:teuthology.orchestra.run.smithi121.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.40221.log.gz 2024-09-17T18:37:54.680 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.38620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35544.log 2024-09-17T18:37:54.680 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.40746.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38169.log.gz 2024-09-17T18:37:54.680 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35272.log 2024-09-17T18:37:54.681 INFO:teuthology.orchestra.run.smithi121.stderr: 89.5%/var/log/ceph/ceph-client.admin.35544.log: -- replaced with /var/log/ceph/ceph-client.admin.38620.log.gz 2024-09-17T18:37:54.681 INFO:teuthology.orchestra.run.smithi121.stderr:gzip 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40746.log.gz 2024-09-17T18:37:54.681 INFO:teuthology.orchestra.run.smithi121.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.43028.log 2024-09-17T18:37:54.681 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38746.log 2024-09-17T18:37:54.681 INFO:teuthology.orchestra.run.smithi121.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35544.log.gz 2024-09-17T18:37:54.681 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.43028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37245.log 2024-09-17T18:37:54.681 INFO:teuthology.orchestra.run.smithi121.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35272.log.gz 2024-09-17T18:37:54.681 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.38746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36943.log 2024-09-17T18:37:54.681 INFO:teuthology.orchestra.run.smithi121.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43028.log.gz 2024-09-17T18:37:54.681 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37245.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-17T18:37:54.681 INFO:teuthology.orchestra.run.smithi121.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.38746.log.gz 2024-09-17T18:37:54.681 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.36943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38039.log 2024-09-17T18:37:54.681 INFO:teuthology.orchestra.run.smithi121.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37245.log.gz 2024-09-17T18:37:54.681 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35970.log 2024-09-17T18:37:54.681 INFO:teuthology.orchestra.run.smithi121.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36943.log.gz 2024-09-17T18:37:54.682 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.38039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38127.log 2024-09-17T18:37:54.682 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35247.log 2024-09-17T18:37:54.682 INFO:teuthology.orchestra.run.smithi121.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38039.log.gz 2024-09-17T18:37:54.682 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.38127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39307.log 2024-09-17T18:37:54.682 INFO:teuthology.orchestra.run.smithi121.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35970.log.gz 2024-09-17T18:37:54.682 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39604.log 2024-09-17T18:37:54.682 INFO:teuthology.orchestra.run.smithi121.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.38127.log.gz 2024-09-17T18:37:54.682 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.39307.log: gzip -5 --verbose 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35247.log.gz 2024-09-17T18:37:54.682 INFO:teuthology.orchestra.run.smithi121.stderr: -- /var/log/ceph/ceph-client.admin.40830.log 2024-09-17T18:37:54.682 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.39604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40008.log 2024-09-17T18:37:54.682 INFO:teuthology.orchestra.run.smithi121.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39307.log.gz 2024-09-17T18:37:54.682 INFO:teuthology.orchestra.run.smithi121.stderr: 88.5%/var/log/ceph/ceph-client.admin.40830.log: -- replaced with /var/log/ceph/ceph-client.admin.39604.log.gz 2024-09-17T18:37:54.682 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36306.log 2024-09-17T18:37:54.682 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.40008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41563.log 2024-09-17T18:37:54.682 INFO:teuthology.orchestra.run.smithi121.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40830.log.gz 2024-09-17T18:37:54.683 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.36306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36348.log 2024-09-17T18:37:54.683 INFO:teuthology.orchestra.run.smithi121.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40008.log.gz 2024-09-17T18:37:54.683 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.41563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41266.log 2024-09-17T18:37:54.683 INFO:teuthology.orchestra.run.smithi121.stderr: 87.9%/var/log/ceph/ceph-client.admin.36348.log: -- replaced with /var/log/ceph/ceph-client.admin.36306.log.gz 2024-09-17T18:37:54.683 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38662.log 2024-09-17T18:37:54.683 INFO:teuthology.orchestra.run.smithi121.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.41563.log.gz 2024-09-17T18:37:54.683 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.41266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43070.log 2024-09-17T18:37:54.683 INFO:teuthology.orchestra.run.smithi121.stderr:gzip 90.3% -- replaced with /var/log/ceph/ceph-client.admin.36348.log.gz 2024-09-17T18:37:54.683 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.38662.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41859.log 2024-09-17T18:37:54.683 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.43070.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.41266.log.gz 2024-09-17T18:37:54.683 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41434.log 2024-09-17T18:37:54.683 INFO:teuthology.orchestra.run.smithi121.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38662.log.gz 2024-09-17T18:37:54.683 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.41859.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43070.log.gz 2024-09-17T18:37:54.683 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39646.log 2024-09-17T18:37:54.683 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43281.log 2024-09-17T18:37:54.684 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.41434.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41859.log.gz 2024-09-17T18:37:54.684 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.39646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39000.log 2024-09-17T18:37:54.684 INFO:teuthology.orchestra.run.smithi121.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41434.log.gz 2024-09-17T18:37:54.684 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.43281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35249.log 2024-09-17T18:37:54.684 INFO:teuthology.orchestra.run.smithi121.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.39646.log.gz 2024-09-17T18:37:54.684 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.39000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38916.log 2024-09-17T18:37:54.684 INFO:teuthology.orchestra.run.smithi121.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43281.log.gz 2024-09-17T18:37:54.684 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31958.log 2024-09-17T18:37:54.684 INFO:teuthology.orchestra.run.smithi121.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.39000.log.gz 2024-09-17T18:37:54.684 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.38916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41166.log 2024-09-17T18:37:54.684 INFO:teuthology.orchestra.run.smithi121.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35249.log.gz 2024-09-17T18:37:54.684 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.31958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36851.log 2024-09-17T18:37:54.684 INFO:teuthology.orchestra.run.smithi121.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38916.log.gz 2024-09-17T18:37:54.684 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.41166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39730.log 2024-09-17T18:37:54.684 INFO:teuthology.orchestra.run.smithi121.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.31958.log.gz 2024-09-17T18:37:54.685 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.36851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35245.log 2024-09-17T18:37:54.685 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.39730.log: gzip 89.2% -- replaced with /var/log/ceph/ceph-client.admin.41166.log.gz 2024-09-17T18:37:54.685 INFO:teuthology.orchestra.run.smithi121.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.36851.log.gz 2024-09-17T18:37:54.685 INFO:teuthology.orchestra.run.smithi121.stderr: -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-17T18:37:54.685 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34836.log 2024-09-17T18:37:54.685 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-mon.c.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.39730.log.gz 2024-09-17T18:37:54.685 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42591.log 2024-09-17T18:37:54.685 INFO:teuthology.orchestra.run.smithi121.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35245.log.gz 2024-09-17T18:37:54.685 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.34836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43743.log 2024-09-17T18:37:54.685 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5/var/log/ceph/ceph-client.admin.42591.log: --verbose -- /var/log/ceph/ceph-client.admin.35886.log 2024-09-17T18:37:54.685 INFO:teuthology.orchestra.run.smithi121.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34836.log.gz 2024-09-17T18:37:54.685 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.43743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41124.log 2024-09-17T18:37:54.685 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35886.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.42591.log.gz 2024-09-17T18:37:54.685 INFO:teuthology.orchestra.run.smithi121.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43743.log.gz 2024-09-17T18:37:54.685 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34636.log 2024-09-17T18:37:54.686 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39688.log 2024-09-17T18:37:54.686 INFO:teuthology.orchestra.run.smithi121.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35886.log.gz/var/log/ceph/ceph-client.admin.41124.log: 2024-09-17T18:37:54.686 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.34636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37073.log 2024-09-17T18:37:54.686 INFO:teuthology.orchestra.run.smithi121.stderr: 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-17T18:37:54.686 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.39688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39856.log 2024-09-17T18:37:54.686 INFO:teuthology.orchestra.run.smithi121.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34636.log.gz 2024-09-17T18:37:54.686 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39476.log 2024-09-17T18:37:54.686 INFO:teuthology.orchestra.run.smithi121.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.41124.log.gz 2024-09-17T18:37:54.686 INFO:teuthology.orchestra.run.smithi121.stderr: 89.8%/var/log/ceph/ceph-client.admin.39856.log: -- replaced with /var/log/ceph/ceph-client.admin.39688.log.gz 2024-09-17T18:37:54.686 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39171.log 2024-09-17T18:37:54.686 INFO:teuthology.orchestra.run.smithi121.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37073.log.gz 2024-09-17T18:37:54.686 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41040.log 2024-09-17T18:37:54.686 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.39476.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39856.log.gz 2024-09-17T18:37:54.686 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.39171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43112.log 2024-09-17T18:37:54.686 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.41040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39560.log 2024-09-17T18:37:54.687 INFO:teuthology.orchestra.run.smithi121.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39171.log.gz 2024-09-17T18:37:54.687 INFO:teuthology.orchestra.run.smithi121.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.39476.log.gz 2024-09-17T18:37:54.687 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.43112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37329.log 2024-09-17T18:37:54.687 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.39560.log: gzip -5 89.8% -- replaced with /var/log/ceph/ceph-client.admin.41040.log.gz 2024-09-17T18:37:54.687 INFO:teuthology.orchestra.run.smithi121.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40914.log 2024-09-17T18:37:54.687 INFO:teuthology.orchestra.run.smithi121.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43112.log.gz 2024-09-17T18:37:54.687 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37329.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39560.log.gz 2024-09-17T18:37:54.687 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40389.log 2024-09-17T18:37:54.687 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39924.log 2024-09-17T18:37:54.687 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.40914.log: /var/log/ceph/ceph-client.admin.40389.log: 90.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35020.log 2024-09-17T18:37:54.687 INFO:teuthology.orchestra.run.smithi121.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37329.log.gz 2024-09-17T18:37:54.687 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.39924.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40914.log.gz 2024-09-17T18:37:54.687 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37573.log 2024-09-17T18:37:54.687 INFO:teuthology.orchestra.run.smithi121.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40389.log.gz 2024-09-17T18:37:54.687 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35844.log 2024-09-17T18:37:54.687 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35020.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39924.log.gz 2024-09-17T18:37:54.688 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36222.log 2024-09-17T18:37:54.688 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35844.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35020.log.gz 2024-09-17T18:37:54.688 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-17T18:37:54.688 INFO:teuthology.orchestra.run.smithi121.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37573.log.gz 2024-09-17T18:37:54.688 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38494.log 2024-09-17T18:37:54.688 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.36222.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35844.log.gz 2024-09-17T18:37:54.688 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36138.log 2024-09-17T18:37:54.688 INFO:teuthology.orchestra.run.smithi121.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36222.log.gz 2024-09-17T18:37:54.688 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.38494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42986.log 2024-09-17T18:37:54.688 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.36138.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38494.log.gz 2024-09-17T18:37:54.688 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40872.log 2024-09-17T18:37:54.688 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.42986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40788.log 2024-09-17T18:37:54.688 INFO:teuthology.orchestra.run.smithi121.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.36138.log.gz 2024-09-17T18:37:54.688 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.40872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34752.log 2024-09-17T18:37:54.689 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42737.log 2024-09-17T18:37:54.689 INFO:teuthology.orchestra.run.smithi121.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40872.log.gz 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42986.log.gz 2024-09-17T18:37:54.689 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:54.689 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.40788.log: /var/log/ceph/ceph-client.admin.34752.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.40788.log.gz 2024-09-17T18:37:54.689 INFO:teuthology.orchestra.run.smithi121.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34752.log.gz 2024-09-17T18:37:54.689 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43491.log 2024-09-17T18:37:54.689 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.42737.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-17T18:37:54.689 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.43491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36096.log 2024-09-17T18:37:54.689 INFO:teuthology.orchestra.run.smithi121.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42737.log.gz 2024-09-17T18:37:54.689 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43491.log.gz 2024-09-17T18:37:54.689 INFO:teuthology.orchestra.run.smithi121.stderr: -- /var/log/ceph/ceph-client.admin.35676.log 2024-09-17T18:37:54.689 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.36096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37787.log 2024-09-17T18:37:54.689 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35676.log: 89.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34661.log 2024-09-17T18:37:54.689 INFO:teuthology.orchestra.run.smithi121.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36096.log.gz 2024-09-17T18:37:54.689 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37371.log 2024-09-17T18:37:54.689 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37787.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35676.log.gz 2024-09-17T18:37:54.690 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.34661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35928.log 2024-09-17T18:37:54.690 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37371.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37787.log.gz 2024-09-17T18:37:54.690 INFO:teuthology.orchestra.run.smithi121.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34661.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42818.log 2024-09-17T18:37:54.690 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:54.690 INFO:teuthology.orchestra.run.smithi121.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37371.log.gz 2024-09-17T18:37:54.690 INFO:teuthology.orchestra.run.smithi121.stderr:gzip/var/log/ceph/ceph-client.admin.35928.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41605.log 2024-09-17T18:37:54.690 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.42818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43867.log 2024-09-17T18:37:54.690 INFO:teuthology.orchestra.run.smithi121.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35928.log.gz 2024-09-17T18:37:54.690 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.41605.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42818.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43196.log 2024-09-17T18:37:54.690 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:54.690 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.43867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42339.log 2024-09-17T18:37:54.690 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.43196.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.41605.log.gz 2024-09-17T18:37:54.690 INFO:teuthology.orchestra.run.smithi121.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43867.log.gz 2024-09-17T18:37:54.690 INFO:teuthology.orchestra.run.smithi121.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43196.log.gz 2024-09-17T18:37:54.691 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36264.log 2024-09-17T18:37:54.691 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.42339.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-17T18:37:54.691 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.36264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43323.log 2024-09-17T18:37:54.691 INFO:teuthology.orchestra.run.smithi121.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42339.log.gz 2024-09-17T18:37:54.691 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36264.log.gz 2024-09-17T18:37:54.691 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35252.log 2024-09-17T18:37:54.691 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.43323.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43323.log.gz 2024-09-17T18:37:54.691 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34794.log 2024-09-17T18:37:54.691 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40347.log 2024-09-17T18:37:54.691 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.34794.log: 91.3% 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35252.log.gz 2024-09-17T18:37:54.691 INFO:teuthology.orchestra.run.smithi121.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2024-09-17T18:37:54.691 INFO:teuthology.orchestra.run.smithi121.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34794.log.gz 2024-09-17T18:37:54.691 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38874.log 2024-09-17T18:37:54.691 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.40347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40092.log 2024-09-17T18:37:54.691 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.38874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37287.log 2024-09-17T18:37:54.691 INFO:teuthology.orchestra.run.smithi121.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.40347.log.gz 2024-09-17T18:37:54.692 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40431.log 2024-09-17T18:37:54.692 INFO:teuthology.orchestra.run.smithi121.stderr: 87.7%/var/log/ceph/ceph-client.admin.40092.log: -- replaced with /var/log/ceph/ceph-client.admin.38874.log.gz 2024-09-17T18:37:54.692 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37287.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40092.log.gz 2024-09-17T18:37:54.692 INFO:teuthology.orchestra.run.smithi121.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37287.log.gz 2024-09-17T18:37:54.692 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35718.log 2024-09-17T18:37:54.692 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40704.log 2024-09-17T18:37:54.692 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.40431.log: /var/log/ceph/ceph-client.admin.35718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38958.log 2024-09-17T18:37:54.692 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.40704.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.40431.log.gz 2024-09-17T18:37:54.692 INFO:teuthology.orchestra.run.smithi121.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35718.log.gz 2024-09-17T18:37:54.692 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38361.log 2024-09-17T18:37:54.692 INFO:teuthology.orchestra.run.smithi121.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.40704.log.gz 2024-09-17T18:37:54.692 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42507.log 2024-09-17T18:37:54.692 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.38958.log: /var/log/ceph/ceph-client.admin.38361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36987.log 2024-09-17T18:37:54.692 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.42507.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38958.log.gz 2024-09-17T18:37:54.692 INFO:teuthology.orchestra.run.smithi121.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38361.log.gz 2024-09-17T18:37:54.692 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41817.log 2024-09-17T18:37:54.693 INFO:teuthology.orchestra.run.smithi121.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42507.log.gz 2024-09-17T18:37:54.693 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42944.log 2024-09-17T18:37:54.693 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.36987.log: /var/log/ceph/ceph-client.admin.41817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31657.log 2024-09-17T18:37:54.693 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.42944.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36987.log.gz 2024-09-17T18:37:54.693 INFO:teuthology.orchestra.run.smithi121.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41817.log.gz 2024-09-17T18:37:54.693 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42087.log 2024-09-17T18:37:54.693 INFO:teuthology.orchestra.run.smithi121.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42944.log.gz 2024-09-17T18:37:54.693 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40305.log 2024-09-17T18:37:54.693 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.31657.log: /var/log/ceph/ceph-client.admin.42087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38832.log 2024-09-17T18:37:54.693 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.40305.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.42087.log.gz 89.2% -- replaced with /var/log/ceph/ceph-client.admin.31657.log.gz 2024-09-17T18:37:54.693 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:37:54.693 INFO:teuthology.orchestra.run.smithi121.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40305.log.gz 2024-09-17T18:37:54.693 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42633.log 2024-09-17T18:37:54.693 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41392.log 2024-09-17T18:37:54.693 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.38832.log: /var/log/ceph/ceph-client.admin.42633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35760.log 2024-09-17T18:37:54.694 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.41392.log: 89.1% 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38832.log.gz 2024-09-17T18:37:54.694 INFO:teuthology.orchestra.run.smithi121.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42633.log.gz 2024-09-17T18:37:54.694 INFO:teuthology.orchestra.run.smithi121.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41392.log.gz 2024-09-17T18:37:54.694 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39966.log 2024-09-17T18:37:54.694 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42213.log 2024-09-17T18:37:54.694 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35760.log: /var/log/ceph/ceph-client.admin.39966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38280.log 2024-09-17T18:37:54.694 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.42213.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.35760.log.gz 2024-09-17T18:37:54.694 INFO:teuthology.orchestra.run.smithi121.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.39966.log.gz 2024-09-17T18:37:54.694 INFO:teuthology.orchestra.run.smithi121.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42213.log.gz 2024-09-17T18:37:54.694 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35016.log 2024-09-17T18:37:54.694 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.38280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40956.log 2024-09-17T18:37:54.694 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37871.log 2024-09-17T18:37:54.694 INFO:teuthology.orchestra.run.smithi121.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38280.log.gz 2024-09-17T18:37:54.694 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.40956.log: gzip -5 --verbose 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35016.log.gz 2024-09-17T18:37:54.694 INFO:teuthology.orchestra.run.smithi121.stderr: -- /var/log/ceph/ceph-client.admin.41082.log 2024-09-17T18:37:54.695 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43407.log 2024-09-17T18:37:54.695 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.41082.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40956.log.gz 2024-09-17T18:37:54.695 INFO:teuthology.orchestra.run.smithi121.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37871.log.gz 2024-09-17T18:37:54.695 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36432.log 2024-09-17T18:37:54.695 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.43407.log: 90.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38788.log 2024-09-17T18:37:54.695 INFO:teuthology.orchestra.run.smithi121.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41082.log.gz 2024-09-17T18:37:54.695 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40515.log 2024-09-17T18:37:54.695 INFO:teuthology.orchestra.run.smithi121.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43407.log.gz 2024-09-17T18:37:54.695 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.36432.log: /var/log/ceph/ceph-client.admin.38788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35584.log 2024-09-17T18:37:54.695 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.40515.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36432.log.gz 2024-09-17T18:37:54.695 INFO:teuthology.orchestra.run.smithi121.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38788.log.gz 2024-09-17T18:37:54.695 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- 89.7% /var/log/ceph/ceph-client.admin.43533.log 2024-09-17T18:37:54.695 INFO:teuthology.orchestra.run.smithi121.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40515.log.gz 2024-09-17T18:37:54.695 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41689.log 2024-09-17T18:37:54.695 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40137.log 2024-09-17T18:37:54.695 INFO:teuthology.orchestra.run.smithi121.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35584.log.gz 2024-09-17T18:37:54.696 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.43533.log: /var/log/ceph/ceph-client.admin.41689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37955.log 2024-09-17T18:37:54.696 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.40137.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43533.log.gz 2024-09-17T18:37:54.696 INFO:teuthology.orchestra.run.smithi121.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41689.log.gz 2024-09-17T18:37:54.696 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41731.log 2024-09-17T18:37:54.696 INFO:teuthology.orchestra.run.smithi121.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40137.log.gz 2024-09-17T18:37:54.696 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42381.log 2024-09-17T18:37:54.696 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37955.log: /var/log/ceph/ceph-client.admin.41731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40263.log 2024-09-17T18:37:54.696 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.42381.log: 88.4% 87.4% -- replaced with /var/log/ceph/ceph-client.admin.37955.log.gz 2024-09-17T18:37:54.696 INFO:teuthology.orchestra.run.smithi121.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41731.log.gz 2024-09-17T18:37:54.696 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34580.log 2024-09-17T18:37:54.696 INFO:teuthology.orchestra.run.smithi121.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.42381.log.gz 2024-09-17T18:37:54.696 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37413.log 2024-09-17T18:37:54.696 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.40263.log: /var/log/ceph/ceph-client.admin.34580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43239.log 2024-09-17T18:37:54.696 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37413.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.40263.log.gz 2024-09-17T18:37:54.697 INFO:teuthology.orchestra.run.smithi121.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34580.log.gz 2024-09-17T18:37:54.697 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39518.log 2024-09-17T18:37:54.697 INFO:teuthology.orchestra.run.smithi121.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.37413.log.gz 2024-09-17T18:37:54.697 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42129.log 2024-09-17T18:37:54.697 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.43239.log: /var/log/ceph/ceph-client.admin.39518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35485.log 2024-09-17T18:37:54.697 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.42129.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43239.log.gz 2024-09-17T18:37:54.697 INFO:teuthology.orchestra.run.smithi121.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.39518.log.gz 2024-09-17T18:37:54.697 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43154.log 2024-09-17T18:37:54.697 INFO:teuthology.orchestra.run.smithi121.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42129.log.gz 2024-09-17T18:37:54.697 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36012.log 2024-09-17T18:37:54.697 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35485.log: /var/log/ceph/ceph-client.admin.43154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41921.log 2024-09-17T18:37:54.697 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.36012.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35485.log.gz 2024-09-17T18:37:54.697 INFO:teuthology.orchestra.run.smithi121.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43154.log.gz 2024-09-17T18:37:54.697 INFO:teuthology.orchestra.run.smithi121.stderr:gzip 87.8% -5 -- replaced with /var/log/ceph/ceph-client.admin.36012.log.gz --verbose 2024-09-17T18:37:54.698 INFO:teuthology.orchestra.run.smithi121.stderr: -- /var/log/ceph/ceph-client.admin.43659.log 2024-09-17T18:37:54.698 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37115.log 2024-09-17T18:37:54.698 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.41921.log: gzip/var/log/ceph/ceph-client.admin.43659.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.37829.log 2024-09-17T18:37:54.698 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37115.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43659.log.gz 2024-09-17T18:37:54.698 INFO:teuthology.orchestra.run.smithi121.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41921.log.gz 2024-09-17T18:37:54.698 INFO:teuthology.orchestra.run.smithi121.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37115.log.gz 2024-09-17T18:37:54.698 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43785.log 2024-09-17T18:37:54.698 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39814.log 2024-09-17T18:37:54.698 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.43785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34878.log 2024-09-17T18:37:54.698 INFO:teuthology.orchestra.run.smithi121.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.37829.log.gz 2024-09-17T18:37:54.698 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.39814.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43785.log.gz 2024-09-17T18:37:54.698 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40179.log 2024-09-17T18:37:54.698 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.34878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41350.log 2024-09-17T18:37:54.698 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.40179.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.39814.log.gz 2024-09-17T18:37:54.698 INFO:teuthology.orchestra.run.smithi121.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34878.log.gz 2024-09-17T18:37:54.699 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41647.log 2024-09-17T18:37:54.699 INFO:teuthology.orchestra.run.smithi121.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40179.log.gz 2024-09-17T18:37:54.699 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35017.log 2024-09-17T18:37:54.699 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.41350.log: /var/log/ceph/ceph-client.admin.41647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36054.log 2024-09-17T18:37:54.699 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35017.log: 87.9% 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41350.log.gz 2024-09-17T18:37:54.699 INFO:teuthology.orchestra.run.smithi121.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41647.log.gz 2024-09-17T18:37:54.699 INFO:teuthology.orchestra.run.smithi121.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35017.log.gz 2024-09-17T18:37:54.699 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.36054.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36054.log.gz 2024-09-17T18:37:54.708 INFO:teuthology.orchestra.run.smithi173.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.35260.log.gz 2024-09-17T18:37:54.709 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31896.log 2024-09-17T18:37:54.709 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.31816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36652.log 2024-09-17T18:37:54.709 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.31896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36599.log 2024-09-17T18:37:54.709 INFO:teuthology.orchestra.run.smithi173.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.31816.log.gz 2024-09-17T18:37:54.709 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.36652.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.31896.log.gz 2024-09-17T18:37:54.709 INFO:teuthology.orchestra.run.smithi173.stderr: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.36271.log.gz 2024-09-17T18:37:54.709 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35561.log 2024-09-17T18:37:54.709 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.36599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35220.log 2024-09-17T18:37:54.709 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.35561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35384.log 2024-09-17T18:37:54.709 INFO:teuthology.orchestra.run.smithi173.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.36652.log.gz 2024-09-17T18:37:54.709 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.35220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35030.log 2024-09-17T18:37:54.709 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.35384.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.36599.log.gz 2024-09-17T18:37:54.709 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35729.log 2024-09-17T18:37:54.709 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.35030.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.35903.log.gz 2024-09-17T18:37:54.709 INFO:teuthology.orchestra.run.smithi173.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.35384.log.gz 2024-09-17T18:37:54.710 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36549.log 2024-09-17T18:37:54.710 INFO:teuthology.orchestra.run.smithi173.stderr: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.36386.log.gz 2024-09-17T18:37:54.710 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.35729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35844.log 2024-09-17T18:37:54.710 INFO:teuthology.orchestra.run.smithi173.stderr: 91.5%/var/log/ceph/ceph-client.admin.36549.log: -- replaced with /var/log/ceph/ceph-client.admin.35030.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35349.log 2024-09-17T18:37:54.710 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T18:37:54.710 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.35844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36887.log 2024-09-17T18:37:54.710 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.35349.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.35561.log.gz 90.4% -- replaced with /var/log/ceph/ceph-client.admin.35729.log.gz 2024-09-17T18:37:54.710 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T18:37:54.710 INFO:teuthology.orchestra.run.smithi173.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.36549.log.gz 2024-09-17T18:37:54.710 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35070.log 2024-09-17T18:37:54.710 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.36887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36886.log 2024-09-17T18:37:54.710 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35948.log 2024-09-17T18:37:54.710 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.35070.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.35844.log.gz 2024-09-17T18:37:54.710 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.36886.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.35349.log.gz 2024-09-17T18:37:54.710 INFO:teuthology.orchestra.run.smithi173.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.35070.log.gz 2024-09-17T18:37:54.711 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35907.log 2024-09-17T18:37:54.711 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.35948.log: gzip -5 --verbose -- 90.9% -- replaced with /var/log/ceph/ceph-client.admin.36887.log.gz 2024-09-17T18:37:54.711 INFO:teuthology.orchestra.run.smithi173.stderr: /var/log/ceph/ceph-client.admin.35110.log 2024-09-17T18:37:54.711 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.35907.log: gzip -5 88.2% --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-17T18:37:54.711 INFO:teuthology.orchestra.run.smithi173.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35948.log.gz 2024-09-17T18:37:54.711 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.35110.log: gzip -5 --verbose -- 90.4% -- replaced with /var/log/ceph/ceph-client.admin.36886.log.gz 2024-09-17T18:37:54.711 INFO:teuthology.orchestra.run.smithi173.stderr: /var/log/ceph/ceph.log 2024-09-17T18:37:54.711 INFO:teuthology.orchestra.run.smithi173.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35907.log.gz 2024-09-17T18:37:54.711 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36550.log 2024-09-17T18:37:54.711 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-17T18:37:54.711 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.36550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36361.log 2024-09-17T18:37:54.711 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-mgr.x.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.35110.log.gz 2024-09-17T18:37:54.711 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36883.log 2024-09-17T18:37:54.711 INFO:teuthology.orchestra.run.smithi173.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.36550.log.gz 2024-09-17T18:37:54.712 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.36361.log: 87.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-17T18:37:54.712 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-17T18:37:54.712 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.36883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35153.log 2024-09-17T18:37:54.712 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-mon.b.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.36883.log.gz 2024-09-17T18:37:54.712 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36651.log 2024-09-17T18:37:54.712 INFO:teuthology.orchestra.run.smithi173.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.36361.log.gz 2024-09-17T18:37:54.712 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.35153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35536.log 2024-09-17T18:37:54.712 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.36651.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.36651.log.gz 2024-09-17T18:37:54.712 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31936.log 2024-09-17T18:37:54.712 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.35536.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.35536.log.gz 2024-09-17T18:37:54.712 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36022.log 2024-09-17T18:37:54.712 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.31936.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.31936.log.gz 2024-09-17T18:37:54.712 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36254.log 2024-09-17T18:37:54.713 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.36022.log: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.36022.log.gz 2024-09-17T18:37:54.713 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36653.log 2024-09-17T18:37:54.724 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.36254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35906.log 2024-09-17T18:37:54.727 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.36653.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.36653.log.gz 2024-09-17T18:37:54.738 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35511.log 2024-09-17T18:37:54.741 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.35906.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.35906.log.gz 2024-09-17T18:37:54.744 INFO:teuthology.orchestra.run.smithi173.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.36254.log.gz 2024-09-17T18:37:54.744 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35767.log 2024-09-17T18:37:54.747 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.35511.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.35511.log.gz 2024-09-17T18:37:54.747 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35399.log 2024-09-17T18:37:54.749 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.35767.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35767.log.gz 2024-09-17T18:37:54.749 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 94.7% -- replaced with /var/log/ceph/ceph-client.admin.35220.log.gz 2024-09-17T18:37:54.749 INFO:teuthology.orchestra.run.smithi173.stderr: --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-17T18:37:54.749 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.35399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35046.log 2024-09-17T18:37:54.753 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-osd.3.log: 93.3% -- replaced with /var/log/ceph/ceph-client.admin.35153.log.gz 2024-09-17T18:37:54.757 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35720.log 2024-09-17T18:37:54.773 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.35046.log: /var/log/ceph/ceph-client.admin.35720.log: 93.3% -- replaced with /var/log/ceph/ceph-client.admin.35046.log.gz 2024-09-17T18:37:54.776 INFO:teuthology.orchestra.run.smithi173.stderr: 93.5% -- replaced with /var/log/ceph/ceph-client.admin.35720.log.gz 2024-09-17T18:37:54.990 INFO:teuthology.orchestra.run.smithi121.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-17T18:37:55.126 INFO:teuthology.orchestra.run.smithi173.stderr: 92.0% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-17T18:37:55.204 INFO:teuthology.orchestra.run.smithi173.stderr: 93.4% -- replaced with /var/log/ceph/ceph-client.admin.35399.log.gz 2024-09-17T18:37:55.962 INFO:teuthology.orchestra.run.smithi173.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.35135.log.gz 2024-09-17T18:37:55.977 INFO:teuthology.orchestra.run.smithi121.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-17T18:37:56.173 INFO:teuthology.orchestra.run.smithi173.stderr: 92.0% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-17T18:38:00.819 INFO:teuthology.orchestra.run.smithi173.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-17T18:38:01.602 INFO:teuthology.orchestra.run.smithi173.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-17T18:38:01.721 INFO:teuthology.orchestra.run.smithi121.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-17T18:38:01.946 INFO:teuthology.orchestra.run.smithi121.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-17T18:38:01.947 INFO:teuthology.orchestra.run.smithi173.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-17T18:38:01.949 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T18:38:01.949 INFO:teuthology.orchestra.run.smithi173.stderr:real 0m7.351s 2024-09-17T18:38:01.949 INFO:teuthology.orchestra.run.smithi173.stderr:user 0m23.824s 2024-09-17T18:38:01.949 INFO:teuthology.orchestra.run.smithi173.stderr:sys 0m0.824s 2024-09-17T18:38:02.657 INFO:teuthology.orchestra.run.smithi121.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-17T18:38:02.659 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-17T18:38:02.659 INFO:teuthology.orchestra.run.smithi121.stderr:real 0m8.102s 2024-09-17T18:38:02.659 INFO:teuthology.orchestra.run.smithi121.stderr:user 0m23.715s 2024-09-17T18:38:02.659 INFO:teuthology.orchestra.run.smithi121.stderr:sys 0m0.873s 2024-09-17T18:38:02.659 INFO:tasks.ceph:Archiving logs... 2024-09-17T18:38:02.660 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909241/remote/smithi121/log 2024-09-17T18:38:02.661 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-17T18:38:06.494 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909241/remote/smithi173/log 2024-09-17T18:38:06.495 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-17T18:38:10.080 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-17T18:38:10.092 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-17T18:38:10.092 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-17T18:38:10.095 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-17T18:38:10.179 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-17T18:38:10.188 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-17T18:38:10.188 DEBUG:teuthology.orchestra.run.smithi121:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T18:38:10.191 DEBUG:teuthology.orchestra.run.smithi173:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T18:38:10.205 INFO:teuthology.orchestra.run.smithi121.stderr:bash: line 1: ntpq: command not found 2024-09-17T18:38:10.206 INFO:teuthology.orchestra.run.smithi173.stderr:bash: line 1: ntpq: command not found 2024-09-17T18:38:10.209 INFO:teuthology.orchestra.run.smithi121.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T18:38:10.209 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================== 2024-09-17T18:38:10.210 INFO:teuthology.orchestra.run.smithi121.stdout:^* hv01.front.sepia.ceph.com 3 6 377 26 -77us[ -76us] +/- 59ms 2024-09-17T18:38:10.210 INFO:teuthology.orchestra.run.smithi121.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 28 +397us[ +398us] +/- 76ms 2024-09-17T18:38:10.210 INFO:teuthology.orchestra.run.smithi121.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 29 -298us[ -297us] +/- 81ms 2024-09-17T18:38:10.210 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T18:38:10.211 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T18:38:10.211 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-09-17T18:38:10.212 INFO:teuthology.orchestra.run.smithi173.stdout:^* hv01.front.sepia.ceph.com 3 6 377 27 -71us[ -66us] +/- 59ms 2024-09-17T18:38:10.212 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 28 +395us[ +399us] +/- 76ms 2024-09-17T18:38:10.212 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 28 -301us[ -296us] +/- 81ms 2024-09-17T18:38:10.212 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T18:38:10.212 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-17T18:38:10.224 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-17T18:38:10.225 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-17T18:38:10.240 DEBUG:teuthology.orchestra.run.smithi121:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-17T18:38:10.320 DEBUG:teuthology.orchestra.run.smithi173:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-17T18:38:10.385 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T18:38:10.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T18:38:10.415 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T18:38:10.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T18:38:10.443 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-17T18:38:10.466 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-17T18:38:10.544 INFO:teuthology.task.internal:Duration was 1554.541678 seconds 2024-09-17T18:38:10.544 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-17T18:38:10.557 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-17T18:38:10.557 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-17T18:38:10.560 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-17T18:38:10.598 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T18:38:10.600 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T18:38:10.848 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-17T18:38:10.848 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi121.front.sepia.ceph.com 2024-09-17T18:38:10.848 DEBUG:teuthology.orchestra.run.smithi121:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-17T18:38:10.908 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi173.front.sepia.ceph.com 2024-09-17T18:38:10.908 DEBUG:teuthology.orchestra.run.smithi173:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-17T18:38:10.943 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-17T18:38:10.943 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T18:38:10.950 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T18:38:11.009 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-17T18:38:11.010 DEBUG:teuthology.orchestra.run.smithi121:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-17T18:38:11.017 DEBUG:teuthology.orchestra.run.smithi173:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-17T18:38:11.150 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-17T18:38:11.176 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-17T18:38:11.177 DEBUG:teuthology.orchestra.run.smithi121:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-17T18:38:11.201 DEBUG:teuthology.orchestra.run.smithi173:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-17T18:38:11.227 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-17T18:38:11.241 DEBUG:teuthology.orchestra.run.smithi121:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:38:11.244 DEBUG:teuthology.orchestra.run.smithi173:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:38:11.264 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = core 2024-09-17T18:38:11.292 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = core 2024-09-17T18:38:11.303 DEBUG:teuthology.orchestra.run.smithi121:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:38:11.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T18:38:11.340 DEBUG:teuthology.orchestra.run.smithi173:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:38:11.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T18:38:11.357 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-17T18:38:11.368 INFO:teuthology.task.internal:Transferring archived files... 2024-09-17T18:38:11.368 DEBUG:teuthology.misc:Transferring archived files from smithi121:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909241/remote/smithi121 2024-09-17T18:38:11.369 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-17T18:38:11.414 DEBUG:teuthology.misc:Transferring archived files from smithi173:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909241/remote/smithi173 2024-09-17T18:38:11.414 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-17T18:38:11.466 INFO:teuthology.task.internal:Removing archive directory... 2024-09-17T18:38:11.466 DEBUG:teuthology.orchestra.run.smithi121:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-17T18:38:11.468 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-17T18:38:11.498 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-17T18:38:11.508 INFO:teuthology.task.internal:Not uploading archives. 2024-09-17T18:38:11.508 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-17T18:38:11.518 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-17T18:38:11.518 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-17T18:38:11.524 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-17T18:38:11.536 INFO:teuthology.orchestra.run.smithi121.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 17 18:38 /home/ubuntu/cephtest 2024-09-17T18:38:11.552 INFO:teuthology.orchestra.run.smithi173.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 17 18:38 /home/ubuntu/cephtest 2024-09-17T18:38:11.553 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-17T18:38:11.563 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-17T18:38:11.594 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{centos_latest} thrashers/force-sync-many workloads/rados_api_tests} duration: 1554.5416781902313 flavor: default owner: scheduled_skanta@teuthology success: true 2024-09-17T18:38:11.595 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-17T18:38:11.671 INFO:teuthology.run:pass